#This fork I've forked this repo to work on ASL related coding for an accessibility based app.
Links I'm exploring: https://devpost.com/software/search?query=asl
INTRODUCTION
We are a group of female Deaf students who recently attended the 2023 WiHacks event at Rochester Institute of Technology. At the event, we were challenged to build a proof of concept project that demonstrates how AI technologies can augment people’s lives in positive and meaningful ways. Almost immediately, we thought about how fruitful the intersection of AI technologies with American Sign Language (ASL) could be. All of us primarily use ASL in our daily lives, and it is our belief that ASL is a beautiful and powerful language that casts a bright light on American Deaf Culture and is thus deserving of recognition and appreciation. However, we are also fully cognizant of the fact that Deaf people are very much a linguistic and cultural minority in the U.S., and for that reason many people are not familiar with ASL. So we thought it would be wonderful if the power of AI could be harnessed to facilitate ASL learning and thus increase awareness of ASL and Deaf culture. We are particularly interested in how ChatGPT can be used to develop ASL awareness and proficiency because ChatGPT is freely available to everyone and has low barriers to entry.
WHAT IS SignGPT?
SignGPT is a plugin integrated with ChatGPT in order to extend its capabilities and it is a tool for learning and practicing ASL as well as learning about Deaf culture. Like ChatGPT, SignGPT can provide a personalized and adaptive learning experience that caters to the individual needs and preferences of users.
WHY SignGPT?
Although there are already some AI-driven solutions for ASL learning and translation, we don’t know of any that specifically leverages ChatGPT technology to teach ASL.
CAPABILITIES
SignGPT will be able to produce ASL videos on request. A user will be able to type a word or a phrase, after which the AI will provide videos associated with that word or phrase. Ideally, the AI will be able to display videos directly in the SignGPT, but if the AI is not able to display videos for a particular word or phrase for some reason, then it will provide links to relevant ASL dictionary websites such as Handspeak or Lifeprint. SignGPT will also be capable of giving users personalized feedback. There will be a toggleable button on the page called “practice.” If the user has it set to ON, they will be able to access personalized feedback from the AI. Once the user toggles it to ON, they will see the SignGPT screen split into 2, with the left screen devoted to AI output/feedback and the right screen devoted to user input. On the right screen, the camera will be on and the user will be able to view themselves signing. Once the user presses the red button on the right screen, it will begin recording. After the user has finished signing, they can stop recording by pressing the red button again. Then the AI will scan the video for any mistakes or issues. After scanning, it will provide the user with feedback on the left screen. It will generate videos of the signs for words / phrases that the user got wrong, and also generate links to resources such as ASL dictionaries and other ASL grammar / linguistics websites. On the other hand, if the user does not wish to get feedback from the AI and just wants to practice signing, they can simply sign into the camera without hitting the record button.
SignGPT will also be able to provide explanations and clarifications about Deaf culture and history and provide links to appropriate resources upon request, although most of the content will be geared towards American Deaf culture and American Deaf history since that is what the majority of SignGPT’s dataset will cover.
TECHNICAL PROCESS
First of all, we brainstormed ideas on a shared Google Doc. Once we hit upon the idea of SignGPT, we initially used paper and pen to sketch our ideas. Then we transitioned our design to Figma to make it look more clean and attractive. After we created wireframes and mockups with interaction flows in Figma, we began thinking about how to translate our design to code. We decided to use a combination of HTML, CSS, and Javascript (especially React) to implement our vision and use Visual Studio Code during the process. We used React bootstrap and CSS to customize the style of SignGPT and Javascript libraries to implement the webcam and other functionalities.
LIMITATIONS
SignGPT may occasionally generate signs that are incorrect in form or context. SignGPT may also lack domain-specific knowledge for signs in fields such as healthcare, science, technology, and so on. SignGPT may also lack knowledge of ASL dialects and regional variations, so it may not be able to recognize a signed variant of a particular word or phrase. Also, since the current publicly available version of ChatGPT has limited knowledge of the world and events after 2021, that means SignGPT will have similarly limited knowledge. Another major limitation of SignGPT is that it only supports ASL.
TARGET AUDIENCE
SignGPT is intended to be broadly accessible, but our target audience will likely consist of the following groups: hearing people who don’t know ASL, parents who would like to learn ASL for their Deaf or hard-of-hearing children, and Deaf and hard-of-hearing people who were never taught ASL or lost their hearing late in life.
FUTURE GOALS We previously mentioned that SignGPT currently only supports ASL. Our goal, however, is to eventually add support for all of the signed languages in the world, including International Sign Language, informally known as “Gestuno.” We also have a particularly ambitious goal to enable ASL-English and English-ASL translations in SignGPT. Users would be able to record themselves signing something in SignGPT, and the AI would be able to translate what they signed into written English. The AI would also be able to translate written English into ASL. Another ambitious goal of ours is to add captioning capabilities to SignGPT. SignGPT would be able to add captions and subtitles to videos, thus making uncaptioned media much more accessible. Users would be able to input links to the videos they would like to be captioned, and the AI would be able to grab those videos and add captions/subtitles to them. We also want to make SignGPT accessible to as many people as possible, so we would like to implement the following accessibility features in future iterations of SignGPT: bigger text, dyslexic font, and invert colors. We also would love to add other accessibility features as per user feedback/request. Examples of additional accessibility features would be grayscale, saturation, contrast, tooltips, and so on.