A Novel Program for the Detection and Translation of the ASL Alphabet through the Use of Deep Learning

Overview

Objectives American Sign Language (ASL) is a common way for people who are deaf, hard of hearing, or verbally impaired to communicate, with over one million people using it as a primary method of communication. While this language is prevalent in terms of usage, and has been cited as one of the mostly commonly-used languages used today, many people who don t rely on ASL to communicate have no knowledge or rudimentary grasp on the language. Because of this, there is currently an almost insurmountable language barrier between hearing people and the hard of hearing/deaf community. The goal of this project was thus to create a program that translates the ASL Alphabet in order to provide a means of communication that bridges the gap between these isolated communities. Methods I collected data from a variety of different sources, including both a self-generated dataset and data compiled from various publicly available services. This data was then sorted into classes by letter, and uploaded to an Amazon S3 server in preparation for training my model. I then trained my model with Pytorch, using a modified version of the standard Resnet18 architecture to accurately classify the data through deep learning. By using transfer learning to hasten the training process, I was able to achieve accurate results that could generalize well to other datasets that the model had not yet seen. Using this model, I was then able to create a program that could process webcam input from a user to translate the ASL alphabet in real time. I used OpenCV to pass camera input to the deployed model, which would then output the most probable letter as a translation. This end-to-end system created a reliable way to classify and translate the alphabet. Results By evaluating the model on a set of data that was different than that given to the model during training, I found that the model had an overall accuracy of 82%, which exceeded my goal for my model s accuracy. This accuracy data is also backed up by the accuracy data given while training, which showed an accuracy of over 90% with the training and validation data it had. Conclusions My program was also able to generalize well to real time usage, which shows that my model was successful at translation. My detection program also performed admirably and is able to accurately detect letters in under a second, meeting my second criteria for this project. This speed allows people fluent in ASL to sign naturally and means that my program adapts best to the real need of people in the ASL community.

Competition history

• CSEF 2019 Computational Systems & Analysis (Senior Division Only) · Entry S0814

Resources

• CSEF project summary ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.

Browse more like this

• All Systems Software and Computer Science projects
• CSEF Systems Software and Computer Science projects
• Every CSEF 2019 project