Improving Sign Language Translation to Text Using Machine Learning

Overview

Over 430 million people worldwide struggle with disabling hearing loss, and among those that are deaf, 80% of individuals resort to using a form of sign language to communicate. Today, we rely on human interpreters to bridge the communication gap between sign language and non-sign language users, but interpreters are becoming increasingly inaccessible, and online media offers no translation support. In this project, I tackle the problem of automated American Sign Language (ASL) translation from glosses (transcribed representations for a sequence of signs) to English text and vice versa. I addressed the main challenge of low-resource constraints in sign language translation by developing a novel approach to generate synthetic data samples for rare words in the dataset. The algorithm involved building a dictionary of English text to glosses and performing masked word prediction using BERT, a deep learning language model. Using this method, the well known NCSLGR training set of 651 samples was augmented to 3858 samples. A deep learning transformer model was trained on the augmented dataset to predict text from glosses and demonstrated a BLEU score of 41.22, more than doubling the performance of the model from the most recent work on ASL translation. Another transformer was also trained to predict glosses from text with a BLEU score of 42.02. As a demonstration of the work’s real-world applicability, a web application was developed to allow users to input English text and output a series of ASL sign videos corresponding to the predicted glosses.

Competition history

• AJAS 2024 Category not listed

Related projects

Closest projects by meaning, across every fair and year in the corpus.

Browse more like this

• All Robotics and Intelligent Machines projects
• AJAS Robotics and Intelligent Machines projects
• Every AJAS 2024 project