Developing an LSTM-Based Model for Accurate Sign Language Recognition Through Hand Keypoint Detection

Overview

Individuals with hearing impairments often receive inadequate medical care due to a lack of communication accessibility. This lack of effective communication is linked to lower medical literacy, increasing the risk of major medical issues. Our project’s aim was to develop a translation model that can translate American Sign Language (ASL) in real-time without a human translator for more efficient communication for the hearing-impaired population. Many current solutions for ASL translation are limited, many of them only work online, while some are restricted to finger spelling. Our project developed a computer vision model that interprets and translates ASL using action recognition. The model relies on three-dimensional positional data from keypoints in each hand, avoiding the need for video data and enabling easy dataset expansion. A Long Short-Term Memory, a type of Recurrent Neural Network, was chosen for the ASL to English translation model due to its ability to model sequential data, a key characteristic of ASL. The end goal of the project was to have a device, which was modeled in Computer Aided Design software, that could utilize the model with a Raspberry Pi, to translate ASL using a webcam and show the results on a display. The development of this project has the potential to significantly improve accessibility for deaf and hard of hearing individuals, promoting inclusivity and enhancing understanding. The program's ability to recognize signs, work offline, and easily expand its dataset could allow for global usage, particularly in developing regions.

Awards (2)

• Sigma Xi, The Scientific Research Honor Society: First Physical Science Award of $1,500 $1,500
• University of Arizona: Renewal Tuition Scholarship

Competition history

• ISEF 2023 Robotics and Intelligent Machines · Entry ROBO032T · Dallas, Texas, United States

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.