Enhancing Healthcare Communication for the Deaf and Hard of Hearing Community Using Landmark-Based ASL Recognition

Overview

Communication barriers in healthcare settings remain a significant challenge for Deaf and Hard-of-Hearing (DHH) individuals, particularly during patient-provider interactions. This can lead to misunderstandings, delays in treatment, and a loss of critical information. Although American Sign Language (ASL) interpreters and tools exist, access remains limited due to a lack of qualified interpreters, high costs, and high-latency systems that restrict real-time translations. This project presents SignBridge, a real-time, scalable application designed for ASL recognition and translation for medically relevant signs. Using MediaPipe Holistic, 21 keypoints from each hand and 25 upper-body pose landmarks were extracted across 40-frame sequences to capture both spatial and temporal features. Multiple Long Short-Term Memory (LSTM) architectures were evaluated, incorporating motion-based velocity, temporal attention mechanisms, and feature normalization strategies to improve robustness. The best-performing architecture, an attention-augmented LSTM with motion features and normalization, achieved an accuracy of 98%, macro F1 of 0.983, and a latency of 51.40 ms. Combining temporal attention, velocities, and normalization improved per-class consistency without significantly increasing latency. The final model was integrated into SignBridge, enabling real-time translation of ASL to text with confidence scoring and bidirectional communication through a combined chat window containing speech-to-text functionality. This approach provides a computationally efficient, low-latency solution that has the potential to enhance healthcare accessibility by bridging communication gaps between the DHH and hearing communities, creating a more equitable system for all.

Awards (1)

• Association for the Advancement of Artificial Intelligence: AAAI Student Memberships for each finalist that is part of the 1st, 2nd, and 3rd Prize Winning projects and 5 Honorable Mention winning projects (up to 3 students per project) (in-kind award / part of the 1st-3rd prize)

Competition history

• ISEF 2026 Software Design · Entry SFTD018

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.