HearMe A Bidirectional Communication System for Hearing Impaired

Overview

Over 430 million people have some degree of hearing loss . Hearing-impaired individuals use sign language (SL) as their primary communication method, but SL literacy in society is low. This creates communication barriers, leading to psychological problems, social isolation, and difficulties in daily activities.Existing SL recognition systems have drawbacks. Image-processing systems are sensitive to lighting and not portable. Glove-based systems restrict hand movements therefore interfere with daily activities.This project presents a near-real-time SL recognition system using sEMG and IMU sensors. sEMG is chosen because it measures muscle activity without interrupting users’ daily activities, while IMU sensors provide hand orientation. By integrating data from both hands, the system can recognize SL gestures.A dataset was collected to train a general model, but using a general dataset is challenging. Sign language is not strictly standardized, leading to variations across regions even individuals. Because general models are trained on static datasets, they inherently struggle to adapt to this diversity. To address this limitation, a personalized model was introduced that learns from user-recorded gestures, adapting to individual signing styles, hand dominance, and regional variations.This hybrid approach combining a general model with a personalized model enables user customization, making it a more inclusive, flexible and adaptable solution than fixed-dataset systems.System test and user survey has been done to evaluate the project. Participants reported that the system could reduce communication barriers and increase social confidence, suggesting strong potential social impact, nevertheless future improvements are needed to enhance communication performance.

Competition history

• ISEF 2026 Technology Enhances the Arts · Entry TECA001T

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.