HearBridge: AR Headset for Interactive Communication via Conversation Textualization and Sign Language Translation for the Hearing Impaired

Overview

Globally, more than 1.5 billion people have a hearing impairment, and they struggle to communicate much more than people with other disabilities. Previously, various methods have been utilized to solve the communication issue for the hearing-impaired, such as hearing aids. However, these methods have considerable downsides; they only assist the hearing-impaired to hear what others are saying, rather than boosting their ability to communicate with others, with their residual hearing limiting their use. The absence of interactive communication goes beyond simply being unable to express; it is a major barrier to their full participation in society and their capacity to make a livelihood. To address these challenges, this project proposes an AR headset called "HearBridge" by textualizing conversations and translating the user’s sign language into speech. While traditional solutions simply textualize the conversation without recognizing who is saying what, HearBridge performs visual speech recognition via lipreading rather than speech-to-text, allowing users to know exactly who is saying what. Also, while existing solutions are not prepared for sign language, HearBridge tracks hand landmarks with the Leap Motion 2 Sensor and translates sign language into speech using the WLASL dataset. HearBridge's lipreading model achieved a WER of 17%, which is 18% lower than previous studies. As a result, HearBridge will significantly enhance their full participation in society by encouraging interactive communication, generating various positive effects. Furthermore, HearBridge will improve usability through its user-friendly design and affordability by reducing costs to under $440, making it more cost-effective compared to hearing aids, which are typically priced at around $2000.

Competition history

• ISEF 2024 Technology Enhances the Arts · Entry TECA011T · Los Angeles, California, United States

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.