fairfolio
Saved Projects

HearBridge: AR Headset for Interactive Communication via Conversation Textualization and Sign Language Translation for the Hearing Impaired

ISEF · 2024 Technology Enhances the Arts

Byun, Seongbin, Cho , Hyunwoo, Park, Seongmin · Kyeongbuk High School

Overview

Globally, more than 1.5 billion people have a hearing impairment, and they struggle to communicate much more than people with other disabilities. Previously, various methods have been utilized to solve the communication issue for the hearing-impaired, such as hearing aids. However, these methods have considerable downsides; they only assist the hearing-impaired to hear what others are saying, rather than boosting their ability to communicate with others, with their residual hearing limiting their use. The absence of interactive communication goes beyond simply being unable to express; it is a major barrier to their full participation in society and their capacity to make a livelihood. To address these challenges, this project proposes an AR headset called "HearBridge" by textualizing conversations and translating the user’s sign language into speech. While traditional solutions simply textualize the conversation without recognizing who is saying what, HearBridge performs visual speech recognition via lipreading rather than speech-to-text, allowing users to know exactly who is saying what. Also, while existing solutions are not prepared for sign language, HearBridge tracks hand landmarks with the Leap Motion 2 Sensor and translates sign language into speech using the WLASL dataset. HearBridge's lipreading model achieved a WER of 17%, which is 18% lower than previous studies. As a result, HearBridge will significantly enhance their full participation in society by encouraging interactive communication, generating various positive effects. Furthermore, HearBridge will improve usability through its user-friendly design and affordability by reducing costs to under $440, making it more cost-effective compared to hearing aids, which are typically priced at around $2000.

Competition history

  • ISEF 2024 Technology Enhances the Arts · Entry TECA011T · Los Angeles, California, United States Affiliated fair in South Korea

Resources

Related projects

Closest projects by meaning, across every fair and year in the corpus.

Browse more like this

Source: Regeneron International Science and Engineering Fair

Save projects to your library

Sign in with Google to keep track of projects you find interesting, organized into folders. An account also raises your daily allowance for “Has this been done?”, and lets you create a key for the MCP server with a much higher limit than anonymous use. Browsing stays public.

Continue with Google