Wearable Translator for Bidirectional Communication Between ASL and Speech

Overview

Communication between the hearing impaired and the general people has long been an unresolved issue. Although sign language can solve this problem to some extent, sign language becomes ineffective when communicating with people who do not know it. Currently, related studies are attempting to achieve sign language recognition, such as using machine vision or smart gloves. For hearing impaired, there are related studies on hearing aids and speech-to-text software. However, none of these studies have truly broken down the barriers to bidirectional communication. Therefore, this project aims to design a wearable device to facilitate seamless communication between hearing impaired and others. Based on the chest-worn prototype made in my preliminary research, I designed a glasses-style wearable device to make the device more convenient and unobtrusive to use. The device includes a machine vision sign language recognition and speech-to-text model deployed on Raspberry Pi5, an eye-mounted display, and a glasses-style head-mounted structure that I designed using Fusion360 and made through 3D printing. I trained the sign language recognition model using VGG-16. The training set contained 104,000 sample photos, and the recognition rate for the ASL alphabet reached 95%. By wearing the device developed in this study, bidirectional communication can be achieved between hearing impaired and others, allowing both parties to communicate in their preferred ways. This project involves the training of machine learning models, programming, debugging, the iteration of prototypes, and feasibility testing, thus verifying the feasibility of the solution. This innovation is expected to empower the hearing-impaired community, promoting social inclusion and their independence.

Competition history

• ISEF 2025 Technology Enhances the Arts · Entry TECA008

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.