HandTalk: A Two-Way Translation System for American Sign Language

Overview

Nearly 5% of the global population, or 430M people, require hearing loss rehabilitation, while in the U.S., about 11M people experience serious hearing difficulties. Many deaf individuals communicate through sign language, but few hearing people understand it, making everyday interactions challenging. This project introduces an affordable, portable, two-way translator between American Sign Language (ASL) and English that uses computer vision techniques for highly accurate fast interpretation. The HandTalk system translates ASL into English through a hand pose estimation neural network and k-Nearest-Neighbors classifier, identifying ASL letters based on finger joint angles. This system runs on various platforms, including Jetson Nano computers, laptops, and tablets. Additionally, a 3D-printed, servo-actuated robot hand uses the HandTalk algorithm in reverse to translate English text into ASL signs, creating a complete ASL-to-English and English-to-ASL translation system. This setup is affordable (under $200) and adaptable to different platforms. The system's performance was tested under real-world conditions, such as varying lighting and backgrounds, with experimental groups assessing the ASL recognition accuracy. The system achieves over 90% accuracy in ASL recognition under controlled conditions, performing well across different environments. The robot-hand learns signs by observing users perform them and can replicate each sign consistently after seeing it once. Future improvements aim to expand the system's vocabulary to include more gestures and enhance its robustness in diverse real-world settings. HandTalk offers a promising solution for automatic two-way ASL translation, with the potential to improve communication for millions of deaf individuals worldwide.

Competition history

• ISEF 2025 Technology Enhances the Arts · Entry TECA010

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.