HandTalk: A Translation System for American Sign Language

Overview

Approximately 11 million people in the U.S. experience severe hearing loss, yet most hearing individuals do not understand American Sign Language (ASL). HandTalk is a cost-effective interactive translation platform that combines gesture recognition with robotic reproduction of ASL signs. HandTalk integrates a six-degree-of-freedom (6-DOF) robotic arm with a 10-DOF 3D-printed robotic hand to track and replicate human upper-limb motion in near real time. Using MediaPipe pose estimation, 47 body and hand landmarks are extracted from video, converted into joint angles, and mapped to servo positions. The 6-DOF arm is driven by an ESP32 micro-controller, while an Arduino controls the 10-DOF hand, enabling synchronized actuation and coordinated signing. A gesture recognition pipeline interprets live video to classify ASL words based on joint angle trajectories, supporting both imitation and replay modes. Five classification models were evaluated using a 27-word dataset: two traditional baselines (KNN and DTW) and three neural networks (Bi-LSTM, 1D-CNN, and Tiny-Transformer). The Bi-LSTM achieved about 92% accuracy, and the 1D-CNN performed similarly, while the transformer and static models trailed. HandTalk supports live signing through webcam input, classifies signs in near real time, and drives robotic output with low latency. The upper-limb robotic prototype allows visualizing and reproducing ASL gestures for common words. Future work will expand the vocabulary, improve arm movement smoothness, and investigate integrating facial expression modeling to improve ASL recognition accuracy. HandTalk demonstrates how machine-learning and robotics can serve as a practical tool for accessibility, education, and communication, especially in underserved or low-resource communities.

Awards (2)

• First Award of $6,000 $6,000
• Qorvo: Qorvo Innovator Award - 3rd Place

Competition history

• ISEF 2026 Engineering Technology: Statics & Dynamics · Entry ETSD048

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.