HandTalk: A Translation System for American Sign Language

Overview

Approximately 11 million people in the U.S. experience severe hearing loss, yet most hearing individuals do not understand American Sign Language (ASL). HandTalk is a cost-effective interactive translation platform that combines gesture recognition with robotic reproduction of ASL signs. HandTalk integrates a six-degree-of-freedom (6-DOF) robotic arm with a 10-DOF 3D-printed robotic hand to track and replicate human upper-limb motion in near real time. Using MediaPipe pose estimation, 47 body and hand landmarks are extracted from video, converted into joint angles, and mapped to servo positions. The 6-DOF arm is driven by an ESP32 microcontroller, while an Arduino controls the 10-DOF hand, enabling synchronized actuation and coordinated signing. A gesture recognition pipeline interprets live video to classify ASL words based on joint angle trajectories, supporting both imitation and replay modes. Five classification models were evaluated using a 20 word dataset: two traditional baselines (KNN and DTW) and three neural networks (Bi LSTM, 1D CNN, and Tiny Transformer). The Bi LSTM achieved about 92% accuracy, and the 1D CNN performed similarly, while the transformer and static models trailed. HandTalk supports live signing through webcam input, classifies signs in near real time, and drives robotic output with low latency. The fully actuated upper-limb robotic prototype allows visualizing and reproducing ASL gestures for common words. Future work will expand the vocabulary, improve arm movement smoothness, and investigate integrating facial expression modeling to improve ASL recognition accuracy. HandTalk demonstrates how low-cost AI‑assisted robotics can serve as a practical tool for accessibility, education, and communication, especially in underserved or low‑resource communities.

Competition history

• AJAS 2026 Category not listed

Related projects

Closest projects by meaning, across every fair and year in the corpus.

Browse more like this

• AJAS None projects
• Every AJAS 2026 project