EZ-Sign: A Nonintrusive Aid for Communication with ASL Using Machine Learning

Overview

Individuals with hearing loss have difficulty communicating effectively through sign language when the majority of the world’s population communicates verbally. Still, over 70 million people, approximately 1 out of every 100 people, use sign language as their primary method of communication. To bridge this communication gap, many devices, such as sign language gloves and video recognition, have been created to translate sign language to aid the deaf community; however, due to cost and lack of vocabulary, these devices have not been widely accepted as they force signers to compromise their style of signing. To solve this problem, a lower cost assistive technology has been engineered to allow the deaf to sign at their comfortable rate. A Leap Motion device, which infers the coordinates of the joints, palm, and fingertips, is connected to a laptop. Training data was created by calculating the distances and angles between the fingers for different positions above the Leap Motion. The data was trained in classification models, which then translated hand gestures in real-time. Currently, the device has all the static numbers 0 – 9 and A – Z trained with an accuracy of over 84% for those gestures, allowing users to reliably communicate. The different models were also compared to see which classification algorithm was the best. Currently, the dynamic gestures model can reliably differentiate between I,J and D,Z – which have otherwise identical static gestures! With this product, signers will be able to better communicate with non-signers, while being able to keep their style of signing.

Competition history

• ISEF 2020 Robotics and Intelligent Machines · Entry ROBO037

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.