Recognition of American Sign Language (ASL) Trigger Words Using RF Sensors in Combination with Deep Neural Networks

Overview

For more than two centuries, deaf communities in the U.S. have benefited from American Sign Language (ASL) as their main mode of communication. However, in the past decade, with the rapidly expanding use of speech recognition and voice-controlled devices (e.g. Siri, Alexa), the deaf are being excluded from the benefit afforded by such technological advances. There has thus been an interest in designing means for automated ASL recognition to serve as an interface between the deaf and modern devices. Current approaches are intrusive, limit user movement, suffer from privacy concerns, cannot function in the dark, or still lack accuracy. By contrast, Radar (radio detection and ranging) has emerged as a promising approach for ASL recognition when coupled with machine learning. Radar based sensors can work in the dark, from a distance without contact, and don’t suffer from privacy concern. Experimentally, this is done by pairing a commercial RF transceiver with a data capture card to record radio frequency (RF) signals associated with a sequence of ASL signs mixed with various body motions (e.g. walking, sitting, standing). In this project, de-identified raw RF data cubes are each processed into 3 distinct RF data domain representations: Range-Doppler and Range-Angle video maps, as well as micro-Doppler spectrograms. These were then used as inputs to a joint domain multi-input, multi-task learning-based Deep Neural Network (DNN) architecture. The trained network could then be used for the recognition of a few ASL ‘trigger’ words and evaluated for the accuracy of such recognition.

Awards (1)

• National Security Agency Research Directorate : Third Place Award “Cybersecurity”

Competition history

• ISEF 2022 Robotics and Intelligent Machines · Entry ROBO034 · Atlanta, Georgia, United States

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.