Sign Language Recognition Using WLASL Sign Language Database and MediaPipe Hands

Overview

Most Intelligent Personal Assistant (IPA) devices are reliant on voice recognition, which poses a major accessibility issue for the deaf and hard-of-hearing (DHH) community. A majority of DHH users do not have IPA experience, and designing an accessible alternative requires investigation to ensure proper design. Although deaf users express interest in seeing an accessible IPA, there is not current technology suitable for this community paralleling IPA functionality. The few sign language recognition systems that do exist require extra sensors, cameras, or powerful computer processing not provided by a smartphone. This research proposes multiple methods for a smartphone interface system with the use of MediaPipe Hands, a hand landmark extraction data pipeline, combined with a machine learning tools such as TensorFlow, to provide a viable and customizable dynamic gesture recognition interface that has the capability to run efficiently on a smartphone. Initial testing performed at around 95% accuracy on a small dataset, but fails to scale to the size necessary for comprehensive sign language recognition. This paper further proposes an alternative method implementing the WLASL database that bypasses many of the limitations of the previous method.

Competition history

• AJAS 2024 Category not listed

Related projects

Closest projects by meaning, across every fair and year in the corpus.

Browse more like this

• All Systems Software and Computer Science projects
• AJAS Systems Software and Computer Science projects
• Every AJAS 2024 project