Meta's Interaction SDK Allows for Accurate American Sign Language Detection

Overview

This project investigated whether Meta's Interaction SDK was sufficient for accurate American Sign Language (ASL) detection. This was tested in a virtual reality (VR) environment in an attempt to bridge language barriers and find new uses for the underutilized technology that is hand tracking. Based on the developer of this project's prior experience and many others' work in the VR space, it was anticipated that accurate sign detection systems would be possible with the SDK. The project was created in Unity and utilizes the Meta Interaction SDK to model the user's hands. Prior versions used the OpenXR Hand Interaction plugin to interpret gestures, but custom systems proved more accurate for detecting complex signs. C# scripts formed the basis of the project's functionality, sensing when the user's hands formed signs and how to react accordingly. Signs are ultimately detected by checking hand gestures, positions, and then stages, only completing a full sign if all three steps succeed. As better sign detection features were implemented, such as checking the relative positions between the hands and other objects, performing signs felt more natural and proper sign detection grew more consistent. This perfectly matched the hypothesis for this project, indicating there is a bright future for this technology and its ability to enhance communication both in and outside of virtual worlds. Though development was a success, further additions to allow a broader range of and greater control over signs are needed to truly bridge language barriers in a new and practical way.

Competition history

• ISEF 2025 Systems Software · Entry SOFT029

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.