Novel Electronic Gloves: Converting Sign Language to Natural Language for People With Hearing and Speech Impairment

Overview

Electronic gloves utilizing flex sensors and micro-electro-mechanical-system (MEMS) inertia measurement units (IMU) have been developed to aid individuals with hearing and speech impairments by translating sign language to natural language. However, existing models have limitations, notably IMU drift, which reduces the word conversion capacity and lacks comprehensive, complete sentence conversions and two-way communication. I built Bluetooth Low Energy (BLE)-enabled electronic gloves with similar sensors, a solution for IMU drift, and seamless integration with a mobile app. Using the Large Language Model (LLM) on a server, the gloves display individual words that can be transformed into natural language sentences. The app can communicate with the server via 4G or Wi-Fi. I developed an application housing a customized Transformers model with 393,888 parameters. Our model detects and normalizes abnormally segmented trajectories, thanks to the simultaneous collection of normal and ToF camera trajectory training data within the IMU via the Colmap-SLAM algorithm. By fine-tuning the Sheared-LLaMA-2.7B model on an online server using the PEFT-LoRA method, we can concatenate individual words into complete sentences displayed as text or sound on the app. The IMU drift has been mitigated, enabling us to draw similar trajectories to typical ones, resulting in the ability to generate 40 words and an unlimited word number. The accuracy in words distinguished reached 99.3% using the Cosine Similarity algorithm. The current gloves can merge 40 individual words into 100 meaningful sentences in English or Vietnamese in the context defined for optimal word use in sentence converting.

Competition history

• ISEF 2024 Technology Enhances the Arts · Entry TECA020 · Los Angeles, California, United States

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.