Beyond Audio: A Multimodal EMG-Visual Speech System for Reconstructing Voice From Silence

Overview

Aphonia—complete loss of voice following stroke or laryngeal surgery—affects millions worldwide yet lacks practical, non-invasive communication solutions. This study was deeply motivated by witnessing a grandmother rendered unable to communicate after a stroke, a condition where the voice is permanently silenced while cognition remains intact. This highlighted the urgent need for accessible communication tools. Traditional clinical electromyography (EMG) equipment costs upwards of $15,000, preventing everyday patient use. To address this, I engineered a custom surface EMG (sEMG) PCB for approximately $285, achieving an over 98% cost reduction. To evaluate the system, I personally recorded a small-scale dataset using my custom hardware. With this data, I analyzed the optimal number of sEMG channels. Results surprisingly appeared as 1 channel accuracy was the highest score, above 2-channel and 3-channel configurations. Character Error Rate (CER): 1-channel (51.05%), 2-channel (55.58%), and 3-channel (55.18%). The design was intentionally capped at three channels to maintain practical, burden-free daily wearability. Regarding the fusion of lip video and EMG signals, experiments on this custom data confirmed that Concatenation achieved the lowest CER (58.11%) compared to cross-attention architectures, proving it the most robust strategy under noisy, limited-data conditions. Finally, I collected a multi-subject dataset from four individuals, including one Aphonia patient, to fine-tune Auto-AVSR, a state-of-the-art visual speech model. This integration reduced the baseline Word Error Rate WER from 19.1% to 9.8%. This study proves that a cost-effective, custom-built EMG device combined with optimal fusion architecture yields a highly practical assistive communication system.

Awards (4)

Fourth Award of $600 $600
King Abdulaziz & his Companions Foundation for Giftedness and Creativity: Full Scholarship from King Fahd University of Petroleum and Minerals(KFUPM) (and a $400 cash prize) $400
King Abdulaziz & his Companions Foundation for Giftedness and Creativity: NOT TO BE READ -- $400 cash prize for each Full Scholarship from King Fahd University award recipient $400
Midjourney: Eight awards of $2,500 $2,500

Competition history

ISEF 2026 Biomedical Engineering · Entry ENBM056

Resources

ISEF abstract page

Related projects

ISEF · 2021

SpeakUp: A Machine Learning-Based Speech Aid to Enable Real-Time Silent Communication for the Paralyzed by Translating Neuromuscular EMG Signals to Speech

CWSF · 2026