ExpressBuddy: An AI-Powered Companion With Impediment-Aware Speech Processing for Children With Autism and Speech Disorders

Overview

In the U.S., about 1 in 14 children has a communication disorder, and 1 in 4 children with speech disorders also experience social and emotional difficulties, making them twice as likely to be bullied. Most receive only 1-3 hours of speech therapy per week, leaving limited opportunities to practice communication in real-world settings. ExpressBuddy is an AI-powered system that provides practice between therapy sessions. Its voice-based companion, Pico, is placed in situations like bullying, peer pressure, or exclusion, and the child becomes the teacher, guiding Pico through how to respond. This role reversal helps children practice articulation, turn-taking, and social reasoning through the protege effect, where teaching reinforces learning. To support use in noisy classrooms on low-compute Chromebooks, we developed an on-device audio pipeline. A CPU-based Audio-Worklet DSP applies noise gating and dynamic compression. Next, a WebAssembly VAD model uses STFT and LSTM/GRU inference to segment speech in real time. A 150ms silence buffer is used to preserve dysfluent pauses to accommodate stuttering students. Because social interaction also relies on reading facial cues, the system includes emotion-recognition activities in which students identify and mirror expressions. An on-device vision pipeline using a YOLOv2-based detector and CNN expression model assessed whether mirrored expressions matched the target emotion with over 85% accuracy. In a school district pilot with more than five students under Speech-Language Pathologist supervision, one participant showed a 56.8% increase in daily verbal output and a 41.2% reduction in AI prompting. These results show ExpressBuddy can increase speech engagement and support more independent conversation outside therapy.

Awards (3)

• First Award of $6,000 $6,000
• Association for the Advancement of Artificial Intelligence: AAAI Membership for the School Libraries of All 8 Winners (in-kind award / part of 1st-3rd prize and honorable mentions' prize)
• Association for the Advancement of Artificial Intelligence: Second Award of $1,000 $1,000

Competition history

• ISEF 2026 Software Design · Entry SFTD022T

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.