Irisense: A Free, Accessible and AI-Powered AAC Application Using Real-Time Eye Tracking for Patients With Severe Speech and Motor Impairments

Overview

Severe Speech and Motor Impairment (SSMI), including conditions like advanced-stage ALS, progressively traps individuals in their bodies. This loss of muscle control and communication ability significantly reduces quality of life and increases social isolation. Current eye-tracking assistive systems are often expensive, complex to set up, and reliant on external hardware, which severely limits their accessibility for disadvantaged individuals. This project utilizes a standard smartphone camera and a multi-stage vision pipeline. Raw frames undergo grayscale conversion, contrast enhancement, and Gaussian blurring to maximize pupil-sclera distinction. Leveraging MediaPipe at the native layer, the system extracts facial landmarks, while the Perspective-n-Point (PnP) algorithm enables 3D head pose estimation for posture-invariant tracking. Stable control is achieved through linear regression and consecutive frame averaging. A Visual tab provides pictogram-based communication for those with low literacy. Furthermore, a Social tab operates as a messaging platform, empowering isolated patients to text distant relatives and connect with peers. For maximum privacy, an on-device Local LLM generates context-aware sentences, supporting 10 languages. Evaluated on 108 participants (including 8 SSMI patients), the system achieved 96.3% detection accuracy (94.1% with reflective glasses) and 93.1% communication success. Ultimately, these findings prove that smartphones can serve as a highly scalable, cost-effective, and accessible alternative to traditional AAC systems.

Awards (2)

• Fourth Award of $600 $600
• Association for the Advancement of Artificial Intelligence: AAAI Membership for the School Libraries of All 8 Winners (in-kind award / part of 1st-3rd prize and honorable mentions' prize)

Competition history

• ISEF 2026 Software Design · Entry SFTD003T

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.