SAMA: A Self-Evolving AI Multimodal Assistant for Dementia Care

Overview

Dementia affects over 57 million individuals worldwide, causing two critical challenges: facial recognition deficit and disordered speech. Current assistive tools for dementia such as facial recognition aids or dementia caregiver apps only offer limited static solutions, which fail to adapt to the patients' changing needs and become less productive as their condition worsens. This project introduces a novel AI agent integrated with augmented reality (AR) glasses that offers facial recognition with on screen prompts, disordered speech correction, and evolves with patients' daily changing data. SAMA is made of three subsystems: (1) a Facial Recognition Self-Evolving System, using the ResNet model to identify known individuals and auto learn/label new faces using Qwen 3 LLM reasoning; (2) an Auto Speech Recognition Self-Evolving System, using the Whisper ASR model to detect disordered speech and continuously label/learn user voice data; and (3) a Multimodal Information Based Second Pass Regeneration System which ranks Whisper's N-best hypotheses using environmental context info to improve prediction accuracy. Furthermore, through multiple iterations of testing and validation, I designed a two stage training pipeline to fine-tune SAMA to auto adapt to daily degraded speech. Stage one utilizes a curriculum learning pipeline to improve Whisper's accuracy in progressively degrading speech, while stage two experimented several training algorithms including SFT and RL training to enhance the text reasoning of Qwen 3 for hypothesis ranking. Testing of the two stage pipeline show significant reduction of WER and facial recognition evaluation also received exemplary accuracy. SAMA presents a unique self-evolving agent to dementia care, benefitting millions of patients world-wide.

Awards (1)

• Association for Computing Machinery: Third Award of $1,500 $1,500

Competition history

• ISEF 2026 Technology Enhances the Arts · Entry TECA018

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.