Integrating Vision-Language Models and Retrieval-Augmented Generation for Adaptive Alzheimer's Assistance

Overview

Alzheimer’s disease (AD) is a progressive neurodegenerative disorder characterized by heterogeneous cognitive decline, including episodic memory loss, semantic impairment, and visuospatial dysfunction. While existing assistive technologies primarily emphasize reactive safety, they lack proactive, context-aware semantic support and often rely on cloud-based processing, raising privacy concerns. This study presents the development and evaluation of a fully offline, privacy-aware multimodal mobile assistant designed to mitigate object and face recognition deficits in individuals with AD. The system integrates MobileVLM for on-device multimodal reasoning, a FaceNet-based biometric pipeline with MTCNN face detection for prosopagnosia support, and ChromaDB for on-device retrieval-augmented generation (RAG). All components operate locally using performance optimizations to ensure efficient execution within resource-constrained mobile environments. A voice interface enables hands-free interaction using native on-device speech recognition. Evaluation across a large-scale corpus of clinical records demonstrated that the RAG-based framework effectively retrieved contextual data, significantly outperforming standard keyword-filtered LLM approaches. Passive linguistic monitoring successfully detected markers of mild cognitive impairment progression with high reliability. Real-time face identification demonstrated near-perfect classification accuracy while maintaining low-latency inference. These results demonstrate the feasibility of a fully localized “memory prosthesis” that provides proactive semantic assistance while preserving patient privacy. The framework establishes a scalable foundation for integrating edge-based multimodal AI into Alzheimer’s care.

Awards (2)

• Association for the Advancement of Artificial Intelligence: AAAI Membership for the School Libraries of All 8 Winners (in-kind award / part of 1st-3rd prize and honorable mentions' prize)
• Association for the Advancement of Artificial Intelligence: Third Award of $500 $500

Competition history

• ISEF 2026 Software Design · Entry SFTD026 Affiliated fair in CT, United States of America

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.