fairfolio
Saved Projects

Integrating Vision-Language Models and Retrieval-Augmented Generation for Adaptive Alzheimer's Assistance

ISEF · 2026 Software Design

Joseph, Ethan · Engineering and Science University Magnet School

Overview

Alzheimer’s disease (AD) is a progressive neurodegenerative disorder characterized by heterogeneous cognitive decline, including episodic memory loss, semantic impairment, and visuospatial dysfunction. While existing assistive technologies primarily emphasize reactive safety, they lack proactive, context-aware semantic support and often rely on cloud-based processing, raising privacy concerns. This study presents the development and evaluation of a fully offline, privacy-aware multimodal mobile assistant designed to mitigate object and face recognition deficits in individuals with AD. The system integrates MobileVLM for on-device multimodal reasoning, a FaceNet-based biometric pipeline with MTCNN face detection for prosopagnosia support, and ChromaDB for on-device retrieval-augmented generation (RAG). All components operate locally using performance optimizations to ensure efficient execution within resource-constrained mobile environments. A voice interface enables hands-free interaction using native on-device speech recognition. Evaluation across a large-scale corpus of clinical records demonstrated that the RAG-based framework effectively retrieved contextual data, significantly outperforming standard keyword-filtered LLM approaches. Passive linguistic monitoring successfully detected markers of mild cognitive impairment progression with high reliability. Real-time face identification demonstrated near-perfect classification accuracy while maintaining low-latency inference. These results demonstrate the feasibility of a fully localized “memory prosthesis” that provides proactive semantic assistance while preserving patient privacy. The framework establishes a scalable foundation for integrating edge-based multimodal AI into Alzheimer’s care.

Awards (2)

  • Association for the Advancement of Artificial Intelligence: AAAI Membership for the School Libraries of All 8 Winners (in-kind award / part of 1st-3rd prize and honorable mentions' prize)
  • Association for the Advancement of Artificial Intelligence: Third Award of $500 $500

Competition history

Resources

Related projects

Closest projects by meaning, across every fair and year in the corpus.

Browse more like this

Source: Regeneron International Science and Engineering Fair

Save projects to your library

Sign in with Google to keep track of projects you find interesting, organized into folders. An account also raises your daily allowance for “Has this been done?”, and lets you create a key for the MCP server with a much higher limit than anonymous use. Browsing stays public.

Continue with Google