Intelligent Object Finding for the Elderly with Alzheimer's Disease Based on Neural-Symbolic Reasoning
ISEF · 2022 Robotics and Intelligent Machines
Overview
Memory impairment caused by Alzheimer's disease seriously affects the daily life and even safety of the elderly. For example, if the emergency medicine cannot be found in time, the valuable emergency time window will be missed. Existing object finding technology based on either computer vision or wireless IoT cannot solve the problem of object occlusion and long time-span. Moreover, it is generally believed that human behavior is the fundamental reason for the invisibility and position change of objects. In this project, we propose a new object finding method based on neural-symbolic reasoning, which predicts object position by inferring both human-object interaction and object state change. In the neural perception part, human actions based on 3D skeleton representation are recognized by a carefully-designed recurrent network. While in the symbolic reasoning part, the interaction rules between human, target object and context objects (e.g., table and other containers) are modeled by a probabilistic logic framework. For model training, we implement joint neural-symbolic optimization by seamlessly integrating the parameter learning of probabilistic logic programming with back propagation of neural network for behavior recognition. To verify the effectiveness of the proposed method, we collect an object finding dataset including 17 sets of 4-camera videos for total 15GB. The comparison results on this dataset demonstrate that our method can find objects more efficiently and accurately in complex scenes with long time-span and occlusion than traditional computer vision-based methods. Our neural-symbolic framework can also provide better interpretability and stronger generalization capabilities.
Competition history
- ISEF 2022
Resources
Related projects
ISEF · 2026
Integrating Vision-Language Models and Retrieval-Augmented Generation for Adaptive Alzheimer's Assistance
ISEF · 2025
EnAct: A Safety-Aware Actionable Guidance System via Multi-Agent Vision-Language Models for People With Vision Impairments
ISEF · 2020
Real-Time Object Search and Detection for the Visually Impaired
ISEF · 2024
An Improved Deep Neural Architecture Search Net-Based Wearable Object Classification System for the Visually Impaired
Closest projects by meaning, across every fair and year in the corpus.
Source: Regeneron International Science and Engineering Fair