AI Companion for ASD: Predicting Listener's Attention Using Multi-Modal Response Analysis

Overview

This research project aims to create an artificial intelligence tool to improve social communication for neurodivergent individuals, especially those with autism spectrum disorder. The test included participants watching a video with a narrated script with embedded distractors, followed by a recall questionnaire to gauge participants' attentiveness. During the test, participants' faces were recorded for data analysis and later used for training machine learning models. Various characteristics, such as mouth movement, eye openness, gaze direction, and head orientation, were analyzed using facial landmarks attained from DLib, etc. The project analyzed multiple Machine Learning models (Logistic Regression, Support Vector Machines, Decision Tree, Random Forest, XGBoost, Neural Network, and Gemini 1.5 Pro Client, etc.), with XGBoost yielding the highest performance with respect to evaluation metrics (such as precision, recall, F1 score, and AUC) and other design considerations (complexity, explainability, and hardware requirement). Evaluation of different hardware (amount of CPU cores) shows that 4 CPUs are optimal for frame processing speed (2.49 seconds per frame) and cost-effectiveness. The results indicate that the model can help alleviate difficulties faced by neurodivergent individuals, such as interpreting social cues and measuring engagement.

Competition history

• ISEF 2025 Behavioral and Social Sciences · Entry BEHA032

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.