Autonomous Wheelchair for Mobility and Communication Assistance for ALS Patients

Overview

Amyotrophic Lateral Sclerosis (ALS) patients with severe motor and speech impairments often rely on eye movements and assistive technology to move and communicate. However, current smart wheelchairs are unable to simultaneously address both mobility and communication needs. This project introduces a novel autonomous wheelchair system that integrates advanced technologies to tackle these two challenges, specifically tailored to the unique needs of ALS patients. Firstly, the patient interacts on a laptop interface via pupil movements by a novel eye-tracking subsystem utilizing YOLO11 for pupil detection and polynomial regression for gaze estimation, enabling virtual mouse control. Secondly, the system offers patients the choice between two functions: mobility and communication. For mobility, the system provides two options: manual control, which emulates joystick signals for eight directions, and an autonomous mode. The autonomous mode employs the SLAM Toolbox for mapping and Nav2 for localization, path planning, and navigation with an adjusted DWB controller to enable automatic reverse maneuvers in confined spaces. For communication, the system combines an eye-operated virtual keyboard with a fine-tuned Gemma2 language model that converts discrete word selections into 3 distinct sentences, supporting both speech output and Telegram messaging. Additionally, a user-friendly GUI application integrates all functions on the laptop interface. Finally, testing demonstrated 97.11% accuracy in the gaze point processing, 0.12 meters average path-following deviation, 93.33% success in autonomous navigation, and sentence generation in under 5.33 seconds. This solution represents the first fully integrated mobility-communication system engineered for ALS patients.

Competition history

• ISEF 2025 Robotics and Intelligent Machines · Entry ROBO065T

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.