Intelligent Face-Controlled Assistive System for Precise Object Manipulation: Assisting Individuals With Upper Limb Motor Impairments

Overview

Paralysis afflicts 15.4 million people globally, growing by 0.5 million annually, with an additional 2.5 million Americans experiencing severe arm mobility impairment, with prosthetics costing patients $30,000-$100,000 and making everyday tasks such as grasping objects overwhelming, leading to devastatingly high depression rates. The project develops an intelligent, highly accurate facially-controlled object handling robotic arm system by integrating AI-based pose estimation, seamless servo-synchronization using trigonometric cosine laws, and user to robotic arm camera coordinate mapping by homography in real time, allowing for complex actions simply by blinking twice at objects for input. Extensive, rigorous testing enabled replacement of sub-optimal solutions, such as cumbersome IMU-systems by precise AI pose-estimation using EAR (Eye Aspect Ratio) rapid blink-detection, and initial >2cm error single dimensional positional-mapping by homography calibration utilizing 6 reference points, enabling consistent near 100% accuracy to 1cm. Results indicate that EAR pose-estimation values of 0.16-0.24 were most accurate as open-eyed values yielded 0.26-0.28, and squinting yielded 0.14-0.16, and that homography outperformed erroneous 1-dimensional mapping systems. This integration of EAR thresholds, servo-control and deterministic homography successfully led to a cost-effective (<$80), facial-input object manipulation system to <1cm error with the intelligent robotic arm working as a wheelchair-mounted prosthetic arm substitute. This system enables fully contactless autonomous performance of virtually all everyday tasks involving object manipulation, enabling those with motor disabilities to live more independently while spending a fraction of what they would traditionally.

Competition history

• ISEF 2026 Embedded Systems · Entry EBED036T

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.