Mouseless Mouse: Enhancing Digital Interaction for Individuals With Limited Hand Mobility via Amplify and Smooth Extracted Data From Head Movement Using Real-Time Facial Tracking and Deep Learning

Overview

Individuals with limited hand mobility usually face challenges in daily activities and technology interactions that require fine motor skills and dexterity. To address this issue, innovative technology has been developed for interaction with machines and controlling a mouse with head movements. Using a laptop's camera for commands eliminates the need for additional hardware for accurate control. The proposed system tackles the problem of limited input options while also minimizing erratic movements for people with neck movement restrictions. The system uses face mesh technology to map facial features accurately and translate head movements into cursor movements, and the data is amplified to fill the entire screen while maintaining comfortable movement. Exponential moving average (EMA) was used to improve cursor control by giving more weight to recent data points, thus reducing jitter. Because of the latency caused by EMA, Kalman filter is utilized for its predictive capabilities, which significantly enhance the system's responsiveness and enable more accurate real-time cursor movements. Due to limitations in actions, eye wink detection utilizes the eye-aspect ratio (EAR) to accurately measure eye wink representation. A specialized Vision in Transformer (ViT) model for detecting facial gestures has been developed. By implementing a two-phase fine-tuning approach. The model is adapted to function under various lighting conditions. By amplifying and smoothing data, the project reduced movement deviation. Lessening jittery movement. Utilizing ViT enhances real-time classification accuracy by expanding command options. Using a laptop's camera for accurate interaction aids individuals with limited hand mobility, benefiting all users and promoting inclusivity in technology.

Awards (2)

• Third Award of $1,000 $1,000
• University of Arizona: Renewal Tuition Scholarship

Competition history

• ISEF 2024 Systems Software · Entry SOFT045T · Los Angeles, California, United States

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.