Gaze Link: A Multi-language Low-Cost Mobile Eye-Gesture Communication System With Large Language Models for People With Amyotrophic Lateral Sclerosis

Overview

People with Amyotrophic Lateral Sclerosis (PALS), who have severe motor and speech impairments, mostly rely on their eyes and assistive technology to communicate. However, existing high-tech products are expensive and hard to access, while low-tech products are inefficient and restrictive. To mitigate the limitations, this research proposes GazeLink, a low-cost mobile application for PALS to communicate efficiently with only eye movements. First, the system recognizes user eye gestures like left or up with machine learning and a template-matching algorithm. Then, it converts the eye gestures to words through a keyboard that supports English, Spanish, and Chinese. For efficiency, the system employs Large Language Models (LLMs) to generate a suitable sentence with words typed by the user and the context. Finally, the system provides text-to-speech and social media post services for both verbal and digital eye-gesture communication. Simulations conclude that sentence generation with LLMs can improve text-entry rate by 81% while maintaining 90% of semantic similarity. Usability studies with 30 participants show that GazeLink can recognize eye gestures with 94.1% accuracy in varying lighting. After rapidly learning the user interface in under 10 attempts, first-time participants typed sentences of various lengths with their eyes at 15.1 words per minute, which is 7.2x faster than the common low-tech solution E-Tran. Experiments demonstrate GazeLink’s efficiency, learnability, and accuracy in eye-gesture text entry. The system is extremely affordable (<$0.1 a month), portable, and easily accessible online. It also supports different users, lighting, smartphones, and languages. Product testing with PALS and personalized LLM models will be the next step.

Awards (5)

• Second Award of $2,000 $2,000
• Association for the Advancement of Artificial Intelligence: AAAI Student Memberships for each finalist that is part of the 1st, 2nd, and 3rd Prize Winning projects and 5 Honorable Mention winning projects (up to 3 students per project) (in-kind award / part of the 1st-3rd prize)
• Association for the Advancement of Artificial Intelligence: AAAI Membership for the School Libraries of All 8 Winners (in-kind award / part of 1st-3rd prize and honorable mentions' prize)
• Association for the Advancement of Artificial Intelligence: Second Award of $1,000 $1,000
• IEEE Foundation: Third Place Award of $400 $400

Competition history

• ISEF 2024 Systems Software · Entry SOFT043 · Los Angeles, California, United States

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.