Just Look, Don't Type: Enabling Intent-Level Communication Through Gaze Grounding and Disambiguation

Overview

Gaze keyboards require people with ALS to spell words one letter at a time, a process that is slow, exhausting, and forces users to translate simple thoughts into tedious typing sequences. Even with modern predictive text, communication remains centered on character entry rather than human meaning. This project presents a system that replaces letter-by-letter typing with direct object-gaze intent (looking at real objects, not a keyboard), treating a look as an expression of purpose rather than a mere cursor. The system is made feasible by addressing two fundamental challenges: gaze grounding and gaze disambiguation. Gaze grounding is achieved by feeding the live scene and the user’s fixation into a structured vision-language inference step that converts a glance into scene-referenced semantic evidence and a small set of plausible intent hypotheses. Disambiguation is handled by a context-aware reasoning step that combines situational signals such as device state, time, and ambient cues with interaction history to rank the candidate intents and select the best match for that moment and user. When multiple interpretations remain genuinely valid, an intuitive two-stage radial wheel enables the user to clarify their intent with a single additional glance, while the system can auto-select when certainty is high. By shifting gaze-based communication from spelling words to expressing meaning through looking, this work reframes gaze interaction around intent rather than text entry. It demonstrates how scene-grounded perception, context-aware reasoning, and ambiguity-aware interface design can work together to make gaze communication more natural, less burdensome, and better aligned with how people actually intend to communicate.

Awards (2)

Second Award of $2,400 $2,400
Association for the Advancement of Artificial Intelligence: AAAI Student Memberships for each finalist that is part of the 1st, 2nd, and 3rd Prize Winning projects and 5 Honorable Mention winning projects (up to 3 students per project) (in-kind award / part of the 1st-3rd prize)