Feel The Ball: Convert Ball Motion to Touch for Vision/Hearing - Impaired Sport Audiences

Overview

My research aims to develop an innovative system that makes sporting events more accessible and engaging to sports enthusiasts with vision or hearing impairments. Unlike existing captioning-based (i.e., see the real-time commentary) or audio-based (i.e., hear the ball) assistance methods, my system enables users to feel the spatial, dynamic ball movements. The architecture of this system includes extracting real-time ball motion information from broadcast sports videos using computer vision and artificial intelligence technologies and transforming it into dynamic touch feeling through a force feedback device (so-called haptics techniques). Specifically, a novel Swin Transformer combined with DeconvNet and Long Short-Term Memory techniques are developed to achieve reliable tracking of small, blurred, fast-moving balls from broadcast videos. Then a novel psychology-guided haptic technique is developed to enable the user to feel the spatiotemporal ball motion through a force-feedback joystick. This system helps visual or hearing-impaired spectators have a more immersive accessibility to live sports and thus experience the excitement accompanying a sports match. Additionally, this system will heighten people’s sense of reality and spatial awareness and potentially change the way people interact with the real world, advancing future immersive technologies in various fields requiring sensory augmentation. Besides personal use, the fundamental technology innovation in reliable tracking of small and fast-moving objects and intuitive haptics of real- world object motion has broader impact on other applications such as video surveillance and airport safety.

Competition history

• ISEF 2024 Systems Software · Entry SOFT009 · Los Angeles, California, United States

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.