Interactive Real-Time Actions Learning System

Overview

Over 5% of the world's population - or 430 million people - require rehabilitation to address their disabling hearing loss. At a personal level, my younger brother has severe difficulties with hearing. This led me to research ways I could help him and his friends in interpreting sign language . My first research step was looking at existing solutions for sign language learning systems and attempt to improve/enhance them. The main disadvantage of current systems is that they are not interactive and don't provide real-time feedback. After analysing computer vision libraries, which included OpenCV, TensorFlow and Mediapipe, I decided to use Mediapipe because of its easily integrated solution for reliably detecting hand movements. I first developed a solution for learning sign language by analysing libraries of computer vision, designing a user interface, and then expanded the application, through rigorous testing using a range of people and evaluating their learning experiences. Having developed a system to help with learning sign language, the aim was then to help other people with specific needs. An example of another use of my algorithm, would be to help people with Alexithymia or ASD, to learn and understand emotions from facial expressions. Consequently, I created another APP for learning Facial Expressions. The real-time accuracy feedback which my system provides, ensures the learning process is engaging, facilitates practice and learning by doing. Overall, this algorithm has the potential to be developed to help people learn other physical movements such as Gross Motor Skills, Phonics, Orthopedic Therapy, etc.

Competition history

• ISEF 2025 Technology Enhances the Arts · Entry TECA014

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.