Live Emotion-Adaptive Backgrounds for Improved Sign Language Translation: An AI-Driven Multimodal Communication System

Overview

Current speech-to-sign language translation systems cannot express the emotions behind the words, making it difficult for deaf and hard-of-hearing individuals to fully grasp the speaker's true intentions, which is a significant communication barrier. The aim of this research was to develop EVA-SL, a novel AI-based system that translates speech into sign language in real-time, while simultaneously visualizing the speaker's emotional state. The system integrates OpenAI's Whisper for speech transcription, GPT-based classification for emotion detection, and Librosa for pitch and tone analysis. Detected emotions and pitch intensity are mapped to a dynamic background color using a Hue-Lightness-Saturation (HLS) model, with color intensity increasing with emotional strength, providing an intuitive visual cue grounded in psychological color-emotion research. The system supports both American and Saudi Sign Languages. To evaluate its effectiveness, 25 participants from the deaf and hard-of-hearing community completed a structured 15-question Likert-scale survey and two real-time interaction tasks involving live speech scenarios. The emotion-to-color feature received a satisfaction score of 4.31 out of 5.0. Furthermore, 88% of participants reported that the adaptive colors helped them understand the emotional intent behind spoken words, and 92% valued the integration of translation and emotion visualization in a single system. This research demonstrates that emotion-adaptive color visualization enables deeper empathy and more complete communication for deaf users. EVA-SL has direct real-world applications in classrooms, hospitals, and public service settings where emotionally accurate communication between hearing and deaf individuals is essential.

Competition history

• ISEF 2026 Technology Enhances the Arts · Entry TECA015

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.