Multimodal Music: Inferring Musical Emotion through Non-auditory Cues

Overview

This study investigates musical emotion identification using non-auditory senses alone. Guided by the valence-arousal model of emotion and multimodal integration theory, we designed a system that translates musical features into synchronized visual and tactile stimuli. The system was tested using songs categorized into four emotions: happy, angry, calm, and sad on the valence-arousal model. Participants rated arousal and valence after experiencing the multimodal device or an audio-only control. 692 samples were collected from 125 different participants. Users with the device distinguished emotions with 70% accuracy. Additionally, 72% of arousal separation and 52% of valence separation were retained in non-auditory experiences, improving upon state-of-the-art haptic emotion instilling methods. These results demonstrate that emotional perception is not modality-restricted, but instead can emerge through multisensory experience, embodied emotion, and crossmodal cognition, providing new insight into non-auditory musical stimuli and enabling accessible music experiences.

Competition history

• CSEF 2026 Behavioral & Social Sciences (Senior Division) · Entry S-03-25

Related projects

Closest projects by meaning, across every fair and year in the corpus.

Browse more like this

• All Behavioural and Social Sciences projects
• CSEF Behavioural and Social Sciences projects
• Every CSEF 2026 project