Multimodal Foundation Models for Understanding Bodily Expressed Emotions

Overview

Humans inherently communicate through body movement. Therefore bodily expressions serve as a powerful but often subconscious channel for emotion and social signaling. Despite progress in visual-language understanding, whether large multimodal foundation models can recognize emotions and expressive intent from full-body motion remains underexplored. In this paper, we present an empirical study evaluating the capability of visual foundation models to understand bodily expressed emotions through the lens of dance. We choose dance as it is a domain rich in intentional movement, affective cues, and embodied semantics. We examine whether the Gemini 2.5 family models can infer dance genre and associate body motions with its corresponding musical accompaniment using the AIST++ dataset. Our results show that Gemini 2.5 Pro achieves 57.14% genre classification accuracy in the video-only setting, outperforming other variants and demonstrating possible paths to understanding of affective bodily motion. However, cross-modal coherence remains limited. The video-to-audio matching accuracy is only 19.52%, which suggests that foundation models still struggle to form robust mappings between movement and emotions. We hope that our findings provide an early exploration of bodily expression understanding in foundation models.We also discuss challenges and opportunities towards embodied emotion modeling in general-purpose AI.

Competition history

• CSEF 2026 Mathematical Sciences (Junior Division) · Entry J-14-08

Related projects

Closest projects by meaning, across every fair and year in the corpus.

Browse more like this

• All Mathematics projects
• CSEF Mathematics projects
• Every CSEF 2026 project