Hearing Is Discovering: Audio Augmented Reality for Celestial Objects Through Music-Theoretic and Spatial Sonification

Overview

This project advances audio augmentation in augmented reality (AR) by developing a music-theoretic spatial sonification method. Although AR is increasingly used in education and entertainment, most research and applications rely almost exclusively on visual augmentation, with limited meaningful audio. Using astronomy as a testbed, this project addresses the gap by augmenting celestial objects such as constellations with musical representations derived from their astronomical features (e.g., position on the celestial sphere, size, brightness, and local deep-sky objects). By translating astronomical data into musically coherent compositions based on music-theoretic principles (e.g., tonality and functional harmony), the method allows learners to perceive and explore celestial objects through auditory modalities. An accompanying AR device extends this experience beyond the classroom by supporting immersive interaction with celestial objects under the night sky through a virtual spatial soundscape. The device performs real-time object identification and audio-guided search by rendering 3D sounds that appear to originate from celestial objects. A user study with 116 participants shows that the method is effective even for novices, with consistently high accuracy across recognition tasks (92-93%). Participants also exhibit a 68 percentage-point increase in perceived connection between constellations and their music, along with strong engagement, increased motivation, and perceived educational value of multimodal learning. Field testing demonstrates the feasibility of the integrated system, including reliable constellation identification and search, responsive spatial audio rendering, and immersive audio-guided exploration under nighttime observation conditions.

Awards (2)

• Fourth Award of $600 $600
• Association for Computing Machinery: Fourth Award of $500 $500

Competition history

• ISEF 2026 Software Design · Entry SFTD061

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.