Synchronome: An Intelligent Music Practice System Providing Real-time Multimodal Feedback for Visually Impaired Musicians

Overview

While technology has revolutionised music education, visually impaired musicians remain marginalised due to the visual-centric design of existing practice platforms. To restore artistic independence, this project developed Synchronome, a novel music practice application integrating real-time audio analysis with auditory-tactile feedback to enable independent accuracy enhancement without visual reliance. The Optical Music Recognition (OMR) mechanism involves converting uploaded PDF scores into an encoded MusicXML structure, enabling pitch, rhythm, and notation data extraction. Incoming user performance audio is analysed in real time using a Digital Signal Processor (DSP), where waveform manipulation algorithms extract fundamental frequencies and determine note onsets for continuous comparison with the expected score data. Upon the detection of user deviations in pitch and rhythm, immediate feedback is delivered through a specialised haptic signal system alongside complementary auditory cues, allowing users to quickly recognise and correct errors. In addition, aggregated performance data are used to create personalised feedback via a generative artificial intelligence model, delivered post-performance through Text-to-Speech (TTS) output. Prototype testing confirmed the system’s ability to analyse and detect pitch and rhythm deviations with low latency, achieving a mean error detection accuracy of 95%. The haptic response system and TTS output also functioned reliably, supporting users in interpreting feedback. These positive results ultimately demonstrate that Synchronome’s multimodal, real-time feedback system not only improves technical music proficiency but fosters an inclusive musical society, proving how technology can reconstruct independent artistic expression.

Awards (1)

• Third Award of $1,200 $1,200

Competition history

• ISEF 2026 Technology Enhances the Arts · Entry TECA010

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.