BrainSync: Advanced Neural Decoding With fMRI and EEG for Reconstruction of Visual Perception in Neurological States

Overview

Tens of millions of individuals worldwide, including those with locked-in syndrome, advanced ALS, and coma, are unable to communicate due to motor paralysis. BrainSync introduces a novel multi-modal approach that integrates large-scale 7-tesla functional magnetic resonance imaging (fMRI) signals with dense-array electroencephalography (EEG) data to reconstruct natural visual stimuli for humans or artists, utilizing a proprietary EEG dataset of 46,080 high-density 64-channel recordings alongside synchronized fMRI acquisitions. The system employs advanced generative models to decode neural activity patterns from the striate (V1) and prestriate (V2) cortices. A custom diffusion prior architecture combined with CLIP-based latent space mapping yields reconstruction fidelity of 0.682 PixCorr, 0.778 SSIM, and a Dynamic Temporal Correlation (DTC) score of 0.687—an improvement of 183.7% over previous methods. Its dual-stream pipeline merges real-time EEG (512 Hz) with high-spatial fMRI (0.8mm³ voxels) via a transformer architecture for precise neural alignment while reducing operational costs by 330x ($1M to $3K using EEG). BrainSync enables real-time visualization of internally generated imagery, converting thought or vision into dynamic art. By capturing complex cognitive processes and varying levels of consciousness, the system empowers artists to document the fluidity of human experience. Bidirectional cross-modal attention and adversarial training techniques boost real-world expression accuracy by 267%, positioning BrainSync as a transformative tool that unites neuroscience and expression, redefining art conception and communication. This breakthrough marks a pivotal moment in technology: art integration, medical use, and expanding creative potential for future exploration.

Competition history

• ISEF 2025 Technology Enhances the Arts · Entry TECA012

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.