BrainStorm: Reconstructing Natural Vision from fMRI Using Generative Models for Communication and Covert Awareness in Neurological States

Overview

With an estimated 60,000,000+ individuals worldwide suffering from disabilities such as mutism and over 850,000 coma patients in the United States each year, enhancing communication capacities and deciphering internal cognitive processes becomes paramount to health. BrainStorm aims to reconstruct natural images and videos from functional magnetic resonance imaging (fMRI) signals with a large-scale 7-tesla dataset by harnessing generative models to translate neural activity from the striate, prestriate, and V3 areas of the brain into visual representations. BrainStorm's techniques facilitate real-time visualization of these processes, aiding comatose stage classification and treatment: a vital step towards understanding neurological functions in decades. BrainStorm also facilitates mental health investigation by elucidating the neural correlates associated with suicidal ideation and reconstructions of negative, salient images. The ability to reconstruct images and videos from fMRI using these methods—which integrates novel techniques such as mapping to latent space via CLIP and employing cosine similarity—provides a powerful tool for addressing these matters. Decoding visual stimuli from fMRI signals, however, presents unique challenges including low temporal resolution, high noise, and intricate nonlinear mappings of the data. This study uses a diffusion prior within the generative model to overcome these obstacles. Further, a secondary image and brain retrieval pipeline is integrated alongside stimuli reconstruction, achieving top metrics such as 0.456 PixCorr, 0.493 SSIM, and a 142.4% improvement in real-world expression. This positions BrainStorm as state-of-the-art research that has momentous implications for medical imaging, communication, and neuroscience.

Awards (1)

• Long Island University: Presidential Scholarships

Competition history

• ISEF 2024 Biomedical Engineering · Entry ENBM083 · Los Angeles, California, United States

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.