Decoding semantic content of visual stimuli from BOLD fMRI data

In vision, the brain is a feature extractor that works from images. We hypothesize that fMRI can mimic the latent space of a classifier, and employ deep diffusion models with BOLD data from the occipital cortex to generate images which are plausible and semantically close to the visual stimuli administered during fMRI. To this end, we mapped BOLD signals onto the latent space of a pretrained classifier and used its gradients to condition a generative model to reconstruct images. The semantic fidelity of our BOLD response to visual stimulus reconstruction model is superior to the state of the art.