This repo accompanies the original paper "Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space" (Dado et al., 2022). This study introduces a novel experimental paradigm that uses synthesized yet highly naturalistic stimuli with a priori known feature representations together with an implementation thereof for HYperrealistic reconstruction of PERception (HYPER) of faces from brain recordings. The goal was to reveal what information was present in the recorded brain responses by reconstructing the original faces presented to the participants.
Click here for the blog post / tutorial.
Two participants were presented with face images while we recorded their brain responses in the MRI scanner. After collecting the (faces, responses) dataset, we trained a decoding model to reconstruct what the participants were seeing from their (held-out test set) fMRI recordings alone.
The faces in the presented photographs do not really exist but are artificially generated by a progressiveGAN (PGGAN) from randomly sampled latent vectors. The results suggest that the PGGAN latent space and the neural face manifold must have an approximate linear relationship that can be exploited during brain decoding. That is, the latent vectors used for face generation effectively capture the same defining stimulus features as the fMRI measurements. As such, we can predict the latents that underlie the perceived face images and feed them to the PGGAN for (re)generation, leading to the most accurate reconstructions of perception to date.
🤖🤖🤖
This repo contains a Jupyter Notebook that presents the approach. All required data to reproduce the results, preprocessing steps and test set images (stimuli and reconstructions) are made available on Google Drive.
We used GANs for neural decoding of perception. It should be noted that the results of this study are valid reconstructions of visual perception regardless of the synthetic nature of the stimuli themselves. Considering the speed of progress in the field of generative modeling, this framework will likely result in even more impressive reconstructions of perception. Our approach constitutes a leap forward in our ability to reconstruct percepts from patterns of human brain activity.