/AIGC-Brain

Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy

Primary LanguageTeXApache License 2.0Apache-2.0

AIGC-Brain

Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy

This project is associated with our survey paper which comprehensively examines the emerging field of AIGC-based brain-conditional multimodal synthesis, termed AIGC-Brain, to delineate the current landscape and future directions.

arXiv Maintenance PR's Welcome GitHub license


Brain-Conditional Multimodal Synthesis via AIGC-Brain Decoder. Sensory stimuli comprising visual stimuli (Image (I), Video (V)) and audio stimuli (Music (M), Speech/Sound (S)) from the external world are first encoded to non-invasive brain signals (EEG, fMRI, or MEG) and then decoded back to perceptual experience via the AIGC-Brain decoder. This survey focuses on passive brain-conditional multimodal synthesis tasks including Image-Brain-Image (IBI), Video-Brain-Video (VBV), Sound-Brain-Sound (SBS), Music-Brain-Music (MBM), Image-Brain-Text (IBT), Video-Brain-Text (VBT), and Speech-Brain-Text (SBT), where IBI refers to image synthesis tasks conditioned on brain signals evoked by image stimuli.

Qualitative Results for AIGC-Brain Tasks. A: IBI results on GOD (left) and EEG-VOA (right) datasets; B: IBI and IBT results on NSD dataset; C: VBV results on DNV dataset; D: VBT (top) and SBT (bottom) results based on CLSR; E: MBM results based on Brain2Music; F: SBS results based on BSR.

Table of Contents (Work in Progress)

AIGC-Brain Tasks & Implementations:

Image-Brain-Image

IBI-fMRI

DREAM: Visual Decoding from Reversing Human Visual System
Weihao Xia, Raoul de Charette, Cengiz Öztireli, Jing-Hao Xue
arXiv 2023.10 [Paper] [Code] [Project]
Dataset [NSD]

(BrainSD) High-resolution image reconstruction with latent diffusion models from human brain activity
Yu Takagi, Shinji Nishimoto
CVPR 2023 [Paper] [Code] [Project]
Dataset [NSD]

(BrainSD-TGD) Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs
Yu Takagi, Shinji Nishimoto
arxiv 2023 [Paper] [Code] [Project]
Dataset [NSD]

(BrainDiffuser) Natural scene reconstruction from fMRI signals using generative latent diffusion
Furkan Ozcelik, Rufin VanRullen
Scientific Reports 2023 [Paper] [Code]
Dataset [NSD]

MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion
Yizhuo Lu, Changde Du, Dianpeng Wang, Huiguang He
ACM MM 2023 [Paper] [Code]
Dataset [NSD]

Mind Reader: Reconstructing complex images from brain activities
Sikun Lin, Thomas Sprague, Ambuj K Singh
NeurIPS 2022 [Paper] [Code]
Dataset [NSD]

(MindEye) Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors
Paul S. Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan J. Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth A. Norman, Tanishq Mathew Abraham
arxiv 2023 [Paper] [Code] [Project]
Dataset [NSD]

BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding
Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, Nanning Zheng
arxiv 2023 [Paper] [Code]
Dataset [NSD] [GOD]

(BrainSCN) Decoding natural image stimuli from fMRI data with a surface-based convolutional network
Zijin Gu, Keith Jamison, Amy Kuceyeski, Mert Sabuncu
MIDL 2023 [Paper] [Code]
Dataset [NSD]

(MindVis) Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, Juan Helen Zhou
CVPR 2023 [Paper] [Code] [Project]
Dataset [HCP] [GOD] [BLOD]

(LEA) Joint fMRI Decoding and Encoding with Latent Embedding Alignment
Xuelin Qian, Yikai Wang, Yanwei Fu, Xinwei Sun, Xiangyang Xue, Jianfeng Feng
arxiv 2023 [Paper]
Dataset [HCP] [GOD] [BLOD]

(CMVDM) Controllable Mind Visual Diffusion Model
Bohan Zeng, Shanglin Li, Xuhui Liu, Sicheng Gao, Xiaolong Jiang, Xu Tang, Yao Hu, Jianzhuang Liu, Baochang Zhang
AAAI 2024 [Paper] [Code]
Dataset [HCP] [GOD] [BLOD]

(CAD) Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities
Jingyuan Sun, Mingxiao Li, Zijiao Chen, Yunhao Zhang, Shaonan Wang, Marie-Francine Moens
NeurIPS 2023 [Paper] [Code]
Dataset [HCP] [GOD] [BLOD]

(BrainICG) Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs
Furkan Ozcelik, Bhavin Choksi, Milad Mozafari, Leila Reddy, Rufin VanRullen
IJCNN 2022 [Paper] [Code]
Dataset [GOD]

(DBDM) Dual-Guided Brain Diffusion Model: Natural Image Reconstruction from Human Visual Stimulus fMRI
Lu Meng, Chuanhao Yang
Bioengineering 2023 [Paper]
Dataset [GOD]

(VQ-fMRI) Rethinking Visual Reconstruction: Experience-Based Content Completion Guided by Visual Cues
Jiaxuan Chen, Yu Qi, Gang Pan
ICML 2023 [Paper]
Dataset [GOD]

(BrainHVAE) Generative Decoding of Visual Stimuli
Eleni Miliotou, Panagiotis Kyriakis, Jason D Hinman, Andrei Irimia, Paul Bogdan
ICML 2023 [Paper]
Dataset [GOD]

(BrainHSG) Semantics-guided hierarchical feature encoding generative adversarial network for natural image reconstruction from brain activities
Lu Meng, Chuanhao Yang
IJCNN 2023 [Paper]
Dataset [GOD]

(SBD) Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimuli
Matteo Ferrante, Tommaso Boccato, Nicola Toschi
arxiv 2023 [Paper] [Code]
Dataset [GOD]

(SSNIR) From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI
Roman Beliy, Guy Gaziv, Assaf Hoogi, Francesca Strappini, Tal Golan, Michal Irani
NeurIPS 2019 [Paper] [Project]
[Code]
Dataset [GOD] [Vim-1]

(SSNIR-SC) Self-supervised Natural Image Reconstruction and Large-scale Semantic Classification from Brain Activity
Guy Gaziv, Roman Beliy, Niv Granot, Assaf Hoogi, Francesca Strappini, Tal Golan, Michal Irani
NeuroImage 2022 [Paper] [Code]
Dataset [GOD] [Vim-1]

(SSDR) More Than Meets the Eye: Self-Supervised Depth Reconstruction From Brain Activity
Guy Gaziv, Michal Irani
arXiv 2021 [Paper] [Code]
Dataset [GOD] [Vim-1]

(BrainSSG) Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GAN
Tao Fang, Yu Qi, Gang Pan
NeurIPS 2020 [Paper] [Code]
Dataset [GOD]]

(BrainBBG) Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN
Milad Mozafari, Leila Reddy, Rufin VanRullen
IJCNN 2020 [Paper]
Dataset [GOD]

(BrainDVG) Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning
Ziqi Ren, Jie Li, Xuetong Xue, Xin Li, Fan Yang, Zhicheng Jiao, Xinbo Gao
NeuroImage 2021 [Paper]
Dataset [GOD] [BRAINS] [BCP] [6-9]

(BrainDCG) Generative adversarial networks for reconstructing natural images from brain activity
K. Seeliger, U. Güçlü, L. Ambrogioni, Y. Güçlütürk, M.A.J. van Gerven
NeuroImage 2018 [Paper] [Code]
Dataset [GOD] [Vim-1] [BRAINS]

(DIR) Deep image reconstruction from human brain activity
Guohua Shen, Tomoyasu Horikawa, Kei Majima, Yukiyasu Kamitani
PLOS Computational Biology 2019 [Paper] [Code]
Dataset [DIR]

(E-DIR) End-to-End Deep Image Reconstruction From Human Brain Activity
Guohua Shen, Kshitij Dwivedi, Kei Majima, Tomoyasu Horikawa, Yukiyasu Kamitani
Frontiers in Computational Neuroscience 2019 [Paper]
Dataset [DIR]

BigGAN-based Bayesian Reconstruction of Natural Images from Human Brain Activity
Kai Qiao, Jian Chen, Linyuan Wang, Chi Zhang, Li Tong, Bin Yan
Neuroscience 2020 [Paper]
Dataset [Vim-1]

(Faces) Reconstructing faces from fMRI patterns using deep generative neural networks
Rufin VanRullen, Leila Reddy
Communications Biology 2019 [Paper] [Code]
Dataset [Faces]

IBI-EEG

DreamDiffusion: Generating High-Quality Images from Brain EEG Signals
Yunpeng Bai, Xintao Wang, Yan-pei Cao, Yixiao Ge, Chun Yuan, Ying Shan
arxiv 2023 [Paper] [Code]
Dataset [EEG-VOA]

(NeuroImagen) Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals
Yu-Ting Lan, Kan Ren, Yansen Wang, Wei-Long Zheng, Dongsheng Li, Bao-Liang Lu, Lili Qiu
arxiv 2023 [Paper]
Dataset [EEG-VOA]

DM-RE2I: A framework based on diffusion model for the reconstruction from EEG to image
Hong Zeng, Nianzhang Xia, Dongguan Qian, Motonobu Hattori, Chu Wang, Wanzeng Kong
BSPC 2023 [Paper]
Dataset [EEG-VOA]

Brain2Image: Converting Brain Signals into Images
Isaak Kavasidis, Simone Palazzo, Concetto Spampinato, Daniela Giordano, Mubarak Shah
ACM MM 2017 [Paper]
Dataset [EEG-VOA]

NeuroVision: perceived image regeneration using cProGAN
Sanchita Khare, Rajiv Nayan Choubey, Loveleen Amar, Venkanna Udutalapalli
Neural Computing and Applications 2022 [Paper]
Dataset [EEG-VOA]

(EEG-VGD) Decoding EEG by Visual-guided Deep Neural Networks
Zhicheng Jiao, Haoxuan You, Fan Yang, Xin Li, Han Zhang, Dinggang Shen
IJCAI 2019 [Paper]
Dataset [EEG-VOA]

(EEG-GAN) Generative Adversarial Networks Conditioned by Brain Signals
Simone Palazzo, Concetto Spampinato, Isaak Kavasidis, Daniela Giordano, Mubarak Shah
ICCV 2017 [Paper]
Dataset [EEG-VOA]

IBI-MEG

(MEG-BD) Brain decoding: toward real-time reconstruction of visual perception
Yohann Benchetrit1, Hubert Banville1, Jean-Remi King
arXiv 2023 [Paper]
Dataset [MEG-Things]

Image-Brain-Image&Text

UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity
Weijian Mai, Zhijun Zhang
arxiv 2023 [Paper]
Dataset [NSD]

Brain Captioning: Decoding human brain activity into images and text
Matteo Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, Nicola Toschi
arxiv 2023 [Paper]
Dataset [NSD]

Video-Brain-Video

(Mind-Video) Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity
Zijiao Chen, Jiaxin Qing, Juan Helen Zhou
arxiv 2023 [Paper] [Code] [Project]
Dataset [DNV] [HCP]

(SSRNM) A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activity
Ganit Kupershmidt, Roman Beliy, Guy Gaziv, Michal Irani
arxiv 2022 [Paper] [Project]
Dataset [DNV]

(f-CVGAN) Reconstructing rapid natural vision with fMRI-conditional video generative adversarial network
Chong Wang, Hongmei Yan, Wei Huang, Jiyi Li, Yuting Wang, Yun-Shuang Fan, Wei Sheng, Tao Liu, Rong Li, Huafu Chen
Cerebral Cortex 2022 [Paper]
Dataset [DNV] [HCP]

(BrainViVAE) Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex
Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, Zhongming Liu
NeuroImage 2019 [Paper]
Dataset [DNV]

(DNV) Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision
Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, Zhongming Liu
Cerebral Cortex 2018 [Paper] [Data]
Dataset [DNV]

Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity
Lynn Le, Luca Ambrogioni, Katja Seeliger, Yağmur Güçlütürk, Marcel van Gerven, Umut Güçlü
Frontiers in Neuroscience 2022 [Paper] [Code]
Dataset [STNS]

Sound-Brain-Sound

(BSR) Sound reconstruction from human brain activity via a generative model with brain-like auditory features
Jong-Yun Park, Mitsuaki Tsukamoto, Misato Tanaka, Yukiyasu Kamitani
arXiv 2023 [Paper] [Code]
Dataset [BSR]

(ETCAS) End-to-end translation of human neural activity to speech with a dual–dual generative adversarial network
Yina Guo, Ting Liu, Xiaofei Zhang, Anhong Wang, Wenwu Wang
Knowledge-Based Systems 2023 [Paper] [Code]
Dataset [ETCAS]

Music-Brain-Music

Brain2Music: Reconstructing Music from Human Brain Activity
Timo I. Denk, Yu Takagi, Takuya Matsuyama, Andrea Agostinelli, Tomoya Nakai, Christian Frank, Shinji Nishimoto
arxiv 2023 [Paper] [Project] [Data-MusicCaption]
Dataset [MusicGenre]

(NDMusic) Neural decoding of music from the EEG
Ian Daly
Scientific Reports 2023 [Paper]
Dataset [MusicAffect]

Image&Video&Speech-Brain-Text

Image-Brain-Text

DreamCatcher: Revealing the Language of the Brain with fMRI using GPT Embedding
Subhrasankar Chatterjee, Debasis Samanta
arxiv 2023 [Paper]
Dataset [NSD]

(GIC-RL) Generation of Viewed Image Captions From Human Brain Activity Via Unsupervised Text Latent Space
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
ICIP 2020 [Paper]
Dataset [GOD]

(GIC-PTL) A neural decoding algorithm that generates language from visual activity evoked by natural images
Wei Huang, Hongmei Yan, Kaiwen Cheng et al.
Neural Networks 2021 [Paper]
Dataset [OCD]

(GIC-CT) A CNN-transformer hybrid approach for decoding visual neural activity into text
Jiang Zhang, Chen Li, Ganwanming Liu et al.
Computer Methods and Programs in Biomedicine 2022 [Paper]
Dataset [OCD]

(DSR) Describing Semantic Representations of Brain Activity Evoked by Visual Stimuli
Subhrasankar Chatterjee, Debasis Samanta
ICSMC 2018 [Paper]
Dataset [VER]

(GNLD) Generating Natural Language Descriptions for Semantic Representations of Human Brain Activity
Eri Matsuo, Ichiro Kobayashi, Shinji Nishimoto, Satoshi Nishida, Hideki Asoh
ACL 2016 student research workshop 2016 [Paper]
Dataset [VER]

Speech-Brain-Text

UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language
Nuwa Xi, Sendong Zhao, Haochun Wang, Chi Liu, Bing Qin, Ting Liu
arxiv 2023 [Paper]
Dataset [Narratives]

Video&Speech-Brain-Text

(CLSR) Semantic reconstruction of continuous language from non-invasive brain recordings
Jerry Tang, Amanda LeBel, Shailee Jain, Alexander G. Huth
Nature Neuroscience 2023 [Paper] [Code]
Dataset [CLSR]

Active Tasks

Brain-to-Image

ThoughtViz: Visualizing Human Thoughts Using Generative Adversarial Network
Praveen Tirupattur, Yogesh Singh Rawat, Concetto Spampinato, Mubarak Shah
ACM MM 2018 [Paper]
Dataset [EEG-Imagery]

NeuroGAN: image reconstruction from EEG signals via an attention-based GAN
Rahul Mishra, Krishan Sharma, R. R. Jha, Arnav Bhavsar
Neural Computing and Applications 2023 [Paper]
Dataset [EEG-Imagery]

EEG2IMAGE: Image Reconstruction from EEG Brain Signals
Prajwal Singh, Pankaj Pandey, Krishna Miyapuram, Shanmuganathan Raman
ICASSP 2023 [Paper] [Code]
Dataset [EEG-Imagery]

Brain-to-Text

Dewave: Discrete eeg waves encoding for brain dynamics to text translation
Yiqun Duan, Jinzhao Zhou, Zhen Wang, Yu-Kai Wang, Chin-Teng Lin
NeurIPS 2023 [Paper]

Invasive Tasks

Brain-to-Speech

Speech synthesis from neural decoding of spoken sentences
Gopala K. Anumanchipalli, Josh Chartier & Edward F. Chang
Nature 2019 [Paper]

Brain-to-Music

Music can be reconstructed from human auditory cortex activity using nonlinear decoding models
Ludovic Bellier, Anaïs Llorens, et al.
PLOS Biology 2023 [Paper]

Neuroimaging Dataset

fMRI-Pretrain

(HCP) The WU-Minn Human Connectome Project: An overview
David C. Van Essen a, Stephen M. Smith b, Deanna M. Barch c, Timothy E.J. Behrens b, Essa Yacoub d, Kamil Ugurbil d, for the WU-Minn HCP Consortium
NeuroImage 2013 [Paper] [Project]

(MOABB) MOABB: trustworthy algorithm benchmarking for BCIs
Vinay Jayaram, Alexandre Barachant
Journal of Neural Engineering 2018 [Paper] [Project]

fMRI-Image

(NSD) A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence
Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Naselaris & Kendrick Kay
Nature Neuroscience 2021 [Paper] [Project]

(GOD) Generic decoding of seen and imagined objects using hierarchical visual features
Tomoyasu Horikawa, Yukiyasu Kamitani
Nature Communications 2017 [Paper] [Code]

(BOLD) BOLD5000, a public fMRI dataset while viewing 5000 visual images
Nadine Chang, John A. Pyles, Austin Marcus, Abhinav Gupta, Michael J. Tarr & Elissa M. Aminoff
Scientific Data 2019 [Paper] [Project]

(DIR) Deep image reconstruction from human brain activity
Guohua Shen, Tomoyasu Horikawa, Kei Majima, Yukiyasu Kamitani
PLOS Computational Biology 2019 [Paper] [Code]

(Vim-1) Identifying natural images from human brain activity
Kendrick N. Kay, Thomas Naselaris, Ryan J. Prenger, Jack L. Gallant
Nature 2008 [Paper]

(Faces) Reconstructing faces from fMRI patterns using deep generative neural networks
Rufin VanRullen, Leila Reddy
Communications Biology 2019 [Paper] [Code] [Data]

(OCD) Long short-term memory-based neural decoding of object categories evoked by natural images
Wei Huang, Hongmei Yan, Chong Wang et al.
Human Brain Mapping 2020 [Paper] [Data]

(BRAINS) Linear reconstruction of perceived images from human brain activity
Sanne Schoenmakers, Markus Barth, Tom Heskes, Marcel van Gerven
NeuroImage 2013 [Paper]

(BCP) Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders
Yoichi Miyawaki, Hajime Uchida, Okito Yamashita et al.
Neuron 2008 [Paper]

(6-9) Neural Decoding with Hierarchical Generative Models
Marcel A. J. van Gerven, Floris P. de Lange, Tom Heskes
Neural Computation 2010 [Paper]

EEG-Image

(EEG-VOA) Deep Learning Human Mind for Automated Visual Classification
Concetto Spampinato, Simone Palazzo, Isaak Kavasidis, Daniela Giordano, Nasim Souly, Mubarak Shah
CVPR 2017 [Paper] [Code]

(EEG-Things) A large and rich EEG dataset for modeling human visual object recognition
Alessandro T. Gifford, Kshitij Dwivedi, Gemma Roig, Radoslaw M. Cichy
NeuroImage 2022 [Paper] [Code] [Data]

(EEG-Imagery) Envisioned speech recognition using EEG sensors
Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Pawan Kumar Sahu, Debi Prosad Dogra
Personal and Ubiquitous Computing 2018 [Paper]

MEG-Image

(MEG-Things) THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior
Martin N Hebart, Oliver ContierLina, TeichmannAdam et al.
Elife 2023 [Paper] [Project]

fMRI-Video

(DNV) Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision
Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, Zhongming Liu
Cerebral Cortex 2018 [Paper] [Data]

(VER) Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies
Shinji Nishimoto, An T. Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, Jack L. Gallant
Current Biology 2011 [Paper]

fMRI-Video&Speech

(STNS) A large single-participant fMRI dataset for probing brain responses to naturalistic stimuli in space and time
K.Seeliger, R.P.Sommers, U.Güçlü, S.E.Bosch, M.A.J.van Gerven
bioRxiv 2019 [Paper] [Project]

(CLSR) Semantic reconstruction of continuous language from non-invasive brain recordings
Jerry Tang, Amanda LeBel, Shailee Jain, Alexander G. Huth
Nature Neuroscience 2023 [Paper] [Code] [Data-Train-Speech] [Data-Test-SpeechVideo]

fMRI-Sound&Speech

(BSR) Sound reconstruction from human brain activity via a generative model with brain-like auditory features
Jong-Yun Park, Mitsuaki Tsukamoto, Misato Tanaka, Yukiyasu Kamitani
arXiv 2023 [Paper] [Code]
Dataset [BSR]

(ETCAS) End-to-end translation of human neural activity to speech with a dual–dual generative adversarial network
Yina Guo, Ting Liu, Xiaofei Zhang, Anhong Wang, Wenwu Wang
Knowledge-Based Systems 2023 Dataset [ETCAS]
[Paper] [Code]

(Narratives) The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension
Samuel A. Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, et al.
Scientific Data 2021 [Paper] [Code]

fMRI-Music

(MusicGenre) Music genre neuroimaging dataset
Tomoya Nakai, Naoko Koide-Majima, Shinji Nishimoto
Data in Brief 2022 [Paper] [Data]

fMRI&EEG-Music

(MusicAffect) Neural and physiological data from participants listening to affective music
Ian Daly, Nicoletta Nicolaou, et al.
Scientific Data 2020 [Paper] [Data-fMRI&EEG] [Data-EEG]

Related Surveys & Projects

Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
Subba Reddy Oota, Manish Gupta, Raju S. Bapi, Gael Jobard, Frederic Alexandre, Xavier Hinaut
arxiv 2023 [Paper]

Natural Image Reconstruction From fMRI Using Deep Learning: A Survey
Zarina Rakhimberdina, Quentin Jodelet, Xin Liu, Tsuyoshi Murata
Frontiers in Neuroscience 2021 [Paper]

Multimodal Image Synthesis and Editing: A Survey and Taxonomy
Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Shijian Lu§, Lingjie Liu, Adam Kortylewski, Christian Theobalt, Eric Xing
TPAMI 2023 [Paper] [Code]

Vision + Language Applications: A Survey
Yutong Zhou, Nobutaka Shimada
CVPRW 2023 [Paper] [Code]

Our Work

UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity
Weijian Mai, Zhijun Zhang
arxiv 2023 [Paper]
Dataset [NSD]

Citation

If you use this project for your research, please cite our papers.

@article{mai2023brain,
  title={Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy},
  author={Mai, Weijian and Zhang, Jian and Fang, Pengfei and Zhang, Zhijun},
  journal={arXiv preprint arXiv:2401.00430},
  year={2023}
}