NeuSpeech/awesome-brain-decoding

collection of awesome research in brain decoding, including interaction with multi-modalities, theories, and foundation models.

Awesome Brain decoding

A curated collection of resources in brain decoding, including interaction with multi-modalities, theories, and foundation models.

Contributing

Feedback and contributions are welcome! If you think I have missed out on something (or) have any suggestions (papers, implementations and other resources), feel free to raise issues and pull requests.

**Here is the Paper Name.**<br>
*[Author 1](homepage), Author 2, and Author 3.*<br>
Short description
Conference or Journal Year. [[PDF](link)] [[Project](link)] [[Code](link)] [[Data](link)]

Table of Contents

• Brain to text
• Brain to speech
• Brain to music
• Brain-to-image
• Brain-to-video
• Foundation models
• Theories
• Team and People
• Datasets

Brain to text

EEG

Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification
Zhenhailong Wang, Heng Ji
AAAI 2022 . Convert EEG signal within each time frame of reading word to token, and translate tokens to text in open vocabulary condition. The start of this open-vocabulary field. [PDF] [ZuCo1] [ZuCo2] [Code]

Are EEG-to-Text Models Working?
Hyejeong Jo*, Yiqian Yang*, Juhyeok Han, Yiqun Duan, Hui Xiong, Won Hee Lee
IJCAI workshop 2024 . Investigated major flaws in EEG-to-text models. Compared model performance on pure noise inputs. [PDF] [Data-ZuCo1] [Data-ZuCo2] [Code]

MEG

NeuSpeech: Decode Neural signal as Speech
Yiqian Yang*, Yiqun Duan*, Qiang Zhang, Hyejeong Jo, Jinni Zhou, Won Hee Lee, Renjing Xu, Hui Xiong
Arxiv 2024 . [PDF] [MEG Data-Gwilliams] [MEG Data-Schoffelen] [Code]

MAD:Multi-Alignment MEG-to-Text Decoding
Yiqian Yang*, Hyejeong Jo*, Yiqun Duan* Qiang Zhang, Jinni Zhou, Won Hee Lee, Renjing Xu, Hui Xiong
Arxiv 2024 . [PDF] [MEG Data-Gwilliams] [Code]

ECoG

High-performance brain-to-text communication via handwriting
Francis R. Willett, Donald T. Avansino, Leigh R. Hochberg, Jaimie M. Henderson, Krishna V. Shenoy
Nature 2021. [PDF] [Code] [Data-Willett2021]

A high-performance speech neuroprosthesis
Francis R. Willett, Erin M. Kunz, Chaofei Fan, Donald T. Avansino, Guy H. Wilson, Eun Young Choi, Foram Kamdar, Matthew F. Glasser, Leigh R. Hochberg, Shaul Druckmann, Krishna V. Shenoy, Jaimie M. Henderson
Decode ecog signal with RNN network. Signal->phonemes with RNN and plus a language model to get word. Nature 2023. [PDF] [Code] [Data-Willett2023]

Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis
Sean L. Metzger, Jessie R. Liu, David A. Moses, Maximilian E. Dougherty, Margaret P. Seaton, Kaylo T. Littlejohn, Josh Chartier, Gopala K. Anumanchipalli, Adelyn Tu-Chan, Karunesh Ganguly & Edward F. Chang
Nature communications 2022. [PDF] [Code] [Data-Metzger2022 should contact edward.chang@ucsf.edu]

A high-performance neuroprosthesis for speech decoding and avatar control
Sean L. Metzger, Kaylo T. Littlejohn, Alexander B. Silva, David A. Moses, Margaret P. Seaton, Ran Wang, Maximilian E. Dougherty, Jessie R. Liu, Peter Wu, Michael A. Berger, Inga Zhuravleva, Adelyn Tu-Chan, Karunesh Ganguly, Gopala K. Anumanchipalli, Edward F. Chang
Nature 2023. ECoG to text using bi-RNN and CTC beam search, to speech using hubert model and to waveform and voice-convert to target speaker, to articulatory gestures with bi-RNN to gesture units and to gestures and to avatar, etc. [PDF] [Code] [Data-Metzger2023]

Decoding and synthesizing tonal language speech from brain activity
YAN LIU, ZEHAO ZHAO, MINPENG XU, HAIQING YU, YANMING ZHU, JIE ZHANG, LINGHAO BU, XIAOLUO ZHANG, JUNFENG LU, JINSONG WU
SCIENCE ADVANCES 9 Jun 2023 . Chinese language decoding [PDF]

fMRI

Semantic reconstruction of continuous language from non-invasive brain recordings
Jerry Tang, Amanda LeBel, Shailee Jain & Alexander G. Huth
Nature Neuroscience 2023 . brain encoding model, generate word by comparing the predicted brain signal with the original one. [PDF] [Data-LeBel2023] [Data-Tang2023] [Code]

Decoding Continuous Character-based Language from Non-invasive Brain Recordings
Cenyuan Zhang, Xiaoqing Zheng, Ruicheng Yin, Shujie Geng, Jianhan Xu, Xuan Gao, Changze Lv, Zixuan Ling, Xuanjing Huang, Miao Cao, Jianfeng Feng
Arxiv 2024. [PDF]

Brain-to-speech

ECoG

Speech synthesis from neural decoding of spoken sentences.
Gopala K. Anumanchipalli*, Josh Chartier*, Edward F. Chang.
Withdrawn recently. Nature. 2019 April [PDF]

Subject-Agnostic Transformer-Based Neural Speech Decoding from Surface and Depth Electrode Signals.
Junbo Chen, Xupeng Chen, Ran Wang, Chenqian Le, Amirhossein Khalilian-Gourtani, Erika Jensen, Patricia Dugan, Werner Doyle, Orrin Devinsky, Daniel Friedman, Adeen Flinker, and Yao Wang
bioRxiv. Preprint. 2024 Mar 14. [PDF]

A neural speech decoding framework leveraging deep learning and speech synthesis.
Xupeng Chen, Ran Wang, Amirhossein Khalilian-Gourtani, Leyao Yu, Patricia Dugan, Daniel Friedman, Werner Doyle, Orrin Devinsky, Yao Wang, Adeen Flinker Nature Machine Intelligence 2024 [PDF] [Data-Chen2024] [Code]

Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRI.
Che Liu, Changde Du, Xiaoyu Chen, Huiguang He Arxiv 2024 Short as C2F-LDM [PDF] [Data- Brain2Sound] [Data- Brain2Music] [Data- Brain2Speech] [Data- Brain2Sound]

MEG

Decoding speech perception from non-invasive brain recordings
Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, Jean-Rémi King
Nature Machine Intelligence 2023 . Decode M/EEG to speech with proposed brain module, trained with CLIP. M/EEG input to the brain module and get features, only choose sentence from candidates, not generate. [PDF] [MEG Data-Gwilliams] [MEG Data-Schoffelen] [EEG Data-Broderick] [EEG Data-Brennan] [Code]

EMG

A Cross-Modal Approach to Silent Speech with LLM-Enhanced Recognition
Tyler Benster, Guy Wilson, Reshef Elisha, Francis R Willett, Shaul Druckmann
Arxiv 2024. [PDF]

EEG

End-to-end translation of human neural activity to speech with a dual–dual generative adversarial network
Yina Guo*, Ting Liu*, Xiaofei Zhang*, Anhong Wang, Wenwu Wang
Knowledge-Based Systems 2023. [PDF] [Code]

fMRI

Open-vocabulary Auditory Neural Decoding Using fMRI-prompted LLM
Xiaoyu Chen, Changde Du, Liu Che, Yizhe Wang, Huiguang He
Arxiv 2024. [PDF] [MEG Data-Gwilliams]

Brain-to-music

fMRI

BRAIN2MUSIC: RECONSTRUCTING MUSIC FROM HUMAN BRAIN ACTIVITY
Timo I. Denk*, Yu Takagi*, Takuya Matsuyama, Andrea Agostinelli, Tomoya Nakai, Christian Frank, Shinji Nishimoto
ICLR 2024. [PDF] [Data-Nakai2022]

ECoG

Music can be reconstructed from human auditory cortex activity using nonlinear decoding models
*Bellier, Anaïs Llorens, Déborah Marciano, Aysegul Gunduz, Gerwin Schalk, Peter Brunner, Robert T. Knight *
PLOS 2023. [PDF]

Brain-to-image

EEG

Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion.
Dongyang Li, Chen Wei, Shiying Li, Jiachen Zou, Quanying Liu
Arxiv 2024. [PDF] [Code] [Data-Things-EEG1] [Data-Things-EEG2] [Data-Things-MEG]

MEG

BRAIN DECODING: TOWARD REAL-TIME RECONSTRUCTION OF VISUAL PERCEPTION.
Yohann Benchetrit*, Hubert Banville*, Jean-R´emi King
ICLR 2024. [PDF] [Data-Things-MEG]

fMRI

Brain captioning: Decoding human brain activity into images and text.
Matteo Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, Nicola Toschi
Arxiv 2023. [PDF] [Data-NSD]

Natural scene reconstruction from fMRI signals using generative latent diffusion.
Furkan Ozcelik, Rufin VanRullen
Nature scientific reports 2023. [PDF] [Data-NSD]

Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding .
Zijiao Chen* Jiaxin Qing*, Tiange Xiang, Wan Lin Yue, Juan Helen Zhou
CVPR2023. [PDF] [Project] [Code] [Data-HCP Young Adult] [Data-GOD] [Data-BOLD5000]

UMBRAE: Unified Multimodal Decoding of Brain Signals.
Weihao Xia, Raoul de Charette, Cengiz Öztireli, Jing-Hao Xue
Arxiv 2024. [PDF] [Project] [Code] [Data-NSD]

Brain-to-video

Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity.
Zijiao Chen*, Jiaxin Qing*, Juan Helen Zhou
NeurIPS 2023. [PDF] [Project] [Code] [Data-HCP Young Adult] [Data-Wen]

Foundation models

LARGE BRAIN MODEL FOR LEARNING GENERIC REPRESENTATIONS WITH TREMENDOUS EEG DATA IN BCI.
Wei-Bang Jiang, Li-Ming Zhao, Bao-Liang Lu
ICLR 2024. [PDF] [Code] [Data-BCI Competition IV-1] [Data-Emobrain] [Data-Grasp and Lift EEG Challenge] [Data-Inria BCI Challenge] [Data-EEGMotorMovement/ImageryDataset] [Data-Raw EEG Data] [Data-Resting State EEG Data] [Data-SEED Series] [Data-Siena Scalp EEG Database] [Data-SPIS Resting State Dataset] [Data-Target Versus Non-Target] [Data-TUAR] [Data-TUEP] [Data-TUSZ] [Data-TUSL] [Data-Self-collected EEG Data]

theory

How Many Bytes Can YouTake Out Of Brain-To-Text Decoding?.
Richard J. Antonello, Nihita Sarma, Jerry Tang, Jiaru Song, Alexander G. Huth
Arxiv 2024. [PDF]

The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning.
Richard J. Antonello, Nihita Sarma, Jerry Tang, Jiaru Song, Alexander G. Huth
Arxiv 2024.

[PDF]

Team and People

• NCC lab, SUSTech, by Quanying Liu.
• Computational Cognitive and Translational Neuroscience Lab, ShanghaiTech, by Yuanning Li.
• Department of Linguistics and Translation, CityU of HK, by Niels O. Schiller.
• Meta/CNRS, École Normale Supérieure, by Jean-Remi King.
• Neural Prosthetics Translational Laboratory, Stanford University, by Jaimie Henderson.
• HuthLab, The University of Texas at Austin, by Alexander G. Huth.
• Nishimoto Lab, Osaka University, by Shinji Nishimoto.
• Research and Development Center for Large Language Models, National Institute of Informatics, by Yu Takagi.
• ISTBI, Fudan University, by Jianfeng FENG.
• ZNLP, CASIA, by some researchers Shaonan WANG,Jingyuan SUN.
• State Key Laboratory of Management and Control for Complex Systems, Chinese Academy of Sciences and ShanghaiTech University, by Huiguang He.

Dataset

fMRI

[Data-NSD] [Data-HCP Young Adult] [Data-GOD] [Data-BOLD5000] [Data-Wen] [Data-LeBel2023] [Data-Tang2023]

EEG

[Data-Things-EEG1] [Data-Things-EEG2] [EEG Data-Broderick] [EEG Data-Brennan] [Data-ZuCo1] [Data-ZuCo2]

MEG

[MEG Data-Gwilliams] [MEG Data-Schoffelen] [Data-Things-MEG]

ECoG

[Data-Metzger2023] [Data-Metzger2022 should contact edward.chang@ucsf.edu] [Data-Willett2023] [Data-Willett2021]