By Subba Reddy Oota (subba-reddy.oota@inria.fr), Mnemosyne Team, Inria, at Bordeaux, Inria, France, and Brain Cognition and Computation Lab, at IIIT-Hyderabad, India. If there are any areas, papers, and datasets I missed, please let me know!
Visio-Linguistic Brain Encoding
When Computational Representation Meets Neuroscience: A Survey on Brain Encoding and Decoding, IJCAI 2021 Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
Fig 9 in our survey paper Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey).
-
Instruction-tuning Aligns LLMs to the Human Brain
preprintKhai Loong Aw, Syrielle Montariol, Badr AlKhamissi, Martin Schrimpf, and Antoine Bosselut. [PDF] Arxiv 2023,
Scaling laws for language encoding models in fMRI, NeurIPS 2023 [code]
Joint processing of linguistic properties in brains and language models, NeurIPS 2023 [code]
Training language models to summarize narratives improves brain alignment, ICLR 2023 [code]
Long-Term Plausibility of Language Models and Neural Dynamics during Narratives Listening, Cogsci 2022
Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?, NAACL 2022
Can fMRI reveal the representation of syntactic structure in the brain?, NeurIPS 2021 [code]
CogniVal: A Framework for Cognitive Word Embedding Evaluation, CoNLL 2019 [code]
The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing, PNAS 2021 [code]
Disentangling Syntax and Semantics in the Brain with Deep Networks, ICML 2021 [slides]
Modeling task effects on meaning representation in the brain via zero-shot MEG prediction, NeurIPS 2020 [code], [slides]
Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain), NeurIPS 2019 [code]
Inducing brain-relevant bias in natural language processing models, NeurIPS 2019 [code]
Incorporating Context into Language Encoding Models for fMRI, NeurIPS 2018
Aligning context-based statistical models of language with brain activity during reading, EMNLP 2014
Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs, NeurIPS 2019
Neural Taskonomy: Inferring the Similarity of Task-Derived Representations from Brain Activity, NeurIPS 2019
Speech language models lack important brain-relevant semantics Arxiv
Meg encoding using word context semantics in listening stories, Interspeech 2023 [code]
[Speech Taskonomy: Which Speech Tasks are the most Predictive of fMRI Brain Activity?], Interspeech 2023 [code]
Self-supervised models of audio effectively explain human cortical responses to speech, ICML 2022
The Hierarchical Cortical Organization of Human Speech Processing, Journal of Neuroscience 2016
How the Human Brain Makes Sense of a World in Motion, CCN 2021 [code]
Brain encoding models based on multimodal transformers can transfer across language and vision, Arxiv 2023
Interpreting Multimodal Video Transformers Using Brain Recordings, ICLR 2023 (Multimodal Representation Learning Workshop)
Brain2Word: Decoding Brain Activity for Language Generation, [code]
Does injecting linguistic structure into language models lead to better alignment with brain recordings?), [code]
Linking artificial and human neural representations of language, EMNLP 2019, [code]
Towards sentence-level brain decoding with distributed representations, AAAI 2019
Toward a universal decoder of linguistic meaning from brain activation, Nature Communications 2018
From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI, NeurIPS 2019 [code]
Reconstructing Perceived Images from Human Brain Activities with Bayesian Deep Multi-view Learning, IEEE TNNLS 2018
Towards reconstructing intelligible speech from the human auditory cortex, Nature Scientific Reports 2019 [code]
Decoding the Semantic Content of Natural Movies from Human Brain Activity, Frontiers 2016
Deep Learning for Brain Encoding and Decoding, Subba Reddy Oota, Cogsci 2022 [code]
Encoding and Decoding Speech From the Human Brain, Edward Chang, ICML 2021
Reconstructing Speech from Human Auditory Cortex
[The Moth Radio Hour]
Contributions to this repository are welcome!
If you find any error or have relevant resources, feel free to open an issue or a pull request.
Paper format:
1. **[paper title].** `[]`
*[authors].* [[PDF]([pdf link])] [[Code]([code link])], published time,  
Please cite the following paper if you find the resource helpful for your research.
@article{oota2023deep,
title={Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)},
author={Oota, Subba Reddy and Gupta, Manish and Bapi, Raju S and Jobard, Gael and Alexandre, Fr{\'e}d{\'e}ric and Hinaut, Xavier},
journal={arXiv preprint arXiv:2307.10246},
year={2023}
}