/papers

Curated repository of notes from papers I'm reading, mostly NLP related. Updated regularly.

Research literature notes 🤓

made-with-Markdown Maintenance Ask me anything! GitHub issues PRs Welcome

Notes from papers I'm reading, ordered by topic and chronologically.

NLP

  1. What’s Going On in Neural Constituency Parsers? An Analysis, Gaddy et al., 2018 [Paper] [Notes] #nlp
  2. Two Methods for Domain Adaptation of Bilingual Tasks: Delightfully Simple and Broadly Applicable, Hangya et al., 2018 [Paper] [Notes] #nlp
  3. What do you learn from context? Probing for sentence structure in contextualized word representations, Tenney et al., 2019 [Paper] [Notes] #nlp
  4. BPE-Dropout: simple and effective subword regularization, Provilkov et al., 2019 [Paper] [Notes] #nlp
  5. From English To Foreign Languages: Transferring Pre-trained Language Models, Tran, 2020 [Paper] [Notes] #nlp
  6. Evaluating NLP models via contrast sets, Gardner et al., 2020 [Paper] [Notes] #nlp
  7. Byte Pair Encoding is Suboptimal for Language Model Pretraining, Bostrom et al., 2020 [Paper] [Notes] #nlp
  8. Translation artifacts in cross-lingual transfer learning, Artetxe et al., 2020 [Paper] [Notes] #nlp
  9. Weight poisoning attacks on pre-trained models, Kurita et al., 2020 [Paper] [Notes] #nlp
  10. SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings, Sabet et al., 2020 [Paper] [Notes] #nlp
  11. Experience Grounds Language, Bisk et al., 2020 [Paper] [Notes] #nlp #linguistics
  12. Beyond accuracy: behavioral testing of NLP models with CheckList, Ribeiro et al., 2020 [Paper] [Notes] #nlp
  13. The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes, Kiela et al., 2020 [Paper] [Notes] #nlp
  14. The Unstoppable Rise of Computational Linguistics in Deep Learning, Henderson, 2020 [Paper] [Notes] #nlp #linguistics
  15. Language (Technology) is Power: A Critical Survey of "Bias" in NLP, Blodgett et al., 2020 [Paper] [Notes] #nlp
  16. Representation Learning for Information Extraction from Form-like Documents, Majumder et al., 2020 [Paper] [Notes] #nlp
  17. Learning to tag OOV tokens by integrating contextual representation and background knowledge, He et al., 2020 [Paper] [Notes] #nlp
  18. It's not just size that matters, small language models are also few-shot learners, Schick and Schütze, 2020 [Paper] [Notes] #nlp
  19. Did you read the next episode? Using textual cues for predicting podcast popularity, Joshi et al., 2020 [Paper] [Notes] #nlp
  20. A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios, Hedderich et al., 2020 [Paper] [Notes] #nlp
  21. Challenges in Deploying Machine Learning: a Survey of Case Studies, Paleyes et al., 2020 [Paper] [Notes] #nlp
  22. Adapting Coreference Resolution to Twitter Conversations, Aktas et al., 2020 [Paper] [Notes] #nlp
  23. Learning from others' mistakes: avoiding dataset biases without modeilng them, Sanh et al., 2020 [Paper] [Notes] #nlp
  24. CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation, Clark et al., 2021 [Paper] [Notes] #nlp

Embeddings

  1. Semi-supervised sequence tagging with bidirectional language models, Peters et al., 2017 [Paper] [Notes] #nlp #embeddings
  2. Mimicking Word Embeddings using Subword RNNs, Pinter et al., 2017 [Paper] [Notes] #nlp #embeddings
  3. Deep contextualized word representations, Peters et al., 2018 [Paper] [Notes] #nlp #embeddings
  4. Linguistic Knowledge and Transferability of Contextual Representations, Liu et al., 2019 [Paper] [Notes] #nlp #embeddings
  5. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates, Kudo, 2018 [Paper] [Notes] #nlp #embeddings
  6. Dissecting contextual word embeddings: architecture and representation, Peters et al., 2018 [Paper] [Notes] #nlp #embeddings
  7. BERT: Pre-training of deep bidirectional transformers for language understanding, Devlin et al., 2018 [Paper] [Notes] #nlp #embeddings
  8. Learning Semantic Representations for Novel Words: Leveraging Both Form and Context, Schick et al., 2018 [Paper] [Notes] #nlp #embeddings
  9. Wikipedia2Vec: An Efficient Toolkit for Learning and Visualizing the Embeddings of Words and Entities from Wikipedia, Yamada et al., 2018 [Paper] [Notes] #nlp #embeddings
  10. Rare Words: A Major Problem for Contextualized Embeddings and How to Fix it by Attentive Mimicking, Schick et al., 2019 [Paper] [Notes] #nlp #embeddings
  11. Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts, Schick et al., 2019 [Paper] [Notes] #nlp #embeddings
  12. BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance, Schick et al., 2019 [Paper] [Notes] #nlp #embeddings
  13. BERT is Not a Knowledge Base (Yet): Factual Knowledge vs. Name-Based Reasoning in Unsupervised QA, Poerner et al., 2019 [Paper] [Notes] #nlp #embeddings

Architectures

  1. Conditional Random Fields: probabilistic models for segmenting and labeling sequence data, Lafferty et al, 2001 [Paper] [Notes] #nlp #architectures
  2. Bidirectional LSTM-CRF Models for sequence tagging, Huang et al., 2015 [Paper] [Notes] #nlp #architectures
  3. Neural Architectures for Named Entity Recognition, Lample et al., 2016 [Paper] [Notes] #nlp #architectures #NER
  4. Named Entity Recognition with Bidirectional LSTM-CNNs, Chiu et al., 2016 [Paper] [Notes] #nlp #architectures
  5. Attention is all you need, Vaswani et al., 2018 [Paper] [Notes] #nlp #architectures
  6. Reasoning with Sarcasm by Reading In-between, Tay et al., 2018 [Paper] [Notes] #sarcasm-detection #architectures
  7. XLNet: generalized autoregressive pretraining for language understanding, Yang et al., 2019 [Paper] [Notes] #nlp #architectures
  8. R-Transformer: Recurrent Neural Network Enhanced Transformer, Wang et al., 2019 [Paper] [Notes] #nlp #architectures
  9. Generalization through Memorization: Nearest Neighbor Language Models, Khandelwal et al., 2019 [Paper] [Notes] #nlp #architectures
  10. Single Headed Attention RNN: Stop Thinking With Your Head, Merity, 2019 [Paper] [Notes] #nlp #architectures
  11. A Transformer-based approach to Irony and Sarcasm detection, Potamias et al., 2019 [Paper] [Notes] #sarcasm-detection #architecture
  12. ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training, Qi et al., 2020 [Paper] [Notes] #nlp #architectures
  13. Pre-trained Models for Natural Language Processing: A Survey, Qiu et al., 2020 [Paper] [Notes] #nlp #architectures
  14. SqueezeBERT: What can computer vision teach NLP about efficient neural networks?, Iandola et al., 2020 [Paper] [Notes] #nlp #architectures #computer-vision
  15. A comparison of LSTM and BERT for small corpus, Ezen-Can, 2020 [Paper] [Notes] #nlp #architectures

Frameworks

  1. Flair: an easy-to-use framework for stat-of-the-art NLP [Paper] [Notes] #nlp #frameworks
  2. HuggingFace's Transformers: State-of-the-art Natural Language Processing, Wolf et al., 2019 [Paper] [Notes] #nlp #frameworks
  3. Selective Brain Damage: Measuring the Disparate Impact of Model Pruning, Hooker et al., 2019 [Paper] [Notes] #frameworks
  4. Why should we add early exits to neural networks?, Scardapane et al., 2020 [Paper] [Notes] #frameworks

Datasets

  1. Introduction to the CoNLL-2003 shared task: language-independent named entity recognition, Sang et al., 2003 [Paper] [Notes] #nlp #datasets
  2. Datasheets for datasets, Gebru et al., 2018 [Paper] [Notes] #nlp #datasets
  3. SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference, Zellers et al., 2018 [Paper] [Notes] #nlp #datasets
  4. A Named Entity Recognition Shootout for German, Riedl and Padó, 2018 [Paper] [Notes] #nlp #NER #datasets
  5. Probing Neural Network Comprehension of Natural Language Arguments, Nivel and Kao, 2019 [Paper] [Notes] #nlp #datasets
  6. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference., McCoy et al., 2019 [Paper] [Notes] #nlp #linguistics #datasets
  7. UR-FUNNY: A Multimodal Language Dataset for Understanding Humor, Hasan et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
  8. HellaSwag: Can a Machine Really Finish Your Sentence?, Zellers et al., 2019 [Paper] [Notes] #nlp #datasets
  9. Sentiment analysis is not solved! Assessing and probing sentiment classification, Barnes et al., 2019 [Paper] [Notes] #nlp #datasets
  10. Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model, Cai et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
  11. Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper), Castro et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
  12. iSarcasm: A Dataset of Intended Sarcasm, Oprea et al., 2019 [Paper] [Notes] #datasets #sarcasm-detection
  13. Lessons from archives: strategies for collecting sociocultural data in machine learning, Seo Jo and Gebru, 2019 [Paper] [Notes] #nlp #datasets
  14. BERTweet: A pre-trained language model for English Tweets, Nguyen et al., 2020 [Paper] [Notes] #nlp #datasets
  15. GAIA: a fine-grained multimedia knowlege extraction system, Li et al., 2020 [Paper, [Notes] #nlp #datasets
  16. It's morphin' time! Combating linguistic discrimination with inflectional perturbations, Tan et al., 2020 [Paper, [Notes] #nlp #datasets
  17. Reactive Supervision: A New method for Collecting Sarcasm Data, Shmueli et al, 2020 [Paper] [Notes] #datasets #sarcasm-detection

NER

  1. Introduction to the CoNLL-2003 shared task: language-independent named entity recognition, Sang et al., 2003 [Paper] [Notes] #nlp #datasets #NER
  2. Neural Architectures for Named Entity Recognition, Lample et al., 2016 [Paper] [Notes] #nlp #architectures #NER
  3. Named Entity Recognition with Bidirectional LSTM-CNNs, Chiu et al., 2016 [Paper] [Notes] #nlp #architectures #NER
  4. Towards Robust Named Entity Recognition for Historic German, Schweter et al., 2019 [Paper] [Notes] #nlp #NER
  5. A Named Entity Recognition Shootout for German, Riedl and Padó, 2018 [Paper] [Notes] #nlp #NER #datasets

Sarcasm detection

summary

  1. Sarcasm Detection on Twitter: A Behavioral Modeling Approach, Rajadesingan et al., 2015 [Paper] [Notes] #sarcasm-detection
  2. Contextualized Sarcasm Detection on Twitter, Bamman and Smith, 2015 [Paper] [Notes] #sarcasm-detection
  3. Harnessing Context Incongruity for Sarcasm Detection, Joshi et al., 2015 [Paper] [Notes] #linguistics #sarcasm-detection
  4. Automatic Sarcasm Detection: A Survey, Joshi et al., 2017 [Paper] [Notes] #sarcasm-detection
  5. Detecting Sarcasm is Extremely Easy ;-), Parde and Nielsen, 2018 [Paper] [Notes] #sarcasm-detection
  6. CASCADE: Contextual Sarcasm Detection in Online Discussion Forums, Hazarika et al., 2018 [Paper] [Notes] #sarcasm-detection
  7. Reasoning with Sarcasm by Reading In-between, Tay et al., 2018 [Paper] [Notes] #sarcasm-detection #architectures
  8. Tweet Irony Detection with Densely Connected LSTM and Multi-task Learning, Wu et al., 2018 [Paper] [Notes] #sarcasm-detection
  9. UR-FUNNY: A Multimodal Language Dataset for Understanding Humor, Hasan et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
  10. Exploring Author Context for Detecting Intended vs Perceived Sarcasm, Oprea and Magdy, 2019 [Paper] [Notes] #sarcasm-detection
  11. Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper), Castro et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
  12. Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model, Cai et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
  13. A2Text-Net: A Novel Deep Neural Network for Sarcasm Detection, Liu et al., 2019 [Paper] [Notes] #sarcasm-detection
  14. Sarcasm detection in tweets, Rajagopalan et al., 2019 [Paper] [Notes] #sarcasm-detection
  15. A Transformer-based approach to Irony and Sarcasm detection, Potamias et al., 2019 [Paper] [Notes] #sarcasm-detection #architecture
  16. Deep and dense sarcasm detection, Pelser et al., 2019 [Paper] [Notes] #sarcasm-detection
  17. iSarcasm: A Dataset of Intended Sarcasm, Oprea et al., 2019 [Paper] [Notes] #datasets #sarcasm-detection
  18. Reactive Supervision: A New method for Collecting Sarcasm Data, Shmueli et al, 2020 [Paper] [Notes] #datasets #sarcasm-detection

Text summarization

  1. Evaluating the Factual Consistency of Abstractive Text Summarization, Kryscinski et al., 2019 [Paper] [Notes] #nlp #text-summarization
  2. TLDR: extreme summarization of scientific documents, Cachola et al, 2020 [Paper] [Notes] #nlp #text-summarization
  3. A survey on text simplification, Sikka and Mago, 2020 [Paper] [Notes] #nlp #text-summarization

Machine translation

  1. Unsupervised Tokenization for Machine Translation, Chung and Gildea, 2009 [Paper] [Notes] #nlp #machine-translation
  2. Neural Machine Translation of Rare Words with Subword Units, Sennrich et al., 2015 [Paper] [Notes] #nlp #machine-translation
  3. Unsupervised neural machine translation, Artetxe et al., 2017 [Paper] [Notes] #nlp #machine-translation
  4. How Much Does Tokenization Affect Neural Machine Translation? Domingo et al., 2018 [Paper] [Notes] #nlp #machine-translation
  5. Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT, Chronopoulou et al., 2020 [Paper] [Notes] #nlp #machine-translation

Text generation

Reinforcement learning

  1. Theory of Minds: Understanding Behavior in Groups Through Inverse Planning, Shum et al., 2019 [Paper] [Notes] #reinforcement-learning #social-sciences
  2. The Hanabi Challenge: A New Frontier for AI Research, Bard et al., 2019 [Paper] [Notes] #reinforcement-learning
  3. Mastering Atari, Go, Chess and Shogi by Planning with a learned model, Schrittwieser et al., 2019 [Paper] [Notes] #reinforcement-learning
  4. Language as a cognitive tool to imagine goals in curiosity-driven exploration, Colas et al., 2020 [Paper] [Notes] #reinforcement-learning
  5. Planning to Explore via Self-Supervised World Models, Sekar et al., 2020 [Paper] [Notes] #reinforcement-learning

Computer vision

  1. Cubic Stylization, Derek Liu and Jacobson, 2019 [Paper] [Notes] #computer-vision
  2. SqueezeBERT: What can computer vision teach NLP about efficient neural networks?, Iandola et al., 2020 [Paper] [Notes] #nlp #computer-vision

Machine learning

  1. Gender shades: intersectional accuracy disparities in commercial gender classification, Buolamwini and Gebru, 2018 [Paper] [Notes] #machine-learning
  2. Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges, Molnar et al., 2020 [Paper] [Notes] #machine-learning

Audio

  1. End-to-End Adversarial Text-to-Speech, Donahue et al., 2020 [Paper] [Notes] #audio
  2. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations, Baevski et al., 2020 [Paper] [Notes] #audio
  3. Large-scale multilingual audio visual dubbing, Yang et al., 2020 [Paper] [Notes] #audio

Linguistics

  1. Moving beyond the plateau: from lower-intermediate to upper-intermediate, Richards, 2015 [Paper] [Notes] #linguistics
  2. Harnessing Context Incongruity for Sarcasm Detection, Joshi et al., 2015 [Paper] [Notes] #linguistics #sarcasm-detection
  3. A Trainable Spaced Repetition Model for Language Learning, Settles and Meeder, 2016 [Paper] [Notes] #linguistics
  4. Targeted synctactic evaluation of language models, Marvin and Linzen, 2018 [Paper] [Notes] #nlp #linguistics
  5. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference., McCoy et al., 2019 [Paper] [Notes] #nlp #linguistics #datasets
  6. Language Models as Knowledge Bases?, Petroni et al., 2019 [Paper] [Notes] #nlp #linguistics
  7. Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche, Coupé et al., 2019 [Paper] [Notes] #linguistics #social-sciences
  8. My English sounds better than yours: Second language learners perceive their own accent as better than that of their peers, Mittlerer et al., 2020 [Paper] [Notes] #linguistics
  9. Experience Grounds Language, Bisk et al., 2020 [Paper] [Notes] #nlp #linguistics
  10. The Unstoppable Rise of Computational Linguistics in Deep Learning, Henderson, 2020 [Paper] [Notes] #nlp #linguistics
  11. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data, Bender et al., 2020 [Paper] [Notes] #nlp #linguistics

Social sciences

  1. Antisocial Behavior in Online Discussion Communities, Cheng et al., 2015 [Paper] [Notes] #social-sciences
  2. How much does education improve intelligence? A meta-analysis, Ritchie et al., 2017 [Paper] [Notes] #social-sciences
  3. Theory of Minds: Understanding Behavior in Groups Through Inverse Planning, Shum et al., 2019 [Paper] [Notes] #reinforcement-learning #social-sciences
  4. Fake news game confers psychological resistance against online misinformation, Roozenbeek and van der Linden, 2019 [Paper] [Notes] #social-sciences #humanities
  5. Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche, Coupé et al., 2019 [Paper] [Notes] #linguistics #social-sciences
  6. Kids these days: Why the youth of today seem lacking, Protzko and Schooler, 2019 [Paper] [Notes] #social-sciences

Humanities

  1. Fake news game confers psychological resistance against online misinformation, Roozenbeek and van der Linden, 2019 [Paper] [Notes] #social-sciences #humanities

Economics

  1. Why do people stay poor? Balboni et al., 2020 [Paper] [Notes] #economics

Physics

  1. First-order transition in a model of prestige bias, Skinner, 2019 [Paper] [Notes] #physics

Neuroscience

  1. A deep learning framework for neuroscience, Richard et al., 2019 [Paper] [Notes] #neuroscience

Algorithms

  1. Replace or Retrieve Keywords In Documents At Scale, Singh, 2017 [Paper] [Notes] #algorithms