otmjka/BERT-related-papers

BERT-related papers

BERT-related Papers

This is a list of BERT-related papers. Any feedback is welcome.

Table of Contents

• Downstream task
• Generation
• Modification (multi-task, masking strategy, etc.)
• Probe
• Inside BERT
• Multi-lingual
• Domain specific
• Multi-modal
• Model compression
• Misc.

Downstream task

QA, MC, Dialogue

• A BERT Baseline for the Natural Questions
• MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (ACL2019)
• A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning (EMNLP2019)
• SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering
• Multi-hop Question Answering via Reasoning Chains
• End-to-End Open-Domain Question Answering with BERTserini (NAALC2019)
• Latent Retrieval for Weakly Supervised Open Domain Question Answering (ACL2019)
• Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering (EMNLP2019)
• Learning to Ask Unanswerable Questions for Machine Reading Comprehension (ACL2019)
• Unsupervised Question Answering by Cloze Translation (ACL2019)
• Reinforcement Learning Based Graph-to-Sequence Model for Natural Question Generation
• Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension (ACL2019)
• Incorporating Relation Knowledge into Commonsense Reading Comprehension with Multi-task Learning (CIKM2019)
• SG-Net: Syntax-Guided Machine Reading Comprehension
• MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension
• A Simple but Effective Method to Incorporate Multi-turn Context with BERT for Conversational Machine Comprehension (ACL2019 WS)
• FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension (ACL2019 WS)
• BERT with History Answer Embedding for Conversational Question Answering (SIGIR2019)
• GraphFlow: Exploiting Conversation Flow with Graph Neural Networks for Conversational Machine Comprehension (ICML2019 WS)
• Beyond English-only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian (RANLP2019)
• Cross-Lingual Machine Reading Comprehension (EMNLP2019)
• Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
• Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension (EMNLP2019)
• BERT for Joint Intent Classification and Slot Filling
• Multi-lingual Intent Detection and Slot Filling in a Joint BERT-based Model
• BERT-DST: Scalable End-to-End Dialogue State Tracking with Bidirectional Encoder Representations from Transformer (Interspeech2019)
• Dialog State Tracking: A Neural Reading Comprehension Approach
• Domain Adaptive Training BERT for Response Selection

Analysis

• Fine-grained Information Status Classification Using Discourse Context-Aware Self-Attention
• GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge (EMNLP2019)
• Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations (EMNLP2019)
• Using BERT for Word Sense Disambiguation
• Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence (NAACL2019)
• BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis (NAACL2019)
• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis (EMNLP2019 WS)
• Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification
• An Investigation of Transfer Learning-Based Sentiment Analysis in Japanese (ACL2019)
• "Mask and Infill" : Applying Masked Language Model to Sentiment Transfer
• Neural Aspect and Opinion Term Extraction with Mined Rules as Weak Supervision (ACL2019)
• Assessing BERT’s Syntactic Abilities
• Does BERT agree? Evaluating knowledge of structure dependence through agreement relations
• Simple BERT Models for Relation Extraction and Semantic Role Labeling
• A Simple BERT-Based Approach for Lexical Simplification
• Multi-headed Architecture Based on BERT for Grammatical Errors Correction (ACL2019 WS)
• KG-BERT: BERT for Knowledge Graph Completion
• Language Models as Knowledge Bases? (EMNLP2019) [github]
• BERT-Based Arabic Social Media Author Profiling

Word segmentation / parsing / NER

• BERT Meets Chinese Word Segmentation
• Toward Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning
• Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERT
• Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing
• NEZHA: Neural Contextualized Representation for Chinese Language Understanding
• Deep Contextualized Word Embeddings in Transition-Based and Graph-Based Dependency Parsing -- A Tale of Two Parsers Revisited (EMNLP2019)
• Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing
• Named Entity Recognition -- Is there a glass ceiling? (CoNLL2019)
• Portuguese Named Entity Recognition using BERT-CRF

Pronoun/coreference resolution

• Resolving Gendered Ambiguous Pronouns with BERT (ACL2019 WS)
• Anonymized BERT: An Augmentation Approach to the Gendered Pronoun Resolution Challenge (ACL2019 WS)
• Gendered Pronoun Resolution using BERT and an extractive question answering formulation (ACL2019 WS)
• MSnet: A BERT-based Network for Gendered Pronoun Resolution (ACL2019 WS)
• Fill the GAP: Exploiting BERT for Pronoun Resolution (ACL2019 WS)
• Look Again at the Syntax: Relational Graph Convolutional Network for Gendered Ambiguous Pronoun Resolution (ACL2019 WS)
• BERT Masked Language Modeling for Co-reference Resolution (ACL2019 WS)
• BERT for Coreference Resolution: Baselines and Analysis (EMNLP2019) [github]
• WikiCREM: A Large Unsupervised Corpus for Coreference Resolution (EMNLP2019)

Relation extraction

• Matching the Blanks: Distributional Similarity for Relation Learning (ACL2019)
• BERT-Based Multi-Head Selection for Joint Entity-Relation Extraction (NLPCC2019)
• Enriching Pre-trained Language Model with Entity Information for Relation Classification
• Span-based Joint Entity and Relation Extraction with Transformer Pre-training
• Fine-tune Bert for DocRED with Two-step Process
• Fine-tuning BERT for Joint Entity and Relation Extraction in Chinese Medical Text

Text classification

• How to Fine-Tune BERT for Text Classification?
• X-BERT: eXtreme Multi-label Text Classification with BERT
• DocBERT: BERT for Document Classification
• Enriching BERT with Knowledge Graph Embeddings for Document Classification
• Classification and Clustering of Arguments with Contextualized Word Embeddings (ACL2019)
• BERT for Evidence Retrieval and Claim Verification
• Conditional BERT Contextual Augmentation

WSC, WNLI, NLI

• Exploring Unsupervised Pretraining and Sentence Structure Modelling for Winograd Schema Challenge
• A Surprisingly Robust Trick for the Winograd Schema Challenge
• Improving Natural Language Inference with a Pretrained Parser

Commonsense

• HellaSwag: Can a Machine Really Finish Your Sentence? (ACL2019) [website]
• Explain Yourself! Leveraging Language Models for Commonsense Reasoning (ACL2019)
• Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
• Informing Unsupervised Pretraining with External Linguistic Knowledge
• Commonsense Knowledge + BERT for Level 2 Reading Comprehension Ability Test

Extractive summarization

• HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization (ACL2019)
• Deleter: Leveraging BERT to Perform Unsupervised Successive Text Compression

IR

• Passage Re-ranking with BERT
• Investigating the Successes and Failures of BERT for Passage Re-Ranking
• Understanding the Behaviors of BERT in Ranking
• Document Expansion by Query Prediction
• CEDR: Contextualized Embeddings for Document Ranking (SIGIR2019)
• Deeper Text Understanding for IR with Contextual Neural Language Modeling (SIGIR2019)
• FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevance (SIGIR2019)

Generation

• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model (NAACL2019 WS)
• Pretraining-Based Natural Language Generation for Text Summarization
• Text Summarization with Pretrained Encoders (EMNLP2019)
• Multi-stage Pretraining for Abstractive Summarization
• MASS: Masked Sequence to Sequence Pre-training for Language Generation (ICML2019) [github], [github]
• Unified Language Model Pre-training for Natural Language Understanding and Generation (NeurIPS2019)
• Towards Making the Most of BERT in Neural Machine Translation
• Improving Neural Machine Translation with Pre-trained Representation
• On the use of BERT for Neural Machine Translation
• Mask-Predict: Parallel Decoding of Conditional Masked Language Models (EMNLP2019)

Modification (multi-task, masking strategy, etc.)

• Multi-Task Deep Neural Networks for Natural Language Understanding (ACL2019)
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning (ICML2019)
• Unifying Question Answering and Text Classification via Span Extraction
• ERNIE: Enhanced Language Representation with Informative Entities (ACL2019)
• ERNIE: Enhanced Representation through Knowledge Integration
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
• SpanBERT: Improving Pre-training by Representing and Predicting Spans [github]
• RoBERTa: A Robustly Optimized BERT Pretraining Approach [github]
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• KERMIT: Generative Insertion-Based Modeling for Sequences
• DisSent: Sentence Representation Learning from Explicit Discourse Relations (ACL2019)
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• SenseBERT: Driving Some Sense into BERT
• Semantics-aware BERT for Language Understanding
• K-BERT: Enabling Language Representation with Knowledge Graph
• Knowledge Enhanced Contextual Word Representations (EMNLP2019)
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP2019)
• Symmetric Regularization based BERT for Pair-wise Semantic Reasoning
• Transfer Fine-Tuning: A BERT Case Study (EMNLP2019)
• Improving Pre-Trained Multilingual Models with Vocabulary Expansion (CoNLL2019)
• ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

Probe

• A Structural Probe for Finding Syntax in Word Representations (NAACL2019)
• Linguistic Knowledge and Transferability of Contextual Representations (NAACL2019) [github]
• Probing What Different NLP Tasks Teach Machines about Function Word Comprehension (*SEM2019)
• BERT Rediscovers the Classical NLP Pipeline (ACL2019)
• Probing Neural Network Comprehension of Natural Language Arguments (ACL2019)
• Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (EMNLP2019 WS)

Inside BERT

• What does BERT learn about the structure of language? (ACL2019)
• Open Sesame: Getting Inside BERT's Linguistic Knowledge (ACL2019 WS)
• Analyzing the Structure of Attention in a Transformer Language Model (ACL2019 WS)
• What Does BERT Look At? An Analysis of BERT's Attention (ACL2019 WS)
• Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains (ACL2019 WS)
• Inducing Syntactic Trees from BERT Representations (ACL2019 WS)
• A Multiscale Visualization of Attention in the Transformer Model (ACL2019 Demo)
• Visualizing and Measuring the Geometry of BERT
• How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (EMNLP2019)
• Are Sixteen Heads Really Better than One? (NeurIPS2019)
• On the Validity of Self-Attention as Explanation in Transformer Models
• Visualizing and Understanding the Effectiveness of BERT (EMNLP2019)
• Attention Interpretability Across NLP Tasks
• Revealing the Dark Secrets of BERT (EMNLP2019)
• Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs (EMNLP2019)
• The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (EMNLP2019)
• Do NLP Models Know Numbers? Probing Numeracy in Embeddings (EMNLP2019)
• How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations (CIKM2019)

Multi-lingual

• Multilingual Constituency Parsing with Self-Attention and Pre-Training (ACL2019)
• Cross-lingual Language Model Pretraining (NeurIPS2019) [github]
• 75 Languages, 1 Model: Parsing Universal Dependencies Universally (EMNLP2019) [github]
• Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (EMNLP2019)
• How multilingual is Multilingual BERT? (ACL2019)

Domain specific

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets (ACL2019 WS)
• BERT-based Ranking for Biomedical Entity Normalization
• PubMedQA: A Dataset for Biomedical Research Question Answering (EMNLP2019)
• Pre-trained Language Model for Biomedical Question Answering
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• Publicly Available Clinical BERT Embeddings (NAACL2019 WS)
• SciBERT: Pretrained Contextualized Embeddings for Scientific Text [github]
• PatentBERT: Patent Classification with Fine-Tuning a pre-trained BERT Model

Multi-modal

• VideoBERT: A Joint Model for Video and Language Representation Learning (ICCV2019)
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (NeurIPS2019)
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Selfie: Self-supervised Pretraining for Image Embedding
• Contrastive Bidirectional Transformer for Temporal Representation Learning
• M-BERT: Injecting Multimodal Information in the BERT Structure
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers (EMNLP2019)
• Fusion of Detected Objects in Text for Visual Question Answering (EMNLP2019)
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Model compression

• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Patient Knowledge Distillation for BERT Model Compression (EMNLP2019)
• Small and Practical BERT Models for Sequence Labeling (EMNLP2019)
• TinyBERT: Distilling BERT for Natural Language Understanding
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (NeurIPS2019 WS) [github]
• Extreme Language Model Compression with Optimal Subwords and Shared Projections
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

Misc.

• Cloze-driven Pretraining of Self-attention Networks
• Learning and Evaluating General Linguistic Intelligence
• To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (ACL2019 WS)
• BERTScore: Evaluating Text Generation with BERT
• Machine Translation Evaluation with BERT Regressor
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Is BERT Really Robust? Natural Language Attack on Text Classification and Entailment
• Glyce: Glyph-vectors for Chinese Character Representations
• Back to the Future -- Sequential Alignment of Text Representations
• Improving Cuneiform Language Identification with BERT (NAACL2019 WS)
• SMILES-BERT: Large Scale Unsupervised Pre-Training for Molecular Property Prediction (ACM-BCB2019)