/KENLU-Papers

An awesome repository for knowledge-enhanced natural language understanding resources, including related papers, codes and datasets.

Awesome Knowledge-Enhanced Natural Language Understanding

An awesome repository for knowledge-enhanced natural language understanding resources, including related papers, codes and datasets. Inspired by KENLG-Reading.

Keywords Convention

Basic NLU Papers for Beginners

  • Attention is All you Need, at NeurIPS 2017. [pdf]
  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, on arXiv 2019. [pdf]
  • RoBERTa: A Robustly Optimized BERT Pretraining Approach, on arXiv 2019. [pdf]
  • SpanBERT: Improving Pre-training by Representing and Predicting Spans, at TACL 2020. [pdf]
  • A Primer in BERTology: What We Know About How BERT Works, at TACL 2020. [pdf]

Tutorials

  • Knowledge-Augmented Methods for Natural Language Processing, at ACL 2022. [pdf]

KENLU

  • DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understanding, at AAAI 2022. [pdf]
  • Does Knowledge Help General NLU? An Empirical Study, on arXiv 2021. [pdf]

Knowledge-Enhanced Pre-training

  • A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models, on arXiv 2022. [pdf]
  • A Survey of Knowledge Enhanced Pre-trained Models, on arXiv 2022. [pdf]
  • Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey, on arXiv 2021. [pdf]
  • Relational World Knowledge Representation in Contextual Language Models: A Review, at EMNLP 2021. [pdf]
  • Combining pre-trained language models and structured knowledge, on arXiv 2021. [pdf]
  • Incorporating Extra Knowledge to Enhance Word Embedding, at IJCAI 2020. [pdf]
  • KALA: Knowledge-Augmented Language Model Adaptation, at NAACL 2022. [pdf]
  • LinkBERT: Pretraining Language Models with Document Links, at ACL 2022. [pdf]
  • Metadata Shaping: A Simple Approach for Knowledge-Enhanced Language Models, at ACL 2022 findings. [pdf]
  • Dict-BERT: Enhancing Language Model Pre-training with Dictionary, at ACL 2022. [pdf]
  • Great Truths are Always Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models, at NAACL findings 2022. [pdf] [code]
  • JAKET: Joint Pre-training of Knowledge Graph and Language Understanding, at AAAI 2022. [pdf]
  • K-ADAPTER: Infusing Knowledge into Pre-Trained Models with Adapters, at EMNLP 2021. [pdf]
  • KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation, at TACL 2021. [pdf]
  • [Medical Knowledge] SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining, at ACL 2021. [pdf]
  • Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees, at EACL 2021. [pdf]
  • Entities as Experts: Sparse Memory Asccess with Entity Supervision, at EMNLP 2020. [pdf]
  • K-BERT: Enabling Language Representation with Knowledge Graph, at AAAI 2020. [pdf]
  • Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models, on arXiv 2020. [pdf]

Knowledge-Enhanced Text Representation

  • Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense, at ACL findings 2022. [pdf]
  • mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models, at ACL 2022. [pdf]
  • Infusing Finetuning with Semantic Dependencies, at TACL 2021. [pdf]
  • Biomedical Interpretable Entity Representations, at ACL findings 2021. [pdf]
  • GENE: Global Event Network Embedding, at TextGraphs 2021. [pdf]
  • Incorporating Extra Knowledge to Enhance Word Embedding, at IJCAI 2021. [pdf]
  • LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention, at EMNLP 2020. [pdf]
  • Interpretable Entity Representations through Large-Scale Typing, at EMNLP findings 2020. [pdf]
  • E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT, at EMNLP findings 2020. [pdf]
  • Breaking Through the 80% Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information, at ACL 2020. [pdf]
  • Semantics-Aware BERT for Language Understanding, at AAAI 2020. [pdf]
  • Contextualized Representations Using Textual Encyclopedic Knowledge, on arXiv 2020. [pdf]
  • Knowledge Enhanced Contextual Word Representations, at EMNLP 2019. [pdf]
  • [Event Representation] Event Representation Learning Enhanced with External Commonsense Knowledge, at EMNLP 2019. [pdf]
  • Offline versus Online Representation Learning of Documents Using External Knowledge, at TOIS 2019. [pdf]

Knowledge-Enhanced Text Classification

  • Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification, at ACL 2022. [pdf]
  • Pre-training and Fine-tuning Neural Topic Model: A Simple yet Effective Approach to Incorporating External Knowledge, at ACL 2022. [pdf]
  • KenMeSH: Knowledge-enhanced End-to-end Biomedical Text Labelling, on arXiv 2022. [pdf]
  • KESA: A Knowledge Enhanced Approach For Sentiment Analysis, on arXiv 2022. [pdf]
  • Compare to The Knowledge: Graph Neural Fake News Detection with External Knowledge, at ACL 2021. [pdf] [code]
  • Enhancing Zero-shot and Few-shot Stance Detection with Commonsense Knowledge Graph, at ACL findings 2021. [pdf]
  • [Fact Verification] Modeling Entity Knowledge for Fact Verification, at FEVER 2021. [pdf]
  • Hierarchical Heterogeneous Graph Representation Learning for Short Text Classification, at EMNLP 2021. [pdf]
  • Knowledge-Guided Paraphrase Identification, at EMNLP findings 2021. [pdf]
  • KinGDOM: Knowledge-Guided DOMain adaptation for sentiment analysis, at ACL 2020. [pdf]
  • GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification, at ACL 2019. [pdf]

Knowledge-Enhanced Information Extraction

  • Pretrained Knowledge Base Embeddings for improved Sentential Relation Extraction, at ACL srw 2022. [pdf]
  • Enhanced Language Representation with Label Knowledge for Span Extraction, at EMNLP 2021. [pdf]
  • Knowledge-Enriched Event Causality Identification via Latent Structure Induction Networks, at ACL 2021. [pdf]
  • Fine-grained Information Extraction from Biomedical Literature based on Knowledge-enriched Abstract Meaning Representation, at ACL 2021. [pdf]
  • Joint Biomedical Entity and Relation Extraction with Knowledge-Enhanced Collective Inference, at ACL 2021. [pdf]
  • Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter, at ACL 2021. [pdf]
  • Adaptive Knowledge-Enhanced Bayesian Meta-Learning for Few-shot Event Detection, at ACL findings 2021. [pdf]
  • Knowledge-aware Named Entity Recognition with Alleviating Heterogeneity, at AAAI 2021. [pdf]
  • Knowledge Enhanced Event Causality Identification with Mention Masking Generalizations, at IJCAI 2020. [pdf]
  • Leverage Lexical Knowledge for Chinese Named Entity Recognition via Collaborative Graph Network, at EMNLP 2019. [pdf]
  • Improving Relation Extraction with Knowledge-attention, at EMNLP 2019. [pdf]

Knowledge-Enhanced Semantics and Syntax Parsing

  • LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution, at ACL 2022. [pdf]
  • A Unified Syntax-aware Framework for Semantic Role Labeling, at EMNLP 2018. [pdf]

Knowledge-Enhanced Information Retrieval

  • Entity-aware Transformers for Entity Search, on arXiv 2022.[pdf]
  • LET: Linguistic Knowledge Enhanced Graph Transformer for Chinese Short Text Matching, at AAAI 2021. [pdf]
  • [Recommender System] KRED: Knowledge-Aware Document Representation for News Recommendations, at RecSys 2020. [pdf]
  • Learning Unsupervised Knowledge-Enhanced Representations to Reduce the Semantic Gap in Information Retrieval, at TIOS 2020. [pdf]
  • Knowledge Enhanced Hybrid Neural Network for Text Matching, at AAAI 2018. [pdf]
  • [Recommender System] DKN: Deep Knowledge-Aware Network for News Recommendation, at WWW 2018. [pdf]

Knowledge-Enhanced Machine Reading Comprehension

  • Improving Machine Reading Comprehension with Contextualized Commonsense Knowledge, at ACL 2022. [pdf]
  • SG-Net: Syntax-Guided Machine Reading Comprehension, at AAAI 2020. [pdf]
  • Incorporating Syntax and Frame Semantics in Neural Network for Machine Reading Comprehension, at COLING 2020. [pdf]
  • Machine Reading Comprehension Using Structural Knowledge Graph-aware Network, at EMNLP 2019. [pdf]
  • Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge, at EMNLP 2018. [pdf]

Knowledge-Enhanced Question Answering

  • GreaseLM: Graph REASoning Enhanced Language Models for Question Answering, at ICLR 2022. [pdf]
  • Instilling Type Knowledge in Language Models via Multi-Task QA, at NAACL findings 2022. [pdf]
  • Unstructured Text Enhanced Open-Domain Dialogue System: A Systematic Survey, at TOIS 2022. [pdf]
  • Fusing Context Into Knowledge Graph for Commonsense Question Answering, at ACL findings 2021. [pdf]
  • QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering, at NAACL 2021. [pdf]
  • Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention, on arXiv 2021. [pdf]
  • Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition, at EMNLP 2020. [pdf]
  • Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering, at AAAI 2020. [pdf]
  • Improving Question Answering with External Knowledge, at EMNLP MRQA 2019. [pdf]
  • Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering, on arXiv 2019. [pdf]
  • Knowledge-aware Attentive Neural Network for Ranking Question Answer Pairs, at SIGIR 2018. [pdf]
  • Dynamic Integration of Background Knowledge in Neural NLU Systems, on arXiv 2018. [pdf]
  • An End-to-End Model for Question Answering over Knowledge Base with Cross-Attention Combining Global Knowledge, at EMNLP 2017. [pdf]

Knowledge-Enhanced Commonsense Reasoning

  • Leveraging Knowledge in Multilingual Commonsense Reasoning, at ACL findings 2022. [pdf]
  • Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts, at ACL findings 2022. [pdf]
  • Knowledge-Augmented Language Models for Cause-Effect Relation Classification, at CSRR 2022. [pdf]
  • KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning, at EMNLP 2019. [pdf]

Memory and Retrieval Augmented PLMs

  • Mention Memory: incorporating textual knowledge into Transformers through entity mention attention, at ICLR 2022. [pdf]
  • A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models, at ACL 2022. [pdf]
  • REALM: Retrieval-Augmented Language Model Pre-Training, on arXiv 2020. [pdf]

Knowledge Probe

  • Probing Pretrained Language Models for Lexical Semantics, at EMNLP 2020. [pdf]
  • How Much Knowledge Can You Pack Into the Parameters of a Language Model?, at EMNLP 2020. [pdf]
  • Language Models as Knowledge Bases?, at EMNLP 2019. [pdf]

Knowledge-Related Datasets

  • CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge, at NeurIPS 2021. [pdf]
  • KILT: a Benchmark for Knowledge Intensive Language Tasks, at NAACL 2021. [pdf]
  • GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding, at EMNLP 2018. [pdf]

Other Related Projects

  • [Prompt] Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing, on arXiv 2021. [pdf]
  • Knowledge-enriched Text Generation Survey, Tutorial and Reading, on github. [home]
  • A Survey of Knowledge-Enhanced Text Generation, on arXiv 2020. [pdf]
  • PromptPapers, on github. [home]
  • RetrivalLMPapers, on github. [home]