KG-LLM-Papers

What can LLMs do for KGs? Or, in other words, what role can KG play in the era of LLMs?

🙌 This repository collects papers integrating knowledge graphs (KGs) and large language models (LLMs).

😎 Welcome to recommend missing papers through Adding Issues or Pull Requests.

🔔 News

2024-02 We preprint our Survey Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey [Repo].
2023-11 We preprint our paper Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering and release the [Repo].
2023-10 We preprint our paper Making Large Language Models Perform Better in Knowledge Graph Completion and release the [Repo].
2023-06 We create this repository to maintain a paper list on Intergrating Knowledge Graphs and Large Language Models.

Todo:

- Fine-grained classification of papers
- Update paper project / code
- Wiki page for brief paper introduction

Content

📜 Papers

Papers

Surveys

[arxiv] Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey. 2024.02
[arxiv] Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey. 2023.11
[arxiv] Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity. 2023.10
[arxiv] On the Evolution of Knowledge Graphs: A Survey and Perspective. 2023.10
[arxiv] Benchmarking the Abilities of Large Language Models for RDF Knowledge Graph Creation and Comprehension: How Well Do LLMs Speak Turtle? 2023.09
[arxiv] Explainability for Large Language Models: A Survey. 2023.09
[arxiv] Generations of Knowledge Graphs: The Crazy Ideas and the Business Impact. 2023.08
[arxiv] Large Language Models and Knowledge Graphs: Opportunities and Challenges. 2023.08
[arxiv] Unifying Large Language Models and Knowledge Graphs: A Roadmap. 2023.06 [Repo]
[arxiv] ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling. 2023.06
[arxiv] A Survey of Knowledge-Enhanced Pre-trained Language Models. 2023.05

Methods

[arxiv] TechGPT-2.0: A large language model project to solve the task of knowledge graph construction. 2024.01
[arxiv] Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs. 2024.01
[arxiv] The Earth is Flat? Unveiling Factual Errors in Large Language Models. 2024.01
[arxiv] keqing: knowledge-based question answering is a nature chain-of-thought mentor of LLM. 2024.01
[arxiv] Chain of History: Learning and Forecasting with LLMs for Temporal Knowledge Graph Completion. 2024.01
[arxiv] Quartet Logic: A Four-Step Reasoning (QLFR) framework for advancing Short Text Classification. 2024.01
[arxiv] Conversational Question Answering with Reformulations over Knowledge Graph. 2023.12
[arxiv] Think and Retrieval: A Hypothesis Knowledge Graph Enhanced Medical Large Language Models. 2023.12
[arxiv] KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph. 2023.12
[arxiv] Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City Environment. 2023.12
[arxiv] Zero-Shot Fact-Checking with Semantic Triples and Knowledge Graphs. 2023.12
[arxiv] KGLens: A Parameterized Knowledge Graph Solution to Assess What an LLM Does and Doesn't Know. 2023.12
[arxiv] LLM-ARK: Knowledge Graph Reasoning Using Large Language Models via Deep Reinforcement Learning. 2023.12
[arxiv] Towards Trustworthy AI Software Development Assistance. 2023.12
[arxiv] KnowGPT: Black-Box Knowledge Injection for Large Language Models. 2023.12
[arxiv] Making Large Language Models Better Knowledge Miners for Online Marketing with Progressive Prompting Augmentation. 2023.12
[arxiv] Conceptual Engineering Using Large Language Models. 2023.12
[arxiv] Beyond Isolation: Multi-Agent Synergy for Improving Knowledge Graph Construction. 2023.12
[arxiv] Zero- and Few-Shots Knowledge Graph Triplet Extraction with Large Language Models. 2023.12
[arxiv] On Exploring the Reasoning Capability of Large Language Models with Knowledge Graphs. 2023.11
[arxiv] Biomedical knowledge graph-enhanced prompt generation for large language models. 2023.11
[arxiv] A Graph-to-Text Approach to Knowledge-Grounded Response Generation in Human-Robot Interaction. 2023.11
[EMNLP 2023]Revisiting the Knowledge Injection Frameworks. 2023.12
[EMNLP 2023]Does the Correctness of Factual Knowledge Matter for Factual Knowledge-Enhanced Pre-trained Language Models? 2023.12
[EMNLP 2023]ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graph. 2023.12
[EMNLP 2023 Findings]KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion. 2023.12
[arxiv] $R^3$-NL2GQL: A Hybrid Models Approach for for Accuracy Enhancing and Hallucinations Mitigation. 2023.11
[arxiv] Biomedical knowledge graph-enhanced prompt generation for large language models. 2023.11
[arxiv] Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting. 2023.11
[arxiv] Fine-tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata. 2023.11
[arxiv] Leveraging LLMs in Scholarly Knowledge Graph Question Answering. 2023.11
[arxiv] Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering. 2023.11
[arxiv] OLaLa: Ontology Matching with Large Language Models. 2023.11
[arxiv] In-Context Learning for Knowledge Base Question Answering for Unmanned Systems based on Large Language Models. 2023.11
[arxiv] Let's Discover More API Relations: A Large Language Model-based AI Chain for Unsupervised API Relation Inference. 2023.11
[arxiv] Form follows Function: Text-to-Text Conditional Graph Generation based on Functional Requirements. 2023.11
[arxiv] Large Language Models Meet Knowledge Graphs to Answer Factoid Questions. 2023.10
[arxiv] Answer Candidate Type Selection: Text-to-Text Language Model for Closed Book Question Answering Meets Knowledge Graphs. 2023.10
[arxiv] Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models. 2023.10
[arxiv] DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text. 2023.10
[arxiv] Generative retrieval-augmented ontologic graph and multi-agent strategies for interpretive large language model-based materials design. 2023.10
[arxiv] A Multimodal Ecological Civilization Pattern Recommendation Method Based on Large Language Models and Knowledge Graph. 2023.10
[arxiv] LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery. 2023.10
[arxiv] Graph Agent: Explicit Reasoning Agent for Graphs. 2023.10
[arxiv] An In-Context Schema Understanding Method for Knowledge Base Question Answering. 2023.10
[arxiv] GraphGPT: Graph Instruction Tuning for Large Language Models. 2023.10
[arxiv] Systematic Assessment of Factual Knowledge in Large Language Models. 2023.10
[arxiv] KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models. 2023.10
[arxiv] MechGPT, a language-based strategy for mechanics and materials modeling that connects knowledge across scales, disciplines and modalities. 2023.10
[arxiv] Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model. 2023.10
[arxiv] ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Models. 2023.10
[arxiv] From Large Language Models to Knowledge Graphs for Biomarker Discovery in Cancer. 2023.10
[arxiv] Making Large Language Models Perform Better in Knowledge Graph Completion. 2023.10
[arxiv] CP-KGC: Constrained-Prompt Knowledge Graph Completion with Large Language Models. 2023.10
[arxiv] PHALM: Building a Knowledge Graph from Scratch by Prompting Humans and a Language Model. 2023.10
[arxiv] InstructProtein: Aligning Human and Protein Language via Knowledge Instruction. 2023.10
[arxiv] Large Language Models Meet Knowledge Graphs to Answer Factoid Questions. 2023.10
[arxiv] Knowledge Crosswords: Geometric Reasoning over Structured Knowledge with Large Language Models. 2023.10
[arxiv] Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning. 2023.10
[arxiv] RelBERT: Embedding Relations with Language Models. 2023.10
[arxiv] Benchmarking the Abilities of Large Language Models for RDF Knowledge Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?. 2023.09
[arxiv] Let's Chat to Find the APIs: Connecting Human, LLM and Knowledge Graph through AI Chain. 2023.09
[arxiv] Graph Neural Prompting with Large Language Models. 2023.09
[arxiv] A knowledge representation approach for construction contract knowledge modeling. 2023.09
[arxiv] Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering. 2023.09
[arxiv] "Merge Conflicts!" Exploring the Impacts of External Distractors to Parametric Knowledge Graphs. 2023.09
[arxiv] FactLLaMA: Optimizing Instruction-Following Language Models with External Knowledge for Automated Fact-Checking. 2023.09
[arxiv] ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning. 2023.09
[arxiv] Code-Style In-Context Learning for Knowledge-Based Question Answering. 2023.09
[arxiv] Unleashing the Power of Graph Learning through LLM-based Autonomous Agents. 2023.09
[arxiv] Knowledge-tuning Large Language Models with Structured Medical Knowledge Bases for Reliable Response Generation in Chinese. 2023.09
[arxiv] Knowledge Solver: Teaching LLMs to Search for Domain Knowledge from Knowledge Graphs. 2023.09
[arxiv] Biomedical Entity Linking with Triple-aware Pre-Training. 2023.08
[arxiv] Exploring Large Language Models for Knowledge Graph Completion. 2023.08
[arxiv] Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering. 2023.08
[arxiv] Leveraging A Medical Knowledge Graph into Large Language Models for Diagnosis Prediction. 2023.08
[arxiv] LKPNR: LLM and KG for Personalized News Recommendation Framework. 2023.08
[arxiv] Knowledge Graph Prompting for Multi-Document Question Answering. 2023.08
[arxiv] Head-to-Tail: How Knowledgeable are Large Language Models (LLM)? A.K.A. Will LLMs Replace Knowledge Graphs?. 2023.08
[arxiv] MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models. 2023.08
[arxiv] Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text. 2023.08
[arxiv] Towards Semantically Enriched Embeddings for Knowledge Graph Completion. 2023.07
[arxiv] AutoAlign: Fully Automatic and Effective Knowledge Graph Alignment enabled by Large Language Models. 2023.07
[arxiv] Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphs. 2023.07
[arxiv] Think-on-Graph: Deep and Responsible Reasoning of Large Language Model with Knowledge Graph. 2023.07
[arxiv] Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs. 2023.07
[arxiv] Exploring Large Language Model for Graph Data Understanding in Online Job Recommendations. 2023.07
[arxiv] RecallM: An Architecture for Temporal Context Understanding and Question Answering. 2023.07
[arxiv] LLM-assisted Knowledge Graph Engineering: Experiments with ChatGPT. 2023.07
[arxiv] Iterative Zero-Shot LLM Prompting for Knowledge Graph Construction. 2023.07
[arxiv] Fine-tuning Large Enterprise Language Models via Ontological Reasoning. 2023.06
[arxiv] Snowman: A Million-scale Chinese Commonsense Knowledge Graph Distilled from Foundation Model . 2023.06
[arxiv] Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering. 2023.06
[arxiv] Fine-tuning Large Enterprise Language Models via Ontological Reasoning. 2023.06
[arxiv] Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks. 2023.05
[arxiv] Enhancing Knowledge Graph Construction Using Large Language Models. 2023.05
[arxiv] ChatGraph: Interpretable Text Classification by Converting ChatGPT Knowledge to Graphs. 2023.05
[ACL 2023] FactKG: Fact Verification via Reasoning on Knowledge Graphs. 2023.05
[arxiv] HiPrompt: Few-Shot Biomedical Knowledge Fusion via Hierarchy-Oriented Prompting. 2023.04
[arxiv] StructGPT: A General Framework for Large Language Model to Reason over Structured Data. 2023.05
[arxiv] Explanations as Features: LLM-Based Features for Text-Attributed Graphs. 2023.05
[arxiv] LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities. 2023.05
[arxiv] Can Language Models Solve Graph Problems in Natural Language? 2023.05
[arxiv] Knowledge Graph Completion Models are Few-shot Learners: An Empirical Study of Relation Labeling in E-commerce with LLMs. 2023.05
[arxiv] Can large language models generate salient negative statements? 2023.05
[arxiv] GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarking. 2023.05
[arxiv] Complex Logical Reasoning over Knowledge Graphs using Large Language Models. 2023.05 [Repo]
[arxiv] Causal Reasoning and Large Language Models: Opening a New Frontier for Causality. 2023.04
[arxiv] Can large language models build causal graphs? 2023.04
[arxiv] Using Multiple RDF Knowledge Graphs for Enriching ChatGPT Responses. 2023.04
[arxiv] Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPT. 2023.04
[arxiv] Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning. 2023.04 [Repo]

Resources and Benchmarking

[arxiv] Benchmarking Large Language Models in Complex Question Answering Attribution using Knowledge Graphs. 2024.01
[arxiv] AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph. 2023.11
[arxiv] A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases. 2023.11
[arxiv] Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attribution. 2023.10
[arxiv] MarkQA: A large scale KBQA dataset with numerical reasoning. 2023.10
[arxiv] Ontology Enrichment from Texts: A Biomedical Dataset for Concept Discovery and Placement. 2023.06
[arxiv] Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation. 2023.06
[arxiv] LambdaKG: A Library for Pre-trained Language Model-Based Knowledge Graph Embeddings 2023.03 [Repo]
[arxiv] Construction of Paired Knowledge Graph-Text Datasets Informed by Cyclic Evaluation. 2023.09
[arxiv] From Large Language Models to Knowledge Graphs for Biomarker Discovery in Cancer. 2023.10

Contribution

👥 Contributors

🎉 Contributing ( welcome ! )

✨ Add a new paper or update an existing KG-related LLM paper.
🧐 Use the same format as existing entries to describe the work.
😄 A very brief explanation why you think a paper should be added or updated is recommended (Not Neccessary) via Adding Issues or Pull Requests.

Don't worry if you put something wrong, they will be fixed for you. Just feel free to contribute and promote your awesome work here! 🤩 We'll get back to you in time ~ 😉

🤝 Cite:

If this Repo is helpful to you, please consider citing our paper. We would greatly appreciate it :)

@article{DBLP:journals/corr/abs-2311-06503,
  author       = {Yichi Zhang and
                  Zhuo Chen and
                  Yin Fang and
                  Lei Cheng and
                  Yanxi Lu and
                  Fangming Li and
                  Wen Zhang and
                  Huajun Chen},
  title        = {Knowledgeable Preference Alignment for LLMs in Domain-specific Question
                  Answering},
  journal      = {CoRR},
  volume       = {abs/2311.06503},
  year         = {2023}
}

@article{DBLP:journals/corr/abs-2310-06671,
  author       = {Yichi Zhang and
                  Zhuo Chen and
                  Wen Zhang and
                  Huajun Chen},
  title        = {Making Large Language Models Perform Better in Knowledge Graph Completion},
  journal      = {CoRR},
  volume       = {abs/2310.06671},
  year         = {2023}
}

WEIYanbin1999/KG-LLM-Papers