A curated list of papers and resources about language model on graphs.
The transformer architecture used in the work, e.g., EncoderOnly, DecoderOnly, EncoderDecoder.
The size of the language model, e.g., medium, LLM.
- Microsoft Academic network (MAG)
Networks from 19 domains including CS, Mathematics, Geology, etc.
[PDF] [Data] [Preprocessing Code] - Amazon Items
Networks from 24 domains including Home, Clothing, Sports, etc.
[PDF] [Data] [Preprocessing Code]
-
Can Language Models Solve Graph Problems in Natural Language?.
preprint
Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, Yulia Tsvetkov. [PDF] [Code], 2023.5,
-
GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarking.
preprint
Jiayan Guo, Lun Du, Hengyu Liu, Mengyu Zhou, Xinyi He, Shi Han. [PDF], 2023.5,
-
Talk Like A Graph: Encoding Graphs For Large Language Models.
preprint
Bahare Fatemi, Jonathan Halcrow, Bryan Perozzi. [PDF], 2023.10,
-
GraphLLM: Boosting Graph Reasoning Ability of Large Language Model.
preprint
Ziwei Chai, Tianjie Zhang, Liang Wu, Kaiqiao Han, Xiaohai Hu, Xuanwen Huang, Yang Yang. [PDF] [Code], 2023.10,
-
SPECTER: Document-level Representation Learning using Citation-informed Transformers.
ACL 2020
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, Daniel S. Weld. [Paper], 2020.4,
-
GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph.
NeurIPs 2021
Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, Xing Xie. [Paper][Code], 2021.5,
-
Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings.
EMNLP 2022
Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, Xing Xie. [Paper][Code], 2022.2,
-
Heterformer: Transformer-based Deep Node Representation Learning on Heterogeneous Text-Rich Networks.
KDD 2023
Bowen Jin, Yu Zhang, Qi Zhu, Jiawei Han. [Paper][Code], 2022.5,
-
Edgeformers: Graph-Empowered Transformers for Representation Learning on Textual-Edge Networks.
ICLR 2023
Bowen Jin, Yu Zhang, Yu Meng, Jiawei Han. [Paper][Code], 2023.1,
-
GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs.
EMNLP 2023
-
Learning Multiplex Embeddings on Text-rich Networks with One Text Encoder.
preprint
Bowen Jin, Wentao Zhang, Yu Zhang, Yu Meng, Han Zhao, Jiawei Han. [Paper][Code], 2023.10,
-
Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need.
CIKM 2021
Zhengyi Ma, Zhicheng Dou, Wei Xu, Xinyu Zhang, Hao Jiang, Zhao Cao, Ji-Rong Wen. [PDF] [Code], 2021.1,
-
LinkBERT: Pretraining Language Models with Document Links.
ACL 2022
Michihiro Yasunaga, Jure Leskovec, Percy Liang. [Paper][Code], 2022.3,
-
Translation between Molecules and Natural Language.
EMNLP 2022
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, Heng Ji. [PDF] [Code], 2022.4,
-
TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter.
KDD 2023
Xinyang Zhang, Yury Malkov, Omar Florez, Serim Park, Brian McWilliams, Jiawei Han, Ahmed El-Kishky. [PDF] [Code], 2022.9,
-
DRAGON: Deep Bidirectional Language-Knowledge Graph Pretraining.
NeurIPs 2022
Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D. Manning, Percy Liang, Jure Leskovec. [Paper][Code], 2022.10,
-
Patton: Language Model Pretraining on Text-rich Networks.
ACL 2023
Bowen Jin, Wentao Zhang, Yu Zhang, Yu Meng, Xinyang Zhang, Qi Zhu, Jiawei Han. [Paper][Code], 2023.5,
-
Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications.
KDD 2023
Han Xie, Da Zheng, Jun Ma, Houyu Zhang, Vassilis N. Ioannidis, Xiang Song, Qing Ping, Sheng Wang, Carl Yang, Yi Xu, Belinda Zeng, Trishul Chilimbi. [Paper], 2023.6,
-
MATCH: Metadata-Aware Text Classification in A Large Hierarchy.
WWW 2021
Yu Zhang, Zhihong Shen, Yuxiao Dong, Kuansan Wang, Jiawei Han. [PDF] [Code], 2021.2,
-
Minimally-Supervised Structure-Rich Text Categorization via Learning on Text-Rich Networks.
WWW 2021
Xinyang Zhang, Chenwei Zhang, Luna Xin Dong, Jingbo Shang, Jiawei Han. [PDF] [Code], 2021.2,
-
Node Feature Extraction by Self-Supervised Multi-Scale Neighborhood Prediction.
ICLR 2022
Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Olgica Milenkovic, Inderjit S Dhillon. [Paper][Code], 2021.11,
-
Metadata-Induced Contrastive Learning for Zero-Shot Multi-Label Text Classification.
WWW 2022
Yu Zhang, Zhihong Shen, Chieh-Han Wu, Boya Xie, Junheng Hao, Ye-Yi Wang, Kuansan Wang, Jiawei Han. [Paper][Code], 2022.2,
-
Learning on Large-scale Text-attributed graphs via variational inference.
ICLR 2023
Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, Jian Tang. [Paper][Code], 2023.1,
-
Explanations as Features: LLM-Based Features for Text-Attributed Graphs.
preprint
Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, Bryan Hooi. [PDF] [Code], 2023.5,
-
Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs.
preprint
Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, Jiliang Tang. [PDF] [Code], 2023.7,
-
Natural Language is All a Graph Needs.
preprint
Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, Yongfeng Zhang. [PDF], 2023.8,
-
GraphText: Graph Reasoning in Text Space.
preprint
Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, Jian Tang. [PDF], 2023.10,
-
Can LLMs Effectively Leverage Graph Structural Information: When and Why.
preprint
Jin Huang, Xingjian Zhang, Qiaozhu Mei, Jiaqi Ma. [PDF] [Code], 2023.9,
-
Label-free Node Classification on Graphs with Large Language Models (LLMS).
preprint
Zhikai Chen, Haitao Mao, Hongzhi Wen, Haoyu Han, Wei Jin, Haiyang Zhang, Hui Liu, Jiliang Tang. [PDF], 2023.9,
-
Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries.
EMNLP 2021
Carl Edwards, ChengXiang Zhai, Heng Ji. [PDF] [Code], 2021.10,
-
GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning.
preprint
Haiteng Zhao, Shengchao Liu, Chang Ma, Hannan Xu, Jie Fu, Zhi-Hong Deng, Lingpeng Kong, Qi Liu. [PDF] [Code], 2023.6,
-
Can Large Language Models Empower Molecular Property Prediction?.
preprint
Chen Qian, Huayi Tang, Zhirui Yang, Hong Liang, Yong Liu. [PDF] [Code], 2023.7,
-
SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design.
preprint
Carl N Edwards, Aakanksha Naik, Tushar Khot, Martin D Burke, Heng Ji, Tom Hope. [PDF] [Code], 2023.7,
-
InstructProtein: Aligning Human and Protein Language via Knowledge Instruction.
preprint
Zeyuan Wang, Qiang Zhang, Keyan Ding, Ming Qin, Xiang Zhuang, Xiaotong Li, Huajun Chen. [PDF], 2023.10,
-
GNN-LM: Language Modeling based on Global Contexts via GNN.
ICLR 2022
Yuxian Meng, Shi Zong, Xiaoya Li, Xiaofei Sun, Tianwei Zhang, Fei Wu, Jiwei Li. [PDF] [Code], 2021.10,
-
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering.
NAACL 2021
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, Jure Leskovec. [PDF] [Code], 2021.4,
-
GreaseLM: Graph Reasoning Enhanced Language Models for Question Answering.
ICLR 2022
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning and Jure Leskovec. [PDF] [Code], 2022.1,
-
Graph Neural Prompting with Large Language Models.
preprint
Yijun Tian, Huan Song, Zichen Wang, Haozhu Wang, Ziqing Hu, Fang Wang, Nitesh V. Chawla, Panpan Xu. [PDF], 2023.9,
-
Text Generation from Knowledge Graphs with Graph Transformers.
NAACL 2019
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, Hannaneh Hajishirzi. [PDF] [Code], 2019.4,
-
Explaining Relationships Between Scientific Documents.
ACL 2021
Kelvin Luu, Xinyi Wu, Rik Koncel-Kedziorski, Kyle Lo, Isabel Cachola, Noah A. Smith. [PDF], 2020.2,
-
Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPT.
preprint
-
StructGPT: A General Framework for Large Language Model to Reason over Structured Data.
preprint
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, Ji-Rong Wen. [PDF] [Code], 2023.5,
-
Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph.
preprint
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung-Yeung Shum, Jian Guo. [PDF] [Code], 2023.7,
-
Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning.
preprint
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, Shirui Pan. [PDF] [Code], 2023.10,
-
Efficient and effective training of language and graph neural network models.
AAAI 2023
Vassilis N Ioannidis, Xiang Song, Da Zheng, Houyu Zhang, Jun Ma, Yi Xu, Belinda Zeng, Trishul Chilimbi, George Karypis. [Paper], 2022.6,
-
Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs.
PKDD 2023
C. Mavromatis, V. N. Ioannidis, S. Wang, D. Zheng, S. Adeshina, J. Ma, H. Zhao, C. Faloutsos, G. Karypis. [Paper], 2023.4,
Contributions to this repository are welcome!
If you find any error or have relevant resources, feel free to open an issue or a pull request.