/Awesome-Language-Model-on-Graphs

A curated list of papers and resources about language model on graphs.

MIT LicenseMIT

Awesome-Language-Model-on-Graphs Awesome

A curated list of papers and resources about language model on graphs.

Contents

Keywords Convention

The transformer architecture used in the work, e.g., EncoderOnly, DecoderOnly, EncoderDecoder.

The size of the language model, e.g., medium, LLM.

Datasets

Text-attributed network

Basic

  1. Can Language Models Solve Graph Problems in Natural Language?. preprint

    Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, Yulia Tsvetkov. [PDF] [Code], 2023.5,

  2. GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarking. preprint

    Jiayan Guo, Lun Du, Hengyu Liu, Mengyu Zhou, Xinyi He, Shi Han. [PDF], 2023.5,

  3. Talk Like A Graph: Encoding Graphs For Large Language Models. preprint

    Bahare Fatemi, Jonathan Halcrow, Bryan Perozzi. [PDF], 2023.10,

  4. GraphLLM: Boosting Graph Reasoning Ability of Large Language Model. preprint

    Ziwei Chai, Tianjie Zhang, Liang Wu, Kaiqiao Han, Xiaohai Hu, Xuanwen Huang, Yang Yang. [PDF] [Code], 2023.10,

Representation Learning

  1. SPECTER: Document-level Representation Learning using Citation-informed Transformers. ACL 2020

    Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, Daniel S. Weld. [Paper], 2020.4,

  2. GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph. NeurIPs 2021

    Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, Xing Xie. [Paper][Code], 2021.5,

  3. Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings. EMNLP 2022

    Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, Xing Xie. [Paper][Code], 2022.2,

  4. Heterformer: Transformer-based Deep Node Representation Learning on Heterogeneous Text-Rich Networks. KDD 2023

    Bowen Jin, Yu Zhang, Qi Zhu, Jiawei Han. [Paper][Code], 2022.5,

  5. Edgeformers: Graph-Empowered Transformers for Representation Learning on Textual-Edge Networks. ICLR 2023

    Bowen Jin, Yu Zhang, Yu Meng, Jiawei Han. [Paper][Code], 2023.1,

  6. GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs. EMNLP 2023

    Yichuan Li, Kaize Ding, Kyumin Lee. [Paper][Code], 2023.10,

  7. Learning Multiplex Embeddings on Text-rich Networks with One Text Encoder. preprint

    Bowen Jin, Wentao Zhang, Yu Zhang, Yu Meng, Han Zhao, Jiawei Han. [Paper][Code], 2023.10,

Pretraining

  1. Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need. CIKM 2021

    Zhengyi Ma, Zhicheng Dou, Wei Xu, Xinyu Zhang, Hao Jiang, Zhao Cao, Ji-Rong Wen. [PDF] [Code], 2021.1,

  2. LinkBERT: Pretraining Language Models with Document Links. ACL 2022

    Michihiro Yasunaga, Jure Leskovec, Percy Liang. [Paper][Code], 2022.3,

  3. Translation between Molecules and Natural Language. EMNLP 2022

    Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, Heng Ji. [PDF] [Code], 2022.4,

  4. TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter. KDD 2023

    Xinyang Zhang, Yury Malkov, Omar Florez, Serim Park, Brian McWilliams, Jiawei Han, Ahmed El-Kishky. [PDF] [Code], 2022.9,

  5. DRAGON: Deep Bidirectional Language-Knowledge Graph Pretraining. NeurIPs 2022

    Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D. Manning, Percy Liang, Jure Leskovec. [Paper][Code], 2022.10,

  6. Patton: Language Model Pretraining on Text-rich Networks. ACL 2023

    Bowen Jin, Wentao Zhang, Yu Zhang, Yu Meng, Xinyang Zhang, Qi Zhu, Jiawei Han. [Paper][Code], 2023.5,

  7. Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications. KDD 2023

    Han Xie, Da Zheng, Jun Ma, Houyu Zhang, Vassilis N. Ioannidis, Xiang Song, Qing Ping, Sheng Wang, Carl Yang, Yi Xu, Belinda Zeng, Trishul Chilimbi. [Paper], 2023.6,

Node Classification

  1. MATCH: Metadata-Aware Text Classification in A Large Hierarchy. WWW 2021

    Yu Zhang, Zhihong Shen, Yuxiao Dong, Kuansan Wang, Jiawei Han. [PDF] [Code], 2021.2,

  2. Minimally-Supervised Structure-Rich Text Categorization via Learning on Text-Rich Networks. WWW 2021

    Xinyang Zhang, Chenwei Zhang, Luna Xin Dong, Jingbo Shang, Jiawei Han. [PDF] [Code], 2021.2,

  3. Node Feature Extraction by Self-Supervised Multi-Scale Neighborhood Prediction. ICLR 2022

    Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Olgica Milenkovic, Inderjit S Dhillon. [Paper][Code], 2021.11,

  4. Metadata-Induced Contrastive Learning for Zero-Shot Multi-Label Text Classification. WWW 2022

    Yu Zhang, Zhihong Shen, Chieh-Han Wu, Boya Xie, Junheng Hao, Ye-Yi Wang, Kuansan Wang, Jiawei Han. [Paper][Code], 2022.2,

  5. Learning on Large-scale Text-attributed graphs via variational inference. ICLR 2023

    Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, Jian Tang. [Paper][Code], 2023.1,

  6. Explanations as Features: LLM-Based Features for Text-Attributed Graphs. preprint

    Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, Bryan Hooi. [PDF] [Code], 2023.5,

  7. Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs. preprint

    Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, Jiliang Tang. [PDF] [Code], 2023.7,

  8. Natural Language is All a Graph Needs. preprint

    Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, Yongfeng Zhang. [PDF], 2023.8,

  9. GraphText: Graph Reasoning in Text Space. preprint

    Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, Jian Tang. [PDF], 2023.10,

  10. Can LLMs Effectively Leverage Graph Structural Information: When and Why. preprint

    Jin Huang, Xingjian Zhang, Qiaozhu Mei, Jiaqi Ma. [PDF] [Code], 2023.9,

  11. Label-free Node Classification on Graphs with Large Language Models (LLMS). preprint

    Zhikai Chen, Haitao Mao, Hongzhi Wen, Haoyu Han, Wei Jin, Haiyang Zhang, Hui Liu, Jiliang Tang. [PDF], 2023.9,

Graph Classification

  1. Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries. EMNLP 2021

    Carl Edwards, ChengXiang Zhai, Heng Ji. [PDF] [Code], 2021.10,

  2. GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning. preprint

    Haiteng Zhao, Shengchao Liu, Chang Ma, Hannan Xu, Jie Fu, Zhi-Hong Deng, Lingpeng Kong, Qi Liu. [PDF] [Code], 2023.6,

  3. Can Large Language Models Empower Molecular Property Prediction?. preprint

    Chen Qian, Huayi Tang, Zhirui Yang, Hong Liang, Yong Liu. [PDF] [Code], 2023.7,

  4. SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design. preprint

    Carl N Edwards, Aakanksha Naik, Tushar Khot, Martin D Burke, Heng Ji, Tom Hope. [PDF] [Code], 2023.7,

  5. InstructProtein: Aligning Human and Protein Language via Knowledge Instruction. preprint

    Zeyuan Wang, Qiang Zhang, Keyan Ding, Ming Qin, Xiang Zhuang, Xiaotong Li, Huajun Chen. [PDF], 2023.10,

Language Modeling

  1. GNN-LM: Language Modeling based on Global Contexts via GNN. ICLR 2022

    Yuxian Meng, Shi Zong, Xiaoya Li, Xiaofei Sun, Tianwei Zhang, Fei Wu, Jiwei Li. [PDF] [Code], 2021.10,

Question Answering

  1. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. NAACL 2021

    Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, Jure Leskovec. [PDF] [Code], 2021.4,

  2. GreaseLM: Graph Reasoning Enhanced Language Models for Question Answering. ICLR 2022

    Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning and Jure Leskovec. [PDF] [Code], 2022.1,

  3. Graph Neural Prompting with Large Language Models. preprint

    Yijun Tian, Huan Song, Zichen Wang, Haozhu Wang, Ziqing Hu, Fang Wang, Nitesh V. Chawla, Panpan Xu. [PDF], 2023.9,

Text Generation

  1. Text Generation from Knowledge Graphs with Graph Transformers. NAACL 2019

    Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, Hannaneh Hajishirzi. [PDF] [Code], 2019.4,

  2. Explaining Relationships Between Scientific Documents. ACL 2021

    Kelvin Luu, Xinyi Wu, Rik Koncel-Kedziorski, Kyle Lo, Isabel Cachola, Noah A. Smith. [PDF], 2020.2,

Graph As Tools

  1. Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPT. preprint

    Jiawei Zhang. [PDF] [Code], 2023.4,

  2. StructGPT: A General Framework for Large Language Model to Reason over Structured Data. preprint

    Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, Ji-Rong Wen. [PDF] [Code], 2023.5,

  3. Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph. preprint

    Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung-Yeung Shum, Jian Guo. [PDF] [Code], 2023.7,

  4. Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning. preprint

    Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, Shirui Pan. [PDF] [Code], 2023.10,

Model Efficiency

  1. Efficient and effective training of language and graph neural network models. AAAI 2023

    Vassilis N Ioannidis, Xiang Song, Da Zheng, Houyu Zhang, Jun Ma, Yi Xu, Belinda Zeng, Trishul Chilimbi, George Karypis. [Paper], 2022.6,

  2. Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs. PKDD 2023

    C. Mavromatis, V. N. Ioannidis, S. Wang, D. Zheng, S. Adeshina, J. Ma, H. Zhao, C. Faloutsos, G. Karypis. [Paper], 2023.4,

Contribution

Contributions to this repository are welcome!

If you find any error or have relevant resources, feel free to open an issue or a pull request.