A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applications

This is a repository to help all readers who are interested in pre-training on graphs. If you find there are other resources with this topic missing, feel free to let us know via github issues, pull requests or my email: xiajun@westlake.edu.cn. We will update this repository and paper on a regular basis to maintain up-to-date.

Last update date: 2022-3-31

Papers List
Open-Sourced Pretrained Graph Models
Pretraining Datasets
Citation
Acknowledgement

Papers List

Pretraining Strategies

[Arxiv 2022] Unsupervised Heterophilous Network Embedding via r-Ego Network Discrimination [paper]
[CVPR 2022] Node Representation Learning in Graph via Node-to-Neighbourhood Mutual Information Maximization[paper]
[Arxiv 2022] GraphCoCo: Graph Complementary Contrastive Learning[paper]
[AAAI 2022] Simple Unsupervised Graph Representation Learning[paper]
[SDM 2022] Neural Graph Matching for Pre-training Graph Neural Networks[paper]
[Nature Machine Intelligence 2022] Molecular contrastive learning of representations via graph neural networks [paper]
[WWW 2022] SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation [paper] [code]
[WWW 2022] Dual Space Graph Contrastive Learning [paper]
[WWW 2022] Adversarial Graph Contrastive Learning with Information Regularization [paper]
[WWW 2022] The Role of Augmentations in Graph Contrastive Learning: Current Methodological Flaws & Improved Practices [paper]
[WWW 2022] ClusterSCL: Cluster-Aware Supervised Contrastive Learning on Graphs [paper]
[WWW 2022] Graph Communal Contrastive Learning [paper]
[TKDE 2022] CCGL: Contrastive Cascade Graph Learning [paper][code]
[BIBM 2021] Molecular Graph Contrastive Learning with Parameterized Explainable Augmentations [paper]
[WSDM 2022]Bringing Your Own View: Graph Contrastive Learning without Prefabricated Data Augmentations [paper] [code]
[SDM 2022] Structure-Enhanced Heterogeneous Graph Contrastive Learning [paper]
[AAAI 2022] GeomGCL: Geometric Graph Contrastive Learning for Molecular Property Prediction [paper]
[AAAI 2022] Self-supervised Graph Neural Networks via Diverse and Interactive Message Passing [paper]
[AAAI 2022] Augmentation-Free Self-Supervised Learning on Graphs [paper][code]
[AAAI 2022] Deep Graph Clustering via Dual Correlation Reduction [paper][code]
[ICOIN 2022] Adaptive Self-Supervised Graph Representation Learning [paper]
[arXiv 2022] Graph Masked Autoencoder [paper]
[arXiv 2022] Structural and Semantic Contrastive Learning for Self-supervised Node Representation Learning [paper]
[arXiv 2022] Graph Self-supervised Learning with Accurate Discrepancy Learning [paper]
[arXiv 2021] Multilayer Graph Contrastive Clustering Network [paper]
[arXiv 2021] Graph Representation Learning via Contrasting Cluster Assignments [paper]
[arXiv 2021] Graph-wise Common Latent Factor Extraction for Unsupervised Graph Representation Learning [paper]
[arXiv 2021] Bayesian Graph Contrastive Learning [paper]
[NeurIPS 2021 Workshop] Self-Supervised GNN that Jointly Learns to Augment [paper]
[NeurIPS 2021] Enhancing Hyperbolic Graph Embeddings via Contrastive Learning [paper]
[NeurIPS 2021] Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization [paper]
[NeurIPS 2021] Motif-based Graph Self-Supervised Learning for Molecular Property Prediction [paper]
[NeurIPS 2021] Graph Adversarial Self-Supervised Learning [paper]
[NeurIPS 2021] Contrastive laplacian eigenmaps [paper]
[NeurIPS 2021] Directed Graph Contrastive Learning [paper][code]
[NeurIPS 2021] Multi-view Contrastive Graph Clustering [paper][code]
[NeurIPS 2021] From Canonical Correlation Analysis to Self-supervised Graph Neural Networks [paper][code]
[NeurIPS 2021] InfoGCL: Information-Aware Graph Contrastive Learning [paper]
[NeurIPS 2021] Adversarial Graph Augmentation to Improve Graph Contrastive Learning [paper][code]
[NeurIPS 2021] Disentangled Contrastive Learning on Graphs [paper]
[arXiv 2021] Subgraph Contrastive Link Representation Learning [paper]
[arXiv 2021] Augmentations in Graph Contrastive Learning: Current Methodological Flaws & Towards Better Practices [paper]
[arXiv 2021] Collaborative Graph Contrastive Learning: Data Augmentation Composition May Not be Necessary for Graph Representation Learning [paper]
[CIKM 2021] Contrastive Pre-Training of GNNs on Heterogeneous Graphs [paper]
[CIKM 2021] Self-supervised Representation Learning on Dynamic Graphs [paper]
[CIKM 2021] SGCL: Contrastive Representation Learning for Signed Graphs [paper]
[CIKM 2021] Semi-Supervised and Self-Supervised Classification with Multi-View Graph Neural Networks [paper]
[arXiv 2021] Graph Communal Contrastive Learning [paper]
[arXiv 2021] Self-supervised Contrastive Attributed Graph Clustering [paper]
[arXiv 2021] Adaptive Multi-layer Contrastive Graph Neural Networks [paper]
[arXiv 2021] Graph-MVP: Multi-View Prototypical Contrastive Learning for Multiplex Graphs [paper]
[arXiv 2021] Spatio-Temporal Graph Contrastive Learning [paper]
[IJCAI 2021] Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning [paper]
[IJCAI 2021] Pairwise Half-graph Discrimination: A Simple Graph-level Self-supervised Strategy for Pre-training Graph Neural Networks [paper]
[arXiv 2021] RRLFSOR: An Efficient Self-Supervised Learning Strategy of Graph Convolutional Networks [paper]
[ICML 2021] Graph Contrastive Learning Automated [paper] [code]
[ICML 2021] Self-supervised Graph-level Representation Learning with Local and Global Structure [paper] [code] 52.[arXiv 2021] Group Contrastive Self-Supervised Learning on Graphs [paper]
[arXiv 2021] Multi-Level Graph Contrastive Learning [paper]
[KDD 2021] Pre-training on Large-Scale Heterogeneous Graph [paper]
[KDD 2021] Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning [paper] [code]
[arXiv 2021] Prototypical Graph Contrastive Learning [paper]
[arXiv 2021] Graph Barlow Twins: A self-supervised representation learning framework for graphs [paper]
[arXiv 2021] Self-Supervised Graph Learning with Proximity-based Views and Channel Contrast [paper]
[arXiv 2021] FedGL: Federated Graph Learning Framework with Global Self-Supervision [paper]
[IJCNN 2021] Node Embedding using Mutual Information and Self-Supervision based Bi-level Aggregation [paper]
[arXiv 2021] Graph Representation Learning by Ensemble Aggregating Subgraphs via Mutual Information Maximization [paper]
[arXiv 2021] Self-supervised Auxiliary Learning for Graph Neural Networks via Meta-Learning [paper]
[arXiv 2021] Towards Robust Graph Contrastive Learning [paper]
[arXiv 2021] Pre-Training on Dynamic Graph Neural Networks [paper]
[WWW 2021] Graph Contrastive Learning with Adaptive Augmentation [paper] [code]
[Arxiv 2020] Distance-wise Graph Contrastive Learning [paper]
[Openreview 2020] Motif-Driven Contrastive Learning of Graph Representations [paper]
[Openreview 2020] SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks [paper]
[Openreview 2020] TopoTER: Unsupervised Learning of Topology Transformation Equivariant Representations [paper]
[Openreview 2020] Graph-Based Neural Network Models with Multiple Self-Supervised Auxiliary Tasks [paper]
[NeurIPS 2020] Self-Supervised Graph Transformer on Large-Scale Molecular Data [paper]
[NeurIPS 2020] Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs [paper] [code]
[NeurIPS 2020] Graph Contrastive Learning with Augmentations [paper] [code]
[Arxiv 2020] Deep Graph Contrastive Representation Learning [paper]
[ICML 2020] When Does Self-Supervision Help Graph Convolutional Networks? [paper] [code]
[ICML 2020] Contrastive Multi-View Representation Learning on Graphs. [paper] [code]
[ICML 2020 Workshop] Self-supervised edge features for improved Graph Neural Network training. [paper]
[Arxiv 2020] Self-supervised Training of Graph Convolutional Networks. [paper]
[Arxiv 2020] Self-Supervised Graph Representation Learning via Global Context Prediction. [paper]
[KDD 2020] GPT-GNN: Generative Pre-Training of Graph Neural Networks. [pdf] [code]
[KDD 2020] GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training. [pdf] [code]
[Arxiv 2020] Graph-Bert: Only Attention is Needed for Learning Graph Representations. [paper] [code]
[ICLR 2020] InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization. [paper] [code]
[ICLR 2020] Strategies for Pre-training Graph Neural Networks. [paper] [code]
[KDD 2019 Workshop] SGR: Self-Supervised Spectral Graph Representation Learning. [paper]
[ICLR 2019 workshop] Pre-Training Graph Neural Networks for Generic Structural Feature Extraction. [paper]
[Arxiv 2019] Heterogeneous Deep Graph Infomax [paper] [code]
[ICLR 2019] Deep Graph Informax. [paper] [code]

Knowledge-Enriched Pretraining Strategies

[Nature Machine Itelligence 2022] Geometry-enhanced molecular representation learning for property prediction[paper]
[ICLR 2022] PRE-TRAINING MOLECULAR GRAPH REPRESENTATION WITH 3D GEOMETRY [paper] [code]
[AAAI 2022] Molecular Contrastive Learning with Chemical Element Knowledge Graph [paper]
[KDD 2021] MoCL: Data-driven Molecular Fingerprint via Knowledge-aware Contrastive Learning from Molecular Graph [paper] [code]
[arXiv 2021] 3D Infomax improves GNNs for Molecular Property Prediction [paper] [code]

Hard Negative Mining Strategies

[SDM 2022] Structure-Enhanced Heterogeneous Graph Contrastive Learning [paper]
[Signal Processing 2021] Negative Sampling Strategies for Contrastive Self-Supervised Learning of Graph Representations [paper]
[IJCAI 2021] Graph Debiased Contrastive Learning with Joint Representation Clustering [paper]
[IJCAI 2021] CuCo: Graph Representation with Curriculum Contrastive Learning [paper]
[arXiv 2021] Debiased Graph Contrastive Learning [paper]

Tuning Strategies

[KDD 2021] Adaptive Transfer Learning on Graph Neural Networks [paper]
[BioRxiv 2022] Towards Effective and Generalizable Fine-tuning for Pre-trained Molecular Graph Models[paper]
[AAAI 2022] CODE: Contrastive Pre-training with Adversarial Fine-tuning for Zero-shot Expert Linking [paper] [code] 5 [Arxiv 2022]Fine-Tuning Graph Neural Networks via Graph Topology induced Optimal Transport [paper]

Applications

[Arxiv 22] Protein Representation Learning by Geometric Structure Pretraining [paper]
[Nature Communications 2021] Masked graph modeling for molecule generation [paper]
[NPL 2022] How Does Bayesian Noisy Self-Supervision Defend Graph Convolutional Networks? [paper]
[arXiv 2022] Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data [paper]
[arXiv 2022] Link Prediction with Contextualized Self-Supervision [paper]
[arXiv 2022] Learning Robust Representation through Graph Adversarial Contrastive Learning [paper]
[WWW 2021] Multi-view Graph Contrastive Representation Learning for Drug-drug Interaction Prediction [paper]
[BIBM 2021] SGAT: a Self-supervised Graph Attention Network for Biomedical Relation Extraction [paper]
[ICBD 2021] Session-based Recommendation via Contrastive Learning on Heterogeneous Graph [paper]
[arXiv 2021] Graph Augmentation-Free Contrastive Learning for Recommendation [paper]
[arXiv 2021] TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning [paper]
[NeurIPS 2021 Workshop] Contrastive Embedding of Structured Space for Bayesian Optimisation [paper]
[ICCSNT 2021] Graph Data Augmentation based on Adaptive Graph Convolution for Skeleton-based Action Recognition [paper]
[arXiv 2021] Pre-training Graph Neural Network for Cross Domain Recommendation [paper]
[CIKM 2021] Social Recommendation with Self-Supervised Metagraph Informax Network [paper] [code]
[arXiv 2021] Self-Supervised Learning for Molecular Property Prediction [paper]
[arXiv 2021] Contrastive Graph Convolutional Networks for Hardware Trojan Detection in Third Party IP Cores [paper]
[KBS 2021] Multi-aspect self-supervised learning for heterogeneous information network [paper]
[arXiv 2021] Hyper Meta-Path Contrastive Learning for Multi-Behavior Recommendation [paper]
[arXiv 2021] Generative and Contrastive Self-Supervised Learning for Graph Anomaly Detection [paper]
[IJCAI 2021] CSGNN: Contrastive Self-Supervised Graph Neural Network for Molecular Interaction Prediction [paper]
[arXiv 2021] GCCAD: Graph Contrastive Coding for Anomaly Detection [paper]
[arXiv 2021] Contrastive Self-supervised Sequential Recommendation with Robust Augmentation [paper]
[KDD 2021] Contrastive Multi-View Multiplex Network Embedding with Applications to Robust Network Alignment [paper]
[arXiv 2021] Hop-Count Based Self-Supervised Anomaly Detection on Attributed Networks [paper]
[arXiv 2021] Representation Learning for Networks in Biology and Medicine: Advancements, Challenges, and Opportunities [paper]
[arXiv 2021] Drug Target Prediction Using Graph Representation Learning via Substructures Contrast [paper]
[Arxiv 2021] Self-Supervised Multi-Channel Hypergraph Convolutional Network for Social Recommendation [paper] [code]
[ICLR 2021] How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision [paper] [code]
[WSDM 2021] Pre-Training Graph Neural Networks for Cold-Start Users and Items Representation [paper] [code]
[ICML 2020] Graph-based, Self-Supervised Program Repair from Diagnostic Feedback. [paper]

Others

[arXiv 2022] A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applications [paper]
[NeurIPS 2021 datasets and benchmark track] An Empirical Study of Graph Contrastive Learning [paper]
[arXiv 2021] Evaluating Modules in Graph Contrastive Learning [paper] [code]
[arXiv 2021] Graph Self-Supervised Learning: A Survey [paper]
[arXiv 2021] Self-Supervised Learning of Graph Neural Networks: A Unified Review [paper]
[Arxiv 2020] Self-supervised Learning on Graphs: Deep Insights and New Direction. [paper] [code]
[ICLR 2019 Workshop] Can Graph Neural Networks Go "Online"? An Analysis of Pretraining and Inference. [paper]

Open-Sourced Pretrained Graph Models

PGMs	Architecture	Pretraining Database	# Params.	Download Link
Hu et al.	5-layer GIN	ZINC15 (2M) + ChEMBL (456K)	~ 2M	Link
Graph-BERT	Graph Transformer	Cora + CiteSeer + PubMed	N/A	Link
GraphCL	5-layer GIN	ZINC15 (2M) + ChEMBL (456K)	~ 2M	Link
GPT-GNN	HGT	OAG + Amazon	N/A	Link
GCC	5-layer GIN	Academia + DBLP + IMDB + Facebook + LiveJournal	<1M	Link
JOAO	5-layer GIN	ZINC15 (2M) + ChEMBL (456K)	~ 2M	Link
AD-GCL	5-layer GIN	ZINC15 (2M) + ChEMBL (456K)	~ 2M	N/A
GraphLog	5-layer GIN	ZINC15 (2M) + ChEMBL (456K)	~ 2M	Link
GROVER	GTransformer	ZINC + ChEMBL (10M)	48M ~ 100M	Link
MGSSL	5-layer GIN	ZINC15 (250K)	~ 2M	Link
CPT-HG	HGT	DBLP + YELP + Aminer	N/A	N/A
MPG	MolGNet	ZINC + ChEMBL (11M)	53M	N/A
LP-Info	5-layer GIN	ZINC15 (2M) + ChEMBL (456K)	~ 2M	Link
SimGRACE	5-layer GIN	ZINC15 (2M) + ChEMBL (456K)	~ 2M	Link
MolCLR	GCN + GIN	PubChem (10M)	N/A	Link
DMP	DeeperGCN + Transformer	PubChem (110M)	104.1 M	N/A
ChemRL-GEM	GeoGNN	ZINC15 (20M)	N/A	Link
KCL	GCN + KMPNN	ZINC15 (250K)	< 1M	N/A
3D Infomax	PNA	QM9(50K) + GEOM-drugs(140K) + QMugs(620K)	N/A	Link
GraphMVP	GIN + SchNet	GEOM (50k)	~ 2M	Link

Pretraing Datasets

Name	Category	Download Link
ZINC	Molecular Graph	Link
CheMBL	Molecular Graph	Link
PubChem	Molecular Graph	Link
QM9	Molecular Graph	Link
QMugs	Molecular Graph	Link
GEOM	Molecular Graph	Link

Citation (.bib)

@article{xia2022survey,
title={A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applications},
author={Xia, Jun and Zhu, Yanqiao and Du, Yuanqi and Stan Z., Li},
journal={arXiv preprint arXiv:2202.07893},
year={2022}}

YanyiPU/awesome-pretrain-on-graphs