/structured_uncertainty_metrics

Source code of "A Graph Is More Than Its Nodes: Towards Structured Uncertainty-Aware Learning on Graphs" (NeurIPS GLFrontiers Workshop 2022)

Primary LanguagePythonMIT LicenseMIT

structured_uncertainty_metrics

Source code of the paper "A Graph Is More Than Its Nodes: Towards Structured Uncertainty-Aware Learning on Graphs" [Paper]

Edgewise Uncertainty Metircs

Nodewise Edgewise Agree Disagree
GNNs



Structured
Prediction
Models



Requirements

  • python >= 3.6
  • matplotlib >= 3.2.2
  • numpy >= 1.19.5
  • pathlib2 2.3.5
  • torch 1.7.0+cu101
  • torch-geometric 2.0.1

Install the dependencies from requirements file. PyTorch and PyTorch-Geometric are installed with Cuda 10.1.

pip install -r requirements.txt

Compare GNNs with strutured prediction models

Train

  • Train GNNs(GCN or GAT). GNNs model nodewise marginals and do not model the label dependency.
PYTHONPATH=. python src/train.py --dataset Cora --model GCN
PYTHONPATH=. python src/train.py --dataset Cora --model GAT
  • Train structured prediction models(GMNN or EPFGNN). These models combinine GNNs with markov networks to model the output joint distribution.
PYTHONPATH=. python src/train_gmnn.py --dataset Cora --model GMNN
PYTHONPATH=. python src/train_epfgnn.py --dataset Cora --model EPFGNN

Evaluation

  • Evaluate the trained model (GCN, GAT, GMNN, or EPFGNN) with edgewise uncertainty metrics. Use --reli_diag to plot the reliability diagram.
PYTHONPATH=. python src/evaluation.py --dataset Cora --model <trained_model> --reli_diag

How to use the edgewise uncertainty metrics

We implemented easy-to-use wrappers for the metrics in src/metric.py. For detailed implementation please see src/calibloss.py. An example of evaluating your trained models can be like:

from src.metric import NodewiseMetrics, EdgewiseMetrics

log_porb = ... # Make sure your model output is log probability
gt, test_mask = ...  # label and test mask
eval_edge_index = ... # Edges we want to evaluate

node_eval = NodewiseMetrics(log_prob, gt, test_mask)
node_results = node_eval.acc(), node_eval.nll(), node_eval.brier(), node_eval.ece()
edge_eval = EdgewiseMetrics(log_prob, gt, eval_edge_index)
edge_results = edge_eval.acc(), edge_eval.nll(), edge_eval.brier(), edge_eval.ece()

Citation

@article{hsuh2022A,
  title={A Graph Is More Than Its Nodes: Towards Structured Uncertainty-Aware Learning on Graphs},
  author={Hans Hao-Hsun Hsu and Yuesong Shen and Daniel Cremers},
  journal={New Frontiers in Graph Learning Workshop, NeurIPS 2022},
  year={2022}
}