TorchCP is a Python toolbox for conformal prediction research on deep learning models, using PyTorch. Specifically, this toolbox has implemented some representative methods (including posthoc and training methods) for classification and regression tasks. We build the framework of TorchCP based on AdverTorch. This codebase is still under construction and maintained by Hongxin Wei's research group at SUSTech. Comments, issues, contributions, and collaborations are all welcomed!

Overview

TorchCP has implemented the following methods:

Classification

Year	Title	Venue	Code Link	Implementation
2023	Class-Conditional Conformal Prediction with Many Classes	NeurIPS	Link	classification.predictors.cluster
2023	Conformal Prediction for Deep Classifier via Label Ranking	Arxiv	Link	classification.scores.saps
2021	Learning Optimal Conformal Classifiers	ICLR	Link	classification.loss.conftr
2020	Uncertainty Sets for Image Classifiers using Conformal Prediction	ICLR	Link	classification.scores.raps
2020	Classification with Valid and Adaptive Coverage	NeurIPS	Link	classification.scores.aps
2019	Conformal Prediction Under Covariate Shift	NeurIPS	Link	classification.predictors.weight
2016	Least Ambiguous Set-Valued Classifiers with Bounded Error Levels	JASA		classification.scores.thr
2015	Bias reduction through conditional conformal prediction	Intell. Data Anal.		classification.scores.margin
2013	Applications of Class-Conditional Conformal Predictor in Multi-Class Classification	ICMLA		classification.predictors.classwise

Regression

Year	Title	Venue	Code Link	Implementation
2023	Conformal Prediction via Regression-as-Classification	RegML @ NeurIPS 2023	link	regression.predictors.r2ccp
2021	Adaptive Conformal Inference Under Distribution Shift	NeurIPS	Link	regression.predictors.aci
2019	Conformalized Quantile Regression	NeurIPS	Link	regression.predictors.cqr
2016	Distribution-Free Predictive Inference For Regression	JASA	Link	regression.predictors.split

TODO

TorchCP is still under active development. We will add the following features/items down the road:

Year	Title	Venue	Code Link
2022	Training Uncertainty-Aware Classifiers with Conformalized Deep Learning	NeurIPS	Link
2022	Adaptive Conformal Predictions for Time Series	ICML	Link
2022	Predictive Inference with Feature Conformal Prediction	ICLR	Link
2022	Conformal Prediction Sets with Limited False Positives	ICML	Link
2021	Optimized conformal classification using gradient descent approximation	Arxiv

Installation

TorchCP is developed with Python 3.9 and PyTorch 2.0.1. To install TorchCP, simply run

pip install torchcp

To install from TestPyPI server, run

pip install --index-url https://test.pypi.org/simple/ --no-deps torchcp

Examples

Here, we provide a simple example for a classification task, with THR score and SplitPredictor.

from torchcp.classification.scores import THR
from torchcp.classification.predictors import SplitPredictor

# Preparing a calibration data and a test data.
cal_dataloader = ...
test_dataloader = ...
# Preparing a pytorch model
model = ...

model.eval()

# Options of score function: THR, APS, SAPS, RAPS
# Define a conformal prediction algorithm. Optional: SplitPredictor, ClusteredPredictor, ClassWisePredictor
predictor = SplitPredictor(score_function=THR(), model=model)

# Calibrating the predictor with significance level as 0.1
predictor.calibrate(cal_dataloader, alpha=0.1)

#########################################
# Predicting for test instances
########################################
test_instances = ...
predict_sets = predictor.predict(test_instances)
print(predict_sets)

#########################################
# Evaluating the coverage rate and average set size on a given dataset.
########################################
result_dict = predictor.evaluate(test_dataloader)
print(result_dict["Coverage_rate"], result_dict["Average_size"])

You may find more tutorials in examples folder.

Documentation

The documentation webpage is on readthedocs https://torchcp.readthedocs.io/en/latest/index.html.

License

This project is licensed under the LGPL. The terms and conditions can be found in the LICENSE and LICENSE.GPL files.

Citation

If you find our repository useful for your research, please consider citing the following technical report:

@misc{wei2024torchcp,
      title={TorchCP: A Library for Conformal Prediction based on PyTorch}, 
      author={Hongxin Wei and Jianguo Huang},
      year={2024},
      eprint={2402.12683},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

We welcome you to cite the following works:

@article{huang2023conformal,
  title={Conformal Prediction for Deep Classifier via Label Ranking},
  author={Huang, Jianguo and Xi, Huajun and Zhang, Linjun and Yao, Huaxiu and Qiu, Yue and Wei, Hongxin},
  journal={arXiv preprint arXiv:2310.06430},
  year={2023}
}

@article{xi2024does,
  title={Does Confidence Calibration Help Conformal Prediction?},
  author={Xi, Huajun and Huang, Jianguo and Feng, Lei and Wei, Hongxin},
  journal={arXiv preprint arXiv:2402.04344},
  year={2024}
}

ml-stat-Sustech/TorchCP