/regression_uncertainty

Official implementation of "How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?", TMLR 2023.

Primary LanguagePythonMIT LicenseMIT

regression_uncertainty

overview image

Official implementation (PyTorch) of the paper:
How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?, 2023 [arXiv] [project].
Fredrik K. Gustafsson, Martin Danelljan, Thomas B. Schön.
We propose an extensive benchmark for testing the reliability of regression uncertainty estimation methods under real-world distribution shifts. It consists of 8 publicly available image-based regression datasets with different types of challenging distribution shifts. We use our benchmark to evaluate many of the most common uncertainty estimation methods, as well as two state-of-the-art uncertainty scores from OOD detection. We find that while methods are well calibrated when there is no distribution shift, they all become highly overconfident on many of the benchmark datasets. This uncovers important limitations of current uncertainty estimation methods, and the proposed benchmark thus serves as a challenge to the research community.

If you find this work useful, please consider citing:

@article{gustafsson2023reliable,
  title={How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?},
  author={Gustafsson, Fredrik K and Danelljan, Martin and Sch{\"o}n, Thomas B},
  journal={arXiv preprint arXiv:2302.03679},
  year={2023}
}

Datasets:

Cells:

Cells-Tails:

Cells-Gap:

ChairAngle:

ChairAngle-Tails:

ChairAngle-Gap:

AssetWealth:

  • $ pip install wilds
  • Run AssetWealth/create_datasets.py to download the dataset (13 GB).

VentricularVolume:

  • Download EchoNet-Dynamic.zip from https://echonet.github.io/dynamic/index.html#dataset and extract.
  • Move the resulting EchoNet-Dynamic folder to regression_uncertainty/datasets.
  • Run VentricularVolume/create_datasets.py to generate the train, val and test splits.

BrainTumourPixels:

  • Download Task01_BrainTumor.tar from http://medicaldecathlon.com/ and extract.
  • Move the resulting Task01_BrainTumour folder to regression_uncertainty/datasets.
  • Run BrainTumourPixels/create_datasets.py to generate the train, val and test splits.

SkinLesionPixels:

  • Download the data from https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/DBW86T (HAM10000_images_part_1.zip, HAM10000_images_part_2.zip, HAM10000_metadata, HAM10000_segmentations_lesion_tschandl.zip).
  • Create the folder regression_uncertainty/datasets/ham10000.
  • Move the files to regression_uncertainty/datasets/ham10000 and then extract.
  • Run SkinLesionPixels/create_datasets.py to generate the train, val and test splits.

HistologyNucleiPixels:

AerialBuildingPixels:

  • Download the data from https://project.inria.fr/aerialimagelabeling/ and extract.
  • Move the resulting AerialImageDataset folder to regression_uncertainty/datasets.
  • Run AerialBuildingPixels/create_datasets.py to generate the train, val and test splits.