Out-of-distribution Detection with Posterior Sampling

This project is for the paper: POEM: Out-of-distribution Detection with Posterior Sampling published at ICML 2022 (long talk). Some parts of the codebase are adapted from ODIN, Outlier Exposure, Deep Mahalanobis Detector and NTOM.

Required Packages

The following packages are required to be installed:

Our experiments are conducted on Ubuntu Linux 20.04 with Python 3.8.

Method Overview

In-distribution and Auxiliary Outlier Datasets

In-distribution training set:
- CIFAR: included in PyTorch.

Auxiliary outlier training set:
- 80 Million Tiny Images: to download 80 Million Tiny Images dataset. After downloading it, place it in this directory: datasets/unlabeled_datasets/80M_Tiny_Images
- ImageNet-RC (Downsampled ImageNet Datasets](https://patrykchrabaszcz.github.io/Imagenet32/): we use the ImageNet64x64, which can be downloaded from ImageNet Website. After downloading it, place it in this directory: datasets/unlabeled_datasets/ImageNet.

Out-of-distribution Test Datasets

We provide links and instructions to download each dataset:

SVHN: download it and place it in the folder of datasets/ood_datasets/svhn. Then run python select_svhn_data.py to generate test subset.
Textures: download it and place it in the folder of datasets/ood_datasets/dtd.
Places365: download it and place it in the folder of datasets/ood_datasets/places365/test_subset. We randomly sample 10,000 images from the original test dataset.
LSUN: download it and place it in the folder of datasets/ood_datasets/LSUN.
LSUN-resize: download it and place it in the folder of datasets/ood_datasets/LSUN_resize.
iSUN: download it and place it in the folder of datasets/ood_datasets/iSUN.

For example, run the following commands in the root directory to download LSUN:

cd datasets/ood_datasets
wget https://www.dropbox.com/s/fhtsw1m3qxlwj6h/LSUN.tar.gz
tar -xvzf LSUN.tar.gz

Quick Start

For example, to test the model trained with POEM at 100 epochs on CIFAR-10, please download the corresponding checkpoint to ./checkpoints/CIFAR-10/POEM_Sample_1/checkpoint_100.pth.tar. You can also train from scratch with train_poem.py. For each ID dataset, posterior sampling hyperparameters can be tuned for better performance especially the ones controlling the variances for noises and weights: sigma and sigma_n. We observe that training longer until the model fully converges is helpful. For more stable performance, we recommend training CIFAR-10 for 100 epochs and CIFAR-100 for 200 epochs. To reproduce the results in the paper, we provide checkpoints for tuned models here.

To evaluate the OOD detection performance of POEM, for example, with a checkpoint named POEM_Sample_1:

# Generate evaluation results for POEM:
python get_score.py --name POEM_Sample_1 --in-dataset CIFAR-10 --model-arch densenet --test_epochs 100

# Present OOD detection metrics based on results:
python get_results.py --name POEM_Sample_1  --in-dataset CIFAR-10 --test_epochs 100

For bibtex citations

If you find our work helpful for your research, please consider citing our paper:

@inproceedings{ming2022poem,
  author={Ming, Yifei  and Fan, Ying  and Li, Yixuan},
  title={POEM: Out-of-distribution Detection with Posterior Sampling},
  booktitle={International Conference on Machine Learning},
  year={2022},
  organization={PMLR}
}

Issues and further discussions

Please create an issue for code-related questions. For clarification and further discussions, free feel to drop an email at ming5@wisc.edu.

deeplearning-wisc/poem