An up-to-date version including metrics computation codes is available at https://github.com/m-Just/OoD-Bench
================================================================================================================================================
OoD-Bench
This is the code repository of the paper OoD-Bench: Benchmarking and Understanding Out-of-Distribution Generalization Datasets and Algorithms. Currently, this repository only contains the code (modified from the PyTorch suite DomainBed) for our benchmark experiments. We will release the code for estimating diversity and correlation shift in the future.
Data preparation
Most of the datasets (except for CelebA and NICO) can be downloaded by running the script DomainBed/domainbed/scripts/download.py.
Place them under datasets and make sure the directory structures are as follows:
PACS
└── kfold
├── art_painting
├── cartoon
├── photo
└── sketch
office_home
├── Art
├── Clipart
├── Product
├── Real World
├── ImageInfo.csv
└── imagelist.txt
terra_incognita
├── location_38
├── location_43
├── location_46
└── location_100
WILDS
└── camelyon17_v1.0
├── patches
└── metadata.csv
MNIST
└── processed
├── training.pt
└── test.pt
celeba
├── img_align_celeba
└── blond_split
├── tr_env1_df.pickle
├── tr_env2_df.pickle
└── te_env_df.pickle
NICO
├── animal
├── vehicle
└── mixed_split_corrected
├── env_train1.csv
├── env_train2.csv
├── env_val.csv
└── env_test.csv
Warning: a bug has been found in the split generating code for NICO. Please refrain from using the original version mixed_split. The correct split is now mixed_split_corrected. Corresponding experiment results will be updated to our paper later.
Running the experiments
To replicate the benchmark results, see the scripts under DomainBed/sweep.
Example usage:
bash sweep/ColoredMNIST_IRM/run.sh launch ../datasets 0