CAME: Contrastive Automated Model Evaluation [Paper]
This repository contains:
- the PyTorch implementation of CAME
- the example on CIFAR-10 setup
- Contrastive Accuracy calculation and linear regression methods
- MNIST, CIFAR-100, COCO and TinyImageNet Setups (use imgaug to generate Meta-set).
Please see
PROJECT_DIR/meta_set/
Please follow the instruction below to install it and run the experiment demo.
- Linux (tested on Ubuntu 16.04LTS)
- NVIDIA GPU + CUDA CuDNN (tested on Tesla V100)
- SimCLR-v1 our codebase
- MNIST Dataset (download and unzip to
PROJECT_DIR/datasets/MNIST) - CIFAR-10 Dataset (download and unzip to
PROJECT_DIR/datasets/CIFAR10) - CIFAR10.1 Dataset (download and unzip to
PROJECT_DIR/datasets/CIFAR10_1) - CIFAR-10-C Dataset (download and unzip to
PROJECT_DIR/datasets/CIFAR-10-C) - CIFAR-100 Dataset (download and unzip to
PROJECT_DIR/datasets/CIFAR100) - CIFAR-100-C Dataset (download and unzip to
PROJECT_DIR/datasets/CIFAR-100-C) - COCO 2017 Dataset (download and unzip to
PROJECT_DIR/extra_data) - PASCAL Dataset (download and unzip to
PROJECT_DIR/datasets/PASCAL) - Caltech256 Dataset (download and unzip to
PROJECT_DIR/datasets/Caltech256) - ImageNet Dataset (download and unzip to
PROJECT_DIR/datasets/ImageNet) - TinyImageNet Dataset (download and unzip to
PROJECT_DIR/datasets/tiny-imagenet-200) - TinyImageNet-C Dataset (download and unzip to
PROJECT_DIR/datasets/Tiny-ImageNet-C) - SHVN,USPS,FashionMNIST,KMNIST,STL10 are the ready-made datasets in torchvision.datasets package
- All -C Operation can refers to Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
- You might need to change the file paths, and please be sure you change the corresponding paths in the codes as well
- Please see more details about COCO setup in Simon4Yan/Meta-set#2
-
Install dependencies
# COCOAPI cd $DIR/libs git clone https://github.com/cocodataset/cocoapi.git cd cocoapi/PythonAPI python setup.py build_ext install # CAME conda env create --name contrastive_autoeval --file environment.yml conda activate contrastive_autoeval
-
Co-train classifier
# Save as "PROJECT_DIR/checkpoints/CIFAR10/checkpoint.pth" python run.py -
Creat Meta-set
# By default it creates 400 sample sets python meta_set/synthesize_set_cifar.py -
Test classifier on Meta-set and save the fitted regression model
# Get "PROJECT_DIR/checkpoints/CIFAR10/accuracy_cifar.npy" file python regression.py -
Eval on unseen test sets by regression model
# 1) You will see Rank correlation and Pearsons correlation # 2) The absolute error of linear regression is also shown python eval.py
-
Correlation study
# You will see correlation.pdf; python figure/visualize_corr.py
If you use the code in your research, please cite:
@inproceedings{peng2023came,
title={Came: Contrastive automated model evaluation},
author={Peng, Ru and Duan, Qiuyang and Wang, Haobo and Ma, Jiachen and Jiang, Yanbo and Tu, Yongjun and Jiang, Xiu and Zhao, Junbo},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={20121--20132},
year={2023}
}MIT