Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space

This is the PyTorch implementation of the paper Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space

Self-challenging Fisher Discriminative Analysis (SFDA)

We design a novel model selection method, termed Self-challenging Fisher Discriminative Analysis (SFDA), which is efficient, effective, and robust when measuring the transferability of pre-trained models. Compared with the state-of-the-art method NLEEP, SFDA demonstrates an average of 59.1% gain while bringing 22.5x speedup in wall-clock time.

Comparisons of weighted Kendall's tau on 11 downstream classification datasets when selecting 11 pretrained supervised models.

Method	Aircraft	Caltech101	Cars	CIFAR10	CIFAR100	DTD	Flowers	Food	Pets	SUN397	VOC2007
LEEP	-0.234	0.605	0.367	0.824	0.677	0.486	-0.243	0.491	0.389	0.701	0.446
LogME	0.506	0.435	0.576	0.852	0.692	0.647	0.111	0.385	0.411	0.511	0.478
NLEEP	0.495	0.661	0.265	0.806	0.823	0.777	0.215	0.624	0.599	0.807	0.654
PARC	-0.182	0.374	0.562	0.845	0.692	0.642	-0.082	0.732	0.138	0.698	0.723
SFDA	0.615	0.737	0.487	0.949	0.866	0.597	0.542	0.815	0.734	0.703	0.763

Getting Started

Install PyTorch

Clone the repo:

git clone https://git.woa.com/wenqishao/SFDA.git

Requirements

Install PyTorch==1.7.1 and torchvision==0.8.2 with CUDA==10.1:

conda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.1 -c pytorch

Install timm==0.4.9:

pip install timm==0.4.9

Data Preparation

Download the downstream datasets to ./data/*.

Pipeline of Model selection using transferability

Fine-tune pretrained models with hyper-paramters sweep to obtain ground-truth transferability score

python finetune_group1.py -m resnet50 -d cifar10

Extract features of target data using pretrained models

python forward_feature_group1.py -m resnet50 -d cifar10

Compute transferability scores using SFDA

python evaluate_metric_group1_cpu.py -me sfda -d cifar10

Assess the effectiveness of SFDA