NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search [slides][arXiv]

@inproceedings{
  lu2020nsganetv2,
  title={{NSGANetV2}: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search},
  author={Zhichao Lu and Kalyanmoy Deb and Erik Goodman and Wolfgang Banzhaf and Vishnu Naresh Boddeti},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2020}
}

Overview

NSGANetV2 is an efficient NAS algorithm for generating task-specific models that are competitive under multiple competing objectives. It comprises of two surrogates, one at the architecture level to improve sample efficiency and one at the weights level, through a supernet, to improve gradient descent training efficiency.

Datasets

Download the datasets from the links embedded in the names. Datasets with * can be automatically downloaded.

Dataset	Type	Train Size	Test Size	#Classes
ImageNet	multi-class	1,281,167	50,000	1,000
CINIC-10		180,000	9,000	10
CIFAR-10*		50,000	10,000	10
CIFAR-100*		50,000	10,000	10
STL-10*		5,000	8,000	10
FGVC Aircraft*	fine-grained	6,667	3,333	100
DTD		3,760	1,880	47
Oxford-IIIT Pets		3,680	3,369	37
Oxford Flowers102		2,040	6,149	102

How to evalute NSGANetV2 models

Download the models (net.config) and weights (net.init) from [Google Drive] or [Baidu Yun](提取码：4isq).

""" NSGANetV2 pretrained models
Syntax: python validation.py \
    --dataset [imagenet/cifar10/...] --data /path/to/data \
    --model /path/to/model/config/file --pretrained /path/to/model/weights 
"""

ImageNet	CIFAR-10	CINIC10

FLOPs@225: [Google Drive] FLOPs@312: [Google Drive] FLOPs@400: [Google Drive] FLOPs@593: [Google Drive]	FLOPs@232: [Google Drive] FLOPs@291: [Google Drive] FLOPs@392: [Google Drive] FLOPs@468: [Google Drive]	FLOPs@317: [Google Drive] FLOPs@411: [Google Drive] FLOPs@501: [Google Drive] FLOPs@710: [Google Drive]

Flowers102	Aircraft	Oxford-IIIT Pets

FLOPs@151: [Google Drive] FLOPs@218: [Google Drive] FLOPs@249: [Google Drive] FLOPs@317: [Google Drive]	FLOPs@176: [Google Drive] FLOPs@271: [Google Drive] FLOPs@331: [Google Drive] FLOPs@502: [Google Drive]	FLOPs@137: [Google Drive] FLOPs@189: [Google Drive] FLOPs@284: [Google Drive] FLOPs@391: [Google Drive]

CIFAR-100	DTD	STL-10

FLOPs@261: [Google Drive] FLOPs@398: [Google Drive] FLOPs@492: [Google Drive] FLOPs@796: [Google Drive]	FLOPs@123: [Google Drive] FLOPs@164: [Google Drive] FLOPs@202: [Google Drive] FLOPs@213: [Google Drive]	FLOPs@240: [Google Drive] FLOPs@303: [Google Drive] FLOPs@436: [Google Drive] FLOPs@573: [Google Drive]

How to use MSuNAS to search

""" Bi-objective search
Syntax: python msunas.py \
    --dataset [imagenet/cifar10/...] --data /path/to/dataset/images \
    --save search-xxx \ # dir to save search results
    --sec_obj [params/flops/cpu] \ # objective (in addition to top-1 acc)
    --n_gpus 8 \ # number of available gpus
    --supernet_path /path/to/supernet/weights \
    --vld_size [10000/5000/...] \ # number of subset images from training set to guide search
    --n_epochs [0/5]
"""

Download the pre-trained (on ImageNet) supernet from here.
It supports searching for FLOPs, Params, and Latency as the second objective.
- To optimize latency on your own device, you need to first construct a look-up-table for your own device, like this.
Choose an appropriate --vld_size to guide the search, e.g. 10,000 for ImageNet, 5,000 for CIFAR-10/100.
Set --n_epochs to 0 for ImageNet and 5 for all other datasets.
See here for some examples.
Output file structure:
- Every architecture sampled during search has net_x.subnet and net_x.stats stored in the corresponding iteration dir.
- A stats file is generated by the end of each iteration, iter_x.stats; it stores every architectures evaluated so far in ["archive"], and iteration-wise statistics, e.g. hypervolume in ["hv"], accuracy predictor related in ["surrogate"].
- In case any architectures failed to evaluate during search, you may re-visit them in failed sub-dir under experiment dir.

ImageNet	CIFAR-10

How to choose architectures

Once the search is completed, you can choose suitable architectures by:

You have preferences, e.g. architectures with xx.x% top-1 acc. and xxxM FLOPs, etc.

""" Find architectures with objectives close to your preferences
Syntax: python post_search.py \
    -n 3 \ # number of desired architectures you want, the most accurate archecture will always be selected 
    --save search-imagenet/final \ # path to the dir to store the selected architectures 
    --expr search-imagenet/iter_30.stats \ # path to last iteration stats file in experiment dir 
    --prefer top1#80+flops#150 \ # your preferences, i.e. you want an architecture with 80% top-1 acc. and 150M FLOPs 
    --supernet_path /path/to/imagenet/supernet/weights \
"""

If you do not have preferences, pass None to argument --prefer, architectures will then be selected based on trade-offs.
All selected architectures should have three files created:
- net.subnet: use to sample the architecture from the supernet
- net.config: configuration file that defines the full architectural components
- net.inherited: the inherited weights from supernet

How to validate architectures

To realize the full potential of the searched architectures, we further fine-tune from the inherited weights. Assuming that you have both net.config and net.inherited files.

""" Fine-tune on ImageNet from inherited weights
Syntax: sh scripts/distributed_train.sh 8 \ # of available gpus
    /path/to/imagenet/data/ \
    --model [nsganetv2_s/nsganetv2_m/...] \ # just for naming the output dir
    --model-config /path/to/model/.config/file \
    --initial-checkpoint /path/to/model/.inherited/file \
    --img-size [192, ..., 224, ..., 256] \ # image resolution, check "r" in net.subnet
    -b 128 --sched step --epochs 450 --decay-epochs 2.4 --decay-rate .97 \
    --opt rmsproptf --opt-eps .001 -j 6 --warmup-lr 1e-6 \
    --weight-decay 1e-5 --drop 0.2 --drop-path 0.2 --model-ema --model-ema-decay 0.9999 \
    --aa rand-m9-mstd0.5 --remode pixel --reprob 0.2 --amp --lr .024 \
    --teacher /path/to/supernet/weights \
"""

Adjust learning rate as (batch_size_per_gpu * #GPUs / 256) * 0.006 depending on your system config.

""" Fine-tune on CIFAR-10 from inherited weights
Syntax: python train_cifar.py \
    --data /path/to/CIFAR-10/data/ \
    --model [nsganetv2_s/nsganetv2_m/...] \ # just for naming the output dir
    --model-config /path/to/model/.config/file \
    --img-size [192, ..., 224, ..., 256] \ # image resolution, check "r" in net.subnet
    --drop 0.2 --drop-path 0.2 \
    --cutout --autoaugment --save
"""

More Use Cases (coming soon)

With a different supernet (search space).
NASBench 101/201.
Architecture visualization.

Requirements

Python 3.7
Cython 0.29 (optional; makes non_dominated_sorting faster in pymoo)
PyTorch 1.5.1
pymoo 0.4.1
torchprofile 0.0.1 (for FLOPs calculation)
OnceForAll 0.0.4 (lower level supernet)
timm 0.1.30
pySOT 0.2.3 (RBF surrogate model)
pydacefit 1.0.1 (GP surrogate model)

zhangshuaitao/nsganetv2