Search-RegNet-PyTorch

Implemention for neural architecture search of RegNet using PyTorch and AutoTorch.

This example and Fast AutoAugment will be used in the tutorial on From HPO to NAS: Automated Deep Learning at CVPR 2020.

model	ref	Acc	config
RegNet-0.4GF	official	72.38	link
RegNet-0.4GF	ours	72.18	link
RegNet-4.0GF	official	79.03	link

official: using official configuration. ours: using our searched configuration.

Training HP setting: learning rate: 0.2, batch size: 512, weight decay: 1e-4,

Quick Start

Install Dependencies

Install PyTorch, following the instruction.
Install other dependencies:

pip install autotorch thop torch-encoding

Install Apex (optional):

git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

Test #Params and FLOPs from config file

python test_flops.py --config-file configs/RegNetX-4.0GF.ini

Single Model Training

Prepare ImageNet Dataset

cd scripts/
# assuming you have downloaded the dataset in the current folder
python prepare_imagenet.py --download-dir ./

Train a single model from a config file

python train.py --dataset imagenet --config-file configs/RegNetX-4.0GF.ini --lr-scheduler cos --epochs 120 --checkname default --lr 0.025 --batch-size 64 --amp

Architecture Search

Generate config files with expected GFLOPs

python generate_configs.py --gflops 4 --num-configs 32 --config-file configs/RegNetX-4.0GF

The generated configuration files will be saved as configs/RegNetX-4.0GF-1.ini, configs/RegNetX-4.0GF-2.ini ...

Search best model for the config files in a folder

In this example, each model will be trained using a single gpu for 25 epochs.

python search.py --config-file-folder gen_configs/RegNet-0.4GF/ --output-folder out_configs/ --epochs 25

The accuracy will be written into the output config file after training.

dutxubo/RegNet-Search-PyTorch