IMS-LOMONAS: Parameter-less Pareto Local Search for Multi-objective Neural Architecture Search with the Interleaved Multi-start Scheme

Quan Minh Phan, Ngoc Hoang Luong

Setup

Clone this repo
Install necessary packages and databases.

$ cd IMS-LOMONAS
$ bash install.sh

Download data and database for the set of problems in the CEC'2023 Competition.
Run the below cell to load the data

from evoxbench.database.init import config
config(<database_path>, <data_path>)

For example:

from evoxbench.database.init import config
config('/content/drive/database', '/content/drive/data')

Reproducing the results

This repo have already implemented following NAS algorithms:

(IMS-)NSGA-II
(IMS-)NSGA-III
(IMS-)LOMONAS (ours)

CEC'2023 problems

To experiment on CEC'2023 problems, run the below script:

$ python main.py --optimizer <algo_name>[lomonas, ims-lomonas, nsga2, nsga3, ims-nsga2, ims-nsga3]
                 --test_suite <problem_name>[cec-c10, cec-in1k]
                 --pid <problem_id>[from 1 to 9]
                 --max_eval 10000 --n_run 31
                 --database_path <CEC_database_path> --data_path <CEC_data_path>
                 --using_archive --check_limited_neighbors --neighborhood_check_on_potential_sols --log_results

Other problems

To experiment on other problems, run the below script:

$ python main.py --optimizer <algo_name>[lomonas, ims-lomonas]
                 --test_suite <problem_name>[gecco]
                 --pid <problem_id>[from 1 to 8]
                 --max_eval 3000 --n_run 31
                 --database_path <CEC_database_path> --data_path <CEC_data_path>
                 --using_archive --check_limited_neighbors --neighborhood_check_on_potential_sols --log_results

where:

Problem ID	Search Space	Dataset	Target Objectives	Search Objectives
1	MacroNAS	CIFAR-10	test_err & params	val_err & params
2	NAS-Bench-101	CIFAR-10	test_err & params	val_err_12 & params
3	NAS-Bench-201	CIFAR-10	test_err & params	val_err_12 & params
4	NAS-Bench-201	ImageNet16-120	test_err & params	val_err_12 & params
5	NAS-Bench-201	CIFAR-10	test_err & params	val_err_12 & params
6	NAS-Bench-201	ImageNet16-120	test_err & params	val_err_12 & params
7	NAS-Bench-201	CIFAR-10	test_err & params	synflow & jacov & params
8	NAS-Bench-101	CIFAR-10	test_err & params	synflow & jacov & params

Set pid to 7 and 8 to experiment TF-(IMS-)LOMONAS.

Evaluation (only for NAS-Bench-101 and NAS-Bench-201 (GECCO))

$ python evaluate.py --res_path <result_path>
                     --problem_id <problem_id>
                     --dataset [cifar10, cifar100, ImageNet16-120]
                     --algo_name [lomonas, ims-lomonas]

Acknowledgement

We want to give our thanks to the authors of NAS-Bench-101, NAS-Bench-201, and NAS-Bench-ASR for their search spaces; to the authors of Zero-cost Lightweight NAS and NAS-Bench-Zero-Suite for their zero-cost metric databases.