Api.py: TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'

Hey,

first, thanks for your awesome work.

Describe the bug

While trying NAS-Bench-201, I stumbled across a bug, which occurs calling the get_more_info()-function with an old benchmark file. I am using the benchmark file NAS-Bench-201-v1_0-e61699.pth from here.

To Reproduce

from nas_201_api import NASBench201API as API

api = API('./NAS-Bench-201-v1_0-e61699.pth')
result_dict = api.get_more_info(index=0,
                                dataset='cifar10-valid',
                                iepoch=11,
                                use_12epochs_result=False,
                                is_random=False)

Output:
Traceback (most recent call last):
File "", line 10, in
File "/home/philipp/anaconda3/envs/HPOlib3/lib/python3.6/site-packages/nas_201_api/api.py", line 293, in get_more_info
'train-per-time': train_info['all_time'] / total,
TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'

The error is caused by a missing None-value check here:

AutoDL-Projects/lib/nas_201_api/api.py

Lines 291 to 294 in 91ee265

    
           xinfo = {'train-loss'    : train_info['loss'], 
        
                    'train-accuracy': train_info['accuracy'], 
        
                    'train-per-time': train_info['all_time'] / total, 
        
                    'train-all-time': train_info['all_time']}

Used versions
Benchmark file: [2020.02.25] APIv1.0/FILEv1.0: NAS-Bench-201-v1_0-e61699.pth
nas-bench-201: 1.3
python: 3.6

Expected behavior
This benchmark does not contain values for the fields all_time in the infos dict for train, test and valid. I guess a simple check if the time value is None should be sufficient to fix this. The "old" code had this None-value check.

Thanks in advance

Thanks for pointing out this issue!

1, I would recommend you to use our newest benchmark file (NAS-Bench-201-v1_1-096897.pth): https://drive.google.com/file/d/16Y0UwGisiouVRxW-W5hEtbxmcHw_0hF_/view?usp=sharing , where all all-time are avaliable.

2, I will make a major change to the API codes soon, where this issue will be fixed. I will post the news here once I push the new codes.

Best regards,

@PhMueller I just pushed the new codes, you can have a try. With this new code, you can use the following to query:

from nas_201_api import NASBench201API as API

api = API('./NAS-Bench-201-v1_0-e61699.pth')
result_dict = api.get_more_info(index=0,
                                dataset='cifar10-valid',
                                iepoch=11,
                                hp='200',
                                is_random=False)

where hp='200' indicates using the hyper-parameters of 200 epoch training.

Thanks a lot!

You are welcome~

	xinfo = {'train-loss' : train_info['loss'],
	'train-accuracy': train_info['accuracy'],
	'train-per-time': train_info['all_time'] / total,
	'train-all-time': train_info['all_time']}