Is there a way to query the best network architecture and accuracy
linnanwang opened this issue · 1 comments
linnanwang commented
Hello,
This will be an improvement. Could you please suggest the simplest way to query the best model architecture and its accuracy? Thank you.
chrisying commented
Hi linnanwang,
There are multiple ways to define the "best model architecture" (e.g. highest validation, highest test, mean validation, mean test, etc...). Here's how I would find the best model using any of these metrics:
best_metric = -1 # higher = better
best_cell = None
for h in nasbench.hash_iterator():
fixed, computed = nasbench.get_metrics_from_hash(h)
metric = your_metric_function(computed) # e.g. average test accuracy
if metric > best_metric:
best_metric = metric
best_cell = (fixed['module_adjacency'], fixed['module_operations'])
best_spec = api.ModelSpec(best_cell[0], best_cell[1])
This should only take a few seconds (or a minute or so) to run.
For your reference, the best spec according to MEAN TEST ACCURACY is:
best_spec = api.ModelSpec([[0, 1, 1, 0, 0, 1, 1],
[0, 0, 0, 0, 0, 1, 0],
[0, 0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0],
[0, 0, 0, 0, 0, 1, 0],
[0, 0, 0, 0, 0, 0, 1],
[0, 0, 0, 0, 0, 0, 0]],
[u'input', u'conv1x1-bn-relu', u'conv3x3-bn-relu',
u'maxpool3x3', u'conv3x3-bn-relu',
u'conv3x3-bn-relu', u'output'])
EDIT: correction to best_spec