Why ENAS gets a poor accuracy on NAS-Bench-201?

[xzhou29]

Describe the bug
Why it would have a much higher accuracy with just skip connections?

To Reproduce
CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/ENAS.sh cifar10 1 -1
Ran twice with these two seed numbers (randomly generated): 49363, 65491

Expected behavior
The results just do not make any sense to me.

Screenshots
seed-49363:

Loss=1.238, Accuracy@1=56.10%, Accuracy@5=95.49%
ENAS : run 250 epochs, cost 7187.7 s, last-geno is Structure(4 nodes with |skip_connect~0|+|skip_connect~0|skip_connect~1|+|skip_connect~0|skip_connect~1|skip_connect~2|).

seed-65491:

Loss=3.963, Accuracy@1=10.88%, Accuracy@5=66.97%
ENAS : run 250 epochs, cost 7059.1 s, last-geno is Structure(4 nodes with |avg_pool_3x3~0|+|skip_connect~0|avg_pool_3x3~1|+|skip_connect~0|avg_pool_3x3~1|skip_connect~2|).

[D-X-Y]

This is what we have discussed in our paper. ENAS and DARTS will quickly converge to the models that can converge fast but has low accuracy.

[xzhou29]

I didn't notice that on the paper.
Thank you so much!

[D-X-Y]

No problem, please let me know if you have any other questions.