Cannot re-implement your claimed result
NoOneUST opened this issue · 3 comments
Hello, when I am trying to re-implement your result on Cifar-10 with your code, I search 4 times, and train them with your code for 600 epochs, the best accuracy on validation set are 96.77, 97.32, 97.35, 97.21 separately. But in paper you claim the accuracy on testing set is 97.43+-0.07. Obviously here is a significant gap, why does this happen? Hope to get your response, thank you!
@NoOneUST , the reported accuracy 97.43+-0.07 is the best searched model (not the average accuracy of search four times) which trained four times (we follow the original DARTS paper). As the CIFAR dataset is small and the training period involved many argumentation strategies, the result can not be that stable. The search on ImageNet(We also offer the implementations) is much more stable.
@NoOneUST , the reported accuracy 97.43+-0.07 is the best searched model (not the average accuracy of search four times) which trained four times (we follow the original DARTS paper). As the CIFAR dataset is small and the training period involved many argumentation strategies, the result can not be that stable. The search on ImageNet(We also offer the implementations) is much more stable.
Thanks, when testing, simply use command 'python test.py' right? And can you provide your search results' distribution on both datasets? We hope to compare our improved method with yours.