Kendall Tau in FAIRNAS

[pprp]

Another paper EVALUATING THE SEARCH PHASE OF NEURAL ARCHITECTURE SEARCH tested FairNAS on NASBench101 but get the Kendall Tau of -0.23.

FairNAS using 13 models to evaulate the rank and get the Kendall Tau of 0.9487.

I think the number of models used in FairNAS is the way too little and can not really reflect the rank ability of FairNAS

[cxxgtxy]

You can run the code in that paper and will find that the conclusion is not valid since the super-net is not trained at all.