stanford-futuredata/dawn-bench-entries

AmoebaNet does not use the full validation set

ddkang opened this issue · 8 comments

@bignamehyp as per the other thread, setting "eval_batch_size=1024" is incorrect, as it will skip parts of the validation set. Could you rerun the validation checkpoints using a correct batch size and update the TSVs and json files?

See here: #48

I'm on vacation util June 4th with limited Internet connectivity. Is there a deadline for this? I will try to squeeze some time to run the evaluation again.

@bignamehyp we don't have an explicit deadline, but we would like to resolve this in a timely manner. Can anyone else from your team re-evaluate your submissions?

I will start rerun the evaluation with eval_batch_size=1000 and update the accuracies soon.

@ddkang @codyaustun please take a look. Thanks

Great, thanks. The updated TSVs look good, but can you also update the JSON files to use eval_batch_size=1000?

Done

Thanks!