Resubmission should not be allowed after competition deadline
jgong5 opened this issue · 7 comments
I noticed that there were resubmissions from fast.ai for the ImageNet training track after competition deadline:
https://github.com/stanford-futuredata/dawn-bench-entries/blob/master/ImageNet/train/fastai_pytorch.json
https://github.com/stanford-futuredata/dawn-bench-entries/blob/master/ImageNet/train/fastai_pytorch.tsv
Their new result was baked from lots of code changes including hyper-parameter tuning and model changes. I don't think it is fair to other participants. Resubmission should not be allowed after competition deadline.
As a fair alternative, I would suggest the organizer to create a new ranking list for ImageNet without those blacklisted images. Any submission prior to the deadline can be ranked in the respective list. This can avoid confusion and also honor the game rules.
No code changes were made after the deadline. The re-submission was requested, and the only difference was the learning rate. You are incorrectly using an individual file's commit id.
I would ask that financial sponsors of this competition that also competed be mindful of their clear conflict of interest and refrain from any further behavior of this kind.
I was not referring to the commit ids of an individual file but the change history between these two ids (your first submission before deadline vs. your second submission after the deadline):
e47cc1a0a5e1b14c75b26a3ed67a4346124568a7
c4b225555e333a1a2702d2b291b5082bfa6d6a0a
The change diff fast.ai.zip was attached FYI. You can also get the same diff file with the command below:
git diff e47cc1a0a5e1b14c75b26a3ed67a4346124568a7 c4b225555e333a1a2702d2b291b5082bfa6d6a0a
Moreover, changing the learning rate is also a change of hyper-parameter. I don't think it should be allowed after the deadline.
I repeat my previous response.
I do hope that the other competition sponsors and organizers will take appropriate action against Intel's gross violation here of basic standards around conflicts of interest.
I would like to see a clear public statement from Intel that commits to making no further attempts to influence the competition organizers in this or any future competition in which they are both financial sponsors and competitors.
From what I can see from the diff there are only two possibilities. either the commit hash does not actually correspond to the submissions, or significant code changes (both model and hyperparameters) happened between the two submissions. I think @jgong5 is raising reasonable doubt because either one should not be allowed.
The only place that that commit hash appears in the submission is in this URL: https://github.com/fastai/imagenet-fast/blob/c4b225555e333a1a2702d2b291b5082bfa6d6a0a/imagenet_nv/main.py . The file history for that file clearly shows that there has been no changes to it since the closing date of the competition.
The model implementation in the submission is clearly shown as this URL: pytorch/vision@9889de1 . The date of that commit is from December of last year.
As you can see, the commit hashes are correct, and no code used in the submissions changed between the submissions. (There have been ongoing commits to other files since the original submission since we're continuing to work on content for our next class, but they have nothing to do with the above issues, as explained above.)
Furthermore, engaging with and amplifying statements tarnished by obvious conflicts of interest is highly inappropriate. The appropriate response is to clearly state that such behavior is unacceptable, and to engage with it no further.
I have stated 3 times now that no code changes were made between the submissions. This should not have been necessary.
We will not be responding to any further comments on github.
The model implementation in the submission is clearly shown as this URL: pytorch/vision@9889de1
I think it's very clear that, from the URL you gave: https://github.com/fastai/imagenet-fast/blob/c4b225555e333a1a2702d2b291b5082bfa6d6a0a/imagenet_nv/main.py, the model implementation does NOT come from pytorch/vision, but from import models, i.e. from inside the repo itself at https://github.com/fastai/imagenet-fast/tree/c4b225555e333a1a2702d2b291b5082bfa6d6a0a/imagenet_nv/models. Would you clarify this?
@deepakn94 and I reviewed the diff between e47cc1a0a5e1b14c75b26a3ed67a4346124568a7 and c4b225555e333a1a2702d2b291b5082bfa6d6a0a. Although there were code changes to the repo as @jgong5 mentioned, none of them affected the command @jph00 ran. The changes were to files that aren't in the code path they ran (e.g., CIFAR10 models) or cosmetic improvements to their ResNet implementation that do not impact the model or performance in any way.
For hyperparameter tuning, the command line used before the deadline of the competition included runs with learning rates of 0.3 and 0.4. I don't believe they did additional hyperparameter tuning because they ran for both learning rates originally.
Going forward, we believe @jph00 submission still stands as is and we are closing this issue. It seems like @jph00 just uses the same repo and branch for all of his experiments. As a best practice for future iterations, submissions should use a separate branch that is only changed based on requests during verification. If you have specific questions about the diff or commit, please feel free to comment.