pytorch/hub

Reported accuracy mismatch for `vgg11_bn`

nps1ngh opened this issue · 2 comments

The hub documentation here: https://pytorch.org/hub/pytorch_vision_vgg/
reports the vgg11_bn top-1 error to be 26.70. That is, a top-1 accuracy of 73.30.
This is almost as good as the resnet34 reported here: https://pytorch.org/vision/main/models/generated/torchvision.models.resnet34.html

But the main problem is that torchvision reports a top-1 accuracy of 70.37 for the it here: https://pytorch.org/vision/main/models/generated/torchvision.models.vgg11_bn.html

I guess the latter reported accuracy is correct (70.37) and the one reported here by hub is incorrect?

Or am I missing something? Is this possibly related to pytorch/vision#223? (It's a really old issue though.)

Thanks!

Thanks for the report @nps1ngh . Good catch!

For all torchivsion models, the correct reference for accuracies should be this table: https://pytorch.org/vision/main/models.html#table-of-all-available-classification-weights (or the specific model pages like the one you gave above, which has the same info). Unfortunately, the ones reported on the torchhub website may go out of date. I opened #318 to try to think of solutions / mitigations.

Meanwhile, it's probably worth updating the VGG table with the latest correct values. LMK if you would you like to open a PR for that. Thank you!

Sure, I'll open one!