Incorrect preprocessing for ImageNet-C evaluation
imrahulr opened this issue · 2 comments
imrahulr commented
I see that the ImageNet-C evaluation uses the preprocessing: Resize(256)+CenterCrop(224)+ToTensor()
.
robustbench/robustbench/data.py
Lines 146 to 154 in 61ce9e9
This causes discrepancies with the scores reported in the original papers (DeepAugment, AugMix, Standard RN-50). The ImageNet-C dataset already contains 224x224 images and hence only ToTensor()
should be used for consistency.
Fixing prepr='none'
in load_imagenetc
should solve the issue (assuming all the models are capable of handling 224x224 images as input).
max-andr commented
The suggestion of fixing prepr='none'
in load_imagenetc
definitely makes sense for the current models! We'll look into this!