Corruption acc. of a Resnet50 trained on cifar10 with augmix

[kiranchari]

Hi there, I trained a Resnet50 on CIFAR10 using the cifar.py script in this repository.

The clean acc. was about 95% but corruption accuracy was less than reported in the original paper. I have pasted below accuracies for CIFAR10-C at Severity 5. The mean corruption acc. across all corruptions and severity levels was 81%. I understand the architecture used in the original work was different, so is this an expected corruption acc. variation with the architecture?

Thank you.

Corruption severity: 5
gaussian_noise 0.6097
shot_noise 0.6647
impulse_noise 0.6747
speckle_noise 0.6906
defocus_blur 0.79
glass_blur 0.5383
motion_blur 0.7412
zoom_blur 0.7565
gaussian_blur 0.738
snow 0.7785
frost 0.7448
fog 0.6972
brightness 0.8831
spatter 0.8641
contrast 0.454
elastic_transform 0.6711
pixelate 0.5521
jpeg_compression 0.753
saturate 0.8935

[hendrycks]

I cannot predict how well a ResNet-50 would do on CIFAR-10-C. Because it's an uncommon architecture for CIFAR-10, I don't have prior experience with that specific model in that specific setting.

…

On Mon, Sep 6, 2021 at 2:48 AM kiranchari ***@***.***> wrote: I trained a Resnet50 on CIFAR10 using the cifar.py script in this repository. The clean acc. was about 95% but corruption accuracy was much less than reported in the original paper. I have pasted below accuracies for CIFAR10-C at Severity 5. The mean corruption acc. across all corruptions and severity levels was 81%. I understand the architecture used in the original work was different, so is this an expected corruption acc. variation with the architecture? Thank you. *Corruption severity: 5* gaussian_noise 0.6097 shot_noise 0.6647 impulse_noise 0.6747 speckle_noise 0.6906 defocus_blur 0.79 glass_blur 0.5383 motion_blur 0.7412 zoom_blur 0.7565 gaussian_blur 0.738 snow 0.7785 frost 0.7448 fog 0.6972 brightness 0.8831 spatter 0.8641 contrast 0.454 elastic_transform 0.6711 pixelate 0.5521 jpeg_compression 0.753 saturate 0.8935 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#21>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACZBITVEMJBBCGALBLJWGILUASFAVANCNFSM5DQCRYQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[kiranchari]

This issue can be safely ignored. I get results close to those in the augmix paper when I use the evaluation script in this repository. I was previously using my own evaluation script for CIFAR10-C, there were some minor differences in the preprocessing.