Folders missing from the Zenodo files

[sayakpaul]

Hi @hendrycks.

I downloaded the four different tars as provided here: https://zenodo.org/record/3565846.

I think I am missing four folders here although I ensured the tars are not corrupted:

As per ImageNet-P/test.py I expected to see all the perturbations:

['gaussian_noise', 'shot_noise', 'motion_blur', 'zoom_blur',
                             'spatter', 'brightness', 'translate', 'rotate', 'tilt', 'scale',
                             'speckle_noise', 'gaussian_blur', 'snow', 'shear']

Also, inside the noise perturbations, I did not see separate difficulties as indicated inside the code.

Am I missing out on something?

[hendrycks]

The zenodo file is correct. Difficulty is an argument passed to the code which processes the files differently (rather than processing adjacent frames, it compares frames which are more temporally distant).

[sayakpaul]

So, as per this line, I think we still need to have a path like "ImageNet-P/shot_noise_3", right (in case we are testing shot_noise with difficult 3)? As far as I remember I did not have that kind of folder. I will double-check, though.

(rather than processing adjacent frames, it compares frames which are more temporally distant)

This makes sense. This code block makes that clear.

[hendrycks]

No, the step size parameter determines difficulty. No additional data is necessary.

[sayakpaul]

Okay. But I am still confused as to how the data loader would find a path like ImageNet-P/shot_noise_3 (in case we are testing shot_noise with difficult 3)?

[hendrycks]

You're right. It looks like I didn't add the different difficulties for noises because of file size. I'm not sure exactly where they are. If you want to see performance on different difficulties, perhaps just use the non-noise perturbations.

[sayakpaul]

Okay cool. If I wanted to report the results I will just report them separately instead of averaging them out then. Thanks, Dan.

[sayakpaul]

@hendrycks,

I have collected the TD5s and FRs (standardized with AlexNet TD5 and FR) on the 10 perturbations provided in the Zenodo files for difficulty 1.

Using what's laid out in the "IMAGENET-P Metrics" section of the paper, do I have the correct data to now report mFR and mT5D?

[hendrycks]

Yes.