A question about the result of compute_ablations.py

[LZY-233]

Hi, thanks for your wonderful work.
Here, I have a question about the result of compute_ablations.py. I run the compute_ablations.py with the instruction: python compute_ablations.py --model ViT-B-16 the model is ViT-B-16 as you mentioned and the dataset is imagenet . then I get the result as following
Baseline: 0.008
+MLPs ablation: 0.008
+CLS ablation: 0.01
+MLPs + CLS ablation: 0.01
Attention ablations: [0.008, 0.008, 0.008, 0.008, 0.008, 0.008, 0.008, 0.008, 0.008, 0.008, 0.008, 0.008, 0.01]
It seems that with the ablation, the accuracy rate has increased.

[yossigandelsman]

Hi, thank you for your interest in the work!
I guess that you did not configure the data paths correctly. The baseline should be significantly higher (it's the accuracy on imagenet, so it should be above 0.5).
Try to run clip as is and see what you get

[LZY-233]

Thank you for your response. Now I have discovered that the result is due to an error in the calculation of the labels in the imagenet dataset in the compute_ablations.py. I have create a pull request to solve this bug.