brain-score/vision

Fix Score reproducibility discrepancy with Rank 12 and 18 models

Opened this issue · 1 comments

mike-ferguson commented

Currently the following models do not reproduce on MajajHong2015-IT:

  1. resnet50_finetune_cutmix_e3_robust_linf8255_e0_247x234
    • Website Score: 0.518, Local 2.0 Score: 0.5238, Local 1.0 score: 0.518
    • 1.0 Reproducibility status PR here: #884
  2. resnet50-VITO-8deg-cc
    • Website Score: 0.512, Local 2.0 Score: 0.5079, 1.0 Local score: 0.512
    • 1.0 Reproducibility status PR here: #883
pellegreene commented

Tried running on 2.0 and reverting the package versions in my environment, still received the same scores.