Outdated Database?

[jecummin]

I tried running all the testbed models on the ImageNet validation set but of the ~400 models, only 200 or so actually ran. All the other models fail with the same error as #6 (comment). (This includes the newly added CLIP models).

I did some digging around and discovered that error is a result of an sql query to the model database to fetch the model checkpoints returning no results. I then discovered that the database dump that the code is set to download (which I believe is https://vasa.millennium.berkeley.edu:9000/robustness-eval/robustness_evalutation.db) is outdated and doesn't support all the new additions to the model registry in src/registry.

For example,

imagenet-testbed$ python db.py --list-models-db | grep clip

gets no hits whereas

imagenet-testbed$ python db.py --list-models-registry | grep clip

does.

Evidently, when the newer models were added to the testbed, the database dump link was not updated to stay current. Hopefully, there is a more recent dump lying around such that this problem can be easily fixed.

[rtaori]

Hi @jecummin ,

Thanks for catching this issue!
I re-uploaded the DB file, so everything should work now. If you want to go ahead and delete your current file: rm s3_cache/robustness_evaluation.db, then the next time you run any command it should download the current DB file! Please let me know if you have any further issues.