CLIP models

[penfever]

Hello! I'm having a bit of trouble using the CLIP models in this testbed.

When I run --

python eval.py --gpus 0 --models resnet50_clip_zeroshot --eval-settings val imagenetv2-matched-frequency

or

python eval.py --gpus 0 --models vit_b_32_clip_zeroshot --eval-settings val imagenetv2-matched-frequency

I get the following assert error --

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/benfeuer/miniconda3/envs/robustness/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "../src/inference.py", line 64, in main_worker
    model = py_model.generate_classifier(py_eval_setting)
  File "../src/models/model_base.py", line 76, in generate_classifier
    self.classifier = self.classifier_loader()
  File "../src/models/clip.py", line 260, in classifier_loader
    load_model_state_dict(model, name)
  File "../src/mldb/utils.py", line 93, in load_model_state_dict
    bio = load_model_checkpoint_bytes(name)
  File "../src/mldb/utils.py", line 80, in load_model_checkpoint_bytes
    r_model = m_repo.get_model(name=model_name)
  File "../src/mldb/model_repository.py", line 351, in get_model
    return self.run_get(get_fn, session=session, assert_exists=assert_exists)
  File "../src/mldb/model_repository.py", line 329, in run_get
    return self.run_query_with_optional_session(query, session)
  File "../src/mldb/model_repository.py", line 315, in run_query_with_optional_session
    return query(sess)
  File "../src/mldb/model_repository.py", line 324, in query
    assert len(result) == 1

If I comment out the assert, I get this error --

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/benfeuer/miniconda3/envs/robustness/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "../src/inference.py", line 64, in main_worker
    model = py_model.generate_classifier(py_eval_setting)
  File "../src/models/model_base.py", line 76, in generate_classifier
    self.classifier = self.classifier_loader()
  File "../src/models/clip.py", line 260, in classifier_loader
    load_model_state_dict(model, name)
  File "../src/mldb/utils.py", line 93, in load_model_state_dict
    bio = load_model_checkpoint_bytes(name)
  File "../src/mldb/utils.py", line 81, in load_model_checkpoint_bytes
    data = m_repo.get_checkpoint_data(r_model.final_checkpoint_uuid)
AttributeError: 'NoneType' object has no attribute 'final_checkpoint_uuid'

Advice is appreciated!

[jecummin]

Getting the same error.

[rtaori]

Hi @penfever ,

Sorry for the delay in getting back to you! (Somehow I wasn't getting github notifications).

I re-uploaded the DB file, so everything should work now. If you want to go ahead and delete your current file: rm s3_cache/robustness_evaluation.db, then the next time you run any command it should download the current DB file! Please let me know if you have any further issues.