pretrained weights are probably incorrect

Question

pretrained weights are probably incorrect

BestSonny opened this issue 3 years ago · 7 comments

The pretrained weights seem to be wrong.

For example, the vit_base has a dimension of 1024.

Could you upload the correct version? Thanks

Answer 1 · 2022-05-23T16:06:59.000Z

I have also some issues regarding the pre-trained checkpoints. The checkpoints only include the keys "target_encoder" and "prototypes". If I want to load the checkpoint via the training script, I get errors because the keys "epoch" and "encoder" are missing.

Answer 2 · 2022-06-07T18:55:32.000Z

Hi @BestSonny, There are 1024 prototypes used in the loss, but I just checked the ViT-B/16 and ViT-B/4 pre-trained weights, and they both have the correct output dimension of 768. Please let me know if you would like some more clarification or help loading the models!

Answer 3 · 2022-06-07T18:58:09.000Z

Hi @brewormly, yes the current checkpoints only include the "target_encoder" since those are the network used at the end of pre-training to obtain the results in the paper, but I would be happy to release the full checkpoints as well in case you find these useful! Will ping you once these are online!

Answer 4 · 2022-08-25T05:30:26.000Z

@MidoAssran possible to release the ImageNet-1k specific checkpoints (fine-tuned and / or linear-eval'd)?

By "linear-eval'd" I mean keeping the target encoder frozen and just training a linear layer on top of it. So, essentially, the target encoder params (which are already released) and the linear layer params.

Answer 5 · 2022-08-25T05:51:45.000Z

Also, the target_encoder key in the released weights -- seems like it contains two things - the actual encoder plus the projection head (module.fc* params). Is the projection head needed for downstream tasks?

@MidoAssran

Answer 6 · 2023-02-01T00:33:12.000Z

I have also some issues regarding the pre-trained checkpoints. The checkpoints only include the keys "target_encoder" and "prototypes". If I want to load the checkpoint via the training script, I get errors because the keys "epoch" and "encoder" are missing.

I have the same issue !

Answer 7 · 2023-07-17T14:58:11.000Z

Hi @brewormly, yes the current checkpoints only include the "target_encoder" since those are the network used at the end of pre-training to obtain the results in the paper, but I would be happy to release the full checkpoints as well in case you find these useful! Will ping you once these are online!

Sorry for the late reply after one year. I wonder if there is still a plan to release the full checkpoints? I think they will be very helpful in continuing the training for other tasks.