encoder checkpoint?

[kabouzeid]

Hi, thanks for your work on this!

Why is there an encoder checkpoint (I can't find this mentioned in the paper)? Is it possible to train from scratch with this codebase?

https://github.com/AryanShekarlaban/data2vec-pytorch/blob/640fb8531be9deb5f8f0653802e272a4d39f39db/vision/configs/beit-pretraining.yaml#L4

[arxyzan]

Hi Karim, Thanks for your recent contribution to this repo.
You know, the encoder model is the backbone of data2vec which can be any model like BEiT.
The reason I put encoder checkpoint there is because the encoder model must be loaded from huggingface. I wrote it in a way that the checkpoint is used to load the model config so that AutoModel knows what model is being used. If you look into the encoder.py file when the encoder is initialized in __init__, the encoder gets created automatically from that checkpoint. That doesn't mean we're using any weights from the checkpoint so we're actually training from scratch.
Here is the code that uses it: modality/encoder.py

 def __init__(self, cfg, **kwargs):
        super(Encoder, self).__init__()
        self.cfg = cfg
        checkpoint = cfg.model.encoder_checkpoint
        model_config = AutoConfig.from_pretrained(checkpoint)  # only load the config from the checkpoint not the weights
        self.encoder = AutoModel.from_config(model_config)  # create the model from config

Maybe I should've used a better name for that, do you have any suggestions?

Best,
Aryan

[kabouzeid]

That's what I was looking for, thanks!