Can't load model when I train transformer after training pvqvae

[zhougeAI]

I use pvqvae_imagenet.yaml and transformer_imagenet.yaml to train on my own dataset. Training pvqvae is OK, but when I train transformer, I meet a model loading problem:
RuntimeError: Error(s) in loading state_dict for MaskedImageInpaintingTransformer:
Missing key(s) in state_dict: "pos_emb", "content_codec.encoder.pre_layers.0.weight", "content_codec.encoder.pre_layers.0.bias", "content_codec.encoder.res_layers.0.layers.1.weight", "content_codec.encoder.res_layers.0.layers.1.bias", ......
Unexpected key(s) in state_dict: "encoder.pre_layers.0.weight", "encoder.pre_layers.0.bias", "encoder.res_layers.0.layers.1.weight", "encoder.res_layers.0.layers.1.bias", "encoder.res_layers.0.layers.3.weight", "encoder.res_layers.0.layers.3.bias", "encoder.res_layers.1.layers.1.weight", "encoder.res_layers.1.layers.1.bias", "encoder.res_layers.1.layers.3.weight", .....
How to solve it?

[liuqk3]

Hi @zhougeAI ,

You should provid the path of pvqvae model weights in the configure file of transformer, then it will be automatically loaded. You can refer to this line for example:

PUT/configs/put_cvpr2022/ffhq/transformer_ffhq.yaml

Line 23 in 7de8ce0

ckpt_path: OUTPUT/pvqvae_ffhq/checkpoint/last.pth