yalesong/pvse

word embedding

cuishuhao opened this issue · 1 comments

cuishuhao commented

Thanks for your job!
I try to reproduce your job and I get confused in the word embedding.
I use torchtext 0.4.0 and find 70 words missing in GloVe in Coco.
I got very bad results after training 30 epoch with the same parameters.
Though I use torch 1.2.0 and torchvision 0.4.0, I don't think they are the main reasons.
I want to know how you initialize the missing word embeddings or other problems you think I am facing.
Any help would be appreciated. Thank you!

yalesong commented

Obtaining good performance requires finetuning image and text encoders (by providing --img_finetune' and --txt_finetune) after training the other parts of the model. The missing word embeddings are initialized randomly and fixed throughout training unless --txt_finetune is turned on.