The cross entropy loss in training stage
Ziyang412 opened this issue · 2 comments
Ziyang412 commented
Hi, thank you for the interesting work.
From the teaser image(https://github.com/kohjingyu/fromage/blob/main/teaser.png), I noticed two cross entropy loss, one in image caption and one in retrieval. But according to the total loss in the original paper, I only saw the CE loss in the image caption stage. So, does the second CE loss applied in the training stage or am I missing anything?
Thank you in advance!
kohjingyu commented
Thanks for pointing this out! In this case, the second CE loss is indeed also applied during training. This is to allow the model to learn the proper [RET] embeddings. The retrieval CE loss is applied at lines 453-454 here:
Lines 453 to 454 in 92c6d6f
Ziyang412 commented
Gotcha! Thanks!