Question on reproduction
Closed this issue · 15 comments
Thank you for sharing your work. I cannot reproduce the results of the paper in practice. May I ask how many epochs were trained during training and how much the final loss converged?
Do you use the 6-reference test set? 4 epochs during training and nll_loss on validation set is about 1.62.
I follow the default setting for training the model.
This is the training process.
This is the evaluation. The model output the bad response.
I think the model is trained enough by myself, maybe there is something to pay attention to when testing? My beam search is set to 5. I am so confused about the output when testing.
when you are training and testing, what feature do you use?
Just your default setting('vggish', 'i3d_flow', 'i3d_rgb')
how about text features? do you use the caption, summary, and dialogue?
I haven't modified the code yet, just use your original settings, so the text features should be used according to your original code.
Sorry, I check the code and find a bug in VideoGPT2.py. I think I uploaded the wrong version before. Now it will be the right version and you can try again. Hope you can get the right results. If there are still any problems, please don't hesitate to ask me.
Thank you for your patient reply, I will try again.
When testing, you can use the 6-ref test set to compute the metrics.
Hello, I modified the vidioGPT.py file as you did, but after the modification, after one epoch during my training, the loss no longer drops. The model cannot converge completely. All outputs are tag when testing.
Sorry about that, I will rerun it at once. I will check it again. Because of something fusion with other projects, the version may be in disorder. I will inform you at once when I find out why.
Ok, Thanks.
I fixed the bug. You can try again. It should be ok. Be sure that using the default settings, including batch_size, learning rate, etc. The nll loss can be about 1.62.
Thanks, I can get a reasonable result.