delete_retrieve taking long time + the BLEU scores different from the paper
naveen-kinnal opened this issue · 4 comments
Hello. I am trying to run the 'delete_retrieve' model for the yelp dataset with the same config given in the code. I just set the
"model_type": "delete_retrieve"
However, I see that the model is taking infinite time for just the first iteration
2021-05-04 22:12:07,463 - INFO - MODEL HAS 10364933 params
2021-05-04 22:12:13,904 - INFO - EPOCH: 0 ITER: 0.0/692.2578125 EPS: 8830.61 LOSS: 9.1699 METRIC: 0.0000
and it's freezing there. Is there any other parameter in the config or any other file that I need to tweak to run the 'delete_retrieve' efficiently. Also, do we need 70 epochs here or can be lesser?
Thanks for reaching out. Hmm...are you running this on a GPU? The delete_retrieve model is indeed slow, but it shouldn't hang forever. Each epoch should take a few hours on a gpu.
Well thank you @rpryzant. As you mentioned, the delete_retrieve mode is just slow but is working fine.
May I know the BLEU score you obtained for YELP dataset using DeleteOnly and DeleteAndRetrieve?
For the DeleteOnly model with YELP,
- I was getting a score of about 16.46 (using the reference test set in both validation and testing)
- I was getting a score of about 2.16 (using the different validation and test references given in https://github.com/lijuncen/Sentiment-and-Style-Transfer/tree/master/data/yelp)
However, I see in the paper the score for DeleteOnly is 7.5. Could you please let me know which one seems more appropriate here?
And also, is there BLEU score in your code a normalized one? You seem to do in just one direction neg=> pos (src : ref.0 is negative and tgt : ref.1 is positive), right ?
Hello, I've added a FAQ section to the README which addresses this. Let me know if you have any other questions!
Thank you @rpryzant :)