inference speed and diversity

[kiaia]

Hi!
Thanks for your great work! I'm working on getting results on another paraphrase dataset under T5 + GP prior setting. I have the following two questions:

• I found that the generation speed is relatively slow due to the inference batch size 1, and something get wrong if I change it. Is there any way to speed up the generation?
• if I want to get a trade-off between quality and diversity, is it suitable to set the scalar to 7 just like it used in the paper for the paraphrasing task?

[wyu-du]

Hi,

Thanks for your interest in our work!

• If you want to increase the batch size during generation, you need to change the decode() function here to decode_batch() function (documentation here).
• Yes, you can freely tune the scalar until you find the best quality and diversity trade-off on the dev set. In our experiments, we find setting the scalar to 7 gives the best result.

[kiaia]

Hi.
Thanks for your detailed reply. I have sped up the generation process!