How to calculate entropy?
prixt opened this issue · 2 comments
prixt commented
Dear authors,
Thank you for your paper. It was quite illuminating.
Your proposed noise schedule requires the entropy value of each word/token before noising, but I couldn't find how you calculated it. Is it per sentence/ngram/corpus/etc. ? Any libraries you used to calculate it, or was it manual?
Thank you for your time.
Hzfinfdu commented
Hi,
Thank you for following our work!
We calculate the entropy of each token by the negative logarithm of its frequency in the training corpus. Hope this helps!
prixt commented
Thank you for the answer! Closing issue.