The accuracy of the masked LM and NSP tasks
arm-on opened this issue · 0 comments
arm-on commented
Hi
I've been curious about the pretraining part. More specifically, I saw that in your paper, you did not report the accuracy of the masked language modeling and the next sentence prediction tasks. Would you please provide some details on these parts?
For instance, what was the accuracy for these tasks (MLM & NSP) in the pretraining? Did you use the whole corpus (Miras+Wiki+etc.) for these parts too?