Question about pre-training process

Question

Question about pre-training process

natedingyifeng opened this issue 2 years ago · 1 comments

Hi, I am curious about what validation task you are using during the pre-training process. Could you please share some information about this issue?

Answer 1 · 2022-08-10T09:10:54.000Z

Hi, for each stage (either MSP or NTP task) of pretraining, we employ a small proprotion of training data as the held-out validation set and monitor the corresponding loss (either MSP or NTP loss) on this subset. We stop the pretraining when it converges (or in other words, the validation loss does not decrease).