rpryzant/delete_retrieve_generate

Meaning of WPS

lzzhaha opened this issue · 4 comments

lzzhaha commented

Hi,

In your code, a variable called 'WPS' is recorded in every logging step.

By looking at the source code, it seems to me that this variable represents number of examples being processed by the model in every logging step.

But I don't understand why you want to log this variable and why its value would decrease as the training proceeds.

Could you give some elaboration about this?

Thanks.

rpryzant commented

Thanks for reaching out! WPS stands for "words per second" but in practice corresponds to examples per second (sorry for the confusion! Cleaned things up in this commit: 6f9069e).

It's a measure of system speed (how many example are being processed per second).

lzzhaha commented

Thanks for your explanation.

Do you have any idea why this value would decrease and finally become stable as training proceeds?

Thanks.

rpryzant commented

I don't, sorry. It depends on your operating system and hardware.

My guess would be that the initial variance is due to cache warmup and process scheduling/priority.

But the good news changes in training speed will not affect the system's behavior :)

lzzhaha commented

Got it. Thank you very much.