What is the purpose of --lr-hours config in LibriHeavy recipe?
tienanh28122000 opened this issue · 2 comments
Hi everyone, in the LibriHeavy recipe, there is a --lr-hours config (with the default value of 30000) instead of a --lr-epochs like other recipes (with the default value of 3.5). However, I found that both --lr-hours and --lr-epochs configs have the same purpose in the Eden scheduler. Therefore can anyone explain to me that what is the difference between the --lr-hours config and the --lr-epochs, and why the default value of --lr-hours is higher a lot when compared to the default value of the --lr-epochs? Thank you very much!
@pkufool @marcoyang1998 Could you have a look?
They are just different ways of specifying when the learning rate starts to decrease. Originally we specified it as epochs, but later switched to hours with the idea that for very large datasets, you might not do more than one or two epochs. If you initialize with lr_hours, when you do the step_epoch() call (that's the name IIRC), you have to actually supply a number of hours rather than epochs.