The speed of training gradually decreases when using gpu
zhaoqxu-eth opened this issue · 4 comments
Hi, I'd like to know what I could do to prevent the decreasing of speed (samples/s). When I start training it, the first second it's over 6 samples/s, and it shows (00:00<7:57), but after a few minutes, it gradually decreases to under 2 samples/s, which made the time longer(09:27<10:19). What made this problem and how could I fix it? Thanks!
That’s normal with sortagrad turned on. Is sortagrad turned on?
Sortagrad is a curriculum learning method that feeds the model the sorted dataset (sorted from shortest to longest, shortest first) during training.
hi , I'm just running the example speech.yml with BaiduSpeech model. I find sortagrad is duration, what should I do to turn it off? Is that important? thanks!
I tried deleting it directly, which makes training speed too slow
That's normal because now you are training with the average utterance length all the time versus the shortest of the files with sortagrad.