What loss values are reasonable?
grey-area opened this issue · 3 comments
What train/test values are reasonable after 1 or 2 days of training?
For samples which I have uploaded to generated_samples folder I've stopped training after ~35 hours. This is how tensorboard looked like.
I'm struggling a bit to correlate your loss values with the results I'm getting (although I'm using a heavily altered implementation).
I just want to make sure that I'm not mistaken due to some naming changes in your code base, but the total_losses (both train and test) are the L1Loss between eps and eps_hat?
In my experiments audio starts to get good when this loss drops to around 0.03 (albeit on a different batch size, seq len etc.)
I'm also curious as to why test/total_loss is 4 times the train loss? that seems a bit odd.
Thanks for the help and a great repo!
@ohadvb firstly, picture above is irrelevant since a lot of updates have been done to the repo. total_loss is L1-loss between eps and eps_hat. Currently, from my tensorboard I see that train loss is twice less than test loss. Differences between them might appear because for test I set number of iterations to be 50 instead of 1000. Moreover, please, take into account that instead of evaluating a small sequence of audio (during train I cut random audio slice to increase mini-batch size), I calculate test loss between whole test audios, thus it implies test loss to be higher, obviously.