Nan Loss everytime i train
benc1221 opened this issue · 6 comments
Traceback (most recent call last):
File "C:\Users\bencj\Desktop\DDSP\DDSP-SVC-master\train.py", line 92, in
train(args, initial_global_step, model, optimizer, loss_func, loader_train, loader_valid)
File "C:\Users\bencj\Desktop\DDSP\DDSP-SVC-master\solver.py", line 100, in train
raise ValueError(' [x] nan loss ')
ValueError: [x] nan loss
do you know how to resolve this issue?
This does happen occasionally, try reducing lr.
I also fixed it by reducing batch file size even though I have a 4080 but for the next voice I'll reduce learning rate but will that make training go slower?
it is still throwing the same error even though i used low learning rate of 0.0001
Have you tried with default parameters
Hmmm I'm getting this error too. Very odd indeed. It seems to be getting stuck around 54000 steps and throws the error at a seemingly random step.
Ok so I don't know if it's going to mess things up, but I just removed the else raise error statements in the solver.py file and it seems to be doing fine so far.