Nan appears during training
HospitableHost opened this issue · 1 comments
HospitableHost commented
I tried to train Pose-NDF, but after some epochs of training, the problem of Nan always appeared. I believe you should have encountered the same problem. Would you like to tell me how to solve this problem? Thank you very much!
The triangle in the figure represents nan.
garvita-tiwari commented
We figured that this occurs due to weight norm layer of pytorch: pytorch/pytorch#19126
Train without weight normalization