Nan loss occurs while training

Question

Nan loss occurs while training

lyp0413 opened this issue 4 years ago · 2 comments

There is a little bug in src/losses/losses.py 52 & 53,which may cause a nan loss while training.
I think a epsilon like 1e-10 should be added for avoiding log(0).

Answer 1 · 2020-10-22T12:18:06.000Z

What epoch did you face with the NaN loss? I trained the model without NaN loss. Please make sure that you are using the latest code in the repo.

Answer 2 · 2020-10-22T22:48:06.000Z

Hi @lyp0413
I clamped the heatmaps' values with sigmoid activation before computing losses. You can refer to the implementation here