Nan loss in training

Question

Nan loss in training

tranquangchung opened this issue a year ago · 5 comments

Hi
Thanks for sharing your project.
When I trained your model based on your config, however, the val and train loss was NAN.
I tried many times but the results are still the same.
Can you tell me the reasons and how to solve it?

Answer 1 · 2023-05-25T08:55:56.000Z

The problem made NAN is the Language model. So, I solved this problem by modifying your code, and it worked very well.

Answer 2 · 2023-07-16T01:32:03.000Z

Hi @tranquangchung , How did you solve the nan problem? Thank You!

Answer 3 · 2024-04-16T12:08:32.000Z

Hi, could you please explain how do you solve this problem? Thx!

Answer 4 · 2024-04-16T12:41:44.000Z

It turns out the problem is with google/flan-t5-large, this model does not support fp16 training, use fp32 it would be fine.

Answer 5 · 2024-04-16T12:43:33.000Z

Glad to know that it was solved. FYI we have released Tango 2: https://arxiv.org/abs/2404.09956