loss is nan

Question

loss is nan

Closed this issue 6 months ago · 4 comments

After starting training, I found that many loss was nan. I don’t know if this was expected.

Answer 1 · 2024-01-05T14:36:38.000Z

No, there should not be any NaN losses, which losses are NaN?

Answer 2 · 2024-01-16T11:22:53.000Z

It was caused by the installation package version. The problem has been solved. However, there are two problems with the model: (1) There is a difference between the MAE of the model and the value in the paper (40 vs 30.8); (2) There is noise in the blank space. Is it because the parameters are wrong? The parameters are not modified and the data is downloaded directly.

Answer 3 · 2024-01-23T20:51:50.000Z

hi,

Thanks for your interest in our work. Did you set batch size to 4096 as reported in our paper? We re-factor the codebase for this release and are now investigating this issue, and will come back with the full script for reproduction along with the checkpoints.

Best,
Shengyu

Answer 4 · 2024-03-22T17:45:30.000Z

Hi,

We just uploaded the pretrained weights for our dataset and reproduction scripts, you can check it out