Unable to train

Question

Unable to train

lucasjinreal opened this issue 3 years ago · 14 comments

I got Nan loss:

04-08 10:51:43.489 NaN or Inf found in input tensor.
04-08 10:51:43.559 [epoch 28], [iter 8136 / 13576], [train main loss nan], [lr 0.001678]
04-08 10:51:43.560 NaN or Inf found in input tensor.
04-08 10:51:43.632 [epoch 28], [iter 8137 / 13576], [train main loss nan], [lr 0.001678]
04-08 10:51:43.632 NaN or Inf found in input tensor.
04-08 10:51:43.965 [epoch 28], [iter 8138 / 13576], [train main loss nan], [lr 0.001678]
04-08 10:51:43.965 NaN or Inf found in input tensor.
04-08 10:51:44.036 [epoch 28], [iter 8139 / 13576], [train main loss nan], [lr 0.001678]
04-08 10:51:44.037 NaN or Inf found in input tensor.

Answer 1 · 2021-04-08T08:16:38.000Z

I do not meet such error.
I try by myself. It seems normal in the first epoch.

Answer 2 · 2021-04-08T09:03:46.000Z

@lxtGH 1 epoch is converged. but from epoch 2, loss become bigger and bigger:

Answer 3 · 2021-04-10T07:32:39.000Z

What about using pt.1.4?

Answer 4 · 2021-04-11T02:14:40.000Z

@lxtGH Dude, to verify your code logic, I reinstall whole enviroment including pytorch 1.4. Same behavior.
You can test it on pytorch 1.8 .... All people from mmseg can not reproduce your result...

Answer 5 · 2021-04-11T02:45:26.000Z

@jinfagang What is your training setting using pt1.4 ? For mmseg, I will double check its details.

Answer 6 · 2021-04-11T02:51:53.000Z

Also. what dataset you use ? Cityscapes or your custom datasets?

Answer 7 · 2021-04-11T03:48:03.000Z

@lxtGH I am training on custom datasets. I am just following exactly same steps like cityspaces. However I think the biggest issue is that I also tried cityscapes same problem.

Answer 8 · 2021-04-11T05:07:59.000Z

In summary, using pt.1.4 on cityscapes still have this problem?

Answer 9 · 2021-04-11T15:33:11.000Z

@lxtGH Yes

Answer 10 · 2021-04-11T15:34:13.000Z

@lxtGH Besides. I double checked pytorch 1.7. You can try pytorch 1.7, this should not be pytorch version issue. there are some place went wrong. Try debug with pytorch 1.7 to see what caused the issue.

Answer 11 · 2021-04-12T03:02:19.000Z

@jinfagang OK I will re-train the model using pt1.4. I will also train the code using pt.1.7 using our lab server but the time maybe longer. Keep tune it.

Answer 12 · 2021-04-12T03:10:25.000Z

@jinfagang Another thing I want to ask, which config you use for training cityscape ? Could you share the detailed training setting of cityscapes for reference.

Answer 13 · 2021-04-13T03:08:27.000Z

@lxtGH I didn't made any modifications. I just using the sfresnet18 model without dsn since dsn has bug.

Answer 14 · 2021-04-17T06:50:33.000Z

Hi! @jinfagang I did not meet any error,. The results are normal using pt.1.6 and pt.1.4