loss=nan (nan), lr=0.000460
912602337 opened this issue · 3 comments
loss=0.0136 (0.0136), lr=0.000460, local_step=1: : 0% 1/7272 [00:00<47:33, 2.55it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.000460, local_step=2: : 0% 2/7272 [00:00<45:48, 2.64it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.000460, local_step=3: : 0% 3/7272 [00:01<45:23, 2.67it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.000460, local_step=4: : 0% 4/7272 [00:01<44:59, 2.69it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.000460, local_step=5: : 0% 5/7272 [00:01<44:43, 2.71it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.000460, local_step=6: : 0% 6/7272 [00:02<44:30, 2.72it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.000460, local_step=7: : 0% 7/7272 [00:02<44:24, 2.73it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.000460, local_step=8: : 0% 8/7272 [00:02<44:24, 2.73it/s]NaN or Inf found in input tensor.
只有第一轮的第一次是正常的，而且已经很小了，其他全是nan；请问是什么问题
loss=0.0136 (0.0136), lr=0.000460, local_step=1: : 0% 1/7272 [00:00<47:33, 2.55it/s]NaN or Inf found in input tensor. loss=nan (nan), lr=0.000460, local_step=2: : 0% 2/7272 [00:00<45:48, 2.64it/s]NaN or Inf found in input tensor. loss=nan (nan), lr=0.000460, local_step=3: : 0% 3/7272 [00:01<45:23, 2.67it/s]NaN or Inf found in input tensor. loss=nan (nan), lr=0.000460, local_step=4: : 0% 4/7272 [00:01<44:59, 2.69it/s]NaN or Inf found in input tensor. loss=nan (nan), lr=0.000460, local_step=5: : 0% 5/7272 [00:01<44:43, 2.71it/s]NaN or Inf found in input tensor. loss=nan (nan), lr=0.000460, local_step=6: : 0% 6/7272 [00:02<44:30, 2.72it/s]NaN or Inf found in input tensor. loss=nan (nan), lr=0.000460, local_step=7: : 0% 7/7272 [00:02<44:24, 2.73it/s]NaN or Inf found in input tensor. loss=nan (nan), lr=0.000460, local_step=8: : 0% 8/7272 [00:02<44:24, 2.73it/s]NaN or Inf found in input tensor. 只有第一轮的第一次是正常的，而且已经很小了，其他全是nan；请问是什么问题
我也一样，请问你解决了吗