WARNING:tensorboardX.x2num:NaN or Inf found in input tensor.
Opened this issue · 1 comments
During the annual training process, a warning will be issued
$ python -m torch.distributed.launch --nproc_per_node=1 train.py --data /home/a/CustomMap --object yepian
Loading Model...
ready to train!
WARNING:tensorboardX.x2num:NaN or Inf found in input tensor.
Train Epoch: 1 [0/5000 (0%)] Loss: 0.035234406590462 Local Rank: 0
Train Epoch: 1 [1600/5000 (32%)] Loss: 0.004101661965251 Local Rank: 0
Train Epoch: 1 [3200/5000 (64%)] Loss: 0.002820491790771 Local Rank: 0
Train Epoch: 1 [4800/5000 (96%)] Loss: 0.002345604822040 Local Rank: 0
WARNING:tensorboardX.x2num:NaN or Inf found in input tensor.
Train Epoch: 2 [0/5000 (0%)] Loss: 0.003404101822525 Local Rank: 0
Train Epoch: 2 [1600/5000 (32%)] Loss: 0.003067462937906 Local Rank: 0
Train Epoch: 2 [3200/5000 (64%)] Loss: 0.003346179146320 Local Rank: 0
Train Epoch: 2 [4800/5000 (96%)] Loss: 0.003464318113402 Local Rank: 0
We had this warning forever and I never was able to figure out why. You can ignore it.