Loss became infinite or NaN at iteration=xxxx!

[jasoncho608]

File "xx/lvc/detectron2/engine/train_loop.py", line 254, in _detect_anomaly
raise FloatingPointError(
FloatingPointError: Loss became infinite or NaN at iteration=1029!
loss_dict = {'loss_box_reg_stage0': tensor(nan, device='cuda:0', grad_fn=), 'loss_box_reg_stage1': tensor(nan, device='cuda:0', grad_fn=), 'loss_box_reg_stage2': tensor(nan, device='cuda:0', grad_fn=), 'data_time': 0.007915766909718513}

I used my own datasets to train the correct model.And this problem confused me so much.Looking forward to your reply！