xuxw98/ESAM

inst_loss: nan

Closed this issue · 1 comments

Hello author, when I run this command, CUDA_VISIBLE_DEVICES=0 python tools/train.py configs/ESAM_CA/ESAM_sv_scannet200_CA.py --work-dir work_dirs/ESAM_sv_scannet200_CA/,naN appears in the loss obtained by training. 
After debugging, I found that in this command, mask_bce_losses.append(F.binary_cross_entropy_with_logits(
            pred_mask, tgt_mask.float())),pred_mask and tgt_mask.float() sometimes become tenser[], which leads to loss is naN. 
Do you konw how to solve this problem?

2024-09-27 14-17-09 的屏幕截图
2024-09-27 14-36-03 的屏幕截图

That's ok. This nan will not affect the training.