The single card training is normal, but there is nan with multiple cards

[mttsky]

hi, Adam W.
did you encounter single card training is normal, but there is nan with multiple cards

[aharley]

I have not encountered this! I in fact used multi-gpu training for the models I released. Can you share more about your setup, and your train.sh?

[mttsky]

Thank you for your reply

python train_nuscenes.py
--exp_name=${EXP_NAME}
--max_iters=25000
--log_freq=1000
--dset='trainval'
--batch_size=4
--grad_acc=5
--use_scheduler=True
--data_dir=$DATA_DIR
--log_dir='logs_nuscenes'
--ckpt_dir='checkpoints'
--res_scale=2
--ncams=6
--encoder_type='res50'
--do_rgbcompress=True
--device_ids=[0,1,2,3]

[aharley]

OK. So basically I don't know the answer here, but my strategy would be to try to simplify until it works.

For a start, how about: comment out the loss terms: https://github.com/aharley/simple_bev/blob/main/train_nuscenes.py#L203-L208

and just say total_loss = loss_fn(seg_bev_e, seg_bev_g, valid_bev_g)