Losses turned into NaN on ADE20K dataset
TrickyGo opened this issue · 3 comments
Hello there, I'm trying to train the model on ADE20K dataset, but after a few epoches of training, the losses turned into NaN and D_real turned into 0.000(generated blank images). Would you please tell me what could be the problem? Thanks a lot!
Here's how I trained:
python train.py --name ADE --load_size 256 --crop_size 256 --dataset_mode custom --label_dir /home/ADEChallengeData2016/annotations/training --image_dir /home/ADEChallengeData2016/images/training --label_nc 151 --no_instance --batchSize 2 --gpu_ids 7
I am not sure how you implemented the model on ADE20K dataset. You may need to use torch.nn.ModuleList to manage your per-region Conv. For simplicity, you can try to relax the constraint of per-region Conv and replace it with general 3*3 Convs.
Will try. Thanks for your advice!
Will try. Thanks for your advice!
Hi, have you finished the training on ADE20K? what's the performance, can you share the trained model?