milesial/Pytorch-UNet

Predicted output is black.

Opened this issue · 5 comments

akshay9396 commented

I trained the model on Carvana dataset. Below is the screen shot of training. There i you can clearly see that validation Dice score is constant for 5 epochs and loss is "nan". I used one checkpoint.pth for predication and i got black output. could you please help me to resolved this issue.
image
Capture

Feng2100 commented

Have you solved it? I'm having the exact same problem

chenshans commented

I'm having the exact same problem，too

berda-ak commented

I'm having the exact same problem，too

CZG0712 commented

I'm having the exact same problem，too

taipain commented

I trained the model on Carvana dataset. Below is the screen shot of training. There i you can clearly see that validation Dice score is constant for 5 epochs and loss is "nan". I used one checkpoint.pth for predication and i got black output. could you please help me to resolved this issue. image Capture

Probably the reason is that torch.float16 is used in autocast() when using AMP.
So, change torch.float16 to torch.bfloat16.
But, if you wanna use torch.float16, you add grad_scaler in init_scale=4096．
I solved this issue.
I referred to the following sites．
This is my first time commenting on github, so sorry if there is something wrong and my poor English．
https://qiita.com/takeuchiseijin/items/909c48b57127a37fbd12
https://qiita.com/bowdbeg/items/71c62cf8ef891d164ecd

  • 👍1