The training process is stopping after loading weights and doesn't go on
sms95 opened this issue · 1 comments
sms95 commented
The terminal shows the number of iteration and doesn't go on .
This program doesn't stand the cuda memory.
wzt9332 commented
@sms95 Did you train the model? There was a gradient explosion in my training. Training data is cowc.
Region Avg IOU: nan, Class: nan, Obj: -nan, No Obj: -nan, Avg Recall: 0.000000, count: 18
Region Avg IOU: nan, Class: nan, Obj: -nan, No Obj: -nan, Avg Recall: 0.000000, count: 12
Region Avg IOU: nan, Class: nan, Obj: -nan, No Obj: -nan, Avg Recall: 0.000000, count: 18
Do you know the solution?