loss behaviour
Opened this issue · 4 comments
Thank you for the very nice work. I have a small doubt though. I am trying to train the model on a custom dataset and the loss decreases initially from ~8 to ~5 at the end of 60 epochs but then, it begins to increase again. I was wondering if someone else has faced a similar issue.
Thank you
Same issue @chinmay5 on my custom dataset. Have you addressed the problem of loss increasing?
My loss decreased at the first some steps and increased later.
Hi, @ha-lins I managed to do it by playing around with the crop-size and batch-size hyper-parameters. In short, it was more of exploring the hyper-parameters. Still, there was a specific scenario in which I could not manage to handle the issue. It may so be that your data clusters collapse (I got nan
loss in that case).
I got NAN loss after the 250 iterations on my dataset. Can you share with me how you tackle it, please?