facebookresearch/swav

loss behaviour

Opened this issue · 4 comments

Thank you for the very nice work. I have a small doubt though. I am trying to train the model on a custom dataset and the loss decreases initially from ~8 to ~5 at the end of 60 epochs but then, it begins to increase again. I was wondering if someone else has faced a similar issue.

Thank you

Same issue @chinmay5 on my custom dataset. Have you addressed the problem of loss increasing?
My loss decreased at the first some steps and increased later.

Hi, @ha-lins I managed to do it by playing around with the crop-size and batch-size hyper-parameters. In short, it was more of exploring the hyper-parameters. Still, there was a specific scenario in which I could not manage to handle the issue. It may so be that your data clusters collapse (I got nan loss in that case).

I got NAN loss after the 250 iterations on my dataset. Can you share with me how you tackle it, please?

GSusan commented