facebookresearch/swav

Question about loss and hyperparameters

TopTea1 opened this issue · 4 comments

TopTea1 commented

Hi, first, thanks for sharing your great work !

I'm trying to train my network using swav in 2x160 + 4x96 setting, I'm using the hyperparameters provided in the bs_256 script. The loss start to decrease, and seems to be stuck after 3-4epochs did I need to adapt the hyperparameters ? Or made some other changes ?

Thanks for your helps

Erfun76 commented

Hi, first, thanks for sharing your great work !

I'm trying to train my network using swav in 2x160 + 4x96 setting, I'm using the hyperparameters provided in the bs_256 script. The loss start to decrease, and seems to be stuck after 3-4epochs did I need to adapt the hyperparameters ? Or made some other changes ?

Thanks for your helps

Reduce queue length. I encountered this issue in some of my training and using a smaller queue solved the problem.

  • ❤️1
xcvil commented

I had the same issue. During my warm up training, the loss was not reduced anymore and kept the same value.

mathildecaron31 commented

Hi @TopTea1
Thanks for your interest and your kind words.
As suggested by @Erfun76 I would suggest to reduce the queue length or starting the queue later on in training (i.e. --epoch_queue_starts 50 for example). Also feel free to take a look at this section for tips on how to get the model training https://github.com/facebookresearch/swav#common-issues

ibro45 commented

If I understood correctly, you still keep the same number of prototypes (3000) and batch size (64), while reducing the queue size from 3840 to, e.g., 50? How does that influence the equipartition constraint?