meets problem when train ssp head
Opened this issue · 2 comments
Firstly I trained a teacher model and its accuracy is correct, then I train ssp head of teacher model, but I found loss of ssp head falling slowly. the initial loss is 3.37 at the start of training, and falls to 3.25 at the end of training. Did I make something wrong? dataset is ImageNet, and training config is consistent to that in student.py.
maybe the code of train ssp head is wrong? you can read my issue.
The training hyper-parameters (e.g. batchsize, epoch, LR) of CIFAR and ImageNet are different. For ImageNet, we use the hyper-parameters in pytorch/example.
Besides the hyper-parameters, the reason that ssp loss does not fall may be that the backbone of teacher is fixed. The trainable module contains only a 2-layer FC. As stated in the paper, the self-supervision may be not accurate, but it still transfer some structured information. So maybe you could try continuing the experiments and see the results.