BYOL and SimSiam Model Collapse Problem

[jqwangai]

In training BYOL or SimSiam, the model tends to fall into collapse. Have you ever encountered this problem and how did you solve it? Looking forward to your reply!

[Tian0426]

Hi @jqwangai , we also noticed that BYOL and SimSiam are sensitive to the choice of hyper-parameters and easy to collapse. For BYOL, we empirically found using a larger learning rate for the predictor effectively prevent collapse. You might want to consider using the settings in the script we provided in the README file, FYR:

python main.py --framework 'byol'    --backbone 'DCL' --dataset 'ucihar' --aug1 't_warp' --aug2 'negate' --n_epoch 60  --batch_size 128 --lr 5e-4 --lr_cls 0.3
python main.py --framework 'simsiam' --backbone 'DCL' --dataset 'ucihar' --aug1 't_warp' --aug2 'negate' --n_epoch 60  --batch_size 128 --lr 5e-4 --lr_cls 0.3

Hope this helps.