regarding lbfgs

[homelifes]

Hi @ae-foster. Thanks for your work. In simclr.py, you already do linear evaluation after training for some epochs using a constant reg_weight=1e-5:
clf = train_clf(X, y, net.representation_dim, num_classes, device, reg_weight=1e-5)

In that case, is there a big difference between the results obtained here (constant reg_weight=1e-5) and the results obtained in lbfgs_linear_clf.py? May you please tell me the score you got on CIFAR10 with constant reg_weight=1e-5?

[ae-foster]

Hi @homelifes thanks for your question. The reg_weight=1e-5 is close to the optimal reg weight, however, you can always use the lbfgs_linear_clf.py script to scan across different reg weights. From my experiments on CIFAR, carefully optimizing the reg weight can lead to an improvement <<1% accuracy, so not that much. This could be more important if you totally change the model in which case the representation scaling could be different

[homelifes]

I see. Thanks a lot for letting me know that. And great work! Finally an implementation which i was looking for: (1) clean and (2) reproduces the results correctly. All other implementations lack one of those :(

[ae-foster]

Thanks @homelifes ! Please help get the word out about this repo :)