regarding lbfgs
homelifes opened this issue · 3 comments
Hi @ae-foster. Thanks for your work. In simclr.py, you already do linear evaluation after training for some epochs using a constant reg_weight=1e-5:
clf = train_clf(X, y, net.representation_dim, num_classes, device, reg_weight=1e-5)
In that case, is there a big difference between the results obtained here (constant reg_weight=1e-5) and the results obtained in lbfgs_linear_clf.py? May you please tell me the score you got on CIFAR10 with constant reg_weight=1e-5?
Hi @homelifes thanks for your question. The reg_weight=1e-5
is close to the optimal reg weight, however, you can always use the lbfgs_linear_clf.py
script to scan across different reg weights. From my experiments on CIFAR, carefully optimizing the reg weight can lead to an improvement <<1%
accuracy, so not that much. This could be more important if you totally change the model in which case the representation scaling could be different
I see. Thanks a lot for letting me know that. And great work! Finally an implementation which i was looking for: (1) clean and (2) reproduces the results correctly. All other implementations lack one of those :(
Thanks @homelifes ! Please help get the word out about this repo :)