What learning rate/scheduler/optimizer are suitable for training mlp-mixer?

[QiushiYang]

Thanks for your codes!

I find it is very important to set suitable lr/scheduler/optimizer for training res-mlp models. In my experiments with a small dataset, the classification performance is very poor when I train models with lr=1e-3 or 1e-4, weight-decay=05e-4, scheduler=WarmupCosineLrScheduler, optim='sgd'. The results increase remarkably when lr=5e-3, weight-decay=0.2, scheduler=WarmupCosineLrScheduler, optim='lamb'.

While the results are still much lower than CNN models with comparable params. trained from scratch. Could you provide any suggestions for training res-mlp?