LRBench++ is a framework for effective learning rate benchmarking and tuning, which will help practitioners efficiently evaluate, select, and compose good learning rate policies for training DNNs.
The following figure shows the impacts of different learning rates. The FIX (black, k=0.025) reached the local optimum, while the NSTEP (red, k=0.05, γ=0.1, l=[150, 180]) converged to the global optimum. For TRIEXP (yellow, k0=0.05, k1=0.3, γ=0.1, l=100), even though it was the fastest, it failed to converge with high fluctuation.
See the people page for the full listing of contributors.