Optimizer-PyTorch
Package of Optimizer implemented with PyTorch .
Optimizer Lists
SGD: stochastic gradient descent
Adam: A Method for Stochastic Optimization
- https://arxiv.org/abs/1412.6980
- https://openreview.net/forum?id=ryQu7f-RZ
- https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py
Adabound: Adaptive Gradient Methods with Dynamic Bound of Learning Rate
- https://arxiv.org/abs/1902.09843
- https://openreview.net/forum?id=Bkg3g2R9FX
- https://github.com/Luolc/AdaBound
RAdam: On the Variance of the Adaptive Learning Rate and Beyond
Lookahead: Lookahead Optimizer: k steps forward, 1 step back
Optimistic
OptimAdam
OMD
ExtraGradient
STORM: STOchastic Recursive Momentum
- < Momentum-Based Variance Reduction in Non-Convex SGD >
- http://papers.nips.cc/paper/9659-momentum-based-variance-reduction-in-non-convex-sgd
- https://github.com/google-research/google-research/blob/master/storm_optimizer/storm_optimizer.py (TensorFlow)