A Comparison Among Different Variants of Gradient Descent Algorithm

Introduction

This script implements and visualizes the performance the following algorithms, based on the MNIST hand-written digit recognition dataset:

Stochastic Gradient Descent(SGD)
Momentum
Nesterov Accelerated Gradient(NAG)
Adagrada
Adadelta
RMSprop
Adam

All the detail of the algorithms are described in the blog post An overview of gradient descent optimization algorithms by Sebastian Ruder

Dataset

The MNIST dataset contains 60,000 samples for training and 10,000 for validating. It is naturally divided into 10 classes corresponding to digit 0 to 9. The amount of samples for each class is well-balanced. Each digit image is pre-processed and rescaled into a 28*28 gray scale array, ranging from 0 to 255

Model

A traditional 3 layer neural network is adopted, with 28*28 input units, 25 hidden units and a softmax output layer

Performance

Here are the training and validating accuracy of each algorithm, with 30 epochs and 100 mini-batch:

SGD: 95.36% vs 94.06%
Momentum: 97.52% vs 94.88%
NAG: 97.47% vs 94.33%
Adagrad: 96.17% vs 93.95%
Adadelta: 94.65% vs 93.84%
RMSprop: 96.35% vs 94.51%
Adam: 96.54% vs 94.12%

Visualization

Here is the visualization of cost decreasing w.r.t each mini-batch within the first 10 epochs:

Conclusion

The variants of gradient descent algorithm can be roughly divided into 2 types: Momentum-like SGD and Adaptive learning rate SGD.

As mentioned in the blog post Adaptive learning rate SGD is suitable for large-scale sparse optimization problem (e.g, predict CTR). While in this case, data is not sparse, Momentum-like SGD significantly outperforms the others.