/torch7-sdprop

Torch7 implementation of SDProp.

Primary LanguageLuaOtherNOASSERTION

Torch7 implementation of SDProp

This is a Torch7 implementation of SDProp described in the paper of Adaptive Learning Rate via Covariance Matrix Based Preconditioning for Deep Neural Networks(Y. Ida et al., IJCAI, 2017).

SDProp is an adaptive learning rate algorithm that effectively trains deep neural networks.

Usage

Import:

require 'sdprop.lua'

You can use SDProp as a part of optim package as follows:

sdpropState = sdpropState or {}
optim.sdprop(feval, parameters, sdpropState)

You can also set hyper-parameters as follows:

sdpropState = sdpropState or {
  learningRate = 0.0001,
  gamma = 0.999
}
optim.sdprop(feval, parameters, sdpropState)

Demos

You can try demos in folders of demos-mnist and demos-cifar.

These demos are created based on Demos and Tutorials for Torch7.

Usage in demos-mnist:

th train-on-mnist.lua -optimization SDProp

Results:

  • training accuracies

mnist_train

  • test accuracies

mnist_test

Usage in demos-cifar:

th train-on-cifar.lua -optimization SDProp

Results:

  • training accuracies

cifar_train

  • test accuracies

cifar_test

Citation

@inproceedings{ijcai2017-267,
  author    = {Yasutoshi Ida, Yasuhiro Fujiwara, Sotetsu Iwamura},
  title     = {Adaptive Learning Rate via Covariance Matrix Based Preconditioning for Deep Neural Networks},
  booktitle = {Proceedings of the Twenty-Sixth International Joint Conference on
               Artificial Intelligence, {IJCAI-17}},
  pages     = {1923--1929},
  year      = {2017},
  doi       = {10.24963/ijcai.2017/267},
  url       = {https://doi.org/10.24963/ijcai.2017/267},
}