Chainer-based implementation of Entropy-Adam https://arxiv.org/abs/1611.01838
Evaluation on Chainer's examples/mnist/train_mnist.py.
Adam with default parameters:
epoch main/loss validation/main/loss main/accuracy validation/main/accuracy elapsed_time
1 0.192966 0.105841 0.941117 0.9661 9.41056
2 0.0748482 0.0808591 0.976499 0.9752 15.5148
3 0.0451277 0.0630382 0.985082 0.9801 21.5921
4 0.0368719 0.065199 0.988048 0.9808 27.7178
5 0.0268794 0.0830525 0.991265 0.9751 33.7963
6 0.0245737 0.0639368 0.991848 0.9811 39.9647
7 0.0204058 0.0763049 0.993348 0.9826 46.1338
8 0.0187804 0.0894077 0.993765 0.9781 52.5168
9 0.0144037 0.0961628 0.995549 0.9814 59.1598
10 0.0153753 0.0868186 0.995215 0.9794 66.281
11 0.0152795 0.093921 0.994949 0.9805 73.9723
12 0.010641 0.0901965 0.996649 0.982 82.661
13 0.0122108 0.087358 0.996215 0.9806 91.7917
14 0.0118934 0.0842234 0.996415 0.9845 101.728
15 0.0110066 0.112114 0.996416 0.9818 112.785
16 0.00892282 0.102957 0.997416 0.9817 124.153
17 0.0120498 0.0935951 0.996582 0.984 135.8
18 0.00869097 0.1243 0.997316 0.9796 148.009
19 0.009857 0.107859 0.997233 0.9834 160.997
20 0.00926358 0.110751 0.997132 0.983 173.989
Entropy-Adam with default parameters:
epoch main/loss validation/main/loss main/accuracy validation/main/accuracy elapsed_time
1 0.274215 0.115892 0.924217 0.9644 7.66172
2 0.0914476 0.104436 0.972383 0.9677 14.8432
3 0.0583779 0.0817859 0.982215 0.9754 21.8186
4 0.0399681 0.0661899 0.987698 0.98 28.9435
5 0.0285334 0.0646427 0.991015 0.9799 35.9796
6 0.0211337 0.0643336 0.993482 0.9801 43.052
7 0.0155652 0.0674905 0.995382 0.9802 50.072
8 0.0123355 0.0834323 0.996215 0.9793 57.1658
9 0.0129329 0.0748796 0.995949 0.9816 64.2196
10 0.0101271 0.0816437 0.996732 0.9797 71.3355
11 0.0112042 0.0877273 0.996366 0.9792 78.9264
12 0.00777579 0.0995326 0.997399 0.9753 86.4652
13 0.00567992 0.0767617 0.998333 0.9818 94.3747
14 0.00854904 0.0949963 0.997315 0.9785 102.63
15 0.00890467 0.0911491 0.996799 0.9789 111.19
16 0.00593435 0.0767955 0.998199 0.9831 119.9
17 0.0037807 0.0798007 0.998883 0.9828 128.76
18 0.006826 0.100073 0.997799 0.9768 139.226
19 0.00419141 0.0863279 0.9988 0.9812 151.545
20 0.00405932 0.089084 0.998733 0.9815 161.919