/cudnn_rnn_theano_benchmarks

Primary LanguagePython

cudnn-rnn-benchmarks

All benchmarks are reported for a host with the following specifications :

* NVIDIA GeForce GTX TITAN X GPU

* Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz

* CUDA 8.0, cudnnv5105

These benchmarks are aimed at understanding the performance gains with using the cuDNN RNN implementation (https://devblogs.nvidia.com/parallelforall/optimizing-recurrent-neural-networks-cudnn-5/) in theano.

The benchmarks are evaluated similar to https://github.com/glample/rnn-benchmarks that compares RNN implementations in different deep learning frameworks. Results will be integrated into the above repository eventually.

Note: Results on regular RNNs cannot be compared as is between the two repositories as this benchmark uses the new theano GPU backend libgpuarray https://github.com/Theano/libgpuarray and different hardware specifications.

The Recurrent Networks take as input a 3D Tensor batch_size x seq_length x hidden_size and output all hidden states, compute a MSE loss at each step and compute the gradients of error with respect to each parameter. The hidden_size specifies the size of the output and input layer of the networks.

The code of the scripts we ran are available. The code for the regular theano RNN implementations were borrowed from the rnn-benchmarks repository.

The reported Train time is the average time needed to run (forward, backward) for a single training example, the smaller the better.

A more exhaustive grid search will be done soon.

Note: The compile times, although not reported are much faster for the cuDNN implementation.

LSTM - cuDNN LSTM and GRU vs FastLSTM in rnn.py

This LSTM implementation used for these benchmarks does not use peephole connections between cell and gates.

Depth 1

Batch Size 32 x Seq Len 30

Hidden Size 128

Version Train (µs) Forward only (µs)
Theano LSTM 204.5 57.1
cuDNN Theano LSTM 118.8 59.5
cuDNN Theano GRU 117.4 57.6

Hidden Size 512

Version Train (µs) Forward only (µs)
Theano LSTM 530.9 148.1
cuDNN Theano LSTM 223.2 102.4
cuDNN Theano GRU 184.6 77.6

Hidden Size 1024

Version Train (µs) Forward only (µs)
Theano LSTM 1102.0 294.0
cuDNN Theano LSTM 601.8 161.1
cuDNN Theano GRU 394.8 136.2

Batch Size 128 x Seq Len 30

Hidden Size 128

Version Train (µs) Forward only (µs)
Theano LSTM 200.8 52.8
cuDNN Theano LSTM 33.4 15.0
cuDNN Theano GRU 32.2 14.4

Hidden Size 512

Version Train (µs) Forward only (µs)
Theano LSTM 491.0 138.2
cuDNN Theano LSTM 100.8 31.7
cuDNN Theano GRU 83.3 26.5

Hidden Size 1024

Version Train (µs) Forward only (µs)
Theano LSTM 1000.1 291.8
cuDNN Theano LSTM 221.2 69.0
cuDNN Theano GRU 181.3 59.1

Depth 3

Batch Size 128 x Seq Len 30

Hidden Size 512

Version Train (µs) Forward only (µs)
Theano LSTM 778.3 418.3
cuDNN Theano LSTM 244.9 70.2
cuDNN Theano GRU 197.1 55.7

Hidden Size 1024

Version Train (µs) Forward only (µs)
Theano LSTM 1592.8 882.7
cuDNN Theano LSTM 820.6 256.8
cuDNN Theano GRU 639.5 195.2

Batch Size 128 x Seq Len 200

Hidden Size 512

Version Train (µs) Forward only (µs)
Theano LSTM 2196.6 1168.1
cuDNN Theano LSTM 1539.5 485.9
cuDNN Theano GRU 1253.8 386.4

Hidden Size 1024

Version Train (µs) Forward only (µs)
Theano LSTM 5711.1 3427.9
cuDNN Theano LSTM 5342.5 1692.1
cuDNN Theano GRU 4163.4 1274.5