Note that the master branch of Caffe supports LSTM now. (Jeff Donahue's implementation has been merged.)
This repo is no longer maintained.
Speed comparison (Titan X, 3-layer LSTM with 2048 units)
Jeff's code is more modularized, whereas this code is optimized for LSTM.
This code computes gradient w.r.t. recurrent weights with a single matrix computation.
Batch size = 20, Length = 100
Code
Forward(ms)
Backward(ms)
Total (ms)
This code
248
291
539
Jeff's code
264
462
726
Batch size = 4, Length = 100
Code
Forward(ms)
Backward(ms)
Total (ms)
This code
131
118
249
Jeff's code
140
290
430
Batch size = 20, Length = 20
Code
Forward(ms)
Backward(ms)
Total (ms)
This code
49
59
108
Jeff's code
52
92
144
Batch size = 4, Length = 20
Code
Forward(ms)
Backward(ms)
Total (ms)
This code
29
26
55
Jeff's code
30
61
91
Example
An example code is in /examples/lstm_sequence/.
In this code, LSTM network is trained to generate a predefined sequence without any inputs.
This experiment was introduced by Clockwork RNN.
Four different LSTM networks and shell scripts(.sh) for training are provided.
Each script generates a log file containing the predicted sequence and the true sequence.
You can use plot_result.m to visualize the result.
The result of four LSTM networks will be as follows:
1-layer LSTM with 15 hidden units for short sequence
1-layer LSTM with 50 hidden units for long sequence
3-layer deep LSTM with 7 hidden units for short sequence
3-layer deep LSTM with 23 hidden units for long sequence
