An LSTM that incorporates best practices, designed to be fully compatible with the PyTorch LSTM API. Implements the following best practices: - Weight dropout - Variational dropout in input and output layers - Forget bias initialization to 1
These best practices are based on the following papers: A Theoretically Grounded Application of Dropout in Recurrent Neural Networks Regularizing and Optimizing LSTM Language Models An Empirical Exploration of Recurrent Network Architectures <http://proceedings.mlr.press/v37/jozefowicz15.pdf>
This code is heavily based on the code from this repository: most of the credit for this work goes to the authors. (All I have done is update the code for PyTorch version 1.0 and repackage it).
Install via pip.
$ pip install .
Requires PyTorch version 1.0 or higher.
>>> from better_lstm import LSTM
>>> lstm = LSTM(100, 20, dropoutw=0.2)