This module is an example of how create a recursive neural network language model using TensorFlow.
To train a model run
tfrnnlm train text.txt model
This will use the text in text.txt to train an RNN language model that will be stored in the directory model.
See tfrnnlm train --help for the model parameters that you can set.
Various parameter sets are specified as text files in the parameters directory.
For example, to build a small model run
tfrnnlm train text.txt model @parameters/small
The model here is based on the Penn Treebank language model described in the TensorFlow RNN tutorial.
The text is tokenized on whitespace into words and converted into a sequence of indexes into a vocabulary. Subsequences of these indexes are passed to the model as input and used to predict the following index. Token indexes are embedded in a vector space the same size as the number of hidden layers in the RNN. An RNN predicts a token given a sequence, and the average per-token cost is calculated using the cross-entropy loss. Training runs for a specified number of epochs.