This is a fork of OpenNMT-py that allows for inspection of the activations of intermediate neurons, as well as manual modification of those neurons at inference time. Everything is known to work with the single-direction RNN encoder; others have not been tested. This fork adds the following option to train.py:
-separate_layers
Train the RNN or BRNN with a separate module for each layer. This must be true at training time in order for -dump_layers to work at inference time.
It adds the following option to translate.py:
-dump_layers FILENAME
Dump the activations of all encoder layers to FILENAME. Only works on models trained with -separate_layers (or non-RNN models, theoretically). The file will be in .pt format, consisting of an array of shape (sentences) x (tokens in this sentence) x (layers) x (neurons in this layer), where only the last dimension of the array is a Tensor.
It adds the correlate.py and correlation-to-basis.py scripts in the searchers/ directory. These aren't easy to use yet, but they provide functions that can be called from Python, with a usage example at the bottom of correlate.py.
It also adds the script mask_out.py in the root directory. This is run with the following options:
-mask_out_layer INT
Which layer to apply masking out to
-mask_out_basis FILENAME
A .pt file containing a (# of neurons in this layer) x (# neurons in this layer) matrix, where the (n)th row is the
hypothesized (n)th most important basis vector for the space. When testing individual neurons' importances, this will be
a permutation matrix. The basis must be orthonormal.
-mask_out_cumulative
Use this flag to do the cumulative experiment, where more neurons are masked out at every step. If this is not set, then
instead each interval will be masked out independently (i.e. there will be a run without 0-50, then without 50-100, etc.)
-mask_out_intervals INT
The number of intervals to divide the basis into. E.g. if this is 10, and there are 500 neurons, then at every step 50 more neurons
will be masked out.
mask_out.py also accepts all the same arguments as translate.py. When running mask_out.py, the -output option should specify a directory, and this directory will be populated with files named without-%d-%d.txt with the two integers indicating which range of basis vectors was masked out. Currently, -dump_layers will not work, as every next run will overwrite the dump. This will hopefully be fixed soon.
The README from upstream OpenNMT-py follows.
This is a Pytorch port of OpenNMT, an open-source (MIT) neural machine translation system. It is designed to be research friendly to try out new ideas in translation, summary, image-to-text, morphology, and many other domains.
Codebase is relatively stable, but PyTorch is still evolving. We currently only support PyTorch 0.4 and recommend forking if you need to have stable code.
OpenNMT-py is run as a collaborative open-source project. It is maintained by Sasha Rush (Cambridge, MA), Ben Peters (Saarbrücken), and Jianyu Zhan (Shanghai). The original code was written by Adam Lerer (NYC). We love contributions. Please consult the Issues page for any Contributions Welcome tagged post.
All dependencies can be installed via:
pip install -r requirements.txtNote that we currently only support PyTorch 0.4.
The following OpenNMT features are implemented:
- data preprocessing
- Inference (translation) with batching and beam search
- Multiple source and target RNN (lstm/gru) types and attention (dotprod/mlp) types
- TensorBoard/Crayon logging
- Source word features
- Pretrained Embeddings
- Copy and Coverage Attention
- Image-to-text processing
- Speech-to-text processing
- "Attention is all you need"
- Inference time loss functions.
Beta Features (committed):
- multi-GPU
- Structured attention
- [Conv2Conv convolution model]
- SRU "RNNs faster than CNN" paper
python preprocess.py -train_src data/src-train.txt -train_tgt data/tgt-train.txt -valid_src data/src-val.txt -valid_tgt data/tgt-val.txt -save_data data/demoWe will be working with some example data in data/ folder.
The data consists of parallel source (src) and target (tgt) data containing one sentence per line with tokens separated by a space:
src-train.txttgt-train.txtsrc-val.txttgt-val.txt
Validation files are required and used to evaluate the convergence of the training. It usually contains no more than 5000 sentences.
After running the preprocessing, the following files are generated:
demo.train.pt: serialized PyTorch file containing training datademo.valid.pt: serialized PyTorch file containing validation datademo.vocab.pt: serialized PyTorch file containing vocabulary data
Internally the system never touches the words themselves, but uses these indices.
python train.py -data data/demo -save_model demo-modelThe main train command is quite simple. Minimally it takes a data file
and a save file. This will run the default model, which consists of a
2-layer LSTM with 500 hidden units on both the encoder/decoder. You
can also add -gpuid 1 to use (say) GPU 1.
python translate.py -model demo-model_acc_XX.XX_ppl_XXX.XX_eX.pt -src data/src-test.txt -output pred.txt -replace_unk -verboseNow you have a model which you can use to predict on new data. We do this by running beam search. This will output predictions into pred.txt.
!!! note "Note" The predictions are going to be quite terrible, as the demo dataset is small. Try running on some larger datasets! For example you can download millions of parallel sentences for translation or summarization.
Go to tutorial: How to use GloVe pre-trained embeddings in OpenNMT-py
The following pretrained models can be downloaded and used with translate.py.
@inproceedings{opennmt,
author = {Guillaume Klein and
Yoon Kim and
Yuntian Deng and
Jean Senellart and
Alexander M. Rush},
title = {Open{NMT}: Open-Source Toolkit for Neural Machine Translation},
booktitle = {Proc. ACL},
year = {2017},
url = {https://doi.org/10.18653/v1/P17-4012},
doi = {10.18653/v1/P17-4012}
}