This is an implementation of the models described in the paper "A Character-Level Decoder without Explicit Segmentation for Neural Machine Translation". http://arxiv.org/abs/1603.06147
The majority of the script files are written in pure Theano.
In the preprocessing pipeline, there are the following dependencies.
Python Libraries: NLTK
MOSES: https://github.com/moses-smt/mosesdecoder
Subword-NMT (http://arxiv.org/abs/1508.07909): https://github.com/rsennrich/subword-nmt
This code is based on the dl4mt library.
link: https://github.com/nyu-dl/dl4mt-tutorial
Be sure to include the path to this library in your PYTHONPATH.
We recommend you to use the latest version of Theano.
If you want exact reproduction however, please use the following version of Theano.
commit hash: fdfbab37146ee475b3fd17d8d104fb09bf3a8d5c
The original text corpora can be downloaded from http://www.statmt.org/wmt15/translation-task.html
Once the downloading is finished, use the 'preprocess.sh' in 'preprocess' directory to preprocess the text files.
For the character-level decoders, preprocessing is not necessary however,
in order to compare the results with subword-level decoders and other word-level approaches, we apply the same process to all of the target corpora.
Finally, use 'build_dictionary_char.py' for character-case and 'build_dictionary_word.py' for subword-case to build the vocabulary.
Updating...