This is the repository for the software project on Unsupervised Mandarin-Cantonese Machine Translation. The report can be found here.
1. RNN and Transformer Models leveraged the following repositories. For a step-by-step pipeline for running the RNN model in your own environment, see this google doc.
https://github.com/artetxem/undreamt
https://github.com/facebookresearch/UnsupervisedMT
2. Note that there are several components required to train a model for unsupervised machine translation, as detailed in our paper. Below are some components we used for various models, such as a BPE vocabulary generator and repository for training cross-lingual embeddings. Depending on which model configuartion you would like to run, they may be useful:
To map embeddings to a shared space, apply https://github.com/artetxem/vecmap to the seperately learned embeddings
This Github repository was used for generating BPE vocabulary.
We collected 910k monolingual Cantonese data, in addition to some parallel datasets. The scraping script can be found in crawl.py
and an example can be found in crawling_demo.py
.
Monolingual data is used in the train set, parallel data is used in the dev & test sets.
The complete dataset (Cantonese & Mandarin) can be found here and the binarized version here.
Domains include: wikipedia, articles, corpus, dictionary, reviews, movie / tv subtitles, youtube subtitles, instagram
https://drive.google.com/drive/folders/1XUOht0kdUJPf3byd_5ODLyAhPEEAvvep?usp=sharing
To obtain the pretrained model, run:
python3 main.py --exp_name test --transformer True --n_enc_layers 4 --n_dec_layers 4 --share_enc 3 --share_dec 3 --share_lang_emb True --share_output_emb True --langs 'can,man' --n_mono -1 --mono_dataset 'can:data_bin/mono/can.char.pth,,;man:data_bin/mono/man.char.pth,,' --para_dataset 'can-man:,data_bin/para/dev/dev.XX.pth,data_bin/para/test/test.XX.pth' --mono_directions 'can,man' --word_shuffle 3 --word_dropout 0.1 --word_blank 0.2 --pivo_directions 'man-can-man,can-man-can' --pretrained_emb 'data_bin/all.vec' --pretrained_out True --lambda_xe_mono '0:1,100000:0.1,300000:0' --lambda_xe_otfd 1 --otf_num_processes 30 --otf_sync_params_every 1000 --enc_optimizer adam,lr=0.0001 --epoch_size 500000 --stopping_criterion bleu_can_man_valid,10
on https://github.com/facebookresearch/UnsupervisedMT (NMT)
./generate_vocabulary.sh -s ../yue_100K.txt -t ../zh_100K.txt