Language Drift in Collaborative Translation Game
This is a reproduction of the paper Countering Language Drift with Grounding. This is not the official implementation of the paper, but is part of the ICLR reproducibility challange.
This repo contains the implementation for pretraining as well as policy gradient baselines. It also implements a fine-tuning method coupled with language model and a PPO baseline, both these two haven't been made working yet.
Installation
Clone this repo by
git clone https://github.com/diplomacy-game/language-drift --recursive .
Install pytorch 1.0, NLTK, and torchtext. Install the other dependency when it fits. Then run
pip install -e .
Prepare Dataset
Run ./cli/prepare_dataset. It would put everythin under ./data folder.
Pretrain
Pretrain the fr-de agent and en-de agent by
python ./cli/pretrain.py \
-src_lang .fr -tgt_lang .en \
-logging_steps 100 \
-checkpoint_steps 2000 \
-valid_steps 2000 \
-train_steps 100000 \
-start_decay_steps 30000 \
-dropout 0.1 \
-device cuda \
-optim adam \
-learning_rate 0.001 \
-adam_beta2 0.98 \
-save_dir $SAVE_DIR
python ./cli/pretrain.py \
-src_lang .en -tgt_lang .de \
-logging_steps 100 \
-checkpoint_steps 2000 \
-valid_steps 2000 \
-train_steps 100000 \
-start_decay_steps 30000 \
-dropout 0.1 \
-device cuda \
-optim adam \
-learning_rate 0.001 \
-adam_beta2 0.98 \
-save_dir $SAVE_DIR
Finetune with RL
To fine-tune with RL one can run
python ./cli/communicate.py \
-pretrain_fr_en_ckpt ${PRETRAIN_FR_EN} \
-pretrain_en_de_ckpt ${PRETRAIN_EN_DE} \
-ent_coeff 0.01 \
-fr_en_learning_rate 0.0001 \
-norm_reward \
-logging_steps 10 \
-batch_size 256 \
-checkpoint_steps 100 \
-valid_steps 20 \
-train_steps 10000 \
-device cuda \
-fr_en_optim adam \
-fr_en_start_decay_steps 1000 \
-fr_en_decay_steps 1000 \
-fr_en_learning_rate_decay 1. \
-fr_en_max_grad_norm 1 \
-en_de_optim adam \
-en_de_learning_rate 0.001 \
-en_de_start_decay_steps 1000 \
-en_de_decay_steps 1000 \
-en_de_learning_rate_decay 0.5 \
-value_optim adam \
-value_learning_rate 0.001 \
-value_start_decay_steps 10000 \
-value_decay_steps 1000 \
-dropout 0.1 \
-save_dir $SAVE_DIR
Add -reduce_ent flag to use the opposite of entropy loss. Add -disable_dropout flag to not using dropout.
All flags can be found in ld_research/parser.py. Use tensorboard to view the results. Also to check both BLEU score and token frequency analysis, use
python cli/check_en_drift.py -ckpt [CKPT1] [CKPT2] [CKPT3] \
-out translation.txt -data multi30k