rsennrich/wmt16-scripts

Training process quality control

Closed this issue · 2 comments

khusainovaidar commented

Could you help me with my questions. I’m trying to build NMT system for Tatar-Russian language pair (Ubuntu 16.04, GPU 1080).
Have some doubts whether my training procedure is going well.

  1. Is it ok for BLEU scores to be like
    0.6; 0.77; 0.83; 1.56; 1.42; 1.21; 1.72; 1.05; 3.88; 3.17; 4.98; 4.12; 5.72; 5.26; 0.48; 0.00; 0.54
    For now it’s 36th epoch (~5 days), update 2609000 (we train on 340k parallel sentences).
  2. Can we change valid set during training procedure?
rsennrich commented

  1. I don't know what BLEU scores are realisitc on this data set, but if they go over time down, that's an indicator that your model is overfitting. If you haven't already, it may be worth enabling dropout. You should definitely use early stopping and select the model that performs best in the end.

  2. Yes, you can interrupt and restart training, and if the 'restart' option is set to True, training will resume from the last saved checkpoint.

You may also be interested in enabling some improvements we used for WMT17 - some of them will speed up convergence (like layer normalization) and make your model better. See http://data.statmt.org/wmt17_systems/training/ for the scripts.

khusainovaidar commented

I'll try dropout and wmt'17 scripts.
Thank you!