Decoder's output differs from the officially released one

[TingxunShi]

Hi,

Thanks for the data released and the task you organized. I downloaded the materials you offered but found something unexpected: For En-Zh and En-De (I just tried these two pairs), I found using the provided model checkpoints cannot reproduce the same results you provided in the data/{lang-pair}/word-probas/mt.{phase}.{lang-pair}, 1% of the generated results are different from the released one. Could you please kindly help me to check it? The commands I used are listed below

cat train.ende.df.short.tsv | tail -n +2 | cut -f2 -d$'\t' > train.en
perl moses-scripts/tokenizer/tokenizer.perl -threads 80 -a -l en < train.en > train.tok.en
subword-nmt apply-bpe -c ../../nmt_models/en-de/bpecodes < train.tok.en > train.bpe.en
fairseq-interactive ../../nmt_models/en-de/ --path ../../nmt_models/en-de/en-de.pt --input train.bpe.en --beam 5 --source-lang en --target-lang de --unkpen 5 --batch-size 160 --buffer-size 200 | grep ^H | cut -f3- > trans.train.out

I didn't binarize the test files but I don't think this could be the root cause. I followed your instructions to pip install fairseq==0.8.0 and used PyTorch 1.2.0 as the backbone framework.

Thank you

[mfomicheva]

Hi Tingxun,

Thanks for your interest! Could you please share some of the MT outputs that are different?

[TingxunShi]

Hi Marina,

Thanks for the reply. Below attached the first four of the mismatched outputs

trans_data$ diff trans.train.out ../word-probas/mt.train.ende
11c11
< Si@@ enna Miller por@@ trä@@ tiert The Bar@@ on@@ ess als einen der Haupt@@ ant@@ ag@@ onisten von Stephen Somm@@ ers Film G.@@ I .
---
> Si@@ enna Miller por@@ trä@@ tiert The Bar@@ on@@ ess als einen der Haupt@@ ant@@ ag@@ onisten des Films G.@@ I. von Stephen Somm@@ ers 2009
217c217
< Im Dezember 1999 ver@@ lie@@ h die Northern Illinois University den Ehren@@ preis LL@@ .@@ D .
---
> Im Dezember 1999 ver@@ lie@@ h die Northern Illinois University einen Ehren@@ do@@ kt@@ or@@ titel .
575c575
< Als Pink F@@ loyd Nordamerika tour@@ te , war Wat@@ ers &apos; Radio K.@@ A.@@ O.@@ S .
---
> Als Pink F@@ loyd Nordamerika tour@@ te , Wat@@ ers &apos; Radio K.@@ A.@@ O.@@ S.
583c583
< 2008 spielte R@@ ih@@ anna auf der G@@ low in the Dark Tour neben Kan@@ ye West , Lup@@ e Fi@@ as@@ co und N.@@ E.@@ R.@@ D .
---
> 2008 spielte R@@ ih@@ anna auf der G@@ low in the Dark Tour neben Kan@@ ye West , Lup@@ e Fi@@ as@@ co und N.@@ E.@@ R.@@ D.

[adfors]

Hi Tingxun,

Sorry for the delayed response.
The difference could be caused by a different version of the tokenizer.

[TingxunShi]

@adfors Thanks for the explaination, I got it