Decoder's output differs from the officially released one
Closed this issue · 4 comments
Hi,
Thanks for the data released and the task you organized. I downloaded the materials you offered but found something unexpected: For En-Zh and En-De (I just tried these two pairs), I found using the provided model checkpoints cannot reproduce the same results you provided in the data/{lang-pair}/word-probas/mt.{phase}.{lang-pair}
, 1% of the generated results are different from the released one. Could you please kindly help me to check it? The commands I used are listed below
cat train.ende.df.short.tsv | tail -n +2 | cut -f2 -d$'\t' > train.en
perl moses-scripts/tokenizer/tokenizer.perl -threads 80 -a -l en < train.en > train.tok.en
subword-nmt apply-bpe -c ../../nmt_models/en-de/bpecodes < train.tok.en > train.bpe.en
fairseq-interactive ../../nmt_models/en-de/ --path ../../nmt_models/en-de/en-de.pt --input train.bpe.en --beam 5 --source-lang en --target-lang de --unkpen 5 --batch-size 160 --buffer-size 200 | grep ^H | cut -f3- > trans.train.out
I didn't binarize the test files but I don't think this could be the root cause. I followed your instructions to pip install fairseq==0.8.0
and used PyTorch 1.2.0 as the backbone framework.
Thank you
Hi Tingxun,
Thanks for your interest! Could you please share some of the MT outputs that are different?
Hi Marina,
Thanks for the reply. Below attached the first four of the mismatched outputs
trans_data$ diff trans.train.out ../word-probas/mt.train.ende
11c11
< Si@@ enna Miller por@@ trä@@ tiert The Bar@@ on@@ ess als einen der Haupt@@ ant@@ ag@@ onisten von Stephen Somm@@ ers Film G.@@ I .
---
> Si@@ enna Miller por@@ trä@@ tiert The Bar@@ on@@ ess als einen der Haupt@@ ant@@ ag@@ onisten des Films G.@@ I. von Stephen Somm@@ ers 2009
217c217
< Im Dezember 1999 ver@@ lie@@ h die Northern Illinois University den Ehren@@ preis LL@@ .@@ D .
---
> Im Dezember 1999 ver@@ lie@@ h die Northern Illinois University einen Ehren@@ do@@ kt@@ or@@ titel .
575c575
< Als Pink F@@ loyd Nordamerika tour@@ te , war Wat@@ ers ' Radio K.@@ A.@@ O.@@ S .
---
> Als Pink F@@ loyd Nordamerika tour@@ te , Wat@@ ers ' Radio K.@@ A.@@ O.@@ S.
583c583
< 2008 spielte R@@ ih@@ anna auf der G@@ low in the Dark Tour neben Kan@@ ye West , Lup@@ e Fi@@ as@@ co und N.@@ E.@@ R.@@ D .
---
> 2008 spielte R@@ ih@@ anna auf der G@@ low in the Dark Tour neben Kan@@ ye West , Lup@@ e Fi@@ as@@ co und N.@@ E.@@ R.@@ D.
Hi Tingxun,
Sorry for the delayed response.
The difference could be caused by a different version of the tokenizer.
@adfors Thanks for the explaination, I got it