Omission Error in MQM

[SAI990323]

HI,
I find that the omission error in mqm newstest2020 is labeled on the target side, but according to Figure 1 in your paper, the omission error is detected on the source side.

I wonder: have you transformed the annotations to match your predictions and how did you do it?

[jvamvas]

Hi @SAI990323, we did not evaluate word-level accuracy but only segment-level accuracy due to this limitation of the data (as discussed in google/wmt-mqm-human-evaluation#2). That is, we count a prediction as correct if any one of the MQM human raters has marked the same error type anywhere in the segment.

In order to compensate for this limitation we performed an additional word-level human evaluation (Table 2).

[SAI990323]

Thank you for your reply, this helps a lot. Jannis Vamvas ***@***.***> 于2022年3月22日周二 20:16写道：

…

Hi @SAI990323 <https://github.com/SAI990323>, we did not evaluate word-level accuracy but only segment-level accuracy due to this limitation of the data (as discussed in google/wmt-mqm-human-evaluation#2 <google/wmt-mqm-human-evaluation#2>). That is, we count a prediction as correct if any one of the MQM human raters has marked the same error type anywhere in the segment. In order to compensate for this limitation we performed an additional word-level human evaluation (Table 2). — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJVHP2LD4D34SAMRKHLBSRDVBG223ANCNFSM5RJ5YHQQ> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>