MetricX-23

This is not an officially supported Google product.

This repository contains the code for running inference on MetricX-23 models, a family of models for automatic evaluation of translations that were proposed in the WMT'23 Metrics Shared Task submission MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task. The models were trained in T5X and then converted for use in PyTorch.

Available Models

There are 6 models available on HuggingFace that vary in the number of parameters and whether or not the model is reference-based or reference-free (also known as quality estimation, or QE):

We recommend using the XXL model versions for the best agreement with human judgments of translation quality, the Large versions for best speed, and the XL for an intermediate use case.

Changes to the WMT'23 Submission

These models available here are most similar to the primary submission to the WMT'23 Metrics Shared Task. They are initialized with mT5 then fine-tuned on a combination of direct assessment and MQM data. However, we made some changes that make these models different from the WMT'23 submissions.

First, the models are trained to regress the actual MQM score rather than a normalized score between 0 and 1. That means the output from the MetricX-23 models is a score in the range [0, 25] where lower is better (i.e., it predicts an error score).

Second, these models were trained with a larger variety of synthetic data that makes them more robust to translation edge cases like over- and undertranslation, described in more detail in the following section.

Synthetic Data

In order for our MetricX models to learn to identify certain types of bad translations that are not sufficiently (or at all) represented in the regular training data, we created synthetic examples and mixed them in during training. The synthetic training data was generated from the DA datasets ranging from WMT15 to WMT21 (~ 43 language pairs). In most cases, the synthetic examples have the candidate translation manipulated so as to turn it into a bad translation with a specific issue commonly unrecognized by learned metrics.

The table below provides an overview of the various failure modes that we considered, including brief descriptions of how we prepared the synthetic data to address them.

Failure mode	Synthetic example description
Undertranslation	Candidate translation with an arbitrary sentence removed (if multi-sentence); alternatively, candidate with a certain proportion of words removed from the end.
Overtranslation	Candidate translation duplicated (with space in between).
Fluent but unrelated translation	Arbitrary reference of a similar length from the dataset.
Gibberish	Text of a similar length as the reference, generated by sampling words from the reference translation vocabulary (built from all references in the data).
Missing punctuation	Reference translation with the end punctuation removed (11 punctuation symbols considered).
Latin instead of Chinese/Japanese or Hindi/Bengali punctuation	Candidate translation with the language-specific punctuation symbol at the end replaced with the Latin equivalent (e.g., "." instead of "。" or "।"); alternatively, the punctuation symbol is replaced with the Latin equivalent in the reference, keeping the correct one in the candidate.
Reference-matching translation	Reference translation copied as the candidate translation (unlike the rest of the synthetic data, these examples are meant to train the metric to predict a perfect score for candidates matching the reference).

Examples from the first 4 categories were assigned a label corresponding to the worst score on the given rating scale (e.g., 25 when mixed with MQM training data), whereas the reference-matching translation examples are assigned the best score (e.g., 0 when used with MQM data). The missing/incorrect punctuation examples were labeled with a score slightly worse than perfect.

Note that some of the synthetic datasets are only meaningful in the reference-based scenario, and we thus excluded them when training a QE variant of MetricX. These are the Latin-vs-special punctuation and the reference-matching translation examples.

Most of the synthetic training sets were created using stratified sampling across target languages, taking 500 examples per target language. One exception is the missing punctuation set, which used a stratified sample across different punctuation symbols instead.

When training MetricX, a small proportion of the synthetic examples was mixed with the regular training examples. During the first-stage fine-tuning on DA data, each synthetic training set constituted between 0.1% and 1% of all training examples, whereas in the second-stage fine-tuning on MQM data we used an even smaller proportion, around 0.05%.

As for evaluating the effect of the synthetic training data on the model's performance, the DEMETR challenge set - which we originally used to evaluate the models submitted to the WMT23 Metrics Shared Task - was not adequate anymore. We therefore created a new DEMETR-style test set based on the WMT22 DA data, with examples constructed analogically to the synthetic training examples, as described above. This test set helped us determine the right proportions of synthetic data for fine-tuning in order to make MetricX robust for the failure modes in consideration, without sacrificing the system- and segment-level correlations with human ratings.

Usage

The metricx23/predict.py script contains an example for how to run inference on the models.

Reference-Based

Example usage for a reference-based model:

python -m metricx23.predict \
  --tokenizer google/mt5-xl \
  --model_name_or_path google/metricx-23-xl-v2p0 \
  --max_input_length 1024 \
  --batch_size 1 \
  --input_file input.jsonl \
  --output_file output.jsonl

input.jsonl is expected to have 1 serialized JSON object per line with "reference" and "hypothesis" fields. The output jsonl will be parallel to input.jsonl but additionally contain a "prediction" field with the predicted score.

Note that the model was trained with a maximum input length of 1024 tokens, so significantly increasing that value may lead to unpredictable behavior.

Reference-Free

Example usage for a reference-free model:

python -m metricx23.predict \
  --tokenizer google/mt5-xl \
  --model_name_or_path google/metricx-23-qe-xl-v2p0 \
  --max_input_length 1024 \
  --batch_size 1 \
  --input_file input.jsonl \
  --output_file output.jsonl \
  --qe

input.jsonl is expected to have 1 serialized JSON object per line with "source" and "hypothesis" fields. The output jsonl will be parallel to input.jsonl but additionally contain a "prediction" field with the predicted score.

Meta-Evaluation

The metricx23/evaluate.py script contains code to calculate various correlations between the MetricX-23 scores and MQM ratings of translation quality using the MT Metrics Eval library.

Example usage:

python -m metricx23.evaluate \
  --dataset wmt22 \
  --lp en-de \
  --input_file input.jsonl \
  --output_file output.json

input.jsonl is expected to have one JSON object serialized per line. Each JSON object is expected to contain 4 fields:

"system_id": The name of the system that generated the translation.
"segment_id": The 0-based index of the corresponding segment in the MT Metrics Eval data.
"label": The ground-truth translation quality score (with higher is better).
"prediction": The model predicted translation quality score (with lower is better; the script negates the scores so higher is better).

The script will calculate the 4 agreement/correlations that were used in the WMT'23 Shared Task. Below are the results for the MetricX-23 models on the WMT'22 Metrics Shared Task data:

English-German:

Model	System-Level Accuracy	System-Level Pearson	Segment-Level Pearson	Segment-Level Pairwise Acc
MetricX-23-XXL	0.795	0.835	0.546	0.619
MetricX-23-XL	0.756	0.813	0.540	0.605
MetricX-23-Large	0.769	0.759	0.507	0.595
MetricX-23-QE-XXL	0.769	0.830	0.490	0.606
MetricX-23-QE-XL	0.718	0.684	0.421	0.594
MetricX-23-QE-Large	0.744	0.671	0.387	0.579

English-Russian:

Model	System-Level Accuracy	System-Level Pearson	Segment-Level Pearson	Segment-Level Pairwise Acc
MetricX-23-XXL	0.905	0.943	0.477	0.609
MetricX-23-XL	0.876	0.906	0.498	0.589
MetricX-23-Large	0.876	0.841	0.474	0.569
MetricX-23-QE-XXL	0.895	0.940	0.470	0.602
MetricX-23-QE-XL	0.848	0.861	0.415	0.570
MetricX-23-QE-Large	0.819	0.778	0.411	0.551

Chinese-English:

Model	System-Level Accuracy	System-Level Pearson	Segment-Level Pearson	Segment-Level Pairwise Acc
MetricX-23-XXL	0.868	0.919	0.605	0.551
MetricX-23-XL	0.868	0.924	0.584	0.543
MetricX-23-Large	0.857	0.919	0.555	0.539
MetricX-23-QE-XXL	0.857	0.928	0.573	0.544
MetricX-23-QE-XL	0.802	0.879	0.546	0.529
MetricX-23-QE-Large	0.758	0.904	0.522	0.529

The metricx23/evaluate_wmt23.py script re-calculates the average correlation score that was used to rank submissions from the WMT'23 Shared Task.

Example usage:

python -m metricx23.evaluate_wmt23 \
  --en_de predictions_ende.jsonl \
  --he_en predictions_heen.jsonl \
  --zh_en predictions_zhen.jsonl \
  --output_file output.json

Each of the 3 input files is expected to be in the same format as described above. Each file should correspond to running inference on each of the language pairs from the WMT'23 dataset.

The results for each of the models is the following:

Model	Average Correlation
MetricX-23-XXL	0.812
MetricX-23-XL	0.813
MetricX-23-Large	0.794
MetricX-23-QE-XXL	0.797
MetricX-23-QE-XL	0.767
MetricX-23-QE-Large	0.762

Citation

If you use MetricX-23 in your research, please cite the following publication:

@inproceedings{juraska-etal-2023-metricx,
    title = {{MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task}},
    author = "Juraska, Juraj  and
      Finkelstein, Mara  and
      Deutsch, Daniel  and
      Siddhant, Aditya  and
      Mirzazadeh, Mehdi  and
      Freitag, Markus",
    editor = "Koehn, Philipp  and
      Haddow, Barry  and
      Kocmi, Tom  and
      Monz, Christof",
    booktitle = "Proceedings of the Eighth Conference on Machine Translation",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.wmt-1.63",
    doi = "10.18653/v1/2023.wmt-1.63",
    pages = "756--767",
}

google-research/metricx