/literalness

Code for "Do GPTs Produce Less Literal Translations?"

Primary LanguagePython

Code for the metrics used in "Do GPTs Produce Less Literal Translations?" (ACL 2023).

Get alignments from vyraun/awesome-align, which is a fork of neulab/awesome-align.

bash extractv3.sh source_file target_file

This will produce a .aligned file with alignments in the Pharaoh (i-j) format.

Then, use compute_scores.py to get the USW and NM scores.

python compute_scores.py source_file alignment_file target_file

If you find our code or paper useful, please cite the paper:

@inproceedings{raunak-etal-gpt-literalness,
    title = "Do GPTs Produce Less Literal Translations?",
    author = "Raunak, Vikas and Arul, Menezes and Post, Matt and Awadalla, Hany Hassan",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics",
    publisher = "Association for Computational Linguistics",
    year = "2023"
}