Evaluating Gender Bias in Machine Translation

This repo contains code and data for reproducing the experiments in Evaluating Gender Bias in Machine Translation Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer, (ACL 2019).

Citing

@InProceedings{Stanovsky2019ACL,
  author    = {Gabriel Stanovsky and Noah A. Smith and Luke Zettlemoyer},
  title     = {Evaluating Gender Bias in Machine Translation},
  booktitle = {ACL},
  month     = {June},
  year      = {2019},
  address   = {Florence, Italy},
  publisher = {Association for Computational Linguistics},
  pages     = {(to appear)},
}

Requirements

fast_align: install and point an environment variable called FAST_ALIGN_BASE to its root folder (the one containing the build folder).

Install

./install.sh

Running our experiments

This is the entry point for all our experiments: scripts/evaluate_all_languages.sh. Run all of the following from the src folder. Output logs will be written to the given path.

For the general gender accuracy number, run:

  ../scripts/evaluate_all_languages.sh ../data/aggregates/en.txt  path/to/output/folder/

For evaluating pro-sterotypical translations, run:

  ../scripts/evaluate_all_languages.sh ../data/aggregates/en_pro.txt  path/to/output/folder/

For evaluating anti-sterotypical translations, run:

  ../scripts/evaluate_all_languages.sh ../data/aggregates/en_anti.txt  path/to/output/folder/

Adding an MT system