/corefBias

To analyze and remove gender bias in coreference resolution systems

Primary LanguageCSSMIT LicenseMIT

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang. NAACL-2018 short paper.

Updates:


If you want to try the WinoBias dataset using allennlp, remember to add "pos_tag != '-'" in line 274 here.


We analyze different resolution systems to understand the gender bias issues lying in such systems. Providing the same sentence to the system but only changing the gender of the pronoun in the sentence, the performance of the systems varies. To demonstrate the gender bias issue, we created a WinoBias dataset.

The dataset is generated by the five authors of this paper. We use the professions from the Labor Force Statistics which show gender stereotypes:

Professions and their percentages of women
Male biased Female biased
supervisor 44 cashier 73
janitor 34 teacher 78
cook 38 nurse 90
mover 18 assistant 85
laborer 3.5 secretary 95
constructor 3.5 auditor 61
chief 27 cleaner 89
developer 20 receptionist 90
carpenter 2.1 clerk 72
manager 43 counselors 73
lawyer 35 designer 54
farmer 22 hairdressers 92
driver 6 attendant 76
salesperson 48 writer 63
physician 38 housekeeper 89
guard 22 baker 65
analyst 41 accountant 61
mechanician 4 editor 52
sheriff 14 librarian 84
CEO 39 sewer 80
(Note: to reduce the ambigous of words, we made some modification of these professions: mechanician --> mechanic, sewer --> tailor, constructor --> construction worker, counselors --> counselor, designers --> designer, hairdressers --> hairdresser)