This repo contains the supplementary materials for ACL 2021 paper: "All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text"
It contains:
Priming texts
When generating the machine-generated texts with GPT2 and GPT3, we used a "three-shot" setting, conditioning each passage on 3 in-domain, human-authored texts. The three texts we used for each domain are in priming_texts.tsv.
Training materials
Two of our trainings (Examples and Comparison) showed the evaluators example passages, along with a brief explanation for each example. The exact passages used for training in each domain are in training_materials.tsv.