delegability/data

Confirm label coding

Closed this issue · 3 comments

Katiedaisey commented

Great ideas and a wonderful survey.

Could you please confirm the encoding of label in the csv files and used in the analyses are as:

  1. Human Only
  2. Machine-in-the-loop
  3. Human-in-the-loop
  4. Machine Only

This seems opposite of the encoding in the survey and in the Plots page of the overview.

Katiedaisey commented

@blubars Any update?

blubars commented

@Katiedaisey I'm so sorry, missed this before! Thank you for bringing this up.

The label in the CSV here is the same as the encoding from the survey (the only processing we did was to remove identifying info and attention check questions), but you're right that we reversed it for our analysis in the paper and website. We flipped it in the analysis so that it aligned better with the word delegability, where a more delegable task means less human control. Note the survey encoding was not visible to the survey takers, where the options were presented in an unordered list.

So:

  • Surveys and CSVs encode 4 as Human Only, 1 as Machine Only.
  • Analysis in paper and website uses 1 as Human Only, 4 as Machine Only.

Apologies for the confusion! I will add a readme to this repo clarifying the encodings.

Katiedaisey commented

Awesome! Thanks. That's what I though happened, but I wanted to check.