Quantify and benchmark Daemo's benefits to requesters and workers

[mbernst]

Problem

As we have spoken to many potential requesters, we have been repeatedly asked whether Daemo's quality is actually better than AMT. If not, these requesters have no interest in switching. We don't have a direct answer to their question. Likewise, we should be able to answer with confidence: are we actually supporting workers in achieving a living wage?

Proposal

This strategic proposal is for us to continue focusing on measuring whether Daemo is achieving quality and wage goals. Things we measure become things we optimize, so we have to be careful to align the specific measurements with our long term goals.

We have already initiated some efforts on this, but this strategic proposal is intended to set our course for the next three months and reaffirm our commitment to it. The rough steps in this strategic proposal would be:

• Identify a set of benchmark tasks and test them on AMT and Daemo under different configurations (e.g., qualifications): e.g., #14. As a start, we can draw on tasks we have already used in prior papers and evaluations.
• Analyze the results, and if they aren't as good as we'd like, identify what about Daemo or Boomerang is hurting us, and think through whether we can redesign things to help.
• Ideally, create a dashboard that we can use to monitor these results and cron these tasks to run periodically to give us updated data
• Draw on the recent literature on tracking worker earning rates and see if we can implement a similar wage benchmark for Daemo workers.
• Ideally, add the wage data to the dashboard.
  The actual steps may vary and will be reactive to what we find.

Implications

The short term implications of this proposal are that we would be able to measure whether our efforts are succeeding at achieving high-quality work and a living wage. This seems necessary to attract requesters. Long-term, this proposal will allow us to track how these factors are changing as we grow and have more workers and requesters on the system --- since there's no guarantee the results will be stable as we open to more people.

Contact

@michael Bernstein on Slack

---

To officially join in, add yourself as an assignee to the proposal. To break consensus, comment using this template. To find out more about this process, read the how-to.