ToDo's: Measures

[pfistfl]

Unit Tests:

• Write unit test for datasets [data types, column types, number row, cols, PTA)
• Write unit tests for MeasureFairness [data types, parameters, produces error on wrong inputs)
  This should perhaps error if we try to use it with a task that does not have a pta.
• Write unit tests for pre-defined measures in zzz.R
• Write unit tests for "operations" in Measure Fairness

Code:

• Migrate operations code to a single operations.R.

Implement Measures (auto-loaded in zzz.R):

• Top 10 Measures from Wiki
• Equalized odds might not be a 1-line function. So consider implementing it and then just export it in zzz.R.
  Extend base_measure to a list of Measures. Those will then be added together.
• Are there measures that we currently can not compute with our approach?
  We should at least consider one probability score / calibration based measure!

Vignettes

• Write a vignette on measuring fairness of a classifier / regressor.
  • Should explain the idea behind measuring fairness: Why do we do it?
  • Should have a table of pre-defined fairness metrics (in zzz.R) with an explanation, perhaps link to literature
  • Should contain 1-2 examples (COMPAS, Adult) for measuring fairness (3 different measures in total, using benchmark
    and how to score a Prediction object.
    Predictions: prd$score(msr("fairness.fpr"), t)
    Benchmark: bmr$aggregate(....)
  • Should explain how to build your own measure by using MeasureFairness

Organization:

• Go through fairness related issues and migrate them here / check whether they should be closed

[superp0tat0]

Current: Improving the vignettes