One of your articles is NOT a best practice
stucchio opened this issue · 3 comments
ProPublica: Machine Bias https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
This article is actually a worst practice - it's a truly dishonest and misleading piece of journalism. If you read their own R-script you'll discover that they could not find statistically significant evidence of bias (and this is in spite of failing to correct for multiple comparisons). Lines 36 and 46 of the R script are key.
That's why the article is full of anecdotes and ignores the statistical results that the authors found.
This debate is a long one so I'll focus on just a few side points.
- The use of anecdotes is pretty common in journalism and their presence should not be seen as a signal of illegitimacy. Besides their utility in storytelling, anecdotes serve as a form of accountability as it becomes difficult to hide behind your analysis when real people are identified and have an opportunity to rebut or file lawsuit against you.
- The article comes with a Github repo that contains the code, the methodology, and the raw data used, including jail data that was web-scraped from Broward County and then provided as an easy-to-use SQLite database. Hard to be much more transparent than that.
My criticism is not the use of anecdotes, but the use of anecdotes that contradict their own analysis. Every single anecdote presented pairs a white person/false negative with a black person/false positive. If they chose anecdotes honestly, they'd have shown a lot of true positives/true negatives of both races (their own analysis says the score is predictive with p < 0.001), and a few false positives/false negatives of both races.
This article is like taking a statistical analysis that says "cannot reject the null hypothesis that Jews and Gentiles commit crime at the same rate" and illustrating it with 10 anecdotes of Jewish serial killers contrasted against Christian pacifists.
I agree that providing the github repo is good policy. But it's not a "get out of jail free" card.
Although the list is opinionated, I added some criteria for choosing the visualizations.