Algorithmic Fairness in Broward County: COMPAS and the Cost of Classification Errors

Belinda Torres, a fictional data analyst at the Broward County Office of the Public Defender, is asked by her chief to evaluate the COMPAS recidivism risk assessment algorithm following ProPublica’s “Machine Bias” investigation. Torres has two weeks to produce a memo with three deliverables: an independent statistical evaluation of COMPAS, an explanation suitable for judges, and a recommendation: challenge the algorithm in court, push for modification, or advocate for transparency and alternatives. Torres confirms the COMPAS company’s calibration defense, documents base rate differences and distributional asymmetries, and flags that the recidivism measure reflects charges rather than crimes. The case concludes before she examines the algorithm’s classification errors — the analysis students must perform.

Analyzing the case requires basic facility with descriptive statistics and the ability to compute and compare group-level rates from a dataset. The in-class exercise uses a prepared spreadsheet and short Python scripts (runnable in Jupyter, Colab, or within Stata), but assumes no prior background in classification metrics, confusion matrices, or ROC curves — these are introduced during the session.

• Compute and interpret classification error rates (false positive rate, false negative rate) from a real dataset, disaggregated by racialized identity.
• Distinguish between calibration and error rate parity as competing definitions of algorithmic fairness, and explain why satisfying both simultaneously is mathematically impossible when base rates differ.
• Trace the data generating process from policing to arrest to intake to algorithm to bail decision to recidivism outcome, identifying where bias can enter at each stage.
• Evaluate the normative implications of algorithmic error asymmetry in a pretrial justice context, connecting false positive rates to the presumption of innocence.
• Recommend a course of action by integrating statistical evidence, legal precedent, and institutional constraints.