Biased data teaches algorithms how to discriminate

Math is a tool that doesn't discriminate. There's no bias in it; the numbers either add up or they don't. Algorithms depend on math, but they're data driven -- sometimes the information being fed into one is incorrect or doesn't represent the actual goals of the algorithm. Cathy O'Neil, the author of Weapons of Math Destruction, cautions us against trusting the data being fed into our judicial systems: And what ProPublica found was the compass model, which is one version of a recidivism model, made mistakes by sending people to prison longer, that kind of mistake, twice as often for African-American defendants as for white defendants, at least in Broward County Florida. There's another kind of mistake you can make which is: you look like you're not coming back, you look low-risk but you actually do come back that kind of risk, that kind of mistake, was made twice as often for white defendants as for African-American defendants.