The Numbers Do Lie
By some estimates, humanity now produces 2.5 quintillion bytes of data every day—more than a hundred times the amount of data in the entire Library of Congress. This data ranges from Facebook posts to military-grade satellite photos. Increasingly, this data is analyzed by complex mathematical models that determine more and more aspects of our lives, from the advertisements we see to whether we have access to private insurance. Yet despite their growing importance, these models often remain hidden.
Advocates of such mathematical modeling, in both the public and private sectors, portray it as a neutral and efficient alternative to fallible and biased human decision-making. Mathematician, data scientist and popular blogger Cathy O’Neil, author of Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, doesn’t agree. She argues that many mathematical models are ideological tools that exacerbate oppression and inequality. Her examples range from the crime models used by police departments to determine which neighborhoods to patrol, to the recidivism models used by judges to hand out prison sentences.
O’Neil is passionate about exposing the harmful effects of Big Data–driven mathematical models (what she calls WMDs), and she’s uniquely qualified for the task. She earned a Ph.D. in math from Harvard and landed a tenure-track at Barnard. But she became bored with the pace and insularity of academia, and left to work as a quantitative analyst at the hedge fund D.E. Shaw. There, she had a front-row seat for the 2008 financial crisis.
This experience fundamentally changed O’Neil’s relationship to mathematics. She realized that far from being a neutral object of study, mathematics was not only “deeply entangled in the world’s problems but also fueling many of them.” People in power were “deliberately [wielding] formulas to impress rather than clarify.” This disillusionment led O’Neil to get involved with Occupy Wall Street and start educating the public about the dangers of WMDs through her blog, MathBabe.
She is careful to point out that there is nothing inherently destructive about mathematical modeling. Sophisticated data modeling enables much of modern technology, from wireless communication to drug discovery. How can one distinguish a destructive math model from an ordinary, or even helpful, one? O’Neil identifies three key features of WMDs: lack of transparency, lack of fairness and, most importantly, operation on a massive scale.
O’Neil grounds her argument in case studies of WMDs in a variety of settings: finance, higher education, the criminal justice system, online advertising, employment decisions and scheduling, and credit and insurance provision. The “value-added” model for teacher evaluation, which looks at improvements in individual students’ test scores—a favorite of the so-called “educational reform” movement—is touted as an objective measure of a teacher’s worth. Yet this is far from the truth. O’Neil cites an analysis by blogger and educator Gary Rubinstein of New York’s 2010 value-added scores for public school teachers. O’Neil explains that “Of teachers who taught the same subject in consecutive years, one in four registered a 40-point difference. [This] suggests that the evaluation data is practically random.” O’Neil argues that this is because the value-added model, which relies on predictions of student performance, suffers from a built-in logical flaw: No statistical model can accurately make predictions about a class of 25 or 30 students—the sample size is too small. Yet the highstakes testing regime continues to wreak havoc on the trajectories of students and teachers alike.
Other WMDs, known as e-scores, use data such as ZIP codes, web-surfing patterns and recent purchases to evaluate a person’s credit-worthiness. Unlike the more familiar FICO credit scores that are freely available and regulated by the government, these secretive e-scores are “unaccountable, unregulated and often unfair.” Whereas FICO scores are based on your own financial history, e-scores compare you to other people with similar profiles. This may seem benign, but it can result in feedback loops that reinforce existing social inequities. If you live in a poor ZIP code, then your e-score will drop, meaning less credit and higher interest rates—essentially, an algorithmic redlining of the poor and working class.
Through these examples, O’Neil makes a convincing case that many mathematical models today are engineered to benefit the powerful at the expense of the powerless. The problem with most WMDs, O’Neil argues, is that it’s “almost always their objective” to maximize “the bottom line” rather than “fairness” or other “ethical priorities.” By enacting these choices, WMDs serve an important ideological function in modern capitalism. Far from eliminating human bias, O’Neil suggests, WMDs “camouflage it with technology.”
To paraphrase the great evolutionary biologist Richard Lewontin, WMDs are “math as ideology.” This argument is implicit throughout Weapons of Math Destruction, but the book would have benefited from an explicit discussion of the relationship between WMDs and capitalism. This is a minor shortcoming, however. Cathy O’Neil has written an entertaining and timely book that gives readers the tools to cut through the ideological fog obscuring the dangers of the Big Data revolution.