Weapons of Math Destruction

I do a fair amount of modeling at work, nothing as consequential as a credit score or a sentencing algorithm, more like predicting failure rates on parts, but the underlying instinct is the same one O’Neil is going after here. Models are opinions embedded in mathematics, and most people never see the opinion, only the number that comes out the other end.

The three things she keeps circling back to are opacity, scale, and damage, and that framing is what’s stuck with me since finishing it. A bad model that everyone can inspect gets fixed. A bad model nobody can see into just keeps running, and the people it’s grinding through have no path to even understand why they got the outcome they got, let alone appeal it. The teacher-evaluation chapter got under my skin the worst, a system so noisy that a good teacher gets fired based on what’s essentially statistical noise dressed up as objectivity.

It’s not a technical book, if you wanted equations you won’t find many, and I get why some readers found it a bit too pop-magazine in its prose for their taste. I didn’t mind that. The examples do the work, hiring algorithms, insurance pricing, predictive policing, and by the fourth or fifth one you start seeing the same shape everywhere: a proxy standing in for something that would be illegal to measure directly, and for the person on the other end, the computer just says no.

The one thing I wanted more of was what to actually do about it, beyond “demand transparency,” which is true but a little thin as a call to action for anyone who isn’t a regulator. Still, if you work in tech, this is the one I’ve been telling people they need to read, including two coworkers who build the kind of systems this book is warning about and didn’t love hearing it.

Big data should come with a warning label, and this book is it. Five stars, and I mean it as a warning as much as a recommendation.