We’re only beginning to understand the extent to which targeted advertisements played a role in the 2016 presidential election, and we’re still reeling from the fallout; the Equifax leak has left thousands of people’s personally identifiable information vulnerable to exploitation; and every day, a rash of new startups make grand claims about using Big Data to do anything from recognize a person’s sexual orientation, to predict their likelihood of leaving school. Of course, this is not the first time we’ve seen how lack of privacy harms us, but it has been hard to find a non-technical introduction to the topic. Now, we have one: Cathy O’Neil’s Weapons of Math Destruction. Her book was first inspired by another rough period in our recent political-economic history, the market crash of 2008. In this book, she examines the way individuals and companies use Big Data, laying out the faults in their methods—and the impacts.

After earning her PhD at Harvard, Cathy O’Neil started as a mathematics professor in the combined Barnard-Columbia math department. However, she didn’t start worrying about the practical use of mathematics until later. After leaving academia, she started working for a hedge fund, where she put her theoretical mathematical background into practice. She became what is known as a quant (short for “quantitative analyst”): she searched for market inefficiencies that could prove profitable. Socially, she felt isolated at the company; working teams kept their activity secret from one another in order to reduce the damage if an employee left. Her largest issue with the work, however, was the moral ambiguity of what she was doing: while she and her coworkers only saw the numbers, in reality they were playing with people’s retirement funds and mortgages. The financial collapse of 2008 drove that point home, and it was then that she began to think about how we rely too much on mathematical models but are blind to their biases. She started a blog, Mathbabe, which soon gave rise to this book.

Weapons of Math Destruction looks at how mathematical models and algorithms make decisions. While these mathematical tools have the potential to improve people’s lives, they more often than not harm the average person and benefit only a select few. When this happens, O’Neil calls the models “weapons of math destruction.” She creates a clear definition of a weapon of math destruction (or WMD) with three words: Opacity, Scale, and Harm. When a mathematical model is opaque, it operates as a black box; the people whom the model examines do not know what and why the model does what it does. Scale indicates the model is used across large populations. This is where Big Data becomes powerful, because it gathers and exploits a massive amount of data across a wide population. This is what gives these WMDs their power. Finally, WMDs do damage. They only serve those who already have money and power, and they do so at the expense of everyone else. If a model or algorithm uses Big Data and doesn’t have these three characteristics, then it wouldn’t be a WMD. Mathematical models and algorithms are not intrinsically weapons—they tend to be.

Yet we can use the technology that produces WMDs for good. O’Neil herself works on programs that use this technology to help homeless people find the right housing for their situation. But O’Neil is not a naïve algorithm writer; she recognizes that her model is similar to models that anticipate whether prisoners will end up incarcerated again. She reminds us that if we recognize our bias, we can minimize the harm our mathematical models do, and turn them from weapons of math destruction into something beneficial. Disarming weapons of math destruction is a long, difficult process, and its first step is recognition. O’Neil’s book helps with that first step, taking its readers by the hand and pointing out the injustices in a system we once thought impartial.

Often, we see mathematics as objective, representing the truth, because numbers are harder to twist than words. However, when dealing with complex algorithms, bias and subjectivity can slip in. Whether it’s by unknowingly using data that acts as a proxy for race or socioeconomic status, or failing to account for some unmeasured factor, models can be flawed in any one of harmful ways O’Neil describes in her book. Doing “good” math—math that has some positive effect on the world with the least bias possible—is harder than it first appears. Because Big Data is so new and is ever-evolving, we struggle to fit it into our legal, social, and moral frameworks, and that means no one is held accountable for how they use our data. To ensure that we do the most good in the world as we can, we need to keep a critical eye on technology and how we use it—a critical eye that has not yet become commonplace.

Kai Matheson and I interviewed Cathy O’Neil just before winter break about subjects ranging from gender in STEM, to computer science and ethics, to how politics and technology interact. The text below is a transcript of the interview, which has been edited for clarity and brevity.

— Nora Culik, Science + Tech Editor

Nora Culik: I know that you used to be a professor at Barnard, and then you were a quant, and now you write and audit algorithms. What do you think was most distinctive about each of the places that you’ve worked in the realms that you’ve worked?

Cathy O’Neil: Most distinctive? Mathematically? Culturally?

NC: I guess culturally, but also mathematically. What struck you as most different in each of the places you’ve worked from each other?

Kai Matheson: What made you transition from one job to another? What marks those transitions and marks those decisions that you were making?

CO: Well, I can tell you why I did each thing. I became a quant because I thought it would be a way for me to be measured objectively, rather than in a way that was subjective and relatively sexist. I was wrong about that. You don’t become more objective just because you’re dealing with money. I don’t want to be overly cynical to young people, and I don’t think that the world is awful—but there were some things I was going toward, and some things I was going away from. I left academia in part to get away from this idea that, as a woman who was good at organizing, I would be taking charge of a lot of organizational tasks in the math department. But I was also going toward this idea that my work would have more of an effect in a short-term way on the world. I was essentially wrong about the first thing. As for the second thing, I was surprised that I did have an effect on the world, but I had a bad effect. The truth is, when you’re an academic and a mathematician, your frustration is that no one cares what you’re doing. Then, when you’re working at a hedge fund during a financial crisis, you realize that people do care—and not in a good way. At that point, I didn’t just want to have influence; I wanted to have a positive influence. I quit for that reason.

I went to work in a risk firm in finance thinking that I could make things better. Maybe I could do better math in the context of risk, so the world would get better rather than worse. One positive side effect of going from a hedge fund to a risk firm is that it became a lot friendlier. I realized in retrospect how competitive my workplace had been. I hadn’t even noticed it because it had always been like that. I had worked at MIT, then Columbia, then a hedge fund. All of those are very competitive, very hierarchical in certain ways, very status-oriented. By the time I got to the risk firm, it was much less like that. There were more women, and it was a nicer place to work. But then the company I was working for was acquired by another company about a year in, and the new company was much more competitive. The culture of the company I was working for was ruined by the new company, so I left there soon after that. Although, I would’ve left anyway because I wasn’t improving the world, and I was being used kind of as a window dressing. It was like we were doing math better there, but we weren’t really doing anything better.

Then I left finance altogether and I joined a startup. It was once again a very male, very competitive environment. But I didn’t take it very seriously in terms of its effect. I was so glad to get away from finance that at first I thought, “What damage can we possibly be doing here?” It took me a while to figure out that we could be making exactly the same mistakes I’d seen happen in finance, with destructive and very large-scale algorithms giving us the wrong impression and making some problems worse. At that point, I decided to quit that job as well. That was the fourth job I quit. I became a professional quitter and started writing a book. Since I wrote the book, I haven’t really had a job-job. I’ve had a lot of consulting gigs, but consulting is a very different thing than having a boss.

NC: With what you’re doing now, do you feel like you’re fulfilling those goals of trying to create some good in the world?

CO: Yeah, because now I’m my own company, which is the only way I’ve found I can work for a boss that I can stand. And I’m explicitly only taking clients that I actually want to take.

NC: That’s a very good way to do that.

CO: Yeah, and I’ve said “no” to a bunch of clients and I’ve said “yes” to a few. I take clients where I think I’m actually going to make the world a better place in some meaningful way, and where I think the work is worthwhile and interesting. And it’s so mathematical, that’s the cool thing.

KM: Is the work a lot of applied math modeling or more stats?

CO: I’m auditing algorithms, making sure that algorithms are working as expected and they’re fair, with some definition of “fair” that must be discussed at length. That’s what I find the most interesting question of all.

NC: A lot of biases can be hidden, and sometimes people don’t even realize what their biases are. Have you ever been auditing someone, and when you look at the math you realize it isn’t fair and it’s contrary to the what their mission is? Can you tell us a little bit about that, if that’s the case?

CO: That situation hasn’t occurred yet, and it’s because, right now, I’m getting clients that want to prove to the outside world that they’re doing what they say they’re doing. If, in the future, it becomes a requirement for business to have an outside auditor for their algorithms, then I’m sure it will come to people who say, “We say we’re doing X but we’re actually doing something else, Y, which in conflict with X.” I’m sure that will be the case. I’m sure it’s happening all over the place. Those people aren’t coming to me and asking me to look at their algorithms… And they probably won’t. People like that will be the ones that are forced to get their algorithms audited. I think people know that their algorithms don’t live up to their mission. They don’t want to think about it very hard.

NC: It’s easier to pretend you’re doing the right thing than have someone else tell you you’re not.

CO: Exactly.

NC: I’m going to bring it back to the gender thing you were mentioning at the beginning of our conversation. I feel like I see a lot of stereotypes where men tend to be the dominant gender within tech stuff. For example, I see Silicon Valley as a “boys’ club.” Do you think that gender plays any role in some of what you have noticed, specifically related to your book?

CO:

Gender plays a role in every competitive environment I’ve ever been in—just as much in academia as in hedge funds or tech.

Finance tech and math academics all have a power issue and a male issue. I don’t think it just has to do with tech alone. I think it has to do with status. There seems to be a zero-sum game where it seems like there’s going to be like an alpha guy, and everybody wants to be that guy. That’s when men behave badly.

NC: So it wasn’t different across the different places you’ve worked?

CO: It was slightly different. It came up in slightly different behaviors, slightly different one-upmanship. But it was essentially a competition.

KM: There’s something I’m curious about—you worked at Barnard, which is an all-women’s college. Do you think that working in that department was different than it could have been in a different academic setting, or do you think all of the same problems existed there as well?

CO: By the way, I was a Barnard professor, but I was working in the combined Columbia-Barnard math department. There were very few Barnard professors. There were three of us, and there were about forty Columbia professors. I was expected to help the Columbia math department. I was expected to advise Columbia students, even though I was already advising Barnard students. I could go on, but the short version is that if I had been in a liberal arts college environment without that weird combined math department, if I was with just my students and it was all-women’s college, I think would been a very different experience. I will add that I never had any problem with the Barnard administration at all. I really liked them. My negative interactions were due to this weird combined math department. Having said that, one of the benefits of working at Barnard was that I would be able to take graduate students and I’d take on graduate courses to teach. That was only possible because of the Columbia connection. It wasn’t all bad.

NC: In your recent opinion piece for The New York Times, you talk about how academia is failing in regards to educating students about the role of technology in our lives and the way it shapes us. And you recognize that it’s a fairly large problem, and we can’t solve it overnight. As students, do you have any advice for us for filling the gaps in our knowledge?

CO: In that piece, I was trying to criticize STEM departments for not teaching very important information to their future engineers and data scientists of the world, very important information that actually does get taught—but not to STEM students. I caught quite a bit of flak for forgetting to mention that. I do want to mention that I messed up. This probably is mean, and I don’t know your particular situation, but there are probably currently good classes being taught in the history of science or sociology or anthropology departments that would prepare you for a future life of an engineer or computer scientist in the sense that they will make you think harder about what historical practices led to this data collection, in the context of policing. For example, at UC Berkeley there is an ethics class in terms of ethics and computing but it’s not required for computer science majors. It’s a very different world. The problem is that it’s not integrated. It’s not a requirement. Secondarily, that it’s not available. There are places like Berkeley where it is available but not required, and then there’s other places where it’s not available at all. The short answer is if it’s available, definitely take it. Of course, the problem is that the people who are going to go ahead and do that are the exactly the people I’m not as worried about.

NC: Something like “computer science and ethics” would be the obvious class to take. Are there any more obscure or not-so-obvious subjects that you think that STEM students should search out to study? You mentioned history of science, which didn’t strike me as one of the first choices I would take.

CO: People say ethics a lot. I think ethics, depending on what you mean by that, could be enough. I’m going to enlarge it to mean accountability, especially algorithmic accountability. Of course, depending on what you mean by that, that could be smaller than ethics. Something larger than just fairness and “what is fairness?”—something much closer to, “How do you know that this algorithm works?” and “Who’s responsible for it breaking?” And depending on how it broke, “How do we fix it?” When I say “broke,” I don’t necessarily mean it delivered the wrong answer; it could mean it delivered a racist answer or it delivered a sexist answer. Who is responsible for making sure this is working and is fair and is legal and is not discriminating illegally or generally? There’s a lot of questions there, some of which are in the realm of ethics, many of which are not. They’re still not being covered in machine learning classes.

KM: In the last year, I’ve had the opportunity to go to the Grace Hopper Conference for Women in Computing, and I also went to Joint Mathematics Meeting (JMM). JMM was a lot of old white men, and it was straight up math all the time. It was what I expected. But then Grace Hopper was an entirely different experience. There were research talks, but a lot of the research talks were about how to ethically create AI, or that diversity in the workplace is important so we don’t create AI that are only based on one type of person. I was struck by how different the types of conversations at the events were. What do you think of those two atmospheres? As someone kind of between math and computer science at my own college, the math department definitely feels more reminiscent of JMM compared to the type of environment that exists in the computer science department here, which I know is different than a lot of other colleges. I feel like there are these two very different groups of people that are working on these issues. What has your experience been with that?

CO: Yeah, that’s a really good point. I talked to a woman who’s interested in the accountability of algorithms. She’s a computer science assistant professor. She told me that she’s afraid of talking about it publicly because her colleagues in the computer science department—which is a good computer science department—will dismiss her as not serious, as soft, if they hear that she’s interested in these questions. I feel like that says everything you need to know about the problem of STEM fields and how they dismiss questions of accountability and ethics as “girlish” and therefore uninteresting and not hard work. By contrast, it’s the most fascinating question for me. Those are some of the hardest questions to answer.

It’s a real shame that we have this kind of macho lens through which everything is seen, because it prevents us from actually putting resources into these kinds of questions and making progress.

I’m very glad that I don’t work in a computer science department right now. It frees me up to think about things that I find interesting, rather than what is actually rewarded. That was kind of what I was trying to say in my essay: the reward system in computer science, and to some extent in math, statistics, and physics, is always set up explicitly to not think through these questions of accountability and what could go wrong and how does it affect society. If you want to think about that stuff, you’re not going to get resourced. You’re going to get isolated into a field that computer scientists don’t think is interesting. It’s awful. Meanwhile, the lobbyists of big tech companies are paid millions of dollars to whisper into the ear of Congress.

NC: I also noticed that the lobbyists and the people who run these corporations and stand to profit seem to be the sources of a lot of information. Like you said, they’re telling congresspeople what matters, and they’re paying lots of money. I feel like it’s obvious that these are biased sources—they’re sharing this information because it will directly benefit them. I’m curious if you have a theory of your own as to why congresspeople—and people in general—see these as credible sources of information.

CO: Well, I think you first need to understand how information flows in Washington. I’m not an expert on this. Basically, senators and congresspeople are not experts on this at all. They don’t have time for this. They are constantly trying to raise money, so they’re on the phone with donors all the time. They have staffers who work for the congressperson, who are typically 25 to 30 years old. They’re really energetic and very smart. Their jobs are to figure out the information that the congressperson or the senator should know. There are very few of them that are devoted to technology, but there are some of them. Those technology people are not tech nerds themselves; they’re staffers to senators, so they’re political science nerds. They are smart, but they can only read certain types of information and understand it. And the lobbying firms produce content that these people can understand. These people’s job is to learn this stuff to tell the congresspeople in soundbite form what’s going on. Content related to their topic that’s understandable is at a premium. That’s one of the things that was exciting to staffers I talked to in the Senate and Congress; they said my book is readable to them. They were like, “Finally, a book that we can actually read and understand that is critical of big data!” They had very few other things that they could read that would educate them in a way that they can understand.

KM: That’s the whole point, though, right? They’re trying to make it hard to understand so that there isn’t much oversight or questions about what’s really happening.

CO: I agree. I would go a little further; in the world of technology criticism, this question gets raised every few weeks. Is this something new, or is this something we’ve actually seen before? People are always like “we’ve seen this before.” There’s always been some hyped new thing that’s going to change everything. But I do think there’s something about Big Data that is new, and it’s so difficult to pin down. That’s partly because it’s complicated, partly because the technology is massive in the amount of data going into it. It’s gigabytes and terabytes and petabytes. But it’s also because of this lack of accountability that we were talking about. No one in particular is at fault for a mistake. It’s also partly because the definitions of success embedded inside the algorithms are proprietary because the algorithms are proprietary, so we don’t even know what we’re fighting against.

We don’t even know what a failure looks like because we don’t know how success is defined.

When you add it all up, it is actually almost impenetrable. As I said, congresspeople and senators are really not that evil, but they don’t have a million years to learn something. They’re not on top of this, and you’ve got to get them a little bit more on top to get them to be more wary and skeptical of trusting every single thing that’s handed to them technology-wise. You have to compete with the lobbyists, and we’re not doing that.

NC: Since the election, something I’ve been thinking about is that the Facebook advertising for Donald Trump clearly played a very large role in the election. Obviously, there’s an issue there.

KM: The other day, we heard a lecture by someone who worked at Facebook about how a lot of the Facebook algorithms think they are being apolitical by boosting whatever content is getting shared the most, and that creates echo chambers within Facebook for different people. Facebook’s algorithms predicted they want to see this kind of content so they only see that kind of content. If Facebook’s choosing to share Donald Trump advertisements, then they’re implicitly endorsing what he’s doing. But then there’s also this huge push for tech companies to be “apolitical.” The same goes for academic research even, which is something that I’ve been really bothered by. In being apolitical, they are being inherently political.

NC: Being neutral is picking a side.

CO: I agree with you. I would go further and say there’s no such thing as objective. There’s never something objective; we just have a bias in a way we’re okay with. That certainly goes for any kind of so-called neutral approach to newsfeed algorithms. You’re doing somethings; you’re not missing everything, so obviously you’re making choices. You might not think of it as political, but as we’ve seen, Congress has been acting in a way that seems like they were paying attention to Facebook’s influence on the election. It will get politicians’ attention every time if they think their election is being screwed with because, after all, they care a lot about that. So that’s a good thing.

NC: Net neutrality is the pervasive internet/tech/data issue that everyone is thinking about. But aside from that, is there something that you’re concerned about? Is there a pressing issue that needs action?

CO: I’m not sure where I am on the net neutrality debate. I think it’s probably not great, but I don’t think it’s the end of the internet either. I’m much more worried about algorithms that are used to incarcerate people or decide how police should act. For that matter, I’m much more worried about police practices and how uneven practices are creating biased data sets, and then we use those data sets as perfect.

I think that, when it comes to technology concerns, I’m more and more worried about microtargeting in politics than I am about net neutrality.

But everybody gets to decide what to worry about, and one of the reasons I don’t worry about net neutrality is because of so many other very powerful people are worried about it. That frees me up to worry about other things.

As long as we’re on this topic, I’ll say that I also don’t worry about self-driving cars. My explanation for that isn’t that I think they’re perfect. It’s that when they fail, everyone will notice. And that’s exactly what I worry about when I worry about algorithms. Failure will be invisible. We’ll notice when someone dies in a car crash because an algorithm fucked up, but we won’t know when people don’t get a job because of an algorithm fucked up, or if they’re sent to prison for too long because the algorithm is based on racist proxies. I worry about invisible failures by technology. There was an article, maybe a year ago, about how Google and other big companies have been trying to build delivery drones, but they’ve come across technical barriers that mean they’ve wasted enormous amounts of money and time with these delivery drones. The question that the article posed was “How can they be so good at the data algorithms, but so bad at hardware?” So good at software, so bad at hardware. I wondered, “Why do you think they’re good at software?” We only think they’re good at software because it’s not obvious when they fail. And I think that’s a really important point: that people assume, because these things seemed to work, that they are working. That’s a very dangerous problem. That’s what I want to address. BP