The Fine Art of Sniffing Out Crappy Science

Stephen Brashear

Facts and figures are like cow pastures. Unless you squint, you can’t always tell how full of bullshit they are.

Carl T. Bergstrom and Jevin West, a pair of scientists at the University of Washington, think it’s time to arm students with boots and shovels. They have published the outline of a course, titled “Calling Bullshit,” which would try to teach how to spot bad data and misleading graphs at a time when bending statistics has become a popular art form.

“Pending approval from the administrative powers-that-be at the University of Washington, we hope to offer the seminar in the near future,” they wrote on a website they built for the course. “In the meantime, connoisseurs of bullshit may enjoy the course syllabus, readings, and case studies that we have lovingly curated.”

The Chronicle caught up with Mr. Bergstrom, a biologist, and Mr. West, an information scientist, to talk about their course. The following interview has been edited and condensed.

Q. In the wake of the election season, your course and its title arrive with a sense of urgency. But you don’t just implicate politicians in the spreading of bullshit. You also call out advertisers, the techno-optimists of start-up culture, and administrators of all stripes. How are you defining bullshit?

Carl Bergstrom: There’s a lot of discussion of how to define bullshit in the philosophy literature, rather to one’s surprise. Harry Frankfurt defines it as producing verbiage without concern whatsoever for the truth. And that’s bullshitting, according to him. There are others who argue bullshit is broader than that, and would include deliberate attempts to deceive.

How we define it is: “language, statistical figures, graphics, and other forms of presentation intended to persuade by impressing and overwhelming a reader or listener with a blatant disregard for truth and logical coherence.”

Q. So something that has that seductive, kind of musical quality of truth but which is unreliable when subjected to scrutiny.

Mr. Bergstrom: Yes, or kind of impressing you and making you feel like you cannot challenge it because I dump a bunch of numbers at you and say, Look, here are the statistics, and we ran the latest machine-learning algorithm on it, and here’s a fancy data visualization.

Q. The theater of rigor.

Mr. Bergstrom: Yes, exactly. And that could be done verbally, but more and more we see it done quantitatively with figures, data graphics, and with appeal to algorithms that generate results but which no one really understands.

Q. Bullshit is, I think we can agree, both pervasive and often cleverly disguised. How do we know this class is not also bullshit?

Mr. Bergstrom: That’s a good question … [long pause] ... Do you have a witty, snap answer to that, Jevin?

Jevin West: After the students take the course, hopefully they’ll be able to assess whether that’s true or not. The class isn’t a set of lectures only produced from our brains. We’re using readings and real material that you find in texts that are about this very topic. We’re aggregating for the students this kind of information that’s around and sort of gluing it together, so hopefully by the end they could assess that. But it’s a great question.

Mr. Bergstrom: And, you know, unfortunately, good bullshit might do all of the things that Jevin just mentioned. I guess I would say, ‘Trust us.’

Q. Well, that’s ultimately what so much of it comes down to, right? Politicians are successful, as we’ve recently seen, not because the rigor of their arguments checks out according to some refereed system of bullshit identification, but because enough people trust them that whatever data and data graphics and extras they’re bringing to the table amount to something credible.

Mr. Bergstrom: That’s right. Let me give you a serious answer to the question of whether the course is bullshit: What we’ll do in the course is teach people a set of tools and techniques for assessing quantitative evidence that will allow them to make claims about that evidence that will generally be accepted by the scientific community as valid.

The Post-Truth Issue

“Post-truth—adjective; relating to or denoting circumstances in which objective facts are less influential in shaping public opinion than appeals to emotion and personal belief.” So says Oxford Dictionaries, announcing their 2016 word of the year. If we really have entered a post-truth era, as so many have written, what does that mean for the scholar and the student? For the citizen and the state? In our special issue, we wrestle with these and other urgent questions.

• The Plots Against America
• Fighting Fake
• Facebook and Falsehood
• Google and the Misinformed Public
• Our Graduates Are Rubes
• When Truth Becomes a Commodity

Q. I went to college not too long ago, and I remember learning skills for identifying bullshit. I also remember learning how to bullshit. And I’m sure as professors you’ve read papers, possibly submitted at the 11th hour ...

Mr. Bergstrom: No, we’ve never seen anything like that. [They laugh.]

Q. … and said, ‘This student is bullshitting right now.’ And then, sure enough, you take out your grading pen and say, ‘But, they bullshitted well enough to pass this paper.’ Do you think that colleges are, at some level, complicit in teaching students how to become effective bullshitters?

Mr. Bergstrom: I think that what we’re teaching in the classroom, at least in the sciences, is how to cut through numerical bullshit of various sorts. But I do concede that the assessment structure may favor those who practice the art on their own time.

Q. Strictly speaking, bullshit — like, literal bull shit — can be used as a fertilizer to grow things that are necessary for survival. Similarly, political bullshit can be used to inspire good acts. Start-up bullshit can help raise money for useful products. Administrative bullshit, um … can, uh …

Mr. Bergstrom: It must be useful for something. [They laugh.]

Mr. West: Maybe to generate funds to build a new building or something.

Q. Yeah, OK, there you go. So, is bullshit always bad?

Mr. West: Since our focus will be on number or algorithmic or graphic manipulation, I think in that case it is all bad. What you’re talking about has more of a social context, and I think in that case, you’ve couched it pretty well — maybe there are circumstances where it’s OK.

Mr. Bergstrom: You have to distinguish a little bit between the types of bullshit you’re talking about. I mean, we’ve got to repeatedly, every day, say, ‘Hey it’s great to see you,’ even when we’re thinking, ‘Oh God, I wish I’d ducked around the corner.’ So, there are these white-lie, minor acts of bullshit that are essential to lubricate social function.

Q. Is there such a thing as a white lie in statistical bullshit?

Mr. Bergstrom: I don’t believe so. I believe the whole purpose of statistics is to impose a sort of rigor on messy data and make strong claims, or formally specify how strong of claims we can make, given the data. Some people will say, ‘Well, the statistics I used was actually not sufficiently conservative and so my confidence interval is probably a little broader — but, it’s close enough.’ And I really dislike that.

Q. The full title of your course is “Calling Bullshit in the Age of Big Data.” How does big data exacerbate the problem, and in what ways can it be used as an antidote?

Mr. Bergstrom: In the past, you had very small data sets, because that’s all you could get. If you had a hypothesis that was wrong, and you tested it, the data would tell you you’re on the wrong track. But once you’re looking at a data set with a hundred million observations, almost anything you test is going to come out to be statistically significant. You think you’ve found this strong signal, and it turns out that it’s all the result of some spurious correlation in the data.

Mr. West: In my field we spend a lot of time building software tools, visualization tools, algorithmic tools. So more and more of the population is engaging with this kind of data. More and more beautiful graphs are being produced. Which is great! But with that come lots and lots of mistakes in arguments and the way they’re presenting the data. We see more and more of this stuff, and more kinds of bullshit in graphical representation and in numbers and stats.

Q. In my experience, calling bullshit often will cause other people call you cynical. Cynicism isn’t necessarily a bad trait to cultivate, but is there a way that calling bullshit can go wrong?

Mr. Bergstrom: I expect there’s all kinds of ways that calling bullshit can go wrong, and I expect we’ll find out most of them pretty soon. One thing I do want to think about is that I want to be very good-spirited in everything I put together in the class. I want to challenge myself to think well of actors who have put together bullshit, rather than saying, “Oh, this person is working for this agency I don’t like, so they’re probably trying to deceive me.”

I think the effort to humanize the text you’re looking at is very important. We often do too little of that in the academy, and people are very quick to say, ‘Oh these people are just stupid,’ ‘Oh, that discipline is garbage.’ There’s a difference between being a hard-minded skeptic and being a domineering jerk.

Q. So alongside the tools students can use to identify and call out bullshit, you’re also teaching the discretion to know when it’s appropriate to do so?

Mr. Bergstrom: It’s very, very important in interdisciplinary work. Because when you sit down and listen to somebody from another field, they are bringing a whole different set of assumptions, possibly a different jargon. You’re thinking, “None of this can be right — this is bullshit!” And then you realize, “No, what they’re saying is completely consistent. They’re just working off different assumptions than the ones you were making.”

It’s very important to have that generosity of spirit … [laughs] at least, until you’re pretty sure it’s not merited.

Mr. West: We’ll be making fun of some of the issues we’ve had in our own work and our research, just because we know humans are fallible.

Mr. Bergstrom: My first case study attacks my own textbook. Just to say, “There’s bullshit in all kinds of forms, and here, I put some in my textbook, and it’s interesting to look at how that happened to me.”

Something I think I need to work on is learning how to call bullshit on a claim without attacking a person. I’d love to get better at that, and help other people learn how to do that as well.

Steve Kolowich writes about how colleges are changing, and staying the same, in the digital age. Follow him on Twitter @stevekolowich, or write to him at steve.kolowich@chronicle.com.