Law Professor’s New Book Puts Ethnography on Trial

Replication controversies have plagued the social sciences in recent years. But one research method is uniquely difficult to verify: ethnography. Ethnographers, who often work in sociology and anthropology, immerse themselves in the daily lives of people they study. Their written narratives tend to cloak both subjects and field sites behind a curtain of anonymity. Only rarely does one ethnographer go back to another’s site to reexamine the original researcher’s findings.

Steven Lubet, a law professor at Northwestern University, thinks that’s unfortunate. He got interested in ethnography after writing a scathing review of Alice Goffman’s controversial 2014 study of young black men caught up in the criminal-justice system, On the Run (University of Chicago Press), a work of urban ethnography that he viewed as littered with ethical and factual problems. That made him wonder: Did the whole field suffer from similar issues?

Mr. Lubet, who has dedicated much of his academic career to the study of legal and historical evidence, tried to find out by plunging into the literature. He read more than 50 ethnographic monographs and an equivalent number of articles. Focusing on sociologists’ studies of American cities, he hunted for facts that could be documented — or not. He verified details by consulting experts and pulling public records.

The result of his investigation is a new book, Interrogating Ethnography: Why Evidence Matters (Oxford University Press). Its conclusion: Ethnography suffers from an accuracy problem, one that scholars in the field have largely overlooked.

Mr. Lubet did not identify problems as sweeping as those uncovered in psychology, where a recent review of 100 experiments found “more than half of the findings did not hold up when retested.” Most of what he read withstood scrutiny relatively well, he writes. But Mr. Lubet — a self-described fan of ethnography — also found many assertions that were “dubious, exaggerated, tendentious, or just plain wrong.”

The problem, as Mr. Lubet sees it, is ethnographers’ inconsistent approach to reporting and documenting facts. Often, he says in an interview, they state as fact what is really “hearsay": something they heard, rather than observed. He argues that ethnography should adopt methods for dealing with evidence that more closely resemble those practiced in other fields, like law and journalism.

“When a fact is stated in a newspaper, if it was told to the reporter by somebody, it’s presented that way,” Mr. Lubet says. “If it’s something the reporter saw, it’s presented that way. If it comes from a document, it’s presented that way. And, in ethnography, those three were often confused — or not drawn precisely.”

A New Round of Debate

Mr. Lubet’s book has touched off a debate about what ethnographers might learn from legal scholars, and vice versa.

“Tone deaf” is how John Van Maanen, an organizations ethnographer at the Massachusetts Institute of Technology, describes Mr. Lubet’s writing.

“If ethnography was concerned only with verification and, quote, facts, it would be a kind of dismal endeavor,” he says. “We wouldn’t get the narrative. We wouldn’t get the story. We wouldn’t get the feel for the culture that is being explored. And to think about ethnography as just seeking out facts is sort of missing the point.”

But another ethnographer, Colin Jerolmack of New York University, finds much of Mr. Lubet’s book compelling. He cringed reading the book’s accounts of ethnographers who reported their informants’ stories as facts, without much effort at verification.

“When you just go through hundreds of pages, with dozens of examples of it, in well-regarded ethnographies, it makes it hard for me to just say, ‘Oh, well, those are the few outliers who aren’t really careful,’” says Mr. Jerolmack, who joined a prominent cast of scholars at Northwestern for a symposium on Mr. Lubet’s book in October.

One example of Mr. Lubet’s critique of ethnography is a 2015 book that he returns to repeatedly: $2.00 a Day: Living on Almost Nothing in America (Houghton Mifflin Harcourt).

This study, by Kathryn J. Edin (the Johns Hopkins University) and H. Luke Shaefer (University of Michigan), mixes ethnographic field work and quantitative analysis to examine a swell in the number of U.S. families living in extreme poverty. William Julius Wilson, a Harvard University sociologist, has called it an “essential book.”

But Mr. Lubet, while also an admirer of the book, questions some of its details. Among them: its account of a woman who lost her job as a cashier at the pseudonymous “Stars Music,” a position she had held for eight years, because one day her cash drawer came up $10 short. Mr. Lubet checked that story with an expert, Caitlin Kelly, a journalist who writes about the retail industry. Given the high employee turnover in such stores, Mr. Lubet doubts that a veteran worker would lose her job for a one-time shortfall. He suggests she might have “exaggerated the length of her work history, and perhaps the cause of her termination.” He faults Ms. Edin and Mr. Shaefer for failing to check tax and employment records that could have verified the story.

Elsewhere in $2.00 a Day, Mr. Lubet flags the story of some sixth-graders from the Mississippi Delta who see an elevator for the first time during a class trip to Washington, D.C. Ms. Edin and Mr. Shaefer report that some of the kids, disbelieving that it could transport them between floors, imagined their teachers to be playing a joke on them. Mr. Lubet devotes several pages to debunking this “glaringly implausible hearsay.” His quest involves interviewing roughly a half-dozen people in Mississippi — teachers, social workers, a former mayor — and analyzing the prevalence of elevators in TV shows to which these kids might have been exposed. Lubet concludes that “the claim of elevator bafflement was indeed a joke — but it was played by the students who feigned ignorance, not by the chaperones who appear to have fallen for it.”

Why bother with all this? Mr. Lubet uses the tale as a case study for how a legal method, the pursuit of circumstantial evidence, can vet a story in the absence of direct proof.

Defending his work, Mr. Shaefer criticizes Mr. Lubet for privileging the reporting of experts, while failing to recognize those experts’ stake in the issues. Mr. Shaefer expresses confidence that both the music-store and elevator episodes happened in the way he and Ms. Edin portrayed them — portrayals grounded in testimony from people who were there as well as other community members and knowledgeable sources.

For example, he points to a former student of his who taught fourth grade in the Delta region. She relayed a similar experience with students encountering an elevator for the first time. It’s hardly surprising that a mayor would deem the story implausible, Mr. Shaefer says. The mayor has an incentive to say that.

University professors are often disconnected from the life experiences of the poorest Americans, Mr. Shaefer adds. “That it’s hard for us to think that that type of story would happen,” he says, “is exactly the reason why we need ethnography.”

Murder and Anonymity

Mr. Lubet, it’s worth noting, has worked for the Legal Assistance Foundation, a poverty-law organization in Chicago, and he served for many years as a defense lawyer in the Cook County juvenile and criminal courts.

Beyond the elevator dispute, his book points to potentially more serious problems with other ethnographies.

He reports that he was unable to verify two murders detailed in Sudhir Venkatesh’s Off the Books (Harvard University Press, 2006), about the underground economy of poor people living in Chicago. In one of those episodes, Mr. Venkatesh describes how a hustler named “Babycake Jackson” was shot 22 times while sleeping in an abandoned building because he didn’t want to keep serving as a drug dealer’s lookout. Mr. Lubet combed Chicago homicide records but found no killing that fit Mr. Venkatesh’s description.

“Murder is something that leaves a paper trail,” Mr. Lubet says. “Even in Chicago, police statistics about murder victims, at least the occurrence of murders, tend to be pretty reliable — very reliable.”

Mr. Venkatesh is not new to controversy. Other scholars have faulted the former Columbia University sociologist for his perceived exaggeration and sensationalism. Asked to comment on Mr. Lubet’s findings, Mr. Venkatesh, who now works at Facebook, stands by his work.

“I conducted this research under very specific rules and guidelines set forth by my university,” Mr. Venkatesh says in an email. “I am under obligation to reduce risk to subjects, which means that I cannot divulge identifying information and, on occasion, I need to alter identities to protect confidentiality. I am also ethically bound by promises to my research subjects that I will not enable others to find them. All the events and incidents described in the book are real.”

Mr. Venkatesh’s response points to another general concern that Mr. Lubet raises about ethnography: anonymity. Mr. Lubet understands why individual identities may need to be concealed. But he sees much less reason for the convention of anonymizing research sites, which impedes other scholars’ efforts to reexamine those locations.

Here, again, he has company among some sociologists. In recent years, Mr. Jerolmack and a colleague, Alexandra K. Murphy of the University of Michigan at Ann Arbor, have published a paper and online conversation that challenge anonymization as a default ethnographic practice.

Mr. Jerolmack described how masking had hindered his research about the impact of fracking on rural life. Entering his field site, a town in Pennsylvania called Williamsport, he discovered that an earlier ethnography had been done about the same area before fracking began. He could have used it to help design his study. But he hadn’t known of its existence, because the earlier book did not identify the town.

He was further hindered by how the earlier ethnographer had concealed details about civic leaders, historical events, and community groups. It meant Mr. Jerolmack couldn’t examine how those groups had changed over time with the advent of fracking.

“It inhibits what I would call cumulative social science: the capacity of a study to contribute to future studies,” Mr. Jerolmack says.

Marc Parry is a senior reporter who writes about ideas, focusing on research in the humanities and social sciences. Email him at marc.parry@chronicle.com, or follow him on Twitter @marcparry.