The Criminologist Accused of Cooking the Books

Last May, 11 criminologists received an email from someone using the name “John Smith.” The pseudonymous message outlined statistical irregularities in five published papers on which those criminologists are co-authors. These were more than mere rounding errors or minor oversights. These were giant red flags: percentages that couldn’t be true, standard deviations that didn’t add up, and a host of other issues that defy easy explanation. The numbers didn’t make sense.

The obvious next step in determining what went awry would be to reanalyze the data. That data is in the possession of Eric Stewart, a professor of criminology at Florida State University and the only co-author who appears on all five papers. It’s Stewart who could, presumably, clear up what happened. But so far he’s been less than forthcoming, refusing to share key data even with his co-authors. Meanwhile the information he has offered has only deepened the confusion. For instance, a recently published correction for one of the five papers is itself shot through with numerical oddities, leading to suspicions that the numbers don’t make sense because the numbers might be, at least in part, made-up.

The behind-the-scenes fight over these flawed papers has escalated into something more fraught than your standard academic disagreement. While Stewart has remained publicly mum, and hasn’t responded to multiple interview requests made over several weeks, privately he has been vocal about what he views as a campaign of harassment against him. In emails and text messages sent to colleagues, Stewart has portrayed himself as the target of “data thugs” who are attempting to ruin his career. In an email to Florida State administrators, he accused one of his co-authors of having “essentially lynched me and my academic character.” It’s a loaded verb choice not only because Stewart is black, but also because two of the five papers in question focus on the horrific history of lynching in the United States.

The co-author Stewart referred to in that email is Justin Pickett. For the last few months, Pickett, an associate professor of criminology and criminal justice at New York’s University at Albany, has repeatedly pushed Stewart to share the relevant data and has posted online a 27-page critique of suspicious findings in one of the papers. He has also contacted Florida State administrators and journal editors, urging them to investigate.

Pickett’s actions have divided the field of criminology: Some applaud his zeal in attempting to uncover the truth, while others see him as violating unspoken norms of collegiality. Stewart and Pickett are locked in a dispute that reflects a broader debate in social science about the reliability of results and the transparency of methods.

But Stewart and Pickett aren’t just co-authors. Pickett completed his doctoral work at Florida State and Stewart was his mentor there. In the years since, the two have remained close, commenting on each other’s Facebook posts, grabbing beers together at conferences, talking about their growing families, trading academic war stories. “Until a couple of months ago, we were really good friends,” says Pickett. “The only reason I’m doing what I’m doing is that I can’t get any answers.”

The paper that fractured their friendship is also the paper that kick-started Pickett’s academic career. When Pickett was a graduate student, Stewart asked him if he wanted to collaborate on what sounded like an important paper about criminal punishment and social inequality. He jumped at the chance. The project took a while and there were some false starts along the way (Pickett remembers writing an introductory section that didn’t quite work and had to be scrapped). Eventually the paper — “Ethnic Threat and Social Control: Examining Public Support for Judicial Use of Ethnicity in Punishment” — was published in Criminology, the field’s flagship journal, in 2011. Getting an opportunity to be a co-author on a high-profile paper came at the perfect time for Pickett, who would soon be on the job market. He knew he had Stewart to thank for that.

All along Stewart had encouraged Pickett, who was his teaching assistant for four years. Stewart once referred to him as “my guy,” a vote of confidence that stuck with Pickett. The letter of recommendation Stewart wrote for Pickett is chock-full of praise for his research prowess and affable demeanor. Stewart wrote that Pickett was “one of the top theoretical students we have in our graduate program” and Stewart was “continually impressed by his efforts to take on extraordinarily difficult intellectual challenges.” Pickett, Stewart wrote, was “one of the most enjoyable people I have ever met.” On Pickett’s graduation, Stewart gave him a set of pens with his name engraved on them. “I really owe him my career,” Pickett says. “If Stewart had not put me on that paper, I wouldn’t have gotten the job that I have now.”

When they were working on the paper together, Pickett never laid eyes on the raw data the findings were based on. Though he regrets that now, Pickett says it would have been unusual to ask to review it given the nature of their relationship. He trusted that the more senior scholar had checked the required boxes.

Once he had doubts about the data, Pickett found himself faced with a personal and ethical dilemma. Clearly something was wrong, but pressing for answers, he knew, came at a cost. It would be awkward, for starters. Stewart might find it insulting. And what if the paper turned out to be a fraud? That wouldn’t be good for anyone’s reputation. The safest course of action for Pickett, who wasn’t even the paper’s lead author, would be to do nothing and move on.

Instead he requested the data from Stewart — gently at first, assuming that he would forward it along. He reminded Stewart, in text messages and phone calls, to send the information as soon as he could. During one of those calls, Stewart warned him, somewhat cryptically, not to send any emails because that would create a “paper trail.” That seemed strange to Pickett but he didn’t object at the time.

As Stewart related the situation to Pickett, he was a victim of shadowy research vigilantes. He even implied at one point that these disreputable people might be trying to steal his data and somehow use it for their own purposes. “It seems very personal. All of the blame is being directed at me,” Stewart texted Pickett. “For some reason, data thugs are after me.”

Stewart had known since at least February that questions had been raised about his data. Before “John Smith” — whoever he was — had sent an email to the co-authors on those five papers, he had contacted Nick Brown and James Heathers, two researchers known for ferreting out and exposing scientific misconduct. Brown and Heathers have scrutinized papers for evidence that the results have been massaged or outright faked. They’ve also written software programs that can detect when certain results are unlikely or statistically impossible. Heathers and Brown, along with a small band of analytically inclined compatriots, have sometimes been referred to as “data thugs” for their efforts to push for more rigor in social science and their willingness to call out shoddy research.

As they’ve done in other cases, Heathers and Brown combed through the data that “John Smith” emailed them in February. (In an interview, Brown wouldn’t answer questions about the identity of “Smith,” though both Heathers and Brown say it’s not them. As Heathers puts it, “We do everything under our own names — always have, and have in this case.”) That email zeroed in on a paper, “A Legacy of Lynchings: Perceived Black Criminal Threat Among Whites,” published this year in Law & Society Review. The paper has five co-authors, including Stewart and Daniel Mears who, like Stewart, is a professor of criminology at Florida State. In the paper, which examines whether the attitudes of white people toward the punishment of black criminals might be related to the history of lynching, there were, according to Heathers, a “host of unusual features in the data.”

Among the most unusual features was the sample size: 2,736 American adults. That’s a sizable survey, the cost of which would almost certainly run into six figures, and yet no funding source was listed. Who footed the bill? Maybe even more remarkable was the response rate — 61 percent — which is extraordinarily high for a phone survey. The decline of landlines has made conducting phone surveys increasingly difficult; people don’t answer calls from unknown numbers, and it’s impossible to know whether a person’s area code matches their actual location. Pew Research Center published a report this year that lists typical response rates for phone surveys at less than 10 percent, so it would be amazing if the majority of the people the survey organization reached out to had actually agreed to participate. Borderline miraculous, in fact.

Speaking of survey organizations, which one had conducted this curious survey? Stewart and Mears hadn’t revealed its name in the paper — and the organization Stewart used in the past, Research Network, had shut down before this survey was carried out.

The survey had caught Pickett’s attention when he heard about it last year. Pickett isn’t a co-author on that paper, but he wrote to Stewart to ask what organization had conducted it. Stewart told Pickett in an email at the time that some “grad school buddies” had trained students to make calls and, in return, Stewart had helped them with some analysis on their projects. That seemed awfully vague to Pickett. Who were these grad school buddies? And why would they take on such a costly and time-consuming project for no pay? Stewart didn’t say, so Pickett shrugged and let it drop.

Heathers and Brown emailed Mears, who was the lead author on the paper. They asked to see the data set and explained their concerns. Mears replied that he would look into it but didn’t send the data and didn’t respond to subsequent emails. (Mears also didn’t reply to interview requests for this story.) Heathers emailed with Stewart about setting up a phone call but that conversation never took place.

Then came the “John Smith” email. That email pointed to apparent problems in four more papers. “There seem to be irregularities in the data and findings in five articles that you published together with two surveys,” the email said. Over nine pages, “Smith” summarized those irregularities and included links to highlighted excerpts and charts. The email is a dry, just-the-facts account of the situation, but those facts are troubling. In a later message sent to Florida State administrators, including the director of research compliance at the university’s Office of Research, “Smith” again explained what he’d found, calling it a “case of probable research misconduct by Eric Stewart.” Stewart replied to that email, promising to investigate the claims. “And if there in fact are problems,” he wrote, “we will address them.”

In a text message, Stewart told Pickett about the email sent to the research office and how it would very likely trigger a university investigation (that investigation, according to a Florida State spokesman, is underway).

“That’s really concerning, man,” Pickett wrote. “I’m very worried about all of this.”

Stewart was apologetic: “Sorry for all this shit man.”

“It’s the new academic life, I guess,” Pickett replied. “A lot of post-publication scrutiny.”

“This feels like more than that but yeah I got what you are saying,” wrote Stewart. “This is the new academic life.”

Pickett also reminded Stewart to send him the data. In the meantime, though, Pickett decided to do some poking around of his own. After looking through some of his files left over from his Florida State days, Pickett discovered that the now-defunct organization that conducted the survey for their 2011 paper had interviewed only 500 people, not the 1,184 reported in the paper. What about those other 684 people? Where had they come from? Stewart, who did eventually send Pickett that original data, first said that somehow “duplicates” had been included. In other words, responses had been mistakenly counted more than once. Then he later said that 425 respondents from a second survey should have been included, but for some reason were not — despite there being no mention of a second survey in the paper, and even though the total still wouldn’t equal 1,184.

This only made Pickett more suspicious. Why did the explanations keep shifting? Why wasn’t Stewart sharing everything, including the second survey? Why did nothing add up?

Even if the question of how many people were surveyed, and by whom, could be answered, other contradictions remained. For example, the article reported that respondents came from 91 counties, but it turns out that the real number, upon closer inspection, was 326. Stewart explained that the 91 listed counties had been misreported; they were actually “county clusters.” But after studying a map, Pickett determined that you couldn’t come up with that number of county clusters, and Stewart wouldn’t explain his math. What’s more, effects reported in the paper as positive were in fact negative. There were also missing values in the surveys, which happens when a survey respondent declines to reveal a particular fact, like salary or age. That, in itself, was not unusual, but Pickett found that the missing numbers, present in 208 of the 500 interviews, had been filled in with imputed values — that is, essentially educated guesses. The paper made no mention of these imputed values.

The paper was so riddled with errors and misleading information that it was hard to see how any of its conclusions could be trusted. It was also hard to see how these blunders could be chalked up to simple sloppiness.

Pickett had lost patience with Stewart. In June, he sent a letter to the editors of Criminology, the journal that had published their paper, which Pickett later posted online as a pre-print and which was first noted by Retraction Watch. Pickett concluded in the letter that “the sample was not just duplicated in the analysis for the published article; the data were also altered, whether intentionally or unintentionally, and those alterations produced the article’s main findings.”

In short, the findings in the paper — the one that Stewart had so generously invited Pickett to be a co-author on at a crucial moment in his career — were not only meaningless but might well be fabricated.

The lead author of that paper is Brian Johnson, a professor of criminology and criminal justice at the University of Maryland and also a co-editor of Criminology. When Pickett first emailed to tell him that Stewart had seemingly included hundreds of duplicates in the data, Johnson realized this was bad news. “That’s a huge fucking mistake. Man, what a shitshow,” Johnson wrote in an email to Pickett. “Obviously we are going to have to retract the article or at a minimum update it with new data and findings.”

In an email to The Chronicle, Johnson wrote that he “wasn’t involved in the data collection so am not able to provide you with details,” though he pushed back at suggestions that the paper might be partially based on fictional results: “I have absolutely no reason to doubt that the surveys are real and I do not think other people making accusations have any evidence of that either.”

The editors of Criminology have since issued a statement in response to Pickett’s letter, saying that they have “asked the authors of the 2011 paper to prepare a response to these concerns.” (Because he’s an author on the paper, Johnson has recused himself from involvement in any review.) So far no response has been posted.

But in an interview, the editor in chief of Criminology, David McDowall, seemed less than eager about getting to the bottom of what might be wrong with this particular paper. He confirmed that he had seen the letter Pickett sent but says that he “didn’t read it in great depth.” As for the possibility of retraction, which Pickett has requested, McDowall was dubious about the concept. “I don’t even quite know what retraction is,” McDowall says. “I imagine that it could occur. I would think there would be legalistic implications.”

What seemed to disturb McDowall more than possible errors or fabrication was Pickett’s motivation in questioning the data in the first place. McDowall and Pickett have a cordial, if not close, professional relationship. McDowall has been a professor in Albany’s School of Criminal Justice since 1996 and his office is just down the hall from Pickett’s. Pickett recently sat in on a class that McDowall taught and says he found it illuminating. But McDowall’s take on the dispute is more critical of Pickett than Stewart. “It seems to me that it’s pretty hostile for Justin to start making these claims,” McDowall says. Pickett says “negative things” about Stewart, according to McDowall, although he couldn’t recall specific statements. “I think he doesn’t like Eric personally and wants to ruin him and make him lose his job,” McDowall says.

It’s a remarkable accusation, in part because McDowall has a primary role in deciding how the field’s flagship journal will handle the allegations. Pickett says the only negative things he’s mentioned involving Stewart have had to do with the data discrepancies. He hoped initially that Stewart would be able to provide reasonable answers to the questions that had been raised, a correction or retraction could be issued, and then everyone could move on. “I’m not out to get Eric,” says Pickett. “I said, ‘If you’ll give me the data and it checks out I’ll defend you.’”

Still, McDowall doesn’t think Pickett handled the situation appropriately. McDowall is no fan of the move toward more scrutiny in the social sciences, which he sees as overly aggressive. In social psychology, for instance, sizable swaths of the literature are now viewed as suspect after much-cited papers failed to replicate. A number of prominent researchers have seen their reputations crumble.

Meanwhile, sociology — of which criminology is a part — has mostly been spared. “I think criminology is maybe a little behind other disciplines because we haven’t adopted the blood sport of ruining other people’s careers,” he says. When the journal has run into similar issues, the editors have dealt with it in what McDowall believes is a more humane manner. “This is not the first time that papers were published in the journal that were complete gibberish,” he says.

McDowall remembers one such paper that was published years ago in Criminology. After it appeared, someone pointed out that it was in fact gibberish and fellow criminologists took note. But there was no investigation or retraction or social-media outcry. McDowall declined to reveal the name of the paper or its author, who he says is still a professor at a major university. “I think, to me, that was a reasonable outcome,” he says. “The point is that there’s no blood-sport aspect to it.”

When told about McDowall’s theory that Pickett was driven by personal animus toward Stewart, Shawn Bushway, a professor of public administration and policy at Albany who specializes in criminal justice and has looked closely at the evidence, responded, “Can I laugh now?” He calls Pickett an “intense, serious guy” and notes that Pickett published a paper last year on data fraud and selective reporting. Bushway was also surprised by McDowall’s take on how such allegations should be addressed. “It’s a damning indictment of criminology,” he says. “Eric is a nice guy and this sucks but at the end of the day, if we claim to be a science, we have to take this seriously.”

That’s what Pickett believes he’s doing — taking this seriously — though he knows that by doing so he’s alienated senior colleagues like McDowall and that there may be more repercussions to come. Pickett is a prolific scholar, but will his papers get accepted as readily in the future? Will he be welcomed at conferences? What are the costs of being known as a troublemaker? Pickett’s worried about all of that, and he’s had more than a few restless nights in the last few months mulling what this means for his own future in the field.

A phone call in late August from a police officer only added to that anxiety. Someone at Florida State — the officer wouldn’t say who — had accused him of making a threat. The accusation was prompted by an email Pickett sent in June to a number of co-authors suggesting that “if you haven’t yet, you may want to request a copy of the data from the article you are on, and examine it yourself.” One of the co-authors, Eric Baumer, who is head of the department of sociology and criminology at Penn State, wrote back and said that he had spoken to Stewart “about each of the matters you mentioned” and that they needed to give him time to respond. Another co-author, Patricia Warren-Hightower, an associate professor of criminology and criminal justice at Florida State, chimed in: “I too choose to give Eric Stewart time to work through all the issues,” she wrote. (Baumer declined to comment, and Warren-Hightower did not respond to interview requests.)

Taken aback at what struck him as a critical tone from Baumer and Warren-Hightower, Pickett replied: “Let me stop this before it turns into an assault.” He was only notifying them out of professional courtesy, he wrote. “You can do whatever you want with this information. I don’t care.” The first line of his email — by which, Pickett says, he meant “let me stop this before it turns into an assault on me” — was apparently interpreted by someone on the email chain as a threat. The police officer Pickett spoke to said the emails had been reviewed and no further action would be taken.

Pickett doesn’t hold out much hope that Stewart will hand over all the data, or reveal who conducted the mysterious survey, or offer a persuasive explanation for the multitude of errors. He’s also resigned to the possibility that the powers-that-be in criminology may view his critique of the flawed and possibly fabricated research, rather than the research itself, as the real problem. So at the moment he’s bracing himself for more backlash and trying to take the long view. “If the field ignores this and everybody hates me, I at least believe the trend is toward transparency,” he says. “I think in 15 years we’re going to look back and say, ‘I can’t believe that this happened.’”