In the Age of Trump, Scientists See Reproducibility as Risky Business

Brent Humphreys

Earlier this month, just a short walk from the White House, Richard M. Shiffrin led dozens of his academic colleagues through a three-day examination of the trustworthiness of science.

He’s not totally sure it was a good idea.

The colloquium that Mr. Shiffrin and two colleagues organized at the National Academy of Sciences headquarters was planned long before Donald J. Trump took up residence in the nearby executive mansion, capping a campaign that repeatedly challenged science and facts as basic tools of understanding.

The scientific community’s current focus on self-improvement, meanwhile, has been underway for more than a decade. A landmark moment came in 2005 when a study by John P.A. Ioannidis, a professor of health research and policy at Stanford University, promised to answer “Why Most Published Research Findings Are False.”

The paper was among the early shots fired in a continuing war over whether scientific experiments can be — or should be — reproducible. Mr. Shiffrin, a professor of psychological and brain sciences at Indiana University at Bloomington, said he recognizes the now-very-clear risk of spotlighting reproducibility while a U.S. president and a like-minded Congress appear eager to question the fundamental tenets of science.

“There are a lot of dangers here,” Mr. Shiffrin acknowledged at the conclusion of the conference. In the current political climate, he said, “the fact that we’re trying to fix things and make things better may not be the message that gets through at the other end of the pipeline.”

So far, there’s little direct indication that the Trump administration or congressional leaders known for attacking scientific research on climate change and human health are looking to exploit reproducibility campaigns as a political opportunity.

But there are some real risks for researchers striving to improve their professional standards, especially as they prepare for a nationwide series of protest marches next month to make the argument that the administration’s generally skeptical approach to science is unwarranted.

One area of concern is the National Academy itself. As part of a wide-ranging science-policy bill approved by Congress in January and signed by President Obama, the federally chartered, independent science-advisory body is being asked to conduct a broad study of the reproducibility and replicability of research and data.

While the National Academy can generally be trusted to give an even-handed assessment, Mr. Shiffrin said, its report could give lawmakers political ammunition to drastically cut funding for science along the lines favored by the Trump administration.

Another concern involves a series of recent academic studies making the case that much of scientific research is not, by some definitions, truly replicable. The latest major study, announced in January, attempted to repeat five of the most-cited cancer-biology papers published between 2010 to 2012. It meaningfully reproduced none of them.

Public Skepticism

That replication effort was led by the Center for Open Science, which was formed by a team of University of Virginia researchers and financed by the Laura and John Arnold Foundation. The foundation is spending tens of millions of dollars a year on such work as an attempt to convince researchers of the need to pay better attention to matters of accuracy and reliability.

The foundation’s co-chairman, the billionaire former hedge-fund manager John D. Arnold, said he’s not worried that his work might lead to reduced federal spending on science. The first priority, Mr. Arnold said, is to improve “the methods of science.” Once that happens, he said, it will be easier to judge whether federal money is being spent efficiently.

Right now, Mr. Arnold said, the incentives for universities, researchers, and journals to pursue and promote “novel results” rather than provable truths lead to news articles that offer conflicting expert advice on matters such as weight loss and causes of cancer. Those contradictions, he said, are a leading driver of public skepticism toward science.

Mr. Arnold cited polling by the Kaiser Family Foundation that found about a quarter of Americans have trouble paying for all their medications. Given such realities, he said, it’s not automatic that in a finite federal budget, more people would be helped by spending more money on medical research. “Maybe it should go to providing more access to prescriptions for medications that we’ve already created rather than putting that to research to come up with new ones that we may or may not be able to afford as a society,” he said.

Over three days, Mr. Shiffrin’s colloquium made clear the need to fix problems of trust and reproducibility in science. Kathleen Hall Jamieson, a professor of communication at the University of Pennsylvania, told the group that scientists often magnify their problems by using words that may have a clear meaning to them but have an ambiguous or even different meaning to the general public. She cited the confusion caused last year when the U.S. Centers for Disease Control and Prevention said measles had been “eliminated.” Among scientists, “eliminated” simply means that a country is no longer producing new cases, while the complete absence of the disease is described by the term “eradicated.” To the public, those distinctions are opaque.

C.K. Gunsalus, an emerita professor of business at the University of Illinois at Urbana-Champaign and director of the National Center for Professional and Research Ethics, said universities knowingly support a “star system” that spares faculty from penalties for actions that students would be suspended for. “We’re not very serious about fixing it,” she said, “because it serves us well.”

‘Unintended Consequences’

Although he is wary that the work of reproducibility watchdogs could be exploited politically, Mr. Shiffrin said there’s no way to halt or even pause their activities. “There’s too many scientists who have been pursuing this for too long,” he said. “There is too much impetus behind this whole thing.”

Mr. Ioannidis, whose name has become synonymous with reproducibility, has himself has become a subject of concern. Multiple news reports in recent weeks described the Trump administration as considering him to head the National Institutes of Health. Mr. Ioannidis isn’t seeking the job, and said in an interview that he has “no tolerance for any overt or covert anti-science agendas.”

But Mr. Shiffrin said he senses that some reproducibility advocates “are making a little living out of doing it, and, you know, they’re getting certain fame and glory, let’s say, by pursuing this.” It’s worrisome, he said, when “people are arguing that most of science is reporting results that aren’t true, and things like that, which isn’t the case.”

Asked if he meant Mr. Ioannidis and his well-known 2005 paper, Mr. Shiffrin said he would not cite individuals. “I don’t think that their intent is to harm science — their intent is probably to help,” he said. “But there’s unintended consequences of many of these things.”

For his part, Mr. Ioannidis said he has “tremendous respect” for Mr. Shiffrin and has “given a lot of thought to the potential harm that he describes.”

But Mr. Ioannidis added: “I find this ‘keep it quiet’ thinking potentially detrimental to science and its cause.”

Paul Basken covers university research and its intersection with government policy. He can be found on Twitter @pbasken, or reached by email at paul.basken@chronicle.com.

Correction (3/21/2017, 11:05 a.m.): This article, which originally indicated that Richard Shiffrin led a colloquium at the National Academy of Sciences, has been updated to clarify that he was one of three organizers.