How do you change an entire scientific culture?
It may sound grandiose, but that is the loaded question now facing the National Institutes of Health, the federal agency that oversees and finances U.S. biomedical research.
While the public remains relatively unaware of the problem, it is now a truism in the scientific establishment that many preclinical biomedical studies, when subjected to additional scrutiny, turn out to be false. Many researchers believe that if scientists set out to reproduce preclinical work published over the past decade, a majority would fail. This, in short, is the reproducibility crisis.
The NIH, if it was at first reluctant to consider the problem, is now taking it seriously. Just over a year ago, the agency’s director, Francis S. Collins, and his chief deputy, Lawrence A. Tabak, announced actions the agency would take to improve the research it finances. Science needs to get its house in order, Dr. Collins said in a recent interview with The Chronicle.
“We can’t afford to waste resources and produce nonreproducible conclusions if there’s a better way,” he said.
The problem may be most visible in preclinical biomedicine and fields like psychology, but it’s by no means limited to them, Dr. Collins added. “No matter what your science is, there’s a risk here. If experimental methods are not well detailed and well planned, then stuff can happen.”
Science today is riven with perverse incentives:
- Researchers judge one another not by the quality of their science — who has time to read all that? — but by the pedigree of their journal publications.
- High-profile journals pursue flashy results, many of which won’t pan out on further scrutiny.
- Universities reward researchers on those publication records.
- Financing agencies, reliant on peer review, direct their grant money back toward those same winners.
- Graduate students, dependent on their advisers and neglected by their universities, receive minimal, ad hoc training on proper experimental design, believing the system of rewards is how it always has been and how it always will be.
But many senior researchers will oppose reform, said Arturo Casadevall, a professor at the Albert Einstein College of Medicine at Yeshiva University. “Scientists themselves are playing this game because once they succeed, the rewards are so great they basically force everyone to do it.”
The tight financing of the past decade has only made the problem worse, said Ferric C. Fang, a microbiology professor at the University of Washington who, like Dr. Casadevall, has called for changes in the system. “We’re ending up with an ecosystem that’s increasingly populated by predators,” he said.
Some researchers fear that “skeptics” who dispute valid scientific work on a host of hot-button issues will seize on the reproducibility crisis as a political weapon.
“This is a very serious problem,” said C. Glenn Begley, chief scientific officer of TetraLogic Pharmaceuticals, who, several years ago, published a prominent paper on how his former company, Amgen, had found it impossible to replicate more than 90 percent of potential cancer drugs.
“It really threatens the general feeling among the community in terms of how important science is,” he said.
The NIH’s response is wide- ranging. Its institutes are revising how they review grants, requiring far more data on experimental design, including validation of past findings that studies purport to build upon. The agency is financing new programs to teach first-year graduate students how to design experiments. It is pressing journals to raise their review standards. It is improving how researchers identify lab materials, and requiring balanced use of males and females in animal studies. And it is experimenting with different ways of financing research.
Those are all worthy steps, Dr. Casadevall said, and he applauds how quickly the NIH, once it understood the scale of the problem, has responded. But even the NIH is limited in how it can change science. “My concern is that the problem here is cultural,” he said, “and the fixes are procedural.”
An Inside Agent
If the NIH could be said to have an internal whistle-blower on the problems with preclinical research, Shai D. Silberberg would be a candidate for the role. A decade ago, Mr. Silberberg, a biophysicist, became a program director at the National Institute of Neurological Disorders and Stroke, and he was shocked to see what was considered high-quality science, especially in studies using animals. Many experiments supporting proposed clinical investigations used low sample sizes and were prone to bias.
As it happened, neurology researchers were ahead of the curve in pointing out those problems. Scientists in Australia and Scotland had begun documenting flaws in animal studies that had seemed to support promising, but failed, stroke treatments. A similar effort found parallel problems in spinal-cord research. And ALS researchers, seeking a cure for Lou Gehrig’s disease, went back and reproduced studies on more than 70 promising drugs. They found no real effects.
“Zero of those were replicable,” Dr. Collins said. “Zero. And a couple of them had already moved into human clinical trials, which is like — oh, man. You’ve started maybe even putting people at risk for exposure to a compound for which there is really no compelling data that it was going to help. This can’t be.”
Armed with such data, Mr. Silberberg made a case for change. He won over the institute’s director, Story C. Landis, who put her grant reviewers on notice that higher standards were expected. But it was clear to Ms. Landis that the problem wasn’t limited to neurology; all of NIH needed to change. That culminated in a 2012 meeting with editors from journals like Nature, Science, and Cell; financing agencies; and prominent reviewers. The first day, everyone blamed each other. The second day, they started talking solutions.
Though replication has now become an NIH-wide concern, Mr. Silberberg remains enmeshed in the agency’s efforts to improve transparency in detailing how studies are conducted. He gave 24 talks about the effort last year.
“I view this like being a Johnny Appleseed,” Mr. Silberberg said. “You’ve got to plant the seeds and let them grow.”
Training Young Scientists
The NIH has realized it has to get to scientists while they’re young. To that end, the agency has asked researchers to develop training modules for young scientists on proper experimental techniques, and will soon choose 25 of them to finance in the hope of creating a library of material, said Jon R. Lorsch, director of the NIH’s National Institute of General Medical Sciences.
The agency will also offer grants this year for universities willing to make more-radical shifts, such as changing how they teach first- and second-year graduate students to emphasize proper use of statistics and experimental design.
Indeed, more than any part of the scientific system, the universities have been ignoring the replication crisis, Dr. Begley said. “I don’t see the institutions taking this on seriously.”
Currently, the experimental rigor of a graduate student’s education is up to the lab where he or she resides, and few universities teach experimental design. It’s an apprenticeship, yet often the master of the shop is sequestered, filing grant applications. Dr. Begley predicted that the first university to tackle the problem would draw much interest from venture capitalists burned by patents that petered out. So far, that incentive hasn’t been enough.
“As long as universities think that the way for investigators to get money is to publish in Nature and Science, then that’s what they’re going to look for in investigators,” Mr. Silberberg said. “They want that money.”
Dr. Begley would like to see the NIH go further, requiring universities to ensure a minimum research standard among their employees. Even if that step cut in half the amount of research universities produced, if it increased the science’s quality by a commensurate amount, Dr. Begley would take it.
The NIH has also pushed scientific journals to raise their review standards for life-science studies. More than 100 journals have agreed to elevated principles, many of which stem from the neurology institute’s recommendations. Most of the journals are instituting checklists that will require submissions to run down and justify the decisions made in designing the study.
Checklists can prevent all kinds of human-scale problems, Dr. Casadevall said. “The next time you take a flight, look left as you’re boarding the plane. The pilots are doing a checklist.”
The institutes are also exploring better ways of identifying materials used in the lab: cell lines, antibodies, specialty chemicals. By some estimates, up to 30 percent of cell lines are mistakenly labeled.
“They get passed around from lab to lab,” Dr. Collins said. “There’s a little mix-up in someone’s incubator, and then, 10 years later, somebody’s been working on this cell line and it turns out it isn’t what they thought it was at all.”
The NIH is considering a focused investment in technology that would make it quicker and cheaper to check the validity of a cell.
While it makes those tweaks and more, the NIH has also been experimenting with different ways of financing research. Knowing that the unstable environment can drive researchers to cut corners or oversell work, the agency has tried systems that finance individual investigators, rather than projects. That may allow more time for careful science, not grantsmanship.
“In some ways, ultimately, that may be the most important thing we can do,” said Mr. Lorsch.
But perhaps the agency’s most radical step is buried in its education proposals. Not only does it want training on experimental methods; it wants lessons on the sociology of science. Why do promotion and tenure work like they do? Are there alternatives? Seeds must be planted. Students need to know that it doesn’t have to be this way.
“We want them questioning, wanting to fix what’s broken in the system,” Mr. Lorsch said. “If they don’t hear what the problems are, and don’t hear their professors reflecting on them, it’s going to be much harder to change things.”
Paul Voosen is a senior reporter covering the sciences. Write him at paul.voosen@chronicle.com; follow him on Twitter @voooos; or see past work at voosen.me.
