A group of leading medical-journal editors, convened by the National Institutes of Health, this week endorsed a set of guidelines intended to tackle the widespread problem of scientific findings that cannot be replicated.
About 40 editors, representing journals that include Science and Nature, reached a “general agreement” about what they must accept as their responsibility for ensuring the reproducibility of their published findings, the NIH director, Francis S. Collins, said on Thursday.
Dr. Collins, addressing a semiannual session of his advisory committee at the agency’s headquarters, in Bethesda, Md., gave only limited details of the agreement, and the NIH did not release a copy of the text. Officials at Science declined to authorize a release, saying the principles were still regarded as a draft.
As one element, however, Dr. Collins said the journals discussed the need to publish articles that identify reproducibility problems with studies they previously published.
If widely adopted across the industry, the changes would “substantially” aid the campaign to make published scientific findings more reliable, he said.
The problem of so-called irreproducibility in science has been festering for years. A landmark moment came in 2005, when John P.A. Ioannidis, a professor of medicine at Stanford University, published an analysis in PLOS Medicine concluding that most published research findings appeared to be substantially false.
Dr. Collins and his principal deputy director, Lawrence A. Tabak, wrote a commentary in Nature in January that cited a growing recognition “that the complex system for ensuring the reproducibility of biomedical research is failing and is in need of restructuring.”
More Steps Planned or Under Way
The meeting with journal editors was one of several actions under way at the NIH to improve the quality of the research the agency finances, as well as to improve equity in the process of carrying it out, Dr. Collins told the advisory panel.
The session with the journals will be followed next month by a similar meeting with representatives of the pharmaceutical industry, Dr. Collins said. NIH officials want to hear the industry’s advice for improving reproducibility in research, and to press the companies to be more transparent about scientific findings that are not publicly released because they did not achieve a desired result.
And within its own grant-application structure, the NIH is moving ahead with a new process designed to help scientists emphasize the quality, more than the quantity, of their past work. Under the new “biosketch” format, researchers will be permitted to describe up to five of their most significant contributions to science, the agency has said.
The NIH began a second round of testing the approach with grant applicants last month. The changes, as well as the meeting with journal editors, are designed to counter the current atmosphere of competitiveness that too often has both scientists and journal editors hurrying to publish papers without taking enough time to ensure the work is as thorough and accurate as possible, Dr. Collins said.
The problem is particularly evident in testing with animals that produces evidence used to shape testing in human beings, with clear implications for both cost and safety, Dr. Collins said.
Failure to replicate a study’s findings, by itself, doesn’t mean the scientists did anything wrong, said one advisory-panel member, Harlan M. Krumholz, a professor of medicine at Yale University. But the prevalence of the problem suggests a cultural attitude that the NIH is uniquely positioned to help fix, Dr. Krumholz said.
Also on Thursday the NIH’s Advisory Committee to the Director presented Dr. Collins with a series of recommendations for pushing forward the agency’s campaign to achieve greater racial balance among its grant recipients. In addition, NIH officials described for the panel their continuing work—which will include tests over the coming year involving grant applications—to strengthen overall fairness and impartiality in the NIH’s grant-review process.
