The peer-review system, a bedrock of modern science, is under a tremendous strain.
The number of published scientific papers has been growing at a rate of 5 percent a year. Facing cutthroat job markets, researchers slice their work into a panoply of studies — salami science, some call it — to bolster their appearance in metrics employed by tenure-and-promotion committees. With each new paper, requests for peer review ripple outward; the ripples are so aggregated that it now feels, especially to experienced scientists, like a daily tsunami.
It’s becoming harder to give studies the attention they deserve, scientists say. Backlogs build. At prestigious journals, it’s not uncommon for submissions to sit for two years, going through four rounds of revision. Careers stall waiting for acceptance or, more crushing, rejection.
Editors have ceded ever more independence to expert reviewers, who in turn can exhibit a host of problems. They show preference to papers showing positive results, or they’re biased against authors from backgrounds unlike their own. They may prefer their own paradigms and stifle innovation. According to statistical models, their opinions are little better than chance at correctly evaluating a study for its scientific soundness.
Some propose radical transparency or other changes. And some want to do away with the system.
Many researchers have had enough. Over the past few years, they have sought to repair, replace, or revolutionize the practice of peer review. Their methods vary. Some propose radical transparency. Some seek to decouple review from journals. Some propose crediting scientists for their review work. And some propose doing away with the system.
Much of this work was highlighted last month, when a small group of publishers held the first Peer Review Week, via online media, to promote the benefits of and debate changes in the existing system.
But as these reformers celebrate their advances, they must be wary. To many researchers, peer review has more benefits than flaws, and the public certainly expects scientific work to be vetted in some manner.
“Does it have problems? Absolutely,” says John R. Inglis, executive director of the Cold Spring Harbor Laboratory Press, about peer review. “Does it have challenges? Absolutely. But I’m not prepared to believe it’s a battlefield of moral bankruptcy.”
Judging for Novelty
Established after World War II, the modern peer-review system proved valuable to the growth of science. Its anonymity allowed referees to criticize without fear of retaliation. Limited interaction among reviewers prevented one voice from bullying the conversation. It added prestige to studies. And it selected, especially at high-profile journals, studies deserving of notice.
Though often not thought of this way, the mission of several open-access journals, like PLOS ONE, to publish any study deemed scientifically sound has already resulted in one great reform. It made clear that peer review does not, intrinsically, have to judge for novelty or significance. Yet such judgment remains an essential part of the process at many journals.
Retaining such criteria results in a glaring inefficiency, however: Many studies are reviewed and found sound at prestigious journals, only to be rejected for their fit or significance.
It’s a problem that’s not gone unnoticed at the major publishers, many of which have started “cascade” systems that give scientists the option of passing along a study first submitted to, say, Philosophical Transactions B, to a less prestigious journal, Royal Society Open Science, with peer review intact.
If scientists don’t want to end up in the PLOS ONE clones of publishers, however, several services, including Peerage of Science, Axios Review, and Rubriq, have begun to offer peer review independent of any journal. Peerage relies on voluntary help, while Axios and Rubriq charge fees.
The mission of Axios, which began in 2013 and works only in the disciplines of ecology and evolution, is to act as an agent for researchers, providing, for a $250 fee, traditional peer review and helping to place the paper at the journal most likely to take it. That helps to avoid the cycle of review and rejection, says Tim Vines, the service’s founder. The researcher chooses four journals, and then Axios’s referees, selected from its editorial board, assess the study and recommend the journal most likely to take it.
Axios, in effect, lowers the rejection rate of science. If you do that, you lower the overall cost of science, says Mr. Vines. “Yes, we do ask authors to pay. But on the whole, we reduce the overall expense of publishing.”
Radical Questions
Rather than simply clean up inefficiencies, some researchers are asking radical questions about peer review: Why does so much of our scientific labor remain invisible to promotion committees? Do studies need to be peer reviewed at all before they’re published?
For generations, scientists have accepted the burden of peer review as a necessary part of the profession, an obligation that provides intangible rewards, in terms of reputation, and sometimes tangible benefits: membership on a prestigious review board. But as science has become dominated by metrics like impact factor — which quantifies the influence of a journal by its citation count — a new generation of scientists finds the recognition of doing referee work vanishingly small.
“All in all, it seems pretty dreadful,” says Andrew R.H. Preston, a physicist who helped start Publons, one of several services seeking to give researchers scientific credit for their reviews.
Started two years ago, Publons has prompted controversy. It allows referees to upload reviews, without journal approval, that could prove embarrassing to authors. It has raised ethical questions that researchers have only begun to ponder, including basic things, like just who owns peer review, anyway?
Despite such qualms, the drive toward quantifying and crediting peer reviewers is real. Publons just announced a trial partnership with the top biology journal of the Royal Society, in Britain.
While Mr. Preston may seem radical to some, he is still a staunch supporter of the notion that every piece of research ought to receive some sort of peer review.
Brandon Stell, on the other hand, prefers to see that system go.
Mr. Stell is one of the founders of PubPeer, a website that allows anyone to anonymously comment on a published research article. The site has become known for its role in exposing several scientific frauds in the past few years; until this summer, Mr. Stell, an American neuroscientist based in France, operated the site anonymously.
“We would like to move away from journals,” which are a medium doomed to fail, he says. “I suspect one day journals will become unimportant, so why invest our time in developing new ways of reviewing for journals?”
Mr. Stell would like to see every study published immediately, with review coming afterward — postpublication review. Granted, such a system may not ensure that every paper gets reviewed, as they do today: Right now, most papers flagged in PubPeer have one or two comments; many don’t have any comments at all.
‘What Scientists Say’
Still, there’s no law that every paper must face peer review. It’s possible to generate new traditions and standards. “If postpublication peer review becomes the norm, it won’t matter where the paper is published,” Mr. Stell says. “What will matter is what scientists say about the paper.”
If any of the reformers’ ambitions grow too fast, though, they need only look to the experience of Ulrich Pöschl, editor at Atmospheric Chemistry and Physics, which has pioneered an open, two-stage system of peer review.
With several colleagues, Mr. Pöschl began ACP in 2001 on a mission for transparency. He saw value in anonymity, but he also saw that veil being used to file “crap referee reports,” he says. Similarly, researchers would file questionable papers, confident that their studies, if denied, would be quietly returned. And it seemed crazy that reviewers’ reports, which often contained valuable insights, never joined the scientific literature.
The process used by ACP has two stages: Papers, after undergoing a cursory check, are published in the journal’s discussion section. All credible scientific comments are welcomed, anonymous or not. The invited reviewers’ comments are posted, along with the author’s responses. An editor then makes the call on a final, revised version of the paper.
Other journals have eschewed a two-stage system but began doing away with reviewer anonymity. While that step is still uncommon, several prominent journals now provide reviewers’ names and comments after publication. Other journals, like Nature, have moved in the opposite direction, offering optional double-blind review, in which neither reviewer nor author knows the identity of the other.
Few open-review journals have adopted the model of Atmospheric Chemistry and Physics, even though it draws top researchers in the field. For example, James E. Hansen, the famed climate scientist and activist, began publishing there several years ago.
His controversial draft paper predicting catastrophic sea-level rise in this century cast a public eye on ACP for the first time this summer, quickly becoming the journal’s most commented-on submission.
While accompanied by jeers from climate-change contrarians that, at least according to one referee, interfered with the review process, the paper has moved forward. Mr. Pöschl is dismayed that more experts didn’t weigh in on Mr. Hansen’s study, and he expects to clarify that the journal does not welcome “unscientifically sound comments.”
But over all, he’s happy with how the system has survived, if disappointed that it remains an exception. “The basic process of putting out a discussion paper for public scrutiny,” he says, “I haven’t seen a better concept yet.”
Paul Voosen is a senior reporter covering the sciences. Write him at paul.voosen@chronicle.com; follow him on Twitter @voooos; or see past work at voosen.me.
