> Skip to content
FEATURED:
  • The Evolution of Race in Admissions
Sign In
  • News
  • Advice
  • The Review
  • Data
  • Current Issue
  • Virtual Events
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Career Resources
    • Find a Job
    • Post a Job
    • Career Resources
Sign In
  • News
  • Advice
  • The Review
  • Data
  • Current Issue
  • Virtual Events
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Career Resources
    • Find a Job
    • Post a Job
    • Career Resources
  • News
  • Advice
  • The Review
  • Data
  • Current Issue
  • Virtual Events
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Career Resources
    • Find a Job
    • Post a Job
    • Career Resources
Sign In
ADVERTISEMENT
News
  • Twitter
  • LinkedIn
  • Show more sharing options
Share
  • Twitter
  • LinkedIn
  • Facebook
  • Email
  • Copy Link URLCopied!
  • Print

Journal Publishers Rethink a Research Mainstay: Peer Review

By  Paul Voosen
October 11, 2015
Ulrich Pöschl helped found an atmospheric-sciences journal that publishes  papers, posts comments, and then settles on a final version of the work.
Franz Pfluegl
Ulrich Pöschl helped found an atmospheric-sciences journal that publishes papers, posts comments, and then settles on a final version of the work.

The peer-review system, a bedrock of modern science, is under a tremendous strain.

The number of published scientific papers has been growing at a rate of 5 percent a year. Facing cutthroat job markets, researchers slice their work into a panoply of studies — salami science, some call it — to bolster their appearance in metrics employed by tenure-and-promotion committees. With each new paper, requests for peer review ripple outward; the ripples are so aggregated that it now feels, especially to experienced scientists, like a daily tsunami.

We’re sorry. Something went wrong.

We are unable to fully display the content of this page.

The most likely cause of this is a content blocker on your computer or network. Please make sure your computer, VPN, or network allows javascript and allows content to be delivered from c950.chronicle.com and chronicle.blueconic.net.

Once javascript and access to those URLs are allowed, please refresh this page. You may then be asked to log in, create an account if you don't already have one, or subscribe.

If you continue to experience issues, contact us at 202-466-1032 or help@chronicle.com

The peer-review system, a bedrock of modern science, is under a tremendous strain.

The number of published scientific papers has been growing at a rate of 5 percent a year. Facing cutthroat job markets, researchers slice their work into a panoply of studies — salami science, some call it — to bolster their appearance in metrics employed by tenure-and-promotion committees. With each new paper, requests for peer review ripple outward; the ripples are so aggregated that it now feels, especially to experienced scientists, like a daily tsunami.

It’s becoming harder to give studies the attention they deserve, scientists say. Backlogs build. At prestigious journals, it’s not uncommon for submissions to sit for two years, going through four rounds of revision. Careers stall waiting for acceptance or, more crushing, rejection.

Editors have ceded ever more independence to expert reviewers, who in turn can exhibit a host of problems. They show preference to papers showing positive results, or they’re biased against authors from backgrounds unlike their own. They may prefer their own paradigms and stifle innovation. According to statistical models, their opinions are little better than chance at correctly evaluating a study for its scientific soundness.

Some propose radical transparency or other changes. And some want to do away with the system.

ADVERTISEMENT

Many researchers have had enough. Over the past few years, they have sought to repair, replace, or revolutionize the practice of peer review. Their methods vary. Some propose radical transparency. Some seek to decouple review from journals. Some propose crediting scientists for their review work. And some propose doing away with the system.

Much of this work was highlighted last month, when a small group of publishers held the first Peer Review Week, via online media, to promote the benefits of and debate changes in the existing system.

But as these reformers celebrate their advances, they must be wary. To many researchers, peer review has more benefits than flaws, and the public certainly expects scientific work to be vetted in some manner.

“Does it have problems? Absolutely,” says John R. Inglis, executive director of the Cold Spring Harbor Laboratory Press, about peer review. “Does it have challenges? Absolutely. But I’m not prepared to believe it’s a battlefield of moral bankruptcy.”

Judging for Novelty

Established after World War II, the modern peer-review system proved valuable to the growth of science. Its anonymity allowed referees to criticize without fear of retaliation. Limited interaction among reviewers prevented one voice from bullying the conversation. It added prestige to studies. And it selected, especially at high-profile journals, studies deserving of notice.

ADVERTISEMENT

Though often not thought of this way, the mission of several open-access journals, like PLOS ONE, to publish any study deemed scientifically sound has already resulted in one great reform. It made clear that peer review does not, intrinsically, have to judge for novelty or significance. Yet such judgment remains an essential part of the process at many journals.

Retaining such criteria results in a glaring inefficiency, however: Many studies are reviewed and found sound at prestigious journals, only to be rejected for their fit or significance.

It’s a problem that’s not gone unnoticed at the major publishers, many of which have started “cascade” systems that give scientists the option of passing along a study first submitted to, say, Philosophical Transactions B, to a less prestigious journal, Royal Society Open Science, with peer review intact.

If scientists don’t want to end up in the PLOS ONE clones of publishers, however, several services, including Peerage of Science, Axios Review, and Rubriq, have begun to offer peer review independent of any journal. Peerage relies on voluntary help, while Axios and Rubriq charge fees.

The mission of Axios, which began in 2013 and works only in the disciplines of ecology and evolution, is to act as an agent for researchers, providing, for a $250 fee, traditional peer review and helping to place the paper at the journal most likely to take it. That helps to avoid the cycle of review and rejection, says Tim Vines, the service’s founder. The researcher chooses four journals, and then Axios’s referees, selected from its editorial board, assess the study and recommend the journal most likely to take it.

ADVERTISEMENT

Axios, in effect, lowers the rejection rate of science. If you do that, you lower the overall cost of science, says Mr. Vines. “Yes, we do ask authors to pay. But on the whole, we reduce the overall expense of publishing.”

Radical Questions

Rather than simply clean up inefficiencies, some researchers are asking radical questions about peer review: Why does so much of our scientific labor remain invisible to promotion committees? Do studies need to be peer reviewed at all before they’re published?

For generations, scientists have accepted the burden of peer review as a necessary part of the profession, an obligation that provides intangible rewards, in terms of reputation, and sometimes tangible benefits: membership on a prestigious review board. But as science has become dominated by metrics like impact factor — which quantifies the influence of a journal by its citation count — a new generation of scientists finds the recognition of doing referee work vanishingly small.

“All in all, it seems pretty dreadful,” says Andrew R.H. Preston, a physicist who helped start Publons, one of several services seeking to give researchers scientific credit for their reviews.

Started two years ago, Publons has prompted controversy. It allows referees to upload reviews, without journal approval, that could prove embarrassing to authors. It has raised ethical questions that researchers have only begun to ponder, including basic things, like just who owns peer review, anyway?

ADVERTISEMENT

Despite such qualms, the drive toward quantifying and crediting peer reviewers is real. Publons just announced a trial partnership with the top biology journal of the Royal Society, in Britain.

While Mr. Preston may seem radical to some, he is still a staunch supporter of the notion that every piece of research ought to receive some sort of peer review.

Brandon Stell, on the other hand, prefers to see that system go.

Mr. Stell is one of the founders of PubPeer, a website that allows anyone to anonymously comment on a published research article. The site has become known for its role in exposing several scientific frauds in the past few years; until this summer, Mr. Stell, an American neuroscientist based in France, operated the site anonymously.

“We would like to move away from journals,” which are a medium doomed to fail, he says. “I suspect one day journals will become unimportant, so why invest our time in developing new ways of reviewing for journals?”

ADVERTISEMENT

Mr. Stell would like to see every study published immediately, with review coming afterward — postpublication review. Granted, such a system may not ensure that every paper gets reviewed, as they do today: Right now, most papers flagged in PubPeer have one or two comments; many don’t have any comments at all.

‘What Scientists Say’

Still, there’s no law that every paper must face peer review. It’s possible to generate new traditions and standards. “If postpublication peer review becomes the norm, it won’t matter where the paper is published,” Mr. Stell says. “What will matter is what scientists say about the paper.”

If any of the reformers’ ambitions grow too fast, though, they need only look to the experience of Ulrich Pöschl, editor at Atmospheric Chemistry and Physics, which has pioneered an open, two-stage system of peer review.

With several colleagues, Mr. Pöschl began ACP in 2001 on a mission for transparency. He saw value in anonymity, but he also saw that veil being used to file “crap referee reports,” he says. Similarly, researchers would file questionable papers, confident that their studies, if denied, would be quietly returned. And it seemed crazy that reviewers’ reports, which often contained valuable insights, never joined the scientific literature.

The process used by ACP has two stages: Papers, after undergoing a cursory check, are published in the journal’s discussion section. All credible scientific comments are welcomed, anonymous or not. The invited reviewers’ comments are posted, along with the author’s responses. An editor then makes the call on a final, revised version of the paper.

ADVERTISEMENT

Other journals have eschewed a two-stage system but began doing away with reviewer anonymity. While that step is still uncommon, several prominent journals now provide reviewers’ names and comments after publication. Other journals, like Nature, have moved in the opposite direction, offering optional double-blind review, in which neither reviewer nor author knows the identity of the other.

Few open-review journals have adopted the model of Atmospheric Chemistry and Physics, even though it draws top researchers in the field. For example, James E. Hansen, the famed climate scientist and activist, began publishing there several years ago.

His controversial draft paper predicting catastrophic sea-level rise in this century cast a public eye on ACP for the first time this summer, quickly becoming the journal’s most commented-on submission.

While accompanied by jeers from climate-change contrarians that, at least according to one referee, interfered with the review process, the paper has moved forward. Mr. Pöschl is dismayed that more experts didn’t weigh in on Mr. Hansen’s study, and he expects to clarify that the journal does not welcome “unscientifically sound comments.”

But over all, he’s happy with how the system has survived, if disappointed that it remains an exception. “The basic process of putting out a discussion paper for public scrutiny,” he says, “I haven’t seen a better concept yet.”

ADVERTISEMENT

Paul Voosen is a senior reporter covering the sciences. Write him at paul.voosen@chronicle.com; follow him on Twitter @voooos; or see past work at voosen.me.

A version of this article appeared in the October 16, 2015, issue.
We welcome your thoughts and questions about this article. Please email the editors or submit a letter for publication.
Scholarship & Research
Paul Voosen
Paul Voosen was a Chronicle reporter. His stories have also appeared in National Geographic, Scientific American, and Greenwire, with reprints in The New York Times.
ADVERTISEMENT
ADVERTISEMENT
  • Explore
    • Get Newsletters
    • Letters
    • Free Reports and Guides
    • Blogs
    • Virtual Events
    • Chronicle Store
    • Find a Job
    Explore
    • Get Newsletters
    • Letters
    • Free Reports and Guides
    • Blogs
    • Virtual Events
    • Chronicle Store
    • Find a Job
  • The Chronicle
    • About Us
    • DEI Commitment Statement
    • Write for Us
    • Talk to Us
    • Work at The Chronicle
    • User Agreement
    • Privacy Policy
    • California Privacy Policy
    • Site Map
    • Accessibility Statement
    The Chronicle
    • About Us
    • DEI Commitment Statement
    • Write for Us
    • Talk to Us
    • Work at The Chronicle
    • User Agreement
    • Privacy Policy
    • California Privacy Policy
    • Site Map
    • Accessibility Statement
  • Customer Assistance
    • Contact Us
    • Advertise With Us
    • Post a Job
    • Advertising Terms and Conditions
    • Reprints & Permissions
    • Do Not Sell My Personal Information
    Customer Assistance
    • Contact Us
    • Advertise With Us
    • Post a Job
    • Advertising Terms and Conditions
    • Reprints & Permissions
    • Do Not Sell My Personal Information
  • Subscribe
    • Individual Subscriptions
    • Institutional Subscriptions
    • Subscription & Account FAQ
    • Manage Newsletters
    • Manage Your Account
    Subscribe
    • Individual Subscriptions
    • Institutional Subscriptions
    • Subscription & Account FAQ
    • Manage Newsletters
    • Manage Your Account
1255 23rd Street, N.W. Washington, D.C. 20037
© 2023 The Chronicle of Higher Education
  • twitter
  • instagram
  • youtube
  • facebook
  • linkedin