As a scientist, I appreciate the value of peer review in the process of publishing scientific papers. Peer reviewers have greatly improved my own work, identifying errors, pointing out areas that required further research, and informing me about prior research that I had overlooked. However, my recent examination of 60 empirical studies of peer review leads me to conclude that the process needs significant improvement.
Scientific journals use peer review for three primary purposes. The first is quality control, to eliminate major errors in papers. The second is fairness, to insure that authors receive equal treatment. The third is to promote innovation, by encouraging and publishing papers that contain new and useful findings.
But in spite of peer review, journals still publish many errors, do not treat all authors fairly, and are far more likely to publish research that supports earlier work than to publish groundbreaking new studies.
A peer-review process that rejects new ideas can do serious harm to scientific progress. Several changes can be made, however, to reduce bias against innovation and to address other problems in our current procedures.
Scientists now spend hundreds or even thousands of hours researching and writing a given paper. Their careers depend on how well they do this. But studies show that most reviewers, who are usually anonymous and whose reputations thus are not on the line, spend no more than six hours per review.
For example, in a survey of 255 reviewers for medical journals, Alfred Yankauer of the University of Massachusetts Medical Center found that the average review took 4.2 hours. And reviewers typically have less expertise than do authors in the specific subfield of research under consideration.
Despite that disparity of effort and expertise between authors and reviewers, the editors of many leading journals rely primarily on reviewers’ recommendations in deciding what to publish. Joel Kupfersmid and Donald Wonderly’s book, An Author’s Guide to Publishing Better Articles in Better Journals in the Behavioral Sciences (Clinical Psychology Publishing, 1994), concluded that papers that receive a single bad review have a low probability of being published by the journal that commissioned the review.
Advocates of the current peer-review process might claim that, despite the problems, the net result is the publication of high-quality papers. But is this really the case? Several types of studies indicate that peer review does not necessarily screen out errors in published papers. Some studies show that as many as a third of the citations of earlier research listed in authors’ footnotes and bibliographic references contain mistakes. Authors’ descriptions of the content of earlier published work are nearly as likely to involve errors.
In a study published in 1986 in The American Economic Review, William G. Dewald, Jerry G. Thursby, and Richard G. Anderson found that the authors of papers printed by the Journal of Money, Credit, and Banking from 1980 to 1984 commonly used incorrect data and made computing errors that significantly affected the authors’ conclusions. And William M. Epstein, of the University of Nevada’s School of Social Work, conducted an experiment in which he obtained reviews from 53 social-work journals of a reworked version of a previously published paper. This version deliberately omitted all information about the control group. Most of the reviewers failed to comment on the omission.
If peer-reviewed papers contain errors when they are published, are their conclusions still generally correct? Raymond Hubbard, a marketing professor at Drake University, has studied papers in such disciplines as economics, marketing, psychology, and sociology that attempted to replicate or extend research described in other papers. He concluded that almost half of the follow-up papers disagreed with at least one important finding in the original paper, and that another quarter disagreed in minor respects. Other studies are needed to determine if this rate of errors is typical; we do not know if authors are more or less likely to replicate studies that they think are flawed, or if editors are more or less likely to publish follow-up papers that demonstrate errors in earlier publications.
One goal of the peer-review process has been to treat all authors identically, yet some scientists claim that journals are more likely to accept papers by authors from prestigious institutions. And treating all authors alike -- perhaps by deciding to publish only papers that receive no negative reviews -- does not allow editors to take into consideration whether a scientist has made major research contributions in the past or has written other papers that contained errors.
Further, the current process often hurts the authors of innovative work, which is quite likely to receive negative reviews. Although journal reviewers claim in surveys that they value"newness,” scientists who try to publish path-breaking work believe that reviewers frequently are biased against them. Some research supports this perception. For instance, a survey of 140 eminent economists, published in the Journal of Economic Perspectives in 1994 by Joshua Gans and George Shepherd, who were graduate students in economics at Stanford University, found that many of the economists reported that they had had difficulty in publishing their most innovative work.
Other similar examples abound. In 1990, Raymond Hubbard and I found that reviewers recommended rejecting all but one of the papers with controversial findings that were submitted to 16 leading psychology journals over a two-year period. Furthermore, when Michael Mahoney, a psychology professor at North Texas State University, obtained reviews from 67 academics of a fictitious psychology paper, he discovered that the reviewers who received a version whose results disagreed with commonly accepted beliefs rated it lower (1.8 on a 6-point scale) than the reviewers who received a version that agreed with accepted beliefs (3.2 on the scale). The typical reason for a lower rating was"poor methodology,” although the methodology in the two versions was identical.
What I’m describing is not a new problem. Many papers that have become classics or have won their authors Nobel Prizes initially received harsh reviews when they were submitted to journals. In one notable instance, Science in 1955 rejected a paper by the medical physicist Rosalyn S. Yalow that reported on work for which she won a Nobel Prize in 1978. The paper also was rejected by the Journal of Clinical Investigation, although after an appeal, that journal published it in 1966.
I believe that peer reviewers often approach their task with an orientation that discourages publication of innovative science. Rather than being concerned about publishing interesting and useful papers, they focus on what is wrong with a paper. To compound the problem, their chief question seems to be:"Do I agree with the findings?” If not, they often conclude that the author must be wrong. I suspect that the vast majority of papers contain some errors (mostly minor), and that many reviewers use such minor errors to reject papers that make them uncomfortable. Or they simply claim:"The author failed to convince me.”
I have received many such reviews, and they do not include any clues as to what would convince the reviewer. Furthermore, in my nearly 30-year history of publications, I cannot recall a single occasion in which I have succeeded in convincing a dissenting reviewer by supplying additional experimental results, surveys, analyses, or text.
Several options are available for improving the peer-review process. One way to change reviewers’ orientation would be to ask them not whether a paper should be published, but rather how it could be improved. Another option to encourage the publication of innovative science is for journals to publish more invited papers. The successful Journal of Economic Perspectives consists primarily of papers by creative researchers invited by the editor to contribute. Unfortunately, some academics view this practice as unfair and unethical because not every researcher has a chance to participate.
Yet another approach to improving peer review is to give authors the option of a"results-blind review,” in which the editor asks reviewers to comment only on the methodology of a paper, and not on the results. (The author of the paper would prepare a version from which the results and conclusions are deleted.) Merely offering this option is useful as a defense against authors who claim that reviewers are biased against findings that contradict their own beliefs. Results-blind reviewing has been available for many years at the International Journal of Forecasting.
In addition to changing the reviewers’ orientation, journal editors should ask authors to provide evidence that their paper had been reviewed by other scientists before it was submitted for publication. Early reviews are especially useful in spotting problems with the design of research, although reviews at any stage of the process can help detect errors. Of course, many authors already ask colleagues to comment on preliminary versions of papers, but a surprising number do not. Journal editors can make the practice more widespread by offering preferential treatment, such as fewer reviews or faster decisions, to authors who can cite feedback from other scientists or who have presented their research at professional meetings.
Furthermore, editors and scientists alike should place more emphasis on peer review after publication. Rather than satisfying two or three reviewers before a paper is published, authors and editors should be concerned about the long-term impact of the published work. Editors can make greater use of a"Letters to the Editors” section to publish scientists’ reactions to papers, as the American Psychologist and other journals do. The Internet also can help make such post-publication review easier. If a journal makes its papers available on the Internet, it could add links to subsequent comments about them.
In the short term, change in publication practices will depend on the decisions of journal editors and editorial boards. Editors must focus more on publishing innovative papers and on the long-term value of what they publish. Editors or publication boards could initiate any of the changes that I have recommended on a trial basis and revert to the status quo if the modifications did not prove helpful.
In the meantime, I have decided to change my own practices. In the future, I plan to write reviews that make no recommendations about publication; I will focus only on how a paper might be improved. Then it will be up to journal editors to decide which papers are likely to make the most-important contributions to their fields.
J. Scott Armstrong is an associate professor of marketing at the Wharton School at the University of Pennsylvania.