> Skip to content
FEATURED:
  • The Evolution of Race in Admissions
Sign In
  • News
  • Advice
  • The Review
  • Data
  • Current Issue
  • Virtual Events
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Find a Job
    • Post a Job
Sign In
  • News
  • Advice
  • The Review
  • Data
  • Current Issue
  • Virtual Events
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Find a Job
    • Post a Job
  • News
  • Advice
  • The Review
  • Data
  • Current Issue
  • Virtual Events
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Find a Job
    • Post a Job
Sign In
ADVERTISEMENT
News
  • Twitter
  • LinkedIn
  • Show more sharing options
Share
  • Twitter
  • LinkedIn
  • Facebook
  • Email
  • Copy Link URLCopied!
  • Print

Leading Humanities Journal Debuts ‘Open’ Peer Review, and Likes It

By  Jennifer Howard
July 26, 2010

Getting published in a humanities journal usually works like this: Submit an article, then hope for the best as the editors send it to a few hand-picked specialists for critique. The reviewers and the authors are not supposed to know one another’s identity.

But now scholars are asking whether this double-blind peer-review system is still the best way to pass judgment. The Internet makes it possible to share work with many people openly and simultaneously, so why not tap the public wisdom of a crowd? One of the top journals in literary studies, Shakespeare Quarterly, decided to put that question to the test.

We’re sorry. Something went wrong.

We are unable to fully display the content of this page.

The most likely cause of this is a content blocker on your computer or network. Please make sure your computer, VPN, or network allows javascript and allows content to be delivered from c950.chronicle.com and chronicle.blueconic.net.

Once javascript and access to those URLs are allowed, please refresh this page. You may then be asked to log in, create an account if you don't already have one, or subscribe.

If you continue to experience issues, contact us at 202-466-1032 or help@chronicle.com

Getting published in a humanities journal usually works like this: Submit an article, then hope for the best as the editors send it to a few hand-picked specialists for critique. The reviewers and the authors are not supposed to know one another’s identity.

But now scholars are asking whether this double-blind peer-review system is still the best way to pass judgment. The Internet makes it possible to share work with many people openly and simultaneously, so why not tap the public wisdom of a crowd? One of the top journals in literary studies, Shakespeare Quarterly, decided to put that question to the test.

For this year’s fall issue, a special publication devoted to Shakespeare and new media, the journal offered contributors the chance to take part in a partially open peer-review process. Authors could opt to post drafts of their articles online, open them up for anyone to comment on, and then revise accordingly. The editors would make the final call about what to publish (hence the “partially open” label). As far as the editors know, it’s the first time a traditional humanities journal has tried out a version of crowd-sourcing in lieu of double-blind review.

The verdict from several scholars who took part: mostly a thumbs up, with a few cautionary notes and a dollop of “It’s about time” mixed in.

“It was on the whole a successful experiment,” said Martin Mueller, a professor of English and classics at Northwestern University, who took part as a reviewer.

ADVERTISEMENT

Michael Witmore, a professor of English at the University of Wisconsin at Madison, co-authored an article on the use of statistical analysis and a text-tagging database to reveal linguistic patterns in Shakespeare. He and his co-author “got some terrific ideas and some citations” from the comments of the six or so people who actively responded to the article, he said. “It produced a more interesting paper.”

Scholars and editors in the sciences have been trying out open peer review for some time, with not entirely rosy results. The journal Nature did a test run in 2006. In a published overview, the editors concluded the venture had not been a success. Many authors expressed interest but few participated, and the quantity and quality of the comments were disappointing.

Interviews with participants in Shakespeare Quarterly‘s open peer-review trial, however, suggest this attempt went much better than Nature‘s did. At least one participant pointed out that the humanities’ subjective, conversational tendencies may make them well suited to open review—better suited, perhaps, than the sciences.

Katherine Rowe, chair of the English department at Bryn Mawr College, guest-edited the special issue. She and the editorial board decided that the issue’s new-media theme offered a chance to investigate how scholarly authority works in a networked environment.

ADVERTISEMENT

“This was genuinely an experiment,” Ms. Rowe said. “We didn’t know what would result.”

The journal’s publisher, the Johns Hopkins University Press, supported the idea. MediaCommons, a digital scholarly network set up to encourage such experiments and conversations, agreed to host the project. So Ms. Rowe and other editors put out a call for papers, culled submissions, then offered authors still in the running a chance to post drafts online. All accepted.

Rounding Up Experts

Ms. Rowe invited about 90 scholars, including Northwestern’s Mr. Mueller, to comment. Anybody willing to publish thoughts under his or her own name could join in, but the guest editor wanted recognized authorities as part of the field.

“‘What’s the nature of expertise?’ is one of the questions that really gets opened up by an open process,” Ms. Rowe said. “Everybody wanted to be sure that experts would be involved.” By her count, about 40 commenters, invited and self-selected, finally participated.

All told, four articles and three review essays were posted on MediaCommons during a two-month review period this past spring. (The articles and comments remain archived on the MediaCommons site.) As it turned out, all seven submissions will appear in the issue.

ADVERTISEMENT

Some of the authors acknowledged doubts going into the experiment. Would open review be rigorous enough? Was it risky to post work in progress? But “the results were terrific,” said Mr. Witmore, who wrote his paper on Shakespearean linguistic analysis with Jonathan Hope, a reader in the English department of the University of Strathclyde, in Glasgow. “It’s very different from getting a two-paragraph reader’s report from a journal,” Mr. Witmore said. “In this case, what you get is individual readers from a wide range of subspecialties zooming in on a particular paragraph, saying ‘Tell me more about this’ or ‘Why did you do this?’ It seemed more like a dialogue.”

Another scholar, Alan Galey, submitted an article about Shakespeare and the history of information. An assistant professor on the Faculty of Information at the University of Toronto, he worried that an article vetted this way might carry less professional weight—a matter of particular concern to a junior professor going for tenure. “It was very much going on faith in a way,” he said.

Mr. Galey’s dean told him to make sure the process would be rigorous and fair. The stature of the journal also helped reassure him on that point. So did Ms. Rowe’s willingness to answer questions and her decision to invite established scholars to join in. Many crowd-sourcing experiments depend on scale, Mr. Galey pointed out, but this relied “on relationships among scholars where you know you can trust somebody. It wasn’t a Wild West by any means. It was as controlled a process as traditional peer review. It was just controlled in a different way.”

Mr. Galey wound up feeling that the experiment paid off. “I got better feedback from this process than I’ve had from any other peer-review process,” he said.

Another participant, Ayanna Thompson, an associate professor of English at Arizona State University, wrote an article focused on race in performance-based teaching of Shakespeare. She called the experiment “a fascinating process.” But she also found it stressful, and saw evidence that some old ways die hard.

ADVERTISEMENT

Better Feedback

On the positive side, she got lots of feedback—"the equivalent of eight single-spaced pages of comments,” she told The Chronicle via e-mail. She felt that the lack of anonymity encouraged reviewers to engage with the material “in a much more thoughtful and thorough way than in blind reviews because their names were attached to the comments.” That helped her clarify and revise sections of her argument. But going through a public critique “was a stressful experience, and I kept saying to myself, ‘I’m glad I have tenure,’” she said.

She also discovered that academic status still played a role. “The people who were commenting online were very established, senior scholars. In fact, several junior scholars notified me offline that they had comments, but that they did not want to post them in case they contradicted the senior scholars,” she said. “So while the process was supposed to democratize the review process, it did keep some of the old hierarchies in place.”

Still, Ms. Thompson is glad she took part. “The revised article is much stronger for the feedback, and I feel lucky to have received so much deep engagement with my work,” she said.

David Schalkwyk is editor of the journal and director of research at the Folger Shakespeare Library. He called the experiment a success and said that Shakespeare Quarterly plans to try it again for its future special issues, whose focused topics are most likely to attract a critical mass of knowledgeable reviewers.

The system did, however, come at a cost from an editor’s perspective, he said. For instance, editors and writers had to keep tabs on the evolving discussion and build in extra time for revisions; authors also tended to make their articles longer and longer in response to multiple comments. “I think we underestimated the amount of time that was required,” Mr. Schalkwyk said.

ADVERTISEMENT

That’s what keeps him from adopting the approach for every issue. He said, “My reservation about applying open peer review to normal issues is entirely practical. It’s not philosophical. If I could do it, I would.”

For Mr. Mueller, the reviewer from Northwestern, the journal’s experiment was welcome—and past due. “I think it will take a decade for scholarly communication to find ways of fitting itself into this new technological environment,” he said. “It takes time. The human problems always take more time to manage and solve than the technological problems.”

We welcome your thoughts and questions about this article. Please email the editors or submit a letter for publication.
Scholarship & Research
Jennifer Howard
Jennifer Howard, who began writing for The Chronicle in 2005, covered publishing, scholarly communication, libraries, archives, digital humanities, humanities research, and technology.
ADVERTISEMENT
ADVERTISEMENT
  • Explore
    • Get Newsletters
    • Letters
    • Free Reports and Guides
    • Blogs
    • Virtual Events
    • Chronicle Store
    • Find a Job
    Explore
    • Get Newsletters
    • Letters
    • Free Reports and Guides
    • Blogs
    • Virtual Events
    • Chronicle Store
    • Find a Job
  • The Chronicle
    • About Us
    • DEI Commitment Statement
    • Write for Us
    • Talk to Us
    • Work at The Chronicle
    • User Agreement
    • Privacy Policy
    • California Privacy Policy
    • Site Map
    • Accessibility Statement
    The Chronicle
    • About Us
    • DEI Commitment Statement
    • Write for Us
    • Talk to Us
    • Work at The Chronicle
    • User Agreement
    • Privacy Policy
    • California Privacy Policy
    • Site Map
    • Accessibility Statement
  • Customer Assistance
    • Contact Us
    • Advertise With Us
    • Post a Job
    • Advertising Terms and Conditions
    • Reprints & Permissions
    • Do Not Sell My Personal Information
    Customer Assistance
    • Contact Us
    • Advertise With Us
    • Post a Job
    • Advertising Terms and Conditions
    • Reprints & Permissions
    • Do Not Sell My Personal Information
  • Subscribe
    • Individual Subscriptions
    • Institutional Subscriptions
    • Subscription & Account FAQ
    • Manage Newsletters
    • Manage Your Account
    Subscribe
    • Individual Subscriptions
    • Institutional Subscriptions
    • Subscription & Account FAQ
    • Manage Newsletters
    • Manage Your Account
1255 23rd Street, N.W. Washington, D.C. 20037
© 2023 The Chronicle of Higher Education
  • twitter
  • instagram
  • youtube
  • facebook
  • linkedin