What if scholarly books were peer reviewed by anonymous blog comments rather than by traditional, selected peer reviewers?
That’s the question being posed by an unusual experiment that began last week. It involves a scholar studying video games, a popular academic blog with the playful title Grand Text Auto, a nonprofit group designing blog tools for scholars, and MIT Press.
The idea took shape when Noah Wardrip-Fruin, an assistant professor of communication at the University of California at San Diego, was talking with his editor at the press about peer reviewers for the book he was finishing, The book, with the not-so-playful title Expressive Processing: Digital Fictions, Computer Games, and Software Studies, examines the importance of using both software design and traditional media-studies methods in the study of video games.
One group of reviewers jumped to his mind: “I immediately thought, you know it’s the people on Grand Text Auto.” The blog, which takes its moniker from the controversial video game Grand Theft Auto, is run by Mr. Wardrip-Fruin and five colleagues. It offers an academic take on interactive fiction and video games.
The blog is read by many of the same scholars he sees at academic conferences, and also attracts readers from the video-game industry and teenagers who are hard-core video-game players. At its peak, the blog has had more than 200,000 visitors per month, he says.
“This is the community whose response I want, not just the small circle of academics,” Mr. Wardrip-Fruin says.
So he called up the folks at the Institute for the Future of the Book, who developed CommentPress, a tool for adding digital margin notes to blogs (The Chronicle, September 28, 2007). Would they help out? He wondered if he could post sections of his book on Grand Text Auto and allow readers, using CommentPress, to add critiques right in the margins.
The idea was to tap the wisdom of his crowd. Visitors to the blog might not read the whole manuscript, as traditional reviewers do, but they might weigh on a section in which they have some expertise.
The institute, an unusual academic center run by the University of Southern California but based in Brooklyn, N.Y., was game. So was Mr. Wardrip-Fruin’s editor at MIT Press, Doug Sery, but with one important caveat. He insisted on running the manuscript through the traditional peer-review process as well.
“We are a peer-review press — we’re always going to want to have an honest peer review,” says Mr. Sery, senior editor for new media and game studies. “The reputation of MIT Press, or any good academic press, is based on a peer-review model.”
So the experiment will provide a side-by-side comparison of reviewing — old school versus new blog. Mr. Wardrip-Fruin calls the new method “blog-based peer review.”
Each day he will post a new chunk of his draft to the blog, and readers will be invited to comment. That should open the floodgates of input, possibly generating thousands of responses by the time all 300-plus pages of the book are posted. “My plan is to respond to everything that seems substantial,” says the author.
The institute is modifying its CommentPress software for the project, with the help of a $10,000 grant from San Diego’s Academic Senate, to create a version that bloggers can more easily add to their existing academic blogs.
Mr. Wardrip-Fruin’s friends have warned him that sorting through all those comments will take over his life, or at least take far more time than he expects. “It’s been said to me enough times by people who are not just naysayers that it is in the back of my mind,” he acknowledges. Still, the book’s review process “will pale in comparison to the work of writing it.”
He expects the blog-based review to be more helpful than the traditional peer review because of the variety of voices contributing. “I am dead certain it will make the book better,” he says.
Mr. Sery isn’t so sure. “I don’t know how this general peer review is going to help,” the editor says, except maybe to catch small errors that have slipped through the cracks. (The journal Nature attempted something similar in 2006 with scientific papers, but found that few papers received truly useful comments.)
Traditional peer review involves carefully chosen experts in the same subject area, who can point to big-picture issues as well as nitpick details. Mr. Sery bets that the blog reviews might merely spark flame wars or other unhelpful arguments about minor points. “I’m curious to see what kind of comments we get back,” he says.
That probably “depends on what you’re writing about,” says Clifford A. Lynch, executive director of the Coalition for Networked Information, a group that supports the use of technology in scholarly communication. “If, God help you, you’re writing about current religious or political issues, you’re going to get a lot of people with agendas who aren’t interested in having a rational discussion. Some of them are just psychos.”
Even without flame wars, Mr. Sery equates the blog review with the kind of informal sharing of drafts that many academics do with close friends. It’s useful, but it’s still not formal peer review, he argues; Carefully choosing reviewers “really allows for the expression of their ideas on the book,” he says. Scholars can say with authority, for instance, that a book just isn’t worth publishing.
A Community Review
Ben Vershbow, editorial director at the Institute for the Future of the Book, concedes that comments on blogs are unlikely to fully replace peer review. But he says academic blogging can play a role in the publishing process.
“The conversational modes of reading and writing on the Web in things like blogs and wikis really chime well with the essential idea of peer review, which is putting out work in development to a peer group and refining the work,” he says.
Whether or not it succeeds, Mr. Wardrip-Fruin expects the experience will be interesting enough to write up in an academic essay, or maybe in the preface to the book, when it finally comes out in the old-fashioned printed form.
http://chronicle.com Section: Information Technology Volume 54, Issue 21, Page A20
