An editor at a traditional scholarly journal knows what to do with a print monograph in the humanities or social sciences: Find a qualified reviewer to read it and write about the contribution it makes—or doesn’t. Yet create an online interactive archive or digital scholarly edition, and the same editor probably won’t know what to do with it.
There were hardly any journal reviews, for instance, of the monographs in the much-pushed Gutenberg-e digital-history experiment in the mid-2000s. More recently, Penelope J. Kaiserlian, director of the University of Virginia Press, told a gathering of digital humanists how hard it had been to get journals to review projects published by Rotunda, her press’s digital imprint.
This issue may be compounding another worry I keep hearing about: how to get academe’s gatekeepers to take digital work seriously. If the members of your tenure-and-promotion committee don’t have the skills to judge your dazzling visualization of Republican Rome or your fluid-text edition of Rasselas, and if it’s not getting written about in the journals they read and respect, how likely are they to give you full points for your work?
At the American Council of Learned Societies’ annual meeting, in Philadelphia in early May, Helen Cullyer, an associate program officer for scholarly communication at the Andrew W. Mellon Foundation, said enough is enough. She called for “a new form of published review, namely a scholarly review of a digital collection.” She spoke at a panel focused on Google Book Search and what it might mean for scholarship, but her point applies more broadly. And several scholars are beginning to identify criteria to make such reviews rigorous and worthwhile.
A few formal outlets do exist. For instance, The Journal of American History runs a twice-yearly section of Web-site reviews in partnership with History Matters, a site run by the Center for History and New Media at George Mason University.
But those are exceptions. A lot of assessment and discussion of digital-humanities work takes place on blogs and in other informal venues. In Britain, for example, a site called arts-humanities.net serves as a hub for write-ups of digital collections and tools.
The Networked Infrastructure for Nineteenth-Century Electronic Scholarship, or Nines, was created in part “to serve as a peer-reviewing body for digital work in the long 19th century (1770-1920), British and American,” it explains on its Web site. It does not make the specifics of its judgments public and so doesn’t provide the same service as the reviews section of a journal. But if your project passes muster with Nines’s peer reviewers, that marks it as serious scholarship. So does getting a grant from the National Endowment for the Humanities’ Office of Digital Humanities or some other grant-making body that supports digital work.
“In practice, granting agencies have become the certifying authorities for this kind of scholarly work,” Stephen Ramsay, an associate professor of English at the University of Nebraska at Lincoln and a fellow at the university’s Center for Digital Research in the Humanities, told me via e-mail. “That is, merit-review committees take the winning of a grant as a Good Housekeeping Seal of approval on the work. But, of course, that more often happens before any work has been done.”
The after part has been trickier. It’s taken a while to retrofit the machinery of scholarly reviewing to handle digital-humanities work. How do you review an online resource like the Walt Whitman Archive, an immense, collaborative, multimedia resource that allows users to work with variant texts, aids for finding things, and audio and visual material? The short answer is that it’s not as easy as picking up a book.
Brett Bobley, director of the NEH’s Office of Digital Humanities, put it like this in an e-mail to me: “In the past, an edition was judged almost entirely on the scholarship (rightly so). But in the digital realm, we also need to judge it on their digital infrastructure. Do they have useful metadata? A sustainability plan? Are they conforming with library/archive standards? Do they have an API [application programming interface] to enable others to repurpose the data or mash it up with other data? Etc. These are all important issues.”
Can’t you just judge a digital edition on the merits of the scholarship alone? “Only looking at the scholarship is sort of like judging a new building strictly on aesthetics,” Mr. Bobley said. “Yes, it’s a gorgeous building, but if it isn’t structurally sound it’s going to fall down. Sometimes people don’t want to look at the plumbing, but I’d argue that it is essential to the functioning of the building or to the digital scholarly project.” So a good reviewer will come equipped to assess, say, the optimal-character-recognition software a digital-humanities project uses to create scans of texts.
4 Elements for Success
Julia Flanders is editor in chief of Digital Humanities Quarterly and director of the Women Writers Project, which is part of the Center for Digital Scholarship, housed in the Brown University Library. A piece of digital scholarship has four components that have to be weighed, she told me when I called to talk about reviewing digital work. The key elements are the content, the digital tools used to build it, how its data are structured, and the interface. All of those help determine how a project can be used, whether it’s a model for other work, and what contributions it makes to the field.
“Each of those four strands is crucial from a reviewing point of view,” Ms. Flanders explained, “because they all contribute to the intellectual impact of the resource.” A successful digital project does not necessarily need to be strong in all four areas; its main contribution could be an innovative interface or a nifty bit of software that will let other researchers do equally nifty things with it. “The value may be 98 percent in the rigor with which the data was prepared,” she said.
The trick for a reviewer is figuring out how to assess that value and convey it to a reader. “It’s a challenging mode to write in, because you need access to information that’s often quite gigantic and detailed and isn’t easy to look at,” Ms. Flanders said. “Because each project is sui generis, each one poses its own kind of challenges of assessment. There’s a comparatively small pool of people who have expertise in the relevant area.”
Reviewing is already an underappreciated activity, which busy scholars may be disinclined to take on. Ms. Flanders thinks that journal editors ought to play up reviews of digital work “so they get foregrounded: ‘So-and-so has done a major review of this site.’”
DHQ has published many self-critiques by digital humanists assessing their own work, Ms. Flanders said. Now it’s getting into reviews that function more like traditional book reviews but focus on digital projects. She thinks there’s a place for assessments of “software tools, sites, other kinds of innovations that need the same kind of critical scrutiny and benefit from the same kind of contextualizing review that a traditional book review offers.”
If such reviews do catch on as a genre, as the Mellon foundation’s Ms. Cullyer suggested, that could help digital humanists explain themselves and their work to colleagues who don’t get it yet. The old guard may be more baffled than unwilling.
Susan Schreibman is trying to help remedy that. She is director of the Digital Humanities Observatory at the Royal Irish Academy, and during the last couple of years, she has helped lead a workshop at the Modern Language Association’s annual conference on how to evaluate digital scholarship. It brings together scholars doing the work and those who have to figure out how to rate it.
After the most recent workshop, Ms. Schreibman and I talked about how digital scholarship has outpaced the formal systems of evaluating humanities work. “We have such a robust system of evaluation in literary studies in the academy, in terms of book and article publication, that we just don’t have for digital scholarship, or maybe not yet,” she said. On the good-news front, she said, “This is something that department chairs really, really want to come to terms with. We’re seeing more and more hires where digital media and scholarship are part of why you’re hired. I think departments want to encourage this work.”
Now let’s see if more scholarly societies and editors of traditional journals will step up and do the same.
