Certifying Online Research

A department chairman from a nearby university recently solicited my advice on how to handle a tenure case in which the candidate’s entire body of scholarship consisted of online publications.

The candidate was a valued colleague, but the department’s faculty members were “extremely conservative” in their definition of acceptable scholarly work, the chairman told me. He worried that he would lose his young colleague to what he termed “generational prejudices.”

“Our university has no formal policy governing electronic scholarship,” he told me. “And, quite frankly, I’m in no position to judge it myself.”

That anxiety was echoed by a dean of education who complained at a recent professional conference that faculty members in her college were “deeply divided” over how to evaluate electronic scholarship. Professors in her college’s curriculum and instruction department readily embraced many forms of “e-scholarship,” while faculty members in other departments seemed unreceptive or even hostile to those emerging forms.

“What am I supposed to do?” she implored in exasperation. “One group sees itself as avant-garde and riding the wave of the future; the other sees itself as upholding standards against the corrosive force of technophilia. I, as dean, am stuck in the middle.”

Those two scenarios illustrate a growing area of concern for tenure committees and academic administrators throughout higher education: Exactly how do we arrive at a judicious assessment of scholarship presented in various forms of new media? How do we acknowledge and reward substantive electronic scholarship that genuinely furthers knowledge in a discipline, while avoiding the awarding of undue credit to less worthy work?

The digital revolution has substantially improved scholarly work, but it has also brought challenges to those of us charged with overseeing our institutions’ tenure, promotion, and rewards processes. While several electronic forms compete for legitimacy, the two most prominent are journals published exclusively online, and Web sites devoted to scholarly subjects.

As more and more electronic journals adopt peer-review processes that replicate the rigorous ones employed by established print journals, many e-journals are acquiring reputations for comparable rigor. Over time, each discipline will arrive at a general consensus about the status of various e-journals in the same way that they once did for print journals.

Scholarly Web sites, however, present a unique set of challenges to college administrators.

In the print world, scholars generally agree about which sources are reputable. All notable scholarly presses thoroughly vet each project, and certain venues have gained reputations for conducting especially rigorous peer reviews. No such formal gatekeeping, in contrast, is in place for the scholarly sites on the Internet.

A Web site devoted to a celebrated literary critic, for example, purports to contain the most comprehensive bibliography of his oeuvre, but the bibliography is far from comprehensive and is riddled with errors. Simply put, the site is not dependable.

As it turns out, it is operated by a student -- which, in and of itself, is not necessarily a problem. But the point is that it is not always possible to ascertain exactly who is operating a site -- a senior scholar, an entry-level assistant professor, a graduate student, or even a nonacademic.

And since no vetting mechanism for scholarly sites exists, even those that are designed by reputable scholars typically undergo no formal review. Such uncertainty disrupts the orderly intercourse of scholarly activity and plays havoc with the tenure-and-promotion system.

Clearly, the scholarly community needs to devise a way to introduce dependability into the world of electronic scholarship. We need a process to certify sites so that we all can distinguish between one that contains reliable material and one that may have been slapped together by a dilettante. We need to be able to ascertain if we can rely on a site for our own scholarship and whether we should give credit toward a colleague’s tenure and promotion for a given site.

With those goals in mind, I propose the following:

• The major professional and scholarly organizations in each discipline should devise a certification process in which a site owner can apply to have a site reviewed and recognized, perhaps for a nominal processing fee. The site would be subjected to a formal and rigorous review by peers in the disciplinary area covered by the site.

• Only those sites meeting the highest standards should be awarded certification.

• Once a site wins certification from the national scholarly society, it should be permitted to display that stamp of approval prominently.

• The certification should remain in effect for a specific and limited amount of time (since a site can change rapidly and without notice). The site should regularly seek renewal of its certification.

• Each disciplinary organization should issue a resolution recommending that departments construe certification of a site as indicating that it has met the highest standards of scholarship.

• Each organization should maintain an online registry of certified sites.

Only an accreditation process of that sort will resolve the principal obstacle to evaluating research published online. And only our most prestigious learned organizations can be trusted to review and endorse works of scholarship to the satisfaction of all involved. A certification process would give working scholars a modicum of faith in digital resources, and would enable tenure committees and administrators to make informed decisions.

While Internet aficionados might object that traditional print forums are not subjected to a similar certification process, conventional venues do not publish works regardless of their quality. All university presses and most scholarly journals are refereed, and the few journals that are not refereed at least have some gatekeeping in the form of the journal’s editor.

Internet sites, in contrast, have no such mechanisms for quality control, and that is precisely why some kind of official recognition is necessary.

Some critics might object that the Internet and the scholars who publish online can regulate themselves, but I am not proposing “regulation.” I am calling for a voluntary process of certification. Site owners would be free to seek certification -- or not.

“Self-regulation” will not solve the primary problem, namely, that there is simply no way to know whether a site is dependable.

Other people might object that it would be difficult to construct guidelines governing site quality that are applicable from site to site. Yet scholars in each disciplinary area already judge the quality of scholarship every time they referee a manuscript for a university press or print journal. That process has worked well.

In fact, who better to judge the quality of scholarship in a field than peers active in that area? Certainly, each institution will need to articulate precisely how it will value electronic scholarship. But no colleague should be lost over “generational prejudices” -- or, for that matter, tenured because a department could not detect substandard scholarship.

It behooves the disciplinary organizations to play an active role in instituting a mechanism of quality control for scholarly Web sites. The alternatives -- chaos, or evaluation by nonexperts -- are unacceptable.