Productivity Metrics

Questions about faculty productivity are nothing new. But the growing use of metrics to assess faculty activity has raised the stakes at a time when colleges already face growing pressure to demonstrate accountability and compete with peer institutions.

Meanwhile, questions about how to measure a scholar’s influence in social media, known as “altmetrics,” are expected to add to the debate over faculty productivity.

One company that colleges turn to for help with metrics is Academic Analytics, which allows colleges to compare the peer-reviewed publications, journal citations, federal research grants, and other honors of their faculty members with those at peer institutions. The company says it has more than doubled the number of colleges it works with, to about 100, over the past five years.

At public colleges, the pressure to measure faculty productivity often comes from legislators. But all types of institutions are increasingly paying more attention to remaining competitive with their peers, says Peter Lange, chief academics adviser at the South Carolina-based company and a former provost of Duke University.

Such efforts haven’t always been executed with finesse. Texas A&M University, for example, issued a report in 2011 that listed faculty members’ names in red or black — like a corporate balance sheet — depending on whether the research and tuition dollars they generated covered their salary and expenses. Such heavy-handed efforts usually crumble under faculty opposition. Texas A&M abandoned its plan amid faculty objections to the perceived corporatization of the university as well as the accuracy of the data.

Winning over faculty members — or at least avoiding a revolt — is key to the long-term success of evaluation efforts, and administrators must strike a balance between their needs and faculty concerns.

“The most important thing,” says Gary A. Olson, president of Daemen College, “is to make sure you have faculty buy-in. If you have them helping in the production of the measurement instrument, you have the best chance of coming up with an instrument that everybody’s happy with.”

Pleasing everyone, though, may be impossible. Many faculty members, especially in the arts and humanities, are distrustful of faculty analytics.

“They’re trying to run creative thinking through a machine,” says Mark Usher, chair of the classics department at the University of Vermont. While the university works with Academic Analytics, it does not require the use of company data for evaluation, and recently asked each of its academic units to develop its own faculty-productivity metrics.

Mr. Usher says the metrics aren’t meaningful without context and often aren’t even accurate. He echoes faculty members on many campuses who have complained that reports based on metrics often show deflated grant awards and incorrect journal citations, and omit publications that should be included (and vice versa).

For example, he says, Academic Analytics had included Acta Astronautica, a publication he’d never heard of, as a classics journal. “What is that?” he asks, “Like, the study of UFOs?” (It’s an astronautics journal.)

Company officials acknowledge that assessing faculty productivity in the arts and humanities is tougher than in the sciences and engineering, where quantified measurements are the norm.

At the Massachusetts Institute of Technology, visiting committees, comprising members from academe, business, industry, and government, have reviewed each department since the 19th century. These days, those committees receive data compiled by Academic Analytics, which allows comparisons with peers. But administrators must be cognizant of disciplinary differences, says Lydia Snover, director of institutional research at MIT, and must put the data in context with other indicators of productivity, which the company’s data don’t measure.

Peer-reviewed publications may be an effective productivity metric for some departments. For others, like computer science, which produce fewer papers, the metrics might be citations per publication, or conferences per faculty member, or honors and awards. Federal grants could be a productivity metric for a department like chemistry, but not for engineering, since engineering faculty members at MIT receive a large share of grants from private sources that aren’t captured by Academic Analytics’ data.

Moreover, Ms. Snover says, colleges must make clear that faculty-productivity metrics will be placed in the context of a faculty member’s broader body of work. “A lot of this,” she says, “is just to be able to provide some comparative data that isn’t hearsay. It’s not perfect. It’s impossible to be perfect in these areas.”

Some say the next faculty-productivity battlefield might be altmetrics, a term used to describe alternative methods of gauging scholarly impact, including the use of blogs, news coverage, and social media. How many times was a tweet about your research retweeted, or “liked” on Facebook? Such measures have made headway in Britain but are still a gray area in the United States, says Anthony J. Olejniczak, chief knowledge officer of Academic Analytics.

“It’s not exactly clear where that line between what is scholarly and what is media is ultimately going to be settled,” he says. “But a blurring of that line is clearly something that has been happening in the last few years.”

Proponents of faculty analytics say the quality and accuracy of data have improved and will continue to do so as technology evolves. But the debate over how meaningful those data are won’t be settled anytime soon.

Vimal Patel covers graduate education. Follow him on Twitter @vimalpatel232, or write to him at vimal.patel@chronicle.com.