As Concerns Grow About Using Data to Measure Faculty, a Company Changes Its Message

Just a few years ago, Academic Analytics, an upstart company providing data on faculty productivity, talked of helping cash-strapped universities save as much as $2 billion by identifying their lowest-performing professors.

At many universities, “an awful lot of the scholarly work is being carried by a relatively small proportion of all of the people,” said a company founder, Lawrence B. Martin, back in 2012. The value of stanching such waste could be “staggering,” Mr. Martin said.

Now the ambitions of his decade-old company are a bit more measured.

Following the defection last week of Georgetown University — whose provost explained a decision to drop the university’s subscription by questioning whether Academic Analytics’ data are comprehensive, accurate, or consistently valuable — the company is now dialing back its promises of huge cost savings.

Academic Analytics combs various databases to supply universities with details on the research activity of their faculty members. The company does not now believe that institutions should use its information to make individual personnel decisions, said a spokeswoman, Tricia Stapleton. Instead, Ms. Stapleton said, data from Academic Analytics should just be one element among many pieces of information that university leaders use to make broad assessments of their schools and departments.

Ms. Stapleton said she did not know why Mr. Martin, one of the company’s two founders, would have encouraged individual evaluations in 2012. Mr. Martin, a professor of anthropology and former dean of the Graduate School at Stony Brook University, part of the State University of New York, was not available for comment, she said.

Mr. Martin helped create Academic Analytics out of a concern that the leading source of information on faculty research productivity — study data issued by the National Research Council — was issued only about once every decade and was therefore unreliable for many uses.

In promoting his company, he emphasized the value of Academic Analytics’ data in helping cash-strapped universities identify faculty members whose relatively low teaching loads were not justified by the volume of their research output.

Georgetown is one of a handful of institutions where faculty members began to chafe at the data’s role on the campus. Georgetown’s provost, Robert M. Groves, posted a notice online last week saying his office began a project this past summer to compare Academic Analytics records with the university’s own understanding of its faculty research output.

The project found various inaccuracies, including a failure to count about 35 percent of the academic papers published by faculty members in the Georgetown School of Public Policy, said Mr. Groves, a professor of mathematics and statistics and of sociology.

“Even with perfect coverage,” he wrote, “the data have differential value across fields that vary in book versus article production and in their cultural supports for citations of others’ work.”

Academic Analytics now has about 100 subscribing universities, after steady growth since its founding, in 2005, Ms. Stapleton said. Some universities have discontinued the service, though often for budgetary reasons, she said. In the case of Georgetown, company representatives said they had not yet had a chance to review the specific findings of its investigation and could not immediately comment on it.

Faculty Discontent

Beyond Georgetown, institutions that have publicly encountered faculty uprisings over the service include Rutgers University at New Brunswick, where the local chapter of the American Association of University Professors has led a persistent campaign for the university to reject Academic Analytics, as Mr. Groves did.

The local union president, David M. Hughes, a professor of anthropology at Rutgers, said he also regarded Academic Analytics’ data as both inaccurate and insufficient for judging the complexities of professors’ value.

“Our objection, fundamentally, is that these kinds of corporate-derived measurement tools, if successful, will narrow the scholarly life,” Mr. Hughes said.

Mr. Hughes said the market demand for Academic Analytics fits with the growing tendency of universities to hire professional administrators. He cited examples such as Peter March, executive dean of the School of Arts and Sciences at Rutgers, saying the former professor of math at Ohio State University has “never been among us as a colleague” and therefore needs Academic Analytics to tell him what’s happening inside his own institution.

University administration has become “an academic aristocracy of people earning very high salaries who move around like traded baseball players among institutions, climbing the ladder,” Mr. Hughes said.

The Rutgers chancellor, Richard L. Edwards, strongly disputed that characterization. Many Rutgers administrators are current or former faculty members, Mr. Edwards said. And those coming from the outside can bring a useful perspective, he said. Sometimes long-serving faculty members “really haven’t been out and about much, and don’t know what’s happening in other places,” he said.

Even administrators with extensive faculty experience can’t possibly have a deep familiarity with all departments under their jurisdictions, said Robert M. Berdahl, a former president of the Association of American Universities who is now serving as a senior academic adviser to Academic Analytics.

As a dean, a provost, and an historian, Mr. Berdahl said, he knew his department. But he was also responsible for departments in physics, chemistry, and economics. To assess them, he said, it would be invaluable to have comparable data from other institutions.

Rather than drop subscriptions to Academic Analytics, some institutions are finding ways of managing faculty discontent. Rutgers is now in the process of establishing a process by which individual faculty members can learn what data goes into their Academic Analytics profile and get inaccuracies corrected.

But a thorough, universitywide assessment of the data’s accuracy, as took place at Georgetown, isn’t likely at Rutgers. The institution is too large for that, Mr. Edwards said.

The Faculty Association at the University of California at Santa Cruz is also challenging its administration’s use of Academic Analytics, said the union’s co-chair, Deborah B. Gould, an associate professor of sociology.

Ms. Gould said she and other faculty members have seen numerous instances of the company’s database failing to include their papers, awards, and other elements it counts.

But a more fundamental objection, she said, is the company’s premise of being able to truly judge a faculty member’s long-term value by making year-by-year tallies of academic output. A researcher might have thin levels of output while working for years on a project that fundamentally changes his or her field, she said. That’s the kind of thing that should be judged by fellow faculty members, Ms. Gould said.

“It seems strange to me to turn to a data-gathering corporation that actually isn’t thinking about the kind of nonlinear, erratic temporalities of intellectual labor,” she said. “We are able to evaluate ourselves on that front, really.”

The faculty union at the University of Connecticut is also pressing for change, saying that professors have been told that Academic Analytics data affect their professional standing. The president of the local chapter of the American Association of University Professors, Diana Rios, an associate professor of communication, said university administrators have been in talks with the union over possible solutions. She said she didn’t want to discuss details, to avoid harming the process.

As Academic Analytics backs away from Mr. Martin’s suggestion of billions of dollars in savings on unjustified salary costs, the company has changed its message about the value of the data it says it has assembled on some 390,000 faculty members and 33,000 journal articles.

Mr. Berdahl, designated by Academic Analytics to speak on its behalf, initially said a university dean might use the company’s product to help determine “what faculty you can identify for outstanding awards, or what faculty are really crucial to the future of the department, and what kind of retention efforts one would want to make on their behalf.”

But he distanced himself from Mr. Martin’s talk of cost savings associated with individual faculty members, saying the Academic Analytics data really is meant for broader assessments. “It’s based upon individual faculty,” he said of the data, “but by and large you’re trying to evaluate the department.”

Paul Basken covers university research and its intersection with government policy. He can be found on Twitter @pbasken, or reached by email at paul.basken@chronicle.com.

Correction (10/11/2016, 10:35 a.m.): This article originally described Mr. Martin’s comments about the potentially “staggering” value of faculty-productivity tracking as being made in 2013. In fact, he made those statements in 2012, and the text has been changed to reflect that.