Skip to content
ADVERTISEMENT
Sign In
  • Sections
    • News
    • Advice
    • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Student Success
    • Technology
    • Transitions
    • The Workplace
  • Magazine
    • Current Issue
    • Special Issues
    • Podcast: College Matters from The Chronicle
  • Newsletters
  • Virtual Events
  • Ask Chron
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Professional Development
    • Career Resources
    • Virtual Career Fair
  • More
  • Sections
    • News
    • Advice
    • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Student Success
    • Technology
    • Transitions
    • The Workplace
  • Magazine
    • Current Issue
    • Special Issues
    • Podcast: College Matters from The Chronicle
  • Newsletters
  • Virtual Events
  • Ask Chron
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Professional Development
    • Career Resources
    • Virtual Career Fair
    Upcoming Events:
    An AI-Driven Work Force
    University Transformation: a Global Leadership Perspective
Sign In
Faculty

As Concerns Grow About Using Data to Measure Faculty, a Company Changes Its Message

By Paul Basken October 11, 2016

Just a few years ago, Academic Analytics, an upstart company providing data on faculty productivity, talked of helping cash-strapped universities save as much as $2 billion by identifying their lowest-performing professors.

At many universities, “an awful lot of the scholarly work is being carried by a relatively small proportion of all of the people,” said a company founder, Lawrence B. Martin, back in 2012. The value of stanching such waste could be “staggering,” Mr. Martin said.

To continue reading for FREE, please sign in.

Sign In

Or subscribe now to read with unlimited access for as low as $10/month.

Don’t have an account? Sign up now.

A free account provides you access to a limited number of free articles each month, plus newsletters, job postings, salary data, and exclusive store discounts.

Sign Up

Just a few years ago, Academic Analytics, an upstart company providing data on faculty productivity, talked of helping cash-strapped universities save as much as $2 billion by identifying their lowest-performing professors.

At many universities, “an awful lot of the scholarly work is being carried by a relatively small proportion of all of the people,” said a company founder, Lawrence B. Martin, back in 2012. The value of stanching such waste could be “staggering,” Mr. Martin said.

Now the ambitions of his decade-old company are a bit more measured.

Following the defection last week of Georgetown University — whose provost explained a decision to drop the university’s subscription by questioning whether Academic Analytics’ data are comprehensive, accurate, or consistently valuable — the company is now dialing back its promises of huge cost savings.

Academic Analytics combs various databases to supply universities with details on the research activity of their faculty members. The company does not now believe that institutions should use its information to make individual personnel decisions, said a spokeswoman, Tricia Stapleton. Instead, Ms. Stapleton said, data from Academic Analytics should just be one element among many pieces of information that university leaders use to make broad assessments of their schools and departments.

Ms. Stapleton said she did not know why Mr. Martin, one of the company’s two founders, would have encouraged individual evaluations in 2012. Mr. Martin, a professor of anthropology and former dean of the Graduate School at Stony Brook University, part of the State University of New York, was not available for comment, she said.

Mr. Martin helped create Academic Analytics out of a concern that the leading source of information on faculty research productivity — study data issued by the National Research Council — was issued only about once every decade and was therefore unreliable for many uses.

In promoting his company, he emphasized the value of Academic Analytics’ data in helping cash-strapped universities identify faculty members whose relatively low teaching loads were not justified by the volume of their research output.

Georgetown is one of a handful of institutions where faculty members began to chafe at the data’s role on the campus. Georgetown’s provost, Robert M. Groves, posted a notice online last week saying his office began a project this past summer to compare Academic Analytics records with the university’s own understanding of its faculty research output.

The project found various inaccuracies, including a failure to count about 35 percent of the academic papers published by faculty members in the Georgetown School of Public Policy, said Mr. Groves, a professor of mathematics and statistics and of sociology.

ADVERTISEMENT

“Even with perfect coverage,” he wrote, “the data have differential value across fields that vary in book versus article production and in their cultural supports for citations of others’ work.”

Academic Analytics now has about 100 subscribing universities, after steady growth since its founding, in 2005, Ms. Stapleton said. Some universities have discontinued the service, though often for budgetary reasons, she said. In the case of Georgetown, company representatives said they had not yet had a chance to review the specific findings of its investigation and could not immediately comment on it.

Faculty Discontent

Beyond Georgetown, institutions that have publicly encountered faculty uprisings over the service include Rutgers University at New Brunswick, where the local chapter of the American Association of University Professors has led a persistent campaign for the university to reject Academic Analytics, as Mr. Groves did.

Our objection, fundamentally, is that these kinds of corporate-derived measurement tools, if successful, will narrow the scholarly life.

The local union president, David M. Hughes, a professor of anthropology at Rutgers, said he also regarded Academic Analytics’ data as both inaccurate and insufficient for judging the complexities of professors’ value.

ADVERTISEMENT

“Our objection, fundamentally, is that these kinds of corporate-derived measurement tools, if successful, will narrow the scholarly life,” Mr. Hughes said.

Mr. Hughes said the market demand for Academic Analytics fits with the growing tendency of universities to hire professional administrators. He cited examples such as Peter March, executive dean of the School of Arts and Sciences at Rutgers, saying the former professor of math at Ohio State University has “never been among us as a colleague” and therefore needs Academic Analytics to tell him what’s happening inside his own institution.

University administration has become “an academic aristocracy of people earning very high salaries who move around like traded baseball players among institutions, climbing the ladder,” Mr. Hughes said.

The Rutgers chancellor, Richard L. Edwards, strongly disputed that characterization. Many Rutgers administrators are current or former faculty members, Mr. Edwards said. And those coming from the outside can bring a useful perspective, he said. Sometimes long-serving faculty members “really haven’t been out and about much, and don’t know what’s happening in other places,” he said.

Robert M. Berdahl, a former president of the Association of American Universities who’s advising the company Academic Analytics, acknowledges faculty concerns about using data the firm collects to measure individual professors’ productivity. “It’s based upon individual faculty,” he says of the data, “but by and large you’re trying to evaluate the department.”
Robert M. Berdahl, a former president of the Association of American Universities who’s advising the company Academic Analytics, acknowledges faculty concerns about using data the firm collects to measure individual professors’ productivity. “It’s based upon individual faculty,” he says of the data, “but by and large you’re trying to evaluate the department.”Joshua Clark

Even administrators with extensive faculty experience can’t possibly have a deep familiarity with all departments under their jurisdictions, said Robert M. Berdahl, a former president of the Association of American Universities who is now serving as a senior academic adviser to Academic Analytics.

ADVERTISEMENT

As a dean, a provost, and an historian, Mr. Berdahl said, he knew his department. But he was also responsible for departments in physics, chemistry, and economics. To assess them, he said, it would be invaluable to have comparable data from other institutions.

Rather than drop subscriptions to Academic Analytics, some institutions are finding ways of managing faculty discontent. Rutgers is now in the process of establishing a process by which individual faculty members can learn what data goes into their Academic Analytics profile and get inaccuracies corrected.

But a thorough, universitywide assessment of the data’s accuracy, as took place at Georgetown, isn’t likely at Rutgers. The institution is too large for that, Mr. Edwards said.

The Faculty Association at the University of California at Santa Cruz is also challenging its administration’s use of Academic Analytics, said the union’s co-chair, Deborah B. Gould, an associate professor of sociology.

ADVERTISEMENT

Ms. Gould said she and other faculty members have seen numerous instances of the company’s database failing to include their papers, awards, and other elements it counts.

But a more fundamental objection, she said, is the company’s premise of being able to truly judge a faculty member’s long-term value by making year-by-year tallies of academic output. A researcher might have thin levels of output while working for years on a project that fundamentally changes his or her field, she said. That’s the kind of thing that should be judged by fellow faculty members, Ms. Gould said.

“It seems strange to me to turn to a data-gathering corporation that actually isn’t thinking about the kind of nonlinear, erratic temporalities of intellectual labor,” she said. “We are able to evaluate ourselves on that front, really.”

The faculty union at the University of Connecticut is also pressing for change, saying that professors have been told that Academic Analytics data affect their professional standing. The president of the local chapter of the American Association of University Professors, Diana Rios, an associate professor of communication, said university administrators have been in talks with the union over possible solutions. She said she didn’t want to discuss details, to avoid harming the process.

ADVERTISEMENT

As Academic Analytics backs away from Mr. Martin’s suggestion of billions of dollars in savings on unjustified salary costs, the company has changed its message about the value of the data it says it has assembled on some 390,000 faculty members and 33,000 journal articles.

Mr. Berdahl, designated by Academic Analytics to speak on its behalf, initially said a university dean might use the company’s product to help determine “what faculty you can identify for outstanding awards, or what faculty are really crucial to the future of the department, and what kind of retention efforts one would want to make on their behalf.”

But he distanced himself from Mr. Martin’s talk of cost savings associated with individual faculty members, saying the Academic Analytics data really is meant for broader assessments. “It’s based upon individual faculty,” he said of the data, “but by and large you’re trying to evaluate the department.”

Paul Basken covers university research and its intersection with government policy. He can be found on Twitter @pbasken, or reached by email at paul.basken@chronicle.com.

Correction (10/11/2016, 10:35 a.m.): This article originally described Mr. Martin’s comments about the potentially “staggering” value of faculty-productivity tracking as being made in 2013. In fact, he made those statements in 2012, and the text has been changed to reflect that.

A version of this article appeared in the October 21, 2016, issue.
We welcome your thoughts and questions about this article. Please email the editors or submit a letter for publication.
Share
  • Twitter
  • LinkedIn
  • Facebook
  • Email
Paul Basken Bio
About the Author
Paul Basken
Paul Basken was a government policy and science reporter with The Chronicle of Higher Education, where he won an annual National Press Club award for exclusives.
ADVERTISEMENT
ADVERTISEMENT

Related Content

Can Data Measure Faculty Productivity? Rutgers Professors Say No
Productivity Metrics
How One University Measured Faculty Productivity — and Nobody Got Hurt

More News

Illustration of a magnifying glass highlighting the phrase "including the requirements set forth in Presidential Executive Order 14168 titled Defending Women From Gender Ideology Extremism and Restoring Biological Truth to the Federal Government."
Policy 'Whiplash'
Research Grants Increasingly Require Compliance With Trump’s Orders. Here’s How Colleges Are Responding.
Photo illustration showing internal email text snippets over a photo of a University of Iowa campus quad
Red-state reticence
Facing Research Cuts, Officials at U. of Iowa Spoke of a ‘Limited Ability to Publicly Fight This’
Photo illustration showing Santa Ono seated, places small in the corner of a dark space
'Unrelentingly Sad'
Santa Ono Wanted a Presidency. He Became a Pariah.
Illustration of a rushing crowd carrying HSI letters
Seeking precedent
Funding for Hispanic-Serving Institutions Is Discriminatory and Unconstitutional, Lawsuit Argues

From The Review

Football game between UCLA and Colorado University, at Folsom Field in Boulder, Colo., Sept. 24, 2022.
The Review | Opinion
My University Values Football More Than Education
By Sigman Byrd
Photo- and type-based illustration depicting the acronym AAUP with the second A as the arrow of a compass and facing not north but southeast.
The Review | Essay
The Unraveling of the AAUP
By Matthew W. Finkin
Photo-based illustration of the Capitol building dome propped on a stick attached to a string, like a trap.
The Review | Opinion
Colleges Can’t Trust the Federal Government. What Now?
By Brian Rosenberg

Upcoming Events

Plain_Acuity_DurableSkills_VF.png
Why Employers Value ‘Durable’ Skills
Warwick_Leadership_Javi.png
University Transformation: a Global Leadership Perspective
Lead With Insight
  • Explore Content
    • Latest News
    • Newsletters
    • Letters
    • Free Reports and Guides
    • Professional Development
    • Virtual Events
    • Chronicle Store
    • Chronicle Intelligence
    • Jobs in Higher Education
    • Post a Job
  • Know The Chronicle
    • About Us
    • Vision, Mission, Values
    • DEI at The Chronicle
    • Write for Us
    • Work at The Chronicle
    • Our Reporting Process
    • Advertise With Us
    • Brand Studio
    • Accessibility Statement
  • Account and Access
    • Manage Your Account
    • Manage Newsletters
    • Individual Subscriptions
    • Group and Institutional Access
    • Subscription & Account FAQ
  • Get Support
    • Contact Us
    • Reprints & Permissions
    • User Agreement
    • Terms and Conditions
    • Privacy Policy
    • California Privacy Policy
    • Do Not Sell My Personal Information
1255 23rd Street, N.W. Washington, D.C. 20037
© 2025 The Chronicle of Higher Education
The Chronicle of Higher Education is academe’s most trusted resource for independent journalism, career development, and forward-looking intelligence. Our readers lead, teach, learn, and innovate with insights from The Chronicle.
Follow Us
  • twitter
  • instagram
  • youtube
  • facebook
  • linkedin