Skip to content
ADVERTISEMENT
Sign In
  • Sections
    • News
    • Advice
    • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Student Success
    • Technology
    • Transitions
    • The Workplace
  • Magazine
    • Current Issue
    • Special Issues
    • Podcast: College Matters from The Chronicle
  • Newsletters
  • Events
    • Virtual Events
    • Chronicle On-The-Road
    • Professional Development
  • Ask Chron
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Professional Development
    • Career Resources
    • Virtual Career Fair
  • More
  • Sections
    • News
    • Advice
    • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Student Success
    • Technology
    • Transitions
    • The Workplace
  • Magazine
    • Current Issue
    • Special Issues
    • Podcast: College Matters from The Chronicle
  • Newsletters
  • Events
    • Virtual Events
    • Chronicle On-The-Road
    • Professional Development
  • Ask Chron
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Professional Development
    • Career Resources
    • Virtual Career Fair
    Upcoming Events:
    College Advising
    Serving Higher Ed
    Chronicle Festival 2025
Sign In
Technology

Rise of ‘Altmetrics’ Revives Questions About How to Measure Impact of Research

By Jennifer Howard June 3, 2013
Steven Roberts, an assistant professor at the U. of Washington who studies how environmental change affects shellfish, tracks social-media metrics to see how his research is used online.
Steven Roberts, an assistant professor at the U. of Washington who studies how environmental change affects shellfish, tracks social-media metrics to see how his research is used online.Matthew Ryan Williams for The Chronicle

Steven B. Roberts’s 103-page tenure package features the usual long-as-your-arm list of peer-reviewed publications. But Mr. Roberts, an assistant professor at the University of Washington who studies the effects of environmental change on shellfish, chose to add something less typical to his dossier: evidence of his research’s impact online.

To continue reading for FREE, please sign in.

Sign In

Or subscribe now to read with unlimited access for as low as $10/month.

Don’t have an account? Sign up now.

A free account provides you access to a limited number of free articles each month, plus newsletters, job postings, salary data, and exclusive store discounts.

Sign Up

Steven B. Roberts’s 103-page tenure package features the usual long-as-your-arm list of peer-reviewed publications. But Mr. Roberts, an assistant professor at the University of Washington who studies the effects of environmental change on shellfish, chose to add something less typical to his dossier: evidence of his research’s impact online.

He listed how many people viewed his laboratory’s blog posts, tweeted about his research group’s findings, viewed his data sets on a site called Figshare, downloaded slides of his presentations from SlideShare, and otherwise talked about his lab’s work on social-media platforms. In his bibliography, whenever he had the data, he detailed not only how many citations each paper received but how many times it had been downloaded or viewed online. The strategy was part of “an attempt to quantify online science outreach,” he explained in his promotion package.

Mr. Roberts can’t say for sure that including the digital footprint of his research—captured in part with alternative metrics, or “altmetrics,” like those listed above—helped him in his bid for tenure. But it certainly didn’t hurt. He won a promotion to associate professor in the School of Aquatic and Fishery Sciences at the university’s College of the Environment.

Adding altmetrics to CVs and dossiers may not be common yet. But interest in altmetrics is growing fast, as scholars begin to realize that it’s possible to track and share evidence of online impact, and publishers and new start-up companies rush to develop altmetric services to help them document that impact.

The term “altmetrics” has only been around since 2010, when Jason Priem, a doctoral candidate at the School of Information and Library Science at the University of North Carolina at Chapel Hill, first used it in, fittingly enough, a tweet. That led to an influential manifesto written by Mr. Priem and three other researchers. which pointed out the limitations of traditional filters of quality like article citations and the journal impact factor. Those take months or years to bubble up; altmetrics can be collected fast, letting researchers see, almost in real time, how an article or data set or blog post is moving through all levels of the scholarly ecosystem.

That appeals to researchers like Mr. Roberts, interested in sharing scholarship quickly and openly to speed the flow of ideas, in keeping with the philosophy of the open-science and open-access movements. Even his lab’s research notebooks are posted online, so colleagues can see one another’s work as it progresses.

But skeptics and some observers wonder whether blog posts and tweets and other social-media activity are sophisticated and reliable enough to capture true impact, which is a slippery concept to begin with. Nick Scott, digital manager at the Overseas Development Institute, observed in a post on a London School of Economics and Political Science blog last December that online reach and real impact—which he defined as “change in the world"—were not necessarily the same thing. “How do we compare tweets, Facebook likes,” and other uncertain votes of confidence? he asked.

Some also worry that altmetrics can be easily gamed. A much-talked-about paper published last year tested how difficult it was to manipulate the metrics in Google Scholar, a free, much-used service that compiles citation data for academic output, potentially competing with commercial bibliometric databases like Thomson Reuters’s Web of Science and Elsevier’s Scopus. The article’s authors created six papers by a fake author, uploaded them to a Web site, and tracked the resulting citations. They concluded that it’s “simple, easy, and tempting” to game the system. Altmetrics supporters acknowledge that gaming is a risk but point out that any kind of metric is vulnerable to corruption. Journals have been called out over the years for inflating their citation rates and thereby their impact factors, for instance.

More pressing is the question of who controls the sources of these data on scholarly impact online, especially as altmetrics become more sophisticated, reliable, and widespread. Those questions grew louder this spring when the publishing giant Elsevier bought Mendeley, a popular reference-management platform where scholars store and share articles. Mendeley is also a hub for group discussions focused on specific research topics and interests—the kind of online activity that altmetrics proponents envision being harnessed as a kind of early-detection system that will pick up on promising new work and trends in a field.

And then there’s the threat that altmetrics could be co-opted or misused by institutional assessors inclined to rely on numbers rather than on more nuanced indicators of quality when judging the worth of professors, research groups, or departments. “When I talk to administrators, they say there’s a huge pressure to be more quantitative,” says Mr. Priem.

Age-Old Debate

The larger conversation about how to measure scholarly impact is probably as old as scholarship itself. Altmetrics use has been most notable so far among scientists and librarians, for whom “quant culture” has long been a fact of life. Jason Baird Jackson, director of the Mathers Museum of World Cultures at Indiana University at Bloomington, says that metrics can be harder for humanists to understand or get behind.

ADVERTISEMENT

“In many humanities fields, those scholars have intuitions and beliefs about the most important journals,” Mr. Jackson says, but they don’t know much about impact factors. “They don’t know which to be more nervous about,” altmetrics or all metrics. “Any kind of metric entails the risk of promoting short-sightedness,” he says. “I think the humanists are particularly sensitive to this.”

Mr. Jackson invokes predigital conversations that folklorists and museum-based anthropologists have long had about how to measure the scholarly impact of, say, exhibitions or other scholarly output that doesn’t fit a traditional academic mold. At Indiana, he has helped lead a series of campus conversations that touched not just on altmetrics but on related issues like how to rewrite tenure-and-promotion guidelines to better reflect shifts in how scholars conduct and share their work.

In the last year, altmetrics has become “a serious matter that people are getting their head around,” Mr. Jackson says. “For many of our department chairs, this is a totally new world.”

Stacy Rose Konkiel, a science data-management librarian at Bloomington, agreed that what’s lagging now is faculty awareness and trust. “Campuswide there’s a little sensitivity toward measuring faculty output,” she says. Altmetrics can reveal that nobody’s talking about a piece of work, at least in ways that are trackable—and a lack of interest is hardly something researchers want to advertise in their tenure-and-promotion dossiers. “What are the political implications of having a bunch of stuff online that nobody has tweeted about or Facebooked or put on Mendeley?”

ADVERTISEMENT

The library at Indiana has been quietly exploring how to do more with altmetrics, operating on the principle that “altmetrics can just be a faster and more reliable way to measure public reaction to output that Indiana faculty have produced,” Ms. Konkiel says. But skepticism and the old ways make that a hard sell in some quarters. “The folks I’ve talked to are like, ‘Yes, it does have some value, but in terms of the reality of my tenure-and-promotion process, I have to focus on other things,’” she says.

Publishers Jump In

Publishers over all need less convincing. Some, like the open-access giant PLOS, have well-developed efforts to track usage of articles they publish (often called “article-level metrics”). John Wiley & Sons just started a trial with Altmetric, a publisher-oriented service that collects data from social-media sites and reference managers and creates an Altmetric score that attempts to pull all that information together. Different sources of data are given different weights; as the Altmetric Web site explains, “a newspaper article contributes more than a blog post which contributes more than a tweet.” Altmetric also provides embeddable color-coded graphic representations, called “donuts,” that reveal specific social-media uptakes, downloads, or mentions for each article.

Such feedback can be useful for editors as well as for researchers.

Martijn Roelandse is publishing editor for neuroscience at Springer, one of the largest commercial scientific publishers. He is also a member of Springer’s Social Lab, a social-media task force.

ADVERTISEMENT

Springer publishes more than 2,200 journals, some 325 of them open access, according to Mr. Roelandse. The company “is changing from a sole focus on the journal impact factor to providing multiple metrics” to authors and editors, Mr. Roelandse told The Chronicle in an e-mail interview. It uses Altmetric for what he calls “social metrics,” the nonprofit content-linking service CrossRef to gauge citations, and its own download statistics to get a quantitative sense of how much use an article is getting. For Springer journals’ editorial boards, “we now provide in-depth insights on the impact factor, citations, downloads, and social mentions to sketch a broader picture of the journal.”

Like many of the people now experimenting with altmetrics, Mr. Roelandse sees them as complementary to the impact factor, not a replacement for it. “Altmetrics are a wonderful means of highlighting those articles that performed very well within a journal,” he says. But the impact factor will continue to be a benchmark of journal quality, he adds.

As scholars, librarians, and administrators figure out how to combine altmetrics with traditional measures of reach and quality, altmetrics pioneers have been busy the last few months building tools to serve those different groups. Along with Altmetric, another leader in the new field is Plum Analytics, which is several months into a pilot project with the Smithsonian Institution and the University of Pittsburgh library. Yet another mover in this expanding space is Academia.edu, where researchers can create profiles, upload papers, and track readership and use.

One example of an altmetrics provider is ImpactStory, an open-source, Web-based tool created by Mr. Priem and Heather A. Piwowar, two of the most active leaders of the burgeoning altmetrics movement. (Ms. Piwowar until recently was a postdoctoral research associate with Duke University and the University of British Columbia, studying the availability and reuse of research data.) Professors can add an ImpactStory widget on their own Web pages to get live altmetrics for papers and other research products. “I love it because it’s easy, too. It doesn’t take much effort,” says Mr. Roberts, of the University of Washington.

ADVERTISEMENT

The widget creates badges that show the different ways a research object—a journal article or a blog post or a SlideShare presentation—has been tapped by users. For instance, a listing on his lab’s Web site for a 2012 paper Mr. Roberts co-wrote and published in the open-access megajournal PLOS ONE includes a badge that describes it as “highly saved,” with 18 readers adding it to their Mendeley libraries. That’s better than 90 percent of the items indexed in 2012 by the Thomson Reuters product Web of Science, according to the ImpactStory assessment, “suggesting it’s highly cited by scholars.”

The article, about the development of resources for genomic sequencing of Pacific herring, also did well on the tweet-tracking service Topsy; it was tweeted more times than 97 percent of the items that were indexed in 2012 by Web of Science, “suggesting it’s highly discussed by the public.”

As that phrasing indicates, altmetrics data can’t reveal everything. Mr. Roberts points out that if someone tweets about a paper, “they could be making fun of it.” If a researcher takes the time to download a paper into an online reference manager like Mendeley or Zotero, however, he considers that a more reliable sign that the work has found some kind of audience. “My interpretation is that because they downloaded it, they found it useful,” he says.

Correction (6/4/2013, 5 p.m.): The original version of this article misstated the number of journals published by Springer. The correct number is more than 2,200, not more than 3,200. The text has been corrected.

We welcome your thoughts and questions about this article. Please email the editors or submit a letter for publication.
Tags
Technology
Share
  • Twitter
  • LinkedIn
  • Facebook
  • Email
About the Author
Jennifer Howard
Jennifer Howard, who began writing for The Chronicle in 2005, covered publishing, scholarly communication, libraries, archives, digital humanities, humanities research, and technology.
ADVERTISEMENT
ADVERTISEMENT

Related Content

New Metrics Providers Help Keep Libraries in the Research Game
Reliance on 3rd-Party Data Creates Uncertainty for ‘Altmetrics’ Providers

More News

Pro-Palestinian student protesters demonstrate outside Barnard College in New York on February 27, 2025, the morning after pro-Palestinian student protesters stormed a Barnard College building to protest the expulsion last month of two students who interrupted a university class on Israel. (Photo by TIMOTHY A. CLARY / AFP) (Photo by TIMOTHY A. CLARY/AFP via Getty Images)
Campus Activism
A College Vows to Stop Engaging With Some Student Activists to Settle a Lawsuit Brought by Jewish Students
LeeNIHGhosting-0709
Stuck in limbo
The Scientists Who Got Ghosted by the NIH
Protesters attend a demonstration in support of Palestinian activist Mahmoud Khalil, March 10, 2025, in New York.
First-Amendment Rights
Noncitizen Professors Testify About Chilling Effect of Others’ Detentions
Photo-based illustration of a rock preciously suspended by a rope over three beakers.
Broken Promise
U.S. Policy Made America’s Research Engine the Envy of the World. One President Could End That.

From The Review

Vector illustration of a suited man with a pair of scissors for a tie and an American flag button on his lapel.
The Review | Opinion
A Damaging Endowment Tax Crosses the Finish Line
By Phillip Levine
University of Virginia President Jim Ryan keeps his emotions in check during a news conference, Monday, Nov. 14, 2022 in Charlottesville. Va. Authorities say three people have been killed and two others were wounded in a shooting at the University of Virginia and a student is in custody. (AP Photo/Steve Helber)
The Review | Opinion
Jim Ryan’s Resignation Is a Warning
By Robert Zaretsky
Photo-based illustration depicting a close-up image of a mouth of a young woman with the letter A over the lips and grades in the background
The Review | Opinion
When Students Want You to Change Their Grades
By James K. Beggan

Upcoming Events

07-31-Turbulent-Workday_assets v2_Plain.png
Keeping Your Institution Moving Forward in Turbulent Times
Ascendium_Housing_Plain.png
What It Really Takes to Serve Students’ Basic Needs: Housing
Lead With Insight
  • Explore Content
    • Latest News
    • Newsletters
    • Letters
    • Free Reports and Guides
    • Professional Development
    • Events
    • Chronicle Store
    • Chronicle Intelligence
    • Jobs in Higher Education
    • Post a Job
  • Know The Chronicle
    • About Us
    • Vision, Mission, Values
    • DEI at The Chronicle
    • Write for Us
    • Work at The Chronicle
    • Our Reporting Process
    • Advertise With Us
    • Brand Studio
    • Accessibility Statement
  • Account and Access
    • Manage Your Account
    • Manage Newsletters
    • Individual Subscriptions
    • Group and Institutional Access
    • Subscription & Account FAQ
  • Get Support
    • Contact Us
    • Reprints & Permissions
    • User Agreement
    • Terms and Conditions
    • Privacy Policy
    • California Privacy Policy
    • Do Not Sell My Personal Information
1255 23rd Street, N.W. Washington, D.C. 20037
© 2025 The Chronicle of Higher Education
The Chronicle of Higher Education is academe’s most trusted resource for independent journalism, career development, and forward-looking intelligence. Our readers lead, teach, learn, and innovate with insights from The Chronicle.
Follow Us
  • twitter
  • instagram
  • youtube
  • facebook
  • linkedin