Alternative metrics have great potential to help track the real-world influence of research, but they’re only as good as the data sources they draw on. And those data sources are a motley crew of open and closed, public and private. Services that want to offer a wide array of metrics operate in a hybrid environment of commercial and nonprofit enterprises where the balance between open and closed can shift at any time.
Look at the anxiety created when Mendeley, a reference-management service with some 2.3 million users, was recently acquired by the commercial publisher Elsevier. Since it was founded in 2007, Mendeley has become part of the working lives of many researchers. That makes it “an important source of data, because it has made great inroads,” says Heather A. Piwowar, co-founder of ImpactStory, a nonprofit that created a Web tool to capture a variety of metrics, including altmetrics. Most papers have not been tweeted or blogged about, she says, but many do get saved in Mendeley or other bookmarking-and-reference managers like the nonprofit Zotero, based at the Center for History and New Media at George Mason University, or CiteULike. Both are also widely used by researchers.
That makes them attractive to a service like ImpactStory, which aims to “make metrics flow like water,” says Ms. Piwowar, whose own research has focused on the availability and reuse of research data. Jason Priem, the project’s co-founder and a doctoral candidate in information science at the University of North Carolina at Chapel Hill, says that ImpactStory wants to help researchers gather “a record of what they’ve done, the impacts of what they’ve done, in an open way.”
Like other companies and organizations that collect altmetrics, ImpactStory pulls data from a large number of sources. Those include Scopus, the huge Elsevier-owned database of journal abstracts and citations; PubMed Central, which contains more than 21 million citations in biomedical fields, and the PMC full-text archive of biomedical and life-sciences journal articles run by the National Institutes of Health and the National Library of Medicine; Facebook; Topsy, a tweet-tracking service; the free online software repository GitHub; and article-level metrics from the open-access publisher PLOS, which give detailed data on individual articles in PLOS journals, down to how many individual IP addresses have viewed a PLOS paper in the PubMed Central database.
Citations present the biggest challenge. “That’s very much a problematic area in terms of getting good data,” Mr. Priem says. “The upshot is that most of the citation databases are still closed.” According to Ms. Piwowar, Scopus allows its citation counts to be displayed on the ImpactStory Web site but won’t let its data be redistributed beyond that; Google Scholar and Thomson Reuters’s Web of Science do not allow ImpactStory to display their citation counts at all. Having rich sources of data walled off frustrates efforts to build new evaluation tools. Mr. Priem calls Google Scholar “a black box that delivers really wonderful things—the most comprehensive available citation numbers—but at the end of the day, it’s still a black box."Both Ms. Piwowar and Mr. Priem are ardent open-access advocates—there’s a good deal of overlap between the altmetrics and open-access communities—and the ImpactStory tool is Web-based and open source. They say that whenever they can, they use open data and work as transparently as possible.
A nonprofit, ImpactStory got its start with a $125,000 grant from the Alfred P. Sloan Foundation, and its founders have applied for a second round of Sloan money. They would like to support the project long-term with revenue from enhanced services for researchers in addition to the basic, free tool. They’re also investigating a site-license model designed for university libraries.
The founders see altmetrics as an essential element in a grass-roots-driven push to rethink “the reward system and the way we think of publishing as making public rather than capital-P publishing,” Mr. Priem says. Although there’s a lot of interest from journal publishers in such tools, ImpactStory plans to keep the focus on individual researchers because “we feel like that’s where the long-term change is going to come.”
Money Is a Factor
While scholars tend not to think in commercial terms, money does factor into the altmetrics conversation. Some altmetrics advocates emphasize a philosophical imperative to help scholarly communication evolve; some see this evolutionary moment as a business opportunity, too. Commercial publishers as well as nonprofits have been incorporating open access into their business models and, as Elsevier’s acquisition of Mendeley suggests, they understand that survival and profitability depend on integrating themselves into the life cycle of research.
As is the case with academic libraries, a healthy future for scholarly publishers depends to some extent on being able to offer services tailored to individual needs. Journals still have their place, but “the future is not in selling things that are infinitely reproducible,” Mr. Priem says. He points to the music industry, where live shows have remained a hot ticket even as record sales decline. “It’s still easy to sell a concert. In the same way, I think it’s going to be easy to sell personalized recommendations online,” he says. Altmetrics has the potential to be able to point researchers toward work that’s of direct interest to them and to show publishers and librarians how their authors’ and patrons’ work is faring in different arenas.
That potential raises some concerns among the providers of altmetrics who rely on data sources that may or may not remain open. “One issue the altmetrics community has is that we’re relying on data from third-party providers,” says Euan Adie, founder of Altmetric, a company whose Web tools help publishers and others track online metrics for scholarly articles. “You can have open intentions, but we’re all collecting data from places that maybe don’t. Twitter and Mendeley are friendly commercial organizations, which is great, but helping people to get tenure for free probably isn’t in their business plan.”
The solution for now is to proceed with eyes open. “If we do want to build well-accepted metrics using data from third parties, then we need to be able to compensate for those companies potentially going away or changing their business model, and the wider altmetrics community losing access,” Mr. Adie says.
The pull of commercialism won’t necessarily trump the commitment to openness associated with altmetrics, but business pressures are real and likely to get stronger. “This is a competitive field, and I think it’s one that’s going to grow very quickly as they tap into the zeitgeist of ‘Let’s measure everything,’” says ImpactStory’s Mr. Priem. “There’s tremendous incentives for companies to own the data sources for altmetrics.”
