Evaluating scientists by their journal-citation counts is a much-despised shorthand at many universities. And so, when a study came out two years ago suggesting Twitter mentions as a supplement or even an alternative to citation counts, it got a fair bit of attention.
Now, according to a much more exhaustive analysis of the question, it appears that any correlation between tweets and citations was greatly overstated.
The previous study, from the University of Toronto, found that highly tweeted articles were almost 11 times as likely as less-tweeted articles to be highly cited. But the study involved only 55 articles in a single journal. The new study, by scientists at the University of Montreal, involved some 1.4 million articles from more than 5,000 journals. It found no meaningful correlation.
“We found very much contradictory results,” said the lead author, Stefanie Haustein, a postdoctoral researcher at the University of Montreal’s School of Library and Information Science.
Her article, “Tweeting Biomedicine: An Analysis of Tweets and Citations in the Biomedical Literature,” was recently published online by the Journal of the American Society for Information Science and Technology.
Ms. Haustein’s review considered all articles and reviews indexed in PubMed and the Web of Science over the three years ending in 2012. Of the 15 most-tweeted articles, each prompting more than 260 tweets, only four had more than a dozen citations by other scientists and four had none at all.
The smaller 2011 study—by Gunther Eysenbach, an associate professor at the University of Toronto’s Institute of Health Policy, Management, and Evaluation—won coverage in major news outlets with the opposite message. But almost immediately, the implied suggestion that researchers pay more attention to Twitter began attracting skeptics.
In addition to their small number, the articles in Dr. Eysenbach’s study all came from the Journal of Medical Internet Research, which itself uses Twitter to promote its articles. Some reviews suggested that his research may have been affected by his ties to the journal—he disclosed in the study that he is editor, publisher, and a shareholder of the journal—and his well-known advocacy of replacing or augmenting journal citations with Twitter and other alternative measures of scientific value.
Indicators of Social Impact
Dr. Eysenbach said on Monday that he saw little in Ms. Haustein’s report to contradict his findings. Both studies, he said, affirm that tweets reflect social impact, “which is exactly what I proposed.”
Ms. Haustein said she agreed that tweets can reflect social impact. Even though she found no evidence that they reflect future article-citation levels, she said, Twitter may provide universities and others with valuable information about how scientific research is valued by the public at large.
And Twitter does provide reaction on a much faster time scale than citations—days rather than years—so it might be possible someday to use tweets to help judge the value of research findings to future scientists if confounding factors can be identified and weighted, she said.
Those factors could include the degree to which journals or authors tweet their own work, or the degree to which the public sharing of a research finding is driven by curiosities not strictly related to scientific value. As examples, she noted that some of the 15 most-tweeted articles in her study concerned such topics as the likelihood of penile fracture during stressful sexual encounters, and the case of a semi-cooked squid that injected its sperm bag into the tongue of a Korean diner.
“There’s funny stuff in there that people don’t tweet for the scientific content, but because it’s interesting,” she said.
It’s also hard to tease out the effects of marketing, said a co-author, Cassidy R. Sugimoto, an assistant professor of library and information science at Indiana University at Bloomington. That further discounts the likelihood that tweets reflect some broad societal importance of studies appreciated “outside of the ivory tower,” Ms. Sugimoto said.
Of the 13 journals in which more than 50 percent of the publication’s articles were tweeted, nine had an official Twitter account and the other four were represented by official Twitter accounts of their publishers or affiliated societies, the study found.
