For some scientists, it’s the height of irony: The scientific method depends on being able to check what others have done. But the growing use of citation statistics and impact factors in academe goes against that basic principle, because researchers say they cannot assess the basic data used to produce those measurements, which are controlled by a private company, Thomson ISI.
“It’s a bit of a paradoxical situation in science, where data are so critical,” says Emilie Marcus, editor of the journal Cell. “If scientists are going to rely on it, it should be verifiable that it’s a valid measure.”
ISI maintains that all citations are listed, and that any subscriber can check on them. But going through thousands of those citations would require a herculean effort.
Another concern is that impact factors capture only one type of activity: published citations to a journal. A paper that clinicians find useful might not generate many citations but nonetheless represent an important work, says Fiona Godlee, editor of BMJ.
As the disenchantment over impact factors grows, some new tools are emerging that will help scientists and administrators evaluate colleagues and their research.
The publisher Current Science Group started an online service three years ago called the Faculty of 1000, which uses a stable of hand-picked specialists to rate new biology papers. A group of about 2,000 scientists evaluates the most important papers they read each month, from some 800 journals, says Vitek Tracz, chairman of Current Science Group, who started the project.
The experts assess a paper and give it a numerical rating, which is used to calculate a Faculty of 1000 factor. Unlike impact factors, the Faculty of 1000 factor “is calculated from a value judgment given by identified experts,” says Mr. Tracz, “and the reason for their value judgment can be traced.”
“We know from Faculty of 1000 that there are many important papers that get published in journals with low impact factors,” he says. The individual subscription in 2006 for the service will be about $100 per year; the average university site license will be close to $3,700. Mr. Tracz’s company expects to start Faculty of 1000 Medicine early next year.
Mr. Tracz’s other enterprise, a group of more than 100 open-access journals published by BioMed Central, recently developed another rating model. Last month the company’s Web site started labeling papers “highly accessed” if they receive a large number of hits — a distinction that will always apply to the paper and hence could be placed on a CV.
Clinicians seeking to stay on top of medical developments can participate in a new service run by McMaster University, in Ontario, called the McMaster Online Rating of Evidence, or MORE.
The system comprises 2,100 practicing doctors, who rate articles on the basis of relevance and newsworthiness. If a paper gets a sufficient number of top ratings, MORE sends out an e-mail message about the study to readers who have signed up for news in that particular discipline. The system reaches more than 200,000 people.
Among the newest rankings to appear is the h-index. Jorge E. Hirsch, a professor of physics at the University of California at San Diego, devised this method to quantify individual scientists’ output by determining the highest number of papers each researcher has published that receive the same number of citations.
Thus, for example, a scientist with an h of 100 has published 100 papers, each of which has been cited at least 100 times. Unlike the impact factor, the h-index identifies researchers who publish important work in less-visible journals. And it is far easier to determine from ISI data, he says, than it would be to add up all the citations to a person’s entire publication list.
After calculating the h value for many physicists, Mr. Hirsch provides a rough guide for evaluating faculty members in that discipline: A scientist at a research university should achieve an h of 12 to receive tenure and 18 for promotion to full professor. Getting into the National Academy of Sciences would require an h of 45 for a physicist, he says.
The physicist with the highest h, 110, is Edward Witten, a theorist at the Insitute for Advanced Study, in Princeton, N.J., who has made key advances in string theory. Curiously, many of the physicists at the top of the h rankings have not received Nobel Prizes — yet.
Prize handicappers should take notice.
http://chronicle.com Section: Research & Publishing Volume 52, Issue 8, Page A17