The National Research Council released on Thursday a revised edition of its 2010 rankings of American doctoral programs that corrects four types of errors discovered in the original report, which was issued last September. But the new rankings do not deal with certain other concerns that scholars have raised about the project.
In the revised edition, almost all programs’ positions on the council’s “ranges of rankings” have changed at least slightly, but in most cases the changes are not substantial. In a few academic fields, however, the numbers have changed significantly for at least 20 percent of the programs. Those include geography, linguistics, and operations research.
A spreadsheet of the new rankings is available for download at the council’s Web site. The council has also released a separate, much smaller spreadsheet that summarizes the changes in programs’ “R” and “S” rankings. (R rankings reflect how similar a program is to the programs in its field with the strongest reputations. S rankings more directly reflect a program’s performance on variables that scholars in the field say are most important, such as faculty research productivity or student diversity.)
The new edition makes four kinds of corrections. The original report in many cases undercounted faculty members’ honors and awards, the proportion of new graduates who find academic jobs, and the proportion of first-year students who are given full financial support. In nonhumanities fields, the report also used faulty data for faculty members’ 2002 publications, which in turn caused errors in calculations of citation counts.
Big Changes for One Program
One program that fares significantly better in the revised report is Cornell University’s doctoral program in French language and literature. In the original report, that program had an R-ranking range from 12 to 30 (meaning that there is a 95-percent probability that the program is between the 12th-best and the 30th-best of the country’s 43 doctoral programs in French). Its S-ranking range was between 16th-best and 30th-best in the country. In the new report, Cornell’s R-ranking range is from 7 to 21, and its S-ranking range is from 11 to 26.
What changed? The number of awards and honors per faculty member has been revised upward from 0.52 to 1.11. And the percentage of new graduates with academic jobs has been revised upward from 62.5 percent to 100 percent.
Those two changes were enough to propel the estimated rankings of Cornell’s French program up substantially. That illustrates how sensitive the council’s model is to relatively modest variations, especially in fields, like French, where the overall number of programs is small. That kind of sensitivity is why Stephen M. Stigler, a professor of statistics at the University of Chicago, has criticized the project. He believes that the ranges of R- and S-rankings do not carry much meaning if tiny changes in a single variable or two can cause large swings in the rankings.
Other programs with significant upward movement in the revised edition include the applied-mechanics program at the California Institute of Technology, the biomedical-engineering program at the City University of New York, the molecular-genetics program at the University of Chicago, and the applied-linguistics and philosophy programs at the University of California at Los Angeles.
One program that The Chronicle highlighted last fall for its unexpectedly strong rankings—the physics department at the University of Hawaii-Manoa—still scores well in the revised edition. In fact, its R- and S-rankings have improved slightly in the new version.
Intensity of Interest
Will graduate-program directors greet the revisions with the same intensity of interest that the original report drew last fall? Lou McClelland, director of institutional analysis at the University of Colorado at Boulder, believes not. She said in an interview on Thursday that she had notified program directors at her campus on Wednesday about the coming revisions, but no one replied to her message.
The new revisions do not address some of the most widespread concerns aired last fall about the council’s report, including complaints about incorrect lists of faculty members and the NRC’s decision not to count book-length publications in the social sciences.
Andrew Bernat, executive director of the Computing Research Association, said in an interview on Thursday that the revisions did little to satisfy his organization’s discomfort with the rankings. He believes that the council used faulty calculations when it tallied computer scientists’ presentations at conferences, which are a prestigious form of scholarly communication in his field.
“I have the utmost respect for the National Academies,” Mr. Bernat said. “But this report was done very poorly. They took on a very difficult, perhaps an impossible, problem, by trying to compute rankings in a concrete way across all of these fields at once. But that doesn’t absolve them of the problems in their data collection.”
The Chronicle will soon update and correct its own presentations of the NRC data. The Chronicle‘s interactive data tool will be updated by Friday afternoon, and its summary tables for each academic discipline will be updated next week.
