Johns Hopkins and Texas A&M Were the Least Productive Research Universities, Study Finds

Researchers at the Johns Hopkins and Texas A&M Universities were among the least productive of any at large academic institutions from 1989 to 2004, measured by the money spent on their research per paper they published, according to a study to be presented at an academic conference this week.

The University of Pennsylvania and Harvard University were among the most productive.

By the same measure, private research universities were more efficient on average than public ones, and institutions with medical schools produced more papers than those without.

What’s more, the collective productivity of 72 of the nation’s largest research universities dropped significantly in the late 1990s and has not recovered, according to the study’s author, Jeffrey M. Litwin, an associate dean at George Brown College, in Toronto.

Mr. Litwin is to present his findings on Tuesday at the annual meeting of the Association for Institutional Research, which will take place in that city.

The study appears to be the first of its kind to rank institutions by this measure, which compares a university’s total research spending from all sources against the number of papers its researchers publish in scholarly journals. Other prominent methods of evaluating research institutions, like the one released last year by the National Research Council, have considered, as one of many factors, a different but related yardstick—the number of papers published per faculty member.

However, at least one statistician disagrees with the spending-per-paper approach. Mr. Litwin relied on data that are “inadequate” to evaluate universities’ research productivity accurately, said Scott L. Zeger, vice provost for research at Johns Hopkins.

In an interview, Mr. Litwin called his method an improvement over counts of publications per faculty member. That approach is problematic given the “vast differences in outputs per capita across disciplines, and even within individual departments,” said Mr. Litwin, who is executive director of the college’s capital campaign and holds an M.B.A. and a doctorate in higher-education management.

Figures on research expenditures offer a way to smooth the variation and produce a single measure reflecting faculty efforts on research that is comparable across institutions, Mr. Litwin said. “I think it’s quite a bit more useful for studies in a financial or economic context,” he said.

A Drop in Productivity

For his study, Mr. Litwin examined expenditures data reported by institutions to the National Science Foundation from 1989 to 2004. He examined counts of published studies compiled by Thomson Reuters, a news and information company. His results are based on 72 institutions that reported the highest research expenditures and that had comparable data on publications. The median expenditure per paper was $72,020.

He acknowledged that his study raised several questions that it did not answer, which he says merit further study. One was a jump in average expenditures per paper among all institutions, from $67,000 in 1996 to $85,000 in 2002.Mr. Litwin said neither he nor his colleagues had an explanation. Nevertheless, he said, “If you have a decline in productivity, the government isn’t getting the same return for its investment, assuming the quality of publications has been consistent,” he said, “and that’s meaningful.”

Mr. Litwin conceded that his suggested approach had methodological limitations. The NSF figures on research expenditures fail to capture all of a faculty member’s time spent on research because they reflect only research that is separately budgeted by the institution, such as that financed by external sponsors like the federal government. At research institutions, faculty members typically split their time between teaching and research, but the portion of their base pay that represents compensation for their research was not captured in the NSF figures.

Mr. Zeger, who has overseen research at Hopkins since 2008 and whose specialty is biostatistics, suggested some other limitations of Mr. Litwin’s measure.

Hopkins’s designation as the least-productive institution may reflect that about half of the university’s reported expenditures were attributable to its research affiliate, the Applied Physics Laboratory, he said. (Including the money from the physics laboratory, Hopkins led all institutions in 2009 with total research expenditures of $1.86-billion.)

Scientists in the Applied Physics Laboratory do work financed by the U.S. Defense Department, and so not all of their scholarly work would necessarily show up in the Thomson Reuters counts of publications, Mr. Zeger said.

What’s more, some institutions may have looked more productive simply because the mix of academic disciplines there was skewed toward fields, like genetics, in which scholars typically publish a lot of papers, Mr. Zeger suggested.

Mr. Litwin’s paper acknowledges this point by suggesting that individual institutions compare themselves against peers that have a similar mix of research expenditures by discipline, For example, the University of Pittsburgh and Duke University were similar in disciplinary mix but different in expenditures per paper: $61,740 at Pittsburgh versus $81,600 at Duke in the year 2000.

Counts of publications “are too simplistic to capture the richness of what we mean by research productivity,” Mr. Zeger aid. To evaluate that requires keeping an eye on faculty members’ volume of publication but also making professional judgments about the importance of research results, he said.

Most and Least Productive Research Universities by Spending per Paper, 1989-2004