Columbia University is ranked No. 2 among national universities by U.S. News & World Report, behind Princeton University and tied with Harvard University and MIT. But a new analysis claims its lofty position may be based on inaccurate numbers that administrators submitted to the magazine.
Michael Thaddeus, a Columbia professor of mathematics, laid out his case last month in more than 11,000 words on his faculty page. Skeptical of the university’s ranking, he purchased a U.S. News subscription that let him see some of the detailed data upon which the rankings are based. Some of Columbia’s data stuck out to him.
So he did his own digging to try to reproduce the figures submitted to U.S. News. He downloaded data from the university’s online directory of classes, which lists course enrollments, and its directory of faculty members, which notes their degrees.
Among other things, Thaddeus alleges administrators submitted inaccurate data on class sizes, the percentage of full-time faculty with doctorates or other terminal degrees, and how much the university spends on instruction. For many of these data points, U.S. News relies on colleges to self-report, so inaccurate data could result in a warped placement on the ubiquitous list.
Columbia University stood by its numbers in an emailed statement. “We take seriously our responsibility to report information accurately to federal and state entities, as well as to private rankings organizations,” the statement reads. “We consistently work in good faith to answer the hundreds of questions across surveys conducted by U.S. News & World Report and others every year, each with criteria that evolve over time.”
Robert Morse, who develops the methodology and surveys for the U.S. News rankings, said in a statement that the magazine’s staff members “rely on schools to accurately report their data and ask academic officials to verify that data.”
Some discrepancies between Thaddeus’s estimates and Columbia’s official numbers are large. For example, U.S. News reports that 83 percent of classes serving undergraduates at Columbia have fewer than 20 students. Using data from the online directory of classes, Thaddeus estimated the true proportion is between 63 percent and 67 percent. The difference adds up to thousands of classes.
Columbia disputed that finding. “The Directory of Classes on which Professor Thaddeus relied is not an official record of enrollment certified by Columbia’s registrar,” the university statement said. “Conclusions he’s drawn from the class directory should be expected to diverge from our official count.”
U.S. News lists Columbia as having the ninth-highest spending per student, out of the 392 colleges it categorizes as “national universities.” Part of that high spending is that the university counts patient care under instructional costs. That is an unfair categorization of the funds and diverges from the practice of other reporting colleges, such as New York University, Thaddeus argued in his analysis.
The university responded that patient care does count as instruction at Columbia because it’s provided by medical-school faculty members who may be training students at the same time. Costs at the NewYork-Presbyterian/Columbia University Irving Medical Center not incurred by medical-school faculty aren’t included in the U.S. News financial data, the Columbia statement said.
The Chronicle sent Thaddeus’s work to Tim Chartier, a professor of mathematics at Davidson College who previously attempted to reverse-engineer U.S. News’s ranking formula, for a broad review. Whether or not his exact figures are correct, Thaddeus’s analysis pointed to some important questions about the data Columbia submits to U.S. News, Chartier said.
“Michael recognizes his analysis may not be foolproof,” Chartier wrote in an email. “The fundamental issue isn’t the absolute accuracy of his numbers. One has to see a fundamental, rather blaring flaw, for the numbers to replicate the U.S. News & Report numbers.”
He explained an example. Columbia’s percentage of under-20-student classrooms is higher than any other institution in the top 100 national universities, and four percentage points higher than the two runners-up. Those exceptional numbers, in addition to how much lower Thaddeus’s estimate is, “does make one stop and want to ask what’s under the hood of Columbia’s calculation,” Chartier wrote. In response to this claim, Columbia pointed to small classes that all undergraduates must take as part of a core humanities curriculum.
Thaddeus said he hopes for two major outcomes from his analysis. The first is an independent investigation of Columbia’s numbers. The second is to undermine the U.S. News rankings altogether.
“Universities have to report data, but they know they have a strong interest in the data looking favorable, so that’s a conflict of interest,” Thaddeus said. “U.S. News wants the data to be accurate, but they also want to look like a professional operation themselves. Any operation they take that reveals inaccurate, or, even worse, fraudulent data discredits them.”
The idea of one all-encompassing ranking doesn’t serve potential students well, Thaddeus argued. “The whole concept of the university ranking is a disastrous idea because everybody needs something different,” he said. Better for colleges to make data such as graduation rates, debt load, and class sizes available, he said, so students can look up what’s important to them.
Thaddeus began teaching at Columbia in 1998, when the university was ranked around ninth, as it had been for many years. He was motivated to do his analysis as he watched that ranking rise over the last two decades. “There was just this gap between that ascent and the reality in my mind, which is that Columbia has a lot of brilliant faculty and students, but we’re all placed together in this physical setting that is historic and beautiful but severely, severely overcrowded,” he said. (He has opposed the administration’s plans to expand Columbia’s undergraduate population.)
“It’s a matter of perception versus reality,” he said. “Administration is very interested in improving the way that Columbia is perceived, but not that interested in improving the way it really is.”
