Analysts Map Some of the Challenges a College-Ratings System Would Face

February 06, 2014

In anticipation of the debate over a proposed new federal ratings system for colleges, two analysts with the American Enterprise Institute have crunched the data to highlight what can be measured today about colleges’ success in achieving the ratings’ goals of improving affordability, quality, and access to students from the lowest socioeconomic classes, all at the same time.

Their conclusion: "Very few institutions are terrible on all three marks," but very few appear to have broken the "iron triangle"—the concept that any change in one of the goals will have an impact on the others. In fact, according to their analysis, only 19 four-year colleges have managed to perform well across all three of the measures. (Four are campuses of the California State University system, and three are part of the City University of New York.)

The two analysts—Awilda Rodriguez and Andrew P. Kelly, of AEI’s Center on Higher Education Reform—write that this "lack of exemplars shows just how rare this kind of well-rounded success is in American higher education" and demonstrates the difficulties ahead for the ratings plan, proposed by President Obama.

"Presumedly, this status quo is what the president hopes to improve on" by linking some federal student aid to how campuses fare in the ratings, they write. "The analysis here suggests it will not be easy."

A report describing their analysis, complete with scatterplots, is being released on Thursday, the same day the U.S. Department of Education will convene a cadre of experts to discuss issues related to the ratings.

How They Crunched the Data

For their analysis, Ms. Rodriguez, a research fellow, and Mr. Kelly, the center’s director, looked at data on the number of Pell Grant recipients enrolled, graduation rates, and net price at more than 1,700 four-year colleges.

They then plotted those results on a graph and found four categories of colleges: a group with high access and low graduation rates, which they call "You Can Get in, but You Can’t Get Out"; a group with middling results in all categories, which accounts for about a third of the sample and which they call the "Muddle in the Middle"; a group with below-average affordability, average rates of access, and above-average completion rates, which they call "Pretty Good but Pretty Expensive"; and a group whose access and affordability are low but completion rates are high, which they classify as places with "Great Outcomes but Restricted Access."

They also note that basing incentives for colleges on student success could end up rewarding institutions for whom they admit rather than for how well they educate their students, and that punishing institutions for enrolling low rates of Pell-eligible students may make them even less likely, or less able, to admit more of them.

Many college leaders have said a poorly designed system could have unintended and unwanted effects on colleges. The authors echo that warning, but with no small hint that their critique is meant to ultimately bolster, not scuttle, a ratings system. An ill-conceived system that does more harm than good, they say, will give critics "the ammunition they need to roll back future efforts to hold colleges accountable."