Early Evidence: The College Scorecard Made a Difference, but Only for Some Groups of Students

Mark Gordon stumbled across the College Scorecard this past fall while searching online for information on college outcomes. Mr. Gordon, whose son, Zeb, is a senior in high school, realized it was the same consumer-information website that President Obama had been promoting.

The Scorecard provided the kind of data that Mr. Gordon was looking for. “The schools are very expensive,” he said. “Now is a time when you’re looking for a great education, but you also want to make sure you’re not overpaying for a brand that doesn’t necessarily bring with it any commensurate value.”

Playing around with the Scorecard’s filters brought new colleges to Mr. Gordon’s attention. Zeb, who plans to study computer science and engineering, applied to one of them, the Rose-Hulman Institute of Technology, in Indiana. “They’re not a school that is high-profile,” Mr. Gordon said, but on the Scorecard the college stacked up well against others that are better known.

Ever since the Scorecard — which replaced President Obama’s controversial college-ratings plan — was unveiled, in September, higher-education experts have been digging through the data. They’ve explored what the numbers can tell us about colleges and documenting their limitations. Researchers have certainly been using the Scorecard. But what about the intended audience: people trying to make informed college choices?

So far, the curious have had to draw their conclusions from anecdotes and the Education Department’s tally of the site’s unique users: 1.3 million and still growing steadily, according to Michael Itzkowitz, director of the College Scorecard.

A new study from the College Board seeks to shed more light on the question, providing the first evidence that the Scorecard had an impact on student behavior.

Earnings Data Matter, to Some

The research, conducted by Michael Hurwitz and Jonathan Smith, both policy research scientists at the College Board, studied the patterns of where students sent their SAT scores — a measure of interest in a college and a proxy for applying to it — before and after the Scorecard was released. Mr. Hurwitz and Mr. Smith investigated whether score-sending had been influenced by each of the three main metrics highlighted on the Scorecard: graduation rate, “average annual cost,” and “salary after attending.” The study was designed to look at the effect of the information featured on the Scorecard, whether students encountered it directly on the government site or indirectly (say, in a news article or by word of mouth).

The data on graduation rate and average annual cost did not change score-sending patterns — possibly, the researchers say, because those two metrics were already available.

The earnings data, however, did make a difference. (While there have been other efforts to measure postcollege income, this particular metric was new.) Colleges with higher-than-median earnings saw higher-than-expected growth in scores sent in the months after the September release, while those with lower-than-median earnings had lower-than-expected growth. The researchers estimate that for every 10-percent increase in earnings, the number of scores sent rose 2.4 percent.

But when the researchers dug deeper, they found that only some students had been swayed by the new earnings information. “The impact,” they wrote, “is driven almost entirely by well-resourced high schools and students.” When the researchers looked at parental education, the Scorecard’s impact was concentrated among students whose parents had at least some college. When they looked at race, it was concentrated among students who were white and Asian. And when they looked at school type, it was concentrated among those attending well-resourced public — and even more so, private — schools.

“The subgroups of students expected to enter the college-search process with the most information and most cultural capital,” the researchers wrote, “are exactly the students who responded most strongly to the Scorecard.”

Zeb, who is African-American, is the sort of student who the research suggests might encounter the Scorecard information. He attends St. Christopher’s School, a private school in Richmond, Va. His father is a hospital executive.

A Potential to ‘Diffuse Down’

What about less-advantaged students? There are a couple of possibilities, and because the researchers measured students’ behavior, but not whether or not they actually saw the data, they can’t be sure which is right. It’s possible that the students did not encounter the new earnings data. It’s also possible that they saw the information but that it did not change their behavior, for any number of reasons. But either way, the study raises important questions about whether giving consumers more information might actually widen gaps between advantaged and disadvantaged students — and, if so, what might be done about it.

“I’m sure the effort was not intended to exacerbate inequalities,” Mr. Smith, one of the College Board researchers, said. If the reason low-income students didn’t respond to the information is that they didn’t see it, the patterns might change in the future. Information often reaches the most-advantaged first. “This has the potential to diffuse down,” he said.

While the College Board research suggests students are responding to the new earnings data, “an open question is the process by which students are influenced by this information,” said Lindsay Page, an assistant professor of education at the University of Pittsburgh. Are students going directly to the government website, or are they hearing about how different colleges compare on its measures from their counselors — or, as Zeb did, from their parents? Figuring out the answer might help people who are trying to spread the information more widely decide how to go about that.

While it’s “really encouraging” to see evidence that the Scorecard has an impact on students’ searches, said Mr. Itzkowitz, the Education Department official, “this study obviously highlights a concerning trend as well.” The department is already working to reach low-income students with the data, he said, and the new research underlines that “we need to stay laser-focused” on that.

What is the department doing on this front? For one, it made the data and programming tools for the Scorecard site available to developers, who can use them to create something students might be more likely to see. In addition, Mr. Itzkowitz said, the department is working with college-access groups that serve disadvantaged students.

While the College Board research poses some interesting questions, it couldn’t possibly give the final answer to the big question of whether or not the Scorecard is working, said David Hawkins, executive director for educational content and policy at the National Association for College Admission Counseling.

Back when the Scorecard was released, part of the hope was that it would help students avoid bad colleges, he said. The colleges that look the worst on those measures are not the sort of places students have to take the SAT to attend, he said, so “the College Board is in no position to say anything about that segment of the population.” But determining whether the information helped steer students away from such colleges, Mr. Hawkins said, is an important part of deciding if it is effective.

There’s plenty more research to be done. The College Board is studying how the Scorecard information reaches and could influence low-income students in particular, as well as investigating whether the data release changed enrollment patterns.

As for Zeb, he’s planning to attend Bucknell University. Bucknell was on the family’s radar before he found the Scorecard, said Mr. Gordon, adding that he likes its focus on the liberal arts. And no matter how the data were cut, he said, “it always ended up on the list.”

Beckie Supiano writes about college affordability, the job market for new graduates, and professional schools, among other things. Follow her on Twitter @becksup, or drop her a line at beckie.supiano@chronicle.com.