The Next Great Hope for Measuring Learning

The long search for an answer to one of higher education’s most pressing questions led here, to the basement of a bistro outside Hartford.

What do students really learn in college?

To find answers, about 20 faculty members from Central Connecticut State University came to spend the waning days of summer break analyzing hundreds of samples of students’ work.

Carl R. Lovitt, their provost, gave them a pep talk over bagels and coffee: “You are engaged in work of meaningful national significance.”

Academe has been pilloried for decades, he said, for its lack of accountability. This project could remedy that. It’s the kind of acronym-heavy, jargon-laced endeavor that’s easily overlooked. But by measuring students’ intellectual skills, it might turn out to provide telling insight into one of higher education’s central functions.

Accountability is often equated with standardized tests, which have attracted support from policy makers and researchers but have failed to catch on with many faculty members. Most tests aren’t connected to the curriculum, and students have little motivation to take them seriously. Other measures, like students’ self-reported attitudes or study habits, are widely used but tend to give institutions few clues for how to improve. So the quest for a faculty-endorsed, broadly useful measure of student learning has continued.

The professors at Central Connecticut State are part of a large-scale project, involving 900 faculty members at 80 public two- and four-year institutions in 13 states, called the Multi-State Collaborative to Advance Quality Student Learning. It’s being led by the State Higher Education Executive Officers Association and the Association of American Colleges & Universities. The project’s scale, novel approach, and strong faculty support have many assessment experts hopeful that it will make a big impact.

Perhaps, they say, this collaboration will help establish common understandings and measurements of some of the most important outcomes of a college education. Though the project is still young — it’s getting ready to publish its second year of results — its leaders hope that by 2019-20 it will have enough data, including from similar efforts at private colleges, to paint an accurate picture of learning nationwide and, in turn, to spark continuing improvement.

What makes the effort notable is its subject of analysis: the authentic stuff of college — the homework, problem sets, and papers that students regularly produce. From those, evaluators like the ones being trained at Central Connecticut State can produce generalizable and comparable findings across disciplines, institutions, and states about students’ critical-thinking, writing, and quantitative-reasoning skills.

To do so, they’re using tools called “Value” rubrics (it’s an acronym for Valid Assessment of Learning in Undergraduate Education). Developed nine years ago by faculty members at more than 100 institutions, under the guidance of AAC&U, the rubrics have a 0-to-4 scale on which evaluators rate how well students demonstrate various components of each skill.

The project is sure to face challenges. Longstanding tensions in assessment aren’t easily resolved. The tradition of faculty control over education makes it difficult for any effort to take root widely. Feeding useful data back to professors to help them improve their teaching is a perennial problem.

But the rubrics’ fundamental connection to the daily work of education, says George D. Kuh, a leading expert on assessment, means this attempt may succeed where others have foundered.

“In terms of trying to assess authentic student learning,” he says, “it’s the most ambitious effort ever.”

Assessment often gets caught in a tug of war between accountability and improvement.

Those who embrace improvement see assessment as the domain of the faculty. Quizzes, tests, essays, and the informal back-and-forth of class discussion reveal what students have learned in a course, allowing professors to take stock and adjust instruction accordingly. The end product is a grade.

But some say that’s not reliable. Maybe grade-point averages used to mean something, before grade inflation. As the price of college continues to rise, assuming without any verifiable proof that students have learned something is unacceptable, the argument goes. Accountability requires some external measure of learning, like a standardized test.

The tensions have produced a stalemate, and educational quality has remained opaque.

“We know less about what our students know and are able to do than just about virtually any other aspect of the enterprise,” says Mr. Kuh, who is evaluating the 13-state effort for the Bill & Melinda Gates Foundation, which is also supporting the project. “It’s a national embarrassment.”

Tensions between accountability and improvement characterized the No Child Left Behind Act, the unpopular federal law that set targets and measured progress in reading and mathematics for elementary- and secondary-school students. Replaced late last year, it has served as a bogeyman for many college educators. They feel they must develop a broadly applicable measure of learning themselves, or something like No Child will be imposed on them.

It hasn’t been easy to come up with one. Standardized tests of core skills, like the Collegiate Learning Assessment, ETS Proficiency Profile, and ACT’s Collegiate Assessment of Academic Proficiency, have attracted widespread interest. But many faculty members have chafed, seeing the tests as disconnected from their courses. The tests may present one way to hold colleges accountable, but on their own they do little to drive improvement. Another measure of the value of college is graduates’ first-year earnings, which figure controversially in the Obama administration’s College Scorecard.

Looking more closely at the existing byproducts of college — the assignments students already do and supposedly learn from — and drawing conclusions from them may be a better way forward, says Robert M. Shireman, a former deputy under secretary at the U.S. Department of Education who is now a senior fellow at the Century Foundation, a think tank.

“The evidence of excellent or inadequate student engagement is student work,” he wrote this year in a report for the foundation. Making students’ work more widely available for inspection would provide a clear indication of what they’re learning.

“What we want,” Mr. Shireman said in an interview, “is faculty members to be creative and push students to their potential.”

Assignments are pivotal to a college education, but professors get little guidance on how to create them. A common approach is to gauge students’ content knowledge. Helping them develop skills like oral communication or creative thinking — and judging those skills — can be more difficult.

In the basement of the bistro, Central Connecticut State’s professors saw that analyzing those underlying skills can get messy. They split into three groups to examine students’ work in quantitative reasoning, writing, and critical thinking.

The last group was led by Cassandra Broadus-Garcia, an associate professor of art. She outlined the ground rules. Start your evaluation by looking at each subcategory of critical thinking in the rubric, beginning at the top of the scale. Assume that the student’s work is a 4 until you can’t justify it. Then move to a 3. Look for evidence of the proper score. Don’t make inferences; stick to what the student actually wrote.

“Take off your professor hat,” Ms. Broadus-Garcia told them. “You’re not grading.”

The distinction between grading and scoring is an essential one for this effort. Grading is second nature to faculty members and reflects their disciplinary judgment about how well students understand the course material. Scoring gauges the intellectual skills and habits that should characterize an educated person from any discipline, and that’s what this project wants to capture.

The shift sometimes proved difficult for the group, even though the process was designed to discourage grading. For one, the professors didn’t know the discipline or the purpose of the two-page homework assignment they were evaluating. All the group had were five prompts and one student’s answers. Knowing the goal of an assignment tends to focus attention on how well students meet expectations. And that leads back to grading.

The first prompt was to choose a health treatment to study. This student opted for the “liberation procedure” for multiple sclerosis, based on the idea that poor blood drainage from the brain causes the disease’s main symptoms.

The next prompt was to evaluate the credibility of two websites describing the treatment. The student chose WebMD and a page published by Singularity University. The latter’s reliability had raised questions for the student because it linked to Wikipedia, and the author seemed to have few relevant credentials.

“The article seems like more of a blog posting,” the student wrote. WebMD, in contrast, was written by people with expertise in medicine, health communications, and journalism.

The five professors — a biologist, two from business, and another two from political science — quickly found themselves drifting toward grading, especially as they debated one category of critical thinking, “influence of context and assumptions.” Were the students supposed to analyze a controversial health treatment? Or was this exercise about information literacy?

“This goes back to the problem of not having the assignment,” said Robbin Smith, an associate professor of political science, with some frustration.

Grading crept into the conversation in other ways. Several professors wondered about Singularity University, which the student identified as a university (it’s a think tank that promotes technological solutions to social problems). Would choosing such a site matter if the point of the exercise was to evaluate sources of health information?

If that was the assignment, then the student, by choosing unequal sources, seemed to be constructing a straw man, said Jason Snyder, an associate professor of business. It was a form of selection bias, he said, as the biologist next to him nodded vigorously. After all, how difficult is it to weigh competing claims when one source is WebMD and the other’s author profile features, the student noted, “a picture of a cartoon”?

But regardless of the assignment, the professors weren’t impressed by the student’s handling of context and assumptions. Four of the scorers rated the sample a 2 out of 4. The student had questioned a few assumptions, but not necessarily his or her own.

Professors using these rubrics have long been able to score individual pieces of student work like the group at Central Connecticut State did. What’s different now is that hundreds of faculty members in 13 states are being trained to do the same thing, allowing researchers to aggregate the numbers and look for patterns.

That’s part of what worries John D. Hathcoat. The assistant professor of graduate psychology at James Madison University will be leading the Multi-State Collaborative to Advance Quality Student Learning there. He counts himself as a supportive skeptic.

Should the numbers be used to hold colleges accountable? If one state’s average score on written communication is 2.3, and another’s is 2.5, does that mean the latter’s public colleges are 0.2 points better at developing that skill?

“It’s worth doing and we need to do it,” Mr. Hathcoat says of the project, but “it could get misused.”

He also has methodological concerns. The nature and rigor of assignments vary widely, and oversampling very easy or difficult ones, he argues, could produce misleading results. And his research suggests that critical thinking and writing are fuzzy things to assess, often bleeding into each other. “If my thoughts are jumbled,” he said, “it’s going to show up in my writing.”

Terrel L. Rhodes, executive director of the Value project for AAC&U, shares some of those concerns. He worries that the rubrics’ scores will be rolled up into an average and used for the wrong purpose. “None of this is intended for rankings,” he says. “It is about, What are you doing on your campus to improve?”

And he agrees that, on some level, categories like critical thinking or writing are artificial. But the more important point, he says, is that this effort has provoked sustained thought and attention among faculty members to intellectual skills, teaching, and assignments. It has served to focus professors’ attention on different aspects of learning, he says, “by naming them and trying to take them on.”

Analyzing student work seems to have energized many professors: 94 percent of participants in an earlier stage of the project said they enjoyed having cross-disciplinary discussions. The forums gave them an opportunity to think deeply about skills they all see as valuable and discuss how to teach and assess them in their own fields.

“When was the last time,” asks Mr. Rhodes, “you had people enthusiastic about assessment?”

Rubrics work best for improving teaching, says Roger Benjamin, president of the Council for Aid to Education, which oversees the Collegiate Learning Assessment. But they’re not as reliable for accountability purposes, he says, as standardized tests like his.

More than two-thirds of institutions use some type of rubric, while fewer than half give standardized tests like the CLA. Colleges tend to see many forms of assessment as a way to satisfy accreditors, according to a 2014 study by the National Institute for Learning Outcomes Assessment. Using such tools for institutional improvement or curricular change ranked far lower.

But attitudes seem to be shifting, says Natasha A. Jankowski, director of the institute, which is updating the study. The Value rubrics and the state collaboration, she says, have changed faculty behavior, chiefly because the approach is directly related to their daily work.

“It’s been maybe the best leverage point we’ve had to help faculty think about improvement,” says Ms. Jankowksi. “It hits them where they live.”

Still, results don’t always make their way back to the classroom. A standard complaint about assessment efforts is not “closing the loop.”

Yvonne Kochera Kirby, Central Connecticut State’s director of institutional research and assessment, wants to avoid that. She provides data to the professors whose assignments are scored, and that has sparked changes. One professor realized she’d focused on the arcana of her discipline, mistaking that for critical thinking. Another saw that, in one assignment, she had unintentionally repeated the same prompt three different ways.

Absent such feedback, professors often assume their assignments achieve what they’re supposed to, says Ms. Kirby. “A lot of faculty members probably say, ‘This assignment aligns,’” she says, “but it really doesn’t.”

Improving teaching can seem like a huge task. It may sound like it requires wholesale changes or a radical rethinking of the professor’s role in the classroom.

The changes driven by the rubrics tend to be comparatively modest. But a small adjustment can still be powerful. It might mean drawing a clearer connection between an assignment and the goals of the course, or giving more-explicit directions. “Those minor modifications have huge impacts on students,” says Ms. Jankowski.

Professors have described how the rubric scores have helped them look with fresh eyes at what they assign students. Bonnie L. Orcutt is one of them.

For a microeconomics assignment, the professor at Worcester State University often asked students to analyze an article from The New York Times on the use of a tobacco tax to balance state budgets. She would have students summarize the states’ position and predict what would happen if they increased taxes.

When that work was analyzed using the quantitative-literacy rubric, her students scored low in one category, evaluating assumptions. It wasn’t because they couldn’t, she realized. She just hadn’t asked them to evaluate assumptions, no small matter to an economist like Ms. Orcutt. “Assumptions,” she says, “inform the models you choose and how you interpret them.”

Ms. Orcutt revised the assignment, making explicit the steps she wanted students to take. She added a prompt: “Indicate any assumptions that underlie your analysis.” She also brought up assumptions during class discussions. They were simple modifications, but since then, Ms. Orcutt has noticed her students demonstrating that skill more consistently.

Focusing on little things has had a big effect on her teaching. She got into assessment almost by happenstance eight years ago, when she was enlisted to help with a general-education revision at Worcester State. She recently finished a three-year term as director of learning-outcomes assessment for the Massachusetts Department of Higher Education.

Ms. Orcutt is an unlikely convert. Early in her career, she thought of assessment as one more hoop to jump through. Teaching well was a matter of how “on” she was during her lectures, she says, and how much course content her students absorbed. She still cares about whether they know their stuff, of course, but now she thinks about how to help them apply it.

Creating the right conditions for that kind of learning is an intellectual challenge, like research, that is both invigorating and aggravating. The work of teaching well is a continuing process, she says, of creating assignments, analyzing the results, and making more changes. It’s work that’s never finished.

Dan Berrett writes about teaching, learning, the curriculum, and educational quality. Follow him on Twitter @danberrett, or write to him at dan.berrett@chronicle.com.