Or subscribe now to read with unlimited access for as low as $10/month.
A free account provides you access to a limited number of free articles each month, plus newsletters, job postings, salary data, and exclusive store discounts.
In 2011 the sociologist Richard Arum bemoaned the lack of rigor in American higher education.
In 2016 he critiqued the “inadequate and wrongheaded approaches,” especially those focusing on labor-market data, that drive most studies of student outcomes.
Now, as dean of the University of California at Irvine’s School of Education, he gets to design his own study, leading a new effort to track students’ academic, social, and personal behaviors to gauge their success.
Irvine is the pilot site of a longitudinal study aiming to rethink how colleges measure their own students’ success, thanks to a $1.1-million grant from the Andrew W. Mellon Foundation. What researchers learn at Irvine will inform a national study of several colleges, managed by a consortium at the University of Michigan at Ann Arbor.
“Every other industry in the world right now has figured out a way to use data to better serve customers and clients, and meet their needs and improve institutional performance,” said Arum, the project’s lead investigator. Although higher education has pushed for more “evidence-based decision making,” he said, “we’re a real laggard in using those techniques and tools to improve our own work.”
Academics, in the liberal arts in particular, have been divided on how to best justify their usefulness to the public — or whether such justification is even necessary. In recent years, partisan divisions have expanded to the point that simply earning a college degree is now seen as both a marker and a driver of political affiliation, troubling a higher-education world already unnerved by declining public funding. The Irvine project seeks to meet skeptical policy makers on their home turf: measurement.
“In a society hungry for data and evidence, simply claiming these values of the liberal arts no longer suffices,” said Mariët Westermann, a vice president at the Mellon Foundation, in a news release. Modeling and measuring student outcomes won’t just prove higher ed’s worth to the public, the thinking goes — it will also improve academic quality, because it will help colleges hone their practices.
New Methods to Measure Student Success
The study, called the Next-Generation Undergraduate Success Measurement Project, will trawl depositories of student data, combining institutional records with survey responses and data from learning-management systems like Canvas and Blackboard. Researchers hope to determine what practices affect how students learn, grow, and enter the work force.
In the first year, Arum’s team will gather experts from more than a dozen institutions, including Michigan, Columbia University, and Indiana University, to flesh out the study’s design.
One of those experts is Charles F. Blaich, director of Wabash College’s Center of Inquiry. He led the Wabash National Study of Liberal Arts Education, a similarly ambitious research effort that used multiple data sources to track student pathways. He has noted the challenges of studying student outcomes.
Those decade-old studies relied on unwieldy surveys, longer time frames, and full-time freshman populations, Blaich said. But faster methods have sprung up to measure student success. And the new effort at Irvine is poised to take advantage of them.
For example, the advent of data-rich learning-management systems, which can track a student’s every moment and click on assignments, adds a layer of information that many colleges have yet to examine. And Irvine’s study of two cohorts, freshmen and juniors, as well its substantial mix of juniors who transferred from California’s community colleges, help shift the focus from traditional first-time freshmen, a less representative slice of college students than they used to be.
In the project’s second and third years, Arum’s team will collect data on 1,050 Irvine students — 500 freshmen, 500 continuing and transfer juniors, and 50 freshman-honors students — and will track not just how they perform academically but also, to a modest degree, what they do outside of course work.
“We make a mistake to think that it’s all about academics in higher education,” Arum said. Students spend most of their time socializing, in extracurricular activities, at work, or “at a place like UCI, time with their family, commuting from home.”
As part of the project, in exchange for a small stipend, a freshman in Irvine’s honors program might fill out a one-minute survey pinged to her smartphone that asks whom she’s with and what she’s doing, Arum said. That could happen 10 times, at random, in a given month.
The goal is not to get an exact picture of what each student is doing at any given time. Rather, if you “multiply that by the number of students you have in the project,” Arum said, “all of a sudden you have a very detailed description of what the student body as a whole is doing, and how they’re feeling about it.”
By combining that “experiential” information with routine administrative data, like a student’s demographic profile and course progression, as well as data from a learning-management system, like the writing assignments a student uploads to Canvas, the project will seek to stitch together an aggregate picture of how students move through an undergraduate education.
The Variables That Matter
It’s too early to use those data to tailor education to each individual student, Arum said. Rather, he plans to use the information to identify what variables matter in measuring student success, and to improve practices at an institutional level. At Irvine, that may mean working with faculty members to streamline how they teach and organize their curricula.
Arum sees health care, businesses, and public administration as models for this multifaceted approach. Hospitals, for example, track mortality and infection rates to improve treatment units and health systems, but they’re on their way to personalizing medicine to individual patients. Colleges, meanwhile, are “just starting” to pull together data from across campus units, and often don’t yet know what variables to track, he said.
“These are kind of the methods you want to be using in public administration, in social-welfare organizations,” Arum said. And they’re just the kinds of methods that he and the funders at the Mellon Foundation hope can better inform colleges as they enter the next few years of public debate.
Follow Steven Johnson on Twitter at @stetyjohn, or email him at steve.johnson@chronicle.com.
