A research university investigates itself
Five years ago, Richard Arum and a crew of researchers set out to turn their institution into a living laboratory to study learning. Now they are turning up some fascinating results.
Just as significant, Arum believes this initial phase at the University of California at Irvine proves that the effort is not only replicable by other campuses but vital. Given public doubt about the value of higher education, “particularly the costly residential model,” says Arum, colleges should be measuring some of what they argue is important. “If not to improve it,” he adds, at least “to defend it.”
Arum, a professor of education and sociology, is probably best-known as the co-author of Academically Adrift, which documented how little time students were spending on their course work. But as he told me several years ago, after that book came out in 2011, he realized the need for “a firmer and more empirical basis” to measure the value of college and especially a liberal-arts education. That’s when he began to create this new data-rich project on what’s happening to students inside and beyond the classroom, now known as the UCI Measuring Undergraduate Success Trajectories Project (MUST Project).
Last month I sat down with Arum and several of his colleagues in Irvine (and a few on Zoom) to hear what they’ve been finding. Side note: It was great to spend time on an actual campus, even if that one, despite its handsome brutalist buildings and lush landscaping, lacks my favorite higher-ed amenity, an on-site creamery.
The scope of the research is still taking shape, but I was impressed by the breadth and focus of what I heard about, including:
- Studies of how university grading policies in STEM courses may be unintentionally steering female, first-generation, and underrepresented-minority students away from majors that lead to lucrative jobs.
- Analyses of more than two-million class-discussion posts over three years, drawn from the UCI learning-management system, that show underrepresented-minority students tend to post less and receive fewer peer responses than their classmates do (Arum says that could be problematic because a lot of learning occurs through peer interaction).
- Inquiries into how students’ choice of roommates may affect the racial and ethnic diversity of who ends up as classmates in courses.
- Plans to track whether students who engage regularly with the career-services office, and those who are active in extracurricular activities, fare better postgraduation than do others. That will draw on data from the career-services platform Handshake and from CampusGroups, UCI’s digital campus-community platform.
Cool, right? The projects, some of which are led by current and former grad students, illustrate how institutions can tap into a range of existing data sources to better understand what’s happening around them. Note to the hundreds of other colleges using Handshake: What are you doing with that info? And credit where due: The research on grading policies was conducted by Oded Mcdossi, a UCI postdoc, who is also tracking the impact of student engagement; the LMS work came from Renzhe Yu, now an assistant professor at Teachers College, Columbia University; and the roommate study was done by Xunfei Li, a UCI Ph.D. student.
The university is also generating a heap of new data. Since 2019, the MUST Project has been regularly giving a set of five tests to about 1,000 freshmen and juniors to measure their intellectual development in five areas: critical thinking, perspective taking, understanding of confirmation bias, collaborative problem solving, and online civic reasoning. It also texts weekly surveys to a subset of that group to collect more real-time insights into emerging questions, like how students managed their stress during Covid, what sources they trusted for pandemic information, and most recently, what their experiences have been with ChatGPT. In case you’re curious, three-quarters said they knew about the generative AI tool; 96 percent said they hadn’t used it in their course work. (Maria Calderon Leon, the MUST Project‘s lab manager, stewarded that work.)
The findings I’m really interested in seeing — evaluations of the intellectual-development tests given to students over four years — aren’t available yet. The final set of assessments for the students who entered as freshmen in the fall of 2019 are being administered this spring. Like Arum, I hope the results offer deeper insights into what students are learning than did the data sets used for Academically Adrift, which some questioned.
The MUST Project was originally funded with $1.5 million from the Andrew W. Mellon Foundation. When that ran out, the university stepped in, forming the Postsecondary Education Research & Implementation Institute, which now helps support the work.
Michael Dennin, vice provost for teaching and learning, says having that research capacity on campus makes it easier to introduce changes. For example, when the university began to explore reasons that low-income students were leaving, he says, an analysis aided by MUST researchers helped to identify that the cause was less financial and more related to an academic policy that made it difficult for students to switch majors. In discussions about altering that policy, Dennin says, “it made more impact on the Academic Senate that the data came from UCI researchers.”
The project’s four-year, $4.1-million partnership with California’s Calbright College is another source of support, as I described in my last newsletter. In effect, MUST is an educational R&D arm for the new online community college dedicated to serving working adults. UCI researchers, along with the University of Virginia’s Benjamin L. Castleman, an associate professor of economics and education, and colleagues from the nonprofit behavioral-design consultancy Ideas 42, are experimenting with nudges to help students stay on track, and studying different approaches to course structure and faculty-student interaction.
Shannon McCarty, Calbright’s vice president for learning and instruction, hopes this work might inform other institutions that are looking to update short-term and competency-based programs. Arum is also developing a series of surveys for Calbright students that will be given when they first enroll and then periodically during their programs, including if they withdraw, and after.
Arum hopes the work with Calbright, and with the John N. Gardner Institute for Excellence in Undergraduate Education, particularly on students’ first-year experiences at broad-access institutions, is just the start. “We’d love for this to be taken up by other campuses,” he says.
I’d love to see that, too. But after hearing what’s involved — for example, collecting and anonymizing millions of data points from an LMS or ensuring that statistically significant numbers of students receive and reply to weekly surveys — I wonder how many institutions are prepared to make that kind of commitment. Arum says he and his colleagues stand ready to help; colleges can borrow surveys and even the coding scripts Yu developed to extract information from Irvine’s LMS.
And let’s not forget, Arum adds, many institutions also have “an incredible internal capacity” in the form of their own professoriate. “The biggest barrier” he says, “is institutions’ willingness.”
Got a tip you’d like to share or a question you’d like me to answer? Let me know, at goldie@chronicle.com. If you have been forwarded this newsletter and would like to see past issues, find them here. To receive your own copy, free, register here. If you want to follow me on Twitter (yeah, for now at least, I’m still there), @GoldieStandard is my handle.