Much current research on how college students are helped or harmed by various experiences may paint a distorted picture, two new studies conclude.
Research on the impact of students’ involvement in certain activities, such as doing volunteer work, may fail to sufficiently account for how students’ predispositions to engage in or reject such activities shape their development, according to one of the studies.
The second study concludes that such research, in trying to measure the impact of experiences such as holding a job outside of class, errs in assuming that positive or negative effects of such experience increase at a steady rate. Experiences may not have any impact in moderation, or, at a certain point, their effects may switch from positive to negative, or vice versa.
“Researchers need to be really careful about the ways in which they analyze the data they have,” said Nicholas A. Bowman, an associate professor of higher education and student affairs at the University of Iowa who was a co-author of both studies.
Some of the things we have been doing for years potentially get the wrong answers.
In an interview last week, Mr. Bowman, the director of Iowa’s Center for Research on Undergraduate Education, said, “Some of the things we have been doing for years potentially get the wrong answers.”
Both studies, presented here on Sunday at the annual conference of the American Educational Research Association, were based on analyses of data from about 8,500 students at 46 four-year colleges gathered as part of the Wabash National Study of Liberal Arts Education. Data from the Wabash study, which tracked thousands of full-time students from their entry into college as freshmen through the spring of their senior year, have generated a wealth of research examining how experiences such as exposure to racial diversity or study abroad influence students’ intellectual development.
Charles F. Blaich, a professor of psychology and director of the Center of Inquiry at Wabash College, which conducted the Wabash study, said last week he was not surprised by the findings of the two new studies. Researchers, he said, have long struggled to account for self-selection bias — the fact that students who chose to study abroad already may possess the characteristics that study abroad is believed to develop.
Mr. Blaich said the idea that the effects of exposure to some experience can change — and, for example, turn from positive to negative — as that exposure increases “has some common sense behind it.”
But Alexander C. McCormick, an associate professor of educational leadership and policy studies at Indiana University at Bloomington and director of the National Survey of Student Engagement, called the results of the study of self-selection into question, especially given that it focused on data gathered on students during the spring of their freshman year. Among his concerns, he said, was that one outcome the study sought to measure, improvement in critical thinking, occurs only modestly during the freshman year.
Mr. McCormick said he was “not sure what to make” of freshmen who reported having engaged in activities, such as internships, that students typically participate in much later in college.
Diminishing Returns
Mr. Bowman was joined in conducting both studies by Teniell L. Trolian, a doctoral student in higher education and student affairs at Iowa. Cindy Ann Kilgo, a doctoral student in the same Iowa program, is a co-author of the study examining self-selection bias.
Where other studies of the impact of college experiences have sought to offset the effects of self-selection bias by trying to mathematically account for the differences between students who choose a given experience and those who turn it down, the Iowa researchers took a more nuanced approach. They divided students into three camps: those who had participated in a given experience such as an internship or undergraduate research, those who had not participated and expressed no interest in doing so, and those who were interested in the experience but had not undergone it yet.
The researchers’ paper on their results says they generally found that students’ propensity to engage in certain activities was more strongly correlated with various educational outcomes than their actual engagement. In other words, students who wanted to engage in an activity, but had not yet done so, had educational outcomes closer to those who had already engaged in the activity than those who had forgone it based on a lack of interest.
Noting that self-selection bias appeared to be present even when the researchers rigorously sought to compensate for the influence of factors such as academic motivation or demographic differences, the paper concludes that research on the impact of college experiences “may often contain misleading results.”
Mr. McCormick of Indiana University said he questioned whether students’ declaration of whether they plan to engage in an activity is an accurate measure of their predisposition toward it.
The other study presented on Sunday found that, for most of the student experiences examined, the relationship between the experience and the student outcomes thought to come from it were nonlinear. For example, students who held jobs for a few hours a week were more likely to graduate than those who did not work at all, but after a certain point putting more hours into a job became correlated with attrition. And some experiences that appeared to have a positive impact when they occurred frequently, such as encountering diversity, seemed to have a negative impact when sporadic.
Peter Schmidt writes about affirmative action, academic labor, and issues related to academic freedom. Contact him at peter.schmidt@chronicle.com.
Correction (4/13/2016, 11:25 a.m.): The text has been updated to make clear that Mr. McCormick was speaking only about critical thinking in questioning the study’s focus on freshman-year outcomes. Moreover, he was pointing out that critical thinking improves only slightly in the first year, not commenting on what is known about its subsequent development.
