Skip to content
ADVERTISEMENT
Sign In
  • Sections
    • News
    • Advice
    • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Student Success
    • Technology
    • Transitions
    • The Workplace
  • Magazine
    • Current Issue
    • Special Issues
    • Podcast: College Matters from The Chronicle
  • Newsletters
  • Events
    • Virtual Events
    • Chronicle On-The-Road
    • Professional Development
  • Ask Chron
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Professional Development
    • Career Resources
    • Virtual Career Fair
  • More
  • Sections
    • News
    • Advice
    • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Student Success
    • Technology
    • Transitions
    • The Workplace
  • Magazine
    • Current Issue
    • Special Issues
    • Podcast: College Matters from The Chronicle
  • Newsletters
  • Events
    • Virtual Events
    • Chronicle On-The-Road
    • Professional Development
  • Ask Chron
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Professional Development
    • Career Resources
    • Virtual Career Fair
    Upcoming Events:
    College Advising
    Serving Higher Ed
    Chronicle Festival 2025
Sign In
Research

We May Know Less Than We Thought About What Helps or Hurts Students

By Peter Schmidt April 10, 2016
Washington

Much current research on how college students are helped or harmed by various experiences may paint a distorted picture, two new studies conclude.

Research on the impact of students’ involvement in certain activities, such as doing volunteer work, may fail to sufficiently account for how students’ predispositions to engage in or reject such activities shape their development, according to one of the studies.

To continue reading for FREE, please sign in.

Sign In

Or subscribe now to read with unlimited access for as low as $10/month.

Don’t have an account? Sign up now.

A free account provides you access to a limited number of free articles each month, plus newsletters, job postings, salary data, and exclusive store discounts.

Sign Up

Much current research on how college students are helped or harmed by various experiences may paint a distorted picture, two new studies conclude.

Research on the impact of students’ involvement in certain activities, such as doing volunteer work, may fail to sufficiently account for how students’ predispositions to engage in or reject such activities shape their development, according to one of the studies.

The second study concludes that such research, in trying to measure the impact of experiences such as holding a job outside of class, errs in assuming that positive or negative effects of such experience increase at a steady rate. Experiences may not have any impact in moderation, or, at a certain point, their effects may switch from positive to negative, or vice versa.

“Researchers need to be really careful about the ways in which they analyze the data they have,” said Nicholas A. Bowman, an associate professor of higher education and student affairs at the University of Iowa who was a co-author of both studies.

Some of the things we have been doing for years potentially get the wrong answers.

In an interview last week, Mr. Bowman, the director of Iowa’s Center for Research on Undergraduate Education, said, “Some of the things we have been doing for years potentially get the wrong answers.”

Both studies, presented here on Sunday at the annual conference of the American Educational Research Association, were based on analyses of data from about 8,500 students at 46 four-year colleges gathered as part of the Wabash National Study of Liberal Arts Education. Data from the Wabash study, which tracked thousands of full-time students from their entry into college as freshmen through the spring of their senior year, have generated a wealth of research examining how experiences such as exposure to racial diversity or study abroad influence students’ intellectual development.

Charles F. Blaich, a professor of psychology and director of the Center of Inquiry at Wabash College, which conducted the Wabash study, said last week he was not surprised by the findings of the two new studies. Researchers, he said, have long struggled to account for self-selection bias — the fact that students who chose to study abroad already may possess the characteristics that study abroad is believed to develop.

Mr. Blaich said the idea that the effects of exposure to some experience can change — and, for example, turn from positive to negative — as that exposure increases “has some common sense behind it.”

But Alexander C. McCormick, an associate professor of educational leadership and policy studies at Indiana University at Bloomington and director of the National Survey of Student Engagement, called the results of the study of self-selection into question, especially given that it focused on data gathered on students during the spring of their freshman year. Among his concerns, he said, was that one outcome the study sought to measure, improvement in critical thinking, occurs only modestly during the freshman year.

ADVERTISEMENT

Mr. McCormick said he was “not sure what to make” of freshmen who reported having engaged in activities, such as internships, that students typically participate in much later in college.

Diminishing Returns

Mr. Bowman was joined in conducting both studies by Teniell L. Trolian, a doctoral student in higher education and student affairs at Iowa. Cindy Ann Kilgo, a doctoral student in the same Iowa program, is a co-author of the study examining self-selection bias.

Where other studies of the impact of college experiences have sought to offset the effects of self-selection bias by trying to mathematically account for the differences between students who choose a given experience and those who turn it down, the Iowa researchers took a more nuanced approach. They divided students into three camps: those who had participated in a given experience such as an internship or undergraduate research, those who had not participated and expressed no interest in doing so, and those who were interested in the experience but had not undergone it yet.

The researchers’ paper on their results says they generally found that students’ propensity to engage in certain activities was more strongly correlated with various educational outcomes than their actual engagement. In other words, students who wanted to engage in an activity, but had not yet done so, had educational outcomes closer to those who had already engaged in the activity than those who had forgone it based on a lack of interest.

ADVERTISEMENT

Noting that self-selection bias appeared to be present even when the researchers rigorously sought to compensate for the influence of factors such as academic motivation or demographic differences, the paper concludes that research on the impact of college experiences “may often contain misleading results.”

Mr. McCormick of Indiana University said he questioned whether students’ declaration of whether they plan to engage in an activity is an accurate measure of their predisposition toward it.

The other study presented on Sunday found that, for most of the student experiences examined, the relationship between the experience and the student outcomes thought to come from it were nonlinear. For example, students who held jobs for a few hours a week were more likely to graduate than those who did not work at all, but after a certain point putting more hours into a job became correlated with attrition. And some experiences that appeared to have a positive impact when they occurred frequently, such as encountering diversity, seemed to have a negative impact when sporadic.

Peter Schmidt writes about affirmative action, academic labor, and issues related to academic freedom. Contact him at peter.schmidt@chronicle.com.

Correction (4/13/2016, 11:25 a.m.): The text has been updated to make clear that Mr. McCormick was speaking only about critical thinking in questioning the study’s focus on freshman-year outcomes. Moreover, he was pointing out that critical thinking improves only slightly in the first year, not commenting on what is known about its subsequent development.

A version of this article appeared in the April 22, 2016, issue.
We welcome your thoughts and questions about this article. Please email the editors or submit a letter for publication.
Tags
Scholarship & Research
Share
  • Twitter
  • LinkedIn
  • Facebook
  • Email
About the Author
Peter Schmidt
Peter Schmidt was a senior writer for The Chronicle of Higher Education. He covered affirmative action, academic labor, and issues related to academic freedom. He is a co-author of The Merit Myth: How Our Colleges Favor the Rich and Divide America (The New Press, 2020).
ADVERTISEMENT
ADVERTISEMENT

More News

Vector illustration of large open scissors  with several workers in seats dangling by white lines
Iced Out
Duke Administrators Accused of Bypassing Shared-Governance Process in Offering Buyouts
Illustration showing money being funnelled into the top of a microscope.
'A New Era'
Higher-Ed Associations Pitch an Alternative to Trump’s Cap on Research Funding
Illustration showing classical columns of various heights, each turning into a stack of coins
Endowment funds
The Nation’s Wealthiest Small Colleges Just Won a Big Tax Exemption
WASHINGTON, DISTICT OF COLUMBIA, UNITED STATES - 2025/04/14: A Pro-Palestinian demonstrator holding a sign with Release Mahmud Khalil written on it, stands in front of the ICE building while joining in a protest. Pro-Palestinian demonstrators rally in front of the ICE building, demanding freedom for Mahmoud Khalil and all those targeted for speaking out against genocide in Palestine. Protesters demand an end to U.S. complicity and solidarity with the resistance in Gaza. (Photo by Probal Rashid/LightRocket via Getty Images)
Campus Activism
An Anonymous Group’s List of Purported Critics of Israel Helped Steer a U.S. Crackdown on Student Activists

From The Review

John T. Scopes as he stood before the judges stand and was sentenced, July 2025.
The Review | Essay
100 Years Ago, the Scopes Monkey Trial Discovered Academic Freedom
By John K. Wilson
Vector illustration of a suited man with a pair of scissors for a tie and an American flag button on his lapel.
The Review | Opinion
A Damaging Endowment Tax Crosses the Finish Line
By Phillip Levine
University of Virginia President Jim Ryan keeps his emotions in check during a news conference, Monday, Nov. 14, 2022 in Charlottesville. Va. Authorities say three people have been killed and two others were wounded in a shooting at the University of Virginia and a student is in custody. (AP Photo/Steve Helber)
The Review | Opinion
Jim Ryan’s Resignation Is a Warning
By Robert Zaretsky

Upcoming Events

07-31-Turbulent-Workday_assets v2_Plain.png
Keeping Your Institution Moving Forward in Turbulent Times
Ascendium_Housing_Plain.png
What It Really Takes to Serve Students’ Basic Needs: Housing
Lead With Insight
  • Explore Content
    • Latest News
    • Newsletters
    • Letters
    • Free Reports and Guides
    • Professional Development
    • Events
    • Chronicle Store
    • Chronicle Intelligence
    • Jobs in Higher Education
    • Post a Job
  • Know The Chronicle
    • About Us
    • Vision, Mission, Values
    • DEI at The Chronicle
    • Write for Us
    • Work at The Chronicle
    • Our Reporting Process
    • Advertise With Us
    • Brand Studio
    • Accessibility Statement
  • Account and Access
    • Manage Your Account
    • Manage Newsletters
    • Individual Subscriptions
    • Group and Institutional Access
    • Subscription & Account FAQ
  • Get Support
    • Contact Us
    • Reprints & Permissions
    • User Agreement
    • Terms and Conditions
    • Privacy Policy
    • California Privacy Policy
    • Do Not Sell My Personal Information
1255 23rd Street, N.W. Washington, D.C. 20037
© 2025 The Chronicle of Higher Education
The Chronicle of Higher Education is academe’s most trusted resource for independent journalism, career development, and forward-looking intelligence. Our readers lead, teach, learn, and innovate with insights from The Chronicle.
Follow Us
  • twitter
  • instagram
  • youtube
  • facebook
  • linkedin