Skip to content
ADVERTISEMENT
Sign In
  • Sections
    • News
    • Advice
    • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Student Success
    • Technology
    • Transitions
    • The Workplace
  • Magazine
    • Current Issue
    • Special Issues
    • Podcast: College Matters from The Chronicle
  • Newsletters
  • Events
    • Virtual Events
    • Chronicle On-The-Road
    • Professional Development
  • Ask Chron
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Professional Development
    • Career Resources
    • Virtual Career Fair
  • More
  • Sections
    • News
    • Advice
    • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Student Success
    • Technology
    • Transitions
    • The Workplace
  • Magazine
    • Current Issue
    • Special Issues
    • Podcast: College Matters from The Chronicle
  • Newsletters
  • Events
    • Virtual Events
    • Chronicle On-The-Road
    • Professional Development
  • Ask Chron
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Professional Development
    • Career Resources
    • Virtual Career Fair
    Upcoming Events:
    College Advising
    Serving Higher Ed
    Chronicle Festival 2025
Sign In
Research

Can Science’s Reproducibility Crisis Be Reproduced?

By Paul Basken March 3, 2016
A 2011 study of whether happier people are more patient used footage of a Broadway performance by Robin Williams (pictured in 2009). But Mr. Williams’s suicide, in 2014, might well have affected researchers’ ability to replicate the original study, as people are now likely to respond differently to his brand of humor.
A 2011 study of whether happier people are more patient used footage of a Broadway performance by Robin Williams (pictured in 2009). But Mr. Williams’s suicide, in 2014, might well have affected researchers’ ability to replicate the original study, as people are now likely to respond differently to his brand of humor.Gary Miller, FilmMagic, Getty Images

More than a year after he committed suicide, is Robin Williams still funny?

The answer, both before his tragic death and after, probably is a matter of personal preference. It’s also now a key to assessing how seriously to take the much-feared crisis of reproducibility in scientific research.

To continue reading for FREE, please sign in.

Sign In

Or subscribe now to read with unlimited access for as low as $10/month.

Don’t have an account? Sign up now.

A free account provides you access to a limited number of free articles each month, plus newsletters, job postings, salary data, and exclusive store discounts.

Sign Up

A 2011 study of whether happier people are more patient used footage of a Broadway performance by Robin Williams (pictured in 2009). But Mr. Williams’s suicide, in 2014, might well have affected researchers’ ability to replicate the original study, as people are now likely to respond differently to his brand of humor.
A 2011 study of whether happier people are more patient used footage of a Broadway performance by Robin Williams (pictured in 2009). But Mr. Williams’s suicide, in 2014, might well have affected researchers’ ability to replicate the original study, as people are now likely to respond differently to his brand of humor.Gary Miller, FilmMagic, Getty Images

More than a year after he committed suicide, is Robin Williams still funny?

The answer, both before his tragic death and after, probably is a matter of personal preference. It’s also now a key to assessing how seriously to take the much-feared crisis of reproducibility in scientific research.

Broad fears over reproducibility were stoked by a 2005 article in PLOS Medicine by John P.A. Ioannidis, a professor of health research and policy at Stanford University, contending that most published research findings are false. Last year a team of hundreds of researchers raised further alarm. After working over three years to faithfully repeat 100 studies that had been published in psychology journals, the team reported that it could not replicate most of the original results.

It’s very easy to come to the wrong conclusion when you try to replicate other people’s research.

Now, two new studies, published on Thursday in Science magazine, are pushing back. One, a Harvard-led critique of the project that repeated 100 psychology studies, suggests that that ambitious effort overlooked some critical factors. The other, an attempt to repeat 18 studies in leading economics journals, found that 61 percent of them replicated successfully.

“Our results were pretty encouraging,” said the lead author of the economics study, Colin F. Camerer, a professor of behavioral economics at the California Institute of Technology.

Together, the two papers this week should help calm the widespread worries about the reliability of science fanned by Mr. Ioannidis, said the lead author of the psychology critique, Daniel T. Gilbert, a professor of psychology at Harvard.

“It’s very easy to come to the wrong conclusion when you try to replicate other people’s research,” Mr. Gilbert said.

Inconclusive Results

On that much, Mr. Ioannidis and other advocates of replication studies say they agree, with Robin Williams and his brand of comedy serving as a case in point. Among the 18 economics studies that Mr. Camerer’s team attempted to replicate was a 2011 report by a pair of Santa Clara University researchers that explored whether happier people are more patient.

ADVERTISEMENT

For the experiment, conducted among 69 students at Santa Clara, the half chosen for the “happy” group were shown a nine-minute clip of Mr. Williams during a 2002 Broadway performance. For Mr. Camerer’s replication attempt, he tested the clip on 131 students at the University of Oxford’s Nuffield College, where he happened to be working at the time.

Mr. Camerer acknowledged the obvious risks in that attempt at replication. For one thing, British and American senses of humor differ. For another, Mr. Williams committed suicide in August 2014, before the replication study was conducted, perhaps coloring test subjects’ emotional response to his work. Such attempts are considered “near replications” because the conditions of the original study cannot be repeated, Mr. Camerer said.

That raises the question, however, of whether any attempt to replicate an experiment involving human subjects warrants a conclusion about the strength of the initial finding.

The coordinator of the 100 replicated psychology studies, Brian A. Nosek, a professor of psychology at the University of Virginia, said he was well aware that there are limits that are only beginning to be understood.

ADVERTISEMENT

In November 2011, Mr. Nosek assembled a team of some 200 researchers, known as the Open Science Collaboration, who spent three years seeing if they could replicate 100 studies published in leading journals. Of those original studies, 97 percent reported a statistically significant finding. In the replication attempts, only 36 percent produced significant results.

Beyond those results, however, Mr. Nosek said he had been driven to learn more about how to really measure reproducibility and how to improve it. The project involved extensive collaboration with the original authors to get their input on how best to conduct the replication attempts. For a follow-up effort, Mr. Nosek said, his team is looking more closely at 11 studies whose original authors did not endorse the replication design, to see if their suggestions might improve the rate of reproducibility.

Despite that commitment to scientifically exploring reproducibility, the findings of the Open Science Collaboration left some bruises. Mr. Gilbert called the project an unwarranted slap at the field of psychology, and said he had set out to find what might have been wrong about the replication attempts. “We were as chagrined as anybody to get this news” from the Open Science Collaboration, he said.

‘Don’t Trust the Headlines’

For their paper this week, Mr. Gilbert and his co-authors pored over the data made available by Mr. Nosek’s team and suggested a variety of flaws in how the Open Science Collaboration had chosen studies to review, selected participants for the replication attempts, and defined statistically significant effects.

ADVERTISEMENT

According to Mr. Gilbert, the Open Science Collaboration’s replications used Italians rather than Americans to repeat a study of racial attitudes, and used students not enrolled in college to repeat an exploration in which college students had been asked to imagine being called on by a professor.

The lesson, Mr. Gilbert said, is: “Don’t trust the headlines when you see that somebody replicated a study. You have to look carefully to see what they really did.”

Mr. Nosek said he agreed that replications can be very difficult. The lesson, however, is to keep trying to make replications even better, he said, and to design studies from the beginning so that replication attempts are easier.

The arguments that the Harvard team is raising are not really very serious.

The attacks by the Harvard group seem especially disingenuous, Mr. Ioannidis said, and motivated more by a desire to defend the field of psychology than to explore ways of improving reproducibility.

ADVERTISEMENT

At least a couple of the Open Science Collaboration’s decisions — trying to reproduce only papers in leading journals, and excluding papers whose authors were reluctant to cooperate — more likely served to underestimate the problem of reproducibility rather than to exaggerate it, Mr. Ioannidis said. “The arguments that the Harvard team is raising are not really very serious,” he said.

And as Mr. Camerer’s group is showing, even success is hard to define. Its attempt to affirm that happier people are more patient was scored a failure — perhaps, Mr. Camerer admitted, because of changes in perceptions of Mr. Williams’s comedic talents. But over all, Mr. Camerer was encouraged to see that 61 percent of the original studies had been effectively replicated.

“So there’s a little bit of room for improvement, but it’s not a disaster,” Mr. Camerer said. That message is important as a counterpoint to some of the “dramatic claims” by Mr. Ioannidis that most research findings are false, he added.

Mr. Nosek was less impressed. “I would certainly hope that we could do better than 60 percent,” he said. Either way, he said, this week’s papers reflect growing attention to the problem and to the pursuit of better ways to define and promote reproducibility. “It’s a victory for openness,” he said.

Paul Basken covers university research and its intersection with government policy. He can be found on Twitter @pbasken, or reached by email at paul.basken@chronicle.com.

A version of this article appeared in the March 18, 2016, issue.
We welcome your thoughts and questions about this article. Please email the editors or submit a letter for publication.
Tags
Scholarship & Research
Share
  • Twitter
  • LinkedIn
  • Facebook
  • Email
Paul Basken Bio
About the Author
Paul Basken
Paul Basken was a government policy and science reporter with The Chronicle of Higher Education, where he won an annual National Press Club award for exclusives.
ADVERTISEMENT
ADVERTISEMENT

Related Content

The Results of the Reproducibility Project Are In. They’re Not Good.
New Center Hopes to Clean Up Sloppy Science and Bogus Research
How to Fix Psychology’s Replication Crisis

More News

Vector illustration of large open scissors  with several workers in seats dangling by white lines
Iced Out
Duke Administrators Accused of Bypassing Shared-Governance Process in Offering Buyouts
Illustration showing money being funnelled into the top of a microscope.
'A New Era'
Higher-Ed Associations Pitch an Alternative to Trump’s Cap on Research Funding
Illustration showing classical columns of various heights, each turning into a stack of coins
Endowment funds
The Nation’s Wealthiest Small Colleges Just Won a Big Tax Exemption
WASHINGTON, DISTICT OF COLUMBIA, UNITED STATES - 2025/04/14: A Pro-Palestinian demonstrator holding a sign with Release Mahmud Khalil written on it, stands in front of the ICE building while joining in a protest. Pro-Palestinian demonstrators rally in front of the ICE building, demanding freedom for Mahmoud Khalil and all those targeted for speaking out against genocide in Palestine. Protesters demand an end to U.S. complicity and solidarity with the resistance in Gaza. (Photo by Probal Rashid/LightRocket via Getty Images)
Campus Activism
An Anonymous Group’s List of Purported Critics of Israel Helped Steer a U.S. Crackdown on Student Activists

From The Review

John T. Scopes as he stood before the judges stand and was sentenced, July 2025.
The Review | Essay
100 Years Ago, the Scopes Monkey Trial Discovered Academic Freedom
By John K. Wilson
Vector illustration of a suited man with a pair of scissors for a tie and an American flag button on his lapel.
The Review | Opinion
A Damaging Endowment Tax Crosses the Finish Line
By Phillip Levine
University of Virginia President Jim Ryan keeps his emotions in check during a news conference, Monday, Nov. 14, 2022 in Charlottesville. Va. Authorities say three people have been killed and two others were wounded in a shooting at the University of Virginia and a student is in custody. (AP Photo/Steve Helber)
The Review | Opinion
Jim Ryan’s Resignation Is a Warning
By Robert Zaretsky

Upcoming Events

07-31-Turbulent-Workday_assets v2_Plain.png
Keeping Your Institution Moving Forward in Turbulent Times
Ascendium_Housing_Plain.png
What It Really Takes to Serve Students’ Basic Needs: Housing
Lead With Insight
  • Explore Content
    • Latest News
    • Newsletters
    • Letters
    • Free Reports and Guides
    • Professional Development
    • Events
    • Chronicle Store
    • Chronicle Intelligence
    • Jobs in Higher Education
    • Post a Job
  • Know The Chronicle
    • About Us
    • Vision, Mission, Values
    • DEI at The Chronicle
    • Write for Us
    • Work at The Chronicle
    • Our Reporting Process
    • Advertise With Us
    • Brand Studio
    • Accessibility Statement
  • Account and Access
    • Manage Your Account
    • Manage Newsletters
    • Individual Subscriptions
    • Group and Institutional Access
    • Subscription & Account FAQ
  • Get Support
    • Contact Us
    • Reprints & Permissions
    • User Agreement
    • Terms and Conditions
    • Privacy Policy
    • California Privacy Policy
    • Do Not Sell My Personal Information
1255 23rd Street, N.W. Washington, D.C. 20037
© 2025 The Chronicle of Higher Education
The Chronicle of Higher Education is academe’s most trusted resource for independent journalism, career development, and forward-looking intelligence. Our readers lead, teach, learn, and innovate with insights from The Chronicle.
Follow Us
  • twitter
  • instagram
  • youtube
  • facebook
  • linkedin