Skip to content
ADVERTISEMENT
Sign In
  • Sections
    • News
    • Advice
    • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Student Success
    • Technology
    • Transitions
    • The Workplace
  • Magazine
    • Current Issue
    • Special Issues
    • Podcast: College Matters from The Chronicle
  • Newsletters
  • Virtual Events
  • Ask Chron
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Professional Development
    • Career Resources
    • Virtual Career Fair
  • More
  • Sections
    • News
    • Advice
    • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Student Success
    • Technology
    • Transitions
    • The Workplace
  • Magazine
    • Current Issue
    • Special Issues
    • Podcast: College Matters from The Chronicle
  • Newsletters
  • Virtual Events
  • Ask Chron
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Professional Development
    • Career Resources
    • Virtual Career Fair
    Upcoming Events:
    An AI-Driven Work Force
    AI and Microcredentials
Sign In
Blog Logo

Letters

Correspondence from Chronicle readers.

The Chronicle welcomes correspondence from readers about our articles and about topics we have covered. Please make your points as concisely as possible. We will not publish letters longer than 350 words, and all letters will be edited to conform to our style.

Send letters to letters@chronicle.com. Please include a daytime phone number and tell us what institution you are affiliated with or what city or town you are writing from.

Scientists Don’t View Reproducibility as ‘Risky Business’

April 20, 2017

To the Editor:

Years ago I participated in a personality test administered to staff at my research institution. The test was intended to aid managers in building well-functioning teams that included scientists, engineers, operations professionals, and outreach experts. Among other insights, the test revealed that each of these groups had different values. Significantly, scientists value getting the answer right to the near exclusion of anything else, over staying on schedule, remaining within budget, following process, and especially caring what anyone else thinks. Scientists will not cover up problems for fear of external perception; they are truth-seekers at their core.

To continue reading for FREE, please sign in.

Sign In

Or subscribe now to read with unlimited access for as low as $10/month.

Don’t have an account? Sign up now.

A free account provides you access to a limited number of free articles each month, plus newsletters, job postings, salary data, and exclusive store discounts.

Sign Up

To the Editor:

Years ago I participated in a personality test administered to staff at my research institution. The test was intended to aid managers in building well-functioning teams that included scientists, engineers, operations professionals, and outreach experts. Among other insights, the test revealed that each of these groups had different values. Significantly, scientists value getting the answer right to the near exclusion of anything else, over staying on schedule, remaining within budget, following process, and especially caring what anyone else thinks. Scientists will not cover up problems for fear of external perception; they are truth-seekers at their core.

Those values were on full display during a three-day colloquium on reproducibility in research held last month at the National Academy of Sciences, in Washington, D.C. So I was surprised to read a recent article in your paper that implied that attendees shied away from exposing thorny issues for fear that detractors of science would use instances of irreproducible research as an excuse to withdraw funding or question the validity of scientific consensus (“In the Age of Trump, Scientists See Reproducibility as Risky Business,” The Chronicle, March 21.). Your story led me to ask: “Were we at the same meeting?” What I witnessed was three days’ worth of enthusiastic and strong support for getting to the root of irreproducible research and developing solutions to make science a stronger and more robust enterprise.

ADVERTISEMENT

One hardly need look any further than the opening direction for the workshop provided by Dr. David Allison from the University of Alabama at Birmingham. Dr. Allison, a member of the organizing committee, reminded everyone that the scientific method is the best approach ever devised for achieving objective knowledge. The validity of the scientific approach is derived from its carefully crafted procedures, which are constantly being improved and updated as better tools and methods are invented and cross-calibrated with established methods. The culture of science demonstrates regular self reflection and self correction on ways to strengthen and sometimes radically change those procedures when shown to be lacking. I note that this culture is also firmly embedded in the reward system for individual scientists. A scientist never achieves recognition and advancement by going along with what someone else did before, but rather by improving upon it or, better yet, by taking the science to a new level.

Victoria Stodden from the University of Illinois at Urbana-Champaign, another member of the organizing committee, provided an overview of irreproducibility that helped to frame the problems and the search for solutions. For example, with empirical reproducibility, the challenge is for independent labs to guarantee that they have indeed conducted an experiment using the same methodology. She gave an example where two excellent labs intent upon reproducing the same experiment concluded after two years that the reason for differing outcomes was that, at a key point in the experiment, one lab used a centrifuge to mix and another stirred a beaker. Both had thought their technique the only reasonable approach. With statistical reproducibility, the problems involve poor experimental design, how data are handled (e.g., treatment of outliers), false discovery rate, etc. These issues result in irreproducible results even when an experiment is exactly repeated in the empirical sense above. Finally, Dr. Stodden mentioned computational reproducibility, with respect to the Google flu example. The internet search giant created an algorithm that predicted how many cases of flu would materialize as the flu season came on that was double that of the Centers for Disease Control. Because Google’s algorithm was not publicly available, it was not possible to understand the discrepancy in the two predictions.

To suggest that attendees might have “pulled their punches” for fear of retribution ignores the presentations on the meta-analyses that illustrate the difficulty in defining a priori when a study has been reproduced and in achieving empirical, statistical, and/or computational reproducibility. For example, Joachim Vandekerckhove from University of California, Irvine, reviewed results from the Open Science Collaboration’s Reproducibility Project: Psychology. In general, those studies with higher statistical power were more likely to have been reproduced. Randy Schekman from University of California, Berkeley, presented the first results from the cancer research reproducibility study. At this point only a few landmark studies have been examined, and some have successfully replicated, but confounding problems such as evolution of key cell lines could prevent re-establishing identical experimental conditions (empirical reproducibility). Brian Nosek from the Center for Open Science summarized where we are as a community on identifying the problems, enforcing solutions, and changing cultural norms.

Your article entirely focused on the brief closing statement by Dr. Richard Shiffrin of Indiana University, Bloomington, which merely pointed out the obvious to the assembled scientists. Indeed there is always the possibility that those who would harm science would punish us for ferreting out our own weaknesses and correcting them. If so, shame on them.

Marcia McNutt
President
National Academy of Sciences
Washington

We welcome your thoughts and questions about this article. Please email the editors or submit a letter for publication.
Share
  • Twitter
  • LinkedIn
  • Facebook
  • Email
ADVERTISEMENT
ADVERTISEMENT

More News

Photo illustration showing Santa Ono seated, places small in the corner of a dark space
'Unrelentingly Sad'
Santa Ono Wanted a Presidency. He Became a Pariah.
Illustration of a rushing crowd carrying HSI letters
Seeking precedent
Funding for Hispanic-Serving Institutions Is Discriminatory and Unconstitutional, Lawsuit Argues
Photo-based illustration of scissors cutting through paper that is a photo of an idyllic liberal arts college campus on one side and money on the other
Finance
Small Colleges Are Banding Together Against a Higher Endowment Tax. This Is Why.
Pano Kanelos, founding president of the U. of Austin.
Q&A
One Year In, What Has ‘the Anti-Harvard’ University Accomplished?

From The Review

Photo- and type-based illustration depicting the acronym AAUP with the second A as the arrow of a compass and facing not north but southeast.
The Review | Essay
The Unraveling of the AAUP
By Matthew W. Finkin
Photo-based illustration of the Capitol building dome propped on a stick attached to a string, like a trap.
The Review | Opinion
Colleges Can’t Trust the Federal Government. What Now?
By Brian Rosenberg
Illustration of an unequal sign in black on a white background
The Review | Essay
What Is Replacing DEI? Racism.
By Richard Amesbury

Upcoming Events

Plain_Acuity_DurableSkills_VF.png
Why Employers Value ‘Durable’ Skills
Warwick_Leadership_Javi.png
University Transformation: a Global Leadership Perspective
  • Explore Content
    • Latest News
    • Newsletters
    • Letters
    • Free Reports and Guides
    • Professional Development
    • Virtual Events
    • Chronicle Store
    • Chronicle Intelligence
    • Jobs in Higher Education
    • Post a Job
  • Know The Chronicle
    • About Us
    • Vision, Mission, Values
    • DEI at The Chronicle
    • Write for Us
    • Work at The Chronicle
    • Our Reporting Process
    • Advertise With Us
    • Brand Studio
    • Accessibility Statement
  • Account and Access
    • Manage Your Account
    • Manage Newsletters
    • Individual Subscriptions
    • Group and Institutional Access
    • Subscription & Account FAQ
  • Get Support
    • Contact Us
    • Reprints & Permissions
    • User Agreement
    • Terms and Conditions
    • Privacy Policy
    • California Privacy Policy
    • Do Not Sell My Personal Information
1255 23rd Street, N.W. Washington, D.C. 20037
© 2025 The Chronicle of Higher Education
The Chronicle of Higher Education is academe’s most trusted resource for independent journalism, career development, and forward-looking intelligence. Our readers lead, teach, learn, and innovate with insights from The Chronicle.
Follow Us
  • twitter
  • instagram
  • youtube
  • facebook
  • linkedin