> Skip to content
FEATURED:
  • Public Perception of College
Sign In
  • News
  • Advice
  • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Technology
    • The Workplace
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Technology
    • The Workplace
  • Current Issue
  • Virtual Events
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Career Resources
    • Find a Job
    • Post a Job
    • Career Resources
Sign In
  • News
  • Advice
  • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Technology
    • The Workplace
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Technology
    • The Workplace
  • Current Issue
  • Virtual Events
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Career Resources
    • Find a Job
    • Post a Job
    • Career Resources
  • News
  • Advice
  • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Technology
    • The Workplace
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Technology
    • The Workplace
  • Current Issue
  • Virtual Events
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Career Resources
    • Find a Job
    • Post a Job
    • Career Resources
Sign In
ADVERTISEMENT
Blog Logo

Letters

Correspondence from Chronicle readers.

  • Twitter
  • LinkedIn
  • Show more sharing options
Share
  • Twitter
  • LinkedIn
  • Facebook
  • Email
  • Copy Link URLCopied!
  • Print

Scientists Don’t View Reproducibility as ‘Risky Business’

April 20, 2017

To the Editor:

Years ago I participated in a personality test administered to staff at my research institution. The test was intended to aid managers in building well-functioning teams that included scientists, engineers, operations professionals, and outreach experts. Among other insights, the test revealed that each of these groups had different values. Significantly, scientists value getting the answer right to the near exclusion of anything else, over staying on schedule, remaining within budget, following process, and especially caring what anyone else thinks. Scientists will not cover up problems for fear of external perception; they are truth-seekers at their core.

We're sorry. Something went wrong.

We are unable to fully display the content of this page.

The most likely cause of this is a content blocker on your computer or network.

Please allow access to our site, and then refresh this page. You may then be asked to log in, create an account if you don't already have one, or subscribe.

If you continue to experience issues, please contact us at 202-466-1032 or help@chronicle.com

To the Editor:

Years ago I participated in a personality test administered to staff at my research institution. The test was intended to aid managers in building well-functioning teams that included scientists, engineers, operations professionals, and outreach experts. Among other insights, the test revealed that each of these groups had different values. Significantly, scientists value getting the answer right to the near exclusion of anything else, over staying on schedule, remaining within budget, following process, and especially caring what anyone else thinks. Scientists will not cover up problems for fear of external perception; they are truth-seekers at their core.

Those values were on full display during a three-day colloquium on reproducibility in research held last month at the National Academy of Sciences, in Washington, D.C. So I was surprised to read a recent article in your paper that implied that attendees shied away from exposing thorny issues for fear that detractors of science would use instances of irreproducible research as an excuse to withdraw funding or question the validity of scientific consensus (“In the Age of Trump, Scientists See Reproducibility as Risky Business,” The Chronicle, March 21.). Your story led me to ask: “Were we at the same meeting?” What I witnessed was three days’ worth of enthusiastic and strong support for getting to the root of irreproducible research and developing solutions to make science a stronger and more robust enterprise.

One hardly need look any further than the opening direction for the workshop provided by Dr. David Allison from the University of Alabama at Birmingham. Dr. Allison, a member of the organizing committee, reminded everyone that the scientific method is the best approach ever devised for achieving objective knowledge. The validity of the scientific approach is derived from its carefully crafted procedures, which are constantly being improved and updated as better tools and methods are invented and cross-calibrated with established methods. The culture of science demonstrates regular self reflection and self correction on ways to strengthen and sometimes radically change those procedures when shown to be lacking. I note that this culture is also firmly embedded in the reward system for individual scientists. A scientist never achieves recognition and advancement by going along with what someone else did before, but rather by improving upon it or, better yet, by taking the science to a new level.

Victoria Stodden from the University of Illinois at Urbana-Champaign, another member of the organizing committee, provided an overview of irreproducibility that helped to frame the problems and the search for solutions. For example, with empirical reproducibility, the challenge is for independent labs to guarantee that they have indeed conducted an experiment using the same methodology. She gave an example where two excellent labs intent upon reproducing the same experiment concluded after two years that the reason for differing outcomes was that, at a key point in the experiment, one lab used a centrifuge to mix and another stirred a beaker. Both had thought their technique the only reasonable approach. With statistical reproducibility, the problems involve poor experimental design, how data are handled (e.g., treatment of outliers), false discovery rate, etc. These issues result in irreproducible results even when an experiment is exactly repeated in the empirical sense above. Finally, Dr. Stodden mentioned computational reproducibility, with respect to the Google flu example. The internet search giant created an algorithm that predicted how many cases of flu would materialize as the flu season came on that was double that of the Centers for Disease Control. Because Google’s algorithm was not publicly available, it was not possible to understand the discrepancy in the two predictions.

ADVERTISEMENT

To suggest that attendees might have “pulled their punches” for fear of retribution ignores the presentations on the meta-analyses that illustrate the difficulty in defining a priori when a study has been reproduced and in achieving empirical, statistical, and/or computational reproducibility. For example, Joachim Vandekerckhove from University of California, Irvine, reviewed results from the Open Science Collaboration’s Reproducibility Project: Psychology. In general, those studies with higher statistical power were more likely to have been reproduced. Randy Schekman from University of California, Berkeley, presented the first results from the cancer research reproducibility study. At this point only a few landmark studies have been examined, and some have successfully replicated, but confounding problems such as evolution of key cell lines could prevent re-establishing identical experimental conditions (empirical reproducibility). Brian Nosek from the Center for Open Science summarized where we are as a community on identifying the problems, enforcing solutions, and changing cultural norms.

Your article entirely focused on the brief closing statement by Dr. Richard Shiffrin of Indiana University, Bloomington, which merely pointed out the obvious to the assembled scientists. Indeed there is always the possibility that those who would harm science would punish us for ferreting out our own weaknesses and correcting them. If so, shame on them.

Marcia McNutt
President
National Academy of Sciences
Washington

ADVERTISEMENT
ADVERTISEMENT
  • Explore Content
    • Latest News
    • Newsletters
    • Letters
    • Free Reports and Guides
    • Professional Development
    • Virtual Events
    • Chronicle Store
    • Chronicle Intelligence
    • Find a Job
    • Post a Job
    Explore Content
    • Latest News
    • Newsletters
    • Letters
    • Free Reports and Guides
    • Professional Development
    • Virtual Events
    • Chronicle Store
    • Chronicle Intelligence
    • Find a Job
    • Post a Job
  • Know The Chronicle
    • About Us
    • Write for Us
    • Work at The Chronicle
    • Our Reporting Process
    • Advertise With Us
    • Brand Studio
    • DEI Commitment Statement
    • Accessibility Statement
    Know The Chronicle
    • About Us
    • Write for Us
    • Work at The Chronicle
    • Our Reporting Process
    • Advertise With Us
    • Brand Studio
    • DEI Commitment Statement
    • Accessibility Statement
  • Account and Access
    • Manage Your Account
    • Manage Newsletters
    • Individual Subscriptions
    • Institutional Subscriptions
    • Subscription & Account FAQ
    Account and Access
    • Manage Your Account
    • Manage Newsletters
    • Individual Subscriptions
    • Institutional Subscriptions
    • Subscription & Account FAQ
  • Get Support
    • Contact Us
    • Reprints & Permissions
    • User Agreement
    • Terms and Conditions
    • Privacy Policy
    • California Privacy Policy
    • Do Not Sell My Personal Information
    Get Support
    • Contact Us
    • Reprints & Permissions
    • User Agreement
    • Terms and Conditions
    • Privacy Policy
    • California Privacy Policy
    • Do Not Sell My Personal Information
1255 23rd Street, N.W. Washington, D.C. 20037
© 2023 The Chronicle of Higher Education
  • twitter
  • instagram
  • youtube
  • facebook
  • linkedin