Skip to content
ADVERTISEMENT
Sign In
  • Sections
    • News
    • Advice
    • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Student Success
    • Technology
    • Transitions
    • The Workplace
  • Magazine
    • Current Issue
    • Special Issues
    • Podcast: College Matters from The Chronicle
  • Newsletters
  • Virtual Events
  • Ask Chron
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Professional Development
    • Career Resources
    • Virtual Career Fair
  • More
  • Sections
    • News
    • Advice
    • The Review
  • Topics
    • Data
    • Diversity, Equity, & Inclusion
    • Finance & Operations
    • International
    • Leadership & Governance
    • Teaching & Learning
    • Scholarship & Research
    • Student Success
    • Technology
    • Transitions
    • The Workplace
  • Magazine
    • Current Issue
    • Special Issues
    • Podcast: College Matters from The Chronicle
  • Newsletters
  • Virtual Events
  • Ask Chron
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Professional Development
    • Career Resources
    • Virtual Career Fair
    Upcoming Events:
    An AI-Driven Work Force
    University Transformation
Sign In
News

A New Theory on How Researchers Can Solve the Reproducibility Crisis: Do the Math

By Paul Basken June 28, 2017
Jeanne Calment of France was believed to be the world’s longest lived person when she died in 1997 at age 122. A recent headline-grabbing study about the limits of the human life span has drawn rebuttals with implications for how universities and scientists might approach the reproducibility crisis in research.
Jeanne Calment of France was believed to be the world’s longest lived person when she died in 1997 at age 122. A recent headline-grabbing study about the limits of the human life span has drawn rebuttals with implications for how universities and scientists might approach the reproducibility crisis in research.Lionel Cironneau, AP Images

From the beginning, it seemed like a difficult prediction.

In an article published last October in Nature, three researchers affiliated with the Albert Einstein College of Medicine in New York City said they had crunched the numbers and concluded that humans will never consistently live much beyond 115 years.

To continue reading for FREE, please sign in.

Sign In

Or subscribe now to read with unlimited access for as low as $10/month.

Don’t have an account? Sign up now.

A free account provides you access to a limited number of free articles each month, plus newsletters, job postings, salary data, and exclusive store discounts.

Sign Up

Jeanne Calment of France was believed to be the world’s longest lived person when she died in 1997 at age 122. A recent headline-grabbing study about the limits of the human life span has drawn rebuttals with implications for how universities and scientists might approach the reproducibility crisis in research.
Jeanne Calment of France was believed to be the world’s longest lived person when she died in 1997 at age 122. A recent headline-grabbing study about the limits of the human life span has drawn rebuttals with implications for how universities and scientists might approach the reproducibility crisis in research.Lionel Cironneau, AP Images

From the beginning, it seemed like a difficult prediction.

In an article published last October in Nature, three researchers affiliated with the Albert Einstein College of Medicine in New York City said they had crunched the numbers and concluded that humans will never consistently live much beyond 115 years.

“From now on, this is it,” one of the three authors, Jan Vijg, a professor of genetics at Albert Einstein, told The New York Times one of several major news outlets that helped promote the sobering news. “Humans will never get older than 115.”

With more statistical expertise on research teams, one expert argues, scientists could think with more nuance about whether a research finding is significant.

But almost immediately, the conclusion was attacked by numerous critics citing various problems with the Albert Einstein team’s statistical analysis. That criticism cascaded Wednesday when Nature published another five rebuttals.

Among the allegations: Mr. Vijg and his partners failed to properly consider what statisticians call the “null hypothesis.” In this case, Bryan G. Hughes and Siegfried Hekimi of McGill University explained in one of the critiques, applying the null hypothesis means statistically including the possibility that the maximum human life span actually will continue to increase.

“There are strong statistical grounds to question the validity of their conclusions,” wrote another team, comprising Maarten P. Rozing, Thomas B.L. Kirkwood, and Rudi G.J. Westendorp of the University of Copenhagen. “There might be a limit to human lifespan, but we believe that their results provide no evidence,” wrote a third, Adam Lenart and James W. Vaupel of the University of Southern Denmark.

Mr. Vijg stands by his work. The “real problem,” he said, “is that some people get hysterical when someone openly sheds doubt on the idea that we can live forever, or at least much longer than we do now.”

Statistical Shortcomings

As scientists across various fields move through a period of soul-searching over the disturbing number of studies that apparently cannot be reproduced, the leading suspects include industry bias, financial and career pressures, poor study design, and wide variations in research methodologies, equipment, and standards.

ADVERTISEMENT

But the conversation over the study on aging points to another possibility: that too much research is hamstrung by a lack of pure statistical ability. Universities, scientists, and advocacy groups may have overlooked the seriousness of that problem as they hunt for more complex or nefarious causes of the reproducibility crisis.

Cory Fournier, an adjunct instructor in mathematics at the University of Massachusetts at Lowell, came to that conclusion earlier this year, after he cobbled together $1,000 in scarce union funds to journey to a big national conference on scientific reproducibility.

Mr. Fournier said he made the trip to the National Academy of Sciences headquarters in Washington, D.C., expecting to commune with fellow statisticians. After all, he reasoned, there are lots of ways that research errors can be tied to poor statistical analyses — including haste-induced shortcuts, technical confusion, and outright manipulation.

Instead, upon arrival in the conference hall, he noticed a strange absence. “I don’t believe that I met any other statisticians,” he said.

ADVERTISEMENT

At least one did speak at the three-day event — Giovanni Parmigiani, a professor of biostatistics at Harvard University. And Mr. Parmigiani and other experts assembled by the National Academies did cite statistical rigor as one of the key areas needing improvement.

But Mr. Fournier sees an oversight at a more fundamental level. In all fields, he said, researchers need either to develop a working knowledge of statistics or to include someone with statistical expertise on their research teams.

And with that expertise, scientists should think with more statistical nuance about questions such as whether a research finding is statistically significant, Mr. Fournier said.

Many studies answer that question with a simple “yes” or “no,” relying on a calculation called a p-value to do so. For a p-value of .05, as is typical, a study’s finding will be deemed significant if researchers identify a 95-percent chance that it is genuine.

ADVERTISEMENT

More useful, Mr. Fournier said, would be a practice in which yes-or-no declarations would be replaced in journal articles by more specific estimates of how likely it is that a particular research observation did not just randomly occur: such as 1 in 20, or 1 in 100, or 1 in 1,000.

That numerical specificity of estimates may already exist inside many articles, Mr. Fournier said. But highlighting it in words, he said, should help emphasize what statisticians know to be true — science cannot make definitive yes-or-no declarations in most cases — and perhaps also encourage the publication of studies now abandoned in the belief they failed to show a useful outcome. Better statistical expertise also could help scientists construct experiments that are more likely to be reliable in the first place, he said.

One of the conference organizers, Victoria Stodden of the University of Illinois at Urbana-Champaign, said she recognizes the ways that biases of various types — financial conflicts of interest, academic promotion incentives, and the allure of fame — can contribute to irreproducibility problems in science.

But Ms. Stodden, an associate professor of information sciences, said she agrees that the ongoing misuse of statistics is a broader problem. While researchers may need to work harder to include statisticians on their teams, she said, statisticians also must to work harder to study how they could be more helpful to their interdisciplinary colleagues.

ADVERTISEMENT

“Developing a research agenda within the statistical community to address issues surrounding reproducibility is imperative,” she said.

‘A Tool, Nothing More’

For his part, Mr. Vijg isn’t convinced his team failed basic statistical analysis. His paper used records from sources that included the International Database on Longevity and the Human Mortality Database. It then made calculations suggesting that, while average human life expectancy may continue to increase, the maximum of age of the oldest surviving humans will not substantially move beyond about 115 years.

“We went through a highly experienced and reputed statistician before submitting the work,” Mr. Vijg said in a written exchange about the criticisms. At the same time, he argued that resolving differences in findings between competing labs is less a matter of procuring advanced statistical expertise and more a matter of the two groups getting together and identifying variations in their experimental conditions.

“Look, statistics is a tool, nothing more,” he said. “It certainly is not the arbiter of scientific truth.”

Look, statistics is a tool, nothing more. It certainly is not the arbiter of scientific truth.

An author of another of the five critiques published Wednesday by Nature, Nicholas J.L. Brown of the University of Groningen, said the case exhibits multiple problems seen across science — including statistical errors and some researchers’ basic pursuit of fame.

ADVERTISEMENT

The statistical errors, wrote Mr. Brown and his colleagues at Groningen, included a failure by Mr. Vijg’s team to compare the fit of its model to alternatives, and the use of small sample sizes that failed to properly handle the case of a lone outlier, Jeanne Calment of France, who died in 1997 at the record age of 122.

Mr. Vijg said repeatedly that his Nature paper made no “definitive statement” about a maximum human age and that he felt “amazement” that anyone might think otherwise. But he acknowledged approving a news release about his study issued by Albert Einstein College with the headline: “Maximum human lifespan has already been reached, Einstein researchers conclude.”

The scientific question at hand never even seemed to make much sense, said Mr. Brown, a doctoral student in health psychology at Groningen, because advances in average human lifespan are far more important than the future maximum age of a single person. “The whole article might as well have been designed to create clickbait headlines,” he said.

That type of low-value scientific pursuit is only becoming more common with the advent of modern computer-processing capabilities, Mr. Brown said. Computers let people “explore a half-million alternative realities in 10 minutes,” and then pick out something that seems interesting, without spending too much time on developing meaningful hypotheses, he said.

ADVERTISEMENT

Without qualified statistical experts to guide them, researchers will continue to encounter big problems, Mr. Brown said. “Statistics is demanding in the same way as flying a plane, but many scientists only have the equivalent of a driver’s license,” he said. “As a result, they’re crashing into the side of a mountain on a rather regular basis.”

Paul Basken covers university research and its intersection with government policy. He can be found on Twitter @pbasken, or reached by email at paul.basken@chronicle.com.

We welcome your thoughts and questions about this article. Please email the editors or submit a letter for publication.
Tags
Scholarship & Research
Share
  • Twitter
  • LinkedIn
  • Facebook
  • Email
Paul Basken Bio
About the Author
Paul Basken
Paul Basken was a government policy and science reporter with The Chronicle of Higher Education, where he won an annual National Press Club award for exclusives.
ADVERTISEMENT
ADVERTISEMENT

Related Content

An Attempt to Replicate Top Cancer Studies Casts Doubt on Reproducibility Itself
Can Science’s Reproducibility Crisis Be Reproduced?
The Results of the Reproducibility Project Are In. They’re Not Good.
New Center Hopes to Clean Up Sloppy Science and Bogus Research

More News

Mangan-Censorship-0610.jpg
Academic Freedom
‘A Banner Year for Censorship’: More States Are Restricting Classroom Discussions on Race and Gender
On the day of his retirement party, Bob Morse poses for a portrait in the Washington, D.C., offices of U.S. News and World Report in June 2025. Morse led the magazine's influential and controversial college rankings efforts since its inception in 1988. Michael Theis, The Chronicle.
List Legacy
‘U.S. News’ Rankings Guru, Soon to Retire, Reflects on the Role He’s Played in Higher Ed
Black and white photo of the Morrill Hall building on the University of Minnesota campus with red covering one side.
Finance & operations
U. of Minnesota Tries to Soften the Blow of Tuition Hikes, Budget Cuts With Faculty Benefits
Photo illustration showing a figurine of a football player with a large price tag on it.
Athletics
Loans, Fees, and TV Money: Where Colleges Are Finding the Funds to Pay Athletes

From The Review

A stack of coins falling over. Motion blur. Falling economy concept. Isolated on white.
The Review | Opinion
Will We Get a More Moderate Endowment Tax?
By Phillip Levine
Photo illustration of a classical column built of paper, with colored wires overtaking it like vines of ivy
The Review | Essay
The Latest Awful EdTech Buzzword: “Learnings”
By Kit Nicholls
William F. Buckley, Jr.
The Review | Interview
William F. Buckley Jr. and the Origins of the Battle Against ‘Woke’
By Evan Goldstein

Upcoming Events

Plain_Acuity_DurableSkills_VF.png
Why Employers Value ‘Durable’ Skills
Warwick_Leadership_Javi.png
University Transformation: A Global Leadership Perspective
Lead With Insight
  • Explore Content
    • Latest News
    • Newsletters
    • Letters
    • Free Reports and Guides
    • Professional Development
    • Virtual Events
    • Chronicle Store
    • Chronicle Intelligence
    • Jobs in Higher Education
    • Post a Job
  • Know The Chronicle
    • About Us
    • Vision, Mission, Values
    • DEI at The Chronicle
    • Write for Us
    • Work at The Chronicle
    • Our Reporting Process
    • Advertise With Us
    • Brand Studio
    • Accessibility Statement
  • Account and Access
    • Manage Your Account
    • Manage Newsletters
    • Individual Subscriptions
    • Group and Institutional Access
    • Subscription & Account FAQ
  • Get Support
    • Contact Us
    • Reprints & Permissions
    • User Agreement
    • Terms and Conditions
    • Privacy Policy
    • California Privacy Policy
    • Do Not Sell My Personal Information
1255 23rd Street, N.W. Washington, D.C. 20037
© 2025 The Chronicle of Higher Education
The Chronicle of Higher Education is academe’s most trusted resource for independent journalism, career development, and forward-looking intelligence. Our readers lead, teach, learn, and innovate with insights from The Chronicle.
Follow Us
  • twitter
  • instagram
  • youtube
  • facebook
  • linkedin