To the Editor:
Your article, “A New Theory on How Researchers Can Solve the Reproducibility Crisis: Do the Math” (The Chronicle, June 28), raises some good points about the need for statistical rigor in scientific research. However, we felt it was unfair of you to single out our article in Nature, especially since it does not suffer from any of the statistical shortcomings enumerated in the article.
First, you report that we did not consider the null hypothesis. However, we did consider the null hypothesis and reported in our paper the p-value, the probability of observing what we did if the null hypothesis was true. (Note that this definition of the p-value is slightly different from the one you provide in your article. There are many misunderstandings regarding the meaning of the p-value — so many that Wikipedia has an entire article devoted to the subject).
Another way of improving statistics identified in the article is the replacement of simple yes-or-no declarations with more probabilistic estimates, such as 1 in 20. We strongly agree with this approach, and that is why we reported in our paper the odds of someone living past age 125, which we calculated to be less than 1 in 10,000. This number was also reported in the press release, and we were careful to emphasize in interviews with journalists that the limit we were describing was not one of impossibility, just extreme improbability. A journalist could stop just at the title of the press release, or could pore over our many statements to cherry-pick one that we neglected to nuance, but that would say more about the quality of reporting than the science itself.
You conclude your article with several media-friendly quotes on statistics by Nick Brown, who also claims our findings are the result of a single outlier and insinuates that our paper was generated for the purpose of “clickbait headlines.” However, both statements are false. When we remove the data point for that outlier — the famous Jeanne Calment — the statistics still point to the same conclusion; we reported this in our reply to his comment in Nature. And far from being fame-seekers, we are simply scientists doing our best to come to a better understanding of the world. Indeed, the furor sparked by our paper has been at times an unwelcome distraction from our research, and we have been disappointed to see careful consideration of our nuanced findings eschewed in favor of baseless speculation about our personal motives.
Xiao Dong
Brandon Milholland
Jan Vijg
Albert Einstein College of Medicine
Bronx, N.Y.