Statistical Errors Are Often Not Due to Mathematical Errors

To the Editor:

I agree with the main claim of your article, “A New Theory on How Researchers Can Solve the Reproducibility Crisis: Do the Math” (The Chronicle, June 28), that scientists would benefit from consulting more regularly with statisticians. However, I believe that you (1) undercut your own message by making some incorrect assertions about statistical concepts, and (2) focus too narrowly on the importance of the role of mathematics in statistical reasoning.

If one assumes that statistics is largely mathematics, as many do, and as you seem to imply in your title, then errors in statistics and data analysis made by scientists will appear to be mathematical errors. But often errors that arise in data analysis — errors that contribute to the reproducibility crisis — are due to the misinterpretation of statistical concepts. For example, it is common among users of statistics to conflate practical and statistical significance, and to misinterpret the p-value of statistical hypothesis testing as the probability that the null hypothesis is true. These errors are made by many, including you; you write, “for a p-value of 0.05…a study’s finding will be deemed significant if researchers identify a 95-percent chance that it is genuine.” With respect to the latter error, a p-value is calculated under the assumption that the null hypothesis is true. So, it cannot, by definition, tell us the probability that the null hypothesis is true or that an alternative hypothesis/finding is “genuine.” These and other misinterpretations of p-values are so common that the American Statistical Association recently published a statement clarifying their proper use and interpretation. Importantly, these misinterpretations are not mathematical issues but philosophical ones.

These errors suggest something important. Mathematical skills are necessary for success in statistics, but they are far from sufficient. Statisticians — and researchers using statistics — ought to have a nuanced understanding of the concepts that mathematics helps them quantify. On my view, to become an expert in the analysis and proper interpretation of statistical concepts, one ought to become a better philosopher of statistics. Otherwise, one is flying a plane with only a driver’s license.

Brian Zaharatos
Department of Applied Mathematics