By Alice Richardson
In interdisciplinary research it’s common for at least some data to be analysed using statistical techniques. Have you been taught to look for ‘p < 0.05’ meaning that there is a less than 5% probability that the finding occurred by chance? Do you look askance at your statistician colleagues when they tell you it’s not so simple? Here’s why you need to believe them.
The whole focus on p < 0.05 to the exclusion of all else is a historical hiccup, based on a throwaway line in a manual for research workers. That manual was produced by none other than R.A. Fisher, giant of statistical inference and inventor of statistical methods ranging from the randomised block design to the analysis of variance. But all he said was that “[p = 0.05] is convenient to take … as a limit in judging whether a deviation is to be considered significant or not.” Convenient, nothing more!
Looking solely for p < 0.05 and deciding that the result is significant or not has the effect of replacing a number that can range between 0 and 1 with a binary decision (significant or not significant). This is a waste of information, an inefficient use of experimental resources.
There’s definitely a feeling amongst statisticians that researchers need to embrace a world beyond p < 0.05. In 2016 the American Statistical Association published a statement on p values (Wasserstein and Lazar 2016). Its aim was to alert the research community to the problems associated with an over-reliance on p < 0.05, and to propose some principles for future research to follow.
However, this statement did not mark the end of a debate. The American Statistical Association has published again (Wasserstein, Schirm and Lazar 2019), and not just the editorial cited but a collection of over 40 individual articles. This collection encompasses everything from the history of the p value debate through further alternatives to the p value, such as effect sizes, Bayesian factors and so on to changing the balance in statistics education.
The science press has picked up on the collection, with a Nature article (Arnheim, Greenland and McShane 2019) attracting the signatures of over 800 eminent statisticians and scientists who are keen to see continued reduction in the weight attached to p < 0.05.
Statisticians are thinking hard about how to do this. Educators are calling for revisions to standard introductory statistics courses to emphasise statistical thinking. Some are taking a hard line, such as the Journal of Basic and Applied Psychology in 2016 banning the publication of p-values in their journal, as was widely reported in the scientific press at the time. It makes me feel as though every research department should have a sign over the door saying:
“Abandon Statistical Significance, all Ye who Enter Here!”
My thoughts on the way forward revolve around two concepts central to considering complexity:
- embracing uncertainty; and,
- thinking critically.
These ideas are hardly new – interdisciplinarians and other researchers have been advocating for this for years, and my view is that now is the time for these practices to become second nature.
Asking questions like: So what? Compared to what? How precise are the estimates? What was the model? Are assumptions of independence and random sampling likely to have been met? How robust are the results to changes in or departures from the model? Being transparent about responses to these are the way that science will advance.
Researchers have been painted into a corner where the maxims of “publish or perish” and “p < 0.05 or it’s not publishable” drive the research agenda. It’s not going to be easy to move to a new world, which is where I think complexity scientists come in.
How do you think we could progress these changes? Are there ideas that statisticians could learn from complexity scientists? How would you encourage moves towards embracing uncertainty and thinking critically?
References:
Arnheim, V., Greenland, S. and McShane, B. (2019). Scientists rise up against statistical signficance. Nature, 576: 305–307.
Wasserstein R.L., and Lazar, N.A. (2016). The ASA’s statement on p-values: context process and purpose. American Statistician, 70: 129-133.
Wasserstein, R.L., Schirm, A.L. and Lazar, N.A. (2019). Moving to a world beyond ‘p < 0.05’. American Statistician, 73, supp1: 1–19.
Biography: Alice Richardson PhD is a biostatistician in the National Centre for Epidemiology and Population Health, Research School of Population Health at The Australian National University (ANU) in Canberra, Australia. Prior to commencing at ANU she taught introductory statistics at the University of Canberra for twenty years, providing a wealth of opportunities for communicating the complexities of “p < 0.05” to a diverse audience. Her research now focuses on imputation of missing data in highly structured data sets in order to extract maximum value from complex data collections.
Alice Richardson is a member of blog partner PopulationHealthXchange, which is in the Research School of Population Health at The Australian National University.