The Need for Nuance in the Null Hypothesis Significance Testing Debate

Introduction

While the human cognitive machinery is impressive in many ways, it has limitations and biases that hamper our ability to spontaneously do good science. Examples of such biases are (a) our tendency in many situations toward overconfidence in our beliefs and (b) an overly trigger-happy pattern detection mechanism that tends to see patterns in what is actually just noise; see, for example, Häggström (2013). This necessitates the use of rigorous statistical methods to circumvent these biases. The most widely used of these, from the 20th century and onward, is the so-called frequentist statistics, of which null hypothesis significance testing (NHST) forms a central part; see Salsburg (2001) for a reasonably balanced history. NHST has, however, come under some fire in recent years from critics who claim that its practice suffers from major problems. And indeed, there are problems, but that does not make all criticisms correct. In this article, I will try to add some much-needed nuance to the discussion, to distinguish various strands of the anti-NHST view, and to evaluate their merits. The main focus will be on the practice of NHST in psychology and related sciences. My eventual verdict will be that, on one hand, we cannot do without NHST in our statistical toolbox—a position that implies that the fiercest critics of NHST who want to abolish the practice altogether go too far—while, on the other hand, there are features of contemporary NHST practice that need to change.

This article is organized as follows. At the end of this introductory section, I will recall basic NHST terminology. The second section is a quick tour through some of the main malpractices that haunts present-day NHST practice. In the third section, I comment on the positions of a few of the most influential contributors to contemporary NHST debate. The reader will come out of that section with the (correct) impression that the debate is multifaceted and somewhat chaotic. In the fourth section, I offer an attempt at providing a least a bit of structure to the debate through a distinction between different strands of anti-NHST position; I believe that if everyone who identifies themselves as anti-NHST would make an effort to clarify their own stance with respect to my classification, then the clarity of the debate as a whole would improve nontrivially. The reason for this is explained in the fifth section, where I also offer some further conclusions regarding what ought to be done in order to improve statistical practice and science as a whole.

There is no need here for a mathematically fully precise treatment of NHST, but we do need the basic terminology. Statistical hypothesis testing is always relative to some null hypothesis, which typically states that some effect or some parameter is zero. Given the data, the p value can loosely be defined as what the probability would be of getting at least as extreme data as those we actually got, if the null hypothesis were true. The looseness of this definition comes from the fuzziness of the term “extreme,” which actually needs to be carefully defined before analyzing the data. The standard so-called Fisherian definition gives much flexibility about defining “extreme,” whereas the other classical approach—that of Neyman and Pearson—requires the specification of an alternative hypothesis, and defining “extreme” so as to maximize power (see below) with respect to that alternative hypothesis. See, for example, Lehmann and Romano (2008) for a more careful discussion.

If the p value ends up below a given threshold—which is called the significance level and which (mostly for historical reasons) is most commonly taken to be 0.05—then statistical significance is declared, which is typically considered to provide (at least some) reason for suspecting that the null hypothesis is false. For a given significance level, the power of a testing procedure is a function of the effect size (or, more generally, of the precise choice of alternative hypothesis), and it is defined as the probability of obtaining statistical significance under that effect size.

A (Fisherian) 95% confidence interval—or, more generally, confidence set—for a parameter ρ is the set of all r such that, with the given data, testing the null hypothesis ρ = r would not yield statistical significance at level 1 − 0.95 = 0.05. This guarantees, regardless of the true value of ρ (but provided all other model assumptions are true), that the probability of getting data that yield a confidence interval that includes the true ρ is at least 0.95. The concept generalizes in the obvious way to confidence intervals with confidence levels other than 95%.

Not All Is Well

It is hard to imagine how 20th and early 21st century science could have done without the NHST machinery, and I will argue later in this article that we still need it. It must, however, be admitted that current NHST practice suffers from a number of endemic problems. The following are among the most detrimental:

Likewise, in a situation where a 95% confidence interval has been calculated, it may be tempting to conclude that the true parameter value sits in that interval with probability 0.95, but again this would be to commit the fallacy of the transposed conditional; see Morey, Hoekstra, Rouder, Lee, and Wagenmakers (2016) for extensive discussion of this instance of the fallacy.

There is a much-employed but highly unfortunate terminological convention in the use of the word “likely,” which I believe contributes to the confusion. Consider the case of statistician Yudi Pawitan’s influential book In All Likelihood. Discussing an experiment where a coin with unknown heads-probability θ is tossed 10 times, resulting in 8 heads, he writes that we are in a position to conclude that θ “is very unlikely to be very small,” and that in contrast “θ = 0.6 or θ = 0.7 is likely” (Pawitan, 2001, p. 21). Since, in ordinary English, “likely” is synonymous to “having high probability,” this very much looks like a case of the fallacy of the transposed conditional. Pawitan, however, is much too sophisticated a statistical thinker to make that mistake; what he does is that he implicitly defines the statement “θ = 0.7 is likely” to mean that the likelihood function L(θ)—defined as the probability of the obtained data provided the parameter value is θ—takes a large value for θ = 0.7. For another example, we may note that the same language appears in a recent column by the president of the Association for Psychological Science:

The information that the binomial likelihood function conveys is extremely intuitive. It says that given that we have observed 7 successes in 10 tries, the probability parameter of the binomial distribution from which we are drawing [ . . . ] is very unlikely to be 0.1; it is much more likely to be 0.7, but a value of 0.5 is by no means unlikely. (Gallistel, 2015)

It is difficult or impossible for the reader, confronted with a passage like this one, to judge whether the author is under the spell of the fallacy of the transposed conditional, or if he is merely adhering to the same unfortunate terminological convention as Pawitan. In any case, the chosen language invites confusion and almost begs statistically unsophisticated readers to commit the fallacy of the transposed conditional.

A very common special case of failure to account for multiple testing arises when both positive and negative values of a parameter ρ are possible, and the null hypothesis is that ρ = 0. Unless there is good reason to expect a nonzero parameter to be in a certain direction (positive or negative) one should normally carry out a so-called two-sided test, which takes the possibilities of both ρ < 0 and ρ > 0 equally into account. If there is good reason to expect ρ < 0 rather than ρ > 0, a one-sided test can be decided on beforehand, but the cheap trick to first look at the data and then decide on the direction of the one-sided test to get the best p value is incorrect; in effect, such a procedure is two-sided but produces a better (half as large) p value than the correct two-sided procedure. Ruthless (or naïve) researchers can do this and then produce an ad hoc explanation for why the chosen direction was to be expected before the data were collected and analyzed. Doing so can be seen as a special case of the next malpractice.

Some Highlights From the Debate

During large parts of the 20th century, intense debate took place within the statistics community between on one hand defenders of NHST and other methods in frequentist statistics, and on the other hand advocates of Bayesian statistics. I shall refrain from discussing this history (but see Salsburg, 2001) and instead focus on how researchers in a wide range of applied fields have increasingly begun to highlight their dissatisfaction with NHST practice. Researchers in psychology and related sciences have been especially active in this discourse. Carver (1978) and Cohen (1994) are two relatively early examples, both of them pointing out several of the misconceptions and malpractices discussed in the previous section, plus a few others.

A Taxonomy of Anti-NHST Positions

I wish that participants of the NHST debate expressing anti-NHST sentiments (including those cited in the third section) would clearly state whether they wish to rule out as unscientific all arguments involving or building on calculation of p values, or whether they merely want to ban explicit reference to such calculation in scientific papers. In other words, they ought to declare whether their position is strongly or weakly anti-NHST, in the terminology that I hereby propose.

Of course, it is conceivable that someone who identifies herself as anti-NHST might find that neither strongly nor weakly anti-NHST accurately captures her position, but in such cases, it is likely to be illuminating if she went on to explain wherein the mismatch lies.

Cumming (2009, 2012a, 2012b, 2014) advocates the use of confidence intervals, as do (albeit to a lesser extent) Ziliak and McCloskey (2008), and since confidence intervals are derived from p value calculations, these authors’ views do not match the strongly anti-NHST position. So their respective positions are at most weakly anti-NHST, but it remains to be heard whether they find the weakly anti-NHST position to be an accurate description of their respective views.

The case of Trafimow and Marks (2015) is more difficult to interpret. While some of their rhetoric suggests the strongly anti-NHST position, the following passage is more suggestive of the weakly anti-NHST position:

Will manuscripts with p-values be desk rejected automatically? [ . . . ] No. If manuscripts pass the preliminary inspection, they will be sent out for review. But prior to publication, authors will have to remove all vestiges of the NHSTP (p-values, t-values, F-values, statements about “significant” differences or lack thereof, and so on). (p. 1)

It may also be noted that as one of their preferred alternatives to NHST, they “encourage the use of larger sample sizes than is typical in much psychology research, because as the sample size increases, descriptive statistics become increasingly stable and sampling error is less of a problem” (Trafimow & Marks, 2015, p. 1). This is noteworthy, because while their statement about the advantage of larger sample sizes is correct, calculation of hypothetical p values is necessary in order to derive this monotonicity property and to work out how large a sample size will be needed to attain a desired level of precision.

In my opinion, this last observation demonstrates the untenability of the strongly anti-NHST position. Apparently even Trafimow and Marks recognize that instability of statistical estimates and sampling error are a problem, but to embrace the strongly anti-NHST position would be to deprive ourselves of the ability to calculate how much of a problem it is, and how large a sample size we would need to bring the problem down to an acceptable level. (One of the referees suggested that one might be able to judge the sample size needed while remaining strongly anti-NHST by calculating standard deviations rather than p values. But this overlooks the underlying reason why standard deviations are relevant for this. To go from a small standard deviation to a small probability of a highly misleading statistical estimate one needs to apply Chebyshev’s inequality. That step, which is so standard that it is usually left implicit, is precisely the bounding of a p value.)

On the other hand, the weakly anti-NHST position strikes me as unacceptable for other reasons. To accept a scientific methodology, but to require that it is kept secret, banning any mention of it in scientific publications, flies straight in the face of the transparency that is a cornerstone of good scientific method.

To summarize, it seems that neither the strongly nor the weakly anti-NHST position is acceptable. Well then, might there be some third strand of anti-NHST that makes better sense—perhaps one that accepts the need to calculate p values as well as the principle of openness about the scientific arguments that are used, at the same time as it recognizes the need to act in ways to keep in check the various misuses and misinterpretations of NHST discussed in the second section? Let me propose the following.

This is in fact a strand of anti-NHST sentiment that I can embrace. I do not claim any originality in taking on this position. On the contrary, I daresay it is held by virtually all professional statisticians. Note that not holding the very weakly anti-NHST position is tantamount to accepting the practice of sizeless science (Item (d) in the second section). Since accepting that practice means accepting the scientific literature to be filled with reports of statistically significant findings without mentioning whether the observed effect sizes are large enough to matter, the very weakly anti-NHST position becomes pretty much obligatory for anyone with a reasonable sense of good scientific practice.

What Needs To Be Done

That practitioners of statistical methods in a variety of applied disciplines have taken over much of the initiative from statisticians in the debate on how to do statistics is in my opinion a welcome development; a broad interest in these matters is likely to help the scientific community move forward on them. I only wish that fewer of these participants took on poorly grounded anti-NHST positions.

In my humble opinion, statisticians still have a lot to contribute to these discussions, and it is therefore important that they take part. Without their participation, much of the body of knowledge of statistical inference accumulated during the 20th century and onward risks getting lost. For instance, if the editors of BASP had had contact with professional statisticians, they would probably have been spared the embarrassment of banning NHST methods on the confused grounds that they did. Another case in point—less dramatic but nonetheless interesting—is the article by cognitive scientist John Kruschke (2010), who advocates the abandonment of NHST methods, arguing that their violation of the so-called strong likelihood principle (SLP) is unacceptable. He does not use that terminology, however, being apparently unaware both of SLP being known at least since Savage (1962), and of the long controversy over SLP since then (see, e.g., Barndorff-Nielsen, 1985; Cox & Hinkley, 1974; Pawitan, 2001). I am sure his article would have been much more interesting and useful to the scientific community if there had been statisticians around to inform him of this discourse so that he could have taken into account the arguments for and against SLP that have accumulated over the years, rather than trying to reinvent the discussion from scratch.

One of the referees for this article asked what role I hope the distinction proposed in the fourth section between strongly and weakly anti-NHST positions will play in the debate. I am a bit reluctant to give away my answer to this question, for reasons analogous to why one might not want to ruin a joke by explaining it, but all right, here it is: none. I do not expect or even hope to encounter future debates whose contestants have, for clarity, signed up for the strongly versus and weakly anti-NHST camp, respectively. Rather, my ambition is that anyone with an anti-NHST sentiment reading the fourth section, will, faced with the strongly versus weakly anti-NHST distinction, realize that both positions are untenable, and therefore give up being anti-NHST altogether (or switch to the third option of being very weakly anti-NHST, which actually is not an anti-NHST position at all, but rather one about what constitutes good NHST practice). Once that is accomplished, there is no further need for the distinction.

Besides the much-needed presence of statisticians in the debate over what constitutes good statistical practice, their presence is even more badly needed in the direct workings of applied sciences—in the actual research projects. Because, as I have argued in the fourth section, NHST cannot be abandoned without seriously hurting science, and therefore the competence level among practitioners of statistical methods needs to improve, so as to avoid statistical pitfalls and malpractices such as those discussed in the second section. This, I believe, requires both a better (qualitatively as well as quantitatively) statistics part of the undergraduate and graduate training in other disciplines, and an increased presence of statisticians in research projects in these disciplines.

A particular point where I think an improved understanding among scientists would greatly benefit science itself is the interpretation of a statement like “p < 0.05.” Suppose everyone understood that statistical significance on level 0.05 on its own is no more than (at most) the observation that data suggest that an effect may well be present, an observation that can be taken to warrant further study of the possible presence of the effect. If scientists in general understood this, then the low success rate of replication studies (Item (g) in the second section) might no longer deserve the term “replication crisis,” but would instead represent a reasonably healthy state of science. As things are today, “p < 0.05” is way too often taken to mean something like “beyond reasonable doubt,” or to serve as an excuse for cocksure statements such as the title of the Bygren et al. (2014) article that I quoted in Item (e) of the second section. For that kind of interpretation of statistical significance, way lower significance levels are needed—although at this point I am reluctant to give a number, because how low it is reasonable to go depends very much on the particulars of the problem, including how a priori plausible the existence of the searched-for effect seems to be. The p value alluded to by Fisher (1956) in the passage quoted in the section “The BASP Declaration” is about 0.00003—which in some contexts I might consider small enough to state with a high degree of confidence that the null hypothesis is probably false, but certainly not in all. For instance, the Big Data revolution in areas like bioinformatics (see, e.g., Efron 2012) has led to routine experiments involving thousands or even millions of hypotheses to be tested simultaneously; in such situations, a raw p value of 0.00003, untouched by multiple inference analysis, does not even warrant a raised eyebrow.

A more well-calibrated and realistic interpretation of the evidential value of statements like “p < 0.05” would likely (or at least should) come hand in hand with placing a higher value on replication studies. Unfortunately, today such studies are generally not held in high regard. The view of Mitchell (2014) is extreme (and utterly confused due to committing fallacy (a) of the second section), but the dominant view in most empirical sciences today is depressingly similar to the one expressed by the anonymous professor in following anecdote, told by Richard Feynman in a commencement speech at Caltech in 1974:

When I was at Cornell, I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment. [ . . . ] It had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do A. So her proposal was to do the experiment under circumstances Y and see if they still did A.

I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person—to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know that the real difference was the thing she thought she had under control.

She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1947 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happens. (Feynman, 1985, p 344)

That was in the 1940s, but the same contemptuous view of replication studies is still dominant. It is time we overcome it.