On Some Assumptions of the Null Hypothesis Statistical Testing

Introduction

In social sciences, the majority of the events are contingent, full of uncertainties and permeated by nuisance variables. For instance, cognitive skills are affected by a number of factors such as education, culture, age, tiredness, genetics, and so on. It is impractical to contemplate all factors that influence a specific cognitive skill. Probability and statistical models are mathematical tools used to handle contingent and uncertain events (Fisher, 1955; Kadane, 2011; McCullagh, 2002). These tools are defined in terms of sets and functions, which are fully consistent with the modern formulation of mathematics. ¹

Statistical models are employed to make inferences about unknown quantities and to test the consistency of scientific statements with the observed data (Fisher, 1955). However, statistical models have domains of applicability, internal rules, principles, limitations, and so on (Fisher, 1922; Dempster, 1968; Hájek, 2008). It is important to understand those internal features in order to avoid inadequate interpretations obtained from prohibited inferential rules (Berger & Sellke, 1987; Fisher, 1955; Kempthorne, 1976; Lavine & Schervish, 1999).

The main goal of this article is to discuss some hidden assumptions underlying the classical statistical models ² and null hypotheses, see sections “Statistical Models” and “Scientific and Statistical Hypotheses.” The section “Definition of p-Values” discusses the formal definition of a p-value. The section “Problems of p-Values and an Alternative Measure of Evidence” presents its limitations and reviews a new classical measure of evidence, called s-value, that overcomes some limitations of the p-value. The penultimate section “Numerical Examples” provides some standard examples on testing population averages that illustrate the following feature of p-values: They do not respect the reasoning of the logical consequence. The reasoning of logical consequence is as follows: If one hypothesis H 01 implies another one H 02 , then, by the logical consequence, we would expect more evidence against H 02 than that against H 01 . For example, let μ 1 and μ 2 be two population means. From p-values, it is possible to obtain the following striking result: with the same observed data, one can find more evidence against μ 1 = μ 2 than against μ 1 = μ 2 = 0 , even though the latter necessarily implies the former. The final section concludes the article recapitulating the main points discussed in the article.

Statistical Models

It is difficult to introduce probability and statistical models by adopting an easy language without ambiguity. This article avoids the set-theoretic notation and will not introduce the primary probability space where all quantities are well defined (e.g., random variables, statistics, estimators, induced spaces, etc.). The reader should be aware that the language used here is informal, and to avoid ambiguities it will be required to make many textual caveats. The reader is referred to Cox and Hinkley (1974), Schervish (1995), Lehmann and Casella (1998), and McCullagh (2002) for a detailed discussion on statistical models.

Roughly speaking, the steps before choosing a statistical model are (not necessarily in this particular order) the following:

The practitioner must have prior knowledge to construct an appropriate experiment to access the quantities of interest, for each field has its idiosyncrasies that must be taken into account. The experiment may be randomized in specific strata or layers or clusters (different treatments, genders, groups of risk, and so on), and these considerations should guide the researcher to choose the class of probability distributions that will be considered in the statistical model. Typically, in scientific experiments, there are direct observable quantities (age, gender, measured height and weight, etc.) and unobservable quantities (intelligence, “feelings of morale,”“sense of belonging,” etc.). These quantities might be either random or nonrandom and are ingredients of a statistical model. All random quantities must be well-defined in a probability space.

In this article, random observable quantities are denoted by uppercase Latin letters, say X or T , and their observed counterparts are denoted by lowercase Latin letters, say x or t . Random and nonrandom unobservable quantities are denoted by the Greek letters γ and θ , respectively. The unobservable random quantities are called latent random variables (Bollen, 2002). Let us informally represent a statistical model by the triplet

( X , γ , M ) ,

where X represents the observable random variables, γ represents the latent random variables and M is a family containing joint probability (density) functions of the random variables, that is, M = { g θ : θ ∈ Θ ⊆ ℝ p } , P ∈ ℕ where g θ is a possible joint probability (density) function of ( X , γ ) , for each θ ∈ Θ . It should be clear that θ ∈ Θ is an indexer of possible probability distributions, it is not a random variable. Through residual analyses, one can verify empirically if the family M is adequate or inadequate to model the observable data. It is not possible to assure that the family M contains the generator mechanism of the data, that is, the mechanism that effectively generates the data. Furthermore, the data’s generator mechanism might not even be translatable in terms of probability distributions.

When the probability distribution that governs the random quantities is known, then M contains only one element, namely M = { g } , where g ( x , γ ) ≡ f γ ( x ) f 0 ( γ ) is the joint probability (density) function of the observable and unobservable random variables, with

X | γ ~ f γ and γ ~ f 0 ,

where f γ is the probability (density) function of the random variable X given γ and f 0 is the probability (density) function of the random variable γ . Recall that in this latter case, it is assumed that the joint probability distribution that governs the random quantities is known. In this context, it is possible to provide full probabilistic descriptions of the random quantities (mean, variance, quantiles, marginal probabilities, joint probabilities, conditional probabilities, etc.). As aforementioned, in practice it is difficult (or even impossible) to known the generator of the random quantities and the family M typically has more than one element.

The formal statistical model is defined with sigma-fields and a family of probability measures (see, for instance, Lehmann & Casella, 1998; Lehmann & Romano, 2005; McCullagh, 2002; Patriota, 2013). The reader must keep in mind that Model (1) is a simplified version that shall help us understand some important features of the classical statistical model and the null hypothesis statistical testing.

Scientific and Statistical Hypotheses

In science, it is common to formulate statistical hypotheses to test scientific statements. A nontrivial step is to translate a scientific statement into statistical language. In the classical paradigm, a statistical hypothesis is a statement about probability distributions that potentially govern the experimental data. That is, in order to create a statistical hypothesis, one must be able to transform a scientific statement in terms of probability distributions. For instance, the statement “this coin is not biased” is typically transformed into “ P ( this coin turns up head ) = 0 . 5 ,” that is, the following is taken as a hidden principle:

The theoretical assumptions are made from the specific features of the chosen experiment. One experiment may be performed by independently throwing n times the coin over a smooth surface. The observable random variable is the number of times the coin turned up heads. In this simplified version, no latent variables are considered. Assuming that the coin cannot land on its edge, one statistical model that can represent this experiment is the binomial model ( X , M ) , where M = { g θ : θ ∈ ( 0 , 1 ) } with

g θ ( k ) = n ! k ! ( n − k ) ! θ k ( 1 − θ ) n − k , for k = 1 , … , n ,

where n ! is the usual factorial notation, θ is the probability that the studied coin turns up head and g θ ( k ) is the probability that the coin turns up heads exactly k times in the performed experiment. The scientific statement and its statistical counterpart are related by

The null hypothesis is then represented by H 0 : θ = 0 . 5 , that is H 0 is a statement about probabilities: “if the coin is not biased, then [by the above principle and model assumptions] the probability that the coin turns up head is 0.5.” Notice that, unless the practitioner is totally certain of the theoretical assumptions, evidence to reject H does not mean evidence to reject the scientific statement. Indeed, we have that not - H implies that either “This coin is biased” or “at least one of the theoretical assumptions is not adequate.”

Under the null hypothesis H 0 , the statistical model reduces to ( X , M 0 ) , where M 0 = { g 0 . 5 } . In general, the alternative hypothesis is defined to be H 1 : θ ≠ 0 . 5 and under this alternative hypothesis the statistical model is ( X , M 1 ) , where M 1 = { g θ : θ ≠ 0 . 5 } . Notice that the union of both restricted families under H 0 and H 1 must be the original family, that is, M 0 ∪ M 1 = M . This means that the original statistical model can be partitioned into two separated statistical models, namely the one generated under H 0 and the other generated under H 1 .

In the binomial model, it is implicitly assumed in the “Theoretical assumptions” that “ P (this coin turns up head)” does not change over all throws. Of course, this assumption is oversimplified for actual processes, since in each throwing the coin is submitted to impacts causing microscopic cracks, warps and, consequently, modifications in “ P (this coin turns up head)” over time. Other statistical models can be implemented by relaxing some of the imposed suppositions: (1) latent random variables can be incorporated to model dependence among the coin flips and (2) covariates may be inserted to model variations in θ . That is, by changing some “Theoretical assumptions,” many statistical models could be used to model the outcomes of the very same experiment.

The concept of coin bias can be further elaborated. One may prefer to relate the statement “this coin is not biased” with the structural topology of the coin, for example, types of symmetries around the mass center of the coin, and so on. Under this latter definition, it is possible to define degrees of bias based on a measure of symmetry and another completely different statistical model will emerge. This simple example illustrates the complexity of statistical models and the problem of translating a simple scientific hypothesis into a statistical language. This example is applied in problems with binary outcomes; for instance, the random variable X may be defined to be the number of allergic patients, out of n , who react positively to a specific treatment.

Definition of p-Values

A p-value is built with the purpose of capturing a disagreement between the observed data and the postulated null hypothesis. In this context, a first step is to define a positive real statistic T ≡ T H 0 , it is a function of the random sample X which depends on the null hypothesis H 0 , such that: the larger its observed value t , the stronger is the disagreement between the observed data and the null hypothesis H 0 (Cox, 1977; Mayo & Cox, 2006; Patriota, 2013). The set C H 0 ( t ) = { x : T ( x ) ≥ t } describes all sample values that have stronger disagreements with the postulated null hypothesis H 0 than the observed one. This set has three important elements, namely, the null hypothesis of interest H 0 , the random statistic T and the observed statistic t . Note that T strongly depends on H 0 .

If C H 0 ( t ) is small compared to the total set C H 0 ( 0 ) , then the observed experiment provides strong evidence against H 0 ; this happens when the observed t is large enough to lie in the extreme right tail of the statistic T ’s distribution. One way to measure the size of C H 0 ( t ) is through probabilities. As the null hypothesis states probability distributions that represent the scientific statement of interest, the p-value is computed for the case with the highest probability in H 0 . Let us consider the model without latent variables ( X , M ) , where M 0 = { g θ : θ ∈ Θ 0 } is the set of probability (density) functions restricted under the specifications of H 0 . Let P θ be the probability measure associated with g θ , that is, if g θ is a probability function, then P θ ( A ) = ∑ x ∈ A g θ ( x ) and if g θ is a probability density function then P θ ( A ) = ∫ A g θ ( x ) dx , where A ⊆ X is a measurable set. The p-value is formally defined by

p ( H 0 , t ) = sup θ ∈ Θ 0 P θ ( C H 0 ( t ) ) .

Therefore, as p ( H 0 , t ) is (greater than or equal to) the case with the highest probability in H 0 , the smaller the value of p ( H 0 , t ) , the larger is the evidence against H 0 . Formula (2) explicitly says that the classical p-value is not a conditional measure in the probabilistic sense, it is instead a conditional measure in the possibilistic sense. The reader should notice that the usual representation p-value = P ( T ≥ t | H 0 ) is inadequate, since (a) the probability P is meaningless in the context of classical statistical models and (b) the conditional probability is being misused, since its formal definition is being ignored. The conditional probability is defined by P ( A | B ) = P ( A ∩ B ) P ( B ) , where P ( B ) > 0 and A and B are events of the same type (they must be listed in the same sigma-field). As for random variables, the conditional probability is defined analogously for the probability (density) function g θ . In classical statistics, the events { x : T ( x ) ≥ t } and H 0 are not of the same type, for they are not listed in the same sigma-field; otherwise it is a Bayesian-like analysis. ⁴ In classical statistics, there is not a probability distribution defined over the subsets of M = { g θ : θ ∈ Θ } and as M cannot (even ideally) list all possible measures, a probability measure over the subsets of M would be conceptually ill-defined. ⁵

Problems of p-Values and an Alternative Measure of Evidence

The p-value is a statistical tool to verify a possible discrepancy between a fixed null hypothesis and the observed data. Nevertheless, there is a serious limitation in the use of p-values in nested hypotheses. Consider that the p-value’s computation under H 0 ( 1 ) is extremely complicated. Let H 0 ( 2 ) be an auxiliary hypothesis such that H 0 ( 1 ) ⇒ H 0 ( 2 ) , that is, if H 0 ( 1 ) is true, then H 0 ( 2 ) is true. By logical reasoning: if H 0 ( 2 ) is false, then H 0 ( 1 ) must also be false. The practitioner, led by this logical reasoning, would compute the p-value under H 0 ( 2 ) and conclude that if there is evidence to reject H 0 ( 2 ) , that is, the p-value computed under H 0 ( 2 ) is significantly small, then there must be evidence to reject H 0 ( 1 ) . However, p-values do not allow this latter logical reasoning. That is, it is not guaranteed that p ( H 0 ( 1 ) , t ) ≤ p ( H 0 ( 2 ) , t ) , see the next section for numerical examples. This happens because the test statistic T is built for a specific null hypothesis, therefore, the respective p-value is valid only for this specific null hypothesis; for more details, see, for instance, Schervish (1996) and Patriota (2013). In previous work, Patriota (2013) proposed an alternative classical measure of evidence that meets the above logical reasoning; it is called s-value and will be presented in what follows.

The general purpose of the s-value is almost the same as of the p-value: to verify a discrepancy of null hypotheses with the observed data, but maintaining all logical consequence among null hypotheses. In order to define s-values, let us consider the simplest statistical model without latent variables ( X , M ) , where M = { g θ : θ ∈ Θ ⊆ ℝ p } and let P θ be the probability measure associated with the probability (density) function g θ . The likelihood-ratio statistic is

λ ( θ ; x ) = g θ ( x ) sup θ ∈ Θ g θ ( x ) ,

provided that sup θ ∈ Θ g θ ( x ) > 0 . Notice that, 0 ≤ λ ( θ ; x ) ≤ 1 for all θ ∈ Θ . The likelihood-ratio confidence region with significance level α is defined by

Λ α ( x ) = { θ ∈ Θ : λ ( θ ; x ) ≥ c α ( θ ) } ,

where

P θ ( λ ( θ ; X ) ≥ c α ( θ ) ) ≥ 1 − α , inf θ ∈ Θ P θ ( λ ( θ ; X ) ≥ c α ( θ ) ) = 1 − α

and 0 ≤ c α ( θ ) ≤ 1 . The following equivalent notation may be used

P θ ( λ ( θ ; X ) ≥ c α ( θ ) ) ≡ P θ ( Λ α ( X ) ∋ θ )

The quantity P θ ( Λ α ( X ) ∋ θ ) is the probability of Λ α ( X ) to contain θ , under the measure P θ . This is the formal definition of a general confidence region for the parameter θ (Schervish, 1995).

For some statistical models (normal distribution in general), the following occurs:

P θ ( λ ( θ ; X ) ≥ c α ( θ ) ) = 1 − α for all θ ∈ Θ ,

in this case, the confidence region is said to be exact. For exact confidence regions, the value c α ( θ ) is the ( 1 − α ) × 100 % quantile of the random variable λ ( θ , X ) . Observe that Λ α ( x ) contains all θ ’s that generate likelihood values greater than (or equal to) c α ( θ ) times the largest likelihood value, namely, sup θ ∈ Θ g θ ( x ) . This set is intuitive, for it contains the optimal values for θ ∈ Θ according to the likelihood function. The definition of the s-value follows.

Definition 1. Let Θ 0 be a nonempty parameter subset related with H 0 and let Λ α ( x ) be the likelihood-ratio confidence region with significance level α . Then, the s-value is defined by

s ( H 0 ; x ) ≡ s ( Θ 0 ; x ) ≡ sup { α ∈ [ 0 , 1 ] : Λ α ( x ) ∩ Θ 0 ≠ ∅ } .

If Θ 0 = ∅ , define

s ( ∅ ; x ) ≡ 0 .

This definition is valid for general hypotheses. Let Θ 01 and Θ 02 be two parameter subsets related with the hypotheses H 0 ( 1 ) and H 0 ( 2 ) , respectively. In this context, if H 0 ( 1 ) ⇒ H 0 ( 2 ) , then Θ 01 ⊆ Θ 02 ; Patriota (2013) showed that the following always occurs s ( Θ 01 ; x ) ≤ s ( Θ 02 ; x ) . A possible interpretation for the s-value, under the regular conditions stated in Patriota (2013) and assuming that Θ 0 is nonempty and closed, reads

“ s ( Θ 0 , x ) is equal to the maximum significance level α M such that Λ α M ( x ) and Θ 0 have at least one element in common.”

The smaller s ( Θ 0 , x ) is, the more distant Θ 0 is from the maximum likelihood estimate of θ and, consequently, the more unlikely H 0 is according to the likelihood-ratio confidence region. Observe that, if H 0 : θ = θ 0 , where θ 0 is a given vector (or number if Θ ⊆ ℝ ), then Θ 0 = { θ 0 } and the s-value reduces to

s ( { θ 0 } ; x ) = max { α ∈ [ 0 , 1 ] : θ 0 ∈ Λ α ( x ) }

and its interpretation reads

“ s ( { θ 0 } , x ) is equal to the maximum significance level α M such that Λ α M ( x ) contains θ 0 .”

Therefore, the farther away θ 0 is from the center of Λ α ( x ) , which in regular conditions is the maximum likelihood estimative, the more the observed evidence is against H 0 . Patriota (2015) studied the likelihood-ratio statistic as a measure of evidence and compared it with the s-value and posterior distributions. González, Castro, Lachos, and Patriota (2016) employed the s-value to study confidence sets for observed samples.

Types of Decisions

In this section, some types of decisions are studied. Let θ ^ be the maximum likelihood estimative of θ , then, under regular conditions (Cox & Hinkley, 1974, Chap. 9), we have that θ ^ ∈ Θ and it exists.

First Case

No alternative hypothesis is defined, then the general advice of this paper is to use the s-value as a thermometer of discrepancy between null hypotheses and the observed data. The smaller is s ( Θ 0 , x ) , the stronger is the evidence against H 0 . Patriota (2013) showed that if θ ^ ∈ Θ 0 , then s ( Θ 0 , x ) = 1 and the observed data produce no evidence against H 0 , which does not mean evidence in favor of H 0 . In a working paper, we are showing that s-values are always greater than p-values (based on the likelihood-ratio statistic) for some specific models. This indicates that if a s-value is small, then the respective p-value must be even smaller. Therefore, one could just compute the s-value to verify discrepancies of the null hypothesis with the observed data. The use of s-values is also justified for general hypotheses, because p-values are much more difficult to compute than s-values and, furthermore, p-values do not satisfy the logical consequence.

Second Case

An alternative hypothesis H 1 is defined and let Θ 1 be its related parameter space. Patriota (2013) showed that, on one hand, if θ ^ ∈ Θ 0 , then s ( Θ 0 , x ) = 1 ; on the other hand if θ ^ ∈ Θ 1 , then s ( Θ 1 , x ) = 1 . If the practitioner wants to decide between H 0 or H 1 , then there are three possibilities

If s ( Θ 1 , x ) = 1 and s ( Θ 0 , x ) = a , then reject H 0 and accept H 1 whenever a is sufficiently small.

If s ( Θ 1 , x ) = b and s ( Θ 0 , x ) = 1 , then accept H 0 and reject H 1 whenever b is sufficiently small.

If s ( Θ 1 , x ) = s ( Θ 0 , x ) = 1 and neither a nor b are sufficiently small, then neither reject nor accept H 0 . More data are required.

The threshold values for a and b are being studied. They depend on the sample size, effect sizes, error of type I and II, power of the test, severity (Mayo & Cox, 2006; Mayo & Spanos, 2006), and/or other factors. Notice also that more than one alternative hypotheses H 1 , … , H k can be defined. It is possible to use the s-value in the latter context, but it is beyond the scope of this article.

Izbicki and Esteves (2015) investigated some properties of statistical test procedures, namely: monotonicity, intersection consonance, union consonance and invertibility. According to Izbicki and Esteves:

Monotonicity is a property related to nested hypothesis: if H 0 → H 0 ′ , then a testing scheme that rejects H 0 ′ should also reject H 0 .

Intersection consonance is a property related to conjunctions: if a testing scheme rejects “ H 0 AND H 0 ′ “, then it should also reject at least one of the hypotheses H 0 or H 0 ′ .

Union consonance is a property related to disjunctions: if a testing scheme rejects each of the hypotheses H 0 and H 0 ′ , then it should also reject the disjunction “ H 0 OR H 0 ′ .”

Invertibility is a property related with the null and alternative hypotheses: If a testing scheme rejects the null hypothesis, then it should accept the alternative one and vice verse.

The s-value satisfies the following property:

∀ Θ 0 ⊆ Θ , s ( Θ 0 , x ) = sup θ ∈ Θ 0 s ( { θ } , x ) .

By the property stated in Equation (3), the following property is entailed: for all Θ 0 ⊆ Θ 0 ′ ⊆ Θ , s ( Θ 0 , x ) ≤ s ( Θ 0 ′ , x ) . Provided that the hypotheses are statements regarding to the parameter space, namely H 0 : θ ∈ Θ 0 and H 0 ′ : θ ∈ Θ 0 ′ , we have that: (1) H 0 → H 0 ′ ⇔ Θ 0 ⊆ Θ 0 ′ ; (2) “ H 0 AND H 0 ′ ” ⇔ θ ∈ Θ 0 ∩ Θ 0 ′ ; and (3) “ H 0 OR H 0 ′ ” ⇔ θ ∈ Θ 0 ∪ Θ 0 ′ . By the property stated in Equation (3), it is straightforward to show that the testing scheme based on the s-value satisfies monotonicity, intersection consonance and union consonance. The testing scheme based on the s-value does not satisfy invertibility, since the s-value allows us to maintain both hypotheses whenever the observed evidence is not strong enough against at least one of the null or the alternative hypotheses.

Some alternative Bayesian measures of evidence can be seen in Diniz, Pereira, Polpo, Stern, and Wechsler (2012). The authors studied some relationships between Bayesian and frequentist significance indices. It is beyond the scope of this article to compare the classical and Bayesian approaches.

Steps to Compute the s-Value

The steps to compute the s-value are the following:

Define the statistical model ( X , M ) . Remember that X represents the observable sample and contains n random variables, namely X = ( X 1 , … , X n )

Define the null hypothesis H 0 and its related set Θ 0

If required, define the alternative hypothesis H 1 and its related set Θ 1

Compute the likelihood-ratio statistic λ ( θ ; x )

Compute c α ( θ )

Compute Λ α ( x )

Compute s ( Θ 0 , x )

If required, compute s ( Θ 1 , x )

Step 5 is somewhat difficult to execute for some complex statistical models, since for those models the distribution of λ ( θ , X ) is not trivial and may depend on θ . In those cases, under regular conditions (Cox & Hinkley, 1974), the practitioner may apply the limiting distribution of − 2 log ( λ ( θ , X ) ) , which is a chi-squared distribution with p degrees of freedom, where dim ( Θ ) = p (the Lebesgue measure). Then, Step 5 reduces to

c α ( θ ) = exp ( − 1 2 χ p , 1 − α 2 ) , for all θ ∈ Θ ,

where χ p , 1 − α 2 is the ( 1 − α ) × 100 % quantile of a chi-squared distribution with p degrees of freedom. This approximation reduces the complexity, since c α ( θ ) does not depend on θ . Under this asymptotic approximation, the “asymptotic”s-value, denoted by s a , reduces simply to

s a ( Θ 0 , x ) = 1 − inf θ ∈ Θ 0 F χ p 2 ( − 2 log ( λ ( θ , x ) ) ) = 1 − F χ p 2 ( − 2 log ( sup θ ∈ Θ 0 λ ( θ , x ) ) ) ,

where F χ p 2 is the cumulative distribution of a chi-squared distribution with p degrees of freedom and log is the natural logarithm function. If Θ 0 = { θ 0 } , then the asymptotic p-value (i.e., the asymptotic approximation for the p-value) based on the likelihood-ratio statistic coincides with the above asymptotic s-value. Nevertheless, if dim ( Θ 0 ) > 0 , the asymptotic p-value and asymptotic s-value will probably differ from each other. In the asymptotic p-value, the degree of freedom of the chi-squared distribution varies with the dimension of Θ 0 ; more precisely, the asymptotic p-value based on the likelihood-ratio statistic is

p a ( Θ 0 ; x ) = 1 − F χ q 2 ( − 2 log ( sup θ ∈ Θ 0 λ ( θ , x ) ) ) ,

where q = dim ( Θ ) − dim ( Θ 0 ) , with dim ( Θ 0 ) < dim ( Θ ) . That is, the cumulative distribution function F χ q 2 varies with the chosen null hypothesis, whereas for the s-value F χ p 2 does not vary with the chosen null hypothesis. Patriota (2013, 2014) showed that the asymptotic s-values and p-values (based on the likelihood-ratio statistic) are connected through the following relation:

s a ( Θ 0 , x ) = 1 − F χ p 2 ( F χ q 2 − 1 ( 1 − p a ( Θ 0 , x ) ) ) .

That is, from a p-value (based on the likelihood-ratio statistic) we can compute the s-value via the above formulae. If p = q , then s ( Θ 0 , x ) = p ( Θ 0 , x ) .

Numerical Examples

In this section, the s-value is applied for univariate and bivariate normal distributions. We consider known variances (and covariances) to maintain the simplicity. All required steps are computed.

Example 1. (Normal distribution, variance known: z test) Let X = ( X 1 , … , X n ) be a sample from a normal distribution with population mean θ and variance 1 . Let H 0 : θ = θ 0 be the null hypothesis of interest. The statistical model is ( X , M ) , where M = { g θ : θ ∈ ℝ } and

g θ ( x ) = 1 ( 2 π ) n 2 exp ( − 1 2 ∑ i = 1 n ( x i − θ ) 2 ) .

The likelihood-ratio statistic is

λ ( θ , x ) = exp ( − 1 2 ∑ i = 1 n ( x i − θ ) 2 + 1 2 ∑ i = 1 n ( x i − x ¯ ) 2 ) = exp ( − n 2 ( x ¯ − θ ) 2 ) ,

where x ¯ = 1 n ∑ i = 1 n x i is the maximum likelihood estimate for θ . It is known that

− 2 log ( λ ( θ , X ) ) ~ P θ χ 1 2 ,

where the symbol “ ~ P θ χ p 2 ” means “follows a chi-squared distribution with p degrees of freedom, under the law P θ .” Then,

c α ( θ ) = exp ( − 1 2 χ 1 , 1 − α 2 )

and

Λ α ( x ) = { θ ∈ ℝ : n ( x ¯ − θ ) 2 ≤ χ 1 , 1 − α 2 } = [ x ¯ − 1 n χ 1 , 1 − α 2 , x ¯ + 1 n χ 1 , 1 − α 2 ] .

The quantity χ 1 , 1 − α 2 coincides with the normal ( 1 − α / 2 ) -quantile z 1 − α / 2 , for instance, for α = 0 . 05 , we have χ 1 , 0 . 95 2 = z 0 . 97 ≈ 1 . 96 . That is, in this example, Λ α is the usual ( 1 − α ) -confidence interval for the population mean.

Let H 0 : θ = θ 0 be the null hypothesis of interest. The s-value is computed by finding the α -value such that the border of the observed confidence interval Λ α ( x ) is θ 0 . The solution is

s ( { θ 0 } , x ) = 1 − F χ 1 2 ( n ( x ¯ − θ 0 ) 2 ) .

As aforementioned, for this simple null hypothesis, the s-value is precisely the p-value based on the likelihood-ratio statistic and coincides with the famous z-test. Table 1 depicts numerical s-values to illustrate the univariate normal distribution example for n = 10 and σ 2 = 1 . The null hypothesis is H 0 : θ = θ 0 , where θ 0 = − 1 , 0 , 1 .

OPEN IN VIEWER

Table 1.

S-Values for Testing H 0 : θ = θ 0 , Where θ 0 = 0 , 1 for Some Observed Values of x ¯ When n = 10 .

x ¯	θ 0 = 0	θ 0 = 1
0.0	1.0000	0.0016
0.1	0.7518	0.0044
0.2	0.5271	0.0114
0.3	0.3428	0.0269
0.4	0.2059	0.0578
0.5	0.1138	0.1138
0.6	0.0578	0.2059
0.7	0.0269	0.3428
0.8	0.0114	0.5271
0.9	0.0044	0.7518
1.0	0.0016	1.0000
1.1	0.0005	0.7518
1.2	0.0001	0.5271
1.3	<0.0001	0.3428
1.4	<0.0001	0.2059
1.5	<0.0001	0.1138
1.6	<0.0001	0.0578
1.7	<0.0001	0.0269
1.8	<0.0001	0.0114
1.9	<0.0001	0.0044
2.0	<0.0001	0.0016

Example 2.(Bivariate normal distribution, with known variances and covariances) Let X = ( X 1 , … , X n ) be a sample from a bivariate normal distribution with population mean θ = ( μ 1 , μ 2 ) T and covariance-variance matrix ( 1 0 0 1 ) . The statistical model is ( X , M ) , where M = { g θ : θ ∈ ℝ 2 } and

g θ ( x ) = 1 ( 2 π ) n exp ( − 1 2 ∑ i = 1 n ( x i − θ ) T ( x i − θ ) ) .

The likelihood-ratio statistic is

λ ( θ , x ) = exp ( − 1 2 ( x ¯ − θ ) T ( x ¯ − θ ) ) ,

where x ¯ = ( x ¯ 1 , x ¯ 2 ) T is the maximum estimate for θ , where x ¯ 1 and x ¯ 2 are the sample averages of the bivariate sample. Observe that, here p = 2 . It is also known that

− 2 log ( λ ( θ , X ) ) ~ P θ χ 2 2 .

Then,

c α ( θ ) = exp ( − 1 2 χ 2 , 1 − α 2 )

and

Λ α ( x ) = { θ ∈ ℝ : n ( x ¯ − θ ) T ( x ¯ − θ ) ≤ χ 2 , 1 − α 2 } .

Null Hypothesis 1: Let H 0 ( 1 ) : θ = θ 0 be the null hypothesis of interest, where θ 0 = ( μ 10 , μ 20 ) T is a given vector; then Θ 01 = { θ 0 } . The s-value is computed by finding the α -value such that the border of the observed confidence interval Λ α ( x ) is θ 0 . The solution is (which is also equal to the p-value based on the likelihood-ratio statistic)

s ( { θ 0 } , x ) = 1 − F χ 2 2 ( n ( x ¯ − θ 0 ) T ( x ¯ − θ 0 ) ) .

Null Hypothesis 2: Let H 0 ( 2 ) : μ 1 = μ 2 be the null hypothesis of interest, then Θ 02 = { θ ∈ ℝ 2 : μ 1 = μ 2 } . The s-value is computed by finding the maximum α -value such that

Λ α ( x ) ∩ Θ 02 = { θ ∈ Θ 02 : n ( x ¯ − θ ) T ( x ¯ − θ ) ≤ χ 2 , 1 − α 2 }

has at least one element. The solution is (which is not equal to the p-value based on the likelihood-ratio statistic)

s ( Θ 02 , x ) = 1 − F χ 2 2 ( n min θ ∈ Θ 02 ( x ¯ − θ ) T ( x ¯ − θ ) ) .

Notice that

min θ ∈ Θ 02 ( x ¯ − θ ) T ( x ¯ − θ ) = min μ ∈ ℝ [ ( x ¯ 1 − μ ) 2 + ( x ¯ 2 − μ ) 2 ] = n 2 ( x ¯ 1 − x ¯ 2 ) 2 .

Then,

s ( Θ 02 , x ) = 1 − F χ 2 2 ( n 2 ( x ¯ 1 − x ¯ 2 ) 2 ) .

Recall that the p-value based on the likelihood-ratio statistic is

p ( Θ 02 ; x ) = 1 − F χ 1 2 ( n 2 ( x ¯ 1 − x ¯ 2 ) 2 )

Table 2 presents numerical s-values to illustrate the bivariate normal distribution example for n = 10 and covariance-variance matrix ( 1 0 0 1 ) . The null hypotheses considered are H 01 : μ 1 = μ 2 = 0 and H 02 : μ 1 = μ 2 for which it is expected to find more evidence against H 01 than H 02 . The s-values were defined to hold this expected behavior. We purposely choose values for x ¯ 1 and x ¯ 2 such that p-values are problematic. The figures of Table 2 show that all p-values fail to hold the logical condition for all sample, except for x ¯ 1 = x ¯ 2 = 0 .

OPEN IN VIEWER

Table 2.

S-Values and p-Values for testing H 01 : μ 1 = μ 2 = 0 (the s-Values and p-Values Are Identical) and H 02 : μ 1 = μ 2 (the s-Values and p-Values Differ) for Some Observed Values of ( x ¯ 1 , x ¯ 2 ) That Generate Problematic p-Values (Showing That p-Values Do Not Respect the Logical Consequence). ^a

		H 01 : μ 1 = μ 2 = 0		H 02 : μ 1 = μ 2
( x ¯ 1 , x ¯ 2 )	x ¯ 1 − x ¯ 2	s-value	p-value	s-value	p-value
(0.00, 0.00)	0.0	1.0000	1.0000	1.0000	1.0000
(0.05, –0.05)	0.1	0.9753	0.9753	0.9753	0.8231
(0.09, –0.11)	0.2	0.9039	0.9039	0.9048	0.6547
(0.14, –0.16)	0.3	0.7977	0.7977	0.7985	0.5023
(0.19, –0.21)	0.4	0.6697	0.6697	0.6703	0.3711
(0.23, –0.27)	0.5	0.5331	0.5331	0.5353	0.2636
(0.28, –0.32)	0.6	0.4049	0.4049	0.4066	0.1797
(0.33, –0.37)	0.7	0.2926	0.2926	0.2938	0.1175
(0.37, –0.43)	0.8	0.2001	0.2001	0.2019	0.0736
(0.42, –0.48)	0.9	0.1308	0.1308	0.1320	0.0442
(0.47, –0.53)	1.0	0.0813	0.0813	0.0821	0.0253
(0.51, –0.59)	1.1	0.0478	0.0478	0.0486	0.0139
(0.56, –0.64)	1.2	0.0269	0.0269	0.0273	0.0073
(0.61, –0.69)	1.3	0.0144	0.0144	0.0146	0.0037
(0.65, –0.75)	1.4	0.0073	0.0073	0.0074	0.0017
(0.70, –0.80)	1.5	0.0035	0.0035	0.0036	0.0008
(0.75, –0.85)	1.6	0.0016	0.0016	0.0017	0.0003
(0.79, –0.91)	1.7	0.0007	0.0007	0.0007	0.0001
(0.84, –0.96)	1.8	0.0003	0.0003	0.0003	0.0001
(0.89, –1.01)	1.9	0.0001	0.0001	0.0001	<0.0001
(0.93, –1.07)	2.0	<0.0001	<0.0001	<0.0001	<0.0001

a

The sample size is n = 10 .

The behavior of p-values depicted in Tables 1 and 2 is not restricted to the examples where dispersion parameters are known. This feature happens also for unknown dispersion parameters, other test statistics, and other statistical models. Here, we consider likelihood-ratio statistics, since we are interested in comparing the p-value with the s-value. The distribution of − 2 log ( λ ( θ ; X ) ) is not trivial when the dispersion parameters are unknown and in order to avoid cumbersome computations, we consider only the case with known dispersion parameters.

Conclusion

This article discusses some conceptual and technical problems related to the null hypothesis statistical testing. The scientific and statistical hypotheses and the theoretical assumptions are connected by rules of inferences called modus ponnes and modus tollens, as studied in the section “Scientific and Statistical Hypotheses.” Unless the practitioner is totally certain of the theoretical assumptions, evidence to reject the null statistical hypothesis does not mean evidence to reject the scientific hypothesis, since the assumptions of a statistical model interfere in this process. Types of decisions in null hypothesis statistical testing depend on important assumptions that are not always made explicit. On one hand, if the practitioner considers the null and alternative statistical hypotheses are mutually exclusive and exhaustive, then procedures to accept–reject the null statistical hypothesis are justifiable (e.g., Neyman–Pearsonian and Bayesian procedures). On the other hand, if the practitioner considers that the null and alternative statistical hypotheses are mutually exclusive but not exhaustive, then procedures to reject the null statistical hypothesis are preferable (e.g., Fisherian procedures or some other procedures that do not use a belief measure that excludes all possibilities outside the null or alternative hypotheses), since a third option “ not - H 0 AND not - H 1 ” must be taken into account. A statistical procedure developed under one assumption will certainly fail to be appropriated under the other, therefore an extra caution must be taken when comparing different statistical procedures (classical vs Bayesian). By construction, p-values do not respect the following logical reasoning: if H 01 ⇒ H 02 , then p-value( H 02 ) ≰ p-value( H 01 ). That is, the practitioner must not use the p-value to extrapolate the inference made for H 02 to H 01 . This is not a defect in the classic statistical reasoning, because s-values do respect this logic and can be employed in the place of p-values. Asymptotic versions of s-values are simpler to compute than p-values. S-values can be used as a complementary measure of evidence and, as any other statistical measure, some care is needed when using it to make inferences; rules of thumb must be avoided, the inferential conclusions must be always complemented with other statistical tools.

My personal view is that models are useful tools, they can be adequate or inadequate in specific contexts. As for null hypotheses, they can be compatible or incompatible with the observed data; their degree of (in)compatibility with the observed data can be verified through measures of evidence (p-values, s-values, etc.). Statistical analyses have hard philosophical issues that should not be taken for granted, namely: translation problems, meaning of uncertainty, domain of applicability of each method, underlying (philosophical, scientific, logical, and statistical) principles and so on. My impression is that science would be more trustful if these issues were taken seriously into account in the statistical analyses. For instance, a p-value (or any other quantitative measure of evidence) smaller than a certain threshold (e.g., 0.05) should not be used directly to reject a scientific hypothesis without further investigations regarding model assumptions, test statistics, sample size, scientific relevance, rules of inferences, adopted principles, and so on.