Non-adjustment for multiple testing in multi-arm trials of distinct treatments: Rationale and justification

The concept of a global null hypothesis

If we are using a multiple testing correction that (as a minimum) aims to reduce or control the FWER, then this implies that we wish to control the false-rejection rate of a global null hypothesis. The global null hypothesis consists of the intersection hypothesis of all null hypotheses that we are interested in (called a ‘family’ of null hypotheses). For i= 1,…, m null hypotheses, we can write the following

H 0 : H 0 , 1 ∩ H 0 , 2 ∩ … ∩ H 0 , m

In the case of multi-arm trials, each of the null hypotheses H_0,1, H_0,2,…, H_0,m could indicate that there is no difference between treatment and control, for example. The alternative hypothesis would then be the global union hypothesis

H A : H A , 1 ∪ H A , 2 ∪ … ∪ H A , m

This is called ‘union-intersection testing’ ¹⁰ and implies that if any of the individual null hypotheses are rejected at an unadjusted alpha significance level, then so would the global null hypothesis.

The global intersection null hypothesis is more likely to be rejected as the number of individual null hypotheses increases with multiple testing, and so multiple testing adjustment is designed to control this inflated probability of rejection. If treatments are distinct and we are interested in individual treatment versus control comparisons, however, then it is difficult to see how the concept of formulating a global intersection null hypothesis could be relevant. If the global intersection null hypothesis is not relevant, then neither is the FWER.

Nevertheless, a valid counter-argument is that although we might not be explicitly interested in the global intersection null hypothesis, the final analysis results may be reported and interpreted in ways that implicitly correspond to an underlying global null hypothesis. This is easily seen if we consider the example of a four-arm trial, with three of the arms consisting of the same treatment given at different doses, and the fourth arm placebo. In this case, if only one of the treatment doses shows a significant effect relative to placebo, then we would conclude that the treatment is effective. It is therefore recommended to control the FWER in this situation so that the probability of incorrectly concluding that the treatment is effective is below a pre-specified level. In this case, the overall treatment efficacy conclusion is the same regardless of which of the individual doses showed treatment efficacy, and therefore, we are implicitly assuming that the rejection of a global intersection hypothesis is clinical meaningful.

In multi-arm trials of distinct treatments, however, we are usually interested in each of the treatment comparisons individually (e.g. whether drug A is effective compared to placebo) and their associated false-positive error rate, often referred to as the comparison-wise error rate. The comparison-wise error rate is not inflated by multiple testing. ¹⁰ Indeed, the expected proportion of incorrectly rejected hypotheses will not exceed the significance level used in the individual tests. ¹⁰

The single claim of effectiveness

Some authors have emphasised the importance of making a ‘single claim of effectiveness’ or defining a ‘clinical win criteria’ as determining the need for multiple testing adjustment.^6,7,9,10 Howard et al. ⁶ for example, have suggested that an FWER adjustment is only necessary if ‘assessing multiple hypotheses within a multi-arm trial has increased the chance of making a single claim of effectiveness’. This decision criteria requires careful definition and consideration because a statement such as ‘treatment A is effective’ (assuming no other treatments are effective) could be regarded as a ‘single claim of effectiveness’, even though it has resulted from individual interpretation of comparison-wise hypothesis tests. As above, comparison-wise testing does not require a multiplicity adjustment. Clearly, the term ‘single claim of effectiveness’ must refer to making a conclusion such as ‘the treatment is effective’ as in the dose–response example, which will occur whenever any of the multiple hypotheses is rejected. If we name specific treatments (e.g. state that ‘treatment A is effective’) in our clinical efficacy conclusion, which we invariably would do when interpreting results from multi-arm trials of distinct treatments, we have already severed the connection between the global intersection hypothesis at the study level and our clinical efficacy conclusion. Therefore, researchers should be clear that it is the content of the clinical efficacy statement that matters and how it is determined, not how many statements of effectiveness are stated or how many conclusions given in a final manuscript. Accordingly, the definition of ‘single claim of effectiveness’ and its specific rationale should be included in any overall justification of a multiple testing adjustment.

Issues with multiplicity adjustment

The idea of controlling the FWER to minimise the risk of a study recommending an ineffective treatment for use in clinical practice sounds appealing from the statistical perspective. Nevertheless, it does not accord with the fact that within medicine or public health, multi-arm trials are usually focussed on the effects of individual treatments. Knowledge of the effectiveness of at least one of the treatments is not by itself clinically useful information – the clinician needs to know which treatments are effective so that they can make an impact on clinical practice or lead to further investigation. If one of the treatments was successful, then we need to know its specific name and pharmacological properties. How many other treatments were tested alongside it in the same trial is irrelevant information if those treatments were distinct. It therefore seems paradoxical to control overall errors resulting from entire projects or studies when our main focus in multi-arm trials is usually in the individual treatment comparisons themselves. After all, as others have commented, if the treatments were evaluated in separate trials, there would be no expectation that the researcher would need to make a multiplicity adjustment.^6,11

More generally, it is not always clear which hypotheses should be included in the ‘family’ when calculating the FWER in multiplicity adjustment. The word ‘family’ in FWER does not have a universally consistent definition and its definition needs to be carefully considered for each multiple testing application.^6,12 In our experience ‘family’ tends to be defined at the level of the study protocol or the collection of primary hypotheses that are assessed in a final analysis. However, it is difficult to rationalise why it should be defined at the study level (e.g. instead of across all trials which include the same treatment versus control comparisons), other than relying on convention and pragmatism. Howard et al. recommend that ‘family’ is taken to encompass all hypotheses which contribute to a ‘single statement of effectiveness’; ⁶ but as we have seen, the very definition of ‘single statement of effectiveness’ is itself uncertain and open to differing interpretations.

Furthermore, setting the type I error rate of the overall study to be below a certain level obscures the importance of setting individually appropriate error rate thresholds according to context and expected costs, which is strongly recommended in the literature.^12–14 If we are prioritising errors at the study level over errors in individual treatment comparisons then we are saying in effect that the impact of the study is more important than the impact of the clinical treatments, which is contrary to common sense.

In addition, treatment recommendations arising from a study are often based on both clinical and statistical significance ¹⁵ and may be based on secondary outcomes and/or safety outcomes as well as primary outcomes. Therefore, the interpretative environment used in clinical trials is often much more complex than that in which a multiple testing adjustment implies. Statistical evidence from a set of primary null hypotheses may not be sufficient in itself to lead to clinical impact or substantial cost from a type I error. ¹⁵ There is a need to consider the totality of the evidence in favour of a treatment. ¹⁶

Another argument against multiplicity adjustment is that it usually over-complicates interpretation of the study results. ¹⁷ Unadjusted and adjusted results may lead to different conclusions, and adjusted results are not easy to interpret individually because the degree of adjustment will depend on the underlying global hypothesis and how many other treatment comparisons have been made.

There is also the danger that by focussing too much on controlling type I error, we overlook the type II error rate (failing to reject the null hypothesis even though it is false). The consequence of reducing the type I error rate is that the type II error rate is increased.^2,13,18,19 This is why performing multiplicity adjustment is unsupported in exploratory or early phase studies where type II error is important.

The confirmatory trials argument

It has been suggested that for confirmatory multi-arm trials, the FWER should be controlled but not for exploratory trials.^1,11 There is some logic in making the distinction between exploratory and confirmatory. For confirmatory trials, or trials in which we wish to make definitive conclusions to settle controversy, the cost of a false-positive finding is much greater because the treatment is likely to gain wide acceptance or proceed to be used in clinical practice. However, just because a multi-arm trial is labelled as ‘confirmatory’ does not mean that any of the issues associated with multiple testing adjustment have been resolved. In particular, adjusting for multiplicity is still inconsistent with how multi-arm trials of distinct treatments are usually interpreted.

Some might argue that it still makes sense to minimise the danger in interpreting a result as definitive and ‘confirmatory’ if only one treatment arm is significant among many tested. But we argue that this problem should not be addressed by multiple testing – it should be addressed instead by replication and multiple experiments to confirm the results, or by reducing the individual α significance levels themselves. Indeed, regulatory agencies usually require ‘statistically compelling and clinically relevant’ results for the licencing of medicines, which typically involves replication of the results in at least one other trial. ²⁰ If there was only one pivotal trial conducted, then the results have to be particularly compelling, with a strong pharmacological rationale and strong statistical significance. ²⁰ Anecdotally this usually involves setting a statistical significance level of 0.25%, equating to two independent trials being significant at 5%. In any case, if a type I error rate is of great concern for a given treatment, then this should be addressed by the individual α-levels themselves – not indirectly via controlling the overall FWER.

Consideration of some further objections

Some researchers might argue that it is better to guard against certain types of risky interpretation in an overall study rather than relying on the fact that individual treatments will be interpreted in isolation. However, it is not easy to predict how results will be interpreted or influence subsequent behaviour, ¹⁵ and inappropriate adjustment has ethical consequences because it leads to a larger sample size, lower statistical power or a combination of these. ⁶ The Bonferroni adjustment in particular imposes an extreme penalty in this regard.

Another potential objection is that if the multi-arm trial has a common control group, then we have to adjust for multiple testing anyway because the treatment comparisons are related in this way. However, Howard et al. ⁶ have demonstrated that this concept is false. The FWER is not increased in multi-arm trials with a common control group compared to what it would be if the treatments were tested in independent trials. Therefore, multiplicity adjustment is not required purely on the basis of sharing control data. ⁶

Some might argue that the more treatment arms in a multi-arm trial, the more likely we are to require adjustment. ¹¹ Again, as for confirmatory studies, this does not solve the real issues covered above. Indeed, the counter-arguments to adjustment may strengthen when the number of arms assessed increases, because the multiplicity adjustment becomes more stringent and the type II error is further inflated. This could be compensated by increasing the sample size, but this is not without cost since it may increase the overall participant burden, study duration, or financial cost of the study.