SCORE Study Report 8: Closed Tests for All Pairwise Comparisons of Means

We compare five closed tests for strong control of family-wide type 1 error while making all pairwise comparisons of means in clinical trials with multiple arms such as the SCORE Study. We simulated outcomes of the SCORE Study under its design hypotheses, and used P values from chi-squared tests to compare performance of a pairwise closed test described below to Bonferroni and Hochberg adjusted P values. Pairwise closed testing was more powerful than Hochberg's method by several definitions of multiple-test power. Simulations over a wider parameter space, and considering other closed methods, confirmed this superiority for P values based on normal, logistic, and Poisson distributions. The power benefit of pairwise closed testing begins to disappear with five or more arms and with unbalanced designs. For trials with four or fewer arms and balanced designs, investigators should consider using pairwise closed testing in preference to Shaffer's, Hommel's, and Hochberg's approaches when making all pairwise comparisons of means. If not all P values from the closed family are available, Shaffer's method is a good choice.