Type I Error Rate and Power of Some Alternative Methods to the Independent Samples t Test

Abstract

Several tests exist for use in treatment-control studies in which a larger treatment mean may be accompanied by a larger treatment variance. Type I error rates and power were estimated for four of these tests and for the independent samples t test and the Welch test. Five factors were manipulated: heteroscedasticity, total sample size, sample size imbalance, distribution, and power. The test due to Brownie, Boos, and Hughes-Oliver may not control the Type I error rate when the data are skewed. Among the other tests, one due to Conover and Salsburg and one due to Johnson, Verrill, and Moore had the best power advantage when compared to the independent sample t test and the Welch test. In the Johnson et al. test, a parameter that strongly affects power is specified, and a poor choice can reduce power below that for the Conover and Salsburg test.

Get full access to this article

View all access options for this article.

References

Alexander, R. A. , & Govern, D. M. (1994). A new and simpler approximation for ANOVA under variance heterogeneity. Journal of Educational Statistics, 19, 91-102.

Blair, R. C. , & Sawilowsky, S. (1993). A note on the operating characteristics of the modified F test. Biometrics, 49, 935-939.

Bradley, J. V. (1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31, 144-152.

Brownie, C. , Boos, D. D. , & Hughes-Oliver, J. (1990). Modifying the t and ANOVA F tests when treatment is expected to increase variability relative to controls. Biometrics, 46, 259-266.

Conover, W. J. , & Salsburg, D. S. (1988). Locally most powerful tests for detecting treatment effects when only a subset of patients can be expected to “respond” to treatment. Biometrics, 44, 198-196.

Johnson, R. A. , Verrill, S. , & Moore, D. H. II (1987). Two-sample rank tests for detecting changes that occur in a small proportion of treated population. Biometrics, 43, 641-655.

Lehmann, E. L. (1953). The power of rank tests. Annals of Mathematical Statistics, 24, 23-43.

Lehmann, E. L. (1959). Testing statistical hypotheses. New York: John Wiley.

Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156-166.

10.

Subrahmaniam, K. , Subrahmaniam, K. , & Messeri, J. Y. (1975). On the robustness of some tests of significance in sampling from compound normal population. Journal of the American Statistical Association, 70, 435-438.

11.

Welch, B. L. (1938). The significance of the difference between two means when the population variances are unequal. Biometrika, 29, 350-362.