The Effects of Nonnormality on Parametric,Nonparametric,and Model Comparison Approaches to Pairwise Comparisons

Abstract

Researchers in the behavioral sciences are often interested in comparing the means of several treatment conditions on a specific dependent measure. When scores on the dependent measure are not normally distributed, researchers must make important decisions regarding the multiple comparison strategy that is implemented. Although researchers commonly rely on the potential robustness of traditional parametric test statistics (e.g., t and F), these test statistics may not be robust under all nonnormal data conditions. This article compared strategies for performing multiple comparisons with nonnormal data under various data conditions, including simultaneous violations of the assumptions of normality and variance homogeneity. The results confirmed that when variances are unequal, use of the traditional two-sample t test can result in severely biased Type I and/or Type II error rates. However, the use of Welch’s two-sample test statistic with the REGWQ procedure, with either the usual means and variances or with trimmed means and Winsorized variances, resulted in good control of Type I error rates. The Kruskal-Wallis nonparametric statistic provided good Type I error control and power when variances were equal, although Type I error rates became severely inflated when variances were unequal. Furthermore, for researchers interested in eliminating intransitive decisions or comparing potential mean configuration models, a protected model-testing procedure suggested by Dayton provided good overall results.

Get full access to this article

View all access options for this article.

References

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, AC-19, 716-723.

Boneau, C. A. (1960). The effects of violations of assumptions underlying the t test. Psychological Bulletin, 57, 49-64.

Bradley, J. V. (1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31, 144-152.

Cressie, N. A. C., & Whitford, H. J. (1986). How to use the two sample t-test. Biometrical Journal, 28, 131-148.

Cribbie, R. A., & Keselman, H. J. (in press). Pairwise multiple comparisons: A model testing approach versus stepwise procedures. British Journal of Mathematical and Statistical Psychology.

Dayton, C. M. (1998). Information criteria for the paired-comparisons problem. The American Statistician, 52, 144-151.

Dunnett, C. W. (1980). Pairwise multiple comparisons in the unequal variance case. Journal of the American Statistical Association, 75, 796-800.

Einot, I., & Gabriel, K. R. (1975). A study of the powers of several methods of multiple comparisons. Journal of the American Statistical Association, 70, 574-583.

Gibbons, J. D., & Chakraborti, S. (1991). Comparisons of the Mann-Whitney, Student’s t, and alternate t tests for means of normal distributions. Journal of Experimental Education, 59, 259-267.

10.

Hayter, A. J. (1986). The maximum familywise error rate of Fisher’s least significant difference test. Journal of the American Statistical Association, 81, 1000-1004.

11.

Hoaglin, D. C. (1985). Summarizing shape numerically: The g- and h- distributions. In D. Hoaglin, F. Mosteller, & J. Tukey (Eds.), Exploring data tables, trends, and shapes (pp. 461-513). New York: Wiley.

12.

Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika, 75, 383-386.

13.

Keselman, H. J., Huberty, C. J., et al. (1998). Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research, 68, 350-386.

14.

Keselman, H. J., Lix, L. M., & Kowalchuk, R. K. (1998). Multiple comparison procedures for trimmed means. Psychological Methods, 3, 123-141.

15.

Kromrey, J. D., & La Rocca, M. A. (1995). Power and Type I error rates of new pairwise multiple comparison procedures under heterogeneous variances. Journal of Experimental Education, 63, 343-362.

16.

Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156-166.

17.

Miller, J. (1988). A warning about median reaction time. Journal of Experimental Psychology: Human Perception and Performance, 14, 539-543.

18.

Mudholkar, A., Mudholkar, G. S., & Srivastava, D. K. (1991). A construction and appraisal of pooled trimmed-t statistics. Communications in Statistics: Theory and Methods, 20, 1345-1359.

19.

Penfield, D. A. (1994). Choosing a two-sample location test. Journal of Experimental Education, 62, 343-360.

20.

Ryan, T. A. (1960). Significance tests for multiple comparison of proportions, variances, and other statistics. Psychological Bulletin, 57, 318-328.

21.

SAS Institute. (1999a). SAS/IML user’s guide, Version 8. Cary, NC: Author.

22.

SAS Institute. (1999b). SAS/STAT user’s guide, Version 7-1. Cary, NC: Author.

23.

Sawilowsky, S. S., & Blair, R. C. (1992). A more realistic look at the robustness and Type II error probabilities of the t test to departures from population normality. Psychological Bulletin, 111, 352-360.

24.

Seaman, M. A., Levin, J. R., & Serlin, R. C. (1991). New developments in pairwise multiple comparisons: Some powerful and practicable procedures. Psychological Bulletin, 110, 577-586.

25.

Sprent, P. (1993). Applied nonparametric statistical methods (2nd ed.). London: Chapman & Hall.

26.

Toothaker, L. E. (1991). Multiple comparisons for researchers. Newbury Park, CA: Sage.

27.

Tukey, J. W. (1953). The problem of multiple comparisons. Unpublished manuscript, Princeton University, Department of Statistics.

28.

Tukey, J. W. (1960). A survey of sampling from contaminated distributions. In I. Olkin, S. G. Ghurye, W. Hoeffding, W. G. Madow, & H. B. Mann (Eds.), Contributions to probabilities and statistics. Stanford, CA: Stanford University Press.

29.

Welch, B. L. (1938). The significance of the difference between two means when population variances are unequal. Biometrika, 38, 330-336.

30.

Welch, B. L. (1951). On the comparison of several mean values: An alternative approach. Biometrika, 38, 330-336.

31.

Welsch, R. E. (1977). Stepwise multiple comparison procedures. Journal of the American Statistical Association, 72, 566-575.

32.

Westfall, P. H., Tobias, R. D., Rom, D., Wolfinger, R. D., & Hochberg, Y. (1999). Multiple comparisons and multiple tests using the SAS system. Cary, NC: SAS Institute.

33.

Wilcox, R. R. (1990). Comparing the means of two independent groups. Biometrics Journal, 32, 771-780.

34.

Wilcox, R. R. (1995). ANOVA: The practical importance of heteroscedastic methods, using trimmed means versus means, and designing simulation studies. British Journal of Mathematical and Statistical Psychology, 48, 99-114.

35.

Wilcox, R. R. (1997). Three multiple comparison procedures for trimmed means. Biometrical Journal, 37, 643-656.

36.

Yuen, K. K. (1974). The two-sample trimmed t for unequal population variances. Biometrika, 61, 165-170.

37.

Zhou, X., Gao, S., & Hui, S. L. (1997). Methods for comparing the means of two independent log-normal samples. Biometrics, 53, 1129-1135.

38.

Zimmerman, D. W. (1996). A note on homogeneity of variance of scores and ranks. Journal of Experimental Education, 64, 351-362.

39.

Zumbo, B. D., & Coulombe, D. (1997). Investigation of the robust rank-order test for non-normal populations with unequal variances: The case of reaction time. Canadian Journal of Experimental Psychology, 51, 139-150.