The Effects of Nonnormal Distributions on Confidence Intervals Around the Standardized Mean Difference: Bootstrap and Parametric Confidence Intervals

Abstract

The standardized group mean difference, Cohen’s d, is among the most commonly used and intuitively appealing effect sizes for group comparisons. However, reporting this point estimate alone does not reflect the extent to which sampling error may have led to an obtained value. A confidence interval expresses the uncertainty that exists between d and the population value, δ, it represents. A set of Monte Carlo simulations was conducted to examine the integrity of a noncentral approach analogous to that given by Steiger and Fouladi, as well as two bootstrap approaches in situations in which the normality assumption is violated. Because d is positively biased, a procedure given by Hedges and Olkin is outlined, such that an unbiased estimate of δ can be obtained. The bias-corrected and accelerated bootstrap confidence interval using the unbiased estimate of δ is proposed and recommended for general use, especially in cases in which the assumption of normality may be violated.

Keywords

effect size standardized effect size confidence intervals bootstrap methods nonnormal data

Get full access to this article

View all access options for this article.

References

Boneau, A. C. (1960). The effects of violations of the assumptions underlying the t test. Psychological Bulletin, 57, 49-64.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Cumming, G. , & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 532-574.

Efron, B. , & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York: Chapman & Hall.

Fleishman, A. I. (1978). A method for simulating non-normal distributions. Psychometrika, 43, 521-532.

Glass, G. V. (1976) Primary, secondary, and meta-analysis. Educational Researcher, 5, 3-8.

Hedges, L. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107-128.

Hedges, L. , & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.

LePage, R. , & Billard, L. (Eds.). (1992). Exploring the limits of the bootstrap. New York: John Wiley.

10.

Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156-166.

11.

Olejnik, S. , & Algina, J. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241-286.

12.

Roberts, J. K. , & Henson, R. K. (2002). Correction for bias in estimating effect sizes. Educational and Psychological Measurement, 62, 241-253.

13.

Sawilowsky, S. S. , & Blair, R. (1992). A more realistic look at the robustness and type II error properties of the t test to departures from population normality. Psychological Bulletin, 111, 352-360.

14.

Siegel, S. (1956). Nonparametric statistics for the behavioral sciences. New York: McGraw-Hill.

15.

Smithson, M. (2001). Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Educational and Psychological Measurement, 61, 605-632.

16.

Steiger, J. H. , & Fouladi, R. T. (1997). Noncentrality interval estimation and the evaluation of statistical methods. In L. L. Harlow , S. A. Mulaik , & J. H. Steiger (Eds.), What if there where no significance tests? (pp. 221-257). Mahwah, NJ: Lawrence Erlbaum.

17.

Stuart, A. , & Ord, J. K. (1994). Kendall’s advanced theory of statistics: Distribution theory (Vol. 1, 6th ed.). New York: John Wiley.

18.

Thompson, B. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31, 25-32.

19.

Wilcox, R. R. (1998). How many discoveries have been lost by ignoring modern statistical methods? American Psychologist, 53, 300-314.

20.

Wilkinson, L. , & Task Force on Statistical Inference . (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.