Power Estimates: The Effect of Dependent Variable Reliability on the Power of One-Factor Anovas

Abstract

The literature indicates that the power of a statistical test is influenced by the reliability of the dependent variable; however the authors disagree about the direction and strength of this effect. The purpose of this study was to quantify and to tabulate the degree to which reliability affects the power of the one-factor Analysis of Variance (ANOVA) by taking into account sample size, level of significance, and true score effect size. True score effect size is defined as the difference between means in true score standard deviation units without variation due to the inconsistency of an instrument or lack of its reliability. Results confirm that the impact on power is substantial, particularly when sample sizes are less than 100 per group and/or when small or moderate effect size differences are expected. It is suggested that the influence of reliability is of considerable interest to those who include subtest analyses in their work as well as those who study low incidence populations. The tables provide easy, accurate, one-step estimates of power for various combinations of reliability, sample size, level of significance, and true score effect size and as such should be quite useful to researchers from various human science fields.

Get full access to this article

View all access options for this article.

References

Cleary, T. A. and Linn, R. L. (1969). Error of measurement and the power of a statistical test. British Journal of Mathematical and Statistical Psychology, 22, 49-55.

Crocker, L. and Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart and Winston.

Hopkins, K. D. and Hopkins, B. R. (1979). The effect of the reliability of the dependent variable on power. The Journal of Special Education, 13, 463-466.

Nicewander, W. A. and Price, J. M. (1978). Dependent variable reliability and the power of statistical tests. Psychological Bulletin, 85, 405-409.

Nicewander, W. A. and Price, J. M. (1983). Reliability of measurement and the power of statistical test. Psychological Bulletin, 94, 524-533.

Rogers, W. T. and Hopkins, K. D. (1988a). Power estimates in the presence of a covariate and measurement error. Educational and Psychological Measurement, 48, 647-656.

Rogers, W. T. and Hopkins, K. D. (1988b). Quick power estimates incorporating the joint effects of measurement error and covariate. Journal of Experimental Education, 57, 86-94.

Sutcliffe, J. P. (1980). On the relationship of reliability to statistical power. Psychological Bulletin, 88, 509-515.

Winer, B. J. (1971). Statistical principles in experimental design (2nd ed.). New York: McGraw Hill.

10.

Zimmerman, D. W. and Williams, R. H. (1986). Note on the reliability of experimental measures and the power of significance tests. Psychological Bulletin, 100, 123-124.