Some Comments on the Relation Between Reliability and Statistical Power

Abstract

Several articles have discussed the curious fact that a difference score with zero reliability can nonetheless allow a powerful test of change. This statistical legerde main should not be overemphasized for three reasons. First, although the reliability of the difference score may be unrelated to power, the reliabilities of the var iables used to create the difference scores are directly related to the power of the test. Second, with what some will regard as additional legerdemain, it is possi ble to define reliability in the context of a difference score in such a way that power is a direct function of reliability. The third and most serious objection to the conclusion that the reliability of a difference score is unimportant is that the underlying statistical model used in its derivation is rarely appropriate for psycho logical data. Index terms: control of individual dif ferences, difference scores, reliability, reliability of the marginal distribution, statistical power, within- group reliabilities.

Get full access to this article

View all access options for this article.

References

Kraemer, H.C. (1985). A strategy to teach the concept and application of power to statistical tests. Journal of Educational Statistics , 10, 173-195.

Nicewander, W.A. , & Price, J.M. (1983). Reliability of measurement and the power of statistical tests: Some new results. Psychological Bulletin, 94, 524-533.

Overall, J.E. , & Woodward, J.A. (1975). Unreliability of difference scores: A paradox for the measurement of change. Psychological Bulletin, 82, 85-86.

Zimmerman, D.W. , & Williams, R.H. (1986). Note on the reliability of experimental measures and the power of statistical tests. Psychological Bulletin, 100, 123-124.