Violation of either of two basic assumptions in classical test theory may lead to biased estimates of reliability. Violation of the assumption of essential tau-equivalence may produce underestimates, and the presence of correlated errors among measurement units may result in overestimates. The ubiquity of circumstances in which this problem may occur is not fully comprehended by many workers. This article surveys a variety of settings in which biased reliability estimates may be found in an effort to increase awareness of the prevalence of the problem.
Get full access to this article
View all access options for this article.
References
1.
AllenM. J.YenW. M. (1979) Introduction to measurement theory.Monterey, CA: Brooks/Cole.
2.
BeckerG. (1991) Alternative methods of reporting research results. American Psychologist, 46, 654–655.
3.
BeckerG. (1996a) Bias in the assessment of gender differences. American Psychologist, 51, 154–155.
4.
BeckerG. (1996b) The meta-analysis of factor analyses: an illustration based on the cumulation of correlation matrices. Psychological Methods, 1, 341–353.
5.
BeckerG. (2000) How important is transient error in estimating reliability? Going beyond simulation studies. Psychological Methods, 5, 370–379.
6.
BeckerG. (in press) Decremental and inflationary effects in reliability estimation resulting from violations of assumptions. Psychological Reports.
7.
ClearyT. A.LinnR. L. (1969) Error of measurement and the power of a statistical test. British Journal of Mathematical and Statistical Psychology, 22, 49–55.
8.
CohenJ. (1994) The earth is round (p < .05). American Psychologist, 49, 997–1003.
9.
CohenJ.CohenP. (1983) Applied multiple regression/correlation analysis for the behavioral sciences. (2nd ed.) Hillsdale, NJ: Erlbaum.
10.
CortinaJ. M. (1993) What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98–104.
11.
CronbachL. J. (1951) Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.
12.
CronbachL. J.AzumaH. (1962) Internal consistency reliability formulas applied to randomly-sampled single-factor tests: an empirical comparison. Educational and Psychological Measurement, 22, 645–665.
13.
CuretonE. E. (1965) Reliability and validity: basic assumptions and experimental designs. Educational and Psychological Measurement, 25, 327–346.
14.
DarlingtonR. B. (1990) Regression and linear models.New York: McGraw-Hill.
15.
FeldtL. S.QuallsA. L. (1996) Bias in coefficient alpha arising from heterogeneity of test content. Applied Measurement in Education, 9, 277–286.
16.
FergusonG. A. (1981) Statistical analysis in psychology and education. (5th ed.) New York: McGraw-Hill
17.
GaitoJ. (1973) Introduction to analysis of variance procedures.New York: MSS Information Corp.
HunterJ. E. (1997) Needed: a ban on the significance test. Psychological Science, 8, 3–7.
21.
HunterJ. E.SchmidtF. L. (1990) Methods of meta-analysis: correcting error and bias in research findings.Newbury Park, CA: Sage.
22.
KirkR. E. (1982) Experimental design: procedures for the behavioral sciences.Belmont, CA: Brooks/Cole.
23.
KomaroffE. (1997) Effect of simultaneous violations of essential τ-equivalence and uncorrected error of coefficient α. Applied Psychological Measurement, 21, 337–348.
24.
LevinJ. R.SubkoviakM. J. (1977) Planning an experiment in the company of measurement error. Applied Psychological Measurement, 1, 331–338.
25.
LordF. M.NovickM. R. (1968) Statistical theories of mental test scores.New York: Addison-Wesley.
26.
LubinskiD.HumphreysL. G. (1990) Assessing spurious “moderator effects”: illustrated substantively with the hypothesized (“synergistic”) relation between spatial and mathematical ability. Psychological Bulletin, 107, 385–393.
27.
MacCallumR. C.MarC. M. (1995) Distinguishing between moderator and quadratic effects in multiple regression. Psychological Bulletin, 118, 405–421.
28.
McDonaldR. P. (1981) The dimensionality of tests and items. British Journal of Mathematical and Statistical Psychology, 34, 100–117.
29.
McNemarQ. (1969) Psychological statistics. (4th ed.) New York: Wiley.
30.
MillerM. B. (1995) Coefficient alpha: a basic introduction from the perspective of classical test theory and structural equation modeling. Structural Equation Modeling, 2, 255–273.
31.
NunnallyJ. C. (1978) Psychometric theory. (2nd ed.) New York: McGraw-Hill.
RaykovT. (1998) Coefficient alpha and composite reliability with interrelated nonhomogeneous items. Applied Psychological Measurement, 22, 375–385.
34.
ReuterbergS.GustafssonJ. (1992) Confirmatory factor analysis and reliability: testing measurement model assumptions. Educational and Psychological Measurement, 52, 795–811.
35.
RoffM. (1937) The relation between results obtainable with raw and corrected correlation coefficients in multiple factor analysis. Psychometrika, 2, 35–39.
36.
RoganJ. C.KeselmanH. J. (1977) Is the ANOVA F-test robust to variance heterogeneity when sample sizes are equal? An investigation via a coefficient of variation. American Educational Research Journal, 14, 493–498.
37.
RozeboomW. W. (1966) Foundations of the theory of prediction.Homewood, IL: Dorsey.
38.
SchmidtF. L. (1996) Statistical significance testing and cumulative knowledge in psychology: implications for training of researchers. Psychological Methods, 1, 115–129.
39.
SchmidtF. L.HunterJ. E. (1996) Measurement error in psychological research: lessons from 26 research scenarios. Psychological Methods, 1, 199–223.
40.
SchmittN. (1996) Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350–353.
41.
SpearmanC. (1927) The abilities of man: their nature and measurement.London: Macmillan.
42.
StraussD. (1981) Testing partial correlations when the third variable is measured with error. Educational and Psychological Measurement, 41, 349–358.
43.
ThorndikeR. L. (1951) Reliability. In LindquistE. F. (Ed.), Educational measurement.Washington, DC: American Council on Education. Pp. 560–620.
44.
Wilkinson, L., & The Task Force on Statistical Inference. (1999) Statistical methods in psychology journals. American Psychologist, 54, 594–604.
45.
ZimmermanD. W.ZumboB. D.LalondeC. (1993) Coefficient alpha as an estimate of test reliability under violation of two assumptions. Educational and Psychological Measurement, 53, 33–49.
