Scientific associations and measurement experts in psychology and education have voiced various standards and best-practice recommendations concerning reliability data over the years. Yet in the counseling psychology literature, there is virtually no single-source compilation and articulation of good practices for reporting, analyzing, and interpreting reliability to guide applied researchers intending to use scales rather than develop them. Therefore, focusing on Cronbach’s alpha internal consistency reliability estimates, this article (a) defines and provides rationales for seven broad categories of good practices for reporting, analyzing, interpreting, and using reliability data and (b) illustrates some pragmatic strategies for implementing the good practices with respect to reliability data in quantitative studies involving already-developed scales. The authors’ recommendations for good rather than best practices acknowledge that additional or alternative practices may be required when scale development is the researcher’s focus. The authors summarize their good practices in tabular form.
Get full access to this article
View all access options for this article.
References
1.
Abelson, R. P. (1997). A retrospective on the significance test ban of 1999 (If there were no significance tests, they would be invented). In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 117-141). Mahwah, NJ: Lawrence Erlbaum.
2.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
3.
Block, C. J., Roberson, L., & Neuger, D. A. (1995). White racial identity theory: A framework for understanding reactions toward interracial situations at work. Journal of Vocational Behavior, 46, 71-88.
4.
Bloom, B. S. (1942). Test reliability for what? Journal of Educational Psychology, 33, 517-526.
5.
Bolt, D. M., & Rounds, J. (2000). Advances in psychometric theory and methods. In S. D. Brown & R. W. Lent (Eds.), Handbook of counseling psychology (3rd ed., pp. 140-176). New York: John Wiley.
6.
Boyle, G. J. (1991). Does item homogeneity indicate internal consistency or item redundancy in psychometric scales? Personality and Individual Differences, 7, 305-310.
7.
Charter, R. A., & Feldt, L. S. (2002). The importance of reliability as it relates to true score confidence intervals. Measurement and Evaluation in Counseling and Development, 35, 104-112.
8.
Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 96-104.
9.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
10.
Dawis, R. V. (1987). Scale construction. Journal of Counseling Psychology, 34, 481-489.
11.
DeVellis, R. F. (1991). Scale development: Theory and applications. Newbury Park, CA: Sage.
12.
Fan, X., & Thompson, B. (2001). Confidence intervals about score reliability coefficients, please: An EPM guidelines editorial. Educational and Psychological Measurement, 61, 517-531.
13.
Feldt, L. S. (1969). A test of the hypothesis that Cronbach’s alpha or Kuder-Richardson coefficient twenty is the same for two tests. Psychometrika, 34, 363-373.
14.
Feldt, L. S. (1980). A test of the hypothesis that Cronbach’s alpha reliability coefficient is the same for two tests administered to the same sample. Psychometrika, 45, 99-105.
15.
Feldt, L. S., & Ankenmann, R. D. (1999). Determining sample size for a test of the equality of alpha coefficients when the number of part-tests is small. Psychological Methods, 4, 366-377.
16.
Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 105-146). New York: American Council on Education and Macmillan.
17.
Feldt, L. S., & Charter, R. A. (2003). Estimating the reliability of a test split into two parts of equal or unequal length. Psychological Methods, 8, 102-109.
18.
Ferketich, S. (1990). Focus on psychometrics: Internal consistency estimates of reliability. Research in Nursing & Health, 13, 437-440.
19.
Green, S. B. (2003). A coefficient alpha for test-retest data. Psychological Methods, 8, 88-101.
20.
Green, S. B., Lisitz, R. W., & Mulaik, S. A. (1977). Limitations of coefficient alpha as an index of test unidimensionality. Educational and Psychological Measurement, 37, 827-838.
21.
Helms, J. E. (1984). Toward a theoretical explanation of the effects of race on counseling: A Black and White model. The Counseling Psychologist, 12, 153-165.
22.
Helms, J. E. (1999). Another meta-analysis of the White Racial Identity Attitudes Scale’s alphas: Implications for validity. Measurement and Evaluation in Counseling and Development, 32, 122-137.
23.
Helms, J. E. (2005). Challenging some misuses of reliability as reflected in evaluations of the White Racial Identity Attitude Scale (WRIAS). In R. T. Carter (Ed.), Handbook of racial-cultural psychology and counseling: Theory and research (Vol. 1, pp. 360-390). New York: John Wiley.
24.
Helms, J. E., & Carter, R. T. (1990). Development of the White Racial Identity Attitudes Scale. In J. E. Helms (Ed.), Black and White racial identity: Theory, research, and practice (pp. 67-80). Westport, CT: Greenwood.
25.
Henson, R. K. (2006). Effect-size measures and meta-analytic thinking in counseling psychology research. The Counseling Psychologist, 34, 601-629.
26.
Holmes, T. H., & Rahe, R. H. (1967). The social readjustment rating scale. Journal of Psychosomatic Research, 11, 213-218.
27.
Hoyt, W. T., Warbasse, R. E., & Chu, E. Y. (in press [TCP special issue, part 2]). Construct validation in counseling psychology research. The Counseling Psychologist, 34.
28.
Iacobucci, D., & Duhachek, A. (2003). Advancing alpha: Measuring reliability with confidence. Journal of Consumer Psychology, 13, 478-487.
29.
Joint Committee on Testing Practices. (2004). Code of fair testing practices in education. Washington, DC: Author. Retrieved October 10, 2005, from http://www.apa.org/science/FinalCode.pdf
30.
Kahn, J. H. (2006). Factor analysis in counseling psychology research, training, and practice: Principles, advances, and applications. The Counseling Psychologist, 34, 684-718.
31.
Kopalle, P. K., & Lehmann, D. R. (1997). Alpha inflation? The impact of eliminating scale items on Cronbach’s alpha. Organizational Behavior and Human Decision Processes, 70, 189-197.
32.
Krus, D. J., & Helmstadter, G. C. (1993). The probabilities of negative reliabilities. Educational and Psychological Measurement, 53, 643-650.
33.
Landrine, H., & Klonoff, E. A. (1996). The schedule of racist events: A measure of racial discrimination and a study of its negative physical and mental health consequences. Journal of Black Psychology, 22, 144-168.
34.
Malecki, C. K., & Demaray, M. K. (2002). Measuring perceived social support: Development of the Child and Adolescent Social Support Scale (CASSS). Psychology in the Schools, 39, 1-18.
35.
Martens, M. P., & Hasse, R. F. (in press). Advanced applications of structural equation modeling in counseling psychology research. The Counseling Psychologist, 34.
36.
Meier, S. T., & Davis, S. R. (1990). Trends in reporting psychometric properties of scales used in counseling psychology research. Journal of Counseling, 37, 113-115.
37.
Nunnally, J. C. (1970). Introduction to psychological measurement. New York: McGraw-Hill.
38.
Onwuegbuzie, A. J., & Daniel, L. G. (2002). Framework for reporting and interpreting internal consistency reliability estimates. Measurement and Evaluation in Counseling and Development, 35, 89-103.
39.
Onwuegbuzie, A. J., & Daniel, L. G. (2004). Reliability generalization: The importance of considering sample specificity, confidence intervals, and subgroup differences. Research in the Schools, 11, 60-71.
40.
Osburn, H. G. (2000). Coefficient alpha and related internal consistency reliability coefficients. Psychological Methods, 5, 343-355.
41.
Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Lawrence Erlbaum.
42.
Peterson, R. A. (1994). A meta-analysis of Cronbach’s coefficient alpha. Journal of Consumer Psychology, 21, 381-391.
43.
Quintana, S. M., & Minami, T. (in press [TCP special issue, part 2]). Guidelines for meta-analyses of counseling psychology research. The Counseling Psychologist, 34.
44.
Reckase, M. D. (1996). Test construction in the 1990s: Recent approaches every psychologist should know. Psychological Assessment, 8, 354-359.
45.
Roberts, J. K., Onwuegbuzie, A. J., & McLean, J. E. (2003). Alternative approaches for interpreting coefficient alpha with homogeneous subsamples. Research in Schools, 10, 63-69.
46.
Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353.
47.
Sherry, A. (2006). Discriminant analysis in counseling psychology research. The Counseling Psychologist, 34, 661-683.
48.
Streiner, D. L. (2003a). Being inconsistent about consistency: When coefficient alpha does and doesn’t matter. Journal of Personality Assessment, 80, 217-222.
49.
Streiner, D. L. (2003b). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80, 99-103.
50.
Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
51.
Thompson, B. (2003). Score reliability: Contemporary thinking on reliability issues. Thousand Oaks, CA: Sage.
52.
Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174-195.
53.
Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.
54.
Vacha-Haase, T., Kogan, L. R., & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60, 509-522.
55.
Weston, R., & Gore, P. A., Jr. (2006). A brief guide to structural equation modeling. The Counseling Psychologist, 34, 719-751.
56.
Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
57.
Worthington, R. L., & Whittaker, T. A. (in press [TCP special issue, part 2]). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34.
58.
Yin, P., & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60, 201-223.
