In multilevel analyses, problems may arise when using Likert-type scales at the lowest level of analysis. Specifically, increases in variance should lead to greater censoring for the groups whose true scores fall at either end of the distribution. The current study used simulation methods to examine the influence of single-item Likert-type scale usage on ICC(1), ICC(2), and group-level correlations. Results revealed substantial underestimation of ICC(1) when using Likert-type scales with common response formats (e.g., 5 points). ICC(2) and group-level correlations were also underestimated, but to a lesser extent. Finally, the magnitude of underestimation was driven in large part to an interaction between Likert-type scale usage and the amounts of within- and between-group variance.
Bliese, P.D. (1998). Group size, ICC values, and group-level correlations: A simulation. Organizational Research Methods, 1, 355-373.
2.
Bliese, P.D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analyses. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349—381). San Francisco: Jossey-Bass.
3.
Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.
4.
Chen, G., Bliese, P.D., & Mathieu, J.E. (2005). Conceptual framework and statistical procedures for delineating and testing multilevel theories of homology. Organizational Research Methods, 8, 375-409.
5.
Cicchetti, D.V., Showalter, D., & Tyrer, P.J. (1985). The effect of number of rating scale categories on levels of interrater reliability: A Monte Carlo investigation. Journal of Applied Psychology, 9, 31-36.
6.
Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249-253.
7.
Gangestad, S., & Snyder, M. (2000). Self-monitoring: Appraisal and reappraisal. Psychological Bulletin,126, 530-555.
8.
George, J.M., & James, L.R. (1993). Personality, affect, and behavior in groups revisited: Comment on aggregation, levels of analysis, and a recent application of within and between analysis. Journal of Applied Psychology, 78, 798-804.
9.
Glass, G.V., Peckham, P.D., & Sanders, J.R. (1972). Consequences of failure to meet assumptions underlying the fixed effects analysis of variances and covariances. Review of Educational Research, 42, 237-288.
10.
Gupta, A.K. (1952). Estimation of the mean and standard deviation of a normal population from a censored sample. Biometrika,39, 260-273.
11.
Hofmann, S.A., Griffin, M.A., & Gavin, M.B. (2000). The application of hierarchical linear modeling to organizational research. In K. J. Klein & S. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 467-511). San Francisco: Jossey-Bass.
12.
Hox, J.J. (2002). Multilevel analysis. Mahwah, NJ: Lawrence Erlbaum.
13.
James, L.R. (1982). Aggregation bias in estimates of perceptual agreement . Journal of Applied Psychology, 67, 219-229.
14.
Jones, A.P., & James, L.R. (1979). Psychological climate: Dimensions and relationships of individual and aggregated work environment perceptions. Organizational Behavior and Human Performance, 23, 201-250.
15.
Komorita, S.S., & Graham, W.K. (1965). Number of scale points and the reliability of scales. Educational & Psychological Measurement, 4, 987-995.
Russell, C.J., & Bobko, P. (1992). Moderated regression analysis and Likert scales: Too coarse for comfort. Journal of Applied Psychology, 77, 336-342.
27.
Scheffé, H. (1959). The analysis of variance. New York: John Wiley.
28.
Shrout, P., & Fleiss, J. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin,86, 420-428.
