Conway used a special case and several supporting examples to argue that the average proportion of method variance in a multitrait-multimethod (MTMM) matrix can be estimated by averaging the correlated uniquenesses (cus). This article builds on Conway in two ways. First, it generalizes the logic, showing that the average cu is a lower bound estimate of the correct value for the average proportion of method variance. The generalization also allows examination of the method’s applicability to a much broader range of MTMM matrices. Second, this article presents a new method based on confirmatory factor analysis of the covariance matrix of cus for measuring method variance. The new method provides more precise and unbiased estimates of the average proportion of method variance associated with each measurement method, and is the first method that uses the correlated uniquenesses model to estimate the proportion of method variance in individual measures.
Get full access to this article
View all access options for this article.
References
1.
Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103, 411-423.
2.
Bagozzi, R. P., & Yi, Y. (1992). Testing hypotheses about methods, traits, and communalities in the direct-product model. Applied Psychological Measurement, 16, 373-380.
3.
Becker, T. E., & Cote, J. A. (1994). Additive and multiplicative method effects in applied psychological research: An empirical assessment of three models. Journal of Management, 20, 625-641.
4.
Bollen, K. A., & Paxton, P. (1998). Detection and determinants of bias in subjective measures. American Sociological Review, 63, 465-478.
5.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
6.
Chou, C. P., & Bentler, P. M. (1995). Estimates and tests in structural equation modeling. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 37-55). Thousand Oaks, CA: Sage.
7.
Conway, J. M. (1996). Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management, 22, 139-162.
8.
Conway, J. M. (1998). Estimation and uses of the proportion of method variance for multitrait-multimethod data. Organizational Research Methods, 1, 209-222.
9.
Coovert, M. D., Craiger, J. P., & Teachout, M. S. (1997). Effectiveness of the direct product versus confirmatory factor model for reflecting the structure of multimethod-multirater job performance data. Journal of Applied Psychology, 82, 271-280.
10.
Crampton, S. M., & Wagner, J. A. III. (1994). Percept-percept inflation in microorganizational research: An investigation of prevalence and effect. Journal of Applied Psychology, 79, 67-76.
11.
Cudeck, R. (1989). Analysis of correlation matrices using covariance structure models. Psychological Bulletin, 105, 317-327.
12.
Doty, D. H., & Glick, W. H. (1998). Common methods bias: Does common methods variance really bias results?Organizational Research Methods, 1, 374-406.
13.
Horn, J. L. (1967). On the subjectivity of factor analysis. Educational and Psychological Measurement, 27, 811-820.
14.
Jöreskog, K. G., & Sörbom, D. (1996). LISREL 8: User’s reference guide. Chicago: Scientific Software International.
15.
Kenny, D. A. (1979). Correlation and causality. New York: Wiley & Sons.
16.
Kenny, D. A., & Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172.
17.
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
18.
Marsh, H. W. (1989). Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361.
19.
Marsh, H. W., & Bailey, M. (1991). Confirmatory factor analyses of multitrait-multimethod data: A comparison of alternative methods. Applied Psychological Measurement, 15, 47-70.
20.
Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-576.
21.
Ones, D. S., Viswesvaran, C., & Reiss, A. D. (1996). Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81, 660-679.
22.
Rummel, R. J. (1970). Applied factor analysis. Evanston, IL: Northwestern University Press.
23.
SAS Institute. (1989). SAS/STAT user’s guide (Version 6, 4th ed., Vol. 1). Cary, NC: Author.
24.
Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1-26.
25.
Williams, L. J., & Anderson, S. E. (1994). An alternative approach to method effects by using latent-variable models: Applications in organizational behavior research. Journal of Applied Psychology, 79, 323-331.
26.
Wothke, W., & Browne, M. W. (1990). The direct product model for the MTMM matrix parameterized as a second order factor analysis model. Psychometrika, 55, 255-262.
