It has been established that construct measures often contain measurement error and method effects. However, the extent of these measurement problems and their effects on personality constructs has not been fully examined. In this study, an analysis of published data sets affords estimates of the amount of trait, method, and error variance of measures used in the social/behavioral sciences. It was found that traits accounted for less than 50 percent of the variance in construct measures among the entire group of data sets included in the study. Measures of attitudes and personality variables contain considerably less trait variance. Implications for theory testing are discussed.
Get full access to this article
View all access options for this article.
References
1.
Allen, J. G. and Hamsher, J. H. (1974). The development and validation of a test of emotional styles. Journal of Consulting and Clinical Psychology, 42, 663-668.
2.
Anderson, J. C. and Gerbing, D. W. (1984). The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis. Psychometrika, 49, 155-173.
3.
Arora, R. (1982). Validation of an S-O-R model for situation, enduring and response components of involvement. Journal of Marketing Research, 19, 505-516.
4.
Avison, W. R. (1978). Auxiliary theory and multitrait-multimethod validation: A review of two approaches. Applied Psychological Measurement, 2, 433-449.
5.
Bagozzi, R. P. and Phillips, L. W. (1982). Representing and testing organizational theories: A holistic construal. Administrative Science Quarterly, 27, 459-489.
6.
Bajtelsmit, J. W. (1979). Convergent and discriminant validation of chartered life underwriter (CLU) examinations by the multitrait-multimethod matrix. Educational and Psychological Measurement, 39, 891-896.
7.
Bentler, P. M. (1984). Theory and implementation of EQS a structural equations program. Los Angeles: BMDP Statistical Software, Inc.
8.
Boomsma, A. (1985). Nonconvergence, improper solutions, and starting values in Lisrel maximum likelihood estimation. Psychometrika, 50, 229-242.
9.
Bouchard, T. J., Jr. (1968). Convergent and discriminant validity of the Adjective Check List and Edwards Personal Preference Schedule. Educational and Psychological Measurement, 28, 1165-1171.
10.
Brinberg, D. , Coleman, J., Hoff, H., Newman, G. and Risk, L. (1980). Interpersonal attraction: A multitrait-multimethod analysis. Representative Research in Social Psychology, 11, 49-54.
11.
Campbell, D. T. (1969). Definitional versus multiple operationalism. Et Al., 2, 14-17.
12.
Campbell, D. T. and Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
13.
Campbell, D. T. and O'Connell, E. J. (1967). Methods factors in multitrait-multimethod matrices: Multiplicative rather than additive?. Multivariate Behavior Research, 2, 409-426.
14.
Campbell, J. P. , Dunnette, M. D., Arvey, R. D., and Kellervik, L. V. (1973). The development and evaluation of behaviorally based rating scales. Journal of Applied Psychology, 57, 15-22.
15.
Cohen, J. (1977). Statistical power analysis for the social sciences. New York: Academic Press.
16.
Cote, J. A. and Buckley, M. R. (1987). Estimating trait, method and error variance: Generalizing across seventy construct validation studies. Journal of Marketing Research, 24, 315-318.
17.
Cote, J. A. and Buckley, M. R. (1988). Measurement error and theory testing in comsumer research: An illustration of the importance of construct validation. Journal of Consumer Research, 14, 579-582.
18.
Dickinson, T. L. and Tice, T. E. (1973). A multitrait-multimethod analysis of scales developed by retranstation. Organizational Behavior and Human Performance, 9, 421-438.
19.
Dillon, W. R. , Kumar, A., and Mulani, N. (1987). Offending estimates in covariance structure analysis: Comments on the cause of and solutions to heywood cases. Psychological Bulletin, 101, 126-135.
20.
Elbert, N. (1979). Questionnaire validation by confirmatory factor analysis: An improvement over multitrait-multimethod matrices. Decision-Sciences, 10, 629-644.
21.
Elig, T. W. and Frieze, I. H. (1979). Measuring causal attributions for success and failure. Journal of Personality and Social Psychology, 37, 621-634.
22.
Evans, M. G. (1969). Convergent and discriminant validities between the Cornell Job Descriptive Index and a measure of goal attainment. Journal of Applied Psychology, 53, 102-106.
23.
Fiske, D. W. (1982). Convergent-discriminant validation in measurement and research strategies. In D. Brindberg, and L. Kidder (Eds.), New directions for methodology of social and behavioral science: Forms of validity in research. San Francisco: Jossey-Bass, 72-92.
24.
Flamer, S. (1983). Assessment of the MTMM matrix validity of Likert scales via confirmatory factor analysis. Multivariate Behavioral Research, 18, 275-308.
25.
Freedman, R. D. and Stumpf, S. A. (1978). Student evaluations of courses and faculty based on a perceived learning criterion: Scale construction, validation, and comparison of results. Applied Psychological Measurement, 2, 189-202.
26.
Futtrell, C. M. (1979). Measurement of salespeople's job satisfaction: convergent and discriminant validity of corresponding INDSALES and Job Descriptive Index scales. Journal of Marketing Research, 16, 594-597.
27.
Ghiselli, E. E. , Campbell, J. P., and Zedeck, S. (1981). Measurement theory for the behavioral sciences. San Francisco: W. H. Freeman and Co.
28.
Haley, G. A. (1970). Item-analysis procedures for enhancing validity of existing personality scales. Psychological Reports, 27, 847-853.
29.
Hicks, J. M. (1967). Comparative validation of attitude measures by the multitrait-multimethod matrix. Educational and Psychological Measurement, 27, 985-995.
30.
House, R. J. and Rizzo, J. R. (1972). Toward the measurement of organizational practices: Scale development and validation. Journal of Applied Psychology, 56, 388-396.
31.
Jackson, D. N. , Ahmed, S. A., and Heapy, N. A. (1976). Is achievement a unitary construct?Journal of Research in Personality, 10, 1-21.
32.
Jackson, D. N. , and Chan, D. W. (1980). Maximum-likelihood estimation in common factor analysis: A cautionary note. Psychological Bulletin, 88, 502-508.
33.
Jackson, D. N. , Hourany, L., and Vidmar, N. J. (1972). A four-dimensional interpretation of risk taking. Journal of Personality, 40, 483-501.
34.
Jacoby, J. (1976). Consumer research: Telling it like it is. Advances in Consumer Research, 3, 1-11.
35.
Jenkins, G. D. Jr. , Nadler, D. A., Lawler III, E. E., and Cammann, C. (1975). Standardized observations: An approach to measuring the nature of jobs. Journal of Applied Psychology, 60, 171-181.
36.
Johnson, S. M. , Smith, P. C. and Tucker, S. M. (1982). Response format of the Job Descriptive Index: Assessment of reliability and validity by the multitrait-multimethod matrix. Journal of Applied Psychology, 67, 500-505.
37.
Karst, T. O. and Most, R. (1973). A comparison of stress measures in an experimental analogue of public speaking. Journal of Consulting and Clinical Psychology, 41, 342-348.
38.
Kassarjian, H. H. (1971). Personality and consumer behavior: A review. Journal of Marketing Research, VIII, 409-418.
39.
Kelley, T. L. (1934). Parsimony in the judgement and measurement of character traits. in T. L. Kelley, and A. C. Krey [Eds.] Tests and measurement in the social sciences, part IV. New York: Charles Scribner's Sons, 403-436.
40.
Kerlinger, F. N. (1986). Foundations of behavioral research, New York: Holt, Rinehart and Winston.
41.
Khan, S. B. (1978). A comparative study of assessing children's school related attitudes. Journal of Educational Measurement, 15, 59-66.
42.
Koppel, M. A. and Sechrest, L. (1970). A multitrait-multimethod matrix analysis of sense of humor. Educational and Psychological Measurement, 30, 77-85.
43.
Kothandapani, V. (1971). Validation of feeling, belief, and intention to act as three components of attitude and their contribution to prediction of contraceptive behavior. Journal of Personality and Social Psychology, 19, 321-333.
44.
Lachman, M. E. , Baltes, P., Nesselroade, R. J., and Wills, S. (1982). Examination of personality-ability relationships in the elderly: The role of the contextual (interface) assessment mode. Journal of Research in Personality, 16, 485-501.
45.
Lamont, D. J. (1983). A three dimensional test for White's effectance motive. Journal of Personality Assessment, 47, 91-99.
46.
Lawler, E. E. (1967). The multitrait-multirater approach to measuring managerial job performance. Journal of Applied Psychology, 51, 369-381.
47.
Lefkowitz, J. and Brigando, L. (1980). The redundancy of work alienation and job satisfaction: Some evidence of convergent and discriminant validity. Journal of Vocational Behavior, 16, 115-131.
48.
Locke, E. A. , Smith, P. C., Kendall, L. M., Hulin, C. L., and Miller, A. M. (1964). Convergent and discriminant validity for M. RONALD BUCKLEY ET AL. areas and methods of rating job satisfaction. Journal of Applied Psychology, 48, 313-319.
49.
Marsh, H. W. (1982). Validity of students' evaluations of college teaching: A multitrait-multimethod analysis. Journal of Educational Psychology, 74, 264-279.
50.
Marsh, H. W. , and Butler, S. (1984). Evaluating reading diagnostic tests: An application of confirmatory factor analysis to MTMM data. Applied Psychological Measurement, 8, 307-320.
51.
Marsh, H. W. , and Smith, I. D., and Barnes, J. (1983). Multitrait multimethod analyses of the self-description questionnaire: Student teacher agreement on multidimensional ratings of student self-concept. American Education Research Journal, 20, 333-357.
52.
Marx, R. W. and Winne, P. H. (1978). Construct interpretations of three self-concept inventories. American Educational Research Journal, 15, 99-109.
53.
Mitchell, T. R. (1970). The construct validity of three dimensions of leadership research. Journal of Social Psychology, 80, 89-94.
54.
Mosher, D. L. (1968). Measurement of guilt in females by self-report inventories. Journal of Consulting and Clinical Psychology, 32, 690-695.
55.
Ostrom, T. (1969). The relationship between the affective, behavioral, and cognitive components of attitude. Journal of Applied Social Psychology, 5, 12-30.
56.
Peter, J. P. (1981). Construct validity: A review of basic issues and marketing practices. Journal of Marketing Research, 18, 133-145.
57.
Peter, J. P. , and Churchill, G. A., Jr. (1986). Relationships among research design choices and psychometric properties of rating scales. Journal of Marketing Research, 23, 1-10.
58.
Ritter, D. R. and Ysseldyke, J. E. (1976). Convergent and discriminant validation of the trait of visual figure-ground perception. Journal of Learning Disabilities, 9, 319-325.
59.
Roberts, M. A. , Milich, R., Loney, J., and Caputo, J. (1981). A multitrait-multimethod analysis of variance of teachers' ratings of aggression, hyperactivity, and inattention. Journal of Abnormal Child Psychology, 9, 371-380.
60.
Rose, G. L. , and Andiappan, P. (1978). Sex effects on managerial hiring decisions. Academy of Management Journal. 21, 104-112.
61.
Roshal, S. M. , Frieze, I., and Wood, J. T. (1971). A multitrait-multimethod validation of measures of student attitudes toward school, toward learning and toward technology in sixth grade children. Educational and Psychological Measurement, 31, 999-1006.
62.
Ryan, M. J. , and Bonefield, E. H. (1975). The Fishbein extended model and consumer behavior. Journal of Consumer Research, 2, 118-136.
63.
Saylor, C. F. , Finch, A. J. Jr., Baskin, C. H., Furey, W., and Kelly, M. M. (1984). Construct validity for measures of childhood depression: Application of MTMM methodology. Journal of Consulting and Clinical Psychology, 52, 977-985.
64.
Schmitt, N. , Coyle, B. W., and Saari, B. B. (1977). A review and critique of analyses of multitrait-multimethod matrices. Multivariate Behavioral Research, 12, 447-478.
65.
Schneider, B. (1970). Relationships between various criteria of leadership in smnall groups. Journal of Social Psychology, 82, 253-261.
66.
Seymour, D. and Lessne, G. (1984). Spousal conflict arousal: Scale development. Journal of Consumer Research, 11, 810-821.
67.
Sullivan, J. L. (1982). Reliability and validity assessment. Newbury Park: Sage Publications.
68.
Tapp, G. S. and Barclay, J. R. (1974). Convergent and discriminant validity of the Barclay classroom climate inventory. Educational and Psychological Measurement, 34, 439-447.
69.
Teas, R. K. , Wacker, J. G., and Hughes, R. E. (1979). A path analysis of causes and consequences of salespeople's perceptions of role clarity. Journal of Marketing Research, 16, 355-369.
70.
Teplin, L. A. (1977). Racial preference as artifact? A multitrait-multimethod analysis. Social Science Quarterly, 57, 834-848.
71.
Thorndike, R. L. (1982). Applied psychometrics. Boston: Houghton Mifflin.
72.
Turner, C. J. (1981). The multitrait-multimethod matrix: 1967-1980. JSAS Catalog of Selected Documents in Psychology, 11, 46.
73.
Van Driel, O. P. (1978). On various causes of improper solutions in maximum likelihood factor analysis. Psychometrika, 43, 225-243.
74.
Watkins, D. and Hattie, J. (1981). An investigation of the construct validity of three recently developed personality instruments: An application of confirmatory multimethod factor analysis. Australian Journal of Psychology, 33, 277-284.
75.
Wicker, A. W. (1969). Attitudes versus action: The relationship of verbal and overt behavioral responses to attitude objects. Journal of Social Issues, 25, 41-78.
76.
Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9(1), 1-26.
77.
Wirth, S. and Pease, D. (1983). Convergent and discriminant validity of the Iowa social competency scale for preschool children. Educational and Psychological Measurement, 43, 305-314.
78.
Zedeck, S. and Baker, H. T. (1972). Nursing performance as measured by behavioral expectation scales: A multitrait-multirater analysis. Organizational Behavior and Human Performance, 7, 457-466.
79.
Zuckerman, M. , Persky, H., Eckman, K. M., and Hopkins, T. R. (1967). A multitrait multimethod measurement approach to the traits (or states) of anxiety, depression and hostility. Journal of Projective Technique and Personality Assessment, 31, 39-48.
