The fundamental assumption underlying the use of 360-degree assessments is that ratings from different sources provide unique and meaningful information about the target manager’s performance. Extant research appears to support this assumption by demonstrating low correlations between rating sources. This article reexamines the support of this assumption, suggesting that past research has been distorted by a statistical artifact—restriction of variance in job performance. This artifact reduces the amount of between-target variance in ratings and attenuates traditional correlation-based estimates of rating similarity. Results obtained from a Monte Carlo simulation and two field studies support this restriction of variance hypothesis. Noncorrelation-based methods of assessing interrater agreement indicated that agreement between sources was about as high as agreement within sources. Thus, different sources did not appear to be furnishing substantially unique information. The authors conclude by questioning common practices in 360-degree assessments and offering suggestions for future research and application.
Get full access to this article
View all access options for this article.
References
1.
Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Prospect Heights, IL: Waveland Press.
2.
Antonioni, D. (1994). The effects of feedback accountability on upward appraisal ratings. Personnel Psychology, 47, 249-256.
3.
Austin, J. T., & Villanova, P. (1992). The criterion problem: 1917-1992. Journal of Applied Psychology, 77, 836-874.
4.
Baltes, B. B., & Parker, C. P. (2000). Reducing the effects of performance expectations on behavioral ratings. Organizational Behavior and Human Decision Processes, 82, 237-267.
5.
Barber, A. E. (1998). Recruiting employees: Individual and organizational perspectives. Thousand Oaks, CA: Sage.
6.
Barrick, M. R., & Mount, M. K. (1991). The Big 5 personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1-26.
7.
Bartko, J. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83, 762-765.
8.
Bernardin, H. J., & Buckley, M. R. (1981). Strategies in rater training. Academy of Management Review, 6, 205-212.
9.
Bernardin, H. J., & Kane, J. S. (1980). A second look at behavioral observation scales. Personnel Psychology, 33, 509-513.
10.
Bernardin, H. J., & Smith, P. C. (1981). A clarification of some issues regarding the development and use of behaviorally anchored rating scales (BARS). Journal of Applied Psychology, 66, 458-463.
11.
Binning, J. F., & Barrett, G. V. (1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478-494.
12.
Borman, W. C. (1991). Job behavior, performance, and effectiveness. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 2.). Palo Alto, CA: Consulting Psychologists Press.
13.
Borman, W. C. (1997). 360° ratings: An analysis of assumptions and a research agenda for evaluating their validity. Human Resource Management Review, 7, 299-315.
14.
Borman, W. C., Buck, D. E., Hanson, M. A., Motowidlo, S. J., Stark, S., & Drasgow, F. (2001). An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. Journal of Applied Psychology, 86, 965-973.
15.
Borman, W. C., & Dunnette, M. D. (1975). Behavior-based versus trait-oriented performance ratings: An empirical study. Journal of Applied Psychology, 60, 561-565.
16.
Brogden, H. E., & Taylor, E. K. (1950). The theory and classification of criterion bias. Educational and Psychological Measurement, 10, 159-186.
17.
Campbell, J. P. (1990). Modeling the performance prediction problem in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, 2nd ed.). Palo Alto, CA: Consulting Psychologists Press.
18.
Cardy, R. L., & Dobbins, G. H. (1994). Performance appraisal: Alternative perspectives. Cincinnati, OH: South-Western.
19.
Cohen, A., Doveh, E., & Eick, U. (2001). Statistical properties of the rWG(j) index of interrater agreement. Psychological Methods, 6, 297-310.
20.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
Conway, J., & Huffcutt, A. (1997). Psychometric properties of multi-source performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331-360.
23.
Cooper, W. H. (1981). Ubiquitous halo. Psychological Bulletin, 90, 218-244.
24.
Craig, S. B., & Kaiser, R. B. (2003). Applying item response theory to multisource performance ratings: What are the consequences of violating the independent observations assumption? Organizational Research Methods, 6, 44-60.
25.
Dalessio, A. T. (1998). Using multi-source feedback for employee development and personnel decisions. In J. W. Smither (Ed.), Performance appraisal. San Francisco: Jossey-Bass.
26.
DeNisi, A. S., & Peters, L. H. (1996). Organization of information in memory and the performance appraisal process: Evidence from the field. Journal of Applied Psychology, 81, 717-737.
27.
Feldman, J. M. (1981). Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psychology, 66, 127-148.
28.
Fisicaro, S. A. (1988). A reexamination of the relation between halo error and accuracy. Journal of Applied Psychology, 73, 239-244.
29.
Fisicaro, S. A., & Lance, C. E. (1990). Implications of three causal models for the measurement of halo error. Applied Psychological Measurement, 14, 419-429.
30.
Fisicaro, S. A., & Vance, R. J. (1994). Comments on the measurement of halo. Educational and Psychological Measurement, 54, 366-371.
31.
Fleenor, J. W., McCauley, C. D., & Brutus, S. (1996). Self-other rating agreement and leader effectiveness. Leadership Quarterly, 7, 487-506.
32.
Funder, D. C. (1987). Errors and mistakes: Evaluating the accuracy of social judgement. Psychological Bulletin, 101, 75-90.
33.
Gatewood, R. D., & Feild, H. S. (2001). Human resource selection (5th ed.). New York: Harcourt College.
34.
George, J. M. (1990). Personality, Affect, and behavior in groups. Journal of Applied Psychology, 75, 107-116.
35.
Goldstein, I. L., & Ford, J. K. (2002). Training in organizations: Needs assessment, development, and evaluation (4th ed.). Belmont, CA: Wadsworth/Thomson Learning.
36.
Greguras, G. J., & Robie, C. (1998). A new look at within-source interrater reliability of 360-degree feedback ratings. Journal of Applied Psychology, 83, 960-968.
37.
Guilford, J. P. (1954). Psychometric methods (2nd ed.). New York: McGraw-Hill.
38.
Guion, R. M. (1961). Criterion measurement and personnel judgments. Personnel Psychology, 14, 141-149.
39.
Gulliksen (1950). Theory of mental tests. New York: John Wiley.
40.
Harris, M. M. (1994). Rater motivation in the performance appraisal context: A theoretical framework. Journal of Management, 20, 737-756.
41.
Harris, M. M., & Schaubroeck, J. (1988). A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. Personnel Psychology, 41, 43-62.
42.
Hedge, J. W., & Kavanagh, M. J. (1988). Improving the accuracy of performance evaluations: Comparison of three methods of performance appraiser training. Journal of Applied Psychology, 73, 68-73.
43.
Huffcutt, A. I., & Arthur, W., Jr. (1994). Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. Journal of Applied Psychology, 79, 184-190.
44.
Huffcutt, A. I., Conway, J. M., Roth, P. L., & Stone, N. J. (2001). Identification and meta-analytic assessment of psychological constructs measured in employment interviews. Journal of Applied Psychology, 86, 897-913.
45.
Hunter, J. E. (1983). Test validation for 12,000 jobs: An application of job classification and validity generalization to the General Aptitude Test Battery (U.S. Employment Service Test Research Report No. 45). Washington, DC: U.S. Department of Labor.
46.
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-98.
47.
Ilgen, D. R., Barnes-Farrell, J. L., & McKellin, D. B. (1993). Performance appraisal process research in the 1980s: What has it contributed to appraisals in use? Organizational Behavior and Human Decision Processes, 54, 321-368.
48.
James, L. R. (1973). Criterion models and construct validity for criteria. Psychological Bulletin, 80, 75-83.
49.
James, L. R. (1982). Aggregation in estimates of perceptual agreement. Journal of Applied Psychology, 67, 219-229.
50.
James, L. R., Demaree, R., G. & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
51.
James, L. J., Demaree, R. G., & Wolf, G. (1993). rWG: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306-309.
52.
James, L. R., & LeBreton, J. M. (2001). Disentangling issues of agreement, disagreement, and lack of agreement using rWG, rWGj, and r*WGj. Unpublished manuscript, University of Tennessee, Knoxville.
53.
James, L. R., & Mazerolle, M. (2001). Personality in work organizations: An integrative approach. Thousand Oaks, CA: Sage.
54.
Jawahar, I. M., & Williams, C. R. (1997). Where all the children are above average: The performance appraisal purpose effect. Personnel Psychology, 50, 905-925.
55.
Judge, T. A., & Bono, J. E. (2001). The relationship of core self-evaluations traits—self-esteem, generalized self-efficacy, locus of control, and emotional stability—with job satisfaction and job performance: A meta-analysis. Journal of Applied Psychology, 86, 80-92.
56.
Judge, T. A., Thoresen, C. J., Bono, J. E., & Patton, G. K. (2001). The job satisfaction–job performance relationship: A qualitative and quantitative review. Psychological Bulletin, 127, 376-407.
57.
Kaiser, R. B., Craig, S. B., Kaplan, R. E., & McArthur (2002). Practical science and the development of Motorola’s leadership standards. In K. B. Brookhouse (Chair), Transforming leadership at Motorola. Practitioner forum presented at the 17th annual meeting of the Society for Industrial and Organizational Psychology, Toronto, Canada.
58.
Kanfer, R. (1990). Motivation theory and Industrial/Organizational psychology. In M. D. Dunnette & L. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 1). Palo Alto, CA: Consulting Psychologists Press.
59.
Kingstrom, P. O., & Bass, A. R. (1981). A critical analysis of studies comparing behaviorally anchored rating scales (BARS) and other rating formats. Personnel Psychology, 34, 263-289.
60.
Kozlowski, S. W. J., & Hattrup, K. (1992). A disagreement about within-group agreement: disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161-167.
61.
Kraiger, K. (1986). Self, peer, and supervisory ratings of performance. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, Chicago.
62.
Kristof, A. L. (1996). Person-organization fit: An integrative review of its conceptualizations, measurement, and implications. Personnel Psychology, 49, 1-49.
63.
Kulik, C. T., & Ambrose, M. L. (1993). Category-based and feature-based processes in performance appraisal: Integrating visual and computerized sources of performance data. Journal of Applied Psychology, 78, 821-830.
64.
Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127, 162-181.
65.
Lance, C. E. (1994). The latent structure of performance ratings derived from Wherry’s (1952) theory of rating. Journal of Management, 20, 757-771.
66.
Landy, F. J. (1986). Stamp collecting versus science: Validation as hypothesis testing. American Psychologist, 41, 1183-1192.
67.
Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87, 72-107.
68.
Latham, G. P., & Wexley, K. N. (1977). Behavioral observation scales for performance appraisal purposes. Personnel Psychology, 30, 225-268.
69.
Lawler, E. E. III. (1967). The multitrait-multirater approach to measuring managerial job performance. Journal of Applied Psychology, 51, 369-381.
70.
Lee, K., Carswell, J. J., & Allen, N. J. (2000). A meta-analytic review of occupational commitment: Relations with person- and work-related variables. Journal of Applied Psychology, 85, 799-811.
71.
Lindsey, E. H., Homes, V., & McCall, M. W. (1987). Key events in executive’s lives. Greensboro, NC: Center for Creative Leadership.
72.
Locke, E. A. (1999). The essence of leadership. New York: Lexington Books.
73.
Lombardo, M., & McCauley, C. D. (1994). Benchmarks: A manual and trainer’s guide. Greensboro, NC: Center for Creative Leadership.
74.
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
75.
Maurer, R. J., Palmer, J. K., & Ashe, D. K. (1993). Diaries, checklists, evaluations, and contrast effects in measurement of behavior. Journal of Applied Psychology, 78, 226-231.
76.
McCall, M., Lombardo, M. M., & Morrison, A. M. (1988). The lessons of experience: How successful executives develop on the job. New York: Free Press.
77.
McCauley, C. D., & Lombardo, M. (1990). Benchmarks: An instrument for diagnosing managerial strengths and weaknesses. In K. E. Clark & M. B. Clark (Eds.), Measures of leadership. West Orange, NJ: Leadership Library of America.
78.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46.
79.
Mintzberg, H. (1989). Mintzberg on management: Inside our strange world of organizations. New York: Free Press.
80.
Motowidlo, S. J., Borman, W. C., & Schmit, M. J. (1997). A theory of individual differences in task and contextual performance. Human Performance, 10, 71-83.
81.
Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-576.
82.
Murphy, K. R., & Balzer, W. K. (1989). Rater errors and rating accuracy. Journal of Applied Psychology, 74, 619-624.
83.
Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational, and goal based perspectives. Thousand Oaks, CA: Sage.
84.
Murphy, K. R., Cleveland, J. N., & Mohler, C. J. (2001). Reliability, validity, and meaningfulness of multisource ratings. In D. W. Bracken & C. W. Timmreck (Eds.), The handbook of multisource feedback: The comprehensive resource for designing and implementing MSF processes. San Francisco: Jossey-Bass.
85.
Murphy, K. R., & DeShon, R. (2000). Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873-900.
86.
Murphy, K. R., & Shiarella, A. H. (1997). Implications of the multidimensional nature of job performance for the validity of selection tests: Multivariate frameworks for studying test validity. Personnel Psychology, 50, 823-854.
87.
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
88.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
89.
Ones, D. S., Viswesvaran, C., & Reiss, A. D. (1996). Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81, 660-679.
90.
Platt, J. R. (1964). Strong inference. Science, 146, 347-353.
91.
Rothstein, H. R. (1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75, 322-327.
92.
Sackett, P. R., & Yang, H. (2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85, 112-118.
93.
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research. Psychological Bulletin, 124, 262-274.
94.
Schmidt, F. L., Hunter, J. E., & Urry, V. W. (1976). Statistical power in criterion-related validation studies. Journal of Applied Psychology, 61, 473-485.
95.
Schmidt, F. L., & Kaplan, L. B. (1971). Composite vs. multiple criteria: A review and resolution of the controversy. Personnel Psychology, 24, 419-434.
96.
Schmidt, F. L., & Rader, M. (1999). Exploring the boundary conditions for interview validity: Meta-analytic validity findings for a new interview type. Personnel Psychology, 52, 445-464.
97.
Schmidt, F. L., Viswesvaran, C., & Ones, D. S. (2000). Reliability is not validity and validity is not reliability. Personnel Psychology, 53, 901-912.
98.
Schmitt, N., & Chan, D. (1998). Personnel selection: A theoretical approach. Thousand Oaks, CA: Sage.
99.
Schneider, B. (1987). The people make the place. Personnel Psychology, 40, 437-453.
100.
Schneider, B., Goldstein, H. W., & Smith, D. B. (1995). The ASA framework: An update. Personnel Psychology, 48, 747-773.
101.
Scullen, S. E. (1997). When ratings from one source have been averaged, but ratings from another source have not: Problems and solutions. Journal of Applied Psychology, 82, 880-888.
102.
Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
103.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
104.
Smith, G. M. (1970). A simplified guide to statistics for psychology and education (4th ed.). New York: Holt, Rinehart & Winston.
105.
Smith, P. C. (1976). Behaviors, results, and organizational effectiveness: The problem of criteria. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology (pp. 745-775). Chicago: Rand McNally.
106.
Sulsky, L. M., & Balzer, W. K. (1988). Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns. Journal of Applied Psychology, 73, 497-506.
107.
Sulsky, L. M., & Day, D. V. (1994). Effects of frame-of-reference training on rater accuracy under alternative time delays. Journal of Applied Psychology, 79, 535-543.
108.
Tears, R. S., O’Leary, R. S., & Facteau, J. D. (2001, April). Do rating sources matter? A replication of Mount, Judge, Scullen, Systma, and Hezlett (1998). Presented in R. B. Kaiser & S. B. Craig (Co-chairs), Modern analytic techniques in the study of 360-degree performance ratings. Symposium presented a the 16th annual conference of the Society for Industrial and Organizational Psychology, San Diego, CA.
109.
Tornow, W. (1993). Perceptions or reality: Is multi-perspective measurement a means or an end? Human Resource Management, 32, 221-230.
110.
Tsui, A. S., & Ohlott, P. (1988). Multiple assessment of managerial effectiveness: Interrater agreement and consensus in effectiveness models. Personnel Psychology, 41, 779-803.
111.
Van Velsor, E., & Leslie, J. B. (1993). Feedback to managers (Vol. II): A review and comparison of sixteen multi-rater feedback instruments. Greensboro, NC: Center for Creative Leadership.
112.
Varma, A., DeNisi, A. S., & Peters, L. H. (1996). Interpersonal affect and performance appraisal: A field study. Personnel Psychology, 49, 341-360.
113.
Vinchur, A. J., Schippmann, J. S., Switzer, F. S., III, & Roth, P. L. (1998). A meta-analytic review of predictors of job performance for salespeople. Journal of Applied Psychology, 83, 586-597.
114.
Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
115.
Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2002). Moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Confounding construct-level convergence and rating difficulty. Journal of Applied Psychology, 87, 345-354.
116.
Woehr, D. J. (1994). Understanding frame-of-reference training: The impact of training on the recall of performance information. Journal of Applied Psychology, 79, 525-534.
117.
Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Organizational and Occupational Psychology, 67, 189-205.
118.
Woehr, D. J., & Miller, M. J. (1997). Distributional ratings of performance: More evidence for a new rating format. Journal of Management, 23, 705-720.
119.
Yukl, G. (1998). Leadership in organizations (4th ed.). Upper Saddle River, NJ: Prentice Hall.
120.
Zedeck, S. (1995). [Review of Benchmarks]. In J. Conoley & J. Impara (Eds.), The twelfth mental measurements yearbook (Vol. 1, pp. 128-129). Lincoln, NE: Buros Institute of Mental Measurements.
121.
Zedeck, S., & Cascio, W. F. (1982). Performance appraisal decisions as a function of rater training and purpose of appraisal. Journal of Applied Psychology, 67, 752-758.
