For continuous constructs, the most frequently used index of interrater agreement (r wg(1))can be problematic. Typically, rwg(1) is estimated with the assumption that a uniform distribution represents no agreement. The authors review the limitations of this uniform nullr wg(1) index and discuss alternative methods for measuring interrater agreement. A new interrater agreement statistic,a wg(1),is proposed. The authors derive thea wg(1)statistic and demonstrate thatawg(1) is an analogue to Cohen’s kappa, an interrater agreement index for nominal data. A comparison is made between agreement estimates based on the uniformr wg(1)and a wg(1), and issues such as minimum sample size and practical significance levels are discussed. The authors close with recommendations regarding the use ofr wg(1)/rwg(J) when a uniform null is assumed,r wg(1)/rwg(J) indices that do not assume a uniform null,awg(1) / a wg(J)indices, and generalizability estimates of interrater agreement.
Alliger, G. M., & Williams, K. J. (1989). Confounding among measures of leniency and halo. Educational and Psychological Measurement, 49, 1-10.
2.
Berry, K. J., & Mielke, P. W. (1988). A generalization of Cohen’s kappa agreement measure to interval measurement and multiple raters. Educational and Psychological Measurement, 48, 921-933.
3.
Borman, W. C. (1987). Personal constructs, performance schemata, and “folk theories” of subordinate effectiveness: Explorations in an army officer sample. Organizational Behavior and Human Decision Processes, 40, 307-322.
4.
Bozeman, D. P. (1997). Interrater agreement in multi-source performance appraisal: A commentary. Journal of Organizational Behavior, 18, 313-316.
5.
Brown R. D. (2000, April). Interrater agreement reconsidered: The role of maximum possible variance. Paper presented at the 15th Annual Conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
6.
Burke, M. J., & Dunlop, W. P. (2002). Estimating interrater agreement with the average deviation index: A user’s guide. Organizational Research Methods, 5, 159-172.
7.
Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49-68.
8.
Cohen, A., Doveh, E., & Eick, U. (2001). Statistical properties of the rwg(J)subindex of agreement. Psychological Methods, 6, 297-310.
9.
Cohen, J. (1960). Acoefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
10.
Dunlap, W. P., Burke, M. J.,& Smith-Crowe, K. (2003). Accurate tests of statistical significance for rwgsuband average deviation interrater agreement indices. Journal of Applied Psychology, 88, 356-362.
11.
Eby, L. T., & Dobbins, G. H. (1997). Collectivistic orientation in teams: An individual and group-level analysis. Journal of Organizational Behavior, 18, 275-295.
12.
Feldman, J. M. (1981). Beyond attribution theory: Cognitive biases in performance appraisal. Journal of Applied Psychology, 66, 127-148.
13.
Finn, R. H. (1970). Anote on estimating the reliability of categorical data. Educational and Psychological Measurement, 30, 71-76.
14.
Fleenor, J. W., Fleenor, J. B.,& Grossnickle, W. F. (1996). Interrater reliability and agreement of performance ratings: A methodological comparison. Journal of Business Psychology, 10, 367-380.
15.
George, J. M. (1990). Personality, affect and behavior in groups. Journal of Applied Psychology, 75, 107-116.
16.
Hauenstein, N. M. A.,& Alexander, R. A. (1991). Rating ability in performance judgments: The joint influence of implicit theories and intelligence. Organizational Behavior and Human Decision Processes, 50, 300-323.
17.
Hoyt, W. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it?Psychological Methods, 5, 64-66.
18.
Hyatt, D. E., & Ruddy, T. M. (1997). An examination of the relationship between work group characteristics and performance: Once more into the breech. Personnel Psychology, 50, 553-586.
19.
Janson, H., & Olsson, U. (2001). A measure of agreement for interval or nominal multivariate observations. Educational and Psychological Measurement, 61, 277-289.
20.
Janson, H., & Olsson, U. (2004). A measure of agreement for interval or nominal multivariate observations by different sets of judges. Educational and Psychological Measurement, 64, 62-70.
21.
James, L. R. (1982). Aggregation bias in estimates of perceptual agreement. Journal of Applied Psychology, 67, 219-299.
22.
James, L. R., Demaree, R. G.,& Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
23.
James, L. R., Demaree, R. G.,& Wolf, G. (1993). rwg:An assessment of within-group agreement. Journal of Applied Psychology, 78, 306-309.
24.
Johnson, J. W.,& Ferstl, K. L. (1999). The effects of interrater and self-other agreement on performance improvement following upward feedback. Personnel Psychology, 52, 271-303.
25.
Judge, T. A.,& Bono, J. E. (2000). Five factor model of personality and transformational leadership. Journal of Applied Psychology, 85, 751-765.
26.
Klein, K. J., Conn, A. B., Smith, D. B., & Sorra, J. S. (2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86, 3-16.
27.
Kozlowski, S. W.,& Hattrup, K. (1992). Adisagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161-167.
28.
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563-575.
29.
LeBreton, J. M., Burgess, J. R. D., Kaiser, R. B., Atchley, E. K. P.,& James, L. R. (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar?Organizational Research Methods, 6, 78-126.
30.
Li, M. F.,& Lautenschlager, G. (1997). Generalizability theory applied to categorical data. Educational & Psychological Measurement, 57, 813-822.
31.
Lindell, M. K.,& Brandt, C. J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21, 271-278.
32.
Lindell, M. K.,& Brandt, C. J. (1999). Assessing interrater agreement on the job relevance of a test: A comparison of the CVI, T, rWG(J),and r* WG(J)indexes. Journal of Applied Psychology, 84, 640-647.
33.
Lindell, M. K., & Brandt, C. J. (2000). Climate quality and climate consensus as mediators of the relationship between organizational antecedents and outcomes. Journal of Applied Psychology, 85, 331-348.
34.
Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23, 127-135.
35.
Lindell, M. K., Clause, C. S., Brandt, C. J.,& Landis, R. S. (1998). Relationship between organizational context and job analysis task ratings. Journal of Applied Psychology, 83, 769-776.
36.
Mossholder, K. W., Bennett, N.,& Martin, C. L. (1998). Amultilevel analysis of procedural justice context. Journal of Organizational Behavior, 19, 131-141.
37.
Mulvey, P. W., & Ribbens, B. A. (1999). The effects of intergroup competition and assigned group goals on group efficacy and group effectiveness. Small Group Research, 30, 651-677.
38.
Murphy, K. R.,& DeShon, R. (2000). Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873-900.
39.
Nathan, B. R.,& Tippins, N. (1990) The consequences of halo “error” in performance ratings: A field study of the moderating effect of halo on test validation results. Journal of Applied Psychology, 75, 290-296.
40.
Neuman, G. A., & Wright, J. (1999). Team effectiveness: Beyond skills and cognitive ability. Journal of Applied Psychology, 84, 376-389.
41.
Rousseau, D. M. (1985). Issues of level in organizational research: Multi-level and cross-level perspectives. Research in Organizational Behavior, 7, 1-37.
42.
Rousseau, D. M. (2000). Multilevel competencies and missing linkages. In K. J. Klein & S. W. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions(pp. 572-582). San Francisco: Jossey-Bass.
43.
Schmidt, A. M., & DeShon, R. P. (2003, April). Problems in the use ofr wgfor assessing interrater agreement. Paper presented at the 18th Annual Conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
44.
Schmidt, F. L., & Hunter, J. E. (1989). Interrater reliability coefficients cannot be computed when only one stimulus is rated. Journal of Applied Psychology, 74, 638-370.
45.
Schrader, B. W.,& Steiner, D. (1996). Common comparison standards: An approach to improving agreement between self and supervisory performance ratings. Journal of Applied Psychology, 81, 813-820.
46.
Schriesheim, C. A. (1981). The effect of grouping or randomizing items on leniency response bias. Educational and Psychological Measurement, 41, 401-411.
47.
Suen, H. K., & Ary, D. (1989). Analyzing quantitative behavioral observation data. Hillsdale, NJ: Lawrence Erlbaum.
48.
Walker, A. G., & Smither, J. W. (1999). A five year study of upward feedback: What managers do with their results matters. Personnel Psychology, 52, 393-423.
49.
Wherry, R. J.,& Bartlett, C. J. (1982). The control of bias in ratings: Atheory of ratings. Personnel Psychology, 35, 521-551.
