Estimating Interrater Agreement with the Average Deviation Index: A User’s Guide

Abstract

The authors present guidelines for establishing a useful range for interrater agreement and a cutoff for acceptable interrater agreement when using Burke, Finkelstein, and Dusig’s average deviation (AD) index as well as critical values for tests of statistical significance with the AD index. Under the assumption that judges respond randomly to an item or set of items in a measure, the authors show that a criterion for acceptable interrater agreement or practical significance when using the AD index can be approximated as c/6, where c is the number of response options for a Likert-type item. The resulting values of 0.8, 1.2, 1.5, and 1.8 are discussed as standards for acceptable interrater agreement when using the AD index with 5-, 7-, 9-, and 11-point items, respectively. Using similar logic, the AD agreement index and interpretive standard are generalized to the case of a response scale that involves percentages or proportions, rather than discrete categories, or at the other extreme, the assessment of interrater agreement with respect to the rating of a single target on a dichotomous item (e.g., yes-no, agree-disagree, true-false item formats). Finally, the usefulness of these guidelines for judging acceptable levels of interrater agreement with respect to the metric (or units) of the original response scale is discussed.

Get full access to this article

View all access options for this article.

References

Borucki, C. C., & Burke, M. J. (1999). An examination of service-related antecedents to retail store performance. Journal of Organizational Behavior, 20, 943-962.

Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49-68.

Campion, M. A., Medsker, G. J., & Higgs, A. C. (1993). Relations between work group characteristics and effectiveness: Implications for designing effective work groups. Personnel Psychology, 46, 823-850.

Cascio, W. F. (1998). Applied psychology in human resource management. Upper Saddle River, NJ: Prentice Hall.

Cronbach, L. J. (1990). Essentials of psychological testing. New York: Harper & Row.

Edgington, E. S. (1980). Randomization tests. New York: Marcel Dekker.

George, J. M. (1990). Personality, affect, and behavior in groups. Journal of Applied Psychology, 75, 107-116.

Hater, J. J., & Bass, B. M. (1988). Superiors’ evaluations and subordinates’ perceptions of transformational and transactional leadership. Journal of Applied Psychology, 73, 695-702.

James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.

10.

James, L. R., Demaree, R. G., & Wolf, G. (1993). rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306-309.

11.

Kaplan, R. M., & Saccuzzo, D. P. (1993). Psychological testing. Pacific Grove, CA: Brooks/ Cole.

12.

Kozlowski, S. W. J., & Hattrup, K. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161-167.

13.

Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23, 127-135.

14.

Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill.

15.

Rothstein, H. R. (1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75, 322-327.

16.

Schmidt, F. L., & Hunter, J. E. (1989). Interrater reliability coefficients cannot be computed when only one stimulus is rated. Journal of Applied Psychology, 74, 368-370.

17.

Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.