Do Intraclass Correlation Coefficients Index Interrater Agreement?

Intraclass correlation coefficients (ICCs) often are used to index the level of interrater agreement for job analysis ratings. Typically, ICCs are calculated from a one-way ANOVA design for each of several rated job characteristics in which multiple job analysts rate multiple jobs. ICCs are highest when between mean square (BMS) is large relative to within mean square (WMS), but they can be low either when (a) WMS is large, in which case ICCs appropriately indicate low interrater agreement, or (b) BMS is small, in which case ICCs reflect the attenuating influence of range restriction on the grouping factor (i.e., jobs). Using two job analysis data sets, we found that (a) ICCs covaried significantly and negatively with WMS components and (b) ICCs covaried significantly and positively with BMS. These restults indicate that ICCs cannot be interpreted unambiguously as indices of interrater agreement because they are substantially affected by the level of between-group variance. Interpretational confounding for ICCs can be circumvented by (a) using statistics designed specifically to index within-group interrater agreement, or (b) applying corrections for attenuation due to range restriction.