Balanced Incomplete Block Designs for Inter-Rater Reliability Studies

Abstract

Occasionally, an inter-rater reliability study must be designed so that each subject is rated by fewer than all the participating raters. If there is interest in comparing the raters' mean levels of rating, and if it is desired that each mean be estimated with the same precision, then a balanced incomplete block design for the reliability study is indicated. Methods for executing the design and for analyzing the resulting data are presented, using data from an actual study for illustration.

Get full access to this article

View all access options for this article.

References

Cochran, W.G. , & Cox, G.M. Experimental designs (2nd ed.). New York: Wiley, 1957 .

Feldt, L.S. The approximate sampling distribution of Kuder-Richardson reliability coefficient twenty. Psychometrika, 1965, 30, 357-370.

Scheffé, H. The analysis of variance. New York: Wiley, 1959.

Shrout, P.E. , & Fleiss, J.L. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 1979, 86, 420-428.

Wald, A. On the analysis of variance in case of multiple classifications with unequal class frequencies. Annals of Mathematical Statistics , 1941, 12, 346-350.

Yates, F. Incomplete randomised blocks. Annals of Eugenics, 1936, 7, 121-140.