Rater Errors in a Clinical Skills Assessment of Medical Students

Abstract

The authors used a many-faceted Rasch measurement model to analyze rating data from a clinical skills assessment of 173 fourth-year medical students to investigate four types of rater errors: leniency, inconsistency, the halo effect, and restriction of range. Students performed six clinical tasks with 6 standardized patients (SPs) selected from a pool of 17 SPs. SPs rated the performance of each student in six skills: history taking, physical examination, interpersonal skills, communication technique, counseling skills, and physical examination etiquette. SPs showed statistically significant differences in their rating severity, indicating rater leniency error. Four SPs exhibited rating inconsistency. Four SPs restricted their ratings in high categories. Only 1 SP exhibited a halo effect. Administrators of objective structured clinical examinations should be vigilant for various types of rater errors and attempt to reduce or eliminate those errors to improve the validity of inferences based on objective structured clinical examination scores.

Keywords

rater errors many-faceted Rasch measurement OSCE standardized patients medical students clinical skills

Get full access to this article

View all access options for this article.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing . Washington, DC: American Educational Research Association.

Anastasi, A. (1988). Psychological testing (6th ed.). New York: Macmillan.

Bernardin, H.J. , & Buckley, M.R. (1981). Strategies in rater training. Academy of Management Review, 6(2), 205-212.

Cooper, W.H. (1981). Ubiquitous halo. Psychological Bulletin, 90(2), 218-244.

Cronbach, L.J. (1990). Essentials of psychological testing (5th ed.). New York: Harper & Row.

Downing, S.M. (2003). Item response theory: Applications of modern test theory in medical education. Medical Education, 37(8), 739-745.

Downing, S.M. , & Haladyna, T.M. (2004). Validity threats: Overcoming interference with proposed interpretations of assessment data. Medical Education, 38(3), 327-333.

Guilford, J.P. (1954). Psychometric methods (2nd ed.). New York: McGraw-Hill.

Hambleton, R.K. , Swaminathan, H. , & Rogers, H.J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.

10.

King, L.M. , Hunter, J.E. , & Schmidt, F.L. (1980). Halo in a multidimensional forced-choice performance evaluation scale. Journal of Applied Psychology, 65(5), 507-516.

11.

Linacre, J.M. (1989). Many-faceted Rasch measurement. Chicago: MESA.

12.

Linacre, J.M. (2002). What do infit and outfit mean-square and standardized mean? Rasch Measurement Transactions, 16(2), 878.

13.

Linacre, J.M. (2005). Facets (Version 3.57) [Computer software]. Chicago: Winsteps.

14.

Linacre, J.M. , & Wright, B.D. (1989). The “length” of a logit. Rasch Measurement Transactions, 3(2), 54-55.

15.

Linacre, J.M. , & Wright, B.D. (2004). Construction of measures from many-facet data . In E. V. Smith, Jr. , & R. M. Smith (Eds.), Introduction to Rasch measurement: Theory, models, and applications (pp. 296-321). Maple Grove, MN: JAM.

16.

Linn, R.L. , & Gronlund, N.E. (2000). Measurement and assessment in teaching (8th ed.). Upper Saddle River, NJ: Prentice Hall.

17.

Miller, G.E. (1990). The assessment of clinical skills/competence/performance . Academic Medicine, 65(9 Suppl.), S63-S67.

18.

Myford, C.M. , & Wolfe, E.W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement , 4(4), 386-422.

19.

Myford, C.M. , & Wolfe, E.W. (2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of Applied Measurement , 5(2), 189-227.

20.

Robbins, S.P. (1989). Organizational behavior (4th ed.). Englewood Cliffs, NJ: Prentice Hall.

21.

Saal, F.E. , Downey, R.G. , & Lahey, M.A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88(2), 413-428.

22.

Scullen, S.E. , Mount, M.K. , & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85(6), 956-970.

23.

Van der Vleuten, C.P.M. , & Swanson, D.B. (1990). Assessment of clinical skills with standardized patients: State of the art. Teaching and Learning in Medicine , 2(2), 58-76.

24.

Wells, F.L. (1907). A statistical study of literary merit. Archives of Psychology, 1, 1-30.

25.

Wright, B.D. , & Linacre, J.M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370.

26.

Wright, B.D. , & Masters, G.N. (1982). Rating scale analysis. Chicago : MESA.

27.

Wright, B.D. , & Panchapakesan, N. (1969). A procedure for sample-free item analysis. Educational and Psychological Measurement, 29, 23-48.

28.

Yudkowsky, R. , Alseidi, A. , & Cintron, J. (2004). Beyond fulfilling the core competencies: An objective structured clinical examination to assess communication and interpersonal skills in a surgical residency. Current Surgery, 61(5), 499-503.