Setting Standards on Performance Assessments of Physicians' Clinical Skills Using Contrasting Groups and Receiver Operating Characteristic Curves

Abstract

During recent years, the use of performance assessmentsfor high-stakes evaluation ofphysicians' clinical skills has increased. As a result, considerable attention has been given to the identification of appropriate procedures for setting standards on performance assessments. Several articles have described application of the contrasting groups procedure. It appears that the procedure is not well understood. Relatively little has been written about the requirements for using this procedure appropriately. This article provides a discussion of the theoretical frameworkfor setting standards using contrasting groups. The discussion is then extended to include the circumstance in which the contrasting groups data are interpreted using receiver operating characteristic curves. Examples from the research literature involving variations on the contrasting groups method for setting standards on performance assessments of clinical skills are discussed. The article concludes with guidelines for evaluators interested in using the contrasting groups procedure.

Get full access to this article

View all access options for this article.

References

Clauser, B. E. , and Clyman, S. G. (1994). A contrasting groups approach to standard setting for performance assessments of clinical skills. Academic Medicine, 69 (RIME Supplement), S42-S44.

Colliver, J. E. , Barnhart, A. J. , Marcy, M. L. , & Verhulst, S. J. (1994). Using a receiver operating characteristic analysis to set passing standards for a standardized-patient examination of clinical competence. Academic Medicine, 69 (RIME Supplement), S37-S39.

Hambleton, R. K. , & Plake, B. S. (1995). Using an extended Angoff procedure to set standards on a complex performance assessment. Applied Measurement in Education, 8, 41-55.

Hanley, J. A. , & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29-36.

Harasym, P. H. , & Woloschuk, W. (1992). The halo effect inherent within OSCE checklists. In R. M. Harden , I. R. Hart , & H. Mulholland (Eds.), Approaches to the Assessment of Clinical Competence (pp. 437-446). Dundee, Scotland: Centre for Medical Education.

Herbers, J. E. , Noel, G. L. , Cooper, G. S. , Harvey, J. , Pangaro, L. N. , & Weaver, M. J. (1989). How accurate are faculty evaluations of clinical competence? Journal of General Internal Medicine, 4, 202-208.

Kane, M. (1994, October). Examinee-centered vs. task-centered standard setting. Paper presented at the Joint Conference on Standard Setting for Large Scale Assessment, Washington, DC.

Klass, D. , & Clauser, B. (1994). Evaluating clinical skills. Archives of Pediatric Medicine, 148, 133-134.

Koffler, S. L. (1980). A comparison of approaches for standard setting. Journal of Educational Measurement, 17, 167-178.

10.

Kraemer, H. C. (1988). Assessment of 2 x 2 associations: Generalization of signal-detection methodology. The American Statistician, 42, 37-49.

11.

Livingston, S. A. , & Zieky, M. J. (1982). Passing Scores. Princeton, NJ: Educational Testing Service.

12.

Livingston, S. A. , & Zieky, M. J. (1989). A comparative study of standard setting methods. Applied Measurement in Education, 2, 121-141.

13.

McNeil, B. J. , Keeler, E. , & Adelstein, S. J. (1975). Primer on certain elements of medical decision making. The New England Journal of Medicine, 293, 211-215.

14.

Noel, G. L. , Herbers, J. E. , Caplow, M. P. , Cooper, G. S. , Pangaro, L. N. , & Harvey, J. (1992). How well do internal medicine faculty evaluate the clinical skills of residents? Annals of Internal Medicine, 117, 757-765.

15.

Poldre, P. A. , Rothman, A. I. , Cohen, R. , Dirks, F. , & Ross, J. (1992, April). A judgmentalempirical approach to standard setting for an O.S.C.E. Paper presented at the meeting of The American Educational Research Association, San Francisco.

16.

Stillman, P. L. , Swanson, D. B. , Regan, M. B. , Philbin, M. M. , Nelson, V. , Ebert, T. H. , Ley, B. , Parrino, T. , Shorey, J. , Stillman, A. , Alpert, E. , Caslowitz, J. , Clive, D. , Florek, J. , Hamolsky, M. , Hatem, C. , Kizirian, J. , Kopelman, R. , Levenson, D. J. , Levinson, G. , McCue, J. , Pohl, H. , Schiffnan, F. , Schwartz, J. , Thane, M. , & Wolf, M. (1991). Assessment of clinical skills of residents using standardized patients. Annals of Internal Medicine, 114, 393-401.

17.

Weinstein, M. C. , Feinberg, H. V. , Elstein, A. S. , Frazier, H. S. , Neuhauser, D. , Neutra, R. R. , & McNeil, B. J. (1980). Clinical Decision Analysis. Philadelphia: W.B. Saunders.