Psychometric Properties of an Instrument Designed to Measure the Educational Quality of Graduate Training Programs

Abstract

To assess the quality of residency education programs at an academic medical center for purposes of enhancing individual graduate medical education programs, we asked residents and fellows (N = 419) to evaluate their training programs using a Web-based questionnaire (response rate = 70%). Kruskal-Wallis tests, factor analysis, correlations, generalizability/decision studies, and mean plots were used to examine trainee responses and to assess the questionnaire’s measurement properties. Exploratory factor analysis indicated that the instrument had a threefactor structure that correlated highly with overall program rating. Cronbach’s alpha exceeded .80 for all factors, and decision studies revealed that 13 to 23 raters were needed to obtain G-coefficients greater than .70. Mean plots showed that the instrument could discriminate within and among training programs at the item level and the factor level, which should help target improvements across graduate training programs within large institutions.

Keywords

program evaluation graduate medical education reliability

Get full access to this article

View all access options for this article.

References

Accreditation Council for Graduate Medical Education (ACGME) . (2003). ACGME outcome project: General competencies. RetrievedNovember 24, 2003, from http://www.acgme.org/outcome/comp/compFull.asp

Bellini, L. , Shea, J. A. , & Asch, D. A. (1997). A new instrument for residency program evaluation. Journal of General Internal Medicine, 12, 707-710.

Brennan, R. L. (2001). Generalizability theory. New York: Springer-Verlag.

Cohen, J. J. (1994). From the president: educational quality is job one. Academic Medicine, 69, 555-555.

Crick, J. E. , & Brennan, R. L. (1983). GENOVA? A generalized analysis of variance software system (Version 2.1) [Computer software]. Iowa City, IA: American College Testing Program.

Elliot, R. L. , Juthani, N. V. , Rubin, E. H. , Greenfeld, D. , Skelton, W. D. ,& Yudkowsky, R. (1996). Quality in residency education: Toward a broader, multidimensional definition. Academic Medicine, 71, 243-247.

Iverson, D. J. (1998). Meritocracy in graduate medical education? Some suggestions for creating a report card. Academic Medicine, 73, 1223-1225.

Seelig, C. B. (1993). Quantitating qualitative issues in residency training: Development and testing of a scaled program evaluation questionnaire. Journal of General Internal Medicine, 8, 610-613.

Seelig, C. B. , DuPre, C. T. , & Adelman, H. M. (1995). Development and validation of a scaled questionnaire for evaluation of residency programs. Southern Medical Journal, 86, 745-750.

10.

Yudkowsky, R. , Elliot, R. , & Schwartz, A. (2002). Two perspectives on the indicators of quality in psychiatry residencies: Program directors’and residents’. Academic Medicine, 77, 57-64.

11.

Yudkowsky, R. ,& Schwartz, A. (2000). Content, culture, and context: Determinants of quality in psychiatry residency programs. Academic Medicine, 75, S99-S101.