The health systems science (HSS) framework articulates systems-relevant topics that medical trainees must learn to be prepared for physician practice. As new HSS-related curricula are developed, measures demonstrating appropriate levels of reliability and validity are needed. The authors describe a collaborative effort between a consortium of medical schools and the National Board of Medical Examiners to create a multiple-choice HSS examination in the areas of evidence-based medicine/population health, patient safety, quality improvement, and teamwork. Fifteen schools administered the 100-question examination through 2 academic years a total of 1887 times to 1837 first-time takers. Total test score mean was 67% (SD 11%). Total test reliability as measured by coefficient α was .83. This examination differentiated between medical students who completed the examination before, during, and after relevant training/instruction. This new HSS examination can support and inform the efforts of institutions as they integrate HSS-related content into their curricula.
SkochelakSE.A decade of reports calling for change in medical education: what do they say?Acad Med. 2010;85(9, suppl):S26-S33.
2.
SkochelakSE. Commentary. A century of progress in medical education: what about the next 10 years?Acad Med. 2010;85:197-200.
3.
LuceyCR.Medical education: part of the problem and part of the solution. JAMA Intern Med. 2013;173:1639-1643.
4.
IrbyDMCookeMO’BrienBC.Calls for reform of medical education by the Carnegie Foundation for the Advancement of Teaching: 1910 and 2010. Acad Med. 2010;85:220-227.
5.
CrossonFJLeuJRoemerBMRossMN.Gaps in residency training should be addressed to better prepare doctors for a twenty-first-century delivery system. Health Aff (Millwood). 2011;30:2142-2148.
6.
SkochelakSEStackSJ.Creating the medical schools of the future. Acad Med. 2017;92:16-19.
7.
GonzaloJDDekhtyarMStarrSR, et al. Health systems science curricula in undergraduate medical education: identifying and defining a potential curricular framework. Acad Med. 2017;92:123-131.
8.
GonzaloJDAhluwaliaAHamiltonMWolpawDRThompsonBM.Aligning education with health care transformation: identifying a shared mental model of “new” faculty competencies for academic faculty. Acad Med. 2018;93:256-264.
9.
DonabedianA.Evaluating the quality of medical care. Milbank Q. 1966;83:691-729.
10.
McDonaldKMSundaramVBravataDM, et al. Closing the Quality Gap: Vol. 7, Care Coordination—A Critical Analysis of Quality Improvement Strategies. Rockville, MD: Agency for Healthcare Research and Quality; 2007.
Institute of Medicine Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academies Press; 2001.
13.
PorterME.What is value in health care?N Engl J Med. 2010;363:2477-2481.
14.
HershWRGormanPNBiagioliFEMohanVGoldJAMejicanoGC.Beyond information retrieval and electronic health record use: competencies in clinical informatics for medical education. Adv Med Educ Pract. 2014;5:205-212.
15.
OteroPHershWJai GaneshAU.Big data: are biomedical and health informatics training programs ready? Contribution of the IMIA Working Group for Health and Medical Informatics Education. Yearb Med Inform. 2014;9:177-181.
16.
World Health Organization. Health impact assessment (HIA): the determinants of health. http://www.who.int/hia/evidence/doh/en/. Accessed September 28, 2018.
17.
Diez RouxAV. Conceptual approaches to the study of health disparities. Annu Rev Public Health. 2012;33:41-58.
18.
GlassTAMcAteeMJ.Behavioral science at the crossroads in public health: extending horizons, envisioning the future. Soc Sci Med. 2006;62:1650-1671.
19.
KilbourneAMSwitzerGHymanKCrowley-MatokaMFineMJ.Advancing health disparities research within the health care system: a conceptual framework. Am J Public Health. 2006;96:2113-2121.
20.
StarrSRReedDAEssaryA, et al. Science of health care delivery as a first step to advance undergraduate medical education: a multi-institutional collaboration. Healthc (Amst). 2017;5:98-104.
21.
GonzaloJDGraafDJohannesBBlattBWolpawDR.Adding value to the health care system: identifying value-added systems roles for medical students. Am J Med Qual. 2017;32:261-270.
22.
WalshDSLazorickSLawsonL, et al. The Teachers of Quality Academy: evaluation of the effectiveness and impact of a health systems science training program. Am J Med Qual. 2019;34:36-44.
23.
BaxleyEGLawsonLGarrisonHG, et al. The Teachers of Quality Academy: a learning community approach to preparing faculty to teach health systems science. Acad Med. 2016;91:1655-1660.
24.
GonzaloJDBaxleyEBorkanJ, et al. Priority areas and potential solutions for successful integration and sustainment of health systems science in undergraduate medical education. Acad Med. 2017;92:63-69.
25.
HawkinsRESwansonDB. Using written examinations to assess medical knowledge and its application. In: HolmboeESHawkinsRE, eds. Practical Guide to the Evaluation of Clinical Competence. Philadelphia, PA: Mosby/Elsevier; 2008:42-59.
26.
SchuwirthLWTvan der VleutenCPM.Different written assessment methods: what can be said about their strengths and weaknesses?Med Educ. 2004;38:974-979.
27.
SchuwirthLWTVerheggenMMvan der VleutenCPMBoshuizenHPADinantGJ. Validation of short case-based testing using a cognitive psychological methodology. Med Educ. 2000;35:348-356.
28.
DowningSM.Validity: on the meaningful interpretation of assessment data. Med Educ. 2003;37:830-837.
29.
MessickS.Standards of validity and the validity of standards in performance assessment. Educ Meas. 1995;14:5-8.
30.
DowningSMHaladynaTM. Handbook of Test Development. Mahwah, NJ: Erlbaum; 2006:176.
31.
PaniaguaMASwygertKA.Constructing Written Test Questions for the Basic and Clinical Sciences. Philadelphia, PA: National Board of Medical Examiners; 2016.
ChernickLPusicMLiuHVazquezHKwokM.A pediatrics-based instrument for assessing resident education in evidence-based practice. Acad Pediatr. 2010;10:260-265.
34.
FrohnaJGParkSM.Promoting the use of evidence-based medicine in pediatrics. J Pediatr. 2002;141:599-600.
35.
HaynesBHainesA.Barriers and bridges to evidence based clinical practice. BMJ. 1998;317:273-276.
36.
RyskinaKLKorensteinDWeissmanAMastersPAlguirePSmithCD.Development of a high-value care subscore on the internal medicine in-training examination. Ann Intern Med. 2014;161:733-739.
37.
EnglanderRFlynnTCallS, et al. Toward defining the foundation of the MD degree: core entrustable professional activities for entering residency. Acad Med. 2016;91:1352-1358.
38.
HavyerRDNorbySMLeep HunderfundAN, et al. Science of health care delivery milestones for undergraduate medical education. BMC Med Educ. 2017;17:145.
39.
Lane-FallMBDavisJJClappJTMyersJSRiesenbergLA.What every graduating resident needs to know about quality improvement and patient safety: a content analysis of 26 sets of ACGME milestones. Acad Med. 2018;93:904-910.
40.
CarraccioCEnglanderRGilhoolyJ, et al. Building a framework of entrustable professional activities, supported by competencies and milestones, to bridge the educational continuum. Acad Med. 2017;92:324-330.
