Abstract
Due to the coronavirus disease 2019 (COVID-19) pandemic, several American Board of Medical Specialties members have implemented board exams in an online format. In response, we decided to evaluate the efficacy and receptiveness of otolaryngology faculty and residents to a web-based virtual mock oral examination (MOE). Faculty and residents from DC-metropolitan institutions were recruited for decentralized virtual MOE in early 2020. A total of 28 faculty and 20 residents signed up. Follow-up included a survey study consisting of Likert scale and free-text questions to evaluate receptiveness. Helpfulness of the exercise was rated as an average of 8.8 and 9.06, respectively, by faculty and residents on a 10-point Likert scale. Likelihood to recommend a similar exercise to others was 9.2 and 9.3, respectively, for faculty and residents. All survey respondents said they would participate again if given the opportunity. We conclude that existing videoconferencing technologies can be effective tools for conducting virtual MOE by otolaryngology residency programs.
Resident oral examinations, encouraged by the Accreditation Council for Graduate Medical Education (ACGME) and American Board of Medical Specialties (ABMS), provide a crucial means for residents to demonstrate the knowledge required for board certification. Upon ACGME introduction of Core Competencies in 1999, innovative assessment tools such as standardized patients, simulations, and oral examinations have been increasingly implemented. 1 The ABMS requires passing both written and oral examination for board certification. 2
Most examiners agree that oral exam scores correlate with other markers of resident performance, such as faculty evaluations. 3 Little has been published in otolaryngology literature regarding mock oral examinations (MOEs); however, in general surgery, MOEs have been studied to prepare residents for examination.4-6 While MOEs have become widely adopted in otolaryngology training programs, an examination of MOE has not been documented in the otolaryngology literature.
The paucity of MOE studies in otolaryngology may be attributed to difficulty in organizing and implementing such an examination. Faculty availability and scheduling are the most common problems in organizing the MOE in surgery. 7 In light of the coronavirus disease 2019 (COVID-19) pandemic, scheduling may be even more challenging. Alternatively, web-based scheduling and administration of MOEs offer flexibility and timely delivery of feedback and scores. In response to restrictions on in-person events, our division coordinated with academic institutions in the DC-metropolitan area to organize a virtual MOE for residents.
Methods
Otolaryngology faculty were contacted in early 2020 to participate as examiners in the MOE. Each examiner was asked to specify their availability, what subjects they wanted to oversee, and how many residents they could test. Examiners were asked to choose 2 to 3 topics from subject areas tested by the Otolaryngology Training Examination (OTE) ( Table 1 ). Examiners were advised to spend 45 minutes per session walking residents through history taking, differential diagnosis, appropriate use of diagnostics, and management strategies. Specifics for standardization of the MOE were summarized in an instruction sheet given to examiners. They were permitted to use their preferred videoconferencing platform.
Mock Oral Examination Topics.
Abbreviations: CSF, cerebrospinal fluid; OSA, obstructive sleep apnea; SNHL, sensorineural hearing loss; ZMC, zygomaticomaxillary complex.
Residents were instructed to enroll in 1 to 3 sessions and allowed to choose their subject area and knew the name of the topic’s associated examiner. Each institution determined which residents were required to participate; examination was considered mandatory. Examinations took place over 30 days with scheduling coordinated by the examiner and examinee. Although no standardized grading sheet or formal grading occurred, examiners were advised to provide unstructured feedback to examinees immediately after each session. In some cases, additional feedback was provided by the examiner to the resident’s program director.
Postexam surveys were sent out to examiners and residents. Surveys included a mix of questions about didactics, quality, and helpfulness of testing (see Suppl. Figures S1-S2 in the online version of the article). Content of survey was developed with guidance from the program directors at each institution. Our protocol was determined to be exempt from review by the George Washington Institutional Review Board.
Results
Altogether, 20 residents and 28 faculty participated in the MOE, of whom 18 of 20 (90%) residents and 24 of 28 (86%) faculty completed postexamination surveys. Of residents who completed the survey, 16 (89%) participated in the maximum allowed of 3 scenarios, with 15 (83%) completing more than 2 cases per scenario ( Figure 1 ). When surveyed, using a 10-point Likert scale, how helpful the exercise was, the mean (SD) response was 8.8 (1.15) for faculty and 9.06 (1.21) for residents ( Figure 2 ). When queried about likelihood to recommend a similar exercise to others, the mean (SD) response for faculty and residents was 9.2 (0.98) and 9.3 (1.13), respectively. When asked to rate anxiety levels, residents reported a mean (SD) of 5.2 (1.7) while faculty-perceived anxiety of residents was 3.5 (1.9), a significant difference (P < .003).
Number of examinations per participant and cases per examination.
Helpfulness of exercise to residents and faculty on a Likert scale of 1 to 10.
All respondents said they would participate again in the MOE if given the opportunity. When asked whether virtual MOE would be a fair replacement for in-person examination, 7 of 18 (39%) residents and 11 of 24 (46%) faculty responded yes, 8 of 18 (44%) residents and 6 of 24 (25%) faculty were undecided, and the rest disagreed. Critiques and improvements suggested by respondents included standardization of the web platform, case scenarios, and grading criteria.
Discussion
We present an educational innovation in response to the COVID-19 pandemic. Resident engagement was high; both residents and faculty found the exercise helpful and were likely to recommend a similar exercise to others. While faculty assessment of examinee anxiety was low, residents reported moderate anxiety that was significantly higher than examiner perception, in keeping with previous studies. 8
Beyond the pandemic, virtual MOEs have salient benefits over in-person examinations. Virtual MOEs ease scheduling, which boosts participation. Importantly, the virtual MOE allowed us to organize a large-scale multi-institutional MOE that eased recruitment of outside faculty by eliminating geographic barriers. Resident interaction with unknown examiners has been shown to better simulate test day conditions. 9
Critiques of the virtual MOE by participants focused on standardization of examination. A web-based MOE in surgery has been described 10 that would allow for such standardization in the future. This study is limited by its nonrandom enrollment process, which allows inherent differences between resident seniority, examiner experience, and institutional factors to preclude any direct comparison of resident ability. However, this study serves as a proof of concept to elicit feedback for future implementations of virtual MOEs.
Conclusion
Virtual MOEs are a practical, cost-effective, and time-effective mode of preparing residents for oral board examinations. Involvement of faculty from multiple institutions increases examiner heterogeneity, simulating exam day conditions. While our study was limited to a select few institutions, we believe it would be feasible to recruit other institutions to participate in virtual MOEs.
Author Contributions
Disclosures
Supplemental Material
sj-docx-1-opn-10.1177_2473974X21997392 – Supplemental material for The Virtual Mock Oral Examination: A Multi-institutional Study of Resident and Faculty Receptiveness
Supplemental material, sj-docx-1-opn-10.1177_2473974X21997392 for The Virtual Mock Oral Examination: A Multi-institutional Study of Resident and Faculty Receptiveness by Joseph F. Goodman, Prashant Saini, Alexander J. Straughan, Christopher D. Badger, Punam Thakkar and Philip E. Zapanta in OTO Open: The Official Open Access Journal of the American Academy of Otolaryngology-Head and Neck Surgery Foundation
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
