Assessing Clinical Skills Via Telehealth Objective Standardized Clinical Examination: Feasibility,Acceptability,Comparability,and Educational Value

Introduction

The use of telehealth in clinical practice has exploded due to the COVID-19 pandemic with a corresponding impetus to increase telehealth training in both undergraduate and graduate medical curricula. ¹ Early teaching of telehealth existed before the pandemic, with 58% of medical schools reporting the inclusion of telemedicine in required or elective courses in the 2016–2017 academic year. ² However, the events of 2020 necessitated expanded applications of telehealth in medical teaching and clinical skills assessment.

In March 2020, in-person standardized patient (SP) assessments became unsafe for students, faculty, SP actors, and administrative staff. To support students' clinical learning through direct observation of integrated clinical skills within a standardized format, we incorporated telehealth. Based on the creation and successful use of a prerecorded telehealth visit in a first-year geriatric curriculum, ³ we adapted the previously scheduled, June three-case formative Objective Standardized Clinical Examination (OSCE) to a live telehealth OSCE for all students in the Principal Clinical Experience (PCE) year.

We had two aims. We sought to assess students' perceptions of the challenges of adapting foundational clinical skills to a telehealth OSCE anchored in the Association of American Medical Colleges (AAMC) draft telehealth competencies for graduating medical students. Secondarily, we sought to compare students' demonstration of clinical skills in this innovative environment to their performance during the in-person three-case OSCE in which they had participated in January 2020.

Our primary hypothesis was that students who possess foundational clinical skills but have had minimal education or training in the use of telehealth and no significant exposure to working with patients via telehealth modalities would be most challenged by the necessity of adapting their physical examination skills to telehealth encounters.

Our secondary hypothesis was that students would find this adaptation of their integrated OSCE experience to a telehealth format to be cognitively challenging. We routinely assess students' feedback on the cognitive and emotional challenge of their sequential formative OSCEs and theorized that the dramatic change in format would increase cognitive load.

Materials and Methods

The PCE is a year-long clinical immersion that occurs during the second year of medical school. All students who are enrolled in the PCE participate in two formative multicase OSCEs during the PCE year, in January and in June. These practical experiences offer students opportunities to demonstrate an integrated approach to the clinical skills of history-taking, physical examination, communication, and diagnostic reasoning. The experience is repeated twice during the year with different cases during January and June to support longitudinal assessment, formative feedback, and facilitated student growth across a breadth of medicine.

In March 2020, all students' in-person clinical experiences paused due to the COVID-19 pandemic. To preserve the longitudinal, integrated assessment and feedback goals of the June OSCE, we determined that a telehealth visit format was our best opportunity to maintain a consistent approach to clinical skills assessment. A telehealth OSCE would include direct, real-time faculty observation of a student's live-video interactions with a remotely located SP, followed by an immediate feedback discussion among the student, SP, and faculty member. In keeping with other OSCEs, we expected students to demonstrate skills in history-taking, physical examination, communication, documentation of clinical reasoning, and oral presentation.

Faculty with expertise in OSCE case development and telehealth modalities collaborated to create preparatory educational materials for students, faculty members, and SPs, as well as realistic telehealth cases. We reviewed the draft AAMC Telehealth Competencies for graduating medical students and chose those competencies deemed to be accessible and fair to students with minimal prior exposure to telehealth education, as the basis for adapting two existing OSCE cases. [Note, the AAMC Telehealth Competencies draft used in this experience was a 2020 version before subsequent AAMC stakeholder revisions.] The domains and competencies that were highlighted in this experience are shown in Table 1.

Using these competencies, we created the following educational materials: (1) an informational document describing the telehealth competencies and “thought questions” to prompt students to consider how they would adapt their current skills to a telehealth encounter with an SP (Appendix A1); (2) a modular online self-paced course in telehealth ⁴ ; and (3) adaptations of two preexisting in-person SP cases to the telehealth visit OSCE format.

The modular online telehealth course was launched in May 2020, as a required learning experience for all PCE students. Students, who had been sent home from campus, SP actors, and faculty members were scheduled for the June telehealth OSCE sessions via Zoom^® (Zoom Video Communications, San Jose, CA, USA) meeting videoconferencing from individual remote locations.

We selected two SP cases that had been administered to other class cohorts in prior-year OSCEs, based on their clinical relevance to the students' level of training and the ease of adapting the requisite physical examination components to a telehealth visit: primarily, a student's ability to describe and explain physical examination maneuvers to a patient and to observe patient movements or behaviors that were indicative of normal or abnormal physical examination findings. The cases were also based on similar case content and illustrative examples in the modular online telehealth course. The cases were as follows: (1) a presentation of biliary colic with right upper quadrant abdominal pain and (2) a presentation of vascular headache with a normal neurologic examination. The OSCE assessment faculty had access to all preparatory materials.

We created a post-OSCE survey, with a focus on the following themes:

Students' perceptions of the challenge of adapting clinical skills to the telehealth format, recognizing that students had had minimal prior experience with telehealth.

We selected these themes based on the telehealth approach to this OSCE, as well as our prior routine post-OSCE assessments, which seek feedback from students on the cognitive and emotional challenge of cases, the safety of the OSCE learning environment, and the relationship of these experiences with students' overall clinical learning.

We collaboratively developed a 20-question survey, using survey design best-practices, and administered it using Qualtrics^®. In addition to multiple choice questions, students could also provide narrative feedback on the telehealth OSCE. The survey opened to students after each OSCE session via an anonymous e-mail link. No potentially identifying information was solicited in the survey.

The OSCE team administered the two-case telehealth OSCE over the month of June 2020 and the post-OSCE survey opened electronically after each session to all students who had participated in the session. During the orientation to each session, OSCE staff asked students to complete the post-OSCE survey following the session and all students received a reminder e-mail after each OSCE session. We compiled the survey results and analyzed data, with attention to general and telehealth-specific questions. We compared students' responses from the June 2020 telehealth OSCE with the analogous responses from the January 2020 in-person OSCE.

Finally, we compared students' performance on the telehealth OSCE, including total score, history-taking, physical examination, and communication skills, as scored by observing faculty on case-specific checklists and documented electronically via simIQ^® software (Education Management Solutions, LLC, Exton, PA, USA), to the same cohort's scores on the three-case in-person OSCE in which they had participated in January 2020.

Statistical analysis consisted of one-sided and two-sided t-tests and descriptive statistics. Statistical significance was predetermined as p ≤ 0.05.

This work was approved by the HMS Program in Medical Education as educational quality assurance and IRB exempt.

Results

One hundred sixty students (N = 160) participated in 29 two-hour telehealth OSCE sessions over the month of June, a total of 58 h of assessment. Each session hosted six students, each of whom engaged in two SP cases. Six SPs, six observing faculty, and five OSCE and information technology support personnel were required for each session. A total of 18 SPs and 29 faculty participated over the 29 sessions.

Each case encounter consisted of a 25-min interview and adapted physical examination with an SP, 10 min for the student to document a structured note of the encounter, 5 min to provide an oral presentation to the observing faculty member, and 15 min for a feedback conversation between the student, SP, and faculty member.

We received 95 post-OSCE surveys between June 5 and July 1 (59% response rate). We excluded from the analysis six surveys that were less than 50% completed. The median time spent on the excluded surveys was 3.5 s. Data analysis included 89 surveys. The median time to complete the survey was 2 min and 25 s. Thirty-five (39%) of the 89 respondents provided narrative feedback on the experience.

Discussion

We found that the adaptation of a formative OSCE from an in-person to a telehealth experience was feasible (planned and completed in 2 months), and resulted in similar educational quality, based on students' feedback, and our assessment purposes, as indicated by total performance scores.

Not surprisingly, there were aspects of the telehealth OSCE that students indicated to be particularly challenging. As per our primary hypothesis, of all the clinical skills that students were expected to demonstrate through this format, students perceived the adaptation of the physical examination to the telehealth environment to be most challenging. This was also borne out in narrative feedback: 14 students (representing 15.7% of all survey respondents, and 35.9% of all narrative feedback responses) indicated an interest in receiving more education in adapting their physical examination skills to a telehealth encounter. As per our secondary hypothesis, students also indicated significant cognitive challenge in engaging in the telehealth OSCE format.

We hypothesize several interrelated explanations for these findings. One reason may be that these students, who were completing their first full clinical year, are relative novices in the application of physical examination skills in clinical contexts. Adapting hands-on physical examination skills to the task of “talking” a patient through physical examination maneuvers over telehealth may have stressed students' current skill levels and increased the cognitive load associated with the telehealth encounter. Instructing or teaching the patient observable physical examination maneuvers and adapting that instruction to the limitations of being physically remote from the patient may create a great clinical challenge, one that is not limited to students, but may be relevant to any novice telehealth practitioner.

The fact that the students rated the telehealth physical examination as challenging parallels the anecdotal perceptions of practicing physicians who were forced to transition much of their clinical practice to a virtual realm during the early COVID pandemic. There is little in the literature on evidence-based telehealth physical examination guidance ^5,6 ; much of this education is focused on the use of technology to connect with the patient, rather than how to perform an accurate or clinically useful physical examination via telehealth.

Another challenge to the novice's telehealth physical examination is that most practicing providers today have honed their clinical skills via in-person, hands-on encounters. The subtleties of focused observation of the physical examination that enable a provider to recognize “sick” or “not sick” are traditionally part of an in-person clinical skill set. Our students' lesser confidence in the competency of recognizing patients at risk for unsafe conditions via telehealth may be related to this challenge of observation and interpretation, potentially magnified by the telehealth modality, similarly contributing to the overall cognitive challenge of the experience.

Despite modest online instruction and minimal practice in telehealth, students indicated moderate confidence in the four telehealth competencies about which we queried. It is notable, but not surprising, that students' self-reported confidence was greater in the technical or factual aspects of telehealth, and less in the competencies that pertain to actual patient engagement and safety during a telehealth visit. We hypothesize that it is easier to learn the technical principles of telehealth, while as with most new learning, demonstration and application of that learning are more challenging and require practice to attain skills confidence. Demonstrating this applied knowledge may have been an additional contributor to the students' reported cognitive challenge.

Students' June telehealth OSCE performance data indicated that on average, overall, students scored similarly to the January OSCE. However, it is interesting to note that there was a statistically significant increase in physical examination performance and a decrease in communication performance scores. This is discordant with the students' reported perceptions of the relative challenges of these aspects of the telehealth encounters. We note that the two telehealth cases required physical examination maneuvers that would have been demonstrated during the online telehealth elective course, and therefore, this result may reflect students' recall of examples that were embedded in that course, rather than their actual physical examination prowess in the virtual environment.

In contrast, all students' OSCE communication skills are rated by trained SPs using a version of the modified Kalamazoo Communication Skills rubric. The comparative decrease in communication scores during the telehealth OSCE is worth exploring. We hypothesize that this finding is multifactorial. Patients' ratings of communication include not only word usage but also establishment of rapport through nonverbal cues that may be difficult to convey during a telehealth encounter. An alternative explanation for this finding may be that this communication skills instrument is less applicable to a telehealth encounter, and that a patient's expectations for optimal communication with a provider via telehealth need further investigation and understanding. The fact that students' perceptions of the challenges of adapting these two skills are discordant with their overall performance indicates a gap that requires more research and training.

Finally, students indicated that the telehealth OSCE inspired them to learn more to a greater extent than they indicated after their first in-person OSCE in January. Because the June experience occurred amid the COVID-19 pandemic when the use of telehealth was rapidly expanding and was frequently being discussed, we suspect this environmental background likely contributed to the students' reactions and interest.

Conclusion

Telehealth and telepractice have exploded by necessity in the last 12 months and are likely to continue as essential tools for patient and client care in the future. Many of the students who participated in the telehealth OSCE are now seeing patients via telehealth in their clinical clerkships. However, there is little in the literature about telehealth education in medical schools and its relationship with successful, patient-centered telehealth in practice. Our findings demonstrate the feasibility of assessment through a telehealth OSCE. More importantly, our findings contribute to larger discussions about education and training in the use of telehealth.

We note that there are certain telehealth competencies that will require thoughtful educational attention when teaching novices how to adapt existing clinical skills to telehealth. Training learners to engage with patients via virtual modalities will require emphasis on the skills of understandable, verbal explanation and the interpretations of focused observation. Adapting physical examination techniques and honing observation and communication skills may be instrumental to the telehealth provider/patient experience in ways that are underappreciated in the in-person setting. However, as they pertain to formal learning, the achievement of such verbal explanatory skills could potentially facilitate students' deeper medical knowledge and clinical understanding. ^7,8

Finally, although the technical connectivity of telehealth may be a competency that is easier to teach and learn, building the competency of empathy and patient connectedness through telehealth will require innovative learning experiences and practice that will be at the core of students' telehealth education in the future.