Abstract
Abstract
Background:
Teaching resuscitation discussions to medical students and residents is time intensive and should be taught by teachers with competence in this area of clinical practice. There are plenty of data that these discussions are often inadequate, and that communication skills training, while time and faculty intensive, improves these conversations. The role of online instruction in teaching communication skills, such as resuscitation discussions, is not established.
Objective:
The study objective was to evaluate the effectiveness of an interactive online curriculum in teaching code status discussions to third-year medical students at one medical school.
Design:
In this study we block randomized third-year medical students to one of two web-based curricula versus a similar written curriculum and used standardized patient scores on a communication exercise to measure differences in performance. We used student surveys to measure student satisfaction with the curricula.
Setting/subjects:
Of 121 students participating in the study between April 2007 and March 2008, 88 were randomized to one of two website formats and 33 to the written curriculum.
Measurements:
We compared average scores on content and communication checklists between groups.
Results:
There were no differences between the three groups in the primary outcome of student performance. We could not analyze data on satisfaction with or completion rates for the three curricula due to poor response rates to the student surveys.
Conclusions:
This block randomized study of web-based curricula versus a written curriculum did not show differences in student performance in code status discussions. The optimal use of online communication training remains unclear and requires further investigation.
Introduction
In practice, code status discussions often fail to meet these expert-recommended criteria. They are often short and do not include discussion of risks, benefits, and outcomes of CPR; or the patient's overall prognosis, goals, and values; or a recommendation.3–6 As a result, patients often fail to understand the outcomes of CPR, and providers fail to understand patients' goals and values.10–12 Many data suggest that patients die in ways that are contrary to their wishes and often in physical, psychosocial, or spiritual distress.13–15 Inadequate training in communication about prognosis, goals of care, and resuscitation preferences is a key barrier to accomplishing expert recommended code status discussions.13–17
Proven teaching methods exist for improving code status discussions; these methods employ observed role-play or practice with patient actors, followed by feedback from experienced faculty.13–16 These methods are faculty and time intensive. We created an online curriculum that would recreate, to the extent possible, the best aspects of face-to-face, faculty facilitated teaching. We used best practices for online curricular development. 4 Our goal was not to replace face-to-face teaching, but to either decrease the overall in-person faculty teaching time or to reinforce the face-to-face teaching that is already being done. Web-based curricula are growing rapidly and have been shown to be equivalent to traditional in-person teaching in many situations. 5 Educators argue that online curricula should supplement but not replace face-to-face teaching. 6
In this paper we describe a randomized controlled evaluation testing the ability of two web-based curricula to improve third-year medical students' scores on standardized patients' evaluations of code status discussions.
Methods
Evaluation setting
University of California San Franciso (UCSF) students practice communication skills in a number of venues during the first three years of medical school. There is great variability in training regarding goals of care discussions, as this training depends on the opportunities a particular student has to watch (and/or practice) such discussions during clinical training. Observed conversations may range from exemplary to inadequate. Dedicated teaching on this topic occurs only if a student works with a particular resident or attending physician with an interest in this subject matter. To address this topic formally in the third-year curriculum, the palliative care and clerkship leadership have together created a three-hour communication workshop on advance care planning and code status discussions at the medical school's clinical skills center. Students participate in this workshop during their medicine rotation and are excused only if the student has a preexisting clinic session that conflicts with the workshop. At the session students receive a 20-minute introduction to advance care planning discussions by the course director. Each student interviews two standardized patients (SPs) for 20 minutes each about their understanding of their illness and desires for life-sustaining treatments. One patient actor has end-stage colon cancer and the other has advanced chronic obstructive pulmonary disease (COPD). Both scenarios were reviewed by palliative care and hospitalist faculty and deemed representative of the types of conversations that frequently occur in the inpatient setting. Further, all SPs were retrained prior to the start of the study to increase the probability of consistency among the actors. Finally, two of the authors watched a sampling of student-SP interactions on video in real time to assure quality and consistency. As part of the formative experience, SPs evaluate student performance during the session using checklists. After each conversation the SPs fill out two checklists evaluating the content of the conversation as well as student communication skills. After the workshop the SPs give feedback to the student.
Interventions and control
Web-based intervention: classic and multimedia
There are two versions of the interactive web-based module: the original, “classic” version and the multimedia version. In both, learners watch short video clips of patient-clinician discussions of code status, observe expert commentary on the video discussions, answer multiple choice questions with immediate feedback, choose phrases that they would use in such discussions, and print out a personalized pocket card that summarizes important lessons from the curriculum and incorporates their own favorite phrases. Learners are stimulated to reflect on what they have seen by listening to critiques from palliative care experts and by answering questions posed directly after the expert commentary. The online tutorial is directed at third- and fourth-year medical students, residents, and advanced practice nursing students, and takes an average of 45 minutes to complete.
The multimedia version was created in response to student and resident feedback that the “classic” version was too slow and traditional for their millennial generation that has grown up with online technology. They argued that a faster-paced version using music and visuals would appeal to their generation. The multimedia version was created from the classic version by a technology-savvy medical student, incorporating more media, including music and text, and quickening the pace of the website. The content of the two versions is identical. Both versions of the web-based curricula were available to students from any computer and the videos were accessible from a wide range of software applications, which were provided free of charge for downloading by students. The multimedia version was submitted for peer review and was published by the Association of American Medical College's MedEdPortal. 7
Written control
The written curriculum contains content identical to the web-based version but without the video scenes, expert commentary, and multimedia elements. Students read the written material on the website and answered the same multiple-choice questions on the website, with answers provided. To make up for the video conversations and expert commentary, we provided a classic article on end-of-life communication, which covered similar concepts. 8
Randomization
This was a randomized trial comparing two web-based curricula to a written curriculum in preparing students for advance care planning discussions. We randomized students to complete one of the web-based curricula or a control written curriculum as pre-work for the SP session. Students at our medical school complete their medical clerkships at one of three sites: a VA hospital, a county hospital, or a tertiary care academic center. Each group of students at a particular hospital was randomized to one of three pre-work assignments to prepare them for the workshop: one of the two web-based curricula or a written curriculum with equivalent content. They were sent a computer link to the web-based curriculum or an electronic version of the written curriculum one week prior to the standardized patient exercise and they were asked to complete their assigned homework prior to the SP experience. We chose this design in an attempt to minimize contamination. Knowing how immersed students are at their assigned hospital during their medicine rotation, we hypothesized that students at the same clerkship site would be more likely to discuss the upcoming SP exercise and their assigned “homework” among themselves than with students at other sites. The UCSF institutional review board approved the study.
Evaluation procedure
The main outcome measure was student performance in conducting code status discussions as measured by SP checklists. After each conversation the SPs filled out two checklists evaluating the content of the conversation as well as student communication skills. The first checklist measured, with four yes/no questions, whether the students had discussed the benefits of CPR, the downsides of CPR, the chances of survival, and the reversibility of the decision. This is denoted as the Content Checklist. At least two hospitalists and three palliative care physicians agreed that these content areas were covered in the web-based and paper curricula. The students received a score that was the sum of the four items so that scores could range from 0 to 4. The second checklist had seven items and measured students' overall communication skills on a four-point Likert scale. Once again there was consensus among our hospitalist and palliative care colleagues that the items on the communication checklist matched nicely with the communication skills modeled and discussed on the website and discussed in the written curriculum. This checklist was adapted from a validated communication scale for medical students. 9 The scores for each item ranged from 0.25 (“Needs improvement”) to 1.0 (“Outstanding”). We summed the seven items and divided by 7 to retain the original scale.
Prior to the study the SPs participated in a focus group to explore their understanding of a good code status conversation. The focus group conversation was recorded and the recordings transcribed. The themes that emerged matched the approach presented in the curricula. This was reassuring, but not surprising, given the SPs were trained by the course director with input from palliative care faculty. In addition, we ran an analysis looking at the variance in SP scoring over many student encounters. There were no SPs who stood out as being particularly harsh or easy graders.
Secondary outcomes were student satisfaction with and time spent on the web-based or written curricula. Directly after the SP exercise we surveyed students electronically on their satisfaction with the web-based or written curricula in preparing them for resuscitation discussions with SPs. All students were asked to estimate the percentage of the website or written materials they completed, the amount of time spent with the electronic or written materials in minutes, and the likelihood they would recommend their assigned curriculum to peers using a Likert scale of 1 to 5 (1=“Strongly disagree,” 5=“Strongly agree”). Students assigned to the web-based intervention were also asked to rate the overall user-friendliness of the website and the usefulness of specific aspects of the curricula (the videos, expert commentary, and multiple-choice questions) using a Likert scale of 1 to 5 (1=“Strongly disagree,” 5=“Strongly agree”). Students assigned to the written curriculum were also asked to rate the overall usefulness of the handout, the multiple-choice questions, and the accompanying article on the same Likert scale.
Analysis
Descriptive statistics were performed on the main outcome measures and on the satisfaction data, including mean scores and standard deviations. The mean scores of the two groups were compared using analysis of variance. Level of significance was set at 0.05. All statistical tests were done using SPSS 17.0 (SPSS Inc., Chicago, IL).
Results
Student performance
Of 121 students participating in the study during the half-day advance care planning workshop from April 2007 to March 2008, 47 were randomized to the classic version of the website, 41 to the multimedia version, and 33 to the written curriculum.
There were no differences between groups for the Content Checklist or the Communication Checklist (see Table 1). All students, regardless of randomization, addressed on average three of the four content areas in the Content Checklist.
Satisfaction with and completion rates for assigned curriculum
We have access to the post-workshop surveys for only 31 of the 88 students (35%) assigned to one of the two website interventions and only 6 of the 33 students (18%) assigned to the written curriculum. We did not do further analyses due to the low number of responses.
Discussion
We block randomized students to three curricula on resuscitation discussions as pre-work for a preexisting standardized patient exercise. We failed to find meaningful differences between the three groups in student performance and therefore were unable to prove our hypothesis that an interactive online curriculum would be more effective than a similar written curriculum, which is not considered an ideal teaching method for communication training. The study may have been underpowered to detect differences between the three groups. With hindsight, it appears overly ambitious to have used two versions of the website; a better study design would have compared one version of the online module with the written curriculum. Unfortunately, we could not analyze differences in satisfaction with and completion of the curricula due to poor student response rates.
Informal discussion with third-year students revealed their anxiety regarding the SP communication exercise (personal communication). Working with patient actors is a relatively new experience for third-year students and they are receiving formative feedback on a topic that has generally not been taught effectively if at all. If the anxiety level of the learners was high, they might not have been able to optimally use what they learned from the pre-work, whether web-based or written.
The dose and timing of the intervention may also explain the negative findings. The website alone may be inadequate to create change in a learner's ability to communicate. After watching the video-based curricula students may come away with new concepts and impressions, but without basis for behavior change. Learners likely need an iterative, interactive approach to learn effective code status conversations, of which a web-based or written curriculum could play one part. Discussion of their impressions of the video module with faculty and peers, and role-playing in a low-stakes setting, might lead to more effective learning. Students may need to practice these communication skills multiple times and receive feedback from experienced instructors who have watched their role-plays or patient interactions. Further, third-year medical students may not be developmentally ready for the online curriculum. More experienced learners, such as residents, with more firsthand experience conducting code status discussions, may benefit more from the websites. A web-based curriculum might serve well as one part of a teaching intervention or as a resource that students or residents could return to multiple times in order to review principles of these discussions. 10
This study has many limitations. First, while the clinical scenarios and assessment tools had face validity, they were not tested more stringently for validity. The SP checklists may not have been able to adequately distinguish differing levels of communication skills among the third-year medical students. Interrater reliability was not assessed for the SPs, leading to questions regarding reliability. The SP focus groups, while informative, did not substitute for measurement of interrater reliability. As stated above, the study may have been underpowered to detect differences in the three groups; it would have been better with hindsight to compare one website group with the written curriculum group.
The poor response rate to the satisfaction and curriculum completion questions was another important limitation. Due to missing data, we don't know whether or how much of the materials were completed prior to the communication workshop. If a minority of students completed their pre-work, it would not be surprising that there were no differences in student performance. Ideally, we would have been able to track how much time each student spent on the website instead of relying on self-report. It would also have been better practice to have the website password-protected to ensure the group assigned to the written curriculum did not obtain access to the website materials.
One lesson learned from this study is the difficulty of implementing a complex, blinded, and unfunded study in the real world of an academic teaching center. We relied heavily and unrealistically on busy staff to make sure that data collection ran smoothly. For example, the satisfaction and completion surveys, unlike the standardized patient checklists, were added to the SP exercise for the purposes of the study. The surveys were downloaded onto the Clinical Skills Center (CSC) computer by CSC staff, who were also asked to remind the students to fill out the survey at the end of the long afternoon. The student surveys were not part of the routine workflow of the SP exercise, making it less surprising that there were deficiencies in collecting these data.
Resuscitation discussions are an important piece of the medical care for patients with serious illness. Patients and their families have a vested interest that they are taught well and in an effective and sensitive manner. The outcomes of these conversations not only impact the weeks and months leading up to death and how and where an individual dies, but also family outcomes, including the grieving process. 11 We owe it to patients, families, and our trainees to teach, model, and perform these discussions well. Online instruction may play a part in teaching these discussions.