Code Status Discussion Skill Retention in Internal Medicine Residents: One-Year Follow-Up

Introduction

Physicians in training vary in their ability to communicate effectively with patients and families.^1,2 This is despite the fact that competence in communication skills is required for residents and fellows in all disciplines. ³ Although baseline competence is not assured, studies show that educational interventions improve skills in areas such as communicating with patients with cancer ⁴ and patient-centered interviewing. ⁵

Communicating with patients at the end of life is especially challenging for trainees. Physicians avoid breaking bad news, ⁶ and concerns have been raised about the ability of residents and faculty to address end-of-life decision making with patients. ⁷ Facilitating a code status discussion (CSD), in which patients facing serious illness have the opportunity to discuss goals and preferences for medical care and treatments, is a challenging and complex task. Yet, it is performed frequently by junior trainees in inpatient settings. ⁸

Earlier research demonstrates that focused interventions improve trainee skill in facilitating end-of-life conversations.^2,9,10 At Northwestern University, internal medicine residents completed a multimodality intervention in CSD skills including small group sessions, deliberate practice, and self-study. ⁹ Skill was significantly better in intervention group residents than controls, as assessed by performance on a simulated CSD with a standardized patient. However, post-intervention skills were assessed only at 2 months, ⁹ similar to short-term follow-up reported in other studies.^2,10 The aim of the current study was to evaluate CSD skill retention one year after a communication skills intervention.

Method

This was a longitudinal cohort study of CSD skill retention. The study was performed at Northwestern University's Feinberg School of Medicine in January 2012. The Northwestern University Institutional Review Board approved the study and all participants provided informed consent.

From July 2010 to January 2011, first-year internal medicine residents (n=38) were randomized to complete an intervention in CSD skills or serve as controls. ⁹ All residents were graduates of United States medical schools; complete demographic information is available elsewhere. ⁹ Nineteen residents completed the intervention including small group teaching, on-line modules, and a log of CSDs performed in actual clinical care. ⁹ Nineteen residents completed clinical training alone and did not participate in the intervention. All 38 residents (intervention and control group) completed a CSD clinical skills examination (CSE) with a trained standardized patient in January, 2011. CSD skills were assessed via an 18-item behavioral checklist composed of three sections (a) patient-centered interviewing skills, (b) CSD skills, and (c) responding to emotion. ⁹ Each checklist item was scored dichotomously (done correctly or done incorrectly) and given equal weight. Each item performed correctly received one point and points were summed for a total score. Intervention group residents performed significantly better than control group residents on the CSE. ⁹ Residents in both groups received 10 to 15 minutes of individualized feedback from one of two trained hospice and palliative care certified faculty members after the CSE.

One year later, residents from both groups (n=38) were invited to participate in the current study to assess CSD skill retention. Residents were asked to perform a CSD with a trained standardized patient recruited from the Northwestern University Clinical Education Center using a standardized clinical scenario. CSD CSE scores from 2012 were compared with CSD CSE scores from 2011.

CSEs in 2011 and 2012 used the same format. In 2011, residents used a scenario of a patient admitted to the hospital with advanced colon cancer. In 2012, residents used a scenario of a patient with recurrent non-Hodgkin's lymphoma admitted to the hospital with fever and neutropenia. Residents were instructed to carry out the CSD as they would with an actual patient. Residents were allowed 15 minutes to complete the simulated CSD. CSDs were videotaped and one of two faculty members watched the interaction through closed-circuit video. This faculty member scored the CSE using the 18-item checklist and provided 10 to 15 minutes of individualized feedback to each resident.

A random sample of 19 of 38 (50%) of 2012 CSEs were rescored by a second rater to obtain inter-rater reliability. One rater was involved in the training intervention in 2011, but the second rater was blind to group assignment (intervention versus control) and results of the first rater's checklist score.

Participants reported self-confidence in leading an actual CSD using a 100-point scale (0=not confident and 100=very confident) and clinical experience with the procedure in actual clinical care. Whether or not participants rotated on the palliative care service 2-week elective during the study period was also collected.

The primary outcome measure was a comparison of group (intervention versus control) mean checklist scores in 2011 and 2012. Secondary outcome measures were CSD self-confidence and experience. Correlations were calculated between secondary outcomes and skill retention.

Checklist inter-rater reliability was estimated using the kappa coefficient. ¹¹ Mean CSE scores, clinical experience, and self-confidence for each occasion were compared using Wilcoxon rank sum tests. Differences between groups were assessed using Mann-Whitney U tests. Relationships between self-confidence, CSD clinical experience, and CSE performance in 2012 were assessed using Spearman correlations.

Results

All 38 residents consented to participate in the study and completed the entire protocol. Inter-rater reliability was substantial (kappa=0.69) across the 18 checklist items.

As shown in Figure 1, there was a statistically significant difference between group mean CSE scores. Intervention group residents retained their CSD skills as mean performance was 75.1% (SD=8.9%) in 2011 and 71.9% (SD=16.0%) in 2012 (p=0.46). Control group residents did not develop additional CSD skills as 2011 checklist mean performance was 53.2% (SD=16.2%) and 2012 performance was 54.7% (SD=17.1%; p=0.78). In both years, the difference between the intervention group and the control group was statistically significant (p<0.001).

OPEN IN VIEWER

FIG. 1.

Performance of internal medicine residents on a code status discussion clinical skills examination in 2011 and at one-year follow up.

Trainees in both groups reported performing approximately 25 CSDs in actual clinical care and reported high self-confidence. There was no significant correlation between 2012 follow-up CSE and self-confidence or CSD clinical experience (Table 1). Six residents rotated on palliative care during the study period. This did not significantly predict follow-up scores.

Discussion

This study demonstrates that CSD skills acquired after a multimodality intervention were retained at one year. New postgraduate education models require that residents demonstrate competence in communicating with patients. ¹² In these models, academic advancement depends on achievement of predetermined competency milestones. However, use of competency-based education is complex because clinical skill retention in residents is rarely measured. Earlier research shows that skills are retained after simulation-based education in advanced cardiac life support and central venous catheter insertion.^13,14 However, to our knowledge this is the first study to address retention of CSD skills, which are complex communication skills.

We believe that the primary reason why internal medicine residents retained CSD skills was that the original intervention was powerful and sustained. ¹⁵ It lasted several months and required self-study and reflection in addition to participation in small group sessions lead by expert faculty. ⁹ Clinical skill retention occurs when trainees participate actively and have time for deliberate practice and feedback. ¹⁶ Similar results cannot be expected after passive interventions such as lectures. ¹⁷ This is confirmed because control group residents did not improve after performing a CSD and receiving 10 to 15 minutes of feedback at the initial session in 2011. Educators should consider these findings when designing interventions to boost communication skills in trainees.

Another important finding is that residents in the control group did not acquire CSD skills despite an additional year of training and clinical experience. First-year residents in the intervention group performed significantly better in 2011 than second-year residents in the control group in 2012. Substantial evidence shows that cumulative experience is not a proxy for clinical skill. ¹⁸ This may be because deliberate practice rarely occurs in clinical settings or because mistakes are passed down from faculty to learners. ¹⁹ Effective health care communication has been linked to improved patient care.^20,21 Our findings suggest that these skills will not develop in the absence of high-quality educational programs and rigorous assessment measures.

Lack of correlation between actual skill in leading a CSD and self-assessment of competency in experienced medical residents is another important finding. This result is similar to studies of emergency medicine, general surgery, and internal medicine residents.^22–24 It emphasizes the need for clinical assessments that yield reliable data rather than reliance on self-assessment for competency determinations.

This study has several limitations. First, it was conducted at one institution with a small sample size. Second, one rater performing checklist scoring was not blind to group assignment. This was addressed by including a second rater blinded to group status. Third, only one scenario was assessed during each CSE. Finally, we did not evaluate clinical outcomes. Further study is needed to link CSD education to improved physician-patient communication and quality of care. Replicating these findings through assessment of actual rather than simulated CSDs is an important next step.

In conclusion, this study demonstrates that a CSD multimodality intervention yields skill improvement that is resistant to decay over one year. Leading a CSD requires complex skills that residents often fail to acquire in clinical training but that can be taught in a systematic way. We are encouraged that CSD skills, once attained, are maintained for one year. Based on these results, we require all residents complete training prior to performing a CSD in actual clinical care.