Training Clinicians in Serious Illness Communication Using a Structured Guide: Evaluation of a Training Program in Three Health Systems

Abstract

Background:

Failure to initiate discussions about patients' values and goals in serious illness remains a common problem. Many clinicians are inadequately trained for these discussions.

Objective:

Evaluate whether a novel train-the-trainer model results in high-quality training that improves clinicians' self-reported competencies in serious illness communication.

Design:

Multimethod evaluation of an educational program.

Setting/Context:

In 2016, three faculty at Ariadne Labs (AL) conducted three train-the-trainer courses to equip faculty trainers at each of the three institutions to teach serious illness communication to clinicians.

Measures:

As collected by a post-training questionnaire, primary evaluation measure is clinicians' self-reported change in skills after the training compared with before. Secondary measures include a course evaluation and qualitative learnings.

Results:

From 2016 to 2018, AL trained 22 trainers (19/22 were palliative care specialists) in three systems, who trained 297 clinicians (49% physicians; 35% advanced practice clinicians; 12% registered nurses, social workers, or chaplain; 4.0% Other) spanning subspecialties (48%); primary care (28%); palliative care (17%); and other (7.1%). Clinicians reported statistically significant improvement in all skills for two of the systems, with a third system demonstrating improvement in all skills with two reaching statistical significance (p < 0.0001). Participants rated the quality of the training highly (95% mostly/extremely effective) and shared a diverse array of takeaways that reflect positive shifts in knowledge, attitudes, and skills.

Conclusion:

Serious illness communication training, delivered through a train-the-trainer model, was highly acceptable and resulted in significant self-reported improvements in competencies of clinicians. This may be a viable method for health systems seeking to train their clinical workforce.

Background

Patients with serious illness regularly experience avoidable suffering and nonbeneficial care at the end of life.^1–4 Inadequate, late, and infrequent communication about patients' values, goals, and preferences (“serious illness communication”) significantly contributes to this problem.^4–7 Achieving excellence in serious illness communication requires clinician competencies in understanding patients' goals, sharing prognostic information, and recognizing and responding to patients' emotions.^8–12 However, numerous studies demonstrate gaps in clinical education that leave clinicians inadequately prepared to engage their patients in high-quality communication.^4,13,14

Many barriers to communication have been identified: lack of knowledge about how to approach end-of-life communication, variation in attitudes about the right time to start, time constraints, and worries about patient reluctance.^13,15–19 While multiple communication training programs have improved clinicians' skills,^20–23 several important challenges remain. First, some courses require two to four days of clinicians' time in workshops outside of their clinical setting.^20,22,24,25 Second, while many courses have been tested with small numbers of clinicians or trainees, they employ different training methods and outcomes, which limits generalizability.^23,26–29 Third, data on impact of training interventions on the attitudinal barriers identified above are limited.^30–32 We need generalizable, scalable workforce training that addresses key barriers to clinician uptake.

Training-of-trainers or faculty development programs have the potential to help meet this need.^33–40 Faculty development models have demonstrated improvements in teaching performance as rated by faculty participants and evaluations.^35,41 While specific faculty development models have been described for serious illness communication and related skills,^40,42 to our knowledge, the effect of the courses taught by newly trained faculty on their learners' competencies remains unknown. More evaluations are needed on the use of train-the-trainer models to guide the development and implementation of scalable, high-quality workforce training in serious illness communication.

Objective

We designed a novel train-the-trainer program with the goal of equipping faculty trainers to teach a short, rigorous training to clinicians in their systems. Our primary objective is to evaluate whether this program results in training that improves clinicians' self-reported competencies in serious illness communication.

Evaluation Design

We conducted an evaluation of a training program from surveys routinely collected as part of quality improvement initiatives at three institutions. In 2016, three faculty at Ariadne Labs (AL) conducted three train-the-trainer courses to equip faculty trainers at each of these institutions to teach serious illness communication. AL faculty were all palliative care physicians with three or more years of experience teaching communication skills and train-the-trainer programs, one of whom had more than 35 years of experience. The two-day train-the-trainer curricula was based on best educational practices and adult learning theories, including attention to knowledge, attitudes, and skills-oriented practice with feedback.^33,37,43,44 The goal was to prepare faculty trainers to deliver a predesigned, structured 2.5–3-hour clinician training on serious illness communication.

The clinician training, tested in two research trials,^45,46 teaches clinicians to have conversations about patients' values, goals, and prognosis using a scalable tool, the Serious Illness Conversation Guide (“Guide”).⁸ The training involves interactive methods, including reflection, demonstration and debriefing, cognitive maps, and skills practice with feedback.³³ Table 1 includes more detail about the training course.

Table 1.

Clinician Training Program on the Serious Illness Conversation Guide

Learning domain	Learning objectives	Training method	Exemplary language
Attitudes e.g., reluctance to have serious illness conversations, varying beliefs about the appropriate timing, concerns about harming patients, dislike of “checklists” or “guides”	Identify personal barriers to starting serious illness conversations Describe my role in improving serious illness conversations	A reflective exercise helps learners transition from their usual work and engages learners at an emotional level by bringing in prior personal and professional experiences (10 minutes) Demonstration and debriefing of the Serious Illness Conversation Guide elicits positive and negative reactions to use of a structured communication guide, and allows clinicians to voice their reservations up front, to facilitate openness to learning new skills (25 minutes)	“Think of a patient with serious illness in which the patient had a good or bad outcome at the end of life, and you think that outcome was related to communication about goals-of-care (or lack thereof). Who would be willing to share a story?” “What are your reactions to the serious illness conversation?” “What are the benefits and tradeoffs of using a Conversation Guide?”
Knowledge e.g., evidence behind serious illness communication and its link to patient outcomes; structure for conversations; patient-tested language to share prognosis and explore values and goals.	Describe the evidence-based benefits of serious illness conversations for patients and families List the components of a structured approach to serious illness communication using a guide	Highlight evidence linking serious illness communication to improved patient outcomes (5 minutes) Provide a framework for understanding the rationale and language in the Serious Illness Conversation Guide (10 minutes)	“The literature suggests that patients' values and goals are most effective when they start before patients are close to the end of life, to give them and their families time to consider their goals and choices and to prepare themselves and their families for what is ahead.” “One of the biggest clinical challenges in conducting serious illness conversations is managing the patient's, family's, and one's own anxiety, which is inevitable in these discussions. The Guide was designed with extensive feedback from patients and clinicians. The language is very open and intentionally hypothetical and gentle to ensure a patient-centered approach that reduces the anxiety and discomfort that accompany these conversations.”
Skills e.g., techniques for patient-centered communication, including using silence, responding to emotion, employing open-ended questions, and titrating the conversation to the patient's emotional readiness to ensure safety.	Practice using the Serious Illness Conversation Guide	Create a safe learning environment by asking the group to identify ground rules and by leading an “I hate role play” exercise (10 minutes) Through a simulated encounter with a “patient actor” and feedback from an expert facilitator, clinicians practice skills in: (80–90 minutes) • Patient-centered, safe initiation of a serious illness conversation • Assessing illness understanding and information preferences • Sharing prognosis tailored to patient preferences • Using silence and responding to emotion • Exploring goals and values with open-ended questions • Making a recommendation based on patients' priorities	“What are the ground rules for a safe small group learning environment?” “The goal of the role play is to practice using the Serious Illness Conversation Guide with a patient actor. As the facilitator, I will call a timeout to highlight a learning point.” “Let's call a timeout. How's it going? What are you doing well? What might you do differently?” “I noticed that when you (e.g., told the patient her prognosis), she seemed overwhelmed and when you asked her what her goals were, the question didn't really register.” What do you think was happening? What might you do differently? I suggest (e.g., pausing and using silence, or using an acknowledging statement such as ‘this must be so hard’ after sharing prognostic information to allow the patient time to process.)”

Setting and Context

Twenty-two faculty trainers at the three health care systems (16 palliative care physicians, 3 palliative care nurses or advanced practice clinicians, 1 hospitalist physician, 1 primary care physician, and 1 oncology physician) completed the train-the-trainer course and then delivered clinician training courses at their health systems. System 1 is an integrated health care system in Texas; System 2 is a community hospital with affiliated practices in Massachusetts; and system 3 is an academic health care system in Pennsylvania.

Training Methods Employed

In Systems 1 and 3, trainers reported by survey that their methods very closely resembled the original course; they used standardized patients or actors for role play for all sessions, provided individualized feedback for nearly all or all clinicians, and maintained a training time of 2.5 hours or more. In System 2, trainers did not use standardized patients, provided individualized feedback about half of the time, and shortened the training to two hours for half the sessions.

Measurements

Questionnaire development

We analyzed responses to a single self-report post-training evaluation questionnaire completed by clinicians who attended the training. Because training can cause respondents to reevaluate and downgrade their prior (preprogram) assessments of their own skills, approaches that compare a prequestionnaire to a post-questionnaire are likely to underestimate the effects.^47–49 For that reason, we used a post-then-pre assessment. We based the content of the questionnaires on the course objectives, structure of the Guide,⁸ and a literature review,^33,50–52 as well as input from experts in communication skills training, survey methods, and monitoring and evaluation, who reviewed the questionnaire for content and face validity.

Clinician respondents

The training evaluation survey asked for system, site, discipline, and specialty; all surveys were anonymous and identified only by a unique ID.

Measures

Our main evaluation measure was clinicians' self-reported change in skills in serious illness communication after the training compared with before. These skills include: setting up a conversation, assessing illness understanding and information preferences, sharing prognosis, allowing silence, responding to emotion, exploring values and goals, and speaking less than half the time. We examined these measures in aggregate, by system, and for physicians and nonphysicians.

Additional measures included participants' rating of the quality of the training by clinician participants' report of the degree to which the training met course objectives and their assessment of the effectiveness of the training and likelihood of recommending it.

Qualitative questions

Open-ended questions sought clinicians' self-reported learning experiences as well as feedback about what could be improved with the course.

Statistical analysis

Participants' professional characteristics were presented as frequencies overall and broken down by system. Clinicians assessed themselves on a 5-point Likert scale (1 = not at all skilled, 2 = somewhat skilled, 3 = skilled, 4 = very skilled, 5 = extremely skilled) on the 13 key skills for carrying out a serious illness conversation using a structured Guide. The mean of each of the self-assessed skills was calculated post and pretraining and compared using a paired t test. This testing was done overall, stratified by system, and stratified by physician/nonphysician healthcare professionals. To correct for multiple comparisons, the Bonferroni correction was utilized. With 78 independent statistical tests performed, a p-value less than 0.0006 was considered statistically significant.

Qualitative analysis

We used thematic content analysis to qualitatively analyze responses to the open-ended questions on the questionnaire. We organized all responses into an excel spreadsheet. Using an open coding approach, two authors (A.K., J.P.) independently read ∼30% of responses to develop a preliminary codebook with agreed-upon categories and themes. A third author with extensive experience in qualitative research methods (J.S.) reviewed and helped refine the codebook. J.P. and A.K. subsequently used this codebook to independently categorize clinician responses and met regularly to iteratively refine the codebook. All discrepancies were resolved by consensus.

Ethics review

This educational evaluation was undertaken as a quality improvement initiative at the three institutions and as such was not submitted for Institutional Review Board review.

Results

Respondent characteristics

Twenty-two faculty trainers trained 297 frontline clinicians in serious illness communication in their respective health systems in the first year of implementation. Table 2 shows clinicians' characteristics for the cohort. With regard to health care setting, 33% (n = 97) came from System 1; 31% (n = 91) from System 2; and 37% (n = 109) from System 3. Of these clinicians, 49% (n = 145) were physicians, 35% (n = 104) were advanced practice clinicians, 12% (n = 36) were nurses, social workers, or chaplains, and the remaining were other health care professionals. This cohort included 48% (n = 142) nonpalliative care subspecialists, 28% (n = 83) primary care or family medicine, 17% (n = 51) palliative care, and 7.1% (n = 21) other.

Table 2.

Clinician Characteristics Overall and by System

	Overall	System 1	System 2	System 3
	n (%)	n (%)	n (%)	n (%)
Total clinicians	297	97 (33)	91 (31)	109 (37)
Clinical discipline
Physician	145 (49)	47 (48)	45 (49)	53 (49)
Advanced practice clinician	104 (35)	22 (23)	40 (44)	42 (39)
Registered nurse, social worker, or chaplain	36 (12)	22 (23)	4 (4.4)	10 (9.2)
Other or unknown	12 (4.0)	6 (6.2)	2 (2.2)	4 (3.7)
Specialty condensed
Subspecialty	142 (48)	45 (46)	15 (16)	82 (75)
Primary care and family medicine	83 (28)	9 (9.3)	73 (80)	1 (0.9)
Palliative care	51 (17)	29 (30)	1 (1.1)	21 (19)
Other/unknown	21 (7.1)	14 (14)	2 (2.2)	5 (4.6)
Clinical specialty (detailed)
Cardiology	12 (4.0)	10 (10)	2 (2.2)	0 (0.0)
Radiation oncology	12 (4.0)	1 (1.0)	0 (0.0)	11 (10)
Oncology	101 (34)	21 (22)	11 (12)	69 (63)
Geriatrics	11 (3.7)	9 (9.3)	2 (2.2)	0 (0.0)
Pediatrics	1 (0.3)	0 (0.0)	1 (1.1)	0 (0.0)
Family medicine	29 (9.8)	0 (0.0)	29 (32)	0 (0.0)
Primary care/internal medicine	53 (18)	9 (9.3)	43 (47)	1 (0.9)
Palliative care	51 (17)	29 (30)	1 (1.1)	21 (19)
Critical care/ICU	2 (0.7)	0 (0.0)	0 (0.0)	2 (1.8)
Pulmonary	2 (0.7)	2 (2.1)	0 (0.0)	0 (0.0)
Nephrology	2 (0.7)	2 (2.1)	0 (0.0)	0 (0.0)
Other/unknown	21 (7.1)	14 (14)	2 (2.2)	5 (4.6)

Change in clinician self-reported skills

For all clinician learners, participants' average self-rating of their skill increased from an average of 3.1 (skilled) before the training to 3.9 (very skilled) after the training; results achieved significance with p < 0.0001 across all skills. At System 1, participants' average self-rating of their skills improved from 2.9 (skilled) to 4.0 (very skilled); results achieved significance with p < 0.0001 for all skills. At System 2, participants' average self-rating of their level of skills (in aggregate) increased from an average of 3.2 to 3.6. More specifically, participants' average self-rating for two skills improved significantly (e.g., exploring views of family involvement and speaking <50% of the time) (p < 0.0001), but the improvements did not reach statistical significance for the remaining skills. At System 3, participants' average self-rating of their skills improved from 3.1 (skilled) to 4.0 (very skilled); results achieved significance with p < 0.0001 for all skills. See Figure 1 for more detail.

FIG. 1.

Clinicians' self-reported skills were measured on a 1–5 scale (1 = not at all skilled to 5 = extremely skilled).

For physicians, average self-rating of their skills improved from 3.2 (skilled) to 4.0 (very skilled), a difference of 0.8; results were significant with p < 0.0001 across all skills. For nonphysician healthcare professionals, participants' average self-rating of their skills improved from 3.0 (skilled) to 3.8 (very skilled), a difference of 0.8; results were significant with p < 0.0001 across all skills.

Participant training assessment

In each system, the vast majority of clinicians who responded (96%) mostly or fully agreed that the course objectives were met, 97% or more rated the training as mostly or extremely effective, and 96% or more reported that they were mostly or extremely likely to recommend the training to their colleagues. We also included an open-ended question about participant views of what could be done to improve the training, with comments reflecting the desire for more time in role play.

Qualitative results

We asked participants an open-ended question about the most important thing they took away from the training that they plan to apply in practice. Qualitative results were analyzed in aggregate across all systems. The response rate for this question was 85%, n = 253. Responses fell into four themes.

The first theme highlighted the benefits of the structure and language of the Serious Illness Conversation Guide. Clinicians reported that the Guide is “a more unified/complete framework to have those difficult conversations” (oncology 2–46) and the “scripting for [the] conversation [is] very helpful, especially with wish/worry language” (oncology 3–266). They highlighted “that the tool actually helps the conversation flow better” (palliative care 3–133) and allows them to “take time to appreciate fears and goals” (family medicine 2–112). Comments also focused on efficiency: “speed of the conversation is faster with scripting” (oncology 1–143). Other takeaways included: “prognostic information sharing statements are extremely helpful” (palliative care 3–285) and “prognosis in terms of function and uncertainty” (social work 3–217).

A second theme focused on concrete communication skills that enable patient-centered conversations. Clinicians described: “ask the patient to give the provider information instead of speaking all the time” (primary care 2–86) and “listen more, talk less” (pulmonary 1–59). They also reported learning: “how to initiate a serious conversation” (primary care 2–38) and “how to normalize the process” (radiation oncology 3–200). Other skills that clinicians identified included: “exploring patients' emotions” (geriatrics 1–1) and “allowing silence and reaffirming patients' emotions” (oncology 3–289).

A third theme focused on specific shifts in attitudes about these conversations, including the timing, importance, and content. Key takeaways included: “start [the] conversation earlier and revisit” (family medicine 2–195), “use this tool when patients are doing well” (oncology 3–239) and “the importance of having these conversations early on and letting them build upon another” (social work 3–218). They also reflected “how important the conversation is and can be; therefore I shouldn't let my own worries deter me” (oncology 3–277). Clinicians named a paradigm shift, from medically focused discussion to a more patient-centered approach: “Shut off the impulse to ‘fix’ and focus on truly accommodating pt. wishes” (cardiology 1–30) and “going through this guide with a focus on the questions in the guide, NOT trying to get a DNR” (oncology 1–148).

A fourth theme described greater self-efficacy and confidence in having conversations and intention of trying the tool in practice. Clinicians expressed this takeaway in multiple ways: “I can do this!” (oncology 3–204), “more comfort in approaching the discussion,” (hospitalist 1–81), and “feeling more comfortable and empowered to have these conversations with my patients” (primary care 2–128).

We asked participants what information in the training they found most surprising. One additional theme focused on emotional resonance and affective expression. Clinicians said: they were surprised “how much this hits home” (family medicine 2–129) and “my emotions as I was acting as the patient” (family medicine 2–95). Some noted: “the relationship with [the] patient can improve” (oncology 3–269) and “it was the inspiration to inquire from patients about their world and illness experience” (1–73, chaplain). A small number of clinicians reported that it was difficult to use a guide for these discussions “The guide- it is what we do but hard to implement naturally” (3–275, oncology).

Discussion

Our evaluation of this short, on-site clinician training course, delivered by faculty who completed a novel train-the-trainer program, demonstrates positive changes in clinicians' reported serious illness communication competencies, including knowledge, attitudes, and skills. Qualitative data revealed that after a single training session, clinicians reported benefits of having concrete language and a framework, positive attitudinal shifts, increased focus on patient-centered communication techniques, enhanced self-efficacy, and intention of trying the new tool in practice. Clinicians in all three systems reported improvements in self-rated skills after the training, reaching statistical significance for all measured skills in two of the three systems. The overall success of using a train-the-trainer model was further reinforced by high participant ratings of the training.

One of the strengths of this program is that it provided a standardized method of serious illness communication training that reached diverse groups of clinicians. Both physicians and nonphysician healthcare professionals reported benefitting from the training with significant improvements in skills, suggesting that one program can be used to train multiple disciplines. This is an important finding: to improve serious illness communication, we need to ensure that clinicians of all professional roles have the competencies for these conversations.⁵³ Our evaluation also revealed that this training is valued across specialties (e.g., primary care, palliative care, oncology, cardiology), which is essential to improve communication for all patients with serious illness.⁴

Some differences will require further evaluation. In System 2, trainers maintained the components of the training (e.g., skills practice) but adapted their methods. They used less individualized feedback, shortened the training, and were unable to use standardized patients for skills practice. They made these adaptations for numerous reasons. Since their setting is a community hospital with affiliated private practices and is not a teaching institution, they do not have access to or funding for standardized patients or patient actors. Time constraints for clinicians in community practice led to shortening the training to two hours. In this context, there were improvements in all self-reported skills, although the improvements were less marked than at the other two systems where trainings hewed more closely to the evidence-based model the trainers had been taught.^43,44,54 Additionally, the trained cohort in this system included mostly primary care clinicians compared with the other two system cohorts, which included predominantly subspecialists. While no conclusions can be drawn, this highlights the need to better understand how training can translate across different clinical groups and in particular those with different resources and staffing.

The training course produced changes in knowledge, attitudes, and skills that align with national recommendations for timely serious illness conversations.⁴ Earlier patient-centered conversations can have significant benefits for patients, including improvements in psychological symptoms, time to prepare for the future, and opportunities to make informed decisions.^46,55–58 A structured training program using a Conversation Guide appears to have catalyzed key shifts in clinical values, including recognition of the importance of the conversation, the need to start earlier, and the focus on patients' values and priorities rather than only on end-of-life procedures. Skills learned in this course may also enable clinicians and patients to safely navigate emotionally complex discussions, including: learning how to initiate a conversation, using wish/worry language for sharing prognosis, responding to emotion, listening more than talking, and strategies for clinicians to manage their own reluctance as well as patient reluctance and anxiety, key elements that can support shared decision making.^10,11,59

We interpret these findings to have implications for clinical education. In this model, a small number of mostly palliative care specialists trained large cohorts of their colleagues in serious illness communication within a relatively short timeframe, important given the known workforce shortage in palliative care.^60,61 The training was efficient, requiring two to three hours of clinicians' time, and focused on the use of a scalable tool with gentle, relatable language. The guide allowed clinicians to obtain important information more efficiently, which has implications in a busy clinical practice.^15,62 The qualitative learnings also revealed enhanced confidence and comfort in initiating conversations and an intention to try the guide in practice. Although we were able to observe significant self-reported improvements, we express caution with these findings, given that these are subjective measures and numerous systematic reviews have concluded that training alone is insufficient to drive lasting improvements in practice.^31,32 However, similar to checklists or decision aids,^63–65 a well-received tool has the potential to reinforce key concepts and support learning after training, helping to scale palliative care practices to more clinicians.

Our evaluation has several limitations. We used self-report measures, which may reflect participants' positive learning experiences or prior relationships with trainers rather than actual improvement in competencies. This evaluation would have been strengthened by having objective behavioral measures and/or educationally sensitive patient outcomes, such as audiorecording, observation, or patient surveys.^9,66 We examined trainers' use of the original training structure and methods using trainer self-report rather than direct observation. In some clinical settings in this analysis, training participants may represent early adopters who may be more positive and more willing to undergo a training; therefore, the results of this analysis may not apply to all clinicians. Finally, the connection to actual behavior change in serious illness communication is unknown through this evaluation. These three systems implemented coaching and system changes as part of their quality improvement efforts to support improvements in communication in practice, which were not analyzed in this evaluation.

This short training program, delivered through a standardized train-the-trainer course, produced improvements in clinicians' self-rated skills across different contexts, specialties, and disciplines. Although some differences are noted that require further evaluation, the streamlined and adaptable nature of this training suggests that this may be a scalable model to meet the goal of serious communication training for the large number of health care professionals across the health system who interact with patients with serious illness.

Footnotes

Acknowledgments

All authors would like to acknowledge the significant contributions of Alfreda Webb, MBA, DMS (Baylor Scott & White Health, Texas A&M University College of Medicine); Robert Fine, MD (Baylor Scott & White Health, Texas A&M University College of Medicine); Frank E. Osborn, MD (Lowell General Hospital); Judy Ling, BA (Lowell General Hospital); Sabine Clasen, MSN, RN, CHPN (Lowell General Hospital); Suzanne Trudel, RN, BSN, CHPN (Lowell General Hospital); Virginia Doherty, ANP-BC, ACHPN (Lowell General Hospital); and Pallavi Kumar, MD (Department of Medicine at the Perelman School of Medicine, University of Pennsylvania).

Authors would also like to acknowledge with gratitude the contributions of all of their palliative care and clinical colleagues in conducting the clinician training.

Funding Information

Supported by funding from Blue Cross Blue Shield of Massachusetts. The funder did not play a role in the design, execution, or writing of this work.

Author Disclosure Statement

No competing financial interests exist.

References

Singer

, Meeker

, Teno

, et al.: Symptom trends in the last year of life from 1998 to 2010: A cohort study. Ann Intern Med, 2015; 162:175–183.

Teno

, Gozalo

, Bynum

, et al.: Change in end-of-life care for Medicare beneficiaries: Site of death, place of care, and health care transitions in 2000, 2005, and 2009. JAMA, 2013; 309:470–477.

Wright

, Keating

, Ayanian

, et al.: Family perspectives on aggressive cancer care near the end of life. JAMA, 2016; 315:284–292.

Committee on Approaching Death. Institute of Medicine: Dying in America: Improving Quality and Honoring Individual Preferences Near the End of Life. Washington, DC: National Academies Press, 2015.

Wright

, Zhang

, Ray

, et al.: Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment. JAMA, 2008; 300:1665–1673.

Mack

, Weeks

, Wright

, et al.: End-of-life discussions, goal attainment, and distress at the end of life: Predictors and outcomes of receipt of care consistent with preferences. J Clin Oncol, 2010; 28:1203–1208.

Mack

, Cronin

, Taback

, et al.: End-of-life care discussions among patients with advanced cancer: A cohort study. Ann Intern Med, 2012; 156:204–210.

Bernacki

, Block

: Communication about serious illness care goals: A review and synthesis of best practices. JAMA Intern Med, 2014; 174:1994–2003.

Tulsky

, Beach

, Butow

, et al.: A research agenda for communication between health care professionals and patients living with serious illness. JAMA Intern Med, 2017; 177:1361–1366.

10.

Sanders

, Curtis

, Tulsky

: Achieving goal-concordant care: A Conceptual model and approach to measuring serious illness communication and its impact. J Palliat Med, 2018; 21:S17–S27.

11.

Street

Jr. , Makoul

, Arora

, Epstein

: How does communication heal? Pathways linking clinician-patient communication to health outcomes. Patient Educ Couns, 2009; 74:295–301.

12.

Sudore

, Lum

, You

, et al.: Defining advance care planning for adults: A consensus definition from a multidisciplinary Delphi Panel. J Pain Symptom Manage, 2017; 53:821–832.e1.

13.

The John

Hartford Foundation: Physicians' Views Toward Advance Care Planning Conversations. https://www.johnahartford.org/images/uploads/resources/ConversationStopper_Poll_Memo.pdf. (Last accessed May 30, 2019).

14.

Sullivan

, Lakoma

, Block

: The status of medical education in end-of-life care: A national report. J Gen Intern Med, 2003; 18:685–695.

15.

Chandar

, Brockstein

, Zunamon

, et al.: Perspectives of health-care providers toward advance care planning in patients with advanced cancer and congestive heart failure. Am J Hosp Palliat Care, 2017; 34:423–429.

16.

Keating

, Landrum

, Rogers

Jr , et al.: Physician factors associated with discussions about end-of-life care. Cancer, 2010; 116:998–1006.

17.

You

, Downar

, Fowler

, et al.: Barriers to goals of care discussions with seriously ill hospitalized patients and their families: A multicenter survey of clinicians. JAMA Intern Med, 2015; 175:549–556.

18.

Knauft

, Nielsen

, Engelberg

, et al.: Barriers and facilitators to end-of-life care communication for patients with COPD. Chest, 2005; 127:2188–2196.

19.

Keating

, Huskamp

, Kouri

, et al.: Factors contributing to geographic variation in end-of-life expenditures for cancer patients. Health Aff (Millwood), 2018; 37:1136–1143.

20.

Fallowfield

, Jenkins

, Farewell

, et al.: Efficacy of a Cancer Research UK communication skills training model for oncologists: A randomised controlled trial. Lancet, 2002; 359:650–656.

21.

Clayton

, Butow

, Waters

, et al.: Evaluation of a novel individualised communication-skills training intervention to improve doctors' confidence and skills in end-of-life communication. Palliat Med, 2013; 27:236–243.

22.

Back

, Arnold

, Baile

, et al.: Efficacy of communication skills training for giving bad news and discussing transitions to palliative care. Arch Intern Med, 2007; 167:453–460.

23.

Szmuilowicz

, el-Jawahri

, Chiappetta

, et al.: Improving residents' end-of-life communication skills with a short retreat: A randomized controlled trial. J Palliat Med, 2010; 13:439–452.

24.

Fallowfield

, Lipkin

, Hall

: Teaching senior oncologists communication skills: Results from phase I of a comprehensive longitudinal program in the United Kingdom. J Clin Oncol, 1998; 16:1961–1968.

25.

Wilkinson

, Leliopoulou

, Gambles

, Roberts

: Can intensive three-day programmes improve nurses' communication skills in cancer care?. Psychooncology, 2003; 12:747–759.

26.

Schell

, Green

, Tulsky

, Arnold

: Communication skills training for dialysis decision-making and end-of-life care in nephrology. Clin J Am Soc Nephrol, 2013; 8:675–680.

27.

Smith

, O'Sullivan

, Lo

, Chen

: An educational intervention to improve resident comfort with communication at the end of life. J Palliat Med, 2013; 16:54–59.

28.

Hales

, Hawryluck

: An interactive educational workshop to improve end of life communication skills. J Contin Educ Health Prof, 2008; 28:241–248; quiz 9–55.

29.

Bristowe

, Shepherd

, Bryan

, et al.: The development and piloting of the REnal specific Advanced Communication Training (REACT) programme to improve Advance Care Planning for renal patients. Palliat Med, 2014; 28:360–366.

30.

Gysels

, Richardson

, Higginson

: Communication training for health professionals who care for patients with cancer: A systematic review of training methods. Support Care Cancer, 2005; 13:356–366.

31.

Lord

, Clark-Carter

, Grove

: The effectiveness of communication-skills training interventions in end-of-life noncancer care in acute hospital-based services: A systematic review. Palliat Support Care, 2016; 14:433–444.

32.

Chung

, Oczkowski

, Hanvey

, et al.: Educational interventions to train healthcare professionals in end-of-life communication: A systematic review and meta-analysis. BMC Med Educ, 2016; 16:131.

33.

Sullivan

, Lakoma

, Billings

, et al.: Teaching and learning end-of-life care: Evaluation of a faculty development program in palliative care. Acad Med, 2005; 80:657–668.

34.

Sullivan

, Lakoma

, Billings

, et al.: Creating enduring change: Demonstrating the long-term impact of a faculty development program in palliative care. J Gen Intern Med, 2006; 21:907–914.

35.

Johansson

, Skeff

, Stratos

: A randomised controlled study of role play in a faculty development programme. Med Teach, 2012; 34:e123–e128.

36.

Widger

, Wolfe

, Friedrichsdorf

, et al.: National impact of the EPEC-pediatrics enhanced train-the-trainer model for delivering education on pediatric palliative care. J Palliat Med, 2018; 21:1249–1256.

37.

Stratos

, Katz

, Bergen

, Hallenbeck

: Faculty development in end-of-life care: Evaluation of a national train-the-trainer program. Acad Med, 2006; 81:1000–1007.

38.

Malloy

, Ferrell

, Virani

, Mazanec

: Promoting palliative care internationally: Building leaders to promote education, practice, and advocacy. J Pain Symptom Manage, 2018; 55:S140–S145.

39.

Ferrell

, Malloy

, Virani

: The end of life nursing education nursing consortium project. Ann Palliat Med, 2015; 4:61–69.

40.

Back

, Arnold

, Baile

, et al.: Faculty development to change the paradigm of communication skills teaching in oncology. J Clin Oncol, 2009; 27:1137–1141.

41.

Skeff

, Stratos

, Berman

, Bergen

: Improving clinical teaching. Evaluation of a national dissemination program. Arch Intern Med, 1992; 152:1156–1161.

42.

Bylund

, Brown

, di Ciccone

, et al.: Training faculty to facilitate communication skills training: Development and evaluation of a workshop. Patient Educ Couns, 2008; 70:430–436.

43.

Back

, Arnold

, Tulsky

, et al.: Teaching communication skills to medical oncology fellows. J Clin Oncol, 2003; 21:2433–2436.

44.

Fryer-Edwards

, Arnold

, Baile

, et al.: Reflective teaching practices: An approach to teaching communication skills in a small-group setting. Acad Med, 2006; 81:638–644.

45.

Lakin

, Koritsanszky

, Cunningham

, et al.: A systematic intervention to improve serious illness communication in primary care. Health Aff (Millwood), 2017; 36:1258–1264.

46.

Paladino

, Bernacki

, Neville

, et al.: Evaluating an intervention to improve communication between oncology clinicians and patients with life-limiting cancer: A cluster randomized clinical trial of the serious illness care program. JAMA Onc, 2019; 5:801–809.

47.

Levinson

, Gordon

, Skeff

: Retrospective versus actual pre-course self-assessments. Eval Health Prof, 1990; 13:445–452.

48.

Pratt

, McGuigan

, Katzev

: Measuring program outcomes: Using retrospective pretest methodology. Am J Eval, 2000; 21:341–350.

49.

Howard

, Dailey

: Response-shift bias: A source of contamination of self-report measures. J Appl Psychol, 1979; 64:144–150.

50.

Robinson

, Sutton

, von Gunten

, et al.: Assessment of the education for physicians on end-of-life care (EPEC) project. J Palliat Med, 2004; 7:637–645.

51.

Emanuel

, Ferris

, von Gunten

: EPEC. Education for physicians on end-of-life care. Am J Hosp Palliat Care, 2002; 19:17.

52.

VanGeest

: Process evaluation of an educational intervention to improve end-of-life care: The Education for Physicians on End-of-Life Care (EPEC) program. Am J Hosp Palliat Care, 2001; 18:233–238.

53.

Lakin

, Benotti

, Paladino

, et al.: Interprofessional work in serious illness communication in primary care: A qualitative study. J Palliat Med, 2019; 22:751–763.

54.

Branch

Jr. , Paranjape

: Feedback and reflection: Teaching methods for clinical settings. Acad Med, 2002; 77:1185–1188.

55.

Bernacki

, Paladino

, Neville

, et al.: Effect of the serious illness care program in outpatient oncology: a cluster randomized clinical trial. JAMA Intern Med, 2019; 179:751–759.

56.

Sudore

, Fried

: Redefining the “planning” in advance care planning: Preparing for end-of-life decision making. Ann Intern Med, 2010; 153:256–261.

57.

Steinhauser

, Christakis

, Clipp

, et al.: Factors considered important at the end of life by patients, family, physicians, and other care providers. JAMA, 2000; 284:2476–2482.

58.

Curtis

, Wenrich

, Carline

, et al.: Understanding physicians' skills at providing end-of-life care: Perspectives of patients, families, and health care workers. J Gen Intern Med, 2001; 16:41–49.

59.

Kon

: The shared decision-making continuum. JAMA, 2010; 304:903–904.

60.

Lupu

; American Academy of Hospice and Palliative Medicine Workforce Task Force: Estimate of current hospice and palliative medicine physician workforce shortage. J Pain Symptom Manage, 2010; 40:899–911.

61.

Morrison

, Augustin

, Souvanna

, Meier

: America's care of serious illness: A state-by-state report card on access to palliative care in our nation's hospitals. J Palliat Med, 2011; 14:1094–1096.

62.

Patel

, Periyakoil

, Moore

, et al.: Delivering end-of-life cancer care: perspectives of providers. Am J Hosp Palliat Care, 2018; 35:497–504.

63.

Weiser

, Haynes

, Lashoher

, et al.: Perspectives in quality: Designing the WHO Surgical Safety Checklist. Int J Qual Health Care, 2010; 22:365–370.

64.

Haynes

, Weiser

, Berry

, et al.: Changes in safety attitude and relationship to decreased postoperative morbidity and mortality following implementation of a checklist-based surgical safety intervention. BMJ Qual Saf, 2011; 20:102–107.

65.

Conley

, Singer

, Edmondson

, et al.: Effective surgical safety checklist implementation. J Am Coll Surg, 2011; 212:873–879.

66.

Kalet

, Gillespie

, Schwartz

, et al.: New measures to establish the evidence base for medical education: Identifying educationally sensitive patient outcomes. Acad Med, 2010; 85:844–851.