Effectiveness of Training Programs About a Palliative Care Approach: A Systematic Review of Intervention Trials for Health Care Professionals

Abstract

Background:

Palliative care (PC) training initiatives have proliferated to assist generalist health care providers (HCPs) develop skills for applying an early PC approach. To date, there is little synthesis of high-level evidence to review the content and effectiveness of these programs. To address this gap in knowledge, we conducted a systematic review of trials of training inventions to build PC competency in HCPs, according to PRISMA guidelines (PROSPERO registration no. 271741).

Materials and Methods:

We searched MEDLINE, Embase, PsycINFO, CINAHL, HealthSTAR, Web of Science, and the Cochrane Database of Systematic Reviews and Clinical Trials for studies published since 2000. Eligible studies were trials assessing PC training for HCPs. Interventions had to address at least two of six PC-related components, adapted from the National Consensus Project: identification or assessment; illness understanding; symptom management; decision making; coping; and referral. Two reviewers independently assessed articles for inclusion, using Rayyan, and extracted relevant data. Risk of bias was assessed using the Cochrane ROB2 or ROBINS-I tools.

Results:

Of 1209 articles reviewed, 22 studies met the inclusion criteria, with the majority being conducted in the United States (n = 9) or Europe (n = 8). Nearly all studies (n = 19) collected data through self-reported surveys; administrative (n = 4), clinical outcomes (n = 4), or interaction analysis (n = 6) data were also or solely used. Interventions featured didactic, skill-based training followed by role-play and/or individual coaching. Communication around illness understanding was the most taught PC component. Few interventions involved comprehensive PC training, with 12 studies representing 3 or less of the 6 framework components. Most studies (n = 16) reported a significant positive impact on at least one outcome, most often HCP self-reported outcomes.

Conclusions:

While many of these interventions demonstrated improved confidence among HCPs in the PC components taught, findings were mixed on more objective outcome measures. Further trial-based evidence is required on comprehensive PC training to help inform these interventions.

Introduction

Despite the clear benefits of palliative care (PC) for seriously ill patients, this care is often offered late in the illness trajectory or not at all.¹ Among health care clinicians and scholars, there is general consensus that PC should not be reserved for end of life, but be integrated with concurrent treatment.^2–4 This early integration, also referred to as a palliative approach to care, can be delivered by nonspecialist PC providers.⁵ As a result, there is a recognized need for PC education and training for generalist health care providers (HCPs).^6,7

Elements of a comprehensive PC approach have been defined by the National Consensus Project⁸ and operationalized as educational components in other interventions.⁹ These components include not only symptom management but also practical skills such as identifying and assessing patients, ensuring illness understanding, and supporting decision making. PC education and training initiatives have proliferated to assist diverse HCPs in developing their skills for applying a comprehensive PC approach.^10–13

While the development of many local and national PC training programs is encouraging, for example, the Gold Standards Framework (GSF) Training Program in the United Kingdom¹⁴ and Learning Essential Approaches to Palliative Care (LEAP) in Canada,¹¹ few systematic reviews of these educational programs exist, which could help inform widespread adaptation. Several reviews have examined high-level evidence of the impact of specialist-led PC programs, but not specifically on training interventions to build provider competency and capacity among non-PC specialist clinicians, for offering this care. Recent relevant reviews have focused on education for end-of-life communication only^12,15,16 or within specific populations, such as for preregistration nurses¹⁷ or in long-term care settings.¹⁸ To date, there has been limited synthesis of the educational components and findings from trial-evaluated interventions. To our knowledge, no current review exists that summarizes the effectiveness of comprehensive PC education programs for interprofessional HCPs.

To address this gap in knowledge, we conducted a systematic review of controlled trials of PC education or training interventions for interprofessional HCPs. Specifically, we wanted to examine teaching methods used and synthesize the evidence on the effectiveness of the programs on various outcomes.

Materials and Methods

Review aims and protocol registration

We conducted a systematic literature review to identify and describe trial studies of training interventions published since 2000. Our review was conducted in summer/fall 2021. The aim was to identify PC education and training initiatives for health care professionals and review: (1) the intervention content and delivery modes, and (2) the evidence of intervention effectiveness on provider and patient outcomes. The review is reported according to the PRISMA guidelines for systematic reviews¹⁹ and the protocol is registered with PROSPERO (reg CRD42022271741).

Search strategy

We searched the following seven bibliographic databases: MEDLINE, Embase, PsycINFO, CINAHL (Cumulative Index to Nursing & Allied Health Literature), HealthSTAR, Web of Science (Arts & Humanities Citation Index; Science Citation Index Expanded, Social Sciences Citation Index, Conference Proceedings Citation Indices), and the Cochrane Database of Systematic Reviews and Clinical Trials, for English language studies published from 2000 to April 2021.

We employed the following subject headings and related terms in our search: PC (including serious illness, hospice, etc.); Education/Training, Health Personnel (e.g., primary care, nurse, etc.); and Health Knowledge, Attitudes, Practice (including behavior, competence, etc.), all combined with the “AND” Boolean operator. Search strategies (hedges) for selecting Trials or Reviews were incorporated into the search algorithm to capture these types of studies (see online Supplementary Material for the full search strategy).²⁰ In addition, we hand-searched the reference lists of both original study articles and reviews identified.

Inclusion and exclusion criteria

Eligible studies were trials or quasi-experimental research (i.e., had a control group), assessing the impact of PC training for health care professionals. The intervention population comprised all HCPs, including health care students and post-graduate medical learners, in any setting. PC was defined as care of any illness that could require palliation or hospice care, which included advanced cancer or other chronic diseases or conditions that result in a steady decline in health and function, leading to death. Training was defined as an intervention involving interactive instruction in a formalized course of study for the acquisition of knowledge and skills. This education could range from early identification of patients requiring PC to end-of-life care.

Furthermore, for the content of the training program to be considered teaching a PC approach (as opposed to focusing only on pain management, for example), the intervention content had to contain a minimum of two PC components, these being: Identification or assessment; Illness understanding (including conversation); Symptom management; Decision making (including advance care planning); Coping (including caregivers); and Referral (including coordination/care planning). These six components were based on Jacobsen et al's⁹ guidelines for outpatient PC, which were originally adapted from the National Consensus Project for Quality Palliative Care.⁸ We excluded studies of interventions that had no interactive (in-person or virtual) component and those only intended to train patients or family caregivers.

Study selection

Studies included proceeded through an assessment of title, then abstract, and then full text screen, and were completed independently in duplicate by a minimum of two reviewers from the research team. All articles marked as possibly eligible by either reviewer passed to the next level of screening. Disagreements at the full text screen stage were resolved through discussion among the full review team. Article screening was managed on the Rayyan collaborative review platform.²¹

Data extraction and analysis

A data extraction spreadsheet with a standardized template was used to collect data by field (fields shown in Tables 1 and 3), including study design, intervention description, comparison group(s), data sources, and reported outcomes. Data from included study publications were extracted independently by two reviewers (D.B. and V.B.) in duplicate and then consolidated. The description of the intervention within each study was further examined for any mention of each of the six PC components being included in the content. We extracted and reported all study outcomes presented in all publications of included studies, along with inferential statistics for all significantly positive (p < 0.05), post-intervention comparisons between study arms. Study outcomes were grouped by prevalent theme for description and comparison across the trials.

Table 1.

Program Format and Palliative Care Components Covered in Education Interventions of Included Studies

Lead author, year, intervention name	Target professions	Objective	Intervention education format^a	Domains (D): D1. Identification or Assessment; D2. Illness Understanding (including conversation skills) D3. Symptom Management; D4. Decision Making (including ACP) D5. Coping (including caregiver); D6. Referral (including coordination or care planning)
Lead author, year, intervention name	Target professions	Objective	Intervention education format^a	D1	D2	D3	D4	D5	D6	Domains Covered
Thoonsen, 2019; 2015^42,43 RADboud university medical center indicators for Palliative Care needs tool (RADPAC) and training program	General practitioners	To increase early identification of patients requiring PC and to facilitate anticipatory, interdisciplinary PC planning	Group training session (5 hours); Individual coaching session through phone; 2 additional group sessions with simulated patients	•	•		•	•	•	5
Van den Block, 2020⁴⁶Palliative Care for Older People (PACE) Steps to Success Program	Nurse, care assistants	To improve quality of PC provision	Workshops (over 6 months); Facilitation (ongoing support and education) (over 4 months); Train-the-trainer approach	•		•	•	•	•	5
Bishop, 2019¹⁵ End-of-Life Nursing Education Consortium (ELNEC) project	Registered nurses (acute and ambulatory care units)	To increase competence in primary PC	Online modules (8) (1 hour per module); Workshops (3 hours), including case study discussions; Educational handouts		•	•	•	•		4
Chang, 2021²⁸ Quality End-of-Life Care for All (QELCA), Palliative Care Education and Practice (PCEP) training Program	Residents	To teach the concept and core principles of PC and develop these skills	Lectures (2 hours); Role-play (2 hours); Clinical practice (over 2 months)	•	•	•		•		4
Kinley, 2014³⁴ Gold Standards Framework for Care Homes (GSFCH) programme	Nurse managers	To facilitate the provision of quality end-of-life care	Workshops (over 9 months); Facilitation (ongoing support and education) (over 2 years); Train-the-trainer approach	•		•		•	•	4
Lamppu, 2019³⁶	Physicians, nurses	To improve quality of PC provision	Group training sessions (4 afternoons), including role-playing, discussion, and educational handouts		•	•	•	•		4
Okumura-Hiroshige, 2020³⁷End-of-Life Nursing Education Consortium-Geriatric (ELNEC-G)	Nurses	To improve attitudes and knowledge in PC provision	2-day workshop covering 9 modules, including role-play		•	•		•	•	4
Omidi, 2020³⁸	Oncology nurses	To improve perceptions of and clinical competency in providing end-of-life care to patients with cancer	3 workshops (30–45 minutes each), including educational handouts of materials covered		•	•		•	•	4
Slort, 2013⁴⁰ ACA (Availability, Current Issues and Anticipation) training program	General practitioners	To improve competence in PC communication	8-step training program over several months, consisting of facilitated education, discussion, role-play exercises, simulated patient consultation, and feedback; Educational booklet		•	•	•	•		4
Uslu-Sahan and Terzioglu, 2020⁴⁵ Interprofessional Gynecologic Oncology Palliative Care Training (IP Gyn-Onc PCT)	Interprofessional students (nursing, medical, nutrition-dietician, and social workers)	To increase gynecologic oncology PC knowledge	Lectures with case discussion (16 hours); patient simulations (30 minutes); Prebriefing and debriefing sessions (4 hours in total)		•	•		•	•	4
Curtis, 2011³⁰	Intensive care unit staff (nurses)	To improve intensive care unit (ICU) end-of-life care	Lectures, workshops, video presentations, hospital feedback, and academic detailing (all over 13 to 20 months)		•	•	•			3
Curtis, 2013²⁹	Internal medicine trainees and nurse practitioner students	To improve provider communication with seriously ill patients	8 workshops (32 hours total), including role-play and practice on simulated patients/family		•		•	•		3
Goelz, 2011³³ COM-ON-p (SPIKES/Oncotalk based)	Physicians	To improve competence in early PC communication	Workshop (11 hours), including role-play; individual coaching session (30 minutes); simulated patient session (1 hour)		•		•	•		3
Kruse, 2008³⁵ Teaching Toolkit for Nursing Excellence at End-of-Life Transition (TENEL) and End-of-Life Nursing Education Consortium (ELNEC) curriculum	Nurses	To change perceptions of end-of-life care	Lecture (3 hours); Individual facilitated clinical experience in hospice (3 hours); Post-experience discussion (1 hour)		•	•		•		3
Szmuilowicz, 2010⁴¹	Internal medical residents	To enhance performance of and confidence with end-of-life conversations	One-day workshop, including practice on simulated patient and feedback	•	•	•				3
Wilkinson, 2008⁴⁷ National Advanced Communication Skills Programme for Senior Health Care Professionals in Cancer Care Training (ACST)	Nurses	To increase awareness of end-of-life care communication skills and to explore strategies to implement these skills in practice	3-day workshop covering 9 modules, including role-play, taped demonstration of acquired skills; Train-the-trainer approach	•	•			•		3
Yeun, 2015⁴⁸ Palliative Care Professional Education (PCPE) program	Nurses	To increase PC knowledge and recognition of a good death	5 lectures (23 hours total), including case study discussion and visit to a hospice center		•	•		•		3
Bernacki, 2019²⁶; Paladino, 2019³⁹ Serious Illness Conversation Guide (SICG) training	Physicians and advanced-practice clinicians	To improve the occurrence, timing, quality, and accessibility of documented serious illness conversations	Clinical tools; clinical training (2.5 hours); system changes (SICG template in EMR); Coaching (in-person, email, or telephone based); Educational handouts		•		•			2
Brown, 2018²⁷ Codetalk	Internal medicine residents, medicine subspecialty fellows, nurse practitioner students, or advanced practice nurses	To improve competence in PC communication	Eight simulation-based workshops (4 hours each), including role-play		•		•			2
Epstein, 2017³¹ Values and Options in Cancer Care (VOICE) program	Medical oncologists	To promote patient-centered communication about disease course, prognosis, treatment decisions, and end-of-life care	2 office training sessions (1.75 hours); simulated patient session		•		•			2
Fischer, 2003³² Based on Education for Physicians on End-of-life Care (EPEC) materials	Interns and senior internal medicine residents	To increase palliative knowledge and decrease anxiety surrounding end-of-life care	4 workshops (1 hour each), including role-play		•		•			2
Thulesius, 2002⁴⁴	Home care staff (nurses, physiotherapists, assistant nurses, and care assistants)	To improve attitudes toward end-of-life care and create local guidelines for home care	Lectures, workshops, and small group work (26 to 100 hours total over 1 year, depending on profession)			•		•		2
Total studies				6	19	14	12	16	6
% of Studies				27.3	86.4	63.6	54.5	72.7	27.3

“Lectures” refers to didactic education (noninteractive); “Workshops” refer to education sessions with interactive demonstrations and group discussion.

EMR, electronic medical record; EPEC, Education in Palliative and End-of-Life Care Program; PACE, Palliative Care for Older People; PC, palliative care; SICG, Serious Illness Conversation Guide.

Study quality and risk of bias assessment

We assessed the risk of bias of the randomized controlled trials (RCTs) and cluster randomized controlled trials (cRCTs) using the Cochrane risk of bias tool (RoB 2), and for the nonrandomized trials and quasi-experimental studies, the ROBINS-I tool (Risk of Bias In Non-randomized Studies—of Interventions).^22,23 These tools include assessments of the randomization process, allocation concealment, deviations from intended interventions, completeness of outcome data, and potential selectivity of reporting.

Criteria items in each of the tool's domains were assessed independently in duplicate by two reviewers (D.B. and V.B.), rated as being at high, low, or uncertain risk of bias. We examined all published articles on the study and, where applicable, the trial registry. Studies were considered to be at overall high risk of bias if judged to be at high risk of bias in any domain; uncertain risk of bias if judged to be at uncertain risk of bias in any one domain (and none high risk); and at overall low risk of bias, if judged to be at low risk of bias in all domains.^24,25

Results

Study selection and characteristics

Our search strategy yielded 1209 unique articles, including original studies and review articles. A total of 50 review studies were identified, 33 of which were deemed relevant, and hand searched for original studies missed in the main search. Following preliminary screening of all original articles, 53 full-text original articles were examined and 22 found eligible (see PRISMA flow chart Fig. 1).^{12,14,15,26–48}

FIG. 1.

PRISMA flow diagram.

Description of studies

Most of the training interventions (n = 14) focused specifically on end-of-life care, while 2 studies focused on early PC (Table 1). Sixteen of the studies included were randomized controlled trials (3 cRCTs and 13 RCTs), along with 3 nonrandomized trials and 3 quasi-experimental studies. The studies were conducted in the United States (n = 9), Europe (n = 8), and Asia (n = 5). The control groups in 18 (82%) of the studies received no intervention, although these groups in 2 studies^28,40 received standardized training on procedures and protocols at the study setting. In the other four studies, the control groups were hybrids of the intervention, one study's control included access to online educational materials,^{26,34,35,39,45} and the other three studies^34,35,45 compared high versus low facilitation, with two or more control groups.

Description of interventions

The target audience of the educational interventions tended to be either nurses (n = 8) or physicians/residents (n = 7) (Table 1). One of these interventions focused solely on medical specialists.³¹ There were fewer interprofessional-based programs (n = 7), almost half of which (n = 3) involved student trainees. All interventions featured didactic, skill-based training, followed by role-play and/or individual coaching. The intervention duration for participants ranged from 4 hours to 20 months, with most totaling 1 to 2 days time, held over several weeks.

The interventions evaluated were diverse, 11 (50%) were based on previously published programs or guidelines, for example, the Gold Standards Framework for Care Homes (GSFCH), Serious Illness Conversation Guide (SICG), Education in Palliative and End-of-Life Care Program (EPEC), Palliative Care for Older People (PACE) Steps to Success Program, End-of-Life Nursing Education Consortium (ELNEC) project, and others. Given the wide range in educational approaches, content taught, outcomes measured, and methodological rigor among studies reviewed, it is difficult to draw any association between intervention format and effectiveness. Both one-day workshop programs⁴¹ and those that spanned over a year⁴⁴ demonstrated some significant effects.

Nearly all studies (n = 19) featured training on provider to patient “Communication,” including facilitating illness understanding, with “Coping,” including care for the family caregiver, as the next most common element (n = 16). In contrast, few interventions (n = 6) provided teaching on either “Identification/Assessment” or “Referral/Coordination.” Overall, none of the interventions represented training for comprehensive PC, which covered all 6 framework components, with 12 studies (55%) representing 3 or less of the components.

Study outcomes

To facilitate the analysis, we grouped all outcome measures into 5 main domains, and 22 subthemes (Table 2). The reviewers reached consensus to derive these domains based on the phenomena being measured in the reviewed studies. Almost all the trials (n = 19) collected data through self-reported surveys; 8 of these studies also assessed the interventions through other types of measures (Table 3). Administrative (n = 4), clinically assessed (n = 4), or interaction analysis data (n = 6) were used exclusively or in combination with other sources of data.

Table 2.

Palliative Care Outcome Domains Assessed and Significant Effects Among Included Studies

Outcome domain	Outcome subthemes	Positive findings among subthemes evaluated Number positive significance/total^a	Positive findings among domains evaluated Number positive significance/total^a
PC awareness	Attitude toward death/PC	5/7	7/10
	PC understanding	4/5
	PC competence	1/3
PC delivery	Quality of PC	1/3	2/4
	Care satisfaction	1/2
	Comfort/symptom management	0/1
Communication	Quality of communication	6/8	8/11
	Goal concordance	0/1
	Communication competence	3/4
	Provider rapport/efficacy	0/4
Well-being	Psychological well-being	1/6	2/6
	Quality of life	1/2
	Social functioning	0/1
	Survival	0/2
	Functional status	0/2
Care processes	Acute care use	0/3	3/6
	Aggressive treatment	0/2
	Clinical tool use	3/3
	Contact/home visits	1/1
	Interprofessional collaboration	0/2
	Needs assessment	1/1
	Place of death	0/2

p < 0.05.

Table 3.

Methods, Outcomes, and Assessed Risk of Bias of Included Studies

Lead author, year	Study type Country, setting	Comparison group(s), Ns	Data sources	Outcomes domains captured overall (positive significant findings in bold ^a )	Outcomes and findings Outcome subtheme (data source) (positive significant findings in bold ^a )	Risk of bias (ROB2 or ROBIN-1) assessment
Bernacki, 2019²⁶; Paladino, 2019³⁹ www.instituteforhumancaring.org/documents/Providers/Serious-Illness-Guide-old.pdf	cRCTUnited StatesDana-Farber Cancer Institute (DFCI) cancer center and 2 affiliated satellite clinics	Intervention: Clinicians n = 48Patients n = 184Caregivers n = 38Control (no intervention): Clinicians n = 43Patients n = 195Caregivers n = 26	Self-completed survey: patient and family caregiver; administrative data	CommunicationWell-beingCare processes	Provider rapport/efficacy (patients), goal concordance (family caregivers), quality of communication (administrative data)Psychological well-being (patients), survival (administrative data)Clinical tool use (administrative data)	Some concerns
Bishop, 2019¹⁵ https://www.relias.com/product/elnec-training	RCTUnited StatesLarge academic medical center in South Carolina	Intervention: n = 28Control (online modules only): n = 26	Provider self-completed survey	PC awareness	PC competence (nurses)	Some concerns
Brown, 2018²⁷	RCTUnited States, University of Washington and Medical University of South Carolina	Intervention: n = 232Control (no intervention): n = 240	Provider self-completed survey	Communication	Communication competence (providers)	Some concerns
Chang, 2020²⁸ https://www.stchristophers.org.uk/qelca https://pallcare.hms.harvard.edu/courses/pcep#:~:text=PCEP%20is%20designed%20for%20palliative,%2C%20teaching%2C%20and%20clinical%20practice.	RCTChinaDepartment of Internal Medicine of Beijing Chaoyang Hospital, Capital Medical University	Intervention: n = 36Control (no intervention): n = 36Note: all groups received standardized training, including training in humanistic medical skills and routine procedure through lecture, case discussion, and teaching	Medical interaction analysis (simulated patients); Provider self-completed survey	PC awareness Communication	Attitude toward death/PC (medical residents)Quality of communication (medical interaction analysis), quality of communication (medical residents), provider rapport/efficacy (medical interaction analysis)	Low
Curtis, 2011³⁰	RCTUnited StatesHospital ICUs in Washington	Intervention: n = 6 hospitalsPatients n = 669Nurses n = 624Control (no intervention): n = 6 hospitalsPatients n = 570Nurses n = 396	Provider self-completed survey; Administrative data	PC deliveryCare processes	Quality of PC (nurses, families), care satisfaction (families)Acute care use (administrative data), aggressive treatment (administrative data)	High
Curtis, 2013²⁹	RCTUnited StatesUniversity of Washington and Medical University of South Carolina	Intervention: n = 232Control (no intervention): n = 240	Self-completed survey: patients, families, and clinician assessors	PC deliveryCommunicationWell-being	Quality of PC (patients)Quality of communication (patients, families, clinician assessors)Psychological well-being (patients, families), functional status (patients)	Some concerns
Epstein, 2017³¹	RCTUnited States community and hospital-based cancer clinics	Intervention: Oncologists n = 19Patients n = 130Control (no intervention): Oncologists n = 19Patients n = 135	Medical interaction analysis (real patients); self-completed survey: Providers and Patients; Administrative data	CommunicationWell-beingCare processes	Quality of communication (medical interaction analysis), provider rapport/efficacy (physicians)Psychological well-being (patients), quality of life (patients), functional status (patients), social functioning (patients), survival (physicians)Aggressive treatment (administrative data), acute care use (administrative data)	Low
Fischer, 2003³²	Nonrandomized quasi-experimental studyUnited StatesUniversity of Colorado Health Sciences Center	Intervention: n = 30Control (no intervention): n = 40	Provider self-completed survey	PC awarenessCommunicationWell-being	PC understanding (medical residents)Provider rapport/efficacy (medical residents)Psychological well-being (medical residents)	Some concerns
Goelz, 2011³³	RCTGermanyUniversity Medical Center Freiberg	Intervention: n = 22Control (no intervention): n = 19	Medical interaction analysis (simulated patients)	Communication	Communication competence (medical interaction analysis)	Low
Kinley, 2014³⁴ https://www.goldstandardsframework.org.uk/cd-content/uploads/files/General%20Files/Prognostic%20Indicator%20Guidance%20October%202011.pdf	cRCTEnglandNursing homes	Intervention Group (high facilitation and action learning): n = 12 homesControl Group (high facilitation only): n = 12 homesObservation Group (local standard facilitation of GSFCH with no action learning): n = 14 homes	Administrative data; Qualitative focus groups	Care processes	Place of death (administrative data), clinical tool use (administrative data)	Some concerns
Kruse, 2008³⁵ https://www.aacnnursing.org/ELNEC https://www.rwjf.org/en/library/research/2009/08/online-courses-for-nurse-educators-and-nurses-on-end-of-life-car.html	Randomized quasi-experimental studyUnited StatesAll acute care areas in a large metropolitan health care system	Intervention: n = 27Control: Group A (read article on end of life) n = 27Group B (partial classroom intervention) n = 27	Provider self-completed survey	PC awareness	Attitude toward death/PC (nurses)	Some concerns
Lamppu, 2019³⁶	RCTFinlandNursing home and assisted living facility wards	Intervention: Staff n = 132Home residents n = 159Control (no intervention): Staff n = 48Home residents n = 181	Provider self-completed survey; Patient clinical data (nurse assessed); Qualitative analysis of free-text comments		No post-comparison data reported (providers, patient clinical data)	High
Okumura-Hiroshige, 2020³⁷	Nonrandomized controlled trialJapanHospitals and care facilities for older adults	Intervention: n = 164Control (no intervention): n = 174	Provider self-completed survey	PC awareness	Attitude toward death/PC (nurses), PC understanding (nurses)	High
Omidi, 2020³⁸	RCTIranShahid Bahonar Hospital Oncology Ward	Intervention: n = 35Control (no intervention): n = 35	Provider self-completed survey	PC awareness	Attitude toward death/PC (nurses), palliative care competence (nurses)	High
Slort, 2013⁴⁰	Nonrandomized controlled trialNetherlandsAn existing post-graduate two-year Palliative Care Peer Group Training Course (PCPT)	Intervention: n = 62Control (no intervention): n = 64Note: all participants were enrolled in Palliative Care Peer Group Training Course	Medical interaction analysis (simulated patients)	Communication	Quality of communication (medical interaction analysis)	Low
Szmuilowicz, 2010⁴¹	RCTUnited StatesBrigham and Women's Hospital	Intervention: n = 23Control (no intervention): n = 26	Medical interaction analysis (simulated patients); Provider self-completed survey	Communication	Quality of communication (medical interaction analysis), communication competence (medical residents)	Some concerns
Thoonsen, 2019; 2015^42,43	RCTNetherlandsCancer center	Intervention: n = 57Control (no intervention): n = 77	Provider self-completed survey (including reporting from medical records)	Care processes	Acute care use (physicians), place of death (physicians), clinical tool use (physicians), contact/home visits (physicians), needs assessment (physicians), interprofessional collaboration (physicians)	High
Thulesius, 2002⁴⁴	Nonrandomized controlled trialSwedenHome care	Intervention: n = 276Control (no intervention): n = 184	Provider self-completed survey	PC awarenessWell-being	Attitude toward death/PC (providers)Psychological well-being (providers)	Some concerns
Uslu-Sahan and Terzioglu, 2020⁴⁵	RCTTurkeyUniversity	Intervention: High Fidelity Simulation [HFS] and Hybrid Simulation [HS]) n = 24HS only n = 24HFS only n = 24Control (no HFS or HS): n = 24Note: all groups received interprofessional gynecologic oncology PC training	Provider self-completed survey	PC awarenessCare processes	PC understanding (providers)Interprofessional collaboration (providers)	Some concerns
Van den Block, 2020⁴⁶	cRCTBelgium, England, Finland, Italy, Netherlands, Poland, and SwitzerlandNursing homes	Intervention: n = 37 homesResident focused outcomes: Staff n = 426Patient relative n = 261Staff focused outcomes n = 1159Control (no intervention): n = 38 homesResident focused outcomes: Staff n = 558Patient relative n = 282Staff focused outcomes n = 1278	Self-completed survey: Providers and Patients	PC awarenessPC deliveryCommunication	Attitude toward death/PC (providers), PC understanding (providers), PC competence (providers)Quality of PC (patients), care satisfaction (relatives), comfort/symptom management (patients and relatives)Quality of communication (relatives)	Some concerns
Wilkinson, 2008⁴⁷	RCTEngland, Scotland, Wales, and IrelandHospices and community nursing services	Intervention: Nurses n = 85Patients n = 125^aControl (no intervention): Nurses n = 87Patients n = 125^aassumed, 250 total between arms	Medical interaction analysis (real patients); self-completed survey: providers and patients	PC deliveryCommunicationWell-being	Care satisfaction (patients)Quality of communication (medical interaction analysis), communication competence (nurses)Psychological well-being (patients), quality of life (patients)	Some concerns
Yeun, 2015⁴⁸	Randomized quasi-experimental studyKoreaHospital	Intervention: n = 42Control (no intervention): n = 44	Provider self-completed survey	PC awareness	Attitude toward death/PC (nurses), PC understanding (nurses)	Low

p < 0.05.

cRCT, cluster randomized controlled trial; GSFCH, Gold Standards Framework for Care Homes.

Most studies (n = 16) reported a significant positive impact on at least one outcome. In terms of the timing of follow-up assessments of provider participants, the majority of trials (n = 16) only examined short-term outcomes (less than or equal to three months post-intervention), recorded immediately or shortly after the intervention. Only one study reassessed providers through self-report beyond this time, at six months post-intervention.³⁷ The studies that utilized administrative data typically followed patients up to two years after the training intervention to capture the period before death. Coverage of the five main domains are as follows:

PC awareness

These outcomes denote the provider's familiarity with the concept of what constitutes PC, including self-perceived competence, understanding (clinical knowledge or conceptualization of what this care entails), or attitude toward death/PC. Overall, 10 studies measured this outcome domain, exclusively through provider self-report, 7 of which reported a significant positive impact of the training intervention.

PC delivery

These outcomes refer to behavior and practice resulting from proficiency in this care. Overall, four studies examined evidence of improved PC in practice.^29,30,46,47 Specific outcomes included patient (n = 4) and family (n = 1) ratings of PC quality, effectiveness of symptom management, or care satisfaction, including one RCT³⁰ where the nurses also assessed patients' quality of death and dying. Both Curtis RCTs^29,30 found no significant difference post-intervention, while in the other two studies, patients in this arm perceived higher quality of care⁴⁶ or reported higher care satisfaction.⁴⁷

Communication

These outcomes included measures of self-rated communication competence (n = 4) and/or quality of communication assessed through patient/family reporting (n = 2), medical record documentation (n = 1), or medical interaction analysis in simulated (n = 4)^28,33,40,41 or real (n = 2)^31,47 patient-based illness understanding conversations. Among the aspects of communication examined were breaking bad news, discussing PC, engaging patients in the conversation, and facilitating illness understanding.

Specifically, four studies examined provider to patient rapport and/or effectiveness of establishing goal concordance, through provider (n = 2), patient (n = 1), or family caregiver (n = 1) report, none of which demonstrated any significant improvement following the training intervention. In contrast, all six trials that assessed provider communication through interaction analysis demonstrated a significant positive effect on these outcomes compared to the control group. Significant improvements in self-rated communication competence were also reported in most (three of four) studies assessing this outcome, but no improvement in provider communication quality was reported from the patient and/or family perspective (n = 2).

Well-being

Aspects of well-being measured relate specifically to psychological, physical, and social states of patients or providers, including quality of life. In five studies, patient well-being was self reported by the patient (n = 3) and/or family member (n = 2). Two studies^26,31 that examined survival through provider report or administrative data and two studies^32,44 that examined provider psychological well-being through self report, found no effect of the training. The only studies reporting a significant impact in the well-being domain were by Bernacki et al²⁶ and Wilkinson et al,⁴⁷ both focused on communication skills training, which found that patients in the intervention group reported lower anxiety and depression,²⁶ and better general health,⁴⁷ respectively. There was no other evidence of improved well-being in patient reports and none from the perspectives of family members.

Care processes

These outcomes refer to incidence of specific care activities, use of clinical tools, occurrence of interprofessional collaboration and teamwork, and health care use, including hospital days and place of death. Most studies (four of six) examined these outcomes through administrative data. Both studies that measured provider assessments of care processes found no improvement in interprofessional collaboration^42,43,45 One of these studies^42,43 where data were derived through chart abstractions completed by the participating physicians on their patients, did find significant increases in clinical tool use, phone contacts with the patient, and needs assessments, but no change in acute care use, home visits, or place of death. In the RCT of the GSFCH program,³⁴ administrative data demonstrated an increased use of the Integrated Care Pathway at end of life, but not in other care tools.

Other process outcomes that were not impacted among studies were aggressiveness of treatment at end of life and patient hospice use.^30,31 The SICG training intervention study^26,39 found, through audit of the electronic medical record template implemented, significant increases in documented conversations (each relating to serious illness conversation, values or goals, prognosis or illness understanding, and life-sustaining treatment preferences, and earlier timing of conversation), compared to the control group (no intervention).^26,39 Overall, increased or improved care process outcomes were apparent in three studies,^{26,34,39,42,43} all assessed from administrative data.

Levels of training evaluation

We mapped the study outcomes onto the Kirkpatrick training evaluation model⁴⁹ to show confluence with the intervention objectives of improving practice. The Kirkpatrick model represents the hierarchical levels of evidence of intervention effectiveness, beginning with participant Reaction (e.g., satisfaction), through to Learning, Behavior, and finally, Results (e.g., impact of behavior change). As previously stated, the measures in most of the studies focused on Learning, assessing provider acquisition of understanding or knowledge.

Evidence at the Behavior level is shown in the participant's ability to apply knowledge, likened to exhibiting the behavior change in a structured examination setting. Again, the only studies that demonstrated this level of evaluation were those that used medical interaction analyses, in assessing PC communication competence or quality.^{28,31,33,40,41,47} Evidence of ongoing, real-world impact of behavior change was more elusive. Of the seven studies we reviewed that used measures of objective sustained change, including clinical tool use, patient reported interactions, and patient/family well-being, only four of these trials found any evidence of impact on these outcomes.^{26,34,39,46,47}

Risk of bias (across studies)

We judged 5 studies as low risk of bias, 12 as uncertain risk, and 5 as high risk. Overall, we were lenient in assessing risk of bias due to both the subjectivity of the self-reported measures used and participant awareness of their allocation. In applying the RoB 2 or ROBINS-I criteria, we took into consideration the inherent challenges of education research trials in blinding participants to training activities and measuring objective behavior changes.⁵⁰ Four of the studies rated as low risk of bias focused primarily on improving provider communication.^28,31,33,40

The assessors in all five studies that analyzed medical interactions were blinded to group allocation of the participants, reducing their risk of bias. For 6 (27%) of the studies, 40% or greater of training participants were lost to follow-up,^{15,29,30,36,37,42,43} a main reason for high risk of bias. Other reasons for studies being deemed as having a high risk of bias were not reporting any post-intervention comparison,³⁶ differences in treatment of study arms,³⁷ and being underpowered.³⁸

Discussion

Our review found evidence that education programs improved PC awareness, especially attitudes toward death and understanding of a PC approach, and communication skills, especially quality and competence. There was little evidence that the programs improved PC delivery, patient well-being, or care processes (with the exception of use of clinical tools): few studies measured these outcomes, of which many showed no significant difference. None of the programs covered all six components of a comprehensive PC approach; most covered only a few of these topics. Virtually all the programs used a multisession format, with a blended educational approach (e.g., didactic, discussion, role-playing, etc.), recognizing that changes in attitudes and behavior often take time and practice.

Several programs primarily focused on improving provider communication skills about serious illness. Many of these studies found evidence that the training improved self-perceived confidence among providers; however, the findings were mixed on more objective measures such as clinical outcomes and observer assessment. None of the trials assessed long-term outcomes beyond six months following intervention completion, with the exception of few studies that examined administrative or patient outcomes before death, retrospectively.

Moreover, even significant positive changes in provider communication found with simulated patients in evaluated role-play situations do not always translate into a change in behavior in real-world clinic settings.^51,52 Just over half of the seven studies that included objective measures of real-world behavior change found some evidence of impact. These results are encouraging, but also call for a closer examination of objective measures for assessing PC training interventions, as well as practical supports and change management processes that address systematic barriers, to facilitate knowledge translation into practice.

Other gaps we found in the evidence among the reviewed studies include the lack of patient-reported outcome measures. Also, pre-post self-report was common across all the studies, which is a less objective measure of impact. There was little objective evidence of behavior change, especially long-term behavior change. Furthermore, there was little consistency between studies making it difficult to compare and synthesize approaches and findings across interventions. For instance, although the studies tended to employ validated scales, there was a great deal of heterogeneity in survey instruments and outcome measures used between studies. They varied widely in intervention content, components of PC targeted, and provider audience. These factors complicated the ability to make any definitive statement about the effectiveness of these training initiatives, particularly regarding the care components taught.

Study limitations

Our review has some limitations. Our analysis of the program content is limited to the descriptions that the authors provided. Because of the blended educational approach, we were not able to isolate specific intervention features that produced the greatest impact. Also, we limited our review to controlled trials to summarize the highest level of evidence, but recognize that our synthesis misses noncontrolled, observational, or qualitative studies that may have implemented efficacious training interventions. Future reviews should examine the actual outcome tools used in high-level studies to support the use of common measures in subsequent studies. This is especially relevant in communication training interventions, as measures about provider behavior change were limited.

Implications and future directions

As indicated from this review and others, particularly those of communication training initiatives for PC-related or serious illness contexts,^12,15–18 education programs can help build generalists' capacity for providing a PC approach to their patients. This, in turn, could serve for greater and earlier access for patients who could benefit from this approach to care. Our review identified many promising examples of training interventions and undoubtedly more exist that have yet to be evaluated in a trial. Further work is required to use consistent measures of effectiveness. The few examples of interdisciplinary representation among the trials also warrant investigation, as holistic PC requires collaboration across diverse health care professions.⁵³ Moreover, future directions should target professional development interventions based on patient populations, clinical settings, and learner needs.

Conclusion

In our systematic review of 22 trials examining PC education programs for generalist health care professionals, we found the programs covered only some of the six components of a comprehensive PC approach. Many programs focused on improving provider communication toward end of life. There was evidence that the training led to higher quality conversations with standardized patients. While many of these interventions improved self-reported outcomes, such as competency and understanding, findings were mixed on objective outcome measures, such as those using clinical outcomes or observer assessment. Further trial-based evidence is required on comprehensive PC training, particularly with effectiveness to affect provider behavior change.

Footnotes

Acknowledgments

The authors would like to acknowledge Dr. Melissa Brouwers for her assistance with study development and guidance.

Authors' Contributions

D.B. contributed to study conceptualization, search of publications, article screening, data abstraction, quality assessment of studies, result interpretation, article drafting, and article revisions. H.S. contributed to study conceptualization, article screening, result interpretation, article drafting, and article revisions. V.B. contributed to search of publications, article screening, data abstraction, quality assessment of studies, results interpretation, and article revisions. J.M., D.M., and K.S. contributed to study conceptualization, article screening, result interpretation, and article revisions. All co-authors have reviewed and approved the article before submission. An abstract containing preliminary findings of this study was submitted as a poster to the 2022 European Association for Palliative Care Research Congress. The full-length article has been submitted solely to this journal and is not published, in press, or submitted elsewhere.

Funding Information

This work was funded by the Canadian Institutes of Health Research (CIHR) (Grant No. 408099). . The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the article.

Author Disclosure Statement

No competing financial interests exist.

Supplementary Material

References

Qureshi

, Tanuseputro

, Perez

, et al. Early initiation of palliative care is associated with reduced late-life acute-hospital use: A population-based retrospective cohort study. Palliat Med, 2019; 33(2):150–159; doi: 10.1177/0269216318815794

Evans

, Mackinnon

, Pereira

, et al. Integrating early palliative care into routine practice for patients with cancer: A mixed methods evaluation of the INTEGRATE Project. Psychooncology, 2019; 28(6):1261–1268.

Tassinari

, Drudi

, Monterubbianesi

, et al. Early palliative care in advanced oncologic and non-oncologic chronic diseases: A systematic review of literature. Rev Recent Clin Trials, 2016; 11(1):63–71.

Vanbutsele

, Van Belle

, Surmont

, et al. The effect of early and systematic integration of palliative care in oncology on quality of life and health care use near the end of life: A randomised controlled trial. Eur J Cancer, 2020; 124:186–193; doi: 10.1016/j.ejca.2019.11.009

Sawatzky

, Porterfield

, Lee

, et al. Conceptual foundations of a palliative approach: A knowledge synthesis. BMC Palliat Care, 2016; 15(1):1–4; doi: 10.1186/s12904-016-0076-9

Quill

, Abernethy

. Generalist plus specialist palliative care—Creating a more sustainable model. NEJM, 2013; 368(13):1173–1175.

Carey

, Zucca

, Freund

, et al. Systematic review of barriers and enablers to the delivery of palliative care by primary care practitioners. Palliat Med, 2019; 33(9):1131–1145.

Ferrell

, Twaddle

, Melnick

, et al. National consensus project clinical practice guidelines for quality palliative care guidelines. J Palliat Med, 2018; 21(12):1684–1689.

Jacobsen

, Jackson

, Dahlin

, et al. Components of early outpatient palliative care consultation in patients with metastatic nonsmall cell lung cancer. J Palliat Med, 2011; 14(4):459–464.

10.

, Chhabra

, Singh

. Palliative care education and its effectiveness: A systematic review. Public Health, 2021; 194:96–108; doi: 10.1016/j.puhe.2021.02.033

11.

Pereira

, Chary

, Faulkner

, et al. Primary-level palliative care national capacity: Pallium Canada. BMJ Support Palliative Care, 2021; 0:1–9; doi:10.1136/bmjspcare-2021-003036

12.

Selman

, Brighton

, Hawkins

, et al. The effect of communication skills training for generalist palliative care providers on patient-reported outcomes and clinician behaviors: A systematic review and meta-analysis. J Pain Symptom Manage, 2017; 54(3):404–416.e5.

13.

DeNofrio

, Verma

, Kosinski

, et al. Palliative care always: Hepatology—Virtual primary palliative care training for hepatologists. Hepatol Commun, 2022; 6(4):920–930.

14.

Shaw

, Clifford

, Thomas

, et al. Improving end-of-life care: A critical review of the Gold Standards Framework in primary care. Palliat Med, 2010; 24(3):317–329.

15.

Bishop

, Mazanec

, Bullington

, et al. Online end-of-life nursing education consortium core curriculum for staff nurses: An education strategy to improve clinical practice. J Hosp Palliat Nurs, 2019; 21(6):531–539.

16.

Chung

, Oczkowski

, Hanvey

, et al. Educational interventions to train healthcare professionals in end-of-life communication: A systematic review and meta-analysis. BMC Med Educ, 2016; 16(1):1–13; doi: 10.1186/s12909-016-0653-x

17.

Bassah

, Seymour

, Cox

. A modified systematic review of research evidence about education for pre-registration nurses in palliative care. BMC Palliat Care, 2014; 13(1):1–10; doi: 10.1186/1472-684X-13-56

18.

Anstey

, Powell

, Coles

, et al. Education and training to enhance end-of-life care for nursing home staff: A systematic literature review. BMJ Support Palliat Care, 2016; 6(3):353–361.

19.

Page

, Moher

, Bossuyt

, et al. PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. BMJ, 2021; 372:n160; doi: 10.1136/bmj.n160

20.

McMaster University, Health Information Research Unit. Search filters for MEDLINE in Ovid Syntax and the PubMed translation; 2022. Available from: https://hiru.mcmaster.ca/hiru/hiru_hedges_medline_strategies.aspx [Last accessed: October 21, 2022].

21.

Ouzzani

, Hammady

, Fedorowicz

, et al. Rayyan—A web and mobile app for systematic reviews. Syst Rev, 2016; 5(1):1–10; doi: 10.1186/s13643-016-0384-4

22.

Higgins

, Altman

, Gøtzsche

, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ, 2011; 343:d5928; doi: 10.1136/bmj.d5928

23.

Sterne

JAC

, Higgins

JPT

, Elbers

, et al. Risk of bias in non-randomized studies of interventions (ROBINS-I): Detailed guidance. 2016. Available from: https://www.msg.ugent.be/wp-content/uploads/2019/10/ROBINS-I_detailed_guidance.pdf [Last accessed October 21, 2022].

24.

Higgins

, Thomas

, Chandler

, et al. Cochrane handbook for systematic reviews of interventions version 6.2; 2019. Available from: www.training.cochrane.org/handbook [Last accessed October 21, 2022]

25.

Sterne

, Savović J, Page MJ, et al. RoB 2: A revised tool for assessing risk of bias in randomised trials. BMJ, 2019; 366:l4898; doi: 366 10.1136/bmj.l4898

26.

Bernacki

, Paladino

, Neville

, et al. Effect of the serious illness care program in outpatient oncology: A cluster randomized clinical trial. JAMA Intern Med, 2019; 179(6):751–759.

27.

Brown

, Back

, Ford

, et al. Self-Assessment scores improve after simulation-based palliative care communication skill workshops. Am J Hosp Palliat Med, 2018; 35(1):45–51.

28.

Chang

, Qi

, Jiang

, et al. The impact of palliative care education and training program on the resident physicians. Ann Palliat Med, 2021; 10(3):2758–2765; doi: 10.21037/apm-20-1625

29.

Curtis

, Back

, Ford

, et al. Effect of communication skills training for residents and nurse practitioners on quality of communication with patients with serious illness: A randomized trial. JAMA, 2013; 310(21):2271–2281.

30.

Curtis

, Nielsen

, Treece

, et al. Effect of a quality-improvement intervention on end-of-life care in the intensive care unit: A randomized trial. Am J Respir Crit Care Med, 2011; 183(3):348–355.

31.

Epstein

, Duberstein

, Fenton

, et al. Effect of a patient-centered communication intervention on oncologist-patient communication, quality of life, and health care utilization in advanced cancer: The VOICE randomized clinical trial. JAMA Oncol, 2017; 3(1):92–100.

32.

Fischer

, Gozansky

, Kutner

, et al. Palliative care education: An intervention to improve medical residents' knowledge and attitudes. J Palliat Med, 2003; 6(3):391–399.

33.

Goelz

, Wuensch

, Stubenrauch

, et al. Specific training program improves oncologists' palliative care communication skills in a randomized controlled trial. J Clin Oncol, 2011; 29(25):3402–3407.

34.

Kinley

, Stone

, Dewey

, et al. The effect of using high facilitation when implementing the Gold Standards Framework in Care Homes programme: A cluster randomised controlled trial. Palliat Med, 2014; 28(9):1099–1109.

35.

Kruse

, Melhado

, Convertine

, et al. Evaluating strategies for changing acute care nurses' perceptions on end-of-life care. Am J Hosp Palliat Med, 2008; 25(5):389–397.

36.

Lamppu

, Laurila

, Finne-Soveri

, et al. Training nursing home staff to improve residents' end-of-life care: Design and baseline findings from a randomized controlled trial. Eur Geriatr Med, 2019; 10(4):649–657.

37.

Okumura-Hiroshige

, Fukahori

, Yoshioka

, et al. Effect of an end-of-life gerontological nursing education programme on the attitudes and knowledge of clinical nurses: A non-randomised controlled trial. Int J Older People Nurs, 2020; 15(3):e12309; doi: 10.1111/opn.12309

38.

Omidi

, Dehghan

, Shahrbabaki

. Effectiveness of a traditional training method in increasing long-term end-of-life care perception and clinical competency among oncology nurses: A pilot clinical trial. Asia Pac J Oncol Nurs, 2020; 7(3):287–294.

39.

Paladino

, Bernacki

, Neville

, et al. Evaluating an intervention to improve communication between oncology clinicians and patients with life-limiting cancer: A cluster randomized clinical trial of the Serious Illness Care Program. JAMA Oncol, 2019; 5(6):801–809.

40.

Slort

, Blankenstein

, Schweitzer

, et al. Effectiveness of the ACA (Availability, Current issues and Anticipation) training programme on GP-patient communication in palliative care; a controlled trial. BMC Fam Pract, 2013; 14(1):1–10; doi: 10.1186/1471-2296-14-93

41.

Szmuilowicz

, El-Jawahri

, Chiappetta

, et al. Improving residents' end-of-life communication skills with a short retreat: A randomized controlled trial. J Palliat Med, 2010; 13(4):439–452.

42.

Thoonsen

, Gerritzen

SHM

, Vissers

KCP

, et al. Training general practitioners contributes to the identification of palliative patients and to multidimensional care provision: Secondary outcomes of an RCT. BMJ Support Palliat Care, 2019; 9(1):e18.

43.

Thoonsen

, Vissers

, Verhagen

, et al. Training general practitioners in early identification and anticipatory palliative care planning: A randomized controlled trial. BMC Fam Pract, 2015; 16(1):1–12.

44.

Thulesius

, Petersson

, et al. Learner-centred education in end-of-life care improved well being in home care staff: A prospective controlled study. Palliat Med, 2002; 16(4):347–354.

45.

Uslu-Sahan

, Terzioglu

. Interprofessional simulation-based training in gynecologic oncology palliative care for students in the healthcare profession: A comparative randomized controlled trial. Nurse Educ Today, 2020; 95:104588; doi: 10.1016/j.nedt.2020.104588

46.

Van den Block

, Honinx

, Pivodic

, et al. Evaluation of a palliative care program for nursing homes in 7 countries: The PACE cluster-randomized clinical trial. JAMA Intern Med, 2020; 180(2):233–242.

47.

Wilkinson

, Perry

, Blanchard

, et al. Effectiveness of a three-day communication skills course in changing nurses' communication skills with cancer/palliative care patients: A randomised controlled trial. Palliat Med, 2008; 22(4):365–375.

48.

Yeun

, Kwon

, Lee

. Development and evaluation of an education program for professional palliative care nursing. J Korean Acad Nurs, 2015; 45(1):139–146.

49.

Kirkpatrick

, Kirkpatrick

. Kirkpatrick's four levels of training evaluation. Association for Talent Development: Alexandria, VA; 2016.

50.

Sullivan

. Getting off the “gold standard”: Randomized controlled trials and education research. J Grad Med Educ, 2011; 3(3):285–289.

51.

Wilbur

, Elmubark

, Shabana

. Systematic review of standardized patient use in continuing medical education. J Contin Educ Health Prof, 2018; 38(1):3–10.

52.

Williams

, Song

JJY

. Are simulated patients effective in facilitating development of clinical competence for healthcare students? A scoping review. Adv Simul (Lond), 2016; 1(1):1–9; doi: 10.1186/s41077-016-0006-1

53.

Bainbridge

, Brazil

, Krueger

, et al. A proposed systems approach to the evaluation of integrated palliative care. BMC Palliat Care, 2010; 9(1):1–12; doi: 10.1186/1472-684X-9-8

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB