Deliberate Practice with Standardized Patient Actors and the Development of Formative Feedback for Advance Care Planning Facilitators

Abstract

Objective:

Multimodal curricular assessment after adding standardized patient (SP) actor-based simulation to an advance care planning (ACP) facilitator training course and development of a formative feedback tool.

Background:

ACP represents a highly valued service requiring more and better trained facilitators.

Methods:

Participants were primarily nurses and social workers in a large multisite health system. The course included a precourse video demonstration of ACP, traditional lectures, and four 30-minute simulations with SPs. Knowledge was tested with a multiple choice question (MCQ) test. In addition to standard postcourse/postsimulation evaluations, learners were surveyed pre/post/30–90 days delayed for self-perceived confidence. A linear mixed-effects model was used to analyze changes over time. Trained faculty rated performance in simulations with an observational mini-clinical examination (mini-CEX)-type rating form with a checklist, global competency, and global communication rating. Inter-rater reliability (IRR) was calculated on randomly selected paired ratings.

Results:

Sixty-seven individuals consented to participate. MCQ scores improved from 83% ± 10% to 92% ± 8% (p < 0.001). Paired learner surveys of self-confidence across six domains were available for 65 pre, 65 post, and 40 delayed with a mean positive change on a 0 to 10 point scale from pre–post (2.32 ± 1.65; p < 0.001) and predelayed (2.34 ± 1.96; p < 0.001) time frames. For the faculty observation ratings of simulation performance, the average raw agreement for critical actions was 82% and IRR was 0.71.

Conclusions:

Learner feedback and self-assessment suggest that actor-based simulation contributed to improved confidence in conducting ACP. The mini-CEX observation form is adequate for formative feedback, with further testing needed to make judgments of competence.

Introduction

Advance care planning (ACP) is recognized by healthcare professionals, patients, and loved ones as a means of navigating complex health decisions. ACP is a facilitated discussion with a patient and/or family members with the goals of discussing complex medical decisions, understanding the risks and benefits of those decisions, sharing preferences with loved ones, and documenting choices in advanced directive documents.¹ ACP discussions cover many difficult topics, including selecting a surrogate decision maker, and deciding on continued curative versus palliative care options. ACP fosters awareness and respect for the patient's goals and values and may thus alleviate the distress of surrogate end-of-life decision makers.² Additionally, it improves the quality of end-of-life care, increases patient and family satisfaction with care,^2–4 and decreases the chance of unwanted treatments.⁵

Realizing the value and importance of ACP, Medicare began allowing providers to bill for this service in 2016.⁶ Previously, it was primarily trained nurses, social workers, and other trained facilitators, in consultation with providers, who provided this vital service. Greater access to ACP has been called for explicitly by the Institute of Medicine⁷ and has been identified as a public health issue.^8,9 However, many patients do not have access to ACP.^10,11 Vulnerable patients with serious chronic illnesses would like to better understand disease diagnosis, prognosis, and treatment options, as would patients' loved ones.^10–15 Data from a 2007 national survey showed 88% of discharged hospice care patients, 65% of nursing home patients, and only 28% home health patients had an advance directive document on record.¹⁶ Additionally, multiple studies identify barriers such as inadequate training of personnel, lack of familiarity with documentation, and insufficient time for ACP completion.^17–19 Providers are often uncomfortable with end-of-life discussions, suggesting the need for training on discussing difficult topics.^20–22 Clearly, effective training and the opportunity for deliberate practice with feedback²³ are needed for facilitators of ACP.

While didactic training exists for those facilitating ACP,^24,25 little simulation-based learning exists, and there are few documented examples of deliberate practice with feedback. Detering et al. demonstrated that a multimodal interactive approach (which included e-simulation recordings) to ACP education increased doctor confidence and performance on ACP.²⁴ Chan et al. included the use of actor-based simulation in a curriculum for resident physicians and successfully increased the comfort level of this group performing ACP.²⁶ Regarding assessment, the use of miniclinical examination (mini-CEX) observation tools, typically done in simulated or bedside settings, has proven useful for evaluating competency with interviewing and counseling skills,^27–30 and Ellman et al. recently developed a mini-CEX in palliative care with medical students interacting with standardized patient (SP) actors and achieved good inter-rater reliability (IRR).³¹

We incorporated simulation with SP actors into our existing ACP facilitator training course known as Care Decisions™. We then implemented a multimodal assessment of the curriculum, including learner evaluations, knowledge testing, and pre/post/delayed self-assessed surveys of confidence and competence. We adapted and refined an observational tool for formative feedback derived from a mini-CEX tool originally designed for graduate medical education and we share our experience with tool development.

Methods

IRB statement

We received IRB approval and collected course data from May 2015 to May 2016. Two pilot courses helped refine the simulation cases, assessments, and evaluations and took place in January/February 2015.

Setting

The course took place at a large healthcare simulation center serving a health system and a regional medical campus. The health system is a large, multihospital, multiclinic health system, with rural, suburban, and urban sites.

Participants

Participant learners were primarily internal to the health network and volunteered for training in the simulation-augmented ACP facilitator course. The course is required of our facilitators as initial training and retraining is required if they conduct less than 10 facilitations/year. Participants were invited through a precourse email, a course day explanation of the project, and written consent was obtained on the day of the course.

Actor training

A SP coordinator trained the actors, including general and course-specific orientation processes. We conducted two pilot courses during which the SP coordinator and course coordinator debriefed and gave performance feedback to the actors. The majority of the four simulations used a stable cadre of the same four actors, with substitutes used for two scenarios on occasion, thus a total of six actors. Actors were trained to portray scenarios with challenging emotional content, including a patient in pain, existential distress, and fear of the burden of caring for a loved one.

Course day

Curricular elements were identified by literature review, content expert consensus discussions, and similar recommendations of a nationally respected program.²⁵ The course included training in the appropriate knowledge, skills, and attitudes to facilitate ACP discussions. All learners are exposed to an 18-minute precourse video³² through an eLearning platform that explains the ACP process, its value, and demonstrates several conversations. Precourse written materials included the ACP discussion guide (a two-page pdf document that summarizes their primary task) and information given to patients to prepare them for ACP. Participants were allowed to continue whether or not they had watched the video. The course day begins with the multiple choice pretest. The following topics are then covered through traditional lecture: advance care planning needs and ethical framework, the ACP model, practitioner orders for life-sustaining treatment, and legal considerations, including the power of attorney for healthcare form. Minimum enrollment was 8 and maximum 16 participants per course. This led to simulation group sizes of two to four participants. Each learner had one to two opportunities to directly interact with the SP actor, while the learner observers used an anonymous observation checklist form purely to help keep them engaged (Supplementary Appendix 1; Supplementary Data are available online at www.liebertpub.com/jpm). Each simulation allows 3 minutes in the hall to read over the learning objectives and focus of the discussion, 12 minutes to interact with the actor, then 10 minutes to debrief.

Pre–post multiple choice question test

A 24-question, pre–post multiple choice question (MCQ) test was conducted at the beginning and the end of the day. Content was sampled from the curriculum to be representative of the covered topics. The questions were vetted by the faculty who are content experts in palliative care. The test was trialed during pilot courses, and several questions were revised or replaced for the final version. The participants were made aware of the correct answers with explanations after the pretest.

Survey of self-perceived confidence and competence

We conducted a pre/post/delayed learner survey of self-perceived confidence and competence in the ACP process. This was administered at the beginning and end of the course day and after a delay of 30–90 days (via email) to allow them to gain experience. Surveys were administered through survey software (Qualtrics© 2015, Qualtrics, Provo, UT) and took ∼5 minutes to complete. The survey presented items scaled from 0 to 10, with anchors at 0 = no confidence, 5 = somewhat confident, and 10 = completely confident. We anticipated that learners would begin with a moderate degree of confidence and considered a two-point change (roughly equivalent to a 2 cm change on a visual analog scale) to be significant. We estimated that 30 learners with paired observations were needed to see changes across time. The survey also contains a self-assessment ability to perform independently with the anchors: needs further instruction, competent with close supervision, competent with minimal supervision, and competent to perform independently. A one-category step change on this scale was considered significant.

Faculty rater observational tool used in simulation

With permission, we adapted a mini-CEX tool developed for internal medicine residents by Han et al.³³ Local palliative care physician and nurse experts (R.S., A.M.F., and L.S.F.) adapted and revised the tool for our audience during the pilot courses. The mini-CEX tool has 13 observation points, which should be demonstrable in any ACP discussion scored as observed versus not observed and five additional observation points that had a not applicable option, plus two to three case-specific items rated observed or not observed. At the end of the rating checklist (CL), we chose to place a global rating of competency (GRSCOMP) consistent with the entrustable professional activities literature.³⁴ We felt this more appropriate for novices than the competency framework of Dreyfus,³⁵ which includes proficiency and mastery. The scale used the anchor needs further instruction, competent with close supervision, competent with minimal supervision, and competent to perform independently. The form also had a global rating of communication (GRSCOMM) in the context of ACP rated from unsatisfactory, satisfactory, good, very good, and outstanding. This was included in an attempt to elicit whether high ratings of competency correlated with good communication skill or more closely paralleled accomplishing checklist tasks. We included a comment section for facilitator's strengths and another on areas for improvement. An example of the formative observational assessment is attached (Supplementary Appendix 2). All assessments and evaluations were conducted using survey software (Qualtrics© 2015; Qualtrics, Provo, UT).

Rater training

Five of the raters (W.F.B., H.C.G., A.M.F., L.S.F., and R.S.) had participated in developing the observational tool and the simulation cases. Additional raters were trained using video of past courses or by live secondary consensus rating. Consensus discussions were held during debriefing sessions for the first two pilot courses in an effort to improve IRR. First ratings were live, and second ratings were done through video review.

Debriefing process

Faculty observed the scenario and rated from behind a two-way glass. Ratings were not shown to participants. Rather, the critical actions were noted by the faculty and drove qualitative formative feedback shared verbally. Two of the debriefers were palliative care experts (L.S.F. and R.S.), other debriefers were clinical educators who had taken a faculty development course on debriefing techniques. Debriefers used the debriefing with good judgment method^36,37 and had scripted debriefing questions as a starting point for discussion. Actors were invited to comment in the first few minutes of debriefing. Observing learners were also engaged. In addition to the ACP task of clarifying wishes, learners were encouraged to reflect on both the emotions they perceived in the patient actor and emotions invoked in themselves.

Postsimulation session and postcourse feedback

A postsimulation session evaluation form (simulation case specific) was given at the end of the course day. Participants evaluated only the cases in which they directly interacted with the SP actor. In addition, a standard continuing education evaluation focused on learning objectives was conducted.

Postcourse follow-up

The ACP discussion record is stored in the electronic health record. The course also demonstrates a separate expectation that facilitators log their ACP sessions through a secure web-based system for meeting individual requirements and programmatic tracking. We used this log to see if learners from our studied courses were logging ACP as evidence of using their education in practice.

Data analysis

The MCQ pre–post test of knowledge was analyzed using a generalized linear mixed model, with a logit link, to evaluate the effects of training on performance. The dependent variable was binary, correct, or incorrect for each question, subject and question were included as random factors to account and control for by subject and by question variability (random intercept model). In addition, a paired t test was used to assess simple pre–post change.

A linear mixed-effects model was used to determine the effects of training on learners self-confidence ratings in the six competency areas. Change in self-competency ratings was assessed using the Kruskal–Wallis rank sum test. Post hoc pairwise Wilcoxon rank sum test with a Bonferroni correction was used to evaluate change in rating over time.

For faculty ratings of simulation-based performance, we assessed IRR with the Gwet's AC1 kappa statistic for multiple raters and raw agreement for critical actions (CL). A random sample of 50% of performances was chosen to perform a second rating. Global rating IRRs were calculated using the Gwet's AC2, with linear weights chosen as 1 for perfect agreement, 0.75 for adjacent agreement, 0.25 for the next agreement, and 0 thereafter.

In an attempt to see if competency was changing over the course of the day, we looked for correlation between run order and GRSCOMP, although this represented different learners at different time points. We also looked for correlation of past ACP experience (EXP) with GRSCOMP on the first simulation of the day and any simulation performance. We considered a one-point change in GRSCOMP or a two-point change in CL score to be significant. Data were analyzed using R (open source, version 3.0.2).

Results

A total of 67/83 individuals consented to participate during seven courses over 13 months ending May 2016. Demographics and experience (EXP) are listed in Table 1.

Table 1.

Participant Demographics

Enrolled/consented	N = 67
Role
Nurse	41
Social worker	16
Advanced practice nurse	7
Physician	1
Counselor	1
RN case manager	1
Female	66
Male	1
Age (years) mean	39.2 ± 10 (range 23–62)
Reported experience with ACP
“I have never seen ACP”	N = 14
“I have observed ACP”	N = 29
“I have conducted ACP with supervision”	N = 5
“I have conducted ACP independently”	N = 17
No response	N = 2

ACP, advance care planning.

Training (both didactic and simulation as one variable) was found to have a significant positive effect on the likelihood of a correct response on the MCQ test (b = 1.01, SE = 0.12, z = 8.26, p < 0.001). Paired pre–post MCQ scores improved from a mean prescore of 83% ± 10% to a post score of 92% ± 8% (t(65df) = 8.29 p < 0.001).

Paired surveys of self-confidence across six domains were available for 65, 65, and 40 learners, respectively, with all competency areas showing a significant positive association from pre–post (Δ2.32 ± 1.65, b = 2.26, SE = 0.31, t = 7.4, p < 0.001) and predelayed time frame (Δ2.34 ± 1.96, b = 2.49, SE = 0.41, t = 6.52, p < 0.001; Fig. 1). Self-perceived competence in discussing values and goals improved over time with 57% pre, 92% post, and 95% delayed choosing minimal supervision or independent (Kruskal–Wallis rank sum chi-square (2df) = 35.74, p < 0.001; Fig. 2). Self-perceived competence with advance directives improved with 48% pre, 86% post, and 92% delayed choosing minimal supervision or independent (Kruskal–Wallis rank-sum chi-square (2df) = 37.90, p < 0.001; Fig. 3). (Note: df, degrees of freedom.)

FIG. 1.

Mean learner self-rating of confidence in six content areas, with delayed responses occurring at 30–90 days. N = 65, 65, and 40 learners at pre, post, and delayed time points, respectively.

FIG. 2.

Self-ratings of competence in having discussions about values and goals over time. Delayed rating was at 30–90 days postcourse. N = 65, 65, and 40 learners at pre, post, and delayed time points.

FIG. 3.

Self-ratings of competence in having discussions about advance directives. Delayed rating was at 30–90 days postcourse. N = 65, 65, and 40 learners at pre, post, and delayed time points.

There were 102 simulations by consented participants (of 112 total) and we randomly selected 56 paired ratings for IRR. The average raw agreement for critical actions (CL) across all scenarios was 82% and IRR was 0.71 (Gwet's AC1). Average agreement was 0.58 for GRSCOMP and 0.60 for GRSCOMM (Gwet's AC2).

We found a positive correlation between CL and GRSCOMP (rho = 0.43, p < 0.05) and between GRSCOMM and GRSCOMP (rho = 0.80, p < 0.05). We also found a significant correlation between CL and MCQ test (rho = 0.21, p < 0.05). We did not find a correlation between run order in the day and GRSCOMP (rho = 0.07, p = 0.47) or between past experience with ACP (EXP) and GRSCOMP (rho = 0.15, p = 0.14).

Postcourse evaluations suggest we were successful in meeting our learning objectives (Supplementary Fig. S1). Postsimulation session evaluations indicated increases in confidence managing each of the cases (Fig. 4). With regard to postcourse use of their knowledge, delayed surveys indicated that 20/40 responders conducted ACP. Of consented individuals with logging data, 28/58 went on to conduct 806 ACPs (mean 2.2/month/facilitator, median 0.0/month/facilitator), with a skewed distribution due to several individuals given ACP as a core job role.

FIG. 4.

For the four simulation scenarios, the learner ratings of confidence in managing the case before and after the simulation with anchors: 0% = not at all confident, 50% = somewhat confident, and 100% = completely confident.

Discussion

Simulation involves a trade-off of realism to allow a complex task to be broken into smaller time epochs and component parts. Each simulated scenario was crafted to focus on a particular challenging element of the ACP discussion and was brief compared with the one hour usually allotted for ACP. Prebriefing was extremely important in establishing the fiction contract and expectations around such compromises as well as ground rules for psychological safety.³⁸ Likewise, a positive learning environment was maintained by debriefers trained to use appreciative inquiry^36,37 and we believe this was key for a positive reception by learners.

Physicians are poor at self-assessment³⁹ and this is likely true for other professions.⁴⁰ However, self-efficacy is important for an individual contemplating a new area of practice.⁴¹ We demonstrated that the course increased self-rated confidence and self-assessment of competence, which we believe are reasonable surrogates for self-efficacy. This is important because many of our ACP facilitators, while doing ACP during paid work hours, have voluntarily taken on this challenging task. Our delayed survey could have been biased toward those who had a positive experience with the course. However, it is reassuring that we had similar percentages of those who self-reported conducting ACP on the delayed survey and those who logged ACP. While we cannot yet comment on the quality of the ACP facilitation in the field, those learners who actively conducted their first ACP, with or without a mentor present, demonstrated early transfer of learning to the workplace. Further insights are needed as to the reasons for not conducting ACP. It may be that patient scheduling logistics is an issue, time is not provided in the practice, or the facilitator does not feel comfortable enough in offering facilitation. Course takers may be exploring the possibility of conducting ACP, rather than entering the course with clear plans in place, although precourse discussions with managers are encouraged. In addition, some may simply see the course as a continuing education opportunity on an important topic or they may be managers who want deeper understanding of what they are asking of facilitators.

Our hope for the mini-CEX type simulation observation tool is that it may be used formatively for novice ACP facilitators in the simulation setting and adapted to the care setting to help mentors give more specific and actionable formative feedback. We achieved good IRR using the observational rating tool. The fact that the checklist correlated with global competency suggests that the critical actions cover the items deemed important to competence and would thus provide relevant feedback. Care must always be taken when using checklist items in simulation or care settings for feedback because not observed may simply mean that the opportunity did not present itself. We suggest that the observational checklist and rating scale are adequate for formative feedback in the hands of faculty trained to use them. While we did not find early evidence of construct validity within the simulation-based performance, our efforts were not designed or powered to do so. Future research will attempt to demonstrate construct validity so that minimum competency of ACP facilitators can be determined.

We applied SP actor-based simulation to a new audience and topic area. Given the challenging nature of these discussions, this represents a rare opportunity for deliberate practice with feedback. We hope to improve ACP training and that future studies will show improved practice through mentored observations using the form developed here, as well has high patient satisfaction with the process. The goal is to raise the quantity and quality of the desired ACP facilitators in the labor force.⁷ While recent Medicare rule changes allow for office-based physicians to bill for ACP,⁶ nonphysician ACP facilitators will likely remain as cost-effective interprofessional team members.

Footnotes

Acknowledgments

The authors wish to thank Connor Schuler, Mounika Paravastu, Karen Renken, Kim Cooley, Robert Jennetten, Lynne Madori, Kyle Mou, and the SP actor cohort.

Author Disclosure Statement

No competing financial interests exist.

References

Wood

, Arnold

: What are Advanced Care Plans and How are They Different from Advance Directives? Evidence-Based Practice of Palliative Medicine. Philadelphia: Elsevier/Saunders, 2013, pp. 258–262.

Detering

, Hancock

, Reade

, Silvester

: The impact of advance care planning on end of life care in elderly patients: Randomised controlled trial. BMJ, 2010; 340:c1345.

Tierney

, Dexter

, Gramelspacher

, et al.: The effect of discussions about advance directives on patients' satisfaction with primary care. J Gen Intern Med, 2001; 16:32–40.

Wright

, Zhang

, Ray

, et al.: Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment. JAMA, 2008; 300:1665–1673.

Hammes

, Rooney

: Death and end-of-life planning in one Midwestern community. Arch Intern Med, 1998; 158:383–390.

Medicare Program; Revisions to Payment Policies Under the Physician Fee schedule and Other Revisions to Part B for CY 2016, 80 FedReg. 220 (November 16, 2015). Federal Register: Department of Health and Human Services, Centers for Medicare & Medicaid Services.

IOM (Institute of Medicine): Dying in America: Improving Quality and Honoring Individual Preferences Near the End of Life. Washington, DC: The National Academies Press, 2014.

Benson

, Aldrich

: Advance Care Planning: Ensuring Your Wishes Are Known and Honored If You Are Unable to Speak for Yourself, Critical Issue Brief: Centers for Disease Control and Prevention; 2012. www.cdc.gov/aging (last accessed September 1, 2016).

Rao

, Anderson

, Smith

: End of life is a public health issue. Am J Prev Med, 2002; 23:215–220.

10.

Dyer

: Doctors should discuss advance care planning with dying patients and those with chronic illness, RCP says. BMJ, 2009; 338:b803.

11.

Rhee

, Zwar

, Kemp

: Uptake and implementation of advance care planning in Australia: Findings of key informant interviews. Aust Health Rev, 2012; 36:98–104.

12.

Curtis

, Engelberg

, Nielsen

, et al.: Patient-physician communication about end-of-life care for patients with severe COPD. Eur Respir J, 2004; 24:200–205.

13.

Davison

: End-of-life care preferences and needs: Perceptions of patients with chronic kidney disease. Clin J Am Soc Nephrol, 2010; 5:195–204.

14.

Schell

, Patel

, Steinhauser

, et al.: Discussions of the kidney disease trajectory by elderly patients and nephrologists: A qualitative study. Am J Kidney Dis, 2012; 59:495–503.

15.

Patel

, Janssen

, Curtis

: Advance care planning in COPD. Respirology, 2012; 17:72–78.

16.

Jones

, Moss

, Harris-Kojetin

: Use of advance directives in long-term care populations. NCHS Data Brief, 2011:1–8.

17.

Knauft

, Nielsen

, Engelberg

, et al.: Barriers and facilitators to end-of-life care communication for patients with COPD. Chest, 2005; 127:2188–2196.

18.

Prendergast

: Advance care planning: Pitfalls, progress, promise. Crit Care Med, 2001; 29:N34–N39.

19.

Smith

, Ries

, Zhang

, et al.: Resident approaches to advance care planning on the day of hospital admission. Arch Intern Med, 2006; 166:1597–1602.

20.

Davis

, O'Brien

, Freemantle

, et al.: Impact of formal continuing medical education: Do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes?. JAMA, 1999; 282:867–874.

21.

Girgis

, Cockburn

, Butow

, et al.: Improving patient emotional functioning and psychological morbidity: Evaluation of a consultation skills training program for oncologists. Patient Educ Couns, 2009; 77:456–462.

22.

Curtis

: Communicating about end-of-life care with patients and families in the intensive care unit. Crit Care Clin, 2004; 20:363–380.

23.

Ericsson

: Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med, 2004; 79:S70–S81.

24.

Detering

, Silvester

, Corke

, et al.: Teaching general practitioners and doctors-in-training to discuss advance care planning: Evaluation of a brief multimodality education programme. BMJ Support Palliat Care, 2014; 4:313–321.

25.

Hammes

, Briggs

: Respecting Choices: Advance Care Planning Facilitator Manual-Revised. La Crosse, WI: Gundersen Lutheran Medical Foundation, 2007.

26.

Chan

, Ward

, Lapin

, et al.: Outpatient advance care planning internal medicine resident curriculum: Valuing our patients' wishes. J Palliat Med, 2016; 19:734–745.

27.

Norcini

, Blank

, Arnold

, Kimball

: The mini-CEX (clinical evaluation exercise): A preliminary investigation. Ann Intern Med, 1995; 123:795–799.

28.

Hawkins

, Margolis

, Durning

, Norcini

: Constructing a validity argument for the mini-Clinical Evaluation Exercise: A review of the research. Acad Med, 2010; 85:1453–1461.

29.

Norcini

, Blank

, Duffy

, Fortna

: The mini-CEX: A method for assessing clinical skills. Ann Intern Med, 2003; 138:476–481.

30.

Holmboe

, Huot

, Chung

, et al.: Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med, 2003; 78:826–830.

31.

Ellman

, Putnam

, Green

, et al.: Demonstrating medical student competency in palliative care: Development and evaluation of a new objective structured clinical examination station. J Palliat Med, 2016; 19:706–711.

32.

Advance Care Planning (online YouTube video). British Columbia, Canada: Fraser Health. www.youtube.com/watch?v=-M31-NiH3yU. (Last accessed September 27, 2016).

33.

Han

PKJ

, Keranen

, Lescisin

, Arnold

: The palliative care clinical evaluation exercise (CEX): An experience-based intervention for teaching end-of-life communication skills. Acad Med, 2005; 80:669–676.

34.

Cate

: Entrustment as assessment: recognizing the ability, the right, and the duty to act. J Grad Med Educ, 2016; 8:261–262.

35.

Green

, Aagaard

, Caverzagie

, et al.: Charting the road to competence: Developmental milestones for internal medicine residency training. J Grad Med Educ, 2009; 1:5–20.

36.

Rudolph

, Simon

, Raemer

, Eppich

: Debriefing as formative assessment: Closing performance gaps in medical education. Acad Emerg Med, 2008; 15:1010–1016.

37.

Rudolph

, Simon

, Rivard

, et al.: Debriefing with good judgment: Combining rigorous feedback with genuine inquiry. Anesthesiol Clin, 2007; 25:361–376.

38.

Simon

, Raemer

, Rudolph

: Debriefing assessment for simulation in Healthcare (DASH)© Rater's Handbook. Boston, Massachusetts: Center for Medical Simulation. https://harvardmedsim.org/_media/DASH.handbook.2010.Final.Rev.2.pdf (last accessed August 25, 2016).

39.

Davis

, Mazmanian

, Fordis

, et al.: Accuracy of physician self-assessment compared with observed measures of competence: A systematic review. JAMA, 2006; 296:1094–1102.

40.

Watson

, Stimpson

, Topping

, Porock

: Clinical competence assessment in nursing: A systematic review of the literature. J Adv Nurs, 2002; 39:421–431.

41.

Eccles

, Grimshaw

, MacLennan

, et al.: Explaining clinical behaviors using multiple theoretical models. Implement Sci, 2012; 7:99.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.12 MB