Entrustable Professional Activities in Palliative Medicine: A Faculty and Learner Development Activity

Abstract

Background:

Faculty development (FD) is critical to the implementation of competency-based medical education (CBME) and yet evidence to guide the design of FD activities is limited. Our aim with this study was to describe and evaluate an FD activity as part of CBME implementation.

Methods:

Palliative medicine faculty were introduced to entrustable professional activities (EPAs) and gained experience estimating a learner’s level of readiness for entrustment by directly observing a simulated encounter. The variation that was found among assessments was discussed in facilitated debrief sessions. Attitudes and confidence levels were measured 1 week and 6 months following debriefs.

Results:

Participants were able to use the EPA framework when estimating the learner’s readiness level for entrustment. Significant improvements in attitudes and level of confidence for several knowledge, skill, and behavior domains were maintained over time.

Conclusions:

Simulated direct observation and facilitated debriefs contributed to preparing both faculty and learners for CBME and EPA implementation.

Background

A key component of assessment in competency-based medical education (CBME) is directly observing learners in clinical settings, which has historically occurred relatively infrequently in postgraduate contexts.¹ Developed to operationalize workplace-based assessment, entrustable professional activities (EPAs) are the units of professional practice that reflect the essential work of a discipline and can be both directly observed and entrusted to a learner.²

Despite faculty development (FD) being a necessary component of effective CBME implementation, there is little evidence to guide the design and delivery of assessor training activities.^3,4 The result is a pressing need for efficient and scalable strategies that address CBME implementation broadly and EPA frameworks more specifically.

This study describes the experience of introducing EPAs through a faculty and learner development (FLD) assessor training activity in a postgraduate palliative medicine training program and reports the impact on participants’ self-assessed attitudes and confidence levels over time.⁵

Methods

Participants and setting

Potential participants in this quasi-experimental study, which took place in September and October 2018, included all palliative medicine clinical supervisors and postgraduate learners in the University of Toronto’s academic Division of Palliative Care. Approval for the study was granted by University of Toronto’s Research Ethics Review board.

Intervention

The assessor training activity involved a self-guided, didactic introduction to the EPA framework and orientation to a specific palliative medicine EPA, i.e., manage the care of a dying patient in the last days, final hours. Participants completed a simulated direct observation and assessment of a learner’s performance, i.e., reviewed a video-recorded clinical encounter between a learner and a standardized family member, and estimating the learner’s readiness for entrustment.

When designing the clinical encounter, authenticity of the assessor experience was prioritized. A specific level of readiness for entrustment was not predetermined, and the unscripted video was shot in one uninterrupted take. Participants viewed the video in its original and unedited form. Participant estimates of the learner’s readiness for entrustment were captured using an entrustment-supervision scale. This included five descriptions of progressively less supervision being required, with the final two reflecting the perception that the learner was ready to be entrusted with the activity (Fig. 1). Each participant was also asked to describe the learner’s strengths and areas for improvement, as well as the approach that was used to estimate readiness for entrustment.

FIG. 1.

Distribution of estimated levels of readiness for EPA elements and the EPA overall. EPA, entrustable professional activities.

Within one week of completing the simulated direct observation, small groups were assembled to debrief the experience. Discussions focused on the substantial variability that was found among estimates of readiness for entrustment as well as the processes used by participants to arrive at an estimate. A qualitative analysis of the debrief discussions and assessor variability is reported separately.

Data collection

Participants’ impressions of the assessor training activity were captured with a one-time survey, and attitudes and confidence levels over time were measured using serial surveys. A FD scoping review served as the source for domains of individual survey items.⁴

Surveys were pilot-tested and finalized with faculty members who were not participating in the activity. The one-time survey was comprised of seven items and completed immediately following debriefs. The survey used serially was comprised of 13 items for faculty participants and five items for learner participants. This was administered at three time points: prior to the simulated direct observation (Pre), and one week (Post) and six months (Post-6) following debrief sessions. Demographic information included sex, number of years in practice, and the frequency with which clinical supervision is provided to a palliative medicine resident who is completing a rotation at their site (always, usually, or occasionally). Demographic information for learner participants was not collected to maintain anonymity.

Data analysis

Data sources for the analysis included demographic information from faculty participants, assessment information from all participants, and data from completed surveys. Descriptive statistics were completed on faculty demographic variables, and chi-squared tests of independence were used to identify any associations with estimates of readiness for entrustment. Odds ratios (ORs) were calculated to reflect the strength of identified associations. For survey items with data from all three time points, repeated measures ANOVA tests detected differences in attitudes and levels of confidence for various knowledge, skills, and behavior domains. Mauchly’s test was used to detect violations of the sphericity assumption and if none were detected, uncorrected within-subjects effects are reported. If violations were detected, the Greenhouse–Geisser correction was applied. Items with significant differences underwent post hoc analysis using the Bonferroni correction to identify the specific time points that differed significantly. All tests of statistical significance were designated at p < 0.05. To examine internal consistency among survey items, Cronbach’s alpha was calculated for each survey administration.

Results

Forty-five faculty members and 3 learners were contacted and offered participation with 17 and 2, respectively, responding, providing consent, and completing the FLD activity. Among the 17 faculty members, 11 (64.7%) were female, 10 (58.8%) had been faculty for 10 years or less, and the frequency of clinical supervision was “always” for 7 (41.2%), “usually” for 4 (23.5%) and “occasionally” for 6 (35.3%). Further demographic information is outlined in Table 1.

Table 1.

Demographic Information by Estimated Readiness Level for Entrustment

	Total(n = 17)	Ready for entrustment(n = 7)	Not yet ready for entrustment(n = 10)	p	Or	95% CI
Gender
Female	11 (64.7%)	4 (57.1%)	7 (70%)	0.59	0.57	0.08; 4.30
Male	6 (35.3%)	3 (42.9%)	3 (30%)	0.59	0.57	0.08; 4.30
Years in practice
≤10	10 (58.8%)	3 (42.9%)	7 (70%)	0.26	0.32	0.04; 2.42
>10	7 (41.2%)	4 (57.1%)	3 (30%)	0.26
Frequency of supervision
Always	7 (41.2%)	1 (14.2%)	6 (60%)	0.06	9.0	0.76; 106
Usually	4 (23.5%)	3 (42.9%)	1 (10%)
Occasionally	6 (35.3%)	3 (42.9%)	3 (30%)

CI, confidence interval; OR, odds ratio; p, p value.

Estimates of readiness for entrustment are summarized in Figure 1. Ten participants (52.6%) determined that the learner was not ready for entrustment, with 4 (21.0%) indicating the learner required major instruction and 6 (31.6%) indicating the learner needed minor advice. Among the 9 participants (47.4%) who determined the learner was ready for entrustment, 7 (36.8%) determined that the supervisor did not need to be present and 2 (10.6%) felt the learner demonstrated a leading practice.

Associations between demographic variables of faculty participants and estimates of readiness are outlined in Table 1. Neither sex (OR = 0.57; 95% confidence interval [CI] 0.08, 4.30; p = 0.59) nor the number of years in practice (OR = 0.32; 95% CI: 0.04, 2.42; p = 0.26) were found to be associated with estimated readiness. Assessors who indicated they always provide clinical supervision, however, were nine times more likely to estimate the learner was not ready for entrustment than those who only usually or occasionally clinically supervise, although this did not reach statistical significance (OR = 9.0; 95% CI: 0.76, 106; p = 0.06).

Table 2 outlines the participant evaluations of the assessor training activity itself. All participants felt the content was helpful and relevant to their role. 94.7% felt the length of the debrief was appropriate, the educational format and delivery methods suited the content, and that the debrief resulted in a deeper understanding of EPAs. 89.5% felt the debrief was well organized and well planned.

Table 2.

Participant Evaluation of FLD Activity

Agree or stronglyAgree (n = 19)		%
The learning objectives were met	15	78.9
The content was helpful for my learning	19	100
The content was relevant to my role	19	100
The debrief provided me with a deeper understanding of the subject matter	18	94.7
The methods used to deliver message suited the content	18	94.7
The length was appropriate	18	94.7
The debrief was presented in an organized, well-planned manner	17	89.5

FLD, faculty and learner development.

Differences in attitudes and confidence levels over time are outlined in Tables 3 and 4. Despite participants holding strongly positive attitudes about EPAs before the activity, attitudes about the usefulness of EPAs for resident learning (F[2,16] = 4.0, p = 0.03) and for guiding direct observation (F[2,14] = 4.0, p = 0.03) significantly improved one week following the activity. This was not maintained over time, however.

Table 3.

Repeated Measures ANOVA Tests to Detect Differences in Means of Attitudes and Levels of Confidence over Time

	Item	n	Pre (SD)	Post (SD)	Post-6 (SD)	df	F	p	η ²
Attitude	EPAs are relevant to CBME^b	17	4.67 (0.49)	4.75 (0.58)	4.50 (0.86)	1.42^a	0.64	0.49	0.02
	EPAs are useful for resident learning^b	17	4.44 (0.51)	4.94 (0.24)	4.59 (0.80)	2	4.00	0.03	0.12
	EPAs are useful for resident assessment^b	17	4.27 (0.88)	4.41 (0.87)	4.41 (0.80)	1.22^a	0.52	0.52	0.01
	EPAs are useful for guiding direct observation	15	4.15 (0.69)	4.73 (0.46)	4.47 (0.64)	2	4.00	0.03	0.15
	EPAs are useful for providing meaningful feedback	15	4.50 (0.67)	4.50 (0.67)	4.50 (0.67)	2	0	1.00	0
Knowledge	Confident explaining an EPA^b	17	3.61 (0.85)	4.28 (0.58)	4.11 (0.68)	2	5.97	<0.01	0.14
Knowledge	Confident defining CBME^b	17	3.39 (0.78)	3.78 (0.55)	3.83 (0.79)	2	2.17	0.13	0.08
	Confident using EPAs to assess a learner	15	3.33 (0.89)	4.07 (0.39)	4.20 (0.58)	2	10.60	<0.01	0.28
	Confident assigning a level of entrustment to a learner	15	3.33 (0.78)	4.00 (0.76)	4.13 (0.64)	2	4.58	0.02	0.29
Skill	Confident using EPAs to generate meaningful feedback	15	3.55 (0.82)	4.18 (0.60)	4.27 (0.65)	2	3.62	0.07	0.17
	Confident using EPAs to coach learners for improvement	15	3.50 (0.91)	4.00 (0.43)	4.17 (0.72)	2	3.04	0.07	0.15
	Confident knowing what to observe when using EPAs	15	3.50 (0.80)	4.13 (0.52)	4.27 (0.45)	2	7.50	<0.01	0.24
Behavior	Able to distinguish levels of performance using an EPA tool	15	3.09 (0.94)	4.00 (0.75)	4.07 (0.54)	2	12.20	<0.01	0.32

Mauchly’s test of sphericity was violated, Greenhouse–Geisser correction applied.

Item completed by both faculty and learners.

SD = standard deviation; df = degrees of freedom; F = F-test statistic; p = p value; η² = eta squared (effect size).

Table 4.

Post Hoc Analysis with Bonferroni Correction

	Pre to Post			Pre to Post-6			Post to Post-6
Item	MD	SE	p	MD	SE	p	MD	SE	p
EPAs are useful for resident learning^a	−0.46	0.13	0.01	−0.07	0.18	0.49	0.40	0.21	0.25
EPAs are useful for guiding direct observation	−0.54	0.22	0.03	−0.39	0.21	0.10	0.15	0.15	0.34
Confident explaining an EPA^a	−0.65	0.23	0.02	−0.47	0.17	0.01	0.18	0.18	0.33
Confident using EPAs to assess a learner	−0.73	0.21	0.01	−0.87	0.21	0.01	−0.13	0.17	0.43
Confident assigning a level of entrustment to a learner	−0.58	0.29	0.07	−0.75	0.31	0.03	−0.17	0.17	0.34
Confident knowing what to observe when using EPAs	−0.67	0.26	0.03	−0.75	0.22	0.01	−0.08	0.15	0.59
Able to distinguish levels of performance using an EPA tool	−1.09	0.29	0.01	−1.00	0.23	<0.01	−0.09	0.21	0.90

Item completed by both faculty and learners.

MD, mean difference; p, p value; SE, standard error.

Participant confidence explaining an EPA (F[2,16] = 5.97, p < 0.01), using EPAs for assessing learners (F[2,14 = 10.6, p < 0.01), using EPAs to determine a level of entrustment (F[2,14] = 4.6, p < 0.01), knowing what to observe when using EPAs (F[2,14] = 7.5, p < 0.01) and using EPAs to distinguish levels of performance (F[2,14 = 12.2, p < 0.01) significantly improved both one week and six months following the activity. Internal consistency was either good or acceptable for all three surveys (Cronbach’s alpha: Pre = 0.86, Post = 0.73, Post-6 = 0.7).

Discussion

The rapid pace of CBME implementation and limited evidence that guides the design of FD experiences creates a pressing need to examine educational approaches to preparing assessors. FD activities should be experiential in nature, have straightforward and practical application, and have immediate relevance to clinical teachers.⁶ Our findings indicate that palliative medicine participants were able to estimate a learner’s readiness for entrustment and, despite being a one-time activity, significant improvements in some self-assessed attitudes were noted post-activity and confidence levels for several knowledge, skill, and behavior domains were maintained over time.

The finding that faculty participants who always clinically supervise residents were nine times more likely to estimate that the learner was not ready for unsupervised practice is notable. It could be that greater experience enables confidence to identify learners who require supervision. The corresponding challenge for programs would be determining how to develop junior faculty along with maintaining assessment reliability. As the assessment-related tasks with CBME are particularly complex, programs have been encouraged to focus on developing a select number of expert assessors.⁷ Our findings may support this strategy, in particular for training programs that have few learners and a comparatively larger number of faculty.

Among the several limitations of this study is that participants self-selected rather than using a formal sampling strategy. It is unclear if similar improvements could be expected among all faculty members. An important limitation is self-assessment being the main data source. This is generally thought to be inadequate when evaluating an educational intervention; however, the current study is among the few reporting longitudinal data for CBME FD experiences. Ideally, assessor training experiences are evaluated using prospective data that capture changes in both workplace behaviors and learner experiences.

Conclusions

This study describes a FLD activity in which simulated direct observation enabled participants to gain experience using an EPA to estimate a learner’s readiness for entrustment. When combined with facilitated small group debriefs, improvements in some attitudes were noted post-activity and confidence levels for several knowledge, skill, and behavior domains were maintained over time. Faculty have identified effective feedback provision as a priority topic, which will serve as our local next step.

References

Cheung

, Patey

, Frank

, et al. Barriers and enablers to direct observation of trainees’ clinical performance: A qualitative study using the theoretical domains framework. Acad Med, 2019; 94(1):101–114.

Ten Cate

. Entrustability of professional activities and competency-based training. Med Educ, 2005; 39(12):1176–1177.

Holmboe

, Ward

, Reznick

, et al. Faculty development in assessment: The missing link in competency-based medical education. Acad Med, 2011; 86(4):460–467.

Sirianni

, Glover Takahashi

, Myers

. Taking stock of what is known about faculty development in competency-based medical education: A scoping review. Med Teach, 2020; 42(8):909–915.

Favreau

, Tewksbury

, Lupi

, et al. AAMC Core Entrustable Professional Activities for Entering Residency Faculty Development Concept Group. Constructing a shared mental model for faculty development for the core entrustable professional activities for entering residency. Acad Med, 2017; 92(6):759–764.

Steinert

, Mann

, Centeno

, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education. Med Teach, 2006; 28:497–526.

Kogan

, Hatala

, Hauer

, Holmboe

. Guidelines: The do’s, don’ts and don’t knows of direct observation of clinical skills in medical education. Perspect Med Educ, 2017; 6(5):286–305.