Myth or Reality: Self-Assessment Is Central to Effective Curriculum in Anatomical Pathology Graduate Medical Education

Introduction

Within regulated health professions, self-assessment has frequently been cited as a necessary component of self-directed learning and the maintenance of competency. ^1,2 We are focused on developing professional learners who not only achieve the necessary competencies throughout their formal education, but also continue to grow and adapt their knowledge, behavior, and skills as their professions evolve over time. ³ Exploring the values, limitations, and potential roles of self-assessment on learning within clinical settings will assist with awareness and appropriate application of self-assessment within our curriculum.

Critical to the latter goal is a need for clarity surrounding how self-assessment is defined. Not surprisingly, the conceptual framework of self-assessment varies by context and has evolved over time. In an early review of self-assessment in the health professions, Gordon considered valid self-assessment as judging one’s performance against appropriate criteria. ¹ An alternative conceptual model of self-assessment is one where the learner’s judgment is informed by standards as well as internal and external data, such as emotions and external feedback from peers and teachers. ^4,5 The terminology, informed self-assessment, has since been adopted by a large group of medical educators to capture self-assessment as a reflective process that is informed by data, including standards and multisource feedback. ^{6 -8} While these differences in how self-assessment is defined are significant, they share the characteristic of self-assessment being a personal evaluation of one’s professional attributes and abilities against a perceived norm. ⁹

In this article, we review the evidence on the accuracy and effectiveness of self-assessment, examine enabling factors for the persistence of generalized uninformed self-assessment in medical professions, and suggest recommended practices surrounding self-assessment in the clinical setting of Anatomical Pathology graduate medical education.

Is Self-Assessment by Adult Learners Accurate?

The accuracy of self-assessment by adult learners has been extensively studied since the 1970s. In 1991, Gordon wrote the first systematic review on the accuracy of self-assessment by adult learners as compared to external assessment by peers, experts, or standardized tests. ¹ Eighteen studies published between 1970 and 1990 were included in this review, of which 14 were on health profession learners. In general, self-assessment correlated poorly with external assessment, with the majority of learners overestimating their actual knowledge or skill performance. A later meta-analysis by Davis et al specifically examined the accuracy of self-assessment among physician practitioners and learners. ² Of the 20 domains examined in the 17 published studies, 13 (65%) demonstrated little, no, or even an inverse relationship between self- and external-assessment. The most recent systematic review on self-assessment was published in 2008 and included 77 studies involving health profession learners or practitioners published between 1990 and 2005. ⁹ Similar to the earlier reviews, health professionals demonstrated poor ability to self-assess. ⁹

Several learner- and curriculum-related factors influencing the accuracy of self-assessment were identified by these review studies (see Table 1). Learner skill has been consistently identified as a major learner-related factor. Studies by Kruger and Dunning on undergraduate university students, as well as other studies on health profession learners, have consistently found that learners who were the least skilled significantly overestimated their performance, while highly skilled learners underestimated their performance. ¹⁰ Several studies have also identified gender differences in the accuracy of self-assessment. A longitudinal study by Edwards et al involving 1152 third-year medical students found that men were 1.7 times more likely to overestimate their grades than women. ¹¹ Other studies have also identified a trend for men to express higher levels of confidence than women. ⁹ Cultural background of the learners was not found to significantly influence the accuracy of self-assessment, although there is a lack of high-quality literature on this topic. ⁹ Several curriculum-related factors were also identified. The content of self-assessment was significantly associated with the accuracy of self-assessment. Content unfamiliar to learners (such as asking a surgeon to self-assess their ability to determine patient’s decision-making capacity), complex concepts particularly those involving cognitive skills and knowledge, “soft” skills (such as communication and personal attitude), and self-assessment of global competence were associated with inaccurate self-assessment. On the other hand, learner’s familiarity with the content, performance skills (such as surgical skills or physical exam skills), and self-assessment of task-specific competency, significantly increased self-assessment accuracy. ^1,9 External feedback also influenced the accuracy of self-assessment, especially when based on explicit and measurable assessment criteria. ^1,9 Performance feedback that pointed out what learners did well and poorly with or without coaching for improvement, and benchmark feedback that informed learners of the expected level of performance, were associated with more accurate self-assessment. ^1,9 Reflection also appeared to improve self-assessment accuracy. Several studies examined the use of videotapes in medical training, and found improved self-assessment accuracy when learners were allowed to review and reflect on their videotaped performance. ^1,9 Self-assessment accuracy also improved after learners were provided with external feedback beforehand, allowing learners to reconcile this with their unguided self-assessment. ^1,9

OPEN IN VIEWER

Table 1.

Factors Influencing the Accuracy of Learner Self-Assessment.

Facilitators to self-assessment	Barriers to self-assessment
Explicit and measurable assessment criteria (C)	Low- or high-performing learner (L)
Review of performance data by learner (C)	Learner of male gender (L)
Performance feedback (C)	Unfamiliar content (C)
Benchmark feedback (C)	Complex cognitive or “soft” skills (C)
Feedback from external sources (C)	Global assessment (C)

Abbreviations: C, curriculum-related factors; L, learner-related factors.

Several theories were proposed to explain this inability to accurately self-assess. Cognitive psychology studies have shown that individuals naturally and subconsciously use defense mechanisms to diffuse negative information and to enhance positive information, the most common mechanisms being self-serving reasoning, biased hypothesis testing, and biased recall. With self-serving reasoning, individuals attribute success to internal factors such as skills and intelligence, and failure to external factors such as bad luck and distraction. Even when external feedback is provided, biased hypothesis testing ensures that, when confronted with negative feedback, one generally disbelieves this feedback unless solid evidence is provided, while positive feedback requires little evidence. When self-assessing, individuals also commonly rely on memories of their past performance for evidence. However, studies have shown that feedback on success is more memorable than feedback on failure, and negative feedback is preferentially forgotten. ¹² To explain why, in their study, learners who were the least skilled were particularly inaccurate in self-assessment, Kruger and Dunning postulated that metacognitive skills essential to learning and performing well are also indispensable for accurate self-assessment, and poorly performing learners cannot accurately self-assess until they are taught these metacognitive skills and their performance improves. ¹⁰ In light of these engrained deficiencies in self-assessment, Eva and Regehr argued that perhaps it is simply not possible to accurately assess one’s performance without systematically and intentionally eliciting external assessment. ¹³

Is Accurate Self-Assessment Central to An Effective Curriculum?

Contrary to the common assumption in health profession education, the systematic review by Colthart et al found no empirical evidence that supports the role of self-assessment in identifying learning needs, modifying learning activities, and improving clinical outcomes. ⁹ Instead, research supports the effectiveness of reflection, self-efficacy, and self-directed assessment seeking in learning, metacognitive skills that are related, but not identical, to self-assessment in learning. ^{13 -15} Reflection, as defined by Donald Schön, includes reflection-in-action, which involves concurrent self-evaluation and self-correction while performing a task, and reflection-on-action, which involves summative evaluation of one’s performance after the end of the task. ¹⁶ A systematic review by Mann et al found that reflection-in- and on-action effectively stimulated deeper understanding of the knowledge content and complex clinical problem, and were associated with self-reported performance improvement and change in clinical practice. ¹⁷ Self-efficacy is defined as the confidence in one’s ability to eventually succeed in performing a task, and is essential for maintaining resilience and psychological well-being. A meta-analysis on academic self-efficacy by Marsh and Martin showed that students with a high level of academic self-efficacy were more likely to achieve subsequent academic success, and past academic success contributed to high academic self-efficacy. ¹⁸ Interesting, high self-efficacy has been associated with overestimating one’s current performance. Conversely, individuals with clinical depression and low self-efficacy have been found to be more accurate in self-assessing their abilities. ¹⁸ Self-directed assessment seeking is the ability to actively seek out formative external assessment. As discussed previously, external assessment improves the accuracy of self-assessment. In the literature on physician workplace-based assessment, high-quality performance feedback from external sources was generally rated as educational by feedback recipients, and was associated with improved objective performance. ¹⁹

Even when accurate self-assessment is possible, Regehr and Eva hypothesized that this may still be insufficient for health professionals to embark on self-directed learning and practice change, as learning in the areas of one’s weaknesses will invariably necessitate more energy and commitment. ²⁰ Therefore, unless mandated by regulatory bodies or other authorities, health professionals continue learning in the areas of expertise, while resist learning in the areas of weaknesses. ²⁰

Why Do the Myths of Self-Assessment Persist?

Myths persist in medical education because they fulfill three essential social purposes: (1) shared meaning-making, (2) a vehicle for values and ideologies, and (3) a means of maintaining social power structures. ²¹ The authors believe that three main factors explain the persistent myth of the central role of self-assessment to effective curriculum in medical education, despite the empirical evidence regarding its limitations.

First, self-assessment is a central tenet of Knowles’ adult learning theory and of other humanistic learning theories, which believe that adult learners are inherently motivated to learn and improve in their self-identified knowledge or skill gaps. ²¹ These theories are in turn the foundation of curricular design for health professions.

Second, health professions’ signature pedagogy is apprenticeship, a model characterized by “learning in the practice settings of the workplace.” ²² This pedagogic model relies heavily on learners’ ability to self-assess and self-direct their learning. On the other hand, putting learning responsibilities mainly on learners lessens the responsibility of teachers, curriculum designers, and educational institutions for learners’ performance.

Lastly and perhaps most importantly, health professionals’ ability to self-assess is central to effective self-regulation, which is in turn a cornerstone of professional autonomy. According to Regehr and Eva, self-regulation involves the following continuous and repeated steps: ongoing self-assessment of performance, self-identification of knowledge or skills that have fallen below standards of practice, self-directed learning to address these gaps, and practice change incorporating newly learned knowledge or skills. ²⁰ Without the ability to self-assess, external assessors will need to be brought in to evaluate health professionals’ competency and the design of their learning plan, and to monitor their learning progress and practice change.

The 2021 Lens on Self-Assessment and Its Incorporation in the Anatomical Pathology Graduate Medical Curriculum

While it is clear that generalized uninformed self-assessment has limited value in the learners’ ability to identify their strengths and weaknesses, the evidence does support continued incorporation of learner self-assessment within the curriculum. ^6,12 The literature identifies three overarching themes: (1) the necessity of learners incorporating standardized outcome measures into self-assessment, (2) the value of learners’ use of feedback (internal and external data) within self-assessment, and (3) the role of self-efficacy in learning and continued professional growth. ^6,12 Further discussion with examples is provided below.

Anatomical Pathology Resident Training

As a background, residency training in Canada occurs after three to four years of medical school training, and is the training that leads to specialty (or subspecialty) certification by the Royal College of Physicians and Surgeons of Canada (RCPSC) or certification as a family physician by the College of Family Physicians of Canada (CFPC). Having achieved the RCPSC or CFPC certificate, the physician is prepared for unsupervised medical practice supported by continuing professional development. The University of Toronto’s Anatomical Pathology residency program, one of the largest in Canada, is a five-year training program with a total of 30 to 35 residents. Training occurs at multiple sites including four academic hospitals, affiliated community hospitals, and a provincial Forensic Pathology service. Residents receive ad hoc verbal feedback from faculty and allied health professionals such as pathologists’ assistants and cytotechnologists in the clinical setting, in addition to formal written rotation evaluations 10 to 12 times per year, and slide, written, and oral exams administered semi-annually. Since 2019, Anatomical Pathology residency programs across Canada have moved into a competency-based medical education model called “Competency-by-Design” (CBD), whereby the residency training program is divided into four distinct stages with a list of learning goals and objectives called Entrustable Professional Activities (EPAs) and Milestones. ²³ The resident must be observed completing each EPA independently and satisfactorily to progress to the next stage. ²³ A key purposed benefit of CBD is increased opportunities for feedback and coaching to guide learning. ²³ While each graduate medical training experience is undoubtedly unique, it is anticipated that some of the suggested applications related to self-assessment described below in the four curricular commonplaces are transferrable to other residency programs (see Table 2).

OPEN IN VIEWER

Table 2.

Strategies for Effective Learner Self-Assessment in Anatomical Pathology Graduate Medical Education.

Learner	Teacher
Normalized feedback-seeking behaviors Learning from tasks at the periphery of competency	Assessment based on direct observation of learner’s performance Fostering a culture of trust between learner and instructor
Context	Content
Frequent, dedicated time for learner feedback and coaching	Skill-specific assessment Explicit and measurable assessment criteria Opportunity for learner self-reflection prior to external assessment Provision of performance and benchmark feedback

Conclusions

Within regulated professions, uninformed personal global assessment of performance has consistently been shown to be inaccurate in a variety of contexts including the medical professions, and has limited value in a workplace-based curriculum. Incorporating known standards of performance with internal and external data on the performance improves learners’ ability to accurately self-assess. ¹¹ Selecting content suitable for self-assessment, ensuring standards are explicit to learners, encouraging feedback-seeking behaviors, supporting a growth mindset, and providing quality feedback in a supportive context are all strategies that support learner self-assessment, learner engagement in reflection, and action on feedback in the Anatomical Pathology graduate medical education. ^9,12,25