Are Pathologists Self-Aware of Their Diagnostic Accuracy? Metacognition and the Diagnostic Process in Pathology

Abstract

Background

Metacognition is a cognitive process that involves self-awareness of thinking, understanding, and performance. This study assesses pathologists’ metacognition by examining the association between their diagnostic accuracy and self-reported confidence levels while interpreting skin and breast biopsies.

Design

We studied 187 pathologists from the Melanoma Pathology Study (M-Path) and 115 pathologists from the Breast Pathology Study (B-Path). We measured pathologists’ metacognitive ability by examining the area under the curve (AUC), the area under each pathologist’s receiver operating characteristic (ROC) curve summarizing the association between confidence and diagnostic accuracy. We investigated possible relationships between this AUC measure, referred to as metacognitive sensitivity, and pathologist attributes. We also assessed whether higher metacognitive sensitivity affected the association between diagnostic accuracy and a secondary diagnostic action such as requesting a second opinion.

Results

We found no significant associations between pathologist clinical attributes and metacognitive AUC. However, we found that pathologists with higher AUC showed a stronger trend to request secondary diagnostic action for inaccurate diagnoses and not for accurate diagnoses compared with pathologists with lower AUC.

Limitations

Pathologists reported confidence in specific diagnostic terms, rather than the broader classes into which the diagnostic terms were later grouped to determine accuracy. In addition, while there is no gold standard for the correct diagnosis to determine the accuracy of pathologists’ interpretations, our studies achieved a high-quality reference diagnosis by using the consensus diagnosis of 3 experienced pathologists.

Conclusions

Metacognition can affect clinical decisions. If pathologists have self-awareness that their diagnosis may be inaccurate, they can request additional tests or second opinions, providing the opportunity to correct inaccurate diagnoses.

Highlights

Metacognitive sensitivity varied across pathologists, with most showing higher sensitivity than expected by chance.

None of the demographic or clinical characteristics we examined was significantly associated with metacognitive sensitivity.

Pathologists with higher metacognitive sensitivity were more likely to request additional tests or second opinions for their inaccurate diagnoses.

Keywords

metacognitive sensitivity diagnostic accuracy cognitive science dermatopathology breast pathology secondary diagnostic actions awareness

Get full access to this article

View all access options for this article.

References

Bhise

Rajan

Sittig

Morgan

Chaudhary

Singh

Defining and measuring diagnostic uncertainty in medicine: a systematic review. J Gen Intern Med. 2018;33(1):103–15.

Nelson

Stuart

Howard

Crowley

Metacognition and clinical psychology: a preliminary framework for research and practice. Clin Psychol Psychother. 1999;6(2):73–9.

Foucar

Diagnostic decision-making in anatomic pathology. Pathol Patterns Rev. 2001;116(suppl 1):S21–33.

Davidson

Deuser

Sternberg

RJ.

The role of metacognition in problem solving. In: Metcalfe

Shimamura

, eds. Metacognition: Knowing about Knowing. Cambridge (MA): MIT Press; 1994. p 207–26.

Fleming

Lau

HC.

How to measure metacognition. Front Hum Neurosci. 2014;8:443.

Friedman

Gatti

Franz

, et al. Do physicians know when their diagnoses are correct? J Gen Intern Med. 2005;20(4):334–9.

Norman

Monteiro

Sherbino

Ilgen

Schmidt

Mamede

The causes of errors in clinical reasoning: cognitive biases, knowledge deficits, and dual process thinking. Acad Med. 2017;92(1):23–30.

Nelson

. Metamemory: a theoretical framework and new findings. In: Bower

, ed. Psychology of Learning and Motivation. Cambridge (MA): Academic Press; 1990:125–73.

Metcalfe

Shimamura

AP.

Metacognition: Knowing about Knowing. Cambridge (MA): MIT Press; 1994.

10.

Maniscalco

Lau

A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings. Conscious Cogn. 2012;21(1):422–30.

11.

Sherbino

Dore

Wood

, et al. The relationship between response time and diagnostic accuracy. Acad Med. 2012;87(6):785–91.

12.

Weber

Böckenholt

Hilton

Wallace

Determinants of diagnostic hypothesis generation: effects of information, base rates, and experience. J Exp Psychol Learn Mem Cogn. 1993;19(5):1151–64.

13.

Groves

O’Rourke

Alexander

Clinical reasoning: the relative contribution of identification, interpretation and hypothesis errors to misdiagnosis. Med Teach. 2003;25(6):621–5.

14.

Desender

Boldt

Yeung

Subjective confidence predicts information seeking in decision-making. Psychol Sci. 2018;29(5):761–78.

15.

Nussenbaum

Cohen

Davis

Halpern

Gureckis

Hartley

CA.

Causal information-seeking strategies change across childhood and adolescence. Cogn Sci. 2020;44(9):e12888.

16.

Elmore

Barnhill

Elder

, et al. Pathologists’ diagnosis of invasive melanoma and melanocytic proliferations: observer accuracy and reproducibility study. BMJ. 2017;357:j2813.

17.

Elmore

Longton

Carney

, et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA. 2015;313(11):1122–32.

18.

Piepkorn

Barnhill

Elder

, et al. The MPATH-Dx reporting schema for melanocytic proliferations and melanoma. J Am Acad Dermatol. 2014;70(1):131–41.

19.

Allison

Reisch

Carney

, et al. Understanding diagnostic variability in breast pathology: lessons learned from an expert consensus review panel. Histopathology. 2014;65(2):240–51.

20.

Dalkey

Brown

Cochran

The Delphi Method, III: Use of Self-Ratings to Improve Group Estimates. Santa Monica (CA): RAND Corporation; 1969.

21.

Carney

Reisch

Piepkorn

, et al. Achieving consensus for the histopathologic diagnosis of melanocytic lesions: use of the modified Delphi method. J Cutan Pathol. 2016;43(10):830–7.

22.

Blanch

Hall

Roter

Frankel

RM.

Medical student gender and issues of confidence. Patient Educ Couns. 2008;72(3):374–81.

23.

Blanch-Hartigan

Medical students’ self-assessment of performance: results from three meta-analyses. Patient Educ Couns. 2011;84(1):3–9.

24.

Metcalfe

Eich

Castel

AD.

Metacognition of agency across the lifespan. Cognition. 2010;116(2):267–82.

25.

Cutting

Saks

NS.

Twelve tips for utilizing principles of learning to support medical education. Med Teach. 2012;34(1):20–4.

26.

Elmore

Tosteson

ANA

Pepe

, et al. Evaluation of 12 strategies for obtaining second opinions to improve interpretation of breast histopathology: simulation study. BMJ. 2016;353:i3069.

27.

Middleton

Feeley

Albright

Walters

Hamilton

SH.

Second-opinion pathologic review is a patient safety mechanism that helps reduce error and decrease waste. J Oncol Pract. 2014;10(4):275–80.

28.

Piepkorn

Longton

Reisch

, et al. Assessment of second-opinion strategies for diagnoses of cutaneous melanocytic lesions. JAMA Netw Open. 2019;2(10):e1912597-e.

29.

Tosteson

ANA

Yang

Nelson

, et al. Second opinion strategies in breast pathology: a decision analysis addressing over-treatment, under-treatment, and care costs. Breast Cancer Res Treat. 2018;167(1):195–203.

30.

Butler

Karpicke

Roediger

HL.

Correcting a metacognitive error: feedback increases retention of low-confidence correct responses. J Exp Psychol Learn Mem Cogn. 2008;34(4):918–28.

31.

Miller

Geraci

Training metacognition in the classroom: the influence of incentives and feedback on exam predictions. Metacogn Learn. 2011;6(3):303–14.

32.

Callender

Franco-Watkins

Roberts

AS.

Improving metacognition in the classroom through instruction, training, and feedback. Metacogn Learn. 2016;11(2):215–35.

33.

El Saadawi

Azevedo

Castine

, et al. Factors affecting feeling-of-knowing in a medical intelligent tutoring system: the role of immediate feedback as a metacognitive scaffold. Adv Health Sci Educ Theory Pract. 2010;15(1):9–30.

34.

Kulik

C-lC

. Timing of feedback and verbal learning. Rev Educ Res. 1988;58(1):79–97.

35.

Gonullu

Artar

Metacognition in medical education. Educ Health. 2014;27(2):225.

36.

Medina

Castleberry

Persky

AM.

Strategies for improving learner metacognition in health professional education. Am J Pharm Educ. 2017;81(4):78.

37.

Quirk

Intuition and Metacognition in Medical Education: Keys to Developing Expertise. New York: Springer; 2006.

38.

Turan

Demirel

Sayek

Metacognitive awareness and self-regulated learning skills of medical students in different medical curricula. Med Teach. 2009;31(10):e477–83.

39.

Fernandez-Duque

Baird

Posner

MI.

Executive attention and metacognitive regulation. Conscious Cogn. 2000;9(2):288–307.

40.

Fleming

Dolan

RJ.

The neural basis of metacognitive ability. Philos Trans R Soc Lond B Biol Sci. 2012;367(1594):1338–49.

41.

Shimamura

AP.

Toward a Cognitive Neuroscience of Metacognition. Cambridge (MA): Academic Press; 2000.

42.

Elder

Piepkorn

Barnhill

, et al. Pathologist characteristics associated with accuracy and reproducibility of melanocytic skin lesion interpretation. J Am Acad Dermatol. 2018;79(1):52-9.e5.

43.

Kincannon

Gleber

Kim

The Effects of Metacognitive Training on Performance and Use of Metacognitive Skills in Self-Directed Learning Situations. Eric No. ED436146. February 1999.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.15 MB