Previous research has described physicians’ reluctance to use computerized diagnostic aids (CDAs) but has never experimentally examined the effects of not consulting an aid that was readily available. Experiment 1. Participants read about a diagnosis made either by a physician or an auto mechanic (to control for perceived expertise). Half read that a CDA was available but never actually consulted; no mention of a CDA was made for the remaining half. For the physician, failure to consult the CDA had no significant effect on competence ratings for either the positive or negative outcome. For the auto mechanic, failure to consult the CDA actually increased competence ratings following a negative but not a positive outcome. Negligence judgments were greater for the mechanic than for the physician overall. Experiment 2. Using only a negative outcome, we included 2 different reasons for not consulting the aid and provided accuracy information highlighting the superiority of the CDA over the physician. In neither condition was the physician rated lower than when no aid was mentioned. Ratings were lower when the physician did not trust the CDA and, surprisingly, higher when the physician believed he or she already knew what the CDA would say. Finally, consistent with our previous research, ratings were also high when the physician consulted and then followed the advice of a CDA and low when the CDA was consulted but ignored. Individual differences in numeracy did not qualify these results. Implications for the literature on algorithm aversion and clinical practice are discussed.
Committee on Diagnostic Error in Health Care, Board on Health Care Services, Institute of Medicine, The National Academies of Sciences, Engineering, and Medicine. Improving Diagnosis in Health Care. BaloghEPMillerBTBallJR, eds. Washington (DC): National Academies Press (US); 2016.
2.
TehraniASSLeeHMathewsSC, et al. 25-year summary of US malpractice claims for diagnostic errors 1986–2010: an analysis from the National Practitioner Data Bank. BMJ Qual Saf. 2013;22(8):672–80.
3.
DawesRMFaustDMeehlPE. Clinical versus actuarial judgment. Science. 1989;243(4899):1668–74.
4.
GroveWMZaldDHLebowBSSnitzBENelsonC. Clinical versus mechanical prediction: a meta-analysis. Psychol Assess. 2000;12(1):19–30.
5.
GargAXAdhikariNKJMcDonaldH, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223–38.
6.
BrightTJWongADhurjatiR, et al. Effect of clinical decision-support systems. Ann Intern Med. 2012;157:29.
7.
LynchCJListonC. New machine-learning technologies for computer-aided diagnosis. Nat Med. 2018;24(9):1304–5.
8.
BernerESLa LandeTJ. Overview of clinical decision support systems. In: BernerES, ed. Clinical Decision Support Systems: Theory and Practice. New York: Springer; 2007. p. 3–22.
9.
ArkesHR. Being an advocate for linear models of judgment is not an easy life. In: DawesRMKruegerJI, eds. Rationality and Social Responsibility: Essays in Honor of Robyn Mason Dawes. Boca Raton (FL): CRC Press; 2008. p. 47–70.
10.
HighhouseS. Stubborn reliance on intuition and subjectivity in employee selection. Ind Organ Psychol. 2008;1(3):333–42.
11.
LaiFMacmillanJDaudelinDHKentDM. The potential of training to increase acceptance and use of computerized decision support systems for medical diagnosis. Hum Factors. 2006;48(1):95–108.
CoieraE. Technology, cognition and error. BMJ Qual Saf. 2015;24(7):417–22.
14.
EcclesMMcCollESteenN, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002;325(7370):941.
15.
KeeffeBSubramanianUTierneyWM, et al. Provider response to computer-based care suggestions for chronic heart failure. Med Care. 2005;43(5):461–5.
16.
LitvinCBOrnsteinSMWessellAMNemethLSNietertPJ. Adoption of a clinical decision support system to promote judicious use of antibiotics for acute respiratory infections in primary care. Int J Med Inform. 2012;81(8):521–6.
17.
LiberatiEGRuggieroFGaluppoL, et al. What hinders the uptake of computerized decision support systems in hospitals? A qualitative study and framework for implementation. Implement Sci. 2017;12(1):113.
18.
SieckWRArkesHR. The recalcitrance of overconfidence and its contribution to decision aid neglect. J Behav Decis Mak. 2005;18(1):29–53.
19.
ArkesHRDawesRMChristensenC. Factors influencing the use of a decision rule in a probabilistic task. Organ Behav Hum Decis Process. 1986;37:93–110.
20.
KaplanB. Evaluating informatics applications—clinical decision support systems literature review. Int J Med Inform. 2001;64(1):15–37.
DawesRM. House of Cards: Psychology and Psychotherapy Built on Myth. New York: Free Press; 1994.
23.
ArkesHRShafferVAMedowMA. Patients derogate physicians who use a computer-assisted diagnostic aid. Med Decis Making. 2007;27(2):189–202.
24.
ShafferVAProbstCAMerkleECArkesHRMedowMA. Why do patients derogate physicians who use a computer-based diagnostic support system?Med Decis Making. 2013;33(1):108–18.
25.
GallagherTHWatermanADEbersAGFraserVJLevinsonW. Patients’ and physicians’ attitudes regarding the disclosure of medical errors. JAMA. 2003;289(8):1001–7.
26.
PezzoMVPezzoSP. Physician evaluation after medical errors: Does having a computer decision aid help or hurt in hindsight?Med Decis Making. 2006;26(1):48–56.
27.
ArkesHRShafferVAMedowMA. The influence of a physician’s use of a diagnostic decision aid on the malpractice verdicts of mock jurors. Med Decis Making. 2008;28(2):201–8.
28.
WolfJR. Do IT Students Prefer Doctors Who Use IT? Working Paper. Normal: Illinois State University; 2014. p. 1–34.
29.
JoshiVWolfJ. Do Indian physicians derogate peers who use computer-based diagnostic aids? In: Conference Presentation. AMCIS 2011 Proceedings. 2011. p. Paper 298.
30.
LoweDJReckersP. The influence of outcome effects, decision aid usage, and intolerance of ambiguity on evaluations of professional audit judgement. Int J Accounting Audit Peformance Eval. 1997;1(December1995):43–58.
31.
GomaaMIHuntonJERoseJM. The effect of control risk and litigation risk on decision aid reliance. Int J Appl Decis Sci. 2008;1(1):80–106.
32.
DiabDLPuiSYYankelevichMHighhouseS. Lay perceptions of selection decision aids in US and non-US samples. Int J Select Assess. 2011;19(2):209–16.
33.
ChanSHLoweDJYaoLJ. The legal implications of auditors using a fraud decision aid vs. professional judgment. J Forens Account. 2008;9(1):63–82.
34.
LoweDJReckersPWhitecottonSM. The effects of decision-aid use on jurors’ evaluations of auditor liability. Rev Acct Stud. 2002;77(1):185–202.
35.
BlackhamJClaridgeTBengerJR. Can patients apply the Ottawa ankle rules to themselves?Emerg Med. 2008;25(11):750–1.
36.
StiellIGGreenbergGHMcKnightRDNairRCMcDowellIWorthingtonJR. A study to develop clinical decision rules for the use of radiography in acute ankle injuries. Ann Emerg Med. 1992;21(4):384–90.
37.
AndersonJCJenningsMMKaplanSEReckersPMJ. The effect of using diagnostic decision aids for analytical procedures on judges’ liability judgments. J Account Public Policy. 1995;14(1):33–62.
38.
Chory-AssadRMTamboriniR. Television exposure and the public’s perceptions of physicians. J Broadcast Electron Media. 2003;47(2):197–215.
39.
CarlsonMSDesaiMDruryJLKwakHYancoHA. Identifying factors that influence trust in automated cars and medical diagnosis systems. In: The Intersection of Robust Intelligence and Trust in Autonomous Systems: Papers from the AAAI Spring Symposium. 2011. p. 20–7.
40.
PezzoMVBecksteadJW. Patients prefer artificial intelligence to a human provider, provided the AI is better than the human: a commentary on Longoni, Bonezzi and Morewedge (2019). Judgm Decis Making. 2020;15(3):443.
41.
PezzoMVBecksteadJW. Algorithm aversion is too often presented as though it were non-compensatory: a reply to Longoni et al. (2020). Judgm Decis Mak. 2020;15(3):449–51.
42.
EstevaAKuprelBNovoaRA, et al. Corrigendum: dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;546(7660):686.
43.
SilverDHuangAMaddisonCJ, et al. Mastering the game of Go with deep neural networks and tree search. Nature. 2016;529(7587):484–9.
44.
WengSFRepsJKaiJGaribaldiJMQureshiN. Can machine-learning improve cardiovascular risk prediction using routine clinical data?PLoS One. 2017;12(4):e0174944.
45.
StricklandE. IBM Watson, heal thyself: how IBM overpromised and underdelivered on AI health care. IEEE Spectrum. 2019;56(4):24–31.
46.
YeomansMShahAMullainathanSKleinbergJ. Making sense of recommendations. J Behav Decis Mak. 2019;29:84.
47.
WellerJADieckmannNFTuslerMMertzCKBurnsWJPetersE. Development and testing of an abbreviated numeracy scale: a Rasch analysis approach. J Behav Decis Mak. 2013;26(2):198–212.
48.
PezzoMVQuinnMSMachacekMGMarchalC. Does Hindsight Bias Mediate Outcome Bias in Judgments of Decision Quality and Negligence? Working Paper. University of South Florida St. Petersburg; 2020.
49.
DietvorstBJSimmonsJPMasseyC. Algorithm aversion: people erroneously avoid algorithms after seeing them err. J Exp Psychol Gen. 2015;144(1):114–26.
50.
MarcusGDavisE. Rebooting AI: Building Artificial Intelligence We Can Trust. New York: Knopf Doubleday; 2019.
51.
KatsikopoulosKVSimsekOBuckmannMGigerenzerG. Classification in the Wild: The Science and Art of Transparent Decision Making. Cambridge (MA): MIT Press; 2021.
52.
Barredo ArrietaADíaz-RodríguezNDel SerJ, et al. Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf Fusion. 2020;58:82–115.
53.
RudinCRadinJ. Why are we using black box models in AI when we don’t need to? A lesson from an explainable AI competition. Harvard Data Science Review. 2019;1(2). Available from: https://hdsr.mitpress.mit.edu/pub/f9kuryi8
54.
KennedyJAAndersonCMooreDA. When overconfidence is revealed to others: testing the status-enhancement theory of overconfidence. Organ Behav Hum Decis Process. 2013;122(2):266–79.
55.
Lima de MirandaKDetlefsenLStolpeM. Overconfidence and Hygiene Non-compliance in Hospitals. Kiel Working Paper; 2020. Report No.: 2156. Available from: https://www.econstor.eu/handle/10419/222296
56.
MeyerANDPayneVLMeeksDWRaoRSinghH. Physicians’ diagnostic accuracy, confidence, and resource requests: a vignette study. JAMA Intern Med. 2013;173(21):1952–8.
57.
BrezisMOrkin-BedolachYFinkDKidermanA. Does physician’s training induce overconfidence that hampers disclosing errors?J Patient Saf. 2019;15(4):296–8.
58.
HoBLiuE. Does sorry work? The impact of apology laws on medical malpractice. J Risk Uncertain. 2011;43(2):141.
59.
HamiltonJGGenoff GarzonMWestermanJS, et al. “A tool, not a crutch”: patient perspectives about IBM Watson for oncology trained by Memorial Sloan Kettering. J Oncol Pract. 2019;JOP1800417.
60.
JussupowESpohrerKHeinzlAGawlitzaJ. Augmenting medical diagnosis decisions? An investigation into physicians’ decision-making process with artificial intelligence. Inform Syst Res. 2021. Available from: https://doi.org/10.1287/isre.2020.0980
61.
GomaaMGomaaA. The effect of time pressure, task complexity, and litigation risk on auditors reliance on decision aids. In: Proceedings of the Hawai’i Accounting Research Conference. Shidler College, University of Hawaii at Manoa; January3–5, 2019. p. 1–33.
62.
ParikhRBObermeyerZNavatheAS. Regulation of predictive analytics in medicine. Science. 2019;363(6429):810–2.
Hernandez-BoussardTBozkurtSIoannidisJPAShahNH. MINIMAR (MINimum Information for Medical AI Reporting): developing reporting standards for artificial intelligence in health care. J Am Med Inform Assoc. 2020;27(12):2011–5.
65.
BatesDWKupermanGJWangS, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10(6):523–30.
66.
BabioneJNOcampoWHaubrichS, et al. Human-centred design processes for clinical decision support: a pulmonary embolism case study. Int J Med Inform. 2020;142:104196.
67.
ChanTMMercuriMTurcotteMGardinerESherbinoJde WitK. Making decisions in the era of the clinical decision rule: how emergency physicians use clinical decision rules. Acad Med. 2020;95(8):1230–7.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
