Likelihood Systems Can Improve Hit Rates in Low-Prevalence Visual Search Over Binary Systems

Abstract

Objective

To study the performance consequences of binary versus likelihood decision support systems in low-prevalence visual search.

Background

Hit rates in visual search are often low if target prevalence is low, an issue that is relevant for numerous real-world visual search tasks (e.g., luggage screening and medical imaging). Given that binary decision support systems produce many false alarms at low prevalence, they have often been discounted as a solution to this low-prevalence problem. By offering additional information about the certainty of target-present indications through splitting these into warnings and alarms, likelihood-based systems could potentially boost hit rates without raising the number of false alarms.

Method

We used a simulated medical search task with low target prevalence in a paradigm where participants sequentially uncovered parts of the stimulus with their mouse. In two sessions, participants completed the task either while being supported by a binary or a likelihood system.

Results

Hit rates were higher when interacting with the likelihood systems than with the binary system, at no cost of higher false alarms.

Conclusion

Likelihood systems are a promising way to tackle the low-prevalence problem, and might further be an effective means to make systems more transparent.

Application

Simple-to-process information about system certainty for each case might be a solution to low hit rates in domains with low target prevalence, such as radiology.

Keywords

human–automation interaction,visual search,decision making,trust in automation,compliance and reliance,radiology and medical imaging

Get full access to this article

View all access options for this article.

References

Aberle

D. R.

Adams

A. M.

Berg

C. D.

Black

W. C.

Clapp

J. D.

Fagerstrom

R. M.

Gareen

I. F.

Gatsonis

Marcus

P. M.

Sicks

J. D.

(2011). Reduced lung-cancer mortality with low-dose computed tomographic screening. New England Journal of Medicine, 365(5), 395–409. https://doi.org/10.1056/nejmoa1102873

Arrieta

A. B.

Diaz-Rodriguez

Del Ser

Bennetot

Tabik

Barbado

Garcia

Gil-Lopez

Molina

Benjamins

Chatila

Herrera

(2020). Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58(1), 82–115. https://doi.org/10.1016/j.inffus.2019.12.012

Bartlett

M. L.

McCarley

J. S.

(2017). Benchmarking aided decision making in a signal detection task. Human Factors, 59(6), 881–900. https://doi.org/10.1177/0018720817700258

Bartlett

M. L.

McCarley

J. S.

(2019). No effect of cue format on automation dependence in an aided signal detection task. Human Factors, 61(2), 169–190. https://doi.org/10.1177/0018720818802961

Bejnordi

B. E.

Veta

van Diest

P. J.

van Ginneken

Karssemeijer

Litjens

van der Laak

J. A. W. M.

Hermsen

Manson

Q. F.

Balkenhol

Geessink

Stathonikos

van Dijk

M. C.

Bult

Beca

Beck

A. H.

Wang

Khosla

Venâncio

(2017). Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA, 318(22), 2199–2210. https://doi.org/10.1001/jama.2017.14585

Berbaum

Franken

Caldwell

Shartz

Madsen

(2019). Satisfaction of search in radiology. The handbook of medical image perception and techniques, 2, 121–166.

Boskemper

M. M.

Bartlett

M. L.

McCarley

J. S.

(2021). Measuring the efficiency of automation-aided performance in a simulated baggage screening task. Human Factors, 64(6), 945–961. https://doi.org/10.1177/0018720820983632

Bricolo

Gianesini

Fanini

Bundesen

Chelazzi

(2002). Serial attention mechanisms in visual search: A direct behavioral demonstration. Journal of Cognitive Neuroscience, 14(7), 980–993. https://doi.org/10.1162/089892902320474454

Bruls

R. J. M.

Kwee

R. M.

(2020). Workload for radiologists during on-call hours: Dramatic increase in the past 15 years. Insights into Imaging, 11(1), 121. https://doi.org/10.1186/s13244-020-00925-z

10.

Burgess

A. E.

Jacobson

F. L.

Judy

P. F.

(2001). Human observer detection experiments with mammograms and power-law noise. Medical Physics, 28(4), 419–437. https://doi.org/10.1118/1.1355308

11.

Buser

Sterchi

Schwaninger

(2020). Why stop after 20 minutes? Breaks and target prevalence in a 60-minute x-ray baggage screening task. International Journal of Industrial Ergonomics, 76(1), Article 102897. https://doi.org/10.1016/j.ergon.2019.102897

12.

Bustamante

E. A.

Bliss

J. P.

(2005). Effects of workload and likelihood information on human response to alarm signals. International Symposium on Aviation Psychology.

13.

Chavaillaz

Schwaninger

Michel

Sauer

(2018). Automation in visual inspection tasks: X-Ray luggage screening supported by a system of direct, indirect or adaptable cueing with low and high system reliability. Ergonomics, 61(10), 1395–1408. https://doi.org/10.1080/00140139.2018.1481231

14.

Coolen

A. M. P.

Voogd

A. C.

Strobbe

L. J.

Louwman

M. W. J.

Tjan-Heijnen

V. C. G.

Duijm

L. E. M.

(2018). Impact of the second reader on screening outcome at blinded double reading of digital screening mammograms. British Journal of Cancer, 119(4), 503–507. https://doi.org/10.1038/s41416-018-0195-6

15.

Cunningham

C. A.

Drew

Wolfe

J. M.

(2016). Analog computer-aided detection (CAD) information can be more effective than binary marks. Attention, Perception, & Psychophysics, 79(2), 679–690. https://doi.org/10.3758/s13414-016-1250-0

16.

de Leeuw

J. R.

Gilbert

R. A.

Luchterhandt

(2023). Jspsych: Enabling an open-source collaborative ecosystem of behavioral experiments. Journal of Open Source Software, 8(85), 5351. https://doi.org/10.21105/joss.05351

17.

Donk

van Zoest

(2008). Effects of salience are short-lived. Psychological Science, 19(7), 733–739. https://doi.org/10.1111/j.1467-9280.2008.02149.x

18.

Dorrius

M. D.

der Weide

M. C. J.-v.

van Ooijen

P. M. A.

Pijnappel

R. M.

Oudkerk

(2011). Computer-aided detection in breast MRI: A systematic review and meta-analysis. European Radiology, 21(8), 1600–1608. https://doi.org/10.1007/s00330-011-2091-9

19.

Drew

Carissa Delos Reyes

Moher

(2022). Incorrect computer aided detection (cad) marks lead to early quitting: A potential mechanism for poor cad performance in clinical practice. Journal of Vision, 22(14), 3697. https://doi.org/10.1167/jov.22.14.3697

20.

Drew

Cunningham

Wolfe

J. M.

(2012). When and why might a computer-aided detection (CAD) system interfere with visual search? An eye-tracking study. Academic Radiology, 19(10), 1260–1267. https://doi.org/10.1016/j.acra.2012.05.013

21.

Evans

K. K.

Birdwell

R. L.

Wolfe

J. M.

(2013). If you don’t find it often, you often don’t find it: Why some cancers are missed in breast cancer screening. PLoS One, 8(5), Article e64366. https://doi.org/10.1371/journal.pone.0064366

22.

Faul

Erdfelder

Lang

A.-G.

Buchner

(2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146

23.

Gandomkar

Mello-Thoms

(2019). Visual search in breast imaging. British Journal of Radiology, 92(1102), Article 20190057. https://doi.org/10.1259/bjr.20190057

24.

Getty

D. J.

Swets

J. A.

Pickett

R. M.

Gonthier

(1995). System operator response to warnings of danger: A laboratory investigation of the effects of the predictive value of a warning on human response time. Journal of Experimental Psychology: Applied, 1(1), 19–33. https://doi.org/10.1037/1076-898x.1.1.19

25.

Hofer

Schwaninger

(2005). Using threat image projection data for assessing individual screener performance. WIT Transactions on The Built Environment, 82(1), 417–426. https://doi.org/10.2495/safe050411

26.

Horowitz

T. S.

(2017). Prevalence in visual search: From the clinic to the lab and back again. Japanese Psychological Research, 59(2), 65–108. https://doi.org/10.1111/jpr.12153

27.

Huegli

Merks

Schwaninger

(2020). Automation reliability, human–machine system performance, and operator compliance: A study with airport security screeners supported by automated explosives detection systems for cabin baggage screening. Applied Ergonomics, 86, Article 103094. https://doi.org/10.1016/j.apergo.2020.103094

28.

Huegli

Merks

Schwaninger

(2022). Benefits of decision support systems in relation to task difficulty in airport security x-ray screening. International Journal of Human-Computer Interaction, 39(19), 3830–3845. https://doi.org/10.1080/10447318.2022.2107775

29.

Langer

Oster

Speith

Hermanns

Kästner

Schmidt

Sesing

Baum

(2021). What do we want from explainable artificial intelligence (XAI)? – a stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artificial Intelligence, 296, Article 103473. https://doi.org/10.1016/j.artint.2021.103473

30.

Leonard

C. J.

Egeth

H. E.

(2008). Attentional guidance in singleton search: An examination of top-down, bottom-up, and intertrial factors. Visual Cognition, 16(8), 1078–1091. https://doi.org/10.1080/13506280701580698

31.

Madhavan

Gonzalez

Brown

J. R.

(2013). The impact of target base rate on training and transfer of learning in airline luggage screening: An examination of three base rate scenarios. Applied Cognitive Psychology, 27(2), 263–273. https://doi.org/10.1002/acp.2905

32.

Madhavan

Gonzalez

Lacson

F. C.

(2007). Differential base rate training influences detection of novel targets in a complex visual inspection task. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 51(4), 392–396. https://doi.org/10.1177/154193120705100451

33.

Matzen

L. E.

Stites

M. C.

Gastelum

Z. N.

(2021). Studying visual search without an eye tracker: An assessment of artificial foveation. Cognitive Research: Principles and Implications, 6(1), 45. https://doi.org/10.1186/s41235-021-00304-2

34.

McDonald

R. J.

Schwartz

K. M.

Eckel

L. J.

Diehn

F. E.

Hunt

C. H.

Bartholmai

B. J.

Erickson

B. J.

Kallmes

D. F.

(2015). The effects of changes in utilization and technological advancements of cross-sectional imaging on radiologist workload. Academic Radiology, 22(9), 1191–1198. https://doi.org/10.1016/j.acra.2015.05.007

35.

McKinney

S. M.

Sieniek

Godbole

Godwin

Antropova

Ashrafian

Back

Chesus

Corrado

G. S.

Darzi

Etemadi

Garcia-Vicente

Gilbert

F. J.

Halling-Brown

Hassabis

Jansen

Karthikesalingam

Kelly

C. J.

King

Shetty

(2020). International evaluation of an AI system for breast cancer screening. Nature, 577(7788), 89–94. https://doi.org/10.1038/s41586-019-1799-6

36.

Meyer

(2004). Conceptual issues in the study of dynamic hazard warnings. Human Factors, 46(2), 196–204. https://doi.org/10.1518/hfes.46.2.196.37335

37.

Morey

R. D.

(2008). Confidence intervals from normalized data: A correction to cousineau (2005). Tutorials in Quantitative Methods for Psychology, 4(2), 61–64. https://doi.org/10.20982/tqmp.04.2.p061

38.

Mosier

K. L.

Manzey

(2020). Humans and automated decision aids: A match made in heaven? In Mouloua

Hancock

P. A.

(Eds.), Human performance in automated and autonomous systems: Current theory and methods (pp. 19–42). CRC Press.

39.

Pashler

(1987). Detecting conjunctions of color and form: Reassessing the serial search hypothesis. Perception & Psychophysics, 41(3), 191–201. https://doi.org/10.3758/bf03208218

40.

Pithayarungsarit

Rieger

Onnasch

Roesler

(2024). The pop-out effect of rarer occurring stimuli shapes the effectiveness of AI explainability. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 68(1), 352–358. https://doi.org/10.1177/10711813241261284

41.

Rieger

Heilmann

Manzey

(2021). Visual search behavior and performance in luggage screening: Effects of time pressure, automation aid, and target expectancy. Cognitive Research: Principles and Implications, 6(1), 12. https://doi.org/10.1186/s41235-021-00280-7

42.

Rieger

Manzey

(2024). Understanding the impact of time pressure and automation support in a visual search task. Human Factors, 66(3), 770–786. https://doi.org/10.1177/00187208221111236

43.

Rieger

Manzey

Meussling

Onnasch

Roesler

(2023). Be careful what you explain: Benefits and costs of explainable AI in a simulated medical task. Computers in Human Behavior: Artificial Humans, 1(2), Article 100021. https://doi.org/10.1016/j.chbah.2023.100021

44.

Rizà Porta

Sterchi

Schwaninger

(2022). How realistic is threat image projection for x-ray baggage screening? Sensors, 22(6), 2220. https://doi.org/10.3390/s22062220

45.

Sargent

Walters

Wickens

(2023). Meta-analysis qualifying and quantifying the benefits of automation transparency to enhance models of human performance. In Human-computer interaction (pp. 243–261). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-35596-7_16

46.

Schwark

Sandry

MacDonald

Dolgov

(2012). False feedback increases detection of low-prevalence targets in visual search. Attention, Perception, & Psychophysics, 74(8), 1583–1589. https://doi.org/10.3758/s13414-012-0354-4

47.

Seong

Bisantz

A. M.

(2008). The impact of cognitive feedback on judgment performance and trust with decision aids. International Journal of Industrial Ergonomics, 38(7-8), 608–625. https://doi.org/10.1016/j.ergon.2008.01.007

48.

Sha

L. Z.

Remington

R. W.

Jiang

Y. V.

(2018). Statistical learning of anomalous regions in complex faux x-ray images does not transfer between detection and discrimination. Cognitive Research: Principles and Implications, 3(1), 48. https://doi.org/10.1186/s41235-018-0144-1

49.

Sorkin

R. D.

Kantowitz

B. H.

Kantowitz

S. C.

(1988). Likelihood alarm displays. Human Factors: The Journal of the Human Factors and Ergonomics Society, 30(4), 445–459. https://doi.org/10.1177/001872088803000406

50.

Sorkin

R. D.

Woods

D. D.

(1985). Systems with human monitors: A signal detection analysis. Human-Computer Interaction, 1(1), 49–75. https://doi.org/10.1207/s15327051hci0101_2

51.

St John

Manes

D. I.

(2002). Making unreliable automation useful. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 46(3), 332–336. https://doi.org/10.1177/154193120204600325

52.

Taylor

J. E. T.

Hilchey

M. D.

Weidler

B. J.

Pratt

(2022). Eliminating the low-prevalence effect in visual search with a remarkably simple strategy. Psychological Science, 33(5), 716–724. https://doi.org/10.1177/09567976211048485

53.

Treisman

A. M.

Gelade

(1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136. https://doi.org/10.1016/0010-0285(80)90005-5

54.

van der Gijp

Ravesloot

C. J.

Jarodzka

van der Schaaf

M. F.

van der Schaaf

I. C.

van Schaik

J. P. J.

ten Cate

T. J.

(2016). How visual search relates to visual diagnostic performance: A narrative systematic review of eye-tracking research in radiology. Advances in Health Sciences Education: Theory and Practice, 22(3), 765–787. https://doi.org/10.1007/s10459-016-9698-1

55.

Wickens

C. D.

(2017). Automation stages & levels, 20 years after. Journal of Cognitive Engineering and Decision Making, 12(1), 35–41. https://doi.org/10.1177/1555343417727438

56.

Wickens

C. D.

Colcombe

(2007). Dual-task performance consequences of imperfect alerting associated with a cockpit display of traffic information. Human Factors, 49(5), 839–850. https://doi.org/10.1518/001872007x230217

57.

Wiczorek

(2017). Investigating users’ mental representation of likelihood alarm systems with different thresholds. Theoretical Issues in Ergonomics Science, 18(3), 221–240. https://doi.org/10.1080/1463922x.2016.1207209

58.

Wiczorek

Manzey

(2014). Supporting attention allocation in multitask environments: Effects of likelihood alarm systems on trust, behavior, and performance. Human Factors, 56(7), 1209–1221. https://doi.org/10.1177/0018720814528534

59.

Wolfe

J. M.

(2021). Guided search 6.0: An updated model of visual search. Psychonomic Bulletin & Review, 28(4), 1060–1092. https://doi.org/10.3758/s13423-020-01859-9

60.

Wolfe

J. M.

Brunelli

D. N.

Rubinstein

Horowitz

T. S.

(2013). Prevalence effects in newly trained airport checkpoint screeners: Trained observers miss rare targets, too. Journal of Vision, 13(3), 33. https://doi.org/10.1167/13.3.33

61.

Wolfe

J. M.

Horowitz

T. S.

Kenner

N. M.

(2005). Cognitive psychology: Rare items often missed in visual searches. Nature, 435(7041), 439–440. https://doi.org/10.1038/435439a

62.

Wolfe

J. M.

Horowitz

T. S.

Van Wert

M. J.

Kenner

N. M.

Place

S. S.

Kibbi

(2007). Low target prevalence is a stubborn source of errors in visual search tasks. Journal of Experimental Psychology: General, 136(4), 623–638. https://doi.org/10.1037/0096-3445.136.4.623

63.

Zirk

Wiczorek

Manzey

(2020). Do we really need more stages? Comparing the effects of likelihood alarm systems and binary alarm systems. Human Factors, 62(4), 540–552. https://doi.org/10.1177/0018720819852023