Human reliance on imperfect automation has been the subject of many laboratory experiments. Across these studies, a diverse set of indices of reliance have been used, even in studies with similar experiment settings. This inconsistency of reliance measures makes the meta-analysis of experimental findings difficult. Moreover, few researchers have rigorously defined the optimal level of automation reliance in their settings, making it difficult to judge the appropriateness of that reliance. This paper attempts to guide researchers in selecting and interpreting measures of automation reliance behavior. We review the reliance analysis methods in existing research and propose four criteria for selecting among them. It is recommended that, where possible, future studies define optimal reliance to make unequivocal judgment about the appropriateness of reliance. In addition, more reliable and insightful conclusions can be obtained through the use of multiple measures.
BagheriN.JamiesonG. A. (2004). The impact of context-related reliability on automation failure detection and scanning behaviour. In Proceedings of the 2004 IEEE International Conference on Systems, Man and Cybernetics (pp. 212–217).
3.
BagheriN. (2004). The effect of automation reliability on operator monitoring. Unpublished master's thesis, University of Toronto, Toronto, Ontario, Canada.
4.
BainbridgeL. (1983). Ironies of automation. Automatica, 19(6), 775–779.
5.
BirosD. P.DalyM.GunschG. (2004). The influence of task load and automation trust on deception detection. Group Decision and Negotiation, 13(2), 173–189.
6.
BisantzA. M.PritchettA. R. (2003). Measuring judgment in complex, dynamic environments: A lens model analysis of collision detection behavior. Human Factors, 45, 266–280.
7.
BrunswikE. (1956). Perception and the representative design of psychological experiments (2nd ed.). Berkeley: University of California Press.
8.
DixonS. R.WickensC. D. (2006). Automation reliability in unmanned aerial vehicle control: A reliance-compliance model of automation dependence in high workload. Human Factors, 48(3), 474–486.
9.
DzindoletM. T.PierceL.PomrankyR.PetersonS.BeckH. (2001). Automation reliance on a combat identification system. In Proceedings of the 45th Annual Meeting of the Human Factors and Ergonomics Society (pp. 532–536. Minneapolis/St. Paul, MN: Human Factors and Ergonomics Society.
10.
EzerN.FiskA. D.RogersW. A. (2007). Reliance on automation as a function of expectation of reliability, cost of verification, and age. In Proceedings of the 51st Annual Meeting of the Human Factors and Ergonomics Society (pp. 6–10. Baltimore, MD: Human Factors and Ergonomics Society.
11.
GalsterS. M.BoliaR. S.RoeM. M.ParasuramanR. (2001). Effects of automated cueing on decision implementation in a visual search task. In Proceedings of the 45th Annual Meeting of the Human Factors and Ergonomics Society (pp. 321–325. Santa Monica, CA: Human Factors and Ergonomics Society.
12.
GreenD. M.SwetsJ. A. (1966). Signal detection theory and psychophysics. New YorkWiley.
13.
LeeJ. D.SeeK. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.
14.
MaltzM.MeyerJ. (2001). Use of warnings in an attentionally demanding detection task. Human Factors, 43(2), 217–226.
15.
MaltzM.ShinarD. (2003). New alternative methods of analyzing human behavior in cued target acquisition. Human factors, 45(2), 281(15) 296.
16.
ManzeyD.BahnerE.HueperA. (2006). Misuse of automated aids in process control: Complacency, automation bias and possible training interventions. In Proceedings of the 50th Annual Meeting of the Human Factors and Ergonomics Society (pp. 220–224). San Francisco, CA: Human Factors and Ergonomics Society.
17.
MeyerJ. (2001). Effects of warning validity and proximity on responses to warnings. Human factors, 43(4), 563–572.
18.
MorayN.InagakiT. (2000). Attention and complacency. Theoretical Issues in Ergonomics Science, 1(4), 354–365.
19.
MosierK. L.SkitkaL. J.HeersS.BurdickM. (1998). Automation bias: Decision making and performance in high-tech cockpits. The International journal of aviation psychology, 8(1), 47–63.
20.
MurrellG. A. (1977). Combination of evidence in a probabilistic visual search and detection task. Organizational Behavior and Human Performance, 18, 3–18.
21.
ParasuramanR.MolloyR.SinghI. (1993). Performance consequences of automation-induced “complacency”. International Journal of Aviation Psychology, 3, 1–23.
22.
ParasuramanR.RileyV. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39, 230–253.
23.
WickensC. D.HollandsJ. G. (2000). Engineering psychology and human performance (3rd ed.). Upper Saddle River, NJ: Prentice Hall.
