A Meta-Analysis of Factors Influencing the Development of Trust in Automation

Abstract

Objective:

We used meta-analysis to assess research concerning human trust in automation to understand the foundation upon which future autonomous systems can be built.

Background:

Trust is increasingly important in the growing need for synergistic human–machine teaming. Thus, we expand on our previous meta-analytic foundation in the field of human–robot interaction to include all of automation interaction.

Method:

We used meta-analysis to assess trust in automation. Thirty studies provided 164 pairwise effect sizes, and 16 studies provided 63 correlational effect sizes.

Results:

The overall effect size of all factors on trust development was ḡ = +0.48, and the correlational effect was $\bar{r}$ = +0.34, each of which represented medium effects. Moderator effects were observed for the human-related (ḡ = +0.49; $\bar{r}$ = +0.16) and automation-related (ḡ = +0.53; $\bar{r}$ = +0.41) factors. Moderator effects specific to environmental factors proved insufficient in number to calculate at this time.

Conclusion:

Findings provide a quantitative representation of factors influencing the development of trust in automation as well as identify additional areas of needed empirical research.

Application:

This work has important implications to the enhancement of current and future human–automation interaction, especially in high-risk or extreme performance environments.

Keywords

human–automation interaction human–robot interaction meta-analysis trust

Introduction

Trust is a key element in the development of effective relationships, including interpersonal relationships as well as human–technology relationships. In particular, trust is an overarching concern that affects system effectiveness, especially as it relates to safety, performance, and use rate (Lee & See, 2004). With this sentiment in mind, trust has become a focal concern within the development and integration of complex autonomous systems. These systems are being designed and built to possess increased levels of independent intelligence and decision-making authority, capacities that will be enacted in uncertain, unplanned circumstances.

The principle that we promulgate—no trust, no use—is central to designing automated and semiautomated technologies and is anticipated to have a similar, if not a greater, impact on the design of future autonomous systems. This principle can be generalized into the somewhat polemic assertion that rates of usage are directly and proportionally related to the degree of trust expressed (Lee & See, 2004). However, there are many questions as to these evolving human–technology trust relationships that remain to be understood.

Trust can be viewed as the tripartite expression of a relational property. The first expression is stable over time and is composed of an individual’s overall, long-term propensity to trust in general (Jarvenpaa, Knoll, & Leidner, 1998; Mayer, Davis, & Schoorman, 1995; Rotter, 1967). On top of this bedrock trust level is erected a more transient or state expression of trust (McAllister, 1995; Merritt & Ilgen, 2008; Schaefer, 2013). Momentary trust is a reactive and somewhat volatile property. As a state expression, it is primarily responsive to the immediate ambient conditions. The third dimension of trust here is how these expressions change and evolve across time (Desai, Kaniarasu, Medvedev, Steinfeld, & Yanco, 2013; Schaefer, 2013; Wilson, Straus, & McEvily, 2006). The process of trust development and trust loss are necessarily dynamic in nature. Figure 1 provides a conceptual organization of the trust process (Hancock, Billings, & Schaefer, 2011). It includes trust development (dotted box), trust outcomes (e.g., reliance, compliance, use, etc.), and trust calibration. The scope of our present meta-analysis focuses principally on the first part of this process: how trust develops in the context of human–system interaction.

Figure 1.

A conceptual organization of trust influences highlighting trust development (adapted from Hancock, Billings, & Schaefer, 2011). One asterisk (*) represents propensity to trust, two asterisks (**) represent the state-based trust that is measured at a specific point, and the dotted box represents the trust development part of the process.

To begin, we examine the initial effects of the (a) human, (b) technology partner (automation or robot), and (c) environment. We look to understand their associated antecedent influences on trust development. Hancock, Billings, Schaefer, Chen, et al. (2011) have previously addressed this same issue but specifically constrained to human–robot interaction (HRI). Using extant studies, the former authors identified critical gaps through their meta-analysis focused on trust development in human–robot relationships. Derived from a synthesis of extant, quantitative studies, Hancock, Billings, Schaefer, Chen, et al. (2011) developed the three-factor model of human–robot trust. This model emphasized the triad of human, robot, and environmental factors and their associated antecedents of trust (Figure 2). One key quantitative outcome highlighted the fact that robot attributes and performance capabilities had the most substantive effect (+.67) on trust development. Yet Hancock, Billings, Schaefer, Chen, et al. (2011) also identified numerous gaps in the literature, especially those related to the human operator and environmental factors.

Figure 2.

The three-factor model of human–robot trust with meta-analytic findings (Hancock, Billings, Schaefer, Chen, et al., 2011).

Although interacting with robots contains certain unique dimensions (e.g., degree of the robot’s anthropomorphic character), understanding human trust development with all of automation promises to render a much more complete picture of the general issue. This elaboration is critical as it will serve to inform developing autonomy in future technological systems. More studies are available on automation in general, as opposed to the specific subset of HRI. Further, the influence of the stable characteristics of the individual on trust development, which rendered scant information in the robotic realm, generates a much larger empirical basis when we examine all of automation. Thus, the specific analysis of Hancock, Billings, Schaefer, Chen, et al. (2011) can be seen as a special case of the present, more broad, and inclusive evaluation. Reflexively, however, we expect that current results can also serve to guide and advance work in other specific subdomains of human–automation interaction (HAI), including robotic systems.

For this work, a meta-analytic approach was taken to extend our knowledge of the factors influencing trust development. Meta-analysis permits a quantitative summary of present scientific understanding predicated upon the reported results from qualifying empirical investigations. Although there are constraints upon this technique (Borenstein, Hedges, Higgins, & Rothstein, 2009), and it has some implied but now explicit stopping rules (Szalma & Hancock, 2011), such a procedure has served to significantly advance the solidity of our comprehension, especially over qualitative, ad hoc reviews of literature.

The relevance of this present work lies in the drastic increase of automation that continues to penetrate into many traditional human systems. As complex automation is now being produced and continues to approach increasingly more advanced intelligent, autonomous systems, the value of any such system resides not in the total replacement of a human controller but rather in the capacity for human–system collaboration. Human–machine partnerships are especially important in so-called dirty, dangerous, and dull conditions, in which there is almost a necessary degradation of human efficiency due to factors such as fatigue, workload, stress, distraction, and so on (Chen & Barnes, 2014; Defense Science Board, 2012; Hancock, 2013; Hancock, Desmond, & Matthews, 2012; Hancock & Warm, 1989).

What follows is (a) a brief theory-oriented review of trust in automation, (b) a description of the analytical method of our meta-analytic procedure, (c) major results and moderator analyses from that meta-analysis, and (d) discussion of the implications of these findings for all of HAI. We also address the limitations of our present work as well as indicate future research needs that are required to advance the understanding of trust.

Review of Literature

Automation is broadly defined as the “machine execution of functions” (Parasuraman, Sheridan, & Wickens, 2000, p. 286). More specifically, it is “technology that actively selects data, transforms information, makes decisions, or controls processes” (Lee & See, 2004, p. 50). For the purposes of the present work, automation refers to the full spectrum of capabilities, from preprogrammed automated systems (Frost, 2010) to autonomous systems that have the ability to monitor the state of the world, diagnose or analyze deviations in the actual and desired state, and generate, select, and execute plans (Krogmann, 1999). Truszkowski and colleagues (2009) further suggested that autonomous systems require a degree of “self-governance” or “self-direction.” For the purpose of greater contrastive clarity, we offer the postulate that automated systems are “designed to accomplish a specific set of largely deterministic steps (often in a repeated pattern) in order to achieve one of an envisaged and finite set of predefined goals,” whereas autonomous systems “learn and evolve through the input of operational and contextual information and thus their actions necessarily become more indeterminate across time” (Hancock, 2016).

Conceptual models and taxonomies of trust in automation have been developed over time, of which many are theoretically driven. Therefore, as more relevant evidence becomes available, these models continue to adapt. In the field of trust in automation, these taxonomies were derived from the interpersonal trust domain. For example, Muir (1994) discussed trust in terms of predictability, dependability, and faith. For many years this taxonomy was the foundation on which empirical studies were established. As more research became available over the following decade, Lee and See (2004) updated this taxonomy to highlight the elements associated with performance (e.g., competence, ability, reliability), process (e.g., persistence, integrity, predictability, dependability), and purpose (e.g., loyalty, benevolence, faith). This updated taxonomy was theoretically driven by both interpersonal literature as well as HAI studies. Since that time, additional theoretical and empirical studies have been conducted, and additional antecedents of trust have been identified. These efforts have not only led to a broader foundation on which trust is founded, but from this work, multiple descriptions of differing types of trust have emerged (Hoff & Bashir, 2015; Merritt & Ilgen, 2008).

However, there is a need to understand how the current foundation not only theoretically but also empirically affects the way in which trust develops. As the future of autonomy-enabled systems is incorporating independent and interdependent collaboration, it is important to empirically assess the effect of each partner (human and machine) as well as the effect of the interaction environment on trust development. Therefore, for the purpose of this work, the empirically derived three-factor model of human–robot trust (Hancock, Billings, Schaefer, Chen, et al., 2011) is expanded upon to incorporate the currently known or potential antecedents related to the human partner, the automation technology, and the environment in which the interaction occurs.

The Human Element

In studies of automation, it is natural to focus on the technical capabilities of the machine system at hand. Yet, arguably, it is the character and nature of the individual user that, a priori, could exert the most significant influence. As we move toward a state of “individuation” (Hancock, Hancock, & Warm, 2009), it may well be possible to calibrate all interactive automated systems to be propensities of each of their individual users. As a consequence, we look to explore the largest possible range of applicable human characteristics as we seek to make progress on a wide front.

In respect to the foregoing remarks, we have evaluated trust as a user-centric construct in which the human is the one trusting the technology. In this context, the human element is often asserted to be of utmost importance, although such user factors are often overlooked in research or becomes a secondary focus well behind the functional capabilities of the system (Hancock, Billings, Schaefer, Chen, et al., 2011b). However, our study of the larger body of relevant empirical and non-empirical HAI literature looks to feature this element. Here, the factors associated with the human are human traits and states as well as cognitive and emotive dimensions. To provide a visual framework for this evaluation, we show in Figure 3 the adapted model of the specific human-related antecedents of trust developed and elaborated from our prior model (Hancock, Billings, Schaefer, Chen, et al., 2011).

Figure 3.

Comparison of the human-related antecedents from the human–robot interaction model to the human–automation interaction (HAI) model. The gray areas mark items that are unique to each model (e.g., confidence and satisfaction with the automation, stress, fatigue, and expectancy are all new elements included in the HAI model of trust). Dotted lines mark the items shared across the two models.

Traits

Existing evidence suggests that a number of stable human traits and demographic characteristics can affect how each individual responds to automation. These traits include age, gender, ethnicity, and personality. However, the power and directionality of these various relationships have not been consistent across differing studies, as they appear to vary as a function of the characteristics of the specific automated task at hand (Szalma & Taylor, 2011).

There is some evidence that an individual’s age can be linked to the types of systems that are trusted and thus influences the likelihood that the automation will be used. Research studies specific to vehicle automation (e.g., driver warning systems) and decision aids (e.g., medication management) both suggest that older adults were more likely to trust the automation than younger adults (Donmez, Boyle, Lee, & McGhee, 2006; Ho, Wheatley, & Scialfa, 2005; Kircher & Thorslund, 2009). However, it is important to point out that empirical literature is very largely limited to only these two areas of automation research.

The importance of the relationship between gender, ethnicity, trust propensity, personality traits, and trust has been discussed across multiple domains (Schaefer, 2013). However, these lines of research have yet to be thoroughly explored in the automation trust literature. In one study, Merritt and Ilgen (2008) used an X-ray weapons screening task and found that these antecedents were related to an individual’s propensity to trust machines or his or her initial trust in an automated decision aid but not posttask trust.

States

The relationship between automation and various dynamic human states, including stress level (Cottrell & Barton, 2013; Finomore, Matthews, Shaw, & Warm, 2009), fatigue (Reichenbach, Onnasch, & Manzey, 2011), and attentional control (Chen & Terrence, 2009), are relatively well-researched areas of HAI. For example, several studies showed that operators with lower attentional control interacted differently with, and rely more heavily on, automated systems than those with higher attentional control (Chen & Barnes, 2012; Chen & Terrence, 2009). Studies also suggest that there may be a direct relationship between mood and affect on trust development (Merritt, 2011; Stokes et al., 2010).

Cognitive factors

The cognitive factors have been suggested to include (a) an understanding of how the automation works, (b) self-perceived ability to use that automation, and (c) expectancy concerning the automation (Ashleigh & Stanton, 2001; Lee & See, 2004; Madhavan & Wiegmann, 2007). Understanding how the automation works is influenced by the ease of learning to use and interact with the automation as well as prior experience with the same or similar automated systems. Many studies affirm that trust is something that builds over time, and past experiences with technology—specifically, automated systems—have a direct impact on such trust development (Merritt & Ilgen, 2008; Rajaonah, Anceaux, & Vienne, 2006; Sullivan, Tsimhoni, & Bogard, 2008). However, investigations into ease of learning and trust have yielded only mixed results. Whereas one study (Wang, Jamieson, & Hollands, 2011) indicated that trust was negatively correlated with ease of learning, others have shown that a novice operator’s trust grows along a typical learning curve with training (Muir & Moray, 1996).

The self-perceived ability to use the automation is influenced by efficacy (Igbaria & Iivari, 1995), workload (Wang et al., 2011), and expertise (Manzey et al., 2009). One frequently expressed purpose of automation is to reduce workload, and this goal has been shown to be successful in both mental workload (Wiegmann, Eggman, ElBardissi, Parker, & Sundt, 2010) and in the implementation of action (Yang, Yenn, & Lin, 2009). Increases in mental workload associated with the automated systems have been shown to lead to degradations of trust in combat identification tasks (Wang et al., 2011). Further, mental workload has also been directly tied to expertise with automation. For example, in a surgical task using image-guided navigation, inexperienced users reported increased time pressure and mental demand when using the systems (Manzey et al., 2009). Overall, increased expertise with a system generally enhances trust development (Rajaonah et al., 2006). However, allied research has shown that a higher experience level may impair an operator’s ability to monitor for unanticipated states in highly reliable systems (Bailey, Scerbo, Freeman, Mikulka, & Scott, 2006).

Finally, expectancy is directly related to a person’s mental model (MM) concerning automation, including its perceived usefulness (Spain & Bliss, 2008), expectations of the automation or its reputation (de Vries & Midden, 2008), and the perceived benefits of that automation (Lee, Kim, & Kim, 2007). Keil, Beranek, and Konsynski (1995) found that the perceived usefulness of an automated system is often dictated by a function of the task or the fit of tool. The perceived benefit of the automation can influence whether or not the automation will be trusted (similar to software systems; Lee et al., 2007). Further, the reputation of the system may affect operator expectations. For example, in an experiment involving a planning aid, de Vries and Midden (2008) found trust to be slightly higher when the aid was known to have a positive reputation.

Emotive factors

Emotive factors can be organized based on the framework proposed by Ashleigh and Stanton (2001), being (a) confidence in the automation, (b) attitudes toward automation (where respect is classed as a type of attitude), (c) commitment to the automation (including feelings of satisfaction), and (d) comfort with automation (a more directed emotion-based term than teamwork). Confidence has often been used as a synonym for trust (e.g., evaluation of a navigation system for assisted surgery; Strauss et al., 2006), and thus we have included this term in our assessment as an antecedent of trust. This term is not to be confused with self-confidence, which is often considered to be a separate construct from trust but one that also in part influences reliance on the automation (Lee & Moray, 1994). Schaefer et al. (2014) have provided additional discussion on the relationship between trust and self-confidence.

Attitudes toward automation influence overall awareness of system behavior. For example, positive attitudes and emotions (e.g., happiness) influence trust, liking, and reliance on a system; of course, they may also lead to overreliance (Bailey et al., 2006; Merritt, Heimbaugh, LaChapell, & Lee, 2012). Conversely, errors or difficulty accessing information from the system often lead to negative attitudes (Ardern-Jones, Hughes, Rowe, Mottram, & Green, 2009) and subsequent disuse (Gao, Lee, & Zhang, 2006). Commitment is discussed within the literature in terms of feelings of satisfaction. As is the case in many other applications, satisfaction influences human perception of automation. Here, satisfaction with the automation is fostered by the quality of information and service that the automation provides (Lee et al., 2007). Most importantly, satisfaction has been shown to be positively correlated with trust development with both vehicle automation (Donmez et al., 2006) and combat identification (Wang et al., 2011).

Whether or not a user is comfortable with an automated aid can influence his or her perception of the aid and, ultimately, whether or not the user trusts the aid. Therefore, the level of comfort with automation may be dependent on familiarity with, and proximity to, the automation (Nakada, 1997; van den Broek & Westerink, 2009). Comfort is also contingent on similarity of intent (Verberne, Ham, & Midden, 2012) and the degree of control the automation has over tactics and strategy (Ward, 2000). The direct relationship between comfort and trust is one that has yet to be fully articulated.

The Automation

Hancock, Billings, Schaefer, Chen, et al. (2011) identified the moderating factors associated with robotic trust as attribute-based and performance-based characteristics. In the present review of the larger automation literature, those moderating factors were updated to include features and capabilities, respectively. Our larger survey here also pointed to the gaps in the HRI trust literature related to error and communication.

Features of the automation

Feature-based antecedents of trust include the level of automation (LOA), mode of automation, aesthetics, and mode of communication. The LOA relates to the amount of control the automation is designed to exercise (Parasuraman et al., 2000). Existing evidence suggests that there may not be a simple direct relationship between trust and LOA (Rani, Sinclair, & Case, 2000). However, in general, when the user perceives the LOA as “good,” it results in a greater trust than one that is perceived as “poor” or “ambiguous” (Merritt et al., 2012). Also, the combination of multiple types of automation for adaptive assistance tends to be more trusted than lower LOAs (Cai & Lin, 2012; Cai, Lin, & Cheng, 2012).

The mode of automation may be fixed, adjustable (e.g., set by the operator), or adaptive (e.g., accounts for operator’s state; see Hancock & Chignell, 1987, 1988; van den Broek & Westerink, 2009). Individuals exhibit greater trust in automation that provides some level of collaboration. For example, users typically trust manually adjustable automation when it provides explicit control (i.e., the human has the authority over system function allocation), compared to implicit control (i.e., the system is given the authority; Moray, Inagaki, & Itoh, 2000; Sauer, Nickel, & Wastell, 2013). Looije, Neerincx, and Cnossen (2010) have also reported that individuals prefer automation that can learn, recognize, and respond to personality differences (see also Skjerve & Skraaning, 2004).

Aesthetic appearance has been shown to be related to both automation likeability and use (Li & Yeh, 2010; Nass & Moon, 2000). Higher levels of anthropomorphism significantly enhance trust development in complex systems (e.g., computer agents, robotics; Pak, Fink, Price, Bass, & Sturre, 2012). Perhaps most importantly, physical form should meet the user’s expectations for functional capability (Goetz, Kiesler, & Powers, 2003). This sentiment is directly tied to the mode of communication of the automation. Mode of communication refers to the human sensory system that the automation cues either through visual, auditory, or tactile modalities (Hancock, Mercado, Merlo, & Van Erp, 2013). Sarter (2006) reviewed guidelines focusing on multimodal information presentation and the research gaps in the area, emphasizing the need to design the various parameters carefully to ensure their consistency with the user’s personality to maximize trust. For example, specific experimentation on auditory communication demonstrated that operators tend to trust systems with human speech more than those with synthetic speech and showed enhanced performance in the human speech condition (Stedmon et al., 2007).

Capability of the automation

In a similar pattern to that of HRI, the capability of the system is one of the most researched areas concerning automation trust. Trust is often mediated most strongly by the system’s consistency and reliability over time (McLeod, Walker, & Moray, 2005). Experimentation on this particular topic has provided support for the importance of system feedback and reduction of error. What was initially established in our HRI meta-analysis (Hancock, Billings, Schaefer, Chen, et al., 2011) has been solidified here with further empirical studies in the area of general automation.

Trust development has been shown to be directly affected by the appropriateness of cues and feedback. For example, amount of feedback (Muir & Moray, 1996), availability of information (Bitan & Meyer, 2007), type of cuing (e.g., spatial cuing for target detection; Wiegmann, McCarley, Kramer, & Wickens, 2006), accuracy and truthfulness of feedback (Sharples et al., 2007; Spain & Bliss, 2008), and effective communication (Stanton, Young, & Walker, 2007) that minimizes conflicting information to the operator (Song & Kuchar, 2003) have each been shown to assert an impact. The effect of the type of alarm on trust development appears to be task dependent. For example, Gupta, Bisantz, and Singh (2002) found no significant difference in trust by type of alarm in a vehicle collision avoidance system (e.g., no alarm, on/off auditory alarm, graded alarm), yet Donmez and her colleagues (2006) found that drivers trust visual more than auditory alarms for distraction mitigation.

The sensitivity of alarms, specifically the frequency and duration, also affects trust development and use. Gupta and colleagues (2002) found that a reduced number of alarms led to more trust in a collision avoidance system. Wickens and colleagues (2009) supported this finding by suggesting that operators perceive excessive alarms to contain elevated levels of false alert and are thereby unnecessary. However, Lees and Lee (2007) found an interesting distinction between false alarms and unnecessary alarms. They found that whereas false alarms decreased trust and compliance levels, unnecessary alarms (instances in which the automated device indicates a hazardous situation but the human may not view it as hazardous) actually increase trust and compliance.

Ultimately, the behaviors of the automation should be predictable (i.e., match the operator’s expectations) and dependable (i.e., consistent and effective), as each of these systems’ propensities influences trust development and its sustenance. If experience with a machine provides predictable outcomes, then operators start to and then continue to trust the system (Cahour & Forzy, 2009; Muir, 1994). However, when operators experience unanticipated reactions from the automation (e.g., unpredictable outcomes), there can be a rapid drop in trust level that often leads to disuse or a disregard for future information provided by the aid (Wiegmann et al., 2010). Therefore, the system’s behaviors and actions should be consistent and effective. Tenney, Rogers, and Pew (1998) found that pilots wanted automation to be dependable and to provide information that was simple and informative. These qualities were ranked as more important than accountability (able to explain its actions), subordination (easily countermanded), flexibility (range of modes and levels), adaptability (reprogrammable), error resistance (refuse erroneous inputs), or error tolerance (compensate for erroneous inputs) of automation behaviors. The report was published some time ago, and so the degree to which it applies to contemporary automation is an issue but is one of perennial concern (Hancock & Hoffman, 2015). As such, dependable systems are strongly related to the operator’s trust (Yamada & Kuchar 2006).

Systems with highly reliable behavior and communication engender trust across differing task contexts (Bagheri & Jamieson, 2004; Bailey et al., 2006; Bliss & Acton, 2003; Cahour & Forzy, 2009; Cummings, Clare, & Hart, 2010; Donmez et al., 2006; Ho et al., 2005; Kazi, Stanton, Walker, & Young, 2007; Moray et al., 2000; Stedmon et al., 2007). Although errors occur in HAI, the reduction of all types of error increases the reliability of a system and thus positively influences trust development (de Vries et al., 2003). Conversely, miss rate and excessive alarms (e.g., increased false alarms) degrade trust and adversely affect performance (Wickens et al., 2009; Yamada & Kuchar, 2006). Further research suggests that it is the degree of difficulty of the task and not the type of error that appears to influence trust level. Madhavan and Wiegmann (2007) suggested that trust degrades when the automation system misses or provides a false alarm while detecting what the operator perceives to be an easily identifiable target. However, trust in the automation system increases when the target is perceived by the operator to be a difficult target to identify. Similarly, Master and colleagues (2005) found a significant change in trust between conservative and risky systems but no difference in trust based on error rates.

The Environment

Although factors specific to each of the human and the automation are clearly central in trust development, there is also qualitative support that trust is mediated by some aspects of the operational environment. For example, Hancock, Billings, Schaefer, Chen, et al. (2011) identified antecedents of trust related to the environment in terms of team collaboration and tasking. A review of HAI research related to team collaboration suggests the importance of the situational elements of interdependence, team composition, MMs, and societal impact. In addition, research related specifically to tasking suggests the importance of both environmental risk and the context of use in general.

Team collaboration

Team collaboration is a concept that has ever-greater relevance to HAI with systems that have more advanced levels of autonomy. However, the underlying antecedents, including societal impact, MMs, interdependence, and team composition, are important to both automated and autonomous systems.

Societal impact includes culture, community cohesion, political system, environment, health and well-being, property rights, and fears and aspirations concerning safety and the future (Clavell, & Frowd, 2015). In the area of vehicle cloud computing, society creates a safety culture that affects feelings about security and privacy. These feelings involve extensive issues building trust in the overall environment of the transportation infrastructure (Yan, Wen, Olariu, & Weigle, 2013). Culture also affects trust and acceptance of automated technologies, such as commercial aircrafts (Rice, Kraemer, Winter, Mehta, & Dunbar, 2014), decision aids (Huerta, Glandon, & Petrides, 2012), and even websites (Cyr, 2008).

Societal impact, prior experience, cultural biases, and beliefs can frame how any individual forms an MM of automation; and system design (e.g., feedback) is used to update his or her MM (Cuevas, Fiore, Caldwell, & Strater, 2007; Endsley, 1996). Individual and collective MMs refer to structured, organized knowledge that humans possess that describe, explain, and predict a system’s purpose, form, function, or state (Rouse & Morris, 1986). In a review of such research from 1969 to 1995, Moray (1999) found this construct to be especially applicable to automated systems and to influence how an individual decides the degree of control necessary to use that system. Although the MM antecedent can be considered to be a direct property of the user, here we are discussing social-level MM of the technology. We thus place it under the team collaboration subheading of the environmental factor since social MM of the technology is influenced by the human, societal impact, technology, environment, and task.

MMs have also been previously associated with trust development in HAI, both directly and indirectly. For example, Fogg and Tseng (1999) have suggested that if an interface matches the user’s MM, it increases the credibility of that system, and thus trust is increased. Beggiato and Krems (2013) assessed this direct relationship with adaptive cruise control (ACC) by manipulating participants’ initial MM of the system (i.e., correct, incomplete, or incorrect ACC description) and measuring trust over time. They indicate that trust increased steadily in the correct-MM condition. However, there was a constant decrease in trust in the incomplete-MM group. Other work suggests more of an indirect relationship, such as a link between MM and attention allocation. For example, MM has been linked to reduced attention allocation to relevant task elements, which in turn degraded situation awareness (SA) and negatively affected trust development in the automation (e.g., cockpit displays; Sarter, Woods, & Billings, 1997; ACC; Stanton & Young, 2005; driving automation; Walker, Stanton, & Young, 2001).

As automation continues to advance in intelligence and independent decision making, the potential for interdependent human–machine teaming becomes an increased reality. Therefore, it is important to understand the effects of task interdependence and team composition on the development of trust in that system. Task interdependence stems from the interpersonal literature (Johnson & Johnson, 2004). Recently, task interdependence has been suggested as an important component to understanding relationships with technology. In a review of the larger body of work on this topic, Hattingh (2002) posited that agents, automated systems, and even autonomous systems can function interdependently with other parts of the larger system through algorithms (e.g., performing search functions), neural networks, data-mining tools, expert systems, and even genetic algorithms.

Similarly, work on team composition also stems from earlier interpersonal research. It is discussed as part of the process of selecting team members (Durlach, 2007). Specific components of team composition, including team size, diversity, roles, and characteristics of the team (e.g., attitudes, personality, values), influence the effectiveness of the team (Higgs, Plewnia, & Ploch, 2005; McCormack, Duchon, Geyer, & Orvis, 2009; Yunicheva, 2012). Within most of this literature, the team members are human participants, and technology is used to facilitate communication between human team members. Virtual teams, also called computer-mediated teams, have been shown to be more flexible, but there can be a decrease in communication, less knowledge sharing, loss in goal acquisition, the potential for an increase in conflict, and reduced trust and cohesion (Bierly, Stark, & Kessler, 2009; Mehandjiev & Odgers, 1999; Ritter, Lyons, & Swindler, 2007). Bhappu and Crews (2005) suggest that the effectiveness of computer-mediated teams depends on the team members’ ability to manage diversity and communication context. Automated technology has also been built to aid in the formation of human teams by matching individual characteristics (e.g., training, rank, experience) to generic functional roles and project requirements (Klimoski & Jones, 1995; Klimoski & Zukin, 1999).

Tasking

The overall automation trust research tends to mirror the sentiments expressed in the interpersonal and HRI domains in that risk and uncertainty prove to be of essential importance in trust development. Risk can derive from both internal (e.g., data corruption; Beckles, Welch, & Basney, 2005) and external sources (e.g., physical terrain; Borst, Mulder, & van Paassen, 2010). Risk has a direct impact on the appraisal of system safety (Lin, Bekey, & Abney, 2008).

The automation literature provides an overview of a number of functions that may be automated across a variety of tasks. More importantly, task- and function-specific aspects of automation play an important role in the development of trust in these systems. Three major areas of automation are (a) cognitive, (b) control, and (c) perceptual aids (Schaefer et al., 2014).

Cognitive aids provide recommendations to users about the current and potential future states of systems. These decision aids or expert systems can make accurate identifications or detections during a screening or inspection, monitoring, or route-planning tasks. This type of aid has been used in many areas, including e-commerce, medication management, and many others. Overall, findings suggest the importance of system reliability and transparency, operator’s expertise and age, task difficulty, and risk. For example, a more reliable aid led to more trust (e.g., target detection; Chen & Terrence, 2009; Wang et al., 2009; medication management system; Ho et al., 2005); a more transparent aid also led to greater trust (e.g., route planning; Mercado et al., in press). In addition, Madhavan and Wiegmann (2007) found that expert users trust a decision aid for an inspection task more than novice users, and users trust the aid more when it is difficult to identify errors in the task. Ezer, Fisk, and Rogers, (2008) also suggest that older adults trusted decision aids more than younger adults.

Control aids replace varying levels of human action, such as vehicle control (e.g., autopilots, ACC) and navigation aids. Trust is generally high for control aids. Initial trust expectations tend to be higher for the control aid than a human aid (Ma & Kaber, 2007), and trust has been shown to build over time (Rajaonah et al., 2006). Similar to cognitive aids, reliability and age were found to be important to trust development in control aids, such that higher reliability led to higher trust (e.g., fault management systems; Moray et al., 2000; and patient monitoring systems; Spain & Bliss, 2008), and older adults trusted driver mitigation systems more than did middle-age drivers (Donmez et al., 2006). This driving context also pointed to the importance of communication, such that individuals had higher trust for systems that used human speech and provided accurate, reliable information (e.g., driver mitigation systems; Donmez et al., 2006; and air traffic management systems; Stedmon et al., 2007). Finally, in vehicle control systems, trust was higher when workload was low (Spain & Bliss, 2008) and the system was perceived to be useful (Donmez et al., 2006).

Perceptual aids are used to assist the operator or user by providing warnings or assist with pattern recognition. For example, automobile warning systems are used for distraction mitigation, collision avoidance, and providing alarms (e.g., speed alerts). Gupta and colleagues (2002) found no differences in trust for different types of alarms in driver warning systems; however, drivers did show higher trust for alarms with a low sensitivity rate. In addition, alarms should be prompt and should occur before a user takes action (Abe & Richardson, 2004, 2005). Alarms for collision warning systems should also be reliable (Bliss & Acton, 2003). Koustanai, Cavallo, Dalhomme, & Mas (2012) suggest that individuals familiar with how the collision warning system functions tend to trust the system more than those who were unfamiliar with the system. Research into perceptual aids suggests that there was no significant difference in trust for varying levels of automation in a fault management system (Lorenz, De Nocera, Rottger, & Parasuraman, 2002).

Theoretical Model of Human–Automation Trust

Founded upon the foregoing observations, we made a number of changes to our original descriptive HRI trust model (Figure 4). The three main factors related to the human, its automated partner (e.g., robot, automation), and the environment of operations were retained. However, changes were made to the specific moderating factors, as was a reorganization of trust antecedents. Specifically, the human-related factors evolved from “ability-based characteristics” and “human characteristics” to include more specific subheadings (i.e., demographic characteristics, personality traits, states, cognitive factors, and emotive factors). The partner-related factors were reorganized from “attribute-based characteristics” and “performance-based characteristics” to include “features” and “capabilities” in order to standardize the model across different types of partners. The organization of environment-related factors (i.e., task-related context and team collaboration) was retained.

Figure 4.

Revised three-factor model of trust following review of the trust-in-automation literature. Asterisk (*) represents experimental and plus symbol (+) represents correlation findings included in human–automation trust meta-analyses; white boxes represent both empirical and non-empirical support from the automation trust literature; gray boxes represent human–robot interaction–specific items that were retained from original trust model.

Analytical Method

To generate the present meta-analysis, a review of articles dealing specifically with human–automation trust was conducted using Web of Science and EBSCOhost databases with automation and trust as the primary search terms. We also used citations from Google and its derivative Google Scholar. After the initial listing of articles was obtained, the references for these papers were all reviewed to determine whether other relevant studies could be identified. It is important to note that rejecting primary studies in a meta-analysis is common and necessary in order to ensure meaningful results when combining effect sizes across studies. All identified studies were inspected to ensure that they met the following five criteria for inclusion:

Each study had to report an empirical examination of trust in which trust was a dependent variable.

Each study had to measure trust, not an associated antecedent (e.g., reliability, confidence) or outcome variable (e.g., reliance, compliance).

The empirical examination of trust was directed toward automation. Studies investigating interpersonal trust and human–robot trust were excluded in the present work.

Each included study had to incorporate human participants who either viewed or participated directly in interactions with automation through physical, virtual, or augmented means.

Each study had to include sufficient information to compute effect sizes.

As a consequence of the aforementioned screening procedure, a total of 56 empirical articles were identified that met the first four criteria; however, 27 of these articles had insufficient information to calculate an effect size. The authors of the latter studies were contacted when possible and were asked to provide the information necessary to compute an effect size. They were given an extensive time to respond with such additional information. At the end of this process, the 30 accepted papers yielded a total of 164 pairwise effect sizes (i.e., Hedge’s ḡ), and an additional 16 papers yielded 63 correlational effect sizes. Studies included in the analysis are detailed in the appendix.

The studies that are here included in effect size calculations contained both correlational and group design data. Within this framework, the correlational effects (Pearson $\bar{r}$ ) represent an association between trust and the given factor, and Hedge’s ḡ indicates the difference between two means in standard deviation units. With both types of meta-analytic outcomes, the more positive the effect, the higher the level of trust. Findings were interpreted using Hedge’s ḡ and Pearson’s $\bar{r}$ with established ranges for small (ḡ ≤ |.20|; $\bar{r}$ ≤ |.10|), medium (ḡ = |.50|; $\bar{r}$ = |.25|), and large (ḡ ≥ |.80|; $\bar{r}$ ≥ |.40|) effect sizes.

Results

All data were analyzed using IBM SPSS Statistics v.19 (2010) through syntax coding and procedures that have been tested and validated in prior meta-analytic investigations (Hancock, Billings, Schaefer, Chen, et al., 2011; Szalma & Hancock, 2011). We depict the present results as forest plots, in which all effect sizes are marked with a symbol. The diamond symbol is used to represent the overarching category effect, the square symbol represents the submoderating effect, and the triangle represents the specific antecedent effect (where applicable). Confidence intervals indicate the associated estimate of the error around the result. They are marked with a horizontal line, where shorter lines represent more precise mean estimates. In conjunction with the visual symbols, each precise numerical effect is shown with the mean effect size; and in parentheses, the 95% confidence interval. There is also a vertical line located at the zero point on the x-axis. If the confidence interval crosses this zero point, it represents the potential for a null effect for that measure. The latter instability may occur in part due to the (a) limitation in the number of studies or (b) a large diversity in the specific characteristics of the studies included in the finding, or (c) it may reflect a true absence of an effect. Such an observation serves to encourage future research in that particular area. The number of studies included for each level of analysis (k) is provided for each effect size estimate. Some studies included effect sizes within multiple moderating variables; thus, the numbers of studies for each moderating variable do not necessarily sum to the global effect.

Global Trust and Moderator Analyses

Our analysis was conducted initially to identify the effects of three main identified moderators (human, automation, and environment) on trust development as well as the potential antecedents of trust (Figure 4). Figure 5 confirms a global importance (+0.48) of the identified factors and their associated antecedents on trust development in automation. It also shows the moderate effects for both the human-related (+0.49) and automation-related (+0.53) factors. The absence of specific environment-related studies serves to highlight the lack of information specific to the operational environment. However, we should note that four studies (Beggiato & Krems, 2013; Cassidy, 2009; Kazi et al., 2007; Stanton & Young, 2005) were directly relevant to the environmental assessment, but because of individual violations of the inclusion criteria, we cannot provide a quantitative assessment at this time.

Figure 5.

Forest plot representing the global effect and the moderating effects (human, automation, and environment) on trust development. Effect sizes are Hedge’s ḡ.

Our subsequent analyses focus on the specifics of the human-related and automation-related findings (Figure 6). Nine studies (54 effect sizes) specific to age, cognitive factors, and emotive factors were included for a more detailed analysis of the human-related category. Results showed a moderate effect of age (+0.44), a small-to-moderate effect for various cognitive factors (+0.39), and a moderate-to-large effect of emotive factors (+0.72) on trust development. Review of the overall confidence interval provides support for the contention that the outcome of the nine studies indicated a substantive effect of human-related factors on trust development. The small number of studies (k) and the large confidence intervals for each subcategory suggest that additional research in these areas can serve to provide clarity as to the specific degree of these effects on trust development. This conclusion was bolstered by the six studies that yielded 11 effect sizes, which were then included in the correlation-based analysis of emotive factors. This analysis yielded a small-to-medium effect ( $\bar{r}$ = +0.16), suggesting that trust and emotive factors are indeed related.

Figure 6.

Forest plot representing the submoderating effects of the human and the automation on trust development. Effect sizes are Hedge’s ḡ.

The experimental analysis of the specific effect of automation features on trust development show a small-to-moderate effect (+0.35); however, the confidence interval here included zero, implying a potentially unstable effect. The comparable correlational analysis showed only a small effect ( $\bar{r}$ = +0.16); however, this latter result was based on only two studies. Analyses of automation capabilities showed a moderate-to-large experimental effect (+0.70) and a moderate correlational effect ( $\bar{r}$ = +0.39) on trust development. The following antecedents of the automation capabilities were also analyzed: behavior (+0.79), error (+0.44), and feedback (+0.45). It is important to note here that feedback also included zero in the confidence interval, again implying that one should not rule out the absence of an effect.

Overall, these results suggest that the capability of the automation is important to trust development in that the behaviors should be reliable and that lower error rate leads to more trust in the system. Review of the confidence intervals and the associated number of available studies suggests that further research should be conducted specifically related to the effects of automation features and feedback on trust development.

Analysis of the Task Type

All previous studies were included in this analysis, which were subdivided into the three types of automation tasks: cognitive (13 studies, 72 statistics), control (eight studies, 68 statistics), and perceptual (six studies, 14 statistics) automation aids. All three types of automation aids had a moderate effect on trust development (Figure 7).

Figure 7.

Forest plot representing the effects of the task type on trust development. Effect sizes are Hedge’s ḡ.

The outcome of these overall findings confirm and extend those of Hancock, Billings, Schaefer, Chen, et al. (2011) implying that robot interaction and overall automation bear understandable and striking similarities. However, our present work has now gone beyond those earlier observations.

Discussion

The present meta-analytic results on the whole genre of human–automation trust relationships extend and elaborate upon our existing state of knowledge about subdomains of HAI. Figure 8 provides an explicit comparison of the results from our prior human–robot trust meta-analysis (Hancock, Billings, Schaefer, Chen, et al., 2011) and the current human–automation trust findings. This figure provides a visual depiction to allow the reader the capability to quickly identify the additional known antecedents of trust (i.e., additional factors) but also identify multiple areas of needed research (i.e., factors that have large confidence intervals, confidence intervals that include zero, or small sample sizes). Additional research in each domain would also provide a more stable effect on trust development in each respective context.

Figure 8.

Comparison of human–robot meta-analysis (Hancock, Billings, Schaefer, Chen, et al., 2011) and the current human–automation meta-analysis. No studies were included in both meta-analyses. Effect sizes are Hedge’s ḡ.

The first important finding was that the current HAI results support that human-related factors have an effect on trust development, which was identified as a major lacuna in our previous HRI analysis. Yet, the human-related antecedents of trust should be considered for further research for both HRI and HAI. This current work identified a number of additional research gaps: (a) Age should be explored across multiple contexts; (b) the relationship between traits, states, and trust is still in its infancy; and (c) more work should be conducted for the antecedents associated with the cognitive and emotive factors in order to provide more stable and thorough understanding of the true impact of those characteristics.

A second major finding was the importance of the partner’s (automation or robot) capabilities, which added stability to our previous HRI findings despite the small number of studies. Our present work also addressed the previously identified gap in the HRI analysis related to error and feedback. However, the unstable results that were found in regard to feedback may suggest that it is context dependent. Therefore, additional research regarding feedback and cuing should be extensively studied in HAI and HRI that incorporate more advanced autonomous systems within contexts that require interdependent operations or teaming. Further, additional research conducted in the areas of reliability, errors, and behaviors may serve to provide a more fine-grained analysis of these trust antecedents across contexts (e.g., specific behaviors, level of reliability, specific types of errors). Results of the HAI meta-analysis also imply the need for more investigations related to the features of the automation. There is an obvious lack of literature in the subareas of appearance and mode of communication, compared to the HRI domain. As such, these two antecedents of trust may be more specific to embodied agents (including robots). Finally, the environment-related factors have been more frequently studied in HRI, which may be a result of the specific efforts at teaming and social interaction requirements needed for current and projected HRI.

Limitations and Needed Areas of Extended Review

Since the present meta-analysis included studies only up to 2012, we also evaluated the subsequent literature through the end of 2014 for the areas that presented inconclusive findings (i.e., mode of communication, LOA, appearance-based anthropomorphism, and feedback). The majority of recent work related to the mode of communication has focused on advancing automated telephone (voice), e-mail, text messaging, or web-based communication systems. In respect of embodied agents, Tang, Charalambous, Webb, and Fletcher (2014) suggested that the design of robotic gestures should be intuitive and humanlike in nature to positively influence trust development. Large and Burnett (2014) found the features of a speech-based interface, such as a navigation system, can invoke personality characteristics of the system and thus influence trust level.

Appearance-based anthropomorphism is a more recent area of research that tends to be restricted to embodied computer agents or robotics. Therefore, a large portion of this research lay outside the scope of the current meta-analysis. However, with the relevance to the design of automation, it cannot be overlooked altogether. The more recent literature provides some important thought-provoking guidelines that should be considered when designing future systems. De Visser and his colleagues (2012) found that increasing humanness of an agent increased trust resilience. However, Culley and Madhavan (2013) suggest caution with respect to anthropomorphic design of computer agents, as different individuals perceive varying degrees of reasoning and motivation, which can lead to an incorrect calibration of trust. Additionally, there is a positive relationship between trust and the similarity of the agent to the human in regard to perceived age and gender (Bass, Goodwin, Brennan, Pak, & McLaughlin, 2013; Pak, McLaughlin, & Bass, 2014). This finding also holds for similarity of movement (Verberne, Ham, Ponnada, & Midden, 2013).

Within the past 2 years, research has emphasized the interaction between feedback, transparency of that feedback, and trust. For example, work by Gressgård, Hansen, and Iversen (2013) reinforced the previously determined relationship of system status feedback as a determinant of trust since it is a crucial element in the decision-making process. Koo and colleagues (2014) take a slightly different perspective in that it is not necessarily only feedback (outcome of system’s actions) that is important to trust development but what they term “feedforward” information (providing information ahead of time). Research continues to point toward the issue of agent transparency as a key to better understanding trust development (Chen, Procci, Boyce, Wright, Garcia, & Barnes, 2014; Helldin, 2014; Mercado et al., in press; Sanders, Wixon, Schafer, Chen, & Hancock, 2014).

We recommend that our model can be used to actively influence trust calibration. Our overview indicates that future work should consider the interplay between system capability and team training, thereby taking into consideration appraisal versus intent errors, task-related versus affective state changes, and the distinction between perception-based and objective system characteristics. Thus, the interaction between the updated three-factor model of trust with design and training can be used to determine effective trust calibration strategies that will in turn enhance mission outcomes.

Concluding Remarks

Meta-analytic results, such as those presented, are contingent on the quality of the input. Our present pattern of findings is also necessarily contingent upon the spectrum of factors already associated with trust that have been identified. In the way Sheridan (2014) frames this question, we are engaged in quantifying the “known unknowns.” What this strategy necessarily omits is the variety of variables that have yet to be experimentally investigated. Again, in Sheridan’s parlance, these are “unknown unknowns,” or the more appealing epithet “unk-unks.” Although we might well argue that there are nearly always going to be such entities, our current findings and those of our associated human–robot trust meta-analysis (Hancock, Billings, Schaefer, Chen, et al., 2011) have served to frame such known issues since the full picture in this important and dynamic area is only now coming into focus. Robots are certainly automation, and as such, many of the same antecedents commonly affect trust in both robots and automation systems but not all.

At the same time we are struggling to understand trust and automation, automation itself is very far from a stationary or unitary entity. We are seeing the birth of much more autonomous agents. Unleashed into complex operational contexts, those new evolving systems are necessarily bound by the trust equation. Our present synthetic effects can only point the way, but in general, systems that are not trusted are not used. What happens when the human is deemed the untrustworthy system? Such pressing conundra need rational and objective evidence to elicit viable solution paths. We offer the present meta-analysis as one stepping-stone along this path.

Key Points

The automation literature, and meta-analytic findings derived therefrom, identified three important functions previously deemed as inconclusive by Hancock, Billings, Schaefer, Chen, and colleagues (2011): the importance of the human, the impact of error rates and false alarms, and the relationship of communication on trust development.

The difference in global effects between human–robot interaction (+.71) and human–automation interaction (+.48) may be partially explained by the possible unstable effect of the automation’s features (i.e., results contain a zero marker in the confidence intervals). Thus, the system’s feature-based characteristics (including mode of communication and appearance) may influence human–robot trust development more than automation.

This meta-analysis identified critical gaps in the trust literature. Future research should be focused on the effects of human states, mode of communication, anthropomorphism, and agent transparency on trust development.

The next step beyond understanding trust development will be to use this model to determine which elements can be adapted or further developed to effectively calibrate trust. As this research avenue is pursued, future work should consider the interplay between design and training (including system capability and team training).

Footnotes

Appendix

Table A1:

Studies Included in Meta-Analysis

Factors	Experimental Effects	Correlational Effects
Human-related factors
Traits^a	Donmez, Boyle, Lee, & McGehee (2006) Ho, Wheatley, & Scialfa (2005) Pak, Fink, Price, Bass, & Sturre (2012)	Merritt & Ilgen (2008)
Cognitive factors	De Vries & Midden (2008) Koustanai, Cavallo, Dalhomme, & Mas (2012) Manzey et al. (2009) Rajaonah, Anceaux, & Vienne (2006)	Dzindolet, Peterson, Pomranky, Pierce, & Beck (2003) Lee, Kim, & Kim (2007) Merritt & Ilgen (2008) Rajaonah, Tricot, Anceaux, & Millot (2008) Spain & Bliss (2008) Wang, Jamieson, & Hollands (2011)
Emotive factors	Koustanai et al. (2012) Neyedli, Hollands, & Jamieson (2009) Verberne, Ham, & Midden (2012)	Donmez et al. (2006) Lee & Chung (2009) Lee et al. (2007) Merritt and Ilgen (2008) Wang, Jamieson, & Hollands (2009) Wang et al. (2011)
Automation-related factors
Features	Cai & Lin (2012) Cai & Chen (2012)Lorenz, Di Nocera, Rottger, & Parasuraman (2002) Merritt, Heimbaugh, LaChapell, & Lee (2012) Moray, Inagaki, & Itoh (2000) Pak et al. (2012) Stedmon et al. (2007) Verberne et al. (2012)	Dzindolet et al. (2003) Lee & Chung (2009)
Capabilities	Abe and Richardson (2004) Abe and Richardson (2005) Bagheri (2004) Bagheri & Jamieson (2004) Bliss and Acton (2003) De Vries, Midden, & Bouwhuis (2003) Donmez et al. (2006) Gupta, Bisantz, & Singh (2002) Ho et al. (2005) Kazi, Stanton, Walker, & Young (2007) Koh & Sundar (2010) Madhavan & Wiegmann (2007) Madhavan, Wiegmann, & Lacson (2006) Moray et al. (2000) Rajaonah et al. (2006) Smyth (2007) Spain and Bliss (2008) Stedmon et al. (2007)	Abe and Richardson (2006) Bagheri & Jamieson (2004) Bagheri (2004) Dzindolet et al. (2003) Lee & Chung (2009) Lee et al. (2007) Muir & Moray (1996) Neyedli et al. (2009) Rajaonah et al. (2006)

Findings are related to the effect of age on trust.

Acknowledgements

This research was sponsored by the U.S. Army Research Laboratory and was accomplished under Cooperative Agreement Nos. W911NF-10-2-0016 and W911-NF-12-2-0019. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. government. The U.S. government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notation herein. The authors would also like to acknowledge the various research assistants and support staff from the University of Central Florida who were associated with the generation and submission of the document. We also acknowledge Susan Hill, Donald Headley, and John F. Lockett III, U.S. Army Research Laboratory, for their internal technical reviews and support as well as our external technical reviewers.

Author(s) Note:

The author(s) of this article are U.S. government employees and created the article within the scope of their employment. As a work of the U.S. federal government, the content of the article is in the public domain.

Kristin E. Schaefer is an ORAU postdoctoral fellow with the U.S. Army Research Laboratory, Human Research and Engineering Directorate, at Aberdeen Proving Ground, Maryland. She received her PhD in modeling and simulation from the University of Central Florida in 2013, under the direction of P. A. Hancock. Her research interests include human–robot interaction, simulation systems, situation awareness, and trust.

Jessie Y. C. Chen is a research psychologist with the U.S. Army Research Laboratory, Human Research and Engineering Directorate, at the field element in Orlando, Florida. She received her PhD in applied experimental and human factors psychology from University of Central Florida in 2000. Her research interests include human–robot interaction, human–agent teaming, agent transparency, supervisory control, and individual differences.

James L. Szalma is an associate professor in the Psychology Department at the University of Central Florida. He received a PhD in applied experimental/human factors psychology in 1999 from the University of Cincinnati. His primary research interests include signal/threat detection (e.g., friend/foe identification), training for threat detection, and how the characteristics of tasks and operators interact to influence performance in the context of tasks that require sustained attention or that include human–automation interaction.

P. A. Hancock is Provost Distinguished Research Professor, Pegasus Professor, and University Trustee Chair at the Department of Psychology and Institute for Simulation and Training at the University of Central Florida. There, he directs the MIT² laboratories and currently has extensive research on human–robot interaction funded by the U.S. Army Research Laboratory and National Science Foundation. He received his PhD in human performance from the University of Illinois in 1983 and DSc in human–machine systems from Loughborough University, England, in 2001. He is also a fellow of seven scientific societies as well as a past president of a number of them, including the Human Factors and Ergonomics Society.

References

*Abe

Richardson

(2004). The effect of alarm timing on driver behavior: An investigation of differences in driver trust and response to alarms according to alarm timing. Transportation Research Part F: Traffic Psychology and Behaviour, 7, 307–322. doi:10.1016/j.trf.2004.09.008

*Abe

Richardson

(2005). The influence of alarm timing on braking response and driver trust in low speed driving. Safety Science, 43, 639–654. doi:10.1016/j.ssci.2005.04.006

*Abe

Richardson

(2006). The influence of alarm timing on driver response to collision warning systems following system failure. Behavior & Information Technology, 25, 443–452. doi:10.1080/01449290500167824

Ardern-Jones

Hughes

D. K.

Rowe

P. H.

Mottram

D. R.

Green

C. F.

(2009). Attitudes and opinions of nursing and medical staff regarding the supply and storage of medicinal products before and after the installation of a drawer-based automated stock-control system. International Journal of Pharmacy Practice, 17, 95–99. doi:10.1211/ijpp/17.02.0004

Ashleigh

M. J.

Stanton

N. A.

(2001). Trust: Key elements in human supervisory control domains. Cognition, Technology & Work, 3, 92–100. doi:10.1007/PL00011527

*Bagheri

(2004). The effect of automation reliability on operator monitoring performance (Master’s thesis). University of Toronto, Toronto, Canada.

*Bagheri

Jamieson

(2004). Considering subjective trust and monitoring behavior in assessing automation-induced complacency. In Vincenzi

D. A.

Mouloua

Hancock

P. A.

(Eds.), Human performance, situation awareness and automation: Current research and trends (Vol. 2, pp. 54–59). Mahwah, NJ: Lawrence Erlbaum.

Bailey

N. R.

Scerbo

M. W.

Freeman

F. G.

Mikulka

P. J.

Scott

L. A.

(2006). Comparison of a brain-based adaptive system and a manual adaptable system for invoking automation. Human Factors, 48, 693–709. doi:10.1518/001872006779166280

Bass

Goodwin

Brennan

Pak

McLaughlin

(2013). Effects of age and gender stereotypes on trust in an anthropomorphic decision aid. In Proceedings of the Human Factors and Ergonomics Society 57th Annual Meeting (pp. 1575–1579). Santa Monica, CA: Human Factors and Ergonomics Society. doi:10.1177/1541931213571351

10.

Beckles

Welch

Basney

(2005). Mechanisms for increasing the usability of grid security. International Journal of Human–Computer Studies, 63, 74–101. doi:10.1016/j.ijhcs.2005.04.017

11.

Beggiato

Krems

J. F.

(2013). The evolution of mental model, trust and acceptance of adaptive cruise control in relation to initial information. Transportation Research Part F, 18, 47–57. doi:10.1016/j.trf.2012.12.006

12.

Bhappu

A. D.

Crews

J. M.

(2005). The effects of communication media & conflict on team identification in diverse teams. In Proceedings of the 38th Hawaii International Conference on System Science (p. 50.1) Washington, DC: IEEE Computer Society. doi:10.1109/HICSS.2005.592

13.

Bierly

P. E.

III Stark

E. M.

Kessler

E. H.

(2009). The moderating effects of virtuality on the antecedents and outcome of NPD team trust. Journal of Product Innovation Management, 26, 551–565.

14.

Bitan

Meyer

(2007). Self-initiated and respondent actions in a simulated control task. Ergonomics, 50, 763–788. doi:10.1080/00140130701217149

15.

*Bliss

J. P.

Acton

S. A.

(2003). Alarm mistrust in automobiles: How collision alarm reliability affects driving. Applied Ergonomics, 34, 499–509. doi:10.1016/j.apergo.2003.07.003

16.

Borenstein

Hedges

L. V.

Higgins

J. P. T.

Rothstein

H. R.

(2009). Introduction to meta-analysis. West Sussex, UK: Wiley.

17.

Borst

Mulder

van Paassen

M. M.

(2010). Design and simulator evaluation of an ecological synthetic vision display. Journal of Guidance, Control, and Dynamics, 33, 1577–1591. doi:10.2514/1.47832

18.

Cahour

Forzy

(2009). Does projection into use improve trust and exploration? An example with a cruise control system. Safety Science, 47, 1260–1270. doi:10.1016/j.ssci.2009.03.015

19.

*Cai

Lin

(2012). Coordinating cognitive assistance with cognitive engagement control approaches in human-machine collaboration. IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans, 42, 286–294. doi:10.1109/TSMCA.2011.2169953

20.

*Cai

Lin

Cheng

(2012). Coordinating multi-level cognitive assistance and incorporating dynamic confidence information in driver-vehicle interfaces. Human Factors and Ergonomics in Manufacturing & Service Industries, 22, 437–449. doi:10.1002/hfm.20399

21.

Cassidy

A. M.

(2009). Mental models, trust, and reliance: Exploring the effect of human perceptions on automation use (Master’s thesis). Naval Postgraduate School, Monterey, CA.

22.

Chen

J. Y. C.

Barnes

M. J.

(2012). Supervisory control of multiple robots: Effects of imperfect automation and individual differences. Human Factors, 54, 157–174. doi:10.1177/0018720811435843

23.

Chen

J. Y. C.

Barnes

M. J.

(2014). Human–agent teaming for multi-robot control: A review of human factors issues. IEEE Transactions on Human–Machine Systems, 44, 13–29.

24.

Chen

J. Y. C.

Procci

Boyce

Wright

Garcia

Barnes

(2014). Situation awareness-based agent transparency (Report No. ARL-TR-6905). Aberdeen Proving Ground, MD: U.S. Army Research Laboratory.

25.

Chen

J. Y. C.

Terrence

P. I.

(2009). Effects of imperfect automation and individual differences on concurrent performance of military and robotics tasks in a simulated multi-tasking environment. Ergonomics, 52, 907–920. doi:10.1080/00140130802680773

26.

Clavell

G. G.

Frowd

P. M.

(2015). Using societal impact assessment (SIA) to improve technological development in the field of crime prevention. Lecture Notes in Computer Science, 8852, 18–124. doi:10.1007/978-3-319-15168-7_16

27.

Cottrell

N. D.

Barton

B. K.

(2013). The role of automation in reducing stress and negative affect while driving. Theoretical Issues in Ergonomics Science, 14, 53–68. doi:10.1081/1464536X.2011.573011

28.

Cuevas

H. M.

Fiore

S. M.

Caldwell

B. S.

Strater

(2007). Augmenting team cognition in human-automation teams performing in complex operational environments. Aviation, Space, and Environmental Medicine, 78(5), B63–B70.

29.

Culley

K. E.

Madhavan

(2013). A note of caution regarding anthropomorphism in HCI agents. Computers in Human Behavior, 29, 577–579.

30.

Cummings

M. L.

Clare

Hart

(2010). The role of human–automation consensus in multiple unmanned vehicle scheduling. Human Factors, 52, 17–27. doi:10.1177/0018720810368674

31.

Cyr

(2008). Modeling web site design across cultures: Relationships to trust, satisfaction, and e-loyalty. Journal of Management Information Systems, 24(4), 47–72. doi:10.2753/MIS0742-1222240402

32.

de Visser

E. J.

Krueger

McKnight

Scheid

Smith

Chalk

Parasuraman

(2012). The world is not enough: Trust in cognitive agents. In Proceedings of the Human Factors and Ergonomics Society 56th Annual Meeting (pp. 263–267). Santa Monica, CA: Human Factors and Ergonomics Society.

33.

*de Vries

Midden

(2008). Effect of indirect information on system trust and control allocation. Behavior & Information Technology, 27, 17–29. doi:10.1080/01449290600874956

34.

*de Vries

Midden

Bouwhuis

(2003). The effects of errors on system trust, self-confidence, and the allocation of control in route planning. International Journal of Human-Computer Studies, 58, 719–735. doi:10.1016/s1071-5819(03)00039-9

35.

Defense Science Board. (2012). The role of autonomy in Department of Defense systems. Task force report. Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology and Logistics. Retrieved from http://www.acq.osd.mil/dsb/reports/AutonomyReport.pdf

36.

Desai

Kaniarasu

Medvedev

Steinfeld

Yanco

(2013). Impact of robot failures and feedback on real-time trust. In Proceedings of the 8th ACM/IEEE International Conference on Human–Robot Interaction (pp. 251–258). Piscataway, NJ: IEEE Press.

37.

*Donmez

Boyle

L. N.

Lee

J. D.

McGehee

D. V.

(2006). Drivers’ attitudes toward imperfect distraction mitigation strategies. Transportation Research Part F: Traffic Psychology and Behaviour, 9, 387–398. doi:10.1016/j.trf.2006.02.001

38.

Durlach

P. J.

(2007). PACERS: Platoon aid for collective employment of robotic systems (Report No. 1876). Arlington, VA: U.S. Army Research Institute for Behavioral and Social Sciences.

39.

*Dzindolet

M. T.

Peterson

S. A.

Pomranky

R. A.

Pierce

L. G.

Beck

H. P.

(2003). The role of trust in automation reliance. International Journal of Human–Computer Studies, 58, 697–710. doi:10.1016/S1071-5819(03)00038-7

40.

Endsley

M. R.

(1996). Automation and situation awareness. In Parasuraman

Mouloua

(Eds.), Automation and human performance: Theory and applications (pp. 163–181). Mahwah, NJ: Lawrence Erlbaum.

41.

Ezer

Fisk

A. D.

Rogers

W. A.

(2008). Age-related differences in reliance behavior attributable to costs within a human–decision aid system. Human Factors, 50, 853–863.

42.

Finomore

Matthews

Shaw

Warm

(2009). Predicting vigilance: A fresh look at an old problem. Ergonomics, 52, 791–808. doi:10.1080/00140130802641627

43.

Fogg

B. J.

Tseng

(1999). The elements of computer credibility. In Proceedings of CHI (pp. 80–87). Pittsburgh, PA: ACM.

44.

Frost

C. R.

(2010). Challenges and opportunities for autonomous systems in space. In Frontiers of engineering: Reports on leading-edge engineering (pp. 89–102). Washington, DC: National Academies Press.

45.

Gao

Lee

J. D.

Zhang

(2006). A dynamic model of interaction between reliance on automation and cooperation in multi-operator multi-automation situations. International Journal of Industrial Ergonomics, 36, 511–526. doi:10.1016/j.ergon.2006.01.013

46.

Goetz

Kiesler

Powers

(2003). Matching robot appearance and behavior to tasks to improve human-robot cooperation. In Proceedings of the 12th IEEE International Workshop on Robot and Human Interactive Communication (RO-MAN 2003) (pp. 55–60). Millbrae, CA: IEEE.

47.

Gressgård

L. J.

Hansen

Iversen

(2013). Automation systems and work process safety: Assessing the significance of human and organizational factors in offshore drilling automation. Journal of Information Technology Management, 24(2), 47.

48.

*Gupta

Bisantz

A. M.

Singh

(2002). The effects of adverse condition warning system characteristics on driver performance: An investigation of alarm signal type on threshold level. Behaviors & Information Technology, 21, 235–248. doi:10.1080/0144929021000013473

49.

Hancock

P. A.

(2013). Driven to distraction and back again. In Regan

M. A.

Lee

J. D.

Victor

T. W.

(Eds.), Driver distraction and inattention: Advances in research and countermeasures (pp. 9–26). Surrey, UK: Ashgate.

50.

Hancock

P. A.

(2016). Imposing limits on autonomous systems. Manuscript in preparation.

51.

Hancock

P. A.

Billings

D. R.

Schaefer

K. E.

(2011). Can you trust your robot? Ergonomics in Design, 19(3), 24–29. doi:10.1177/1064804611415045

52.

Hancock

P. A.

Billings

D. R.

Schaefer

K. E.

Chen

J. Y. C.

de Visser

Parasuraman

(2011). A meta-analysis of factors affecting trust in human–robot interaction. Human Factors, 53, 517–527. doi:10.1177/0018720811417254

53.

Hancock

P. A.

Chignell

M. H.

(1987). Adaptive control in human-machine systems. In Hancock

P. A.

(Ed.), Human factors psychology (pp. 305–345). Amsterdam, Netherlands: North-Holland.

54.

Hancock

P. A.

Chignell

M. H.

(1988). Mental workload dynamics in adaptive interface design. IEEE Transactions on Systems, Man, and Cybernetics, 18, 646–658.

55.

Hancock

P. A.

Desmond

P. A.

Matthews

(2012). Conceptualizing and defining fatigue. In Matthews

Desmond

P. A.

Neubauer

Hancock

P. A.

(Eds.), The handbook of operator fatigue (pp. 63–73). Burlington, VT: Ashgate.

56.

Hancock

P. A.

Hancock

G. M.

Warm

J. S.

(2009). Individuation: The N=1 revolution. Theoretical Issues in Ergonomic Science, 10, 481–488.

57.

Hancock

P. A.

Hoffman

R. R.

(2015). Keeping up with intelligent technology. IEEE Intelligent Systems, 30, 62–65.

58.

Hancock

P. A.

Mercado

J. E.

Merlo

Van Erp

J. B. F.

(2013). Improving target detection in visual search through the augmenting multi-sensory cues. Ergonomics, 56, 729–738.

59.

Hancock

P. A.

Warm

J. S.

(1989). A dynamic model of stress and sustained attention. Human Factors, 31, 519–537.

60.

Hattingh

(2002). An analysis of the current nature, status and relevance of data mining tools to enable organizational learning (Master of commerce thesis). University of Stellenbosch, Stellenbosch, South Africa.

61.

Helldin

(2014). Transparency for future semi-automated systems (Doctoral dissertation). Örebro University, Örebro, Sweden.

62.

Higgs

Plewnia

Ploch

(2005). Influence of team composition and task complexity on team performance. Team Performance Management, 11, 227–250. doi:10.1108/ 13527590510635134

63.

*Ho

Wheatley

Scialfa

(2005). Age differences in trust and reliance of a medication management system. Interacting With Computers, 17, 690–710. doi:10.1016/j.intcom.2005.09.007

64.

Hoff

K. A.

Bashir

(2015). Trust in automation integrating empirical evidence on factors that influence trust. Human Factors, 57, 407–434. doi:10.1177/0018720814547570

65.

Huerta

Glandon

Petrides

(2012). Framing, decision-aid systems, and culture: Exploring influences on fraud investigations. International Journal of Accounting Information Systems, 13, 316–333. doi:10.1016/j.accinf.2012.03.007

66.

Igbaria

Iivari

(1995). The effects of self-efficacy on computer usage. Omega, 23, 587–605. doi:10.1016/0305-0483(95)00035-6

67.

Jarvenpaa

Knoll

Leidner

(1998). Is anybody out there? Antecedents of trust in global virtual teams. Journal of Management Information Systems, 14(4), 29–64.

68.

Johnson

D. W.

Johnson

R. T.

(2004). Cooperation and the use of technology. In Jonassen

D. H.

(Ed.), Handbook of research on educational communications and technology (2nd ed., pp. 785–811). Mahwah, NJ: Lawrence Erlbaum.

69.

*Kazi

T. A.

Stanton

N. A.

Walker

G. H.

Young

M. S.

(2007). Designer driving: Drivers’ conceptual models and level of trust in adaptive-cruise-control. International Journal of Vehicle Design, 45, 339–360. doi:10.1504/IJVD.2007.014909

70.

Keil

Beranek

P. M.

Konsynski

B. R.

(1995). Usefulness and ease of use: Field study evidence regarding task considerations. Decision Support Systems, 13, 75–91. doi:10.1016/0167-9236(94)e0032-m

71.

Kircher

Thorslund

(2009). Effects of road surface appearance and low friction warning systems on driver behaviour and confidence in the warning system. Ergonomics, 52, 165–176. doi:10.1080/00140130802277547

72.

Klimoski

Jones

R. G.

(1995). Staffing for effective group decision making: Key issues in matching people and task. In Guzzo

R. A.

Salas

(Eds.), Team effectiveness and decision making in organizations (pp. 292–332). San Francisco, CA: Jossey-Bass.

73.

Klimoski

Zukin

(1999). Selection and staffing for team effectiveness. In Sundstrom

Associates (Eds.), Supporting work team effectiveness: Best management practices for fostering high performance (pp. 63–94). San Francisco, CA: Jossey-Bass.

74.

*Koh

Y. J.

Sundar

S. S.

(2010). Effects of specialization in computer, web sites, and web agents on e-commerce trust. International Journal of Human–Computer Studies, 68, 899–912. doi:10.1016/j.ijhcs.2010.08.002

75.

Koo

Kwac

Steinert

Leifer

Nass

(2014). Why did my car just do that? Explaining semi-autonomous driving actions to improve driver understanding, trust, and performance. International Journal on Interactive Design and Manufacturing, 9, 269–275.

76.

*Koustanai

Cavallo

Dalhomme

Mas

(2012). Simulator training with a forward collision warning system: Effects on driver-system interactions and driver trust. Human Factors, 54, 709–721. doi:10.1177/0018720812441796

77.

Krogmann

(1999). From automation to autonomy: Trends toward autonomous combat systems (ADP010300). Überlingen, Germany: Bodenseewerk Gerätetechnik.

78.

Large

D. R.

Burnett

G. E.

(2014). The effect of different navigation voices on trust and attention while using in-vehicle navigation systems. Journal of Safety Research, 49, 69–75. doi:10.1016/j.jsr.2014.02.009

79.

*Lee

Kim

(2007). Determinants of success for applications service provider: An empirical test in small businesses. International Journal of Human–Computer Studies, 65, 796–815. doi:10.1016/j.ijhcs.2007.04.004

80.

Lee

J. D.

Moray

(1994). Trust, self-confidence, and operators’ adaptation to automation. International Journal of Human–Computer Studies, 40, 153–184. doi:10.1006/ijhc.1994.1007

81.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46, 50–80. doi:10.1518/hfes.46.1.50_30392

82.

*Lee

Chung

(2009). Understanding factors affecting trust in and satisfaction with mobile banking in Korea: A modified DeLone and McLean’s model perspective. Interacting With Computers, 21, 385–392. doi:10.1016/j.intcom.2009.06.004

83.

Lees

M. N.

Lee

J. D.

(2007). The influence of distraction and driving context on driver response to imperfect collision warning systems. Ergonomics, 50, 1264–1286. doi:10.1080/00140130701318749

84.

Y. M.

Yeh

Y. S.

(2010). Increasing trust in mobile commerce through design aesthetics. Computers in Human Behavior, 26, 673–684. doi:10.1016/j.chb.2010.01.004

85.

Lin

Bekey

Abney

(2008). Autonomous military robotics: Risk, ethics, and design. Retrieved from http://works.bepress.com/palin/11

86.

Looije

Neerincx

M. A.

Cnossen

(2010). Persuasive robotic assistant for health self-management of older adults: Design and evaluation of social behaviors. International Journal of Human–Computer Studies, 68, 386–397. doi:10.1016/j.ijhcs.2009.08.007

87.

*Lorenz

Di Nocera

Rottger

Parasuraman

(2002). Automated fault-management in a simulated spaceflight micro-world. Aviation, Space, and Environmental Medicine, 73, 886–897. Retrieved from http://archlab.gmu.edu/people/rparasur/Documents/LorenzASEM02.pdf

88.

Kaber

D. B.

(2007). Effects of in-vehicle navigation assistance and performance on driver trust and vehicle control. International Journal of Industrial Ergonomics, 37, 665–673.

89.

*Madhavan

Wiegmann

D. A.

(2007). Effects of information source, pedigree, and reliability on operator interaction with decision support systems. Human Factors, 49, 773–785. doi:10.1518/001872007X230154

90.

*Madhavan

Wiegmann

D. A.

Lacson

F. C.

(2006). Automation failures on tasks easily performed by operators undermine trust in automated aids. Human Factors, 48, 241–256. doi:10.1518/001872006777724408

91.

*Manzey

Rottger

Bahner-Heyne

J. E.

Schulze-Kissing

Dietz

Meixensberger

Strauss

(2009). Image-guided navigation: The surgeon’s perspective on performance consequences and human factors issues. International Journal of Medical Robotics and Computer Assisted Surgery, 5, 297–308. doi:10.1002/rcs.261

92.

Master

Jiang

Khasawneh

M. T.

Bowling

S. R.

Grimes

Gramopadhye

A. K.

Melloy

B. J.

(2005). Measurement of trust over time in hybrid inspection systems. Human Factors and Ergonomics in Manufacturing & Service Industries, 15, 177–196. doi:10.1002/hfm.20021

93.

Mayer

R. C.

Davis

J. H.

Schoorman

F. D.

(1995). An integrative model of organizational trust, Academy of Management Review, 20, 709–734. doi:10.2307/258792

94.

McAllister

D. J.

(1995). Affect- and cognition-based trust as foundations for interpersonal cooperation in organizations. Academy of Management Journal, 38, 24–59.

95.

McCormack

R. K.

Duchon

Geyer

Orvis

(2009). A case study in data mining for automated building of teams. In Eyob

(Ed.), Social implications of data mining and information privacy: Interdisciplinary frameworks and solutions (pp. 247–265). New York, NY: IGI Global.

96.

McLeod

R. W.

Walker

G. H.

Moray

(2005). Analysing and modeling train driver performance. Applied Ergonomics, 36, 671–680. doi:10.1016/j.apergo.2005.05.006

97.

Mehandjiev

Odgers

B. R.

(1999). SAMBA: Agent-supported visual interactive control for distributed team building and empowerment. BT Technology Journal, 17(4), 72–77.

98.

Mercado

Rupp

Chen

Barber

Procci

Barnes

(in press). Intelligent agent transparency in human–agent teaming for Multi-UxV management. Human Factors.

99.

Merritt

S. M.

(2011). Affective processes in human–automation interactions. Human Factors, 53, 356–370.

100.

*Merritt

S. M.

Heimbaugh

LaChapell

Lee

(2012). I trust it, but I don’t know why: Effects of implicit attitudes toward automation on trust in an automated system. Human Factors, 55, 520–534. doi:10.1177/0018720812465081

101.

*Merritt

S. M.

Ilgen

D. R.

(2008). Not all trust is created equal: Dispositional and history-based trust in human–automation interactions. Human Factors, 50, 194–210. doi:10.1518/001872008X288574

102.

Moray

(1999). Mental models in theory and practice. In Gopher

Koriat

(Eds.), Attention and performance XVII: Cognitive regulation of performance. Interaction of theory and application (pp. 223–258). Cambridge, MA: MIT Press.

103.

*Moray

Inagaki

Itoh

(2000). Adaptive automation, trust, and self-confidence in fault management of time-critical tasks. Journal of Experimental Psychology: Applied, 6, 44–58. doi:10.1037/1076-898X.6.1.44

104.

Muir

B. M.

(1994). Trust in automation: Part 1. Theoretical issues in the study of trust in automated systems. Ergonomics, 37, 1905–1922. doi:10.1080/00140139408964957

105.

*Muir

B. M.

Moray

(1996). Trust in automation: Part 2. Experimental studies of trust and human intervention in a process control simulation. Ergonomics, 39, 429–460. doi:10.1080/ 00140139608964474

106.

Nakada

(1997). Kansei engineering research on the design of construction machinery. International Journal of Industrial Ergonomics, 19, 129–146. doi:10.1016/S0169-8141 (96)00009-1

107.

Nass

Moon

(2000). Machines and mindlessness: Social responses to computers. Journal of Social Issues, 56, 81–103. doi:10.1111/0022-4537.00153

108.

*Neyedli

H. F.

Hollands

J. G.

Jamieson

G. A.

(2009). Human reliance on an automated combat ID system: effects of display format. In Proceedings of the Human Factors and Ergonomics Society 53rd Annual Meeting (pp. 212–216). Santa Monica, CA: Human Factors and Ergonomics Society. doi:10.1177/154193120905300411

109.

*Pak

Fink

Price

Bass

Sturre

(2012). Decision support aids with anthropomorphic characteristics influence trust and performance in younger and older adults. Ergonomics, 55, 1059–1072. doi:10.1080/00140139.2012.691554

110.

Pak

McLaughlin

A. C.

Bass

(2014). A multi-level analysis of the effects of age and gender stereotypes on trust in anthropomorphic technology by younger and older adults. Ergonomics, 57, 1277–1289.

111.

Parasuraman

Sheridan

T. B.

Wickens

C. D.

(2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 30, 286–297. doi:10.1109/3468.844354

112.

*Rajaonah

Anceaux

Vienne

(2006). Study of driver trust during cooperation with adaptive cruise control. Travail Humain, 69, 99–127. doi:10.3917/th.692.0099

113.

*Rajaonah

Tricot

Anceaux

Millot

(2008). The role of intervening variables in driver-ACC cooperation. International Journal of Human–Computer Studies, 66, 185–197. doi:10.1016/j.ijhcs.2007.09.002

114.

Rani

M. R.

Sinclair

M. A.

Case

(2000). Human mismatches and preferences for automation. International Journal of Production Research, 38, 4033–4039. doi:10.1080/ 00207540050204894

115.

Reichenbach

Onnasch

Manzey

(2011). Human performance consequences of automated decision aids in states of sleep loss. Human Factor, 53, 717–728. doi:10.1177/ 0018720811418222

116.

Rice

Kraemer

Winter

S. R.

Mehta

Dunbar

(2014). Passengers from India and the United States have differential opinions about autonomous auto-pilots for commercial flights. International Journal of Aviation, Aeronautics, and Aerospace, 1(1), Article 3. doi:10.15394/ijaaa.2014.1004

117.

Ritter

Lyons

J. B.

Swindler

S. D.

(2007). Large-scale coordination: Developing a framework to evaluate socio-technical and collaborative issues. Cognition, Technology & Work, 9, 33–38. doi:10.1007/s10111-006-0060-x

118.

Rotter

J. B.

(1967). A new scale for the measurement of interpersonal trust. Journal of Personality, 35, 651–665. doi:10.1111/ j.1467-6494.1967.tb01454

119.

Rouse

W. B.

Morris

N. M.

(1986). On looking into the black box: Prospects and limits in the search for mental models. Psychological Bulletin, 100, 349–363. https://dx-doi-org.web.bisu.edu.cn/10.1037/0033-2909.100.3.349

120.

*Rovira

Parasuraman

(2010). Transitioning to future air traffic management: Effects of imperfect automation on controller attention and performance. Human Factors, 52, 411–425. doi:10.1177/0018720810375692

121.

Sanders

T. L.

Wixon

Schafer

K. E.

Chen

J. Y. C.

Hancock

P. A.

(2014). The influence of modality and transparency on trust in human-robot interaction. In Proceedings of the IEEE International Inter-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA) (pp. 156–159). San Antonio, TX: IEEE. doi:10.1109/CogSIMA.2014.6816556

122.

Sarter

N. B.

(2006). Multimodal information presentation: Design guidance and research challenges. International Journal of Industrial Ergonomics, 36, 439–445. doi:10.1016/j.ergon.2006.01.007

123.

Sarter

N. B.

Woods

D. D.

Billings

C. E.

(1997). Automation surprises. In Salvendy

(Ed.), Handbook of human factors and ergonomics (2nd ed., pp. 1926–1943). Hoboken, NJ: Wiley.

124.

Sauer

Nickel

Wastell

(2013). Designing automation for complex work environments under different levels of stress. Applied Ergonomics, 44, 119–127. doi:10.1016/j.apergo.2012.05.008

125.

Schaefer

K. E.

(2013). The perception and measurement of human–robot trust (Doctoral dissertation). University of Central Florida, Orlando.

126.

Schaefer

K. E.

Billings

D. R.

Szalma

J. L.

Adams

J. K.

Sanders

T. L.

Chen

J. Y. C.

Hancock

P. A.

(2014). A meta-analysis of factors influencing the development of trust in automation: Implications for human-robot interaction (Report No. ARL-TR-6984). Aberdeen Proving Ground, MD: U.S. Army Research Laboratory.

127.

Sharples

Stedmon

Cox

Nicholls

Shuttleworth

Wilson

(2007). Flight deck and air traffic control collaboration evaluation (FACE): Evaluating aviation communication in the laboratory and field. Applied Ergonomics, 38, 388–407. doi:10.1016/j.apergo.2007.01.012

128.

Sheridan

T. B.

(2014). What is God? Can religion be modeled? Washington, DC: New Academia.

129.

Skjerve

A. B. M.

Skraaning

Jr. (2004). The quality of human–automation cooperation in human-system interface for nuclear power plants. International Journal of Human–Computer Studies, 61, 649–677.

130.

*Smyth

C. C.

(2007). Sensitivity of subjective questionnaires to cognitive loading with driving with navigation aids: A pilot study. Aviation, Space, and Environmental Medicine, 78(5), B39–B50.

131.

Song

Kuchar

J. K.

(2003). Dissonance between multiple alerting systems part I: Modeling and analysis. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 33, 366–375. doi:10.1109/TSMCA.2003.817045

132.

*Spain

R. D.

Bliss

J. P.

(2008). The effect of sonification display pulse rate and reliability of operator trust and perceived workload during a simulated patient monitoring task. Ergonomics, 51, 1320–1337. doi:10.1080/00140130802120234

133.

Stanton

N. A.

Young

M. S.

(2005). Driver behavior with adaptive cruise control. Ergonomics, 48, 1294–1313.

134.

Stanton

N. A.

Young

M. S.

Walker

G. H.

(2007). The psychology of driving automation: A discussion with professor Don Norman. International Journal of Vehicle Design, 45, 289–306. doi:10.1504/ijvd.2007.014906

135.

*Stedmon

A. W.

Sharples

Littlewood

Cox

Patel

Wilson

J. R.

(2007). Datalink in air traffic management: Human factors issues in communications. Applied Ergonomics, 38, 473–480. doi:10.1016/j.apergo.2007.01.013

136.

Stokes

C. K.

Lyons

J. B.

Littlejohn

Natarian

Case

Speranza

(2010). Accounting for the human in cyberspace: Effects of mood on trust in automation. In Proceedings of the International Symposium on Collaborative Technologies and Systems (pp. 180–187). Chicago, IL: IEEE. doi:10.1109/CTS.2010.5478512

137.

Strauss

Koulechov

Röttger

Bahner

Trantakis

Hofer

Lűth

(2006). Evaluation of a navigation system for ENT with surgical efficiency criteria. Laryngoscope, 116, 564–572. doi:10.1097/01.MLG.0000202091.34295.05

138.

Sullivan

J. M.

Tsimhoni

Bogard

(2008). Warning reliability and driver performance in naturalistic driving. Human Factors, 50, 845–852. doi:10.1518/001872008X312332

139.

Szalma

J. L.

Hancock

P. A.

(2011). Noise effects on human performance: A meta-analysis synthesis. Psychological Bulletin, 137, 682–707. doi:10.1037/a0023987

140.

Szalma

J. L.

Taylor

G. S.

(2011). Individual differences in response to automation: The five factor model of personality. Journal of Experimental Psychology: Applied, 17, 71–96. doi:10.1037/a0024170

141.

Tang

Charalambous

Webb

Fletcher

S. R.

(2014). Users’ understanding of industrial robot gesture motions and effects on trust. In Proceedings of the International Conference on Ergonomics and Human Factors (pp. 116–123). Southampton, UK: CRC Press.

142.

Tenney

Y. J.

Rogers

W. H.

Pew

R. W.

(1998). Pilot opinions on cockpit automation issues. International Journal of Aviation Psychology, 8, 103–120. doi:10.1207/s15327108ijap0802_2

143.

Truszkowski

Hallock

H. L

Rouff

Karlin

Rash

Hinchey

Sterritt

(2009). Autonomous and autonomic systems: With applications to NASA intelligent spacecraft operations and exploration systems. London, UK: Springer-Verlag:.

144.

van den Broek

E. L.

Westerink

J. H.

(2009). Considerations for emotion-aware consumer products. Applied Ergonomics, 40, 1055–1064. doi:10.1016/j.apergo.2009.04.012

145.

*Verberne

F. M.

Ham

Midden

C. J. H.

(2012). Trust in smart systems: Sharing driving goals and giving information to increase trustworthiness and acceptability of smart systems in cars. Human Factors, 54, 799–810. doi:10.1177/ 0018720812443825

146.

Verberne

F. M.

Ham

Ponnada

Midden

C. J.

(2013). Trusting digital chameleons: The effect of mimicry by a virtual social agent on user trust. In Berkovsky

Freyne

(Eds.), Persuasive technology (pp. 234–245). Berlin, Germany: Springer.

147.

Walker

G. H.

Stanton

N. A.

Young

M.S.

(2001). Where is computing driving cars? International Journal of Human–Computer Interaction, 13, 203–229.

148.

*Wang

Jamieson

G. A.

Hollands

J. G.

(2009). Trust and reliance on an automated combat identification system. Human Factors, 51, 281–291. doi:10.1177/0018720809338842

149.

*Wang

Jamieson

G. A.

Hollands

J. G.

(2011). The effects of design features on users’ trust in and reliance on a combat identification system. In Proceedings of the Human Factors and Ergonomics Society 55th Annual Meeting (pp. 375–379). Santa Monica, CA: Human Factors and Ergonomics Society. doi:10.1177/1071181311551077

150.

Ward

J. N.

(2000). Automation of task processes: An example of intelligent transportation systems. Human Factors and Ergonomics in Manufacturing and Service Industries, 10, 395–408. doi:10.1002/1520-6564.

151.

Wickens

C. D.

Rice

Keller

Hutchins

Hughes

Clayton

(2009). False alerts in air traffic control conflict alerting system: Is there a “cry wolf” effect? Human Factors, 51, 446–462. doi:10.1177/0018720809344720

152.

Wiegmann

Eggman

A. A.

ElBardissi

A. W.

Parker

S. H.

Sundt

T. M.

(2010). Improving cardiac surgical care: A work systems approach. Applied Ergonomics, 41, 701–712. doi:10.1016/j.apergo.2009.12.008

153.

Wiegmann

McCarley

J. S.

Kramer

A. F.

Wickens

C. D.

(2006). Age and automation interact to influence performance of a simulated luggage screening task. Aviation, Space, and Environmental Medicine, 77, 825–831.

154.

Wilson

K. A.

Salas

Priest

H. A.

Andrews

(2007). Errors in the heat of battle: Taking a closer look at shared cognition breakdowns through teamwork. Human Factors, 49, 243–256.

155.

Wilson

J. M.

Straus

S. G.

McEvily

(2006). All in due time: The development of trust in computer-mediated and face-to-face teams. Organizational Behavior and Human Decision Processes, 99, 16–33. doi:10.1016/j.obhdp.2005.08.001

156.

Yamada

Kuchar

J. K.

(2006). Preliminary study of behavioral and safety effects of driver dependence on a warning system in a driving simulator. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 36, 602–610. doi:10.1109/TSMCA.2006.871646

157.

Yan

Wen

Olariu

Weigle

M. C.

(2013). Security challenges in vehicular cloud computing. IEEE Transactions on Intelligent Transportation Systems, 14, 284–294.

158.

Yang

C. W.

Yenn

T. C.

Lin

C. J.

(2009). Assessing team workload under automation based on a subjective performance measure. Safety Science, 48, 914–920. doi:10.1016/j.ssci.2010.03.011

159.

Yunicheva

(2012). Composition of Skunk Works teams (Master’s thesis). Halmstad University, Halmstad, Sweden.