Trusting Automation: Designing for Responsivity and Resilience

From Automation Supervisor or Subordinate to Interactive Partner

In supervisory control, human–automation performance depends on automation reliability, the human operator’s ability to assess the automation’s reliability, and to appropriately rely on, or comply with, the automation (Parasuraman & Riley, 1997). This supervisory arrangement tends to appear in well-established domains that benefit from the precise control of automation, such as industrial control systems. Such domains also tend to incur high costs when the automation fails, which is justification for keeping human experts in the loop. The reverse condition is when automation is used to monitor people—often frontline workers— thereby making them subordinates to automation. This arrangement has come as a consequence of computerizing or “informating” work environments (Zuboff, 1988), alongside the perception that organizations can gain control—and competitive advantage—by oppressing worker agency (Smith et al., 1992). As Zuboff (1985, p. 12) quotes a worker: “Are we all going to be working for a smart machine, or will we have smart people around the machine?”

Yet, these widely used hierarchical arrangements do not address the ambiguous and conflicting goals prevalent in complex work environments (Lewicki et al., 1998). In these environments, system success depends on active communication and coordination between participating agents (Sarter et al., 1997). These agents are more likely part of interdependent, networked relationships in resource-constrained environments that require negotiation, assurance, and cooperation—not just appropriate reliance—to resolve competing goals. These interactive relationships resist fitting neatly into the “types and levels of automation” categories that depict human–automation interaction as function allocation between people and automated systems (Parasuraman et al., 2000).

Several approaches to more interdependent work structures have been proposed, such as interactive function allocation (Pritchett et al., 2014; van Wezel et al., 2011), mixed-initiative interaction (Horvitz, 1999), coactive design (Johnson et al., 2014), interactive visual analytics (Sacha et al., 2016), and interactive machine learning (Amershi et al., 2014). These approaches envision automation working interactively with human counterparts, solving problems collaboratively, also referred to as “teaming” (Bartlett & Cooke, 2015; de Visser & Parasuraman, 2011; Zhang et al., 2015). Such teaming arrangements are typically deployed in complex environments where team goals may not always align with individual goals, risking human–automation breakdowns.

Designing trustworthy automation will be even more challenging as the scope of automation expands in safety-critical areas like healthcare, military, and emergency response, where the expectation for system efficiency is high and the tolerance for breakdowns is very low (Casper & Murphy, 2003; Defense Science Board, 2012; Robinette et al., 2013). Because safety-critical domains tend to have stringent requirements for technology reliability, breakdowns in these settings may have less to do with technology reliability and more to do with failures in goal alignment between units of an organization (Hall, 2003; Romzek & Dubnick, 1987). Therefore, given automation that functions as taskmaster, monitor, or influencer that can undermine human judgment, there is potential automation that promotes goal alignment through teamwork, and human agency through interactivity. Framing trustworthy automation in terms of reliability and reliance considers the known tradeoffs of supervisory control (Kirlik, 1993), but neglects the costs of goal conflict, how to resolve those conflicts, and how to provide trust assurance (Israelsen & Ahmed, 2019). Such conflicts may be frustrating when the costs of failure are trivial. In more consequential settings, such conflicts can degrade worker health and organizational resilience (Hancock et al., 2019).

Organizational Restructuring

The move toward more capable automation not only reflects societal needs in a wide variety of sectors, from healthcare to transportation; it also reflects organizational imperatives to gain advantage in competitive markets. The disaggregation of Tayloristic, hierarchical work structures of the Industrial Revolution, to the more interdependent, lateral work processes of the Information Age (Adler, 1997, Adler, 2001; Blomqvist & Stahle, 2004) led to achievements that were once thought impossible, like landing military aircraft on carriers at sea or reviving patients from deathlike states (Gawande, 2009; Rochlin et al., 1987). Although technological innovation was a crucial component of these achievements, these shifts also reflect increased coordination among human specialists. Recent innovations in automation seem poised to continue this trend, with automation rather than human specialists taking on roles that will require lateral coordination with other agents, including people (Frey & Osborne, 2013, Frey & Osborne, 2017).

Studies of highly reliable organizations and of human teaming in complex work environments show that the proficient use of social processes—such as trust—is essential to coordination, rather than relying on individual skill, static knowledge structures, or having well-defined roles (Cooke et al., 2013; Duhigg, 2016; Rochlin et al., 1987). In dynamic environments, top-down procedures or pressures that are far removed from the situation are less effective than they are in more stable environments. When prespecified controls (like standard operating procedures) are insufficient at reducing task uncertainty, work structures that allow for ad hoc coordination among agents are more effective (Cummings, 1978). Under these flexible work structures, in dynamic environments, coordination and cooperation typically emerge with trust playing a central role. Therefore, rather than calibrating trust to automation capabilities for appropriate reliance and compliance, these environments demand supporting a dynamic process of trusting automation for effective coordination through cooperation. As organizations flatten, the need to perform reliably and robustly will be joined by the need to coordinate dynamically and adeptly (Woods, 2015).

Flexible Systems for Resilience Require Cooperation

Alongside the flattening of organizations, there is also a shift toward more flexible and resilient systems to accommodate surprises and perturbations in the environment (Hollnagel et al., 2006). Resilience refers to a system’s ability to manage sustained adaptability in such environments. This differs from resilience as rebound, restoring a system to previous conditions before a disruption, or resilience as robustness, responding effectively to disturbances. Rather, resilience as sustained adaptability considers how interconnected agents within a network cooperate and draws on shared resources to accommodate surprises (Woods, 2015).

Decisions to cooperate in these networks are affected by different goals and priorities, such as the relative priority of individual goals versus group goals. Cooperative behavior is easiest to identify in actions that benefit a group despite individual costs (Dugatkin et al., 1992). This need for compromise, or reconciliation, applies to teaming as well as to groups of specialists coordinating within a traditionally hierarchical organization (Hall, 2003). Participation in these groups therefore involves coordination between private and public goals (Clark, 1996), self and collective interest (Ostrom, 2000), ego-centric and collaborative interests (Hoc, 2001), or local and global goals (Woods, 2005). These constructs (private, public…local, global) are historically and thus conceptually distinct, but they share the idea that cooperation emerges by reconciling myriad competing goals through social processes, like trusting.

Considering the requirements for cooperation and joint adaptation, increasingly autonomous automation is only part of the solution. Increasingly autonomous automation must also be increasingly responsive, to support a process of trusting. Responsive automation adjusts its goals and interaction strategies to the environment and to others’ shifting goals and intents. Automation responsivity—the degree to which the automation effectively adapts to the person and situation—will be crucial for preventing and recovering from breakdowns with increasingly autonomous automation. Responsive automation requires a relational perspective on the process of trusting that is the focus of this paper.

Trusting From Automation Responsivity

The term “responsivity” has been used to describe the input–output gain of a detector system, reflecting an ability to adjust to sudden, altered conditions in the environment and to resume stable operation. Responsivity captures the notion that automation could affect trust development through its responses to common ground variables. Common ground refers to the mutual knowledge, beliefs, and assumptions required to support interdependent actions in some joint activity (Clark & Brennan, 1991; Klein et al., 2005). This includes expectations for social responses, like collegiality.

As such, the traditional notion of reliability as a quality of automation may be supplemented and enveloped by the notion of responsivity. Whereas automation reliability influences reliance and compliance behaviors through trust, automation responsivity influences patterns of engagement and cooperation, through trust development. Friedland (1990) found that trust was promoted most often when one party demonstrated genuine responsiveness to the needs of the other party, not just due to the reciprocity of trust (trusting because one feels trusted), but also because of the strong contingency between the behaviors of those involved. Given these contingencies, automation designers should consider not only what information people need to appropriately trust automation, but also policies for people to explore its competence envelope, to adjust automation parameters, and to track and learn from norm deviations (Hall, 2003; Hoffman et al., 2013).

An example of designing for automation responsivity could be to consider how trust evolves as automation shifts roles over the lifespan of a relationship, for example, from a tutor to a conversational partner, or from a manager to a teammate (Marin et al., 2005). Such shifts could be due to changing needs of the person or the automation, which can be indicated through interactions. For example, a person’s requests for explanations might signal a lack of knowledge, or a lack of trust in a suggestion. If the former, the automation might assume a tutor role; if the latter, the automation might assume a conversational partner role. Responsive automation could also judge its competence and nudge people into different roles through social signals, such as gaze behavior (Jung et al., 2015). Shifting roles within a coordinative context can develop into trusting relationships through responsivity, to facilitate downstream outcomes like purchasing decisions or learning (Cassell & Bickmore, 2000; Liew et al., 2013; Miller, 2019; Traeger et al., 2020). Designing for responsivity avoids a zero-sum tradeoff between human control and automation autonomy, in which automation is optimized for competence in a discretized role, rather than for teamwork involving coordination and cooperation between roles (Bansal et al., 2020; Garbis & Artman, 1998).

To summarize, automation has become more sophisticated and autonomous, organizations are flattening and might capitalize on teams that include automated agents, but such teams must coordinate and cooperate in novel ways to support organizational resilience. These trends force a rethinking of the role of trust. Most research on trust in automation has focused on whether the automation was competent (i.e., reliable) and how this affects appropriate reliance and compliance behaviors. With more autonomous automation, the focus should broaden to consider whether the automation is coordinative (e.g., times its contributions well) and cooperative (e.g., negotiates goals effectively). Understanding how trust facilitates these processes requires refocusing on how automation responsivity affects trusting, and understanding how trust develops from interactions. Designing for system resilience requires thinking beyond people adapting to automation, or automation adapting to people, to include how people and automation promote mutual and sustained adaptation.

Situation: Cooperation, Complementarity, and Control

A social situation is a concept from interdependence theory that defines the interdependence of a dyad. It describes when trust is needed to achieve a goal, where the misaligned goals of two agents could undermine cooperation, the degree of mutual influence (to what extent an agent has control over an other agent’s decisions), and how they contribute to the overall outcome (Thibaut & Kelley, 1959). For automation design, situations are powerful because they can be constructed within a goal environment, to enable control of interactions, to engender trust, or to allow for one agent to probe the trustworthiness of the other.

Formalized situation structures are typically represented as discrete event matrices that serve as proxies for social reality to which cognitive processes must be adapted (Holmes, 2004; Thibaut & Kelley, 1959). These structures can be used in algorithm development (Wagner & Arkin, 2011) and as frameworks to assess the situation, based on the reward structures of the goal environment (Wickham & Knee, 2012). In its simplest form, a situation structure specifies the choices available to each actor in a dyad, the outcomes of their choices, and how those outcomes depend on the choices of the other agent. Figure 2 shows these elements of a situation structure to indicate where trust is involved in decisions.

OPEN IN VIEWER

Figure 2

Trust plays a role in both coordination and cooperation situations; for coordination, the benefits of trust are interpersonal and affect longer-term relationship outcomes (e.g., collegiality), whereas for cooperation trust directly affects the immediate decision outcome.

The matrix on the left (Figure 2) shows two agents with a common goal, but the best outcome depends on Agent_H and Agent_A arranging their behavior such that both choose the same option [H1, A1] or [H2, A2], that is, coordination. The matrix on the right shows a goal conflict, indicating that trust is needed (in this case, trust that the other agent will not choose to defect) for both parties to avoid negative outcomes, that is, cooperation. Therefore, coordination is about the timing or arrangement of joint decisions, whereas cooperation is about negotiating and aligning individual goals when they differ from the joint goal. Negotiating goals to cooperate—by way of trust—has different outcomes compared to coordination. Importantly, situation structures indicate which decisions require negotiation.

The concept of situation structures departs from previous approaches to trust in automation. Such structures focus not just on a person’s perceptions of the automation’s characteristics (the information processing view); they also highlight the interdependence of trust, including the attributions and expectations of another agent’s intentions, motivations, and goals when deciding one’s own goals or actions. Even in studies of reliance and compliance, situation structures can help explain cases in which an operator’s performance changes the value of an alarm or alert. For example, a skilled operator’s actions may result in lowering the probability of an alarm. Trust in these cases would depend not only on perceived information about the automation, but also on the probability of an alarm, a value that reflects the operators’ actions rather than the automation’s independent performance (Meyer & Bitan, 2002). Approaching trust as a relational concept captures this interdependence.

A focus on the trust situation also reveals when trust calibration matters. Trust calibration—aligning a person’s trust with the automation’s capabilities—is often described as a prerequisite for superior human–automation performance (Lee & See, 2004). However, conflicting results show that trust calibration alone is not sufficient for superior performance (McGuirl & Sarter, 2006; Merritt et al., 2015; Zhang et al., 2020). The situation structures show that when the performance profile of the person and the automation are highly correlated, trust calibration does not matter (Zhang et al., 2020). More generally, the situation structure indicates that it might be appropriate for people to rely on automation, even when it performs worse on a task than they would, because relying on the automation enables them to shift attention to a more important activity. The situation structure encourages an analysis that considers reliance in a broader sociotechnical context.

Semiotics: Revealing Capability and Aligning Intent

Semiotics is the study of how meaning is formed through signs, which unite the signifier (what is perceived) with the signified (the concept communicated; Flach, 2017; Sebeok, 2001; de Souza, 2017). Semiotics helps to explain how people construct meaning from interactions with automation and representations of the automation, which guides and is guided by the process of trusting. Signs of trust, or trust-related information, can be encoded as symbols and names; icons and indexes; symptoms and signals. Symbols include emblems, badges, or certifications that represent extrinsic assurance of trustworthiness, as with software certification. Names designate instance as a member of a larger class and can reflect role and function. Icons and indexes encode in an analog fashion through resemblance or location in time and space. In contrast, symptoms and signals are intrinsic indicators of trust that are incidental to the interaction (Riegelsberger et al., 2005). Symptoms result from a physical process and can demonstrate competence, intent, and motivation. Symptoms and signals underlie how people assess and demonstrate compliance with cultural norms (Parasuraman & Miller, 2004) and how people infer intent from actions (Chiou & Lee, 2016; Chiou et al., 2019; Cooke et al., 2013). Signals are often emitted unwittingly but can be produced wittingly (Sebeok, 2001). For example, in a collaborative task, robot motion is a symptom or signal that can be designed to communicate intent, by exaggerating movements, or to convey competence, by moving smoothly and predictably (Dragan et al., 2013). Symptoms and signals support more resilient performance because they reflect intrinsic qualities of the immediate operating context.

People’s perceptions of automation qualities and behaviors have long been the focus of understanding trust (Hancock et al., 2011; Schaefer et al., 2016), particularly how visual representations affect the analytic and affective interpretation of trust signs (Lee & See, 2004). The interpretations of these signs change fundamentally when moving from teleoperation (Sheridan & Verplank, 1978) to more complex automation that enables—or is enabled by—perceived agency and associated social cues (Takayama, 2009). These social cues act as symptoms and signals whether or not the automation was designed to produce them (Epley et al., 2007; Mutlu et al., 2012; Szafir et al., 2015). Furthermore, the strength of social cues depends on the task context. For example, proxemic behavior (use of physical space) strongly affects a hallway navigation task, and direction of eye gaze strongly affects an object handoff task (Fiore et al., 2013; Mutlu et al., 2012). Designers should consider both instrumental and communicative aspects of automation behavior; yet, understanding what symptoms and signals are available and how they are interpreted in different contexts remains an understudied area.

Contrasting a focus on individuals observing automation, a relational approach would argue that either agent can generate and respond to symbols and symptoms of trust. A person might probe automation to better understand its behavior, which can itself signal uncertainty or a lack of trust. Even with different propensities to trust, a period of unconditional, message-consistent behavior immediately after a message can make low trusters responsive to cooperative messages, and high trusters responsive to competitive messages (Parks et al., 1996). Actions signal information about an agent’s value system or the choices of action available to the agent, and can suggest unilateral progress independent of explicit agreements (Clark, 1996; Schelling, 1960) because actions commit where speech and symbolic meaning cannot. Such actions are therefore epistemic—they serve to communicate meaning and to enhance responsivity, not just to achieve a goal. Other non-goal-directed actions such as mimicry, and coordination without obvious cooperation, can indicate engagement in joint activity and being in tune with the other (Riegelsberger et al., 2005; Tomasello & Carpenter, 2007). Such actions, whether or not they are intentional, helps to establish common ground (Clark & Brennan, 1991).

Sequence: Timing, Initiative, and History

Sequence reflects the timing and order of interactions between two or more agents, whether decisions are made synchronously, asynchronously, in a particular order, or in free form. Sequence complements situation and semiotics; it is the order of actions, signals, and events that influence subsequent actions, signals, and events, as in hysteresis, path dependency, and behavioral lock-in (Barnes et al., 2004). The sequence of interactions defines coordination, and agents that coordinate poorly are likely to be distrusted.

The type of sequence can affect trusting and performance. For instance, a synchronous sequence of interactions (decisions and actions that occur in parallel) may force partners to rely on affective trust, rather than analytical trust, because they have less time to seek out signs to confirm or disconfirm the partner’s trustworthiness, before a decision or action must occur. An asynchronous sequence of interactions (decisions and actions that occur serially) may rely more on past actions or signs, given that its temporal structure allows for a separate analysis of trust and acting trustworthy (Riegelsberger et al., 2003). The initial act or default state of automation may disproportionately affect a person’s decisions, compared to subsequent acts or states, even for high-consequence choices such as retirement savings and organ donation (Goldstein et al., 2008; Johnson & Goldstein, 2003). Defaults have a strong initial effect on interactions and may have a lasting influence on the relationship that develops.

Little is known about the costs and benefits of different interaction sequences on trust. For example, automation that relies on a person to initiate interactions may see reduced use during high workload situations (Chiou & Lee, 2016), which bodes poorly for developing trust through shared adversity (Bastian et al., 2018). On the other hand, automation that always initiates interactions may lead to higher complacency and reduced engagement in a shared task. Interestingly, social robots that took initiative were more trusted when engaged in a service task, rather than small talk (Babel et al., 2021). These findings indicate that sequence matters for joint performance, but that its effect depends on the task (i.e., the situation).

New architectures of interactive agents enact sequences such as turn-taking (e.g., chatbots, Siri) or a pseudorandom order that depends on context (e.g., autocorrect during texting, in-vehicle automated safety features). One approach to interactive human–robot coordination represented the sequences as a Markov decision process that included a representation of the task, the cost of communication, and a model of the human response (Unhelkar et al., 2020). These sequences may engage participants more in the shared task of composing a message, or navigating a roadway, but can crowd out other important tasks in fast-paced settings, such as attending to changes in the goal environment (Lee, 2009).

Transition lists define the sequence of situations

Sequence also addresses potential movement between situations and how agents move from a previous situation to a new one with different choices and outcomes (Rusbult & Van Lange, 2003); i.e., a transition list (Kelley, 1984). More concretely, when two choices and two outcomes are specified for each agent in a dyad, this corresponds to the 2 × 2 situation structure (e.g., Figure 2). When a further consequence of each combination of choices is also specified, and a particular combination of behaviors is enacted, a transition list is defined. Transition lists assume that the agents are interdependent not only in how they control their own and each other’s immediate outcomes but also in their movement through situations. Therefore, interdependence can be evaluated at different time scales. Over time, repeated interactions between the same parties or extended situations (Rusbult & Van Lange, 2003) evolve into patterns, which then define relationships (Figure 3). Such relationships may make a habit of trusting (Neal et al., 2006), which has different properties from initial trust or swift trust (Meyerson et al., 1996).

OPEN IN VIEWER

Figure 3

Interacting time scales from interactions (“I”), influenced by (perceived) intents and signals; that lead to situations (“S”), defined by goals and actions; which together define relationships (“R”), formed by trust frames and social norms.

Interactions across situations that combine to form relationships affect trust in a manner that is not simply based on learning or accumulation of trust-relevant information. Some models describe trust evolution in terms of accumulation; trust increases and decreases according to the occurrence of positive and negative outcomes of a sequence of situations (de Visser et al., 2020; Gao et al., 2006). These approaches typically treat people as passive recipients of trust-relevant information.

An alternate view treats trust as an active sensemaking process guided by a trust frame (Fallon et al., 2010). The trust frame helps form and define relationships, guides what information people seek, and how they interpret this information. A trust frame associates qualitative (categorical) difference with high or low levels of trust. For example, one trust frame is the “perfect automation schema” (Dzindolet et al., 2002; Lyons & Guznov, 2019), and this would lead people to place less weight on poor automation performance and use good performance to sustain the “perfect automation” frame. Algorithm aversion is the opposite trust frame that leads people to overweight poor performance (Dietvorst et al., 2014; Prahl & Van Swol, 2017). The most extreme trust frame might be one of suspicion where people view automation as antagonistic. An ambivalent trust frame that reflects unfamiliarity with the automation might lead to more information seeking and experimentation. Other frames include “cooperation threshold” and “limit of forgivability” (Han & Schulz, 2020). Changing frames requires effort and expertise and so once adopted a frame persists (Klein et al., 2006).

Because trust frames are persistent, and because they guide information seeking and interpretation, the same outcome of an interaction within a particular sequence might have very different consequences for the accumulation of trust. Models of trust evolution might be improved by modeling transitions between trust frames and conditioning the accumulation of trust on the trust frame. Likewise, responsive automation might benefit from using frames to guide its estimate of trust, to not just calibrate trust, but also to reframe trust, such as prompting people to exert a little control over the automation to counteract algorithm aversion (Dietvorst et al., 2018).

Strategy: Navigating Situations

Strategies describe a series of actions, norms, or policies that control interactions to achieve a desired goal. In trusting automation, strategies may include enacting policies that are known to generate cooperative outcomes over time, changing the outcome values of a situation to test an agent’s trustworthiness, or leveraging factors in the goal environment to create a new situation. Some strategies rely on a particular sequence of actions, such as tit-for-tat, a well-known cooperative strategy that also avoids exploitation (Axelrod, 1984). Other strategies can be implemented by specifying a cost function from which coordination and cooperation can emerge, such as assistance games (Fickinger et al., 2020; Sadigh et al., 2018; Woodward et al., 2020). Shared strategies between interacting agents are typically agreed-upon ways of achieving goals that may involve resource tradeoffs.

What differentiates strategies from other patterns of action is that strategies are goal oriented. For example, there may be strategies that are primarily used to reduce the costs of obtaining, processing, and communicating information between two or more agents working together (e.g., Miller et al., 2005; Sadigh et al., 2018). Strategies that can reduce these costs are critical for the performance of knowledge-based organizations (Arrow, 1974), but the success of such strategies depend on trust, to avoid the costs of information verification at every step. A central challenge with enacting cooperative strategies is that an agent working to enhance the resilience of an organization (e.g., Stephens et al., 2011) might perform poorly individually, potentially undermining initial trust in that agent. When executed successfully, however, such strategies can further facilitate trusting.

Implications for Design

This framework specifies why, when, how, and which factors might enhance trusting increasingly capable automation and identifies potential understudied factors of trust in the human–automation interaction literature. We postulate that certain factors may have been understudied because of the wide application of more narrowly scoped frameworks that are based on supervisory control automation, and the information processing view of trust.

For example, design guidelines that promote transparency or “make clear what the system can do” (Amershi et al., 2019, p. 3) offer necessary but insufficient guidance for system designers. Conducting a task analysis to reveal what the system can do is an important step for human-centered design, but task analyses do not address how trust develops during repeated interactions and across situations. Focusing on automation responsivity in joint decision making can help capture key elements of trusting in interactive, goal-oriented environments. A focus on automation responsivity makes clearer the need to distinguish between features used by the automation and the input variables needed for human interpretation (Ribeiro et al., 2016), or to use simpler automation that does not require explanations and can be adjusted by the user (Rudin, 2019). Designing for automation responsivity goes beyond supporting human observations and interventions, to make it possible for a person to influence automation and for the automation to influence the person.

The framework also highlights trust theories that have been neglected in prior empirical work. Together, these theoretical perspectives can inform design at the level of the algorithm, interaction, interface, and even the organization, by way of human–automation joint action. These perspectives may be critical for evaluating the long-term efficacy of current automation and for designing new automation. To support this effort, the four-concept framework for trusting automation can be used to:

Extending This Framework

The foundation of this framework rests on the formal representations of interdependent trust situations. However, real-world situations are not usually so simple. The payoffs in a situation may be difficult to quantify, and for most decision models in the wild, defining a common unit of measure for different outcomes can be challenging. For example, how does one place the perceived risk of injury or death of a pedestrian on the same scale as a 30-s delay for the passengers in a vehicle (Domeyer et al., 2019)? Additionally, it is less clear how specifying the cell values within a structure, typically determined through surveying a sample population (Kelley et al., 2003), would translate to in-the-moment, real-world decisions. It is clear that additional research is needed to distinguish between people’s reflective responses—decisions made when given time to assess—and their in-the-moment decisions with increasingly agentic technology (Schilbach et al., 2013; Takayama, 2009).

The situations themselves also do not fully capture the metasituation, depicted as the “goal environment” in Figures 1 and 4, or how agents may come to control the construction of these situations, such as speeding up to negotiate a “game of chicken,” or slowing down to indicate an intention to allow a pedestrian to cross. Such decisions that occur in-the-moment can change the situation for some people, but not for others who do not notice it. To better represent reality each situation would not be isolated, but embedded in a sequence of situations. Making these patterns clear so far seems to be a retrospective exercise, or prospective only for the next near situation, given the same challenges of forecasting in complex environments.

Although simplistic representations only account for a portion of the factors, at the same time, they can help sort through the complexity of reality—where there are multiple parties with varying interests, varying levels of control, and limited resources. Situation structures thus help with identifying critical decision points, and the likely decisions that will be made, based on the rewards of an agent, or on the probability that one action will be preferred over another. Altogether, this framework for trusting in dynamic, human–automation joint action highlights a new perspective to guide future job design, information design, and interaction design.