Cooperation in Human-Agent Systems to Support Resilience

Abstract

Objective:

This study uses a dyadic approach to understand human-agent cooperation and system resilience.

Background:

Increasingly capable technology fundamentally changes human-machine relationships. Rather than reliance on or compliance with more or less reliable automation, we investigate interaction strategies with more or less cooperative agents.

Method:

A joint-task microworld scenario was developed to explore the effects of agent cooperation on participant cooperation and system resilience. To assess the effects of agent cooperation on participant cooperation, 36 people coordinated with a more or less cooperative agent by requesting resources and responding to requests for resources in a dynamic task environment. Another 36 people were recruited to assess effects following a perturbation in their own hospital.

Results:

Experiment 1 shows people reciprocated the cooperative behaviors of the agents; a low-cooperation agent led to less effective interactions and less resource sharing, whereas a high-cooperation agent led to more effective interactions and greater resource sharing. Experiment 2 shows that an initial fast-tempo perturbation undermined proactive cooperation—people tended to not request resources. However, the initial fast tempo had little effect on reactive cooperation—people tended to accept resource requests according to cooperation level.

Conclusion:

This study complements the supervisory control perspective of human-automation interaction by considering interdependence and cooperation rather than the more common focus on reliability and reliance.

Application:

The cooperativeness of automated agents can influence the cooperativeness of human agents. Design and evaluation for resilience in teams involving increasingly autonomous agents should consider the cooperative behaviors of these agents.

Keywords

autonomous agents human-automation interaction technology acceptance trust in automation team collaboration

Automation is becoming increasingly autonomous. From self-driving vehicles to sophisticated decision support systems, computational advances have led to a plethora of machines capable of automating whole functions that previously required people. Emerging new human-automation relationships pressure existing frameworks for research, design, and evaluation of these joint systems (Woods & Hollnagel, 2005). People and machines increasingly enter into coordinative and cooperative relationships, where successful interactions demand teamwork. Resilience engineering approaches the design of such systems as consisting of interactive agents, collaborating to achieve shared goals in increasingly dynamic and safety-sensitive environments (Hollnagel, Woods, & Leveson, 2006). However, few controlled studies have investigated how resilience depends on automation design, which is the focus of this paper.

Background

For supervisory control, joint performance depends on automation reliability and people’s ability to assess automation reliability, so they can appropriately rely on or comply with automation (Parasuraman & Riley, 1997). However, with more advanced automation, reliability may be less critical than the resilience of the system (Zieba, Polet, Vanderhaegen, & Debernard, 2010). Resilience refers to the ability to manage sustained adaptability of a layered network system (Woods, 2015). This differs from resilience as rebound, to restore a system to previous conditions before disruption, or resilience as robustness, having an expanded set of models to respond to disturbances (Woods, 2015). Rather, resilience as sustained adaptability considers how interacting agents cooperate, drawing on shared resources to accommodate surprises. Designing for resilience suggests collaborative or cooperative control might be an important complement to supervisory control.

For human-automation interaction, collaborative control allows both human and automated agents to combine competencies and negotiate conflicts in dynamic settings (Zieba et al., 2010). Collaborative control supports resilience through adaptation to novel situations, accommodation of bugs in the system itself, and recovery from errors in following procedures (Woods, Johannesen, Cook, & Sarter, 1994). Furthermore, resilience calls for the continual renewal of shared goals, reciprocity, and the willingness to accommodate others as unexpected demands require shifts of individual and shared priorities (Woods, 2004). System performance is less about how well predetermined priorities are reliably executed and more about how well goals and actions are adapted for the greater good of the system. Thus, rather than collaborative control, which often refers to managing functional dependencies by adapting levels of autonomy, cooperative control may be more appropriate for human-automation scenarios where cooperation precedes coordination.

Resilience in uncertain environments with complex interdependencies demands alternatives to the more explicit supervisor-subordinate procedures. These alternatives include informal interactions that promote coordination and cooperation (Rochlin, La Porte, & Roberts, 1987). Coordination is broadly defined as dependency management (Malone & Crowston, 1994) and concerns task scheduling and assignment, whereas cooperation is identified in actions that benefit a group but impose individual costs (adapted from Dugatkin, Mesterton-Gibbons, & Houston, 1992). The need to compromise individual goals for a shared goal can arise from what Clark (1996) calls private and public goals, Ostrom (2000) calls self- and collective interest, Hoc (2001) calls ego-centric and collaborative interests, and Woods (2004) calls local and global goals. These constructs share the idea that cooperation emerges through reconciling competing goals with the help of social processes. When interdependent activity involves shared resources and conflicting goals, cooperation is essential for coordination.

The importance of cooperation in resilience is highlighted in the margin of maneuver—a cache of actions and resources that allows the system to function despite unexpected perturbations in the work environment (Woods & Branlat, 2010). Systems without adequate margins become brittle, unable to withstand unexpected demands. Based on studies of emergency department interactions with other hospital units, Stephens, Woods, Branlat, and Wears (2011) identify three classes of strategies to create or maintain a margin of maneuver: defensive strategies, autonomous strategies, and cooperative strategies. Defensive strategies increase the margin of a unit by restricting another unit. Autonomous strategies involve local focus, such as reducing interactions with other units because the benefits of sharing resources are not believed to be worth the costs. Cooperative strategies involve the effort of two or more units that through coordinated and collective action recognize or create common-pool resources from which both units can draw. Such actions help avoid the tragedy of the commons when resources are limited (Ostrom, 1999) and may enhance system resilience by giving the overall network a larger pool of resources to draw on for greater margin of maneuver.

In light of these properties of resilience in volatile environments and increasingly capable automation, understanding what contributes to human-agent cooperation seems crucial. When problems cascade, demands for cognitive activity increase, as do the demands for coordination across agents (Woods & Patterson, 2001). These increased demands may accentuate bonding or fracturing of teams depending on the nature of their interdependence and the information available regarding the trustworthiness of agents (Gao, Lee, & Zhang, 2006). Cooperative strategies require assuming initial generosity (Axelrod & Hamilton, 1981), trust that the other party will reciprocate and not exploit shared resources, and an understanding that the benefits of cooperation will outweigh the costs. Without such trust between agents, an agent may resort to defensive strategies to preserve individual gain at the cost of another (De Dreu et al., 2010).

A substitute for formal controls, trust guides human behavior toward automation when a complete understanding is difficult or impractical (Lee & See, 2004). Such uncertainty is increasingly common due to technological advances and the environments in which automation are deployed. In high-stress environments, human agents may be even less likely to proactively interact with automation, particularly when experiencing physical and cognitive fatigue (Casper & Murphy, 2003). Although trust in automation has been said to be asymmetrical, in that automation does not trust back (Lee & See, 2004), recent work with more autonomous automation (e.g., agents or robots) suggests a more symmetrical view, and social exchange situations may apply to the concerns raised by resilience engineering (Bray, Anumandla, & Thibeault, 2012; Fink & Weyer, 2014; Wagner & Arkin, 2011). In cooperative exchange, people often choose partners depending on the instrumental value of the exchange, even though people’s trust, affective regard, and sense of solidarity with exchange partners are strongly influenced by the symbolic act of reciprocity (Molm, Schaefer, & Collett, 2007). Reciprocity and trust influence cooperative behavior, especially when novel or complex situations involve risk or uncertainty (Delton, Krasnow, Cosmides, & Tooby, 2011; Molm, Takahashi, & Peterson, 2000; Riedl, Mohr, Kenning, Davis, & Heerkeren, 2011).

Existing approaches to human-automation cooperation and resilience consider how automation can facilitate collaboration with people in dynamic environments (Allen, Guinn, & Horvitz, 1999; Fong et al., 2005; Wagner & Arkin, 2011; Woods, Tittle, Feil, & Roesler, 2004; Zieba, Polet, Vanderhaegen, & Debernard, 2009). Other studies address the effects of socially sensitive automation, such as developing trust through conversational cues, appearance, and behavior (Cassell & Bickmore, 2000; Desteno et al., 2012; Robinette, Wagner, & Howard, 2013), or engaging in good or poor etiquette (Parasuraman & Miller, 2004; Takayama, Groom, & Nass, 2009). Another study explored the effects of adaptive automation on human-robot team fluency and subjective ratings of satisfaction, safety, and comfort (Lasota & Shah, 2015). However, these studies focused on reliability and reliance or were conducted in static task environments. At the time of this writing, we were unable to find studies that considered cooperation with automation in a dynamic, shared resource task environment.

As a first step in understanding cooperation in networked and layered sociotechnical systems, we consider the human-agent dyad (Thibaut & Kelley, 1959; Williams, 2010). Our goal is to demonstrate that for resilient systems, an important complement to reliance on automation is cooperation. To achieve this goal, we developed a microworld that required human and automated agent dyads to cooperate on a dynamic, shared resource task. We present two experiments; Experiment 1 tests the hypothesis that an agent’s cooperation will affect people’s cooperation, leading them to reciprocate with similar resource exchange behaviors. Experiment 2 tests the hypothesis that when people’s initial exposure to an agent is during an unexpected, highlydemanding situation, then the effects of agent cooperation will be particularly prominent.

Method

A microworld hospital scheduling scenario was developed in Java and XML with the Android SDK, to assess cooperative behavior in a joint human-agent task with shared resources. A 15” laptop computer running Genymotion and a standard computer mouse were used. Participants acted as hospital schedulers, whose task involved assigning patients and staff to hospital rooms and coordinating shared staff resources with a neighboring hospital managed by an automated agent. Interactions with the agent were bilateral and limited to requesting resources and responding to requests for resources. The study tested different levels of agent cooperation on participants’ behavior and joint performance. In each of their four experimental trials, 36 participants in Experiment 1 experienced a slow-tempo period followed by a fast-tempo period (the slow-to-fast group), and 36 participants in Experiment 2 experienced a fast-tempo period followed by a slow-tempo period (the fast-to-slow group). Testing these two tempo sequences allowed us to assess the effects of agent cooperation level in the context of recovery following a perturbation and across variation in work environments.

Experimental Design

Study participants

Participants were recruited near a Midwestern university through flyers and online postings and received $10 at the end of the hour-long study. Self-reports showed age ranged from 18 to 56 with a mean age of 22 in Experiment 1 and a mean age of 24 in Experiment 2; gender was roughly 50% female and 50% male; 97% of participants used the computer daily; 47% in Experiment 1 and 64% in Experiment 2 reported using the computer for playing games.

Independent variables

Each experiment tested “agent cooperation level” as a within-subjects variable, meaning each participant was exposed to a high-cooperation and a low-cooperation agent (Figure 1). Agent cooperation levels were operationalized in the agents’ resource-requesting and resource-sharing behaviors. Exposure to the agents was counterbalanced, so participants worked with one agent in their first two trials and the other agent in their final two trials. Within each trial, there was one slow-tempo period and one fast-tempo period, and tempo was tested as a within-subject variable. Scheduler was a within-subject variable to distinguish between the participant and the automated agent. Therefore, each study was a mixed within and between 2 × 2 × 2 × 2 design that tested agent cooperation level (high or low cooperation), tempo (fast-tempo period or slow-tempo period), and scheduler (person or agent) as within-subject variables and cooperation order as a between-subject variable (high cooperation followed by low cooperation, or low cooperation followed by high cooperation).

Figure 1.

In Experiment 1, 18 participants experienced the high-cooperation agent then the low-cooperation agent, in a slow-to-fast tempo sequence. The other 18 participants experienced the counterbalanced order of exposure to the cooperation agents. Experiment 2 was the same design as Experiment 1, except participants experienced the fast-to-slow tempo sequence.

Tempo levels: Slow-tempo and fast-tempo periods

Each eight-minute trial was divided into five 96-second intervals, representing a slow, medium, or fast rate at which patients entered the waiting rooms. In the slow-to-fast sequence of Experiment 1, participants experienced the following interval order: medium, slow, medium, fast, and medium. In the fast-to-slow sequence of Experiment 2, participants experienced the following interval order: medium, fast, medium, slow, and medium (see Figure 1). For easier communication, however, tempo period refers to the half of the trial that includes the named interval, for example, the slow-tempo period in the slow-to-fast sequence was one medium interval, one slow interval, and half a medium interval. The slow interval had two patients enter the waiting room, evenly distributed throughout that interval, whereas the medium interval had five patients, and the fast interval had eight patients. The medium-tempo interval was determined in pilot testing as an engaging pace, without high pressure.

In both experiments, the agent’s tempo sequence was always the opposite of the participant’s tempo sequence. For example, in participants’ slow-to-fast sequence of Experiment 1, agents would experience the fast-to-slow sequence. The complementary sequences allowed for an obvious coordination solution if the dyad cooperated and shared resources, making it possible to assess how differences in cooperative behavior contributed to breakdowns or delays in resource sharing. Participants (or agents) in the slow-tempo period would more likely have excess resources, whereas participants (or agents) in the fast-tempo period would need additional resources to meet demand in their hospital.

Experiment 1 was thus designed to investigate how participants reacted to an agent that requested resources more often and shared resources less often during participants’ slow-tempo period and an agent that requested resources less often and shared resources more often as participants moved into a fast-tempo period. Experiment 2 was designed to investigate how an initial perturbation, or fast-tempo period, might differently affect participant and joint cooperation. Taken together, both experiments allowed for comparison of cooperative behavior prior to a fast tempo (Experiment 1) and following a fast tempo (Experiment 2) to assess team cooperation and ability to adapt to environment changes in a timely manner.

Agent cooperation levels: High-cooperation and low-cooperation agents

We operationalized agent cooperation as high cooperation or low cooperation depending on how much its behaviors emphasized joint outcome or individual outcome, respectively. Table 1 summarizes the resource-sharing and resource-requesting behaviors for the agents.

Table 1:

Agent Resource-Sharing and Resource-Requesting Behavior by Cooperation Level

Agent Cooperation Level	Resource-Sharing Behavior (Acceptance Rate)	Resource-Requesting Behavior
High cooperation	100% 75% 50%	1. Checks if it needs a resource 2. Checks if the participant has this resource available 3. Requests the resource
Low cooperation	50% 25% 0%	1. Checks if it needs a resource 2. Requests the resource

Note. The agents’ acceptance rates are keyed to the number of patients in the agent’s waiting room. From top to bottom: one to two patients, two to three patients, and three to four patients.

The high-cooperation agent was more likely to signal trusting, cooperative behavior when responding to resource requests. When a participant requested an available resource and the high-cooperation agent had two or fewer patients in its waiting room, the agent would accept 100% of the time, 75% of the time with three to four patients, and 50% of the time with five to six patients. The low-cooperation agent was less likely to signal trusting, cooperative behavior by accepting 50% less than the high-cooperation agent in each of the waiting room conditions, or 50%, 25%, and 0%, respectively. The low-cooperation agent thus expressed less individual risk and prioritized its own performance compared to the high-cooperation agent. Note that the low-cooperation agent is not exactly competitive, which could involve requesting all resources and refusing to return them; it is still cooperating in this context by sharing and requesting resources.

Cooperation was also expressed through requesting behavior. The high-cooperation agent would check if it needed a resource, then check if the participant had the resource before requesting. The low-cooperation agent only considered its own need for resources, which made it more likely to make requests insensitive to the participants’ needs. Agents were programmed to have an eight-second and two-second delay when assigning patient or staff and when collecting resources, respectively, to simulate the pace of a human player, established during pilot testing. This delay also avoided continuous interruption of the participant and allowed a window for participants to request unassigned agent resources.

Dependent variables

Because we wanted to know how an agent’s cooperative behaviors would influence a person’s cooperative behaviors, the dependent variables in this study were participants’ resource-requesting and resource-sharing behaviors. These were measured as the number of resources they requested and the number of agents’ requests they accepted. Agents’ behaviors were tallied independently. Individual performance was measured as the sum of patients a scheduler treated in a trial, and joint performance was the sum of patients that both participant and agent schedulers treated in a trial.

Procedure

Participants were introduced to the microworld scheduling task and interface. Participants were told they could request resources at any time from an automated agent, scheduling its own hospital in the background. To avoid guiding participants to use a particular strategy, participants were told to “treat as many patients as possible.” If participants asked if this included cooperating with the agent, the goal was restated and they were told it was up to them how to reach the goal. No information was given about the neighboring hospital agent’s behavior or potential changes in tempo.

Participants were then exposed to two, 2-minute practice trials at a medium tempo. At the end of each trial, including experimental trials, a bar graph displayed the patients they treated in the context of patients treated in both hospitals. The graph emphasized the goal of maximizing a joint score and acted as motivational performance feedback (Figure 2).

Figure 2.

Feedback provided to the participants showed a split bar that combines the sum of patients treated in each hospital, with a darker shade representing the participant’s contribution. The sum of patients missed was also reported as a split bar when applicable.

At the beginning of a trial, participants and their agent partner were each given three nurses, two doctors, and two surgeons to assign to one of six rooms in their hospital, along with one patient in their waiting room queue (Figure 3). Hospital staff were the only shareable resources. There were two types of patients that entered the queue with randomized 50% chance; Patients “A” required a doctor and a nurse, and patients “B” required a surgeon and nurse. Both types of patients therefore required a nurse, and to use all of one hospital’s rooms, a scheduler would need to obtain all nurses from both hospitals and some additional doctors or surgeons.

Figure 3.

A screenshot of the interface at the beginning of a trial. The bottom panel had three button options, Assign Patient to Room, Assign Resource to Room, or Request Resources. At the center were six vacant hospital rooms that also functioned as buttons. The left panel displayed the waiting rooms and available staff resources for each hospital. Each circle in the participant’s waiting room represented one patient. Two squares in the agent’s waiting room indicated the agent also had A and B type patients, but only one color status was displayed. In the top left corner, a timer counted down the trial.

To help with resource management, a side panel displayed available resources, patient waiting rooms, and a “score” for each hospital. Color codes were used to signal waiting room queue length status: green for one to two, yellow for three to four, and red for five to six patients. Six patients was a waiting room’s maximum capacity, and participants were told additional patients scheduled to arrive would be turned away. Participants could see the exact number of patients in their waiting room, with each circle representing one patient (Figure 3). However, only the color status of the agent’s waiting room was available to participants. This decision was inspired by Dabbish and Kraut (2004), which showed summarized displays can be more effective than detailed displays for communicating status information in a cooperative task. A summarized display also contributed to the uncertainty of interacting with agents, allowing a focus on the social-affective aspects of participants’ decision making rather than the information processing aspects.

Table 2 describes the main actions of the scheduling task from the participant’s point of view. Level 1 refers to the bottom control panel options in Figure 3, with subsequent navigation options as the following levels.

Table 2:

Participants’ Main Actions and Navigation Pathways

Level 1	Level 2	Level 3		Level 4
Actions and pathways related to scheduling
Assign patient to room	Patient A Patient B	(Select available room)		—
Assign resource to room	Doctor Nurse Surgeon	(Select available room)		—
Actions and pathways related to resource-exchanging
Request resource	Doctor Nurse Surgeon	“The other hospital has given you a (doctor/nurse/surgeon)”		—
Request resource	Doctor Nurse Surgeon	“Your request has been denied by the other hospital.”		—
“The neighboring hospital requests your assistance! They would like to have a (doctor/nurse/surgeon).”	Accept request	“Send (doctor/nurse/surgeon) to neighboring hospital?”	No	—
		“Send (doctor/nurse/surgeon) to neighboring hospital?”	Confirm	“A (doctor/nurse/surgeon) has been sent to the other hospital.”
	Deny request	—		—

Note. Level 1 indicates the first level of options, and the subsequent levels describe potential pathways for the options selected. Italicized texts refer to button options, and quotations refer to interface communications. Em dashes indicate levels not applicable to that pathway.

Interface highlighting and feedback aided participants in the microworld. For example, when assigning patients and resources, rooms currently treating patients would be greyed out. Once a patient and staff were assigned to a room, “patient treatment” began automatically, lasting 60 seconds. After treatment, participants needed to click a “Collect Resources” button that appeared on top of the respective room to free the room and staff for reassignment (action not included in Table 2). The system generally did not allow erroneous assignments; however, if participants attempted to take action using unavailable patients or resources, floating text would appear with the error and last several seconds before fading.

To request resources, participants could select “Request Resource” from the bottom control panel, then select nurse, doctor, or surgeon. Immediately following this, floating text informed participants if their request was accepted or denied (Table 2, Level 3). If accepted, the resource would transfer from the agent’s hospital to participants’ hospital and the staff numbers in both hospitals would update. Incoming requests to participants blocked the control panel, forcing an interruption (Figure 4). Resources that transferred would remain at the hospital unless requested and accepted back.

Figure 4.

A screenshot of the microworld environment interface that shows the bottom control panel obscured by a resource request.

Results and Discussion

In Experiment 1, where participants experienced a slow-tempo period followed by a fast-tempo period, we assess whether the high- and low-cooperation behaviors affected participants’ behaviors using ANOVA with three within-subjects variables (scheduler, cooperation level, tempo) and one between-subjects variable (cooperation order). Participants’ and agents’ mean requests and mean acceptances are discussed in the context of these resource-sharing variables, and mean performance (number of patients treated, individually and jointly) are considered in the context of agent cooperation. To assess the validity of averaging performance across trials, participants’ mean performance across trials was compared. No significant difference was found, suggesting minimal learning effects.

In Experiment 2, a different group of participants experienced a fast-tempo period followed by a slow-tempo period. We compare resource-sharing to Experiment 1 by how people’s cooperation differed following a perturbation, an initial fast-tempo period. We follow the same assessments as in Experiment 1 and supplement our explanations with additional visualizations and analyses. Analyses were conducted using the “stats” package in R (R Core Team, 2014); data visualizations and figures were created using “ggplot2” (Wickham, 2009). All figures are labeled from the participants’ perspective.

Experiment 1: Slow-Tempo Period Followed by a Fast-Tempo Period

Figure 5 shows agents’ and participants’ mean requests in the background narrow bars, overlaid by their mean acceptances, the foreground wider bars. Data are faceted by cooperation order (rows), cooperation level (columns), and tempo period (columns within cells). There were more requests in the fast-tempo periods (quadrants’ right columns, narrow bars) compared to the slow-tempo periods (quadrants’ left columns, narrow bars), demonstrating that agents generally behaved as designed and that participants understood the joint task and engaged in requesting resources accordingly, F(1, 34) = 91.89, p < .01. Participants’ increased requests moving from slow-tempo to fast-tempo shows the effect of tempo, F(1, 34) = 47.47, p < .01, and that people engaged agents as they needed more resources.

Figure 5.

Agents’ (bars on the left) and participants’ (bars on the right) mean requests (narrow bars) and mean acceptances (wide bars) with 95% CIs (confidence intervals).

Mean acceptances (Figure 5 wide bars) were not significantly different overall between tempo periods (quadrants’ left and right columns), F(1, 34) = 0.48, p = .5, due to the complementary tempo design and joint structure of the task. However, acceptances across cooperation levels did interact with tempo; participants’ mean acceptances were greater in their slow-tempo period compared to their fast-tempo period (dark wide bars, quadrants’ left columns compared to right columns, F(1, 34) = 96.58, p < .01. Furthermore, when participants were supposed to be accepting requests during their slow tempo (quadrants’ left wide bars), their mean acceptances were fewer than agents’ mean acceptances during its slow tempo (quadrants’ right wide bars) in three of the four comparisons. The same was true in participants’ fast tempo compared to agents’ fast tempo (participants’ slow tempo); participants’ mean acceptances were fewer than the agents’ in three of the four comparisons. In addition, mean acceptances were generally higher in the bottom row of Figure 5, particularly in the high-cooperation condition, which reflects an interaction between cooperation order and cooperation level, F(1, 34) = 4.31, p = .05.

Appropriate cooperation means participants should generally accept more requests during their slow-tempo period and make more requests during their fast-tempo period. The question is if agent cooperation differentially affected these behaviors. While agents’ mean requests were greater than participants’ mean requests across conditions, F(1, 34) = 26.46, p < .01, and agents’ mean acceptances were greater than participants’ mean acceptances across conditions, F(1, 34) = 24.37, p < .01, overall mean requests in the high-cooperation condition were fewer compared to the low-cooperation condition (Figure 5, comparing the total narrow bars in the left column with the total narrow bars in the right column), F(1, 34) = 35.62, p < .01. Across conditions, the low-cooperation agents’ mean requests were greater than participants’ mean requests, reflecting the significant interaction for scheduler and cooperation level, F(1, 34) = 13.16, p < .01. In sum, participants requested more from the low-cooperation agent than from the high-cooperation agent.

Mean acceptances, in contrast, were greater in the high-cooperation condition compared to in the low-cooperation condition (Figure 5, comparing wide bars in the left column and right column), F(1, 34) = 37.67, p < .01. Thus, participants contributed to more productive interactions that led to an exchange of resources in the high-cooperation condition compared to the low-cooperation condition. Participants’ mean acceptances were fewer than agents’ mean acceptances and even fewer in the low-cooperation condition compared to the high-cooperation condition, F(1, 34) = 21.33, p < .01. Despite interacting more often with the low-cooperation agent, indicated by the greater number of requests, exchanges were less productive—less likely to lead to resource exchange. This shows participants were still willing to engage the low-cooperation agent, making more requests like the low-cooperation agent rather than saving effort or adopting a more autonomous strategy. It also means participants as a group did not exploit the high-cooperation agent’s relative generosity by requesting more often and refusing to return resources when requested.

The dyad handled resource exchanges differently in high-cooperation versus low-cooperation conditions, particularly in their requests. In Figure 5 (left column, narrow bars), agents’ mean requests in participants’ slow tempo were greater than participants’, and this reverses when participants reached their fast tempo—their mean requests are greater than the agents’. In the low-cooperation condition (right column, narrow bars), agents’ mean requests are greater than participants’ in their slow tempo, although this does not increase participants’ mean acceptances (wide bars). This may have led to agents’ subsequent need for resources and the mutually increased requests in participants’ fast tempo. These observations are supported by the significant interaction for cooperation level and tempo, F(1, 34) = 21.14, p < .01.

Did more productive interactions lead to higher mean performance? Figure 6 shows mean individual performance, the sum of patients treated by each scheduler, and mean joint performance, the sum of participants’ and agents’ performance. Because of task interdependence—staff unused by participants were often used by the agents and vice versa—the difference between cooperation conditions is small. In particular, the low-cooperation agents’ higher mean performance compensates for the participants’ lower mean performance and reflects the low-cooperation agent’s more autonomous strategy favoring individual team member performance. However, the lower mean joint performance supports the observation that the low-cooperation agent’s behavior led to unproductive participant behaviors, with the high-cooperation condition producing higher joint performance (Figure 6), F(1, 34) = 7.37, p = .01. When participants transitioned from the low-cooperation to the high-cooperation agent, the difference between the high-cooperation and low-cooperation conditions appears larger (Figure 6, right column differences are larger compared to the left column differences); however, cooperation order and cooperation level did not produce a significant interaction. These findings for joint performance suggest that an inconsiderate interrupter that was less generous with resources—the low-cooperation agent’s persona—could lead to lower joint performance in a task involving people.

Figure 6.

Mean patients treated with 95% CIs (confidence intervals) show individual performance and joint performance, with joint performance lower in the low-cooperation condition compared to the high-cooperation condition. Joint performance is plotted as the average of participants’ scores and agents’ scores.

While appropriate cooperation was defined as generally accepting more requests during slow-tempo periods and making more requests during fast-tempo periods, results show fewer requests occurred with the high-cooperation agent compared to the low-cooperation agent. However, both agent and participant behaviors contributed to the higher joint performance in the high-cooperation condition. Overall, these results show the value of considering reactive (responding to requests) and proactive (making requests) behaviors as part of evaluating appropriate cooperation in a joint task and that social exchange factors may be relevant for human-agent interaction—agent cooperation can affect human cooperation.

Experiment 2: Fast-Tempo Period Followed by a Slow-Tempo Period

Requesting behavior in Experiment 2 was similar to Experiment 1; agents’ mean requests were greater than participants’ mean requests, F(1, 34) = 124.87, p < .01, and mean requests were greater in the low-cooperation condition compared to the high-cooperation condition, F(1, 34) = 93.72, p < .01. In addition, mean acceptances were greater in the high-cooperation condition compared to the low-cooperation condition, F(1, 34) = 33.27, p < .01, and the two schedulers’ mean requests were greater in their respective fast-tempo periods, demonstrating the agents behaved as designed and that participants were engaged in the microworld task, F(1, 34) = 169.43, p < .01. Furthermore, participants and agents differed in their requests, particularly depending on cooperation levels; the low-cooperation agents’ mean requests were greater than participants’ mean requests across conditions, F(1, 34) = 89.85, p < .01.

Contrary to expectations, mean requests in the slow-tempo period were greater than in the fast-tempo period, F(1, 34) = 223.70, p < .01. This highlights the first difference between Experiment 2 and Experiment 1. Figure 7 (quadrants’ right columns, narrow bars) partially explains these findings; agents’ mean requests during participants’ slow tempo were greater relative to the other conditions and relative to Experiment 1, with an especially large portion of requests made by the low-cooperation agent, F(1, 34) = 41.92, p < .01. This is also reflected in the significant interaction term for scheduler and tempo, where the agents’ mean requests were greater than participants’ mean requests during participants’ slow-tempo period, F(1, 34) = 222.18, p < .01; the significant interaction term for cooperation level and tempo, where mean requests were greatest overall in the low-cooperation, slow-tempo period, F(1, 34) = 41.92, p < .01; and the significant interaction term for scheduler, cooperation level, and tempo, F(1, 34) = 20.48, p < .01.

Figure 7.

Agents’ (bars on the left) and participants’ (bars on the right) mean requests (narrow bars) and mean acceptances (wide bars) with 95% CIs (confidence intervals).

Because agent actions were partially contingent on participants’ actions, a complementary explanation for the higher number of requests in the slow-tempo period is that participants delayed and did not request resources early in their fast-tempo period. This delay caused a backlog of patients in their hospitals, which reverberated through their requesting and accepting behaviors in the second half of the trial. The greater number of agent requests are thus likely due to participants retaining resources leading into the agents’ fast-tempo period (and participant’s slow-tempo period), resulting in agents running out of resources more quickly compared to Experiment 1. This failure of participants to quickly adapt in the early fast-tempo might be described as a failure to maintain their margin of maneuver (Stephens et al., 2011), which led to subsequent system breakdowns and delays in the joint task. A visualization of the request sequence shows that participants did not make requests until the end of their fast-tempo period (Figure 8).

Figure 8.

Aggregate requests of all Experiment 2 participants (outlined bars) and agents (un-outlined bars) plotted sequentially show that peak participant requests occur toward the end of their fast-tempo period.

The effects of participants’ delayed requests also seemed to affect participants’ and agents’ acceptances. Contrary to Experiment 1, participants’ mean acceptances were greater than agents’ mean acceptances in Experiment 2, F(1, 34) = 15.97, p < .01. It seems the low-cooperation agent was more badly in need of resources during participants’ slow-tempo period in Experiment 2 compared to Experiment 1 due to participants’ delayed resource requests and subsequent delayed use of resources at the start of their slow-tempo period, and thus agents accepted less often in Experiment 2.

Without the benefit—or an example—of a requesting agent early in the trial, participants seemed focused on scheduling within their own hospital rather than requesting resources from the agent. This is especially true when comparing the timing of participants’ requests in Experiment 2, which peak at the end of the fast-tempo period in Figure 8, and the timing of participants’ requests in Experiment 1, which peaked earlier, halfway through the fast-tempo period (not pictured). Thus, participants’ mean number of requests during an early fast tempo may not have been sufficient. This would lead to requests increasing toward the end of the fast-tempo period to address buildup in the patient queue. Participants’ tendency to adopt an autonomous strategy for assigning patients and staff to available rooms meant they neglected the opportunity to gain resources to serve patients.

Despite this effect in Experiment 2, agent cooperation had a positive effect on joint performance, supporting our initial hypothesis. The dyad in the high-cooperation condition had greater resource exchange between tempo periods compared to the dyad in the low-cooperation condition. This is reflected in the significant interaction term for cooperation level and tempo, F(1, 34) = 4.46, p < .05; the significant interaction term for scheduler, cooperation level, and tempo, F(1, 34) = 17.02, p < .01; and in Figure 7, where the differences between quadrants’ left and right wide bars are greater in the left column (“HighCoop”) compared to the right column (“LowCoop”). It is also noteworthy that compared to Experiment 1, and particularly in the high-cooperation condition, the fast-tempo period in Experiment 2 did not reduce participants’ tendency to share resources despite the agent demanding much less often.

It is tempting to compare performance between the two experiments given their parallel design. However, plotting when agents released staff against when participants reached maximum queue length shows that even with earlier requests in Experiment 1, participants were still unable to treat the backlog of patients in their waiting room before the trial ended. Thus, the only conclusion we can make about joint performance in Experiment 2 is that it supported the findings from Experiment 1—cooperation affects joint performance. Joint performance was lower with the low-cooperation agent, F(1, 34) = 12.24, p < .01, and even lower when the participants initially interacted with the low-cooperation agent, F(1, 34) = 5.63, p = .02 (Figure 9).

Figure 9.

To further investigate the interdependence of the agent and participant, a linear regression was calculated to predict joint performance based on participant performance. For Experiment 1, participants’ performance accounted for a much smaller proportion of the variance in the joint performance, F(1, 142) = 18.09, p < .01, R² = 0.11, compared to Experiment 2, F(1, 142) = 273.00, p < .01, R² = 0.66, supporting the idea that participants used a more autonomous strategy in Experiment 2.

General Discussion

Both experiments confirm that cooperation with automation is an important complement to reliance on automation. People and automated agents were intentionally placed in parallel roles to focus on the need to coordinate in a dynamic task environment through cooperation. While sharing resources potentially produces conflicting goals, in this study, demand was engineered to minimize conflict. From the researcher’s perspective, there was a relatively clear coordination solution—schedulers in the slow tempo would be able to share resources with their counterparts experiencing a fast-tempo period. This allowed us to test how participants responded to agents expressing different levels of cooperation and to motivate future studies on the social or affective influences of cooperation rather than only the information processing components of the task. Overall, both experiments showed agent cooperation affected participant cooperation, supporting our general hypothesis that agent cooperation could be an important construct in human-agent coordination beyond automation reliability and its effects on reliance and compliance.

Although the goal was to “treat as many patients as possible,” participants could take a defensive approach (Stephens et al., 2011), requesting resources without returning them, to maximize patients treated in their own hospital. However, such defensive behavior would undermine cooperation and reciprocity (Axelrod & Hamilton, 1981) and in the microworld lead to fewer patients treated. Though resource-sharing decisions may have been governed by the instrumental value of resources (Molm et al., 2007), the environment demand was not made explicit to participants. Therefore, the act of sharing resources demonstrates their trust in the agent. Accepting requests came at the cost of losing spare resources, reducing their margin of maneuver, and risking that the agent would not reciprocate and return resources when needed. In both experiments, less cooperative agents induced less cooperative behavior in the participants and more cooperative agents induced more cooperative behavior in the participants.

Our hypothesis on cooperative reciprocity for Experiment 1 was thus supported. The cooperative nature of agents affected both dimensions of participants’ cooperative behaviors—resource sharing and resource requesting. Participants engaged in more productive exchanges—successful requests—with the high-cooperation agent than with the low-cooperation agent despite the low-cooperation agent requesting resources from participants more often. This suggests people reciprocated the cooperative behavior of the agent rather than merely mirroring the number of requests. In addition, participants who previously experienced a low-cooperation agent and were working with the high-cooperation agent demonstrated even more productive exchanges than participants who experienced the high-cooperation agent first. It may be that participants were better at timing their requests in later trials to coincide with when the agent had available resources, but there was no evidence of learning effects. It may be that the high-cooperation agent surpassed low expectations formed with the low-cooperation agent, indicating the interaction history influenced decisions to cooperate. This latter explanation and the higher joint score with the high-cooperation agent support the idea that to maintain resilience, systems must continually invest in a renewal of shared goals and a willingness to accommodate (Woods, 2004).

The hypothesis for Experiment 2—that if people’s initial exposure to an agent is in a highly demanding situation, then the effects of agent cooperation level will be more prominent—was partially supported by participants’ mean acceptances, particularly in the fast-tempo, high-cooperation condition. However, the initial fast-tempo period may have overwhelmed participants and led to more autonomous requesting strategies. Rather than anticipating demand and making sufficient early requests, participants’ delay led to a backlog that undermined their ability to accept requests as agents transitioned into fast tempo (especially the low-cooperation agent). However, when prompted, people still responded in a cooperative way that contrasted their relatively autonomous requesting strategy. This suggests cooperation may have two dimensions—proactive and reactive actions (Hollnagel, 2012). Given that participants started with a medium-tempo margin in each trial and were free to request resources, it remains a question why participants did not anticipate demand in subsequent trials. Certainly expertise with such demanding situations in actual work environments might ameliorate the tendency toward an autonomous, reactive response. However, our results are consistent with the general error tendency of cognitive tunneling when people confront unexpected high-demand situations (Woods et al., 1994). In summary, this autonomous behavior was limited to reducing participants’ requests for resources and did not reduce their tendency to accept requests.

Previous work in human-automation cooperation has focused on performance, with adjustable autonomy typically referring to better coordination (Zieba et al., 2010) or better management of functional dependencies. That speed of assigning resources was an important factor for this particular task might make this study a strange example of a joint human-agent task; automation is known to be much better at this than people (Fitts et al., 1951). However, it is not always the case that automation should be used when the costs of a disengaged human counterpart would be greater than the efficiencies of implemented automation (Kirlik, 1993). Our study focuses on the social processes of coordination in resilience, in circumstances where reciprocity and goal tradeoffs feature more prominently than reliability and mode management. Doing so departs from the idea that teams should be formed purely based on complementary abilities rather than how those team members interact. It also challenges the tempting idea that if functions can be automated, then they should be automated.

Establishing trust between people and autonomous agents may be one of the most daunting problems for the success of human-automation teams (Groom & Nass, 2007), and is important for cooperation and maintaining margins of maneuver. To work to this ideal, we investigated a shared-resource activity between people and automation, where social processes could potentially limit joint performance. As Groom and Nass (2007) observe, future research intent on developing human-robot teams must go beyond technical performance to address the social and organizational qualities that make a successful teammate.

Limitations and Future Directions

Microworlds have the advantage of mimicking interactive situations observed in field studies but with a greater degree of experimental control (Gonzalez, Vanyukov, & Martin, 2005). This experimental control is particularly valuable when investigating the dynamics of interdependent behavior. Microworlds make it possible to manipulate the behavior and interdependencies of dyads, the simplest unit of agent networks in complex sociotechnical systems, but such interdependencies can also lead to challenges in the analysis.

In this study, the high-cooperation agent’s requesting behavior could be affected by participants who were slower or chose not to use their resources because the high-cooperation agent was designed to check if participants’ resources were unused prior to requesting them. The low-cooperation agent’s requesting behavior could be affected by how quickly it ran out of resources it needed, which could be influenced by resources it did or did not have due to the participant’s actions. Both high-cooperation and low-cooperation agents’ acceptances depended in part on when participants made requests. This led to challenges in interpreting the results. However, this interdependence approach echoes the challenge of designing strategies to combat conflict in such joint tasks, where a person’s decisions depend on the agent’s decisions, which depend on the person’s decisions, and so on (Schelling, 1960). Most studies seek to avoid interdependencies, but a systems perspective that considers these interdependencies is becoming more important with the changing role of increasingly autonomous automation.

In practice, it may be difficult to draw conclusions about the impact of cooperative behavior on performance outcomes; it is possible that as cooperative activity goes up, productivity decreases or remains unchanged or other important measures are affected, such as worker turnover and quality of work life. Cooperation may create value where there is none, and the quality of the output is not necessarily measured in terms of responding to demands more quickly (Wicks, Berman, & Jones, 1999), as represented in this microworld. In other words, increased cooperation might not increase performance in routine situations even if it enhances resilience in high-demand situations. As a consequence, organizations might neglect the importance of cooperative automation.

It is challenging but critical to identify performance measures that emphasize priorities consistent with cooperative control. In practice, these priorities can manifest as micro-decisions to cooperate that extend to macro outcomes for a system of interconnected agents. To avoid the tendency of reactive responses, future microworld studies could explore different social exchange structures, moving from negotiated exchange, where exchange decisions are made jointly, to reciprocal exchange, where decisions are more unilateral, like in altruism (Molm et al., 2000). Such a design could refocus human-agent interactions from interrupting to request resources, a structure with more reactive affordances, to interrupting with needed resources, a structure with more proactive affordances. A key question would be to understand how to foster proactive behavior that helps systems maintain margins of maneuver and to understand what causes people to ignore or avoid early opportunities that could enhance cooperation.

Conclusion

This study complements the supervisory control perspective by reframing human-automation interaction in terms of cooperative control for human-agent systems. Cooperation is central to resilience because it enables networks of interdependent agents to pool resources and accommodate a greater range of surprises. People can become less cooperative when interacting with less trusting and inconsiderate agents, more cooperative when interacting with more trusting and considerate agents, and cooperative strategies might be expressed in both proactive and reactive behaviors. These behaviors are qualitatively different than those observed in investigations of supervisory control automation. This study’s approach thus departs from the typical supervisory control approach to human-automation interaction and acts as a starting point for future studies to explore how cooperation in human-agent interactions can enhance system resilience.

Key Points

Rather than supervisory control situations where reliance and compliance are the focus, increased automation capability may produce cooperative control situations where reciprocity and proactive and reactive cooperation are the focus.

People adopt cooperative, defensive, or autonomous strategies to maintain their margin of maneuver in dynamic exchange scenarios with an automated agent, and an automation agent’s strategies influence those adopted by people.

Analyzing human-agent interaction in a dynamic system presents a challenge but is essential in revealing patterns of interactive behavior underlying decisions to cooperate.

Footnotes

Acknowledgements

This work was supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE-1256259. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of the National Science Foundation. The authors would like to thank Anousoune K. Bounket and Kim Le for their work on the microworld, and Ja Young Lee, David Woods, Vindhya Venkatraman, and three anonymous reviewers whose comments improved the manuscript.

Erin K. Chiou is a PhD student and National Science Foundation Graduate Research Fellow in the Department of Industrial and Systems Engineering at the University of Wisconsin-Madison. She received her master’s of science in industrial and systems engineering from the University of Wisconsin-Madison in 2013.

John D. Lee is Emerson Electric Quality and Productivity Professor in the Department of Industrial and Systems Engineering at the University of Wisconsin-Madison. He received his PhD in mechanical engineering from the University of Illinois, Urbana-Champaign in 1992.

References

Allen

Guinn

Horvitz

(1999). Mixed-initiative interaction. IEEE Intelligent Systems and Their Applications, 14(5), 14–23. doi:10.1109/5254.796083

Axelrod

Hamilton

(1981). The evolution of cooperation. Science, 211(4489), 1390–1396.

Bray

Anumandla

Thibeault

(2012). Real-time human–robot interaction underlying neurorobotic trust and intent recognition. Neural Networks, 32, 130–137.

Casper

Murphy

R. R.

(2003). Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade Center. IEEE Transactions on Systems, Man, and Cybernetics. Part B: Cybernetics, 33(3), 367–385. doi:10.1109/TSMCB.2003.811794

Cassell

Bickmore

(2000). External manifestations of trustworthiness in the interface. Communications of the ACM, 43(12), 50–56.

Clark

(1996). Using language. Cambridge, UK: Cambridge University Press.

Dabbish

Kraut

(2004). Controlling interruptions: Awareness displays and social motivation for coordination. In Proceedings of CSCW’14 (pp. 182–191). Chicago, IL: ACM Press.

De Dreu

Greer

Handgraaf

M. J. J.

Shalvi

Van Kleef

Baas

Ten Velden

F. S.

Van Dijk

Feith

S. W. W.

(2010). The neuropeptide oxytocin regulates parochial altruism in intergroup conflict among humans. Science, 328(5984), 1408–1411. doi:10.1126/science.1189047

Delton

Krasnow

Cosmides

Tooby

(2011). Evolution of direct reciprocity under uncertainty can explain human generosity in one-shot encounters. Proceedings of the National Academy of Sciences of the United States of America, 108(32), 13335–13340. doi:10.1073/pnas.1102131108

10.

Desteno

Breazeal

Frank

Pizarro

Baumann

Dickens

Lee

(2012). Detecting the trustworthiness of novel partners in economic exchange. Psychological Science, 23(12), 1549–56. doi:10.1177/0956797612448793

11.

Dugatkin

Mesterton-Gibbons

Houston

(1992). Beyond the prisoner’s dilemma: Toward models to discriminate among mechanisms of cooperation in nature. Trends in Ecology & Evolution, 7(6), 202–205.

12.

Fink

Weyer

(2014). Interaction of human actors and non-human agents: A sociological simulation model of hybrid systems. Science, Technology & Innovation Studies, 10(1), 47–64.

13.

Fitts

Viteles

Barr

Brimhall

Finch

Gardner

. . . Stevens

(1951). Human engineering for an effective air-navigation and traffic-control system. Washington, DC: National Research Council.

14.

Fong

Nourbakhsh

Kunz

Flückiger

Schreiner

Ambrose

Burridge

Simmons

Hiatt

L. M.

Schultz

Trafton

J. G.

Bugajska

Scholtz

(2005). The peer-to-peer human-robot interaction project. In Proceedings from SPACE Conferences and Exposition: Space 2005. Reston, VA: American Institute of Aeronautics and Astronautics. doi:10.2514/6.2005-6750

15.

Gao

Lee

Zhang

(2006). A dynamic model of interaction between reliance on automation and cooperation in multi-operator multi-automation situations. International Journal of Industrial Ergonomics, 36(5), 511–526. doi:10.1016/j.ergon.2006.01.013

16.

Gonzalez

Vanyukov

Martin

(2005). The use of microworlds to study dynamic decision making. Computers in Human Behavior, 21(2), 273–286. doi:10.1016/j.chb.2004.02.014

17.

Groom

Nass

(2007). Can robots be teammates? Interaction Studies, 8(3), 285–301. doi:10.1075/gest.8.3.02str

18.

Hoc

J.-M.

(2001). Towards a cognitive approach to human-machine cooperation in dynamic situations. International Journal of Human-Computer Studies, 54, 509–540. doi:10.1006/ijhc.2000.0454

19.

Hollnagel

(2012, May). Proactive approaches to safety management. The Health Foundation Thought Paper, 1–11.

20.

Hollnagel

Woods

Leveson

(Eds.). (2006). Resilience engineering: Concepts and precepts. Hampshire, UK: Ashgate Publishing Limited.

21.

Kirlik

(1993). Modeling strategic behavior in human-automation interaction: Why an “aid” can (and should) go unused. Human Factors, 35(2), 221–242. doi:10.1177/001872089303500203

22.

Lasota

Shah

(2015). Analyzing the effects of human-aware motion planning on close-proximity human-robot collaboration. Human Factors: The Journal of the Human Factors and Ergonomics Society, 57(1), 21–33. doi:10.1177/0018720814565188

23.

Lee

See

(2004). Trust in automation: Designing for appropriate reliance. Human Factors: The Journal of the Human Factors and Ergonomics Society, 46(1), 50–80.

24.

Malone

Crowston

(1994). The interdisciplinary study of coordination. ACM Computing Surveys, 26(1), 87–119. doi:10.1145/174666.174668

25.

Molm

Schaefer

Collett

(2007). The value of reciprocity. Social Psychology Quarterly, 70(2), 199–217.

26.

Molm

Takahashi

Peterson

(2000). Risk and trust in social exchange: An experimental test of a classical proposition. American Journal of Sociology, 105(5), 1396–1427.

27.

Ostrom

(1999). Revisiting the commons: Local lessons, global challenges. Science, 284(5412), 278–282. doi:10.1126/science.284.5412.278

28.

Ostrom

(2000). Collective action and the evolution of social norms. The Journal of Economic Perspectives, 14(3), 137–158.

29.

Parasuraman

Miller

(2004). Trust and etiquette in high-criticality automated systems. Communications of the ACM, 47(4), 51–55.

30.

Parasuraman

Riley

(1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors: The Journal of the Human Factors and Ergonomics Society, 39(2), 230–253. doi:10.1518/001872097778543886

31.

R Core Team. (2014). A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

32.

Riedl

Mohr

Kenning

Davis

Heerkeren

(2011). Trusting humans and avatars: Behavioral and neural evidence. Presented at the Thirty-Second International Conference on Information Systems, Shanghai, China.

33.

Robinette

Wagner

Howard

(2013). Building and maintaining trust between humans and guidance robots in an emergency. In AAAI Spring Symposium, Stanford University (pp. 78–83). Palo Alto, CA: Stanford University.

34.

Rochlin

La Porte

Roberts

(1987). The self-designing high-reliability organization: Aircraft carrier flight operations at sea. Naval War College Review, 40(4), 76–90.

35.

Schelling

(1960). The strategy of conflict. Cambridge, MA: Harvard University Press.

36.

Stephens

Woods

Branlat

Wears

(2011). Colliding dilemmas: Interactions of locally adaptive strategies in a hospital setting. Presented at the Fourth Symposium on Resilience Engineering, Sophia Antipolis, France.

37.

Takayama

Groom

Nass

(2009). I’m sorry, Dave: I’m afraid I won’t do that: Social aspects of human-agent conflict. In CHI 2009-Studying Intelligent Systems (pp. 2099–2107). Boston, MA: ACM.

38.

Thibaut

Kelley

(1959). The social psychology of groups. New York, NY: John Wiley & Sons, Inc.

39.

Wagner

Arkin

(2011). Recognizing situations that demand trust. In Proceedings-IEEE International Workshop on Robot and Human Interactive Communication (pp. 7–14). Atlanta, GA: IEEE.

40.

Wickham

(2009). Ggplot2: Elegant graphics for data analysis. New York, NY: Springer-Verlag.

41.

Wicks

Berman

Jones

(1999). The structure of optimal trust: Moral and strategic implications. Academy of Management Review, 24(1), 99–116.

42.

Williams

(2010). Dyads can be groups (and often are). Small Group Research, 41(2), 268–274. doi:10.1177/1046496409358619

43.

Woods

(2004). Conflicts between learning and accountability in patient safety. DePaul Law Review, 54, 485–502.

44.

Woods

(2015). Four concepts for resilience and the implications for the future of resilience engineering. Reliability Engineering & System Safety, 141, 5–9. doi:10.1016/j.ress.2015.03.018

45.

Woods

Branlat

(2010). Basic patterns in how adaptive systems fail. In Hollnagel

Pariès

Woods

D. D.

Wreathall

(Eds.), Resilience engineering in practice: A guidebook (pp. 127–144). Farnham, UK: Ashgate.

46.

Woods

Hollnagel

(2005). Joint cognitive systems: Foundations of cognitive systems engineering. Boca Raton, FL: CRC Press.

47.

Woods

Johannesen

Cook

Sarter

(1994). Behind human error: Cognitive systems, computers and hindsight. Dayton, OH: Wright-Patterson Air Force Base.

48.

Woods

Patterson

(2001). How unexpected events produce an escalation of cognitive and coordinative demands. In Hancock

Desmond

(Eds.), Stress workload and fatigue (pp. 290–304). Hillsdale, NJ: Lawrence Erlbaum Associates.

49.

Woods

Tittle

Feil

Roesler

(2004). Envisioning human-robot coordination in future operations. IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 34(2), 210–218. doi:10.1109/TSMCC.2004.826272

50.

Zieba

Polet

Vanderhaegen

Debernard

(2009). Resilience of a human-robot system using adjustable autonomy and human-robot collaborative control. International Journal of Adaptive and Innovative Systems, 1(1), 13–29. doi:10.1504/IJAIS.2009.022000

51.

Zieba

Polet

Vanderhaegen

Debernard

(2010). Principles of adjustable autonomy: A framework for resilient human-machine cooperation. Cognition, Technology and Work, 12(3), 193–203. doi:10.1007/s10111-009-0134-7