Abstract
Objective:
This study uses a dyadic approach to understand human-agent cooperation and system resilience.
Background:
Increasingly capable technology fundamentally changes human-machine relationships. Rather than reliance on or compliance with more or less reliable automation, we investigate interaction strategies with more or less cooperative agents.
Method:
A joint-task microworld scenario was developed to explore the effects of agent cooperation on participant cooperation and system resilience. To assess the effects of agent cooperation on participant cooperation, 36 people coordinated with a more or less cooperative agent by requesting resources and responding to requests for resources in a dynamic task environment. Another 36 people were recruited to assess effects following a perturbation in their own hospital.
Results:
Experiment 1 shows people reciprocated the cooperative behaviors of the agents; a low-cooperation agent led to less effective interactions and less resource sharing, whereas a high-cooperation agent led to more effective interactions and greater resource sharing. Experiment 2 shows that an initial fast-tempo perturbation undermined proactive cooperation—people tended to not request resources. However, the initial fast tempo had little effect on reactive cooperation—people tended to accept resource requests according to cooperation level.
Conclusion:
This study complements the supervisory control perspective of human-automation interaction by considering interdependence and cooperation rather than the more common focus on reliability and reliance.
Application:
The cooperativeness of automated agents can influence the cooperativeness of human agents. Design and evaluation for resilience in teams involving increasingly autonomous agents should consider the cooperative behaviors of these agents.
Keywords
Automation is becoming increasingly autonomous. From self-driving vehicles to sophisticated decision support systems, computational advances have led to a plethora of machines capable of automating whole functions that previously required people. Emerging new human-automation relationships pressure existing frameworks for research, design, and evaluation of these joint systems (Woods & Hollnagel, 2005). People and machines increasingly enter into coordinative and cooperative relationships, where successful interactions demand teamwork. Resilience engineering approaches the design of such systems as consisting of interactive agents, collaborating to achieve shared goals in increasingly dynamic and safety-sensitive environments (Hollnagel, Woods, & Leveson, 2006). However, few controlled studies have investigated how resilience depends on automation design, which is the focus of this paper.
Background
For supervisory control, joint performance depends on automation reliability and people’s ability to assess automation reliability, so they can appropriately rely on or comply with automation (Parasuraman & Riley, 1997). However, with more advanced automation, reliability may be less critical than the resilience of the system (Zieba, Polet, Vanderhaegen, & Debernard, 2010). Resilience refers to the ability to manage sustained adaptability of a layered network system (Woods, 2015). This differs from resilience as rebound, to restore a system to previous conditions before disruption, or resilience as robustness, having an expanded set of models to respond to disturbances (Woods, 2015). Rather, resilience as sustained adaptability considers how interacting agents cooperate, drawing on shared resources to accommodate surprises. Designing for resilience suggests collaborative or cooperative control might be an important complement to supervisory control.
For human-automation interaction, collaborative control allows both human and automated agents to combine competencies and negotiate conflicts in dynamic settings (Zieba et al., 2010). Collaborative control supports resilience through adaptation to novel situations, accommodation of bugs in the system itself, and recovery from errors in following procedures (Woods, Johannesen, Cook, & Sarter, 1994). Furthermore, resilience calls for the continual renewal of shared goals, reciprocity, and the willingness to accommodate others as unexpected demands require shifts of individual and shared priorities (Woods, 2004). System performance is less about how well predetermined priorities are reliably executed and more about how well goals and actions are adapted for the greater good of the system. Thus, rather than collaborative control, which often refers to managing functional dependencies by adapting levels of autonomy, cooperative control may be more appropriate for human-automation scenarios where cooperation precedes coordination.
Resilience in uncertain environments with complex interdependencies demands alternatives to the more explicit supervisor-subordinate procedures. These alternatives include informal interactions that promote coordination and cooperation (Rochlin, La Porte, & Roberts, 1987). Coordination is broadly defined as dependency management (Malone & Crowston, 1994) and concerns task scheduling and assignment, whereas cooperation is identified in actions that benefit a group but impose individual costs (adapted from Dugatkin, Mesterton-Gibbons, & Houston, 1992). The need to compromise individual goals for a shared goal can arise from what Clark (1996) calls private and public goals, Ostrom (2000) calls self- and collective interest, Hoc (2001) calls ego-centric and collaborative interests, and Woods (2004) calls local and global goals. These constructs share the idea that cooperation emerges through reconciling competing goals with the help of social processes. When interdependent activity involves shared resources and conflicting goals, cooperation is essential for coordination.
The importance of cooperation in resilience is highlighted in the margin of maneuver—a cache of actions and resources that allows the system to function despite unexpected perturbations in the work environment (Woods & Branlat, 2010). Systems without adequate margins become brittle, unable to withstand unexpected demands. Based on studies of emergency department interactions with other hospital units, Stephens, Woods, Branlat, and Wears (2011) identify three classes of strategies to create or maintain a margin of maneuver: defensive strategies, autonomous strategies, and cooperative strategies. Defensive strategies increase the margin of a unit by restricting another unit. Autonomous strategies involve local focus, such as reducing interactions with other units because the benefits of sharing resources are not believed to be worth the costs. Cooperative strategies involve the effort of two or more units that through coordinated and collective action recognize or create common-pool resources from which both units can draw. Such actions help avoid the tragedy of the commons when resources are limited (Ostrom, 1999) and may enhance system resilience by giving the overall network a larger pool of resources to draw on for greater margin of maneuver.
In light of these properties of resilience in volatile environments and increasingly capable automation, understanding what contributes to human-agent cooperation seems crucial. When problems cascade, demands for cognitive activity increase, as do the demands for coordination across agents (Woods & Patterson, 2001). These increased demands may accentuate bonding or fracturing of teams depending on the nature of their interdependence and the information available regarding the trustworthiness of agents (Gao, Lee, & Zhang, 2006). Cooperative strategies require assuming initial generosity (Axelrod & Hamilton, 1981), trust that the other party will reciprocate and not exploit shared resources, and an understanding that the benefits of cooperation will outweigh the costs. Without such trust between agents, an agent may resort to defensive strategies to preserve individual gain at the cost of another (De Dreu et al., 2010).
A substitute for formal controls, trust guides human behavior toward automation when a complete understanding is difficult or impractical (Lee & See, 2004). Such uncertainty is increasingly common due to technological advances and the environments in which automation are deployed. In high-stress environments, human agents may be even less likely to proactively interact with automation, particularly when experiencing physical and cognitive fatigue (Casper & Murphy, 2003). Although trust in automation has been said to be asymmetrical, in that automation does not trust back (Lee & See, 2004), recent work with more autonomous automation (e.g., agents or robots) suggests a more symmetrical view, and social exchange situations may apply to the concerns raised by resilience engineering (Bray, Anumandla, & Thibeault, 2012; Fink & Weyer, 2014; Wagner & Arkin, 2011). In cooperative exchange, people often choose partners depending on the instrumental value of the exchange, even though people’s trust, affective regard, and sense of solidarity with exchange partners are strongly influenced by the symbolic act of reciprocity (Molm, Schaefer, & Collett, 2007). Reciprocity and trust influence cooperative behavior, especially when novel or complex situations involve risk or uncertainty (Delton, Krasnow, Cosmides, & Tooby, 2011; Molm, Takahashi, & Peterson, 2000; Riedl, Mohr, Kenning, Davis, & Heerkeren, 2011).
Existing approaches to human-automation cooperation and resilience consider how automation can facilitate collaboration with people in dynamic environments (Allen, Guinn, & Horvitz, 1999; Fong et al., 2005; Wagner & Arkin, 2011; Woods, Tittle, Feil, & Roesler, 2004; Zieba, Polet, Vanderhaegen, & Debernard, 2009). Other studies address the effects of socially sensitive automation, such as developing trust through conversational cues, appearance, and behavior (Cassell & Bickmore, 2000; Desteno et al., 2012; Robinette, Wagner, & Howard, 2013), or engaging in good or poor etiquette (Parasuraman & Miller, 2004; Takayama, Groom, & Nass, 2009). Another study explored the effects of adaptive automation on human-robot team fluency and subjective ratings of satisfaction, safety, and comfort (Lasota & Shah, 2015). However, these studies focused on reliability and reliance or were conducted in static task environments. At the time of this writing, we were unable to find studies that considered cooperation with automation in a dynamic, shared resource task environment.
As a first step in understanding cooperation in networked and layered sociotechnical systems, we consider the human-agent dyad (Thibaut & Kelley, 1959; Williams, 2010). Our goal is to demonstrate that for resilient systems, an important complement to reliance on automation is cooperation. To achieve this goal, we developed a microworld that required human and automated agent dyads to cooperate on a dynamic, shared resource task. We present two experiments; Experiment 1 tests the hypothesis that an agent’s cooperation will affect people’s cooperation, leading them to reciprocate with similar resource exchange behaviors. Experiment 2 tests the hypothesis that when people’s initial exposure to an agent is during an unexpected, highlydemanding situation, then the effects of agent cooperation will be particularly prominent.
Method
A microworld hospital scheduling scenario was developed in Java and XML with the Android SDK, to assess cooperative behavior in a joint human-agent task with shared resources. A 15” laptop computer running Genymotion and a standard computer mouse were used. Participants acted as hospital schedulers, whose task involved assigning patients and staff to hospital rooms and coordinating shared staff resources with a neighboring hospital managed by an automated agent. Interactions with the agent were bilateral and limited to requesting resources and responding to requests for resources. The study tested different levels of agent cooperation on participants’ behavior and joint performance. In each of their four experimental trials, 36 participants in Experiment 1 experienced a slow-tempo period followed by a fast-tempo period (the slow-to-fast group), and 36 participants in Experiment 2 experienced a fast-tempo period followed by a slow-tempo period (the fast-to-slow group). Testing these two tempo sequences allowed us to assess the effects of agent cooperation level in the context of recovery following a perturbation and across variation in work environments.
Experimental Design
Study participants
Participants were recruited near a Midwestern university through flyers and online postings and received $10 at the end of the hour-long study. Self-reports showed age ranged from 18 to 56 with a mean age of 22 in Experiment 1 and a mean age of 24 in Experiment 2; gender was roughly 50% female and 50% male; 97% of participants used the computer daily; 47% in Experiment 1 and 64% in Experiment 2 reported using the computer for playing games.
Independent variables
Each experiment tested “agent cooperation level” as a within-subjects variable, meaning each participant was exposed to a high-cooperation and a low-cooperation agent (Figure 1). Agent cooperation levels were operationalized in the agents’ resource-requesting and resource-sharing behaviors. Exposure to the agents was counterbalanced, so participants worked with one agent in their first two trials and the other agent in their final two trials. Within each trial, there was one slow-tempo period and one fast-tempo period, and tempo was tested as a within-subject variable. Scheduler was a within-subject variable to distinguish between the participant and the automated agent. Therefore, each study was a mixed within and between 2 × 2 × 2 × 2 design that tested agent cooperation level (high or low cooperation), tempo (fast-tempo period or slow-tempo period), and scheduler (person or agent) as within-subject variables and cooperation order as a between-subject variable (high cooperation followed by low cooperation, or low cooperation followed by high cooperation).
In Experiment 1, 18 participants experienced the high-cooperation agent then the low-cooperation agent, in a slow-to-fast tempo sequence. The other 18 participants experienced the counterbalanced order of exposure to the cooperation agents. Experiment 2 was the same design as Experiment 1, except participants experienced the fast-to-slow tempo sequence.
Tempo levels: Slow-tempo and fast-tempo periods
Each eight-minute trial was divided into five 96-second intervals, representing a slow, medium, or fast rate at which patients entered the waiting rooms. In the slow-to-fast sequence of Experiment 1, participants experienced the following interval order: medium, slow, medium, fast, and medium. In the fast-to-slow sequence of Experiment 2, participants experienced the following interval order: medium, fast, medium, slow, and medium (see Figure 1). For easier communication, however, tempo period refers to the half of the trial that includes the named interval, for example, the slow-tempo period in the slow-to-fast sequence was one medium interval, one slow interval, and half a medium interval. The slow interval had two patients enter the waiting room, evenly distributed throughout that interval, whereas the medium interval had five patients, and the fast interval had eight patients. The medium-tempo interval was determined in pilot testing as an engaging pace, without high pressure.
In both experiments, the agent’s tempo sequence was always the opposite of the participant’s tempo sequence. For example, in participants’ slow-to-fast sequence of Experiment 1, agents would experience the fast-to-slow sequence. The complementary sequences allowed for an obvious coordination solution if the dyad cooperated and shared resources, making it possible to assess how differences in cooperative behavior contributed to breakdowns or delays in resource sharing. Participants (or agents) in the slow-tempo period would more likely have excess resources, whereas participants (or agents) in the fast-tempo period would need additional resources to meet demand in their hospital.
Experiment 1 was thus designed to investigate how participants reacted to an agent that requested resources more often and shared resources less often during participants’ slow-tempo period and an agent that requested resources less often and shared resources more often as participants moved into a fast-tempo period. Experiment 2 was designed to investigate how an initial perturbation, or fast-tempo period, might differently affect participant and joint cooperation. Taken together, both experiments allowed for comparison of cooperative behavior prior to a fast tempo (Experiment 1) and following a fast tempo (Experiment 2) to assess team cooperation and ability to adapt to environment changes in a timely manner.
Agent cooperation levels: High-cooperation and low-cooperation agents
We operationalized agent cooperation as high cooperation or low cooperation depending on how much its behaviors emphasized joint outcome or individual outcome, respectively. Table 1 summarizes the resource-sharing and resource-requesting behaviors for the agents.
Agent Resource-Sharing and Resource-Requesting Behavior by Cooperation Level
Note. The agents’ acceptance rates are keyed to the number of patients in the agent’s waiting room. From top to bottom: one to two patients, two to three patients, and three to four patients.
The high-cooperation agent was more likely to signal trusting, cooperative behavior when responding to resource requests. When a participant requested an available resource and the high-cooperation agent had two or fewer patients in its waiting room, the agent would accept 100% of the time, 75% of the time with three to four patients, and 50% of the time with five to six patients. The low-cooperation agent was less likely to signal trusting, cooperative behavior by accepting 50% less than the high-cooperation agent in each of the waiting room conditions, or 50%, 25%, and 0%, respectively. The low-cooperation agent thus expressed less individual risk and prioritized its own performance compared to the high-cooperation agent. Note that the low-cooperation agent is not exactly competitive, which could involve requesting all resources and refusing to return them; it is still cooperating in this context by sharing and requesting resources.
Cooperation was also expressed through requesting behavior. The high-cooperation agent would check if it needed a resource, then check if the participant had the resource before requesting. The low-cooperation agent only considered its own need for resources, which made it more likely to make requests insensitive to the participants’ needs. Agents were programmed to have an eight-second and two-second delay when assigning patient or staff and when collecting resources, respectively, to simulate the pace of a human player, established during pilot testing. This delay also avoided continuous interruption of the participant and allowed a window for participants to request unassigned agent resources.
Dependent variables
Because we wanted to know how an agent’s cooperative behaviors would influence a person’s cooperative behaviors, the dependent variables in this study were participants’ resource-requesting and resource-sharing behaviors. These were measured as the number of resources they requested and the number of agents’ requests they accepted. Agents’ behaviors were tallied independently. Individual performance was measured as the sum of patients a scheduler treated in a trial, and joint performance was the sum of patients that both participant and agent schedulers treated in a trial.
Procedure
Participants were introduced to the microworld scheduling task and interface. Participants were told they could request resources at any time from an automated agent, scheduling its own hospital in the background. To avoid guiding participants to use a particular strategy, participants were told to “treat as many patients as possible.” If participants asked if this included cooperating with the agent, the goal was restated and they were told it was up to them how to reach the goal. No information was given about the neighboring hospital agent’s behavior or potential changes in tempo.
Participants were then exposed to two, 2-minute practice trials at a medium tempo. At the end of each trial, including experimental trials, a bar graph displayed the patients they treated in the context of patients treated in both hospitals. The graph emphasized the goal of maximizing a joint score and acted as motivational performance feedback (Figure 2).
Feedback provided to the participants showed a split bar that combines the sum of patients treated in each hospital, with a darker shade representing the participant’s contribution. The sum of patients missed was also reported as a split bar when applicable.
At the beginning of a trial, participants and their agent partner were each given three nurses, two doctors, and two surgeons to assign to one of six rooms in their hospital, along with one patient in their waiting room queue (Figure 3). Hospital staff were the only shareable resources. There were two types of patients that entered the queue with randomized 50% chance; Patients “A” required a doctor and a nurse, and patients “B” required a surgeon and nurse. Both types of patients therefore required a nurse, and to use all of one hospital’s rooms, a scheduler would need to obtain all nurses from both hospitals and some additional doctors or surgeons.
A screenshot of the interface at the beginning of a trial. The bottom panel had three button options, Assign Patient to Room, Assign Resource to Room, or Request Resources. At the center were six vacant hospital rooms that also functioned as buttons. The left panel displayed the waiting rooms and available staff resources for each hospital. Each circle in the participant’s waiting room represented one patient. Two squares in the agent’s waiting room indicated the agent also had A and B type patients, but only one color status was displayed. In the top left corner, a timer counted down the trial.
To help with resource management, a side panel displayed available resources, patient waiting rooms, and a “score” for each hospital. Color codes were used to signal waiting room queue length status: green for one to two, yellow for three to four, and red for five to six patients. Six patients was a waiting room’s maximum capacity, and participants were told additional patients scheduled to arrive would be turned away. Participants could see the exact number of patients in their waiting room, with each circle representing one patient (Figure 3). However, only the color status of the agent’s waiting room was available to participants. This decision was inspired by Dabbish and Kraut (2004), which showed summarized displays can be more effective than detailed displays for communicating status information in a cooperative task. A summarized display also contributed to the uncertainty of interacting with agents, allowing a focus on the social-affective aspects of participants’ decision making rather than the information processing aspects.
Table 2 describes the main actions of the scheduling task from the participant’s point of view. Level 1 refers to the bottom control panel options in Figure 3, with subsequent navigation options as the following levels.
Participants’ Main Actions and Navigation Pathways
Note. Level 1 indicates the first level of options, and the subsequent levels describe potential pathways for the options selected. Italicized texts refer to button options, and quotations refer to interface communications. Em dashes indicate levels not applicable to that pathway.
Interface highlighting and feedback aided participants in the microworld. For example, when assigning patients and resources, rooms currently treating patients would be greyed out. Once a patient and staff were assigned to a room, “patient treatment” began automatically, lasting 60 seconds. After treatment, participants needed to click a “Collect Resources” button that appeared on top of the respective room to free the room and staff for reassignment (action not included in Table 2). The system generally did not allow erroneous assignments; however, if participants attempted to take action using unavailable patients or resources, floating text would appear with the error and last several seconds before fading.
To request resources, participants could select “Request Resource” from the bottom control panel, then select nurse, doctor, or surgeon. Immediately following this, floating text informed participants if their request was accepted or denied (Table 2, Level 3). If accepted, the resource would transfer from the agent’s hospital to participants’ hospital and the staff numbers in both hospitals would update. Incoming requests to participants blocked the control panel, forcing an interruption (Figure 4). Resources that transferred would remain at the hospital unless requested and accepted back.
A screenshot of the microworld environment interface that shows the bottom control panel obscured by a resource request.
Results and Discussion
In Experiment 1, where participants experienced a slow-tempo period followed by a fast-tempo period, we assess whether the high- and low-cooperation behaviors affected participants’ behaviors using ANOVA with three within-subjects variables (scheduler, cooperation level, tempo) and one between-subjects variable (cooperation order). Participants’ and agents’ mean requests and mean acceptances are discussed in the context of these resource-sharing variables, and mean performance (number of patients treated, individually and jointly) are considered in the context of agent cooperation. To assess the validity of averaging performance across trials, participants’ mean performance across trials was compared. No significant difference was found, suggesting minimal learning effects.
In Experiment 2, a different group of participants experienced a fast-tempo period followed by a slow-tempo period. We compare resource-sharing to Experiment 1 by how people’s cooperation differed following a perturbation, an initial fast-tempo period. We follow the same assessments as in Experiment 1 and supplement our explanations with additional visualizations and analyses. Analyses were conducted using the “stats” package in R (R Core Team, 2014); data visualizations and figures were created using “ggplot2” (Wickham, 2009). All figures are labeled from the participants’ perspective.
Experiment 1: Slow-Tempo Period Followed by a Fast-Tempo Period
Figure 5 shows agents’ and participants’ mean requests in the background narrow bars, overlaid by their mean acceptances, the foreground wider bars. Data are faceted by cooperation order (rows), cooperation level (columns), and tempo period (columns within cells). There were more requests in the fast-tempo periods (quadrants’ right columns, narrow bars) compared to the slow-tempo periods (quadrants’ left columns, narrow bars), demonstrating that agents generally behaved as designed and that participants understood the joint task and engaged in requesting resources accordingly, F(1, 34) = 91.89, p < .01. Participants’ increased requests moving from slow-tempo to fast-tempo shows the effect of tempo, F(1, 34) = 47.47, p < .01, and that people engaged agents as they needed more resources.
Agents’ (bars on the left) and participants’ (bars on the right) mean requests (narrow bars) and mean acceptances (wide bars) with 95% CIs (confidence intervals).
Mean acceptances (Figure 5 wide bars) were not significantly different overall between tempo periods (quadrants’ left and right columns), F(1, 34) = 0.48, p = .5, due to the complementary tempo design and joint structure of the task. However, acceptances across cooperation levels did interact with tempo; participants’ mean acceptances were greater in their slow-tempo period compared to their fast-tempo period (dark wide bars, quadrants’ left columns compared to right columns, F(1, 34) = 96.58, p < .01. Furthermore, when participants were supposed to be accepting requests during their slow tempo (quadrants’ left wide bars), their mean acceptances were fewer than agents’ mean acceptances during its slow tempo (quadrants’ right wide bars) in three of the four comparisons. The same was true in participants’ fast tempo compared to agents’ fast tempo (participants’ slow tempo); participants’ mean acceptances were fewer than the agents’ in three of the four comparisons. In addition, mean acceptances were generally higher in the bottom row of Figure 5, particularly in the high-cooperation condition, which reflects an interaction between cooperation order and cooperation level, F(1, 34) = 4.31, p = .05.
Appropriate cooperation means participants should generally accept more requests during their slow-tempo period and make more requests during their fast-tempo period. The question is if agent cooperation differentially affected these behaviors. While agents’ mean requests were greater than participants’ mean requests across conditions, F(1, 34) = 26.46, p < .01, and agents’ mean acceptances were greater than participants’ mean acceptances across conditions, F(1, 34) = 24.37, p < .01, overall mean requests in the high-cooperation condition were fewer compared to the low-cooperation condition (Figure 5, comparing the total narrow bars in the left column with the total narrow bars in the right column), F(1, 34) = 35.62, p < .01. Across conditions, the low-cooperation agents’ mean requests were greater than participants’ mean requests, reflecting the significant interaction for scheduler and cooperation level, F(1, 34) = 13.16, p < .01. In sum, participants requested more from the low-cooperation agent than from the high-cooperation agent.
Mean acceptances, in contrast, were greater in the high-cooperation condition compared to in the low-cooperation condition (Figure 5, comparing wide bars in the left column and right column), F(1, 34) = 37.67, p < .01. Thus, participants contributed to more productive interactions that led to an exchange of resources in the high-cooperation condition compared to the low-cooperation condition. Participants’ mean acceptances were fewer than agents’ mean acceptances and even fewer in the low-cooperation condition compared to the high-cooperation condition, F(1, 34) = 21.33, p < .01. Despite interacting more often with the low-cooperation agent, indicated by the greater number of requests, exchanges were less productive—less likely to lead to resource exchange. This shows participants were still willing to engage the low-cooperation agent, making more requests like the low-cooperation agent rather than saving effort or adopting a more autonomous strategy. It also means participants as a group did not exploit the high-cooperation agent’s relative generosity by requesting more often and refusing to return resources when requested.
The dyad handled resource exchanges differently in high-cooperation versus low-cooperation conditions, particularly in their requests. In Figure 5 (left column, narrow bars), agents’ mean requests in participants’ slow tempo were greater than participants’, and this reverses when participants reached their fast tempo—their mean requests are greater than the agents’. In the low-cooperation condition (right column, narrow bars), agents’ mean requests are greater than participants’ in their slow tempo, although this does not increase participants’ mean acceptances (wide bars). This may have led to agents’ subsequent need for resources and the mutually increased requests in participants’ fast tempo. These observations are supported by the significant interaction for cooperation level and tempo, F(1, 34) = 21.14, p < .01.
Did more productive interactions lead to higher mean performance? Figure 6 shows mean individual performance, the sum of patients treated by each scheduler, and mean joint performance, the sum of participants’ and agents’ performance. Because of task interdependence—staff unused by participants were often used by the agents and vice versa—the difference between cooperation conditions is small. In particular, the low-cooperation agents’ higher mean performance compensates for the participants’ lower mean performance and reflects the low-cooperation agent’s more autonomous strategy favoring individual team member performance. However, the lower mean joint performance supports the observation that the low-cooperation agent’s behavior led to unproductive participant behaviors, with the high-cooperation condition producing higher joint performance (Figure 6), F(1, 34) = 7.37, p = .01. When participants transitioned from the low-cooperation to the high-cooperation agent, the difference between the high-cooperation and low-cooperation conditions appears larger (Figure 6, right column differences are larger compared to the left column differences); however, cooperation order and cooperation level did not produce a significant interaction. These findings for joint performance suggest that an inconsiderate interrupter that was less generous with resources—the low-cooperation agent’s persona—could lead to lower joint performance in a task involving people.
Mean patients treated with 95% CIs (confidence intervals) show individual performance and joint performance, with joint performance lower in the low-cooperation condition compared to the high-cooperation condition. Joint performance is plotted as the average of participants’ scores and agents’ scores.
While appropriate cooperation was defined as generally accepting more requests during slow-tempo periods and making more requests during fast-tempo periods, results show fewer requests occurred with the high-cooperation agent compared to the low-cooperation agent. However, both agent and participant behaviors contributed to the higher joint performance in the high-cooperation condition. Overall, these results show the value of considering reactive (responding to requests) and proactive (making requests) behaviors as part of evaluating appropriate cooperation in a joint task and that social exchange factors may be relevant for human-agent interaction—agent cooperation can affect human cooperation.
Experiment 2: Fast-Tempo Period Followed by a Slow-Tempo Period
Requesting behavior in Experiment 2 was similar to Experiment 1; agents’ mean requests were greater than participants’ mean requests, F(1, 34) = 124.87, p < .01, and mean requests were greater in the low-cooperation condition compared to the high-cooperation condition, F(1, 34) = 93.72, p < .01. In addition, mean acceptances were greater in the high-cooperation condition compared to the low-cooperation condition, F(1, 34) = 33.27, p < .01, and the two schedulers’ mean requests were greater in their respective fast-tempo periods, demonstrating the agents behaved as designed and that participants were engaged in the microworld task, F(1, 34) = 169.43, p < .01. Furthermore, participants and agents differed in their requests, particularly depending on cooperation levels; the low-cooperation agents’ mean requests were greater than participants’ mean requests across conditions, F(1, 34) = 89.85, p < .01.
Contrary to expectations, mean requests in the slow-tempo period were greater than in the fast-tempo period, F(1, 34) = 223.70, p < .01. This highlights the first difference between Experiment 2 and Experiment 1. Figure 7 (quadrants’ right columns, narrow bars) partially explains these findings; agents’ mean requests during participants’ slow tempo were greater relative to the other conditions and relative to Experiment 1, with an especially large portion of requests made by the low-cooperation agent, F(1, 34) = 41.92, p < .01. This is also reflected in the significant interaction term for scheduler and tempo, where the agents’ mean requests were greater than participants’ mean requests during participants’ slow-tempo period, F(1, 34) = 222.18, p < .01; the significant interaction term for cooperation level and tempo, where mean requests were greatest overall in the low-cooperation, slow-tempo period, F(1, 34) = 41.92, p < .01; and the significant interaction term for scheduler, cooperation level, and tempo, F(1, 34) = 20.48, p < .01.
Agents’ (bars on the left) and participants’ (bars on the right) mean requests (narrow bars) and mean acceptances (wide bars) with 95% CIs (confidence intervals).
Because agent actions were partially contingent on participants’ actions, a complementary explanation for the higher number of requests in the slow-tempo period is that participants delayed and did not request resources early in their fast-tempo period. This delay caused a backlog of patients in their hospitals, which reverberated through their requesting and accepting behaviors in the second half of the trial. The greater number of agent requests are thus likely due to participants retaining resources leading into the agents’ fast-tempo period (and participant’s slow-tempo period), resulting in agents running out of resources more quickly compared to Experiment 1. This failure of participants to quickly adapt in the early fast-tempo might be described as a failure to maintain their margin of maneuver (Stephens et al., 2011), which led to subsequent system breakdowns and delays in the joint task. A visualization of the request sequence shows that participants did not make requests until the end of their fast-tempo period (Figure 8).
Aggregate requests of all Experiment 2 participants (outlined bars) and agents (un-outlined bars) plotted sequentially show that peak participant requests occur toward the end of their fast-tempo period.
The effects of participants’ delayed requests also seemed to affect participants’ and agents’ acceptances. Contrary to Experiment 1, participants’ mean acceptances were greater than agents’ mean acceptances in Experiment 2, F(1, 34) = 15.97, p < .01. It seems the low-cooperation agent was more badly in need of resources during participants’ slow-tempo period in Experiment 2 compared to Experiment 1 due to participants’ delayed resource requests and subsequent delayed use of resources at the start of their slow-tempo period, and thus agents accepted less often in Experiment 2.
Without the benefit—or an example—of a requesting agent early in the trial, participants seemed focused on scheduling within their own hospital rather than requesting resources from the agent. This is especially true when comparing the timing of participants’ requests in Experiment 2, which peak at the end of the fast-tempo period in Figure 8, and the timing of participants’ requests in Experiment 1, which peaked earlier, halfway through the fast-tempo period (not pictured). Thus, participants’ mean number of requests during an early fast tempo may not have been sufficient. This would lead to requests increasing toward the end of the fast-tempo period to address buildup in the patient queue. Participants’ tendency to adopt an autonomous strategy for assigning patients and staff to available rooms meant they neglected the opportunity to gain resources to serve patients.
Despite this effect in Experiment 2, agent cooperation had a positive effect on joint performance, supporting our initial hypothesis. The dyad in the high-cooperation condition had greater resource exchange between tempo periods compared to the dyad in the low-cooperation condition. This is reflected in the significant interaction term for cooperation level and tempo, F(1, 34) = 4.46, p < .05; the significant interaction term for scheduler, cooperation level, and tempo, F(1, 34) = 17.02, p < .01; and in Figure 7, where the differences between quadrants’ left and right wide bars are greater in the left column (“HighCoop”) compared to the right column (“LowCoop”). It is also noteworthy that compared to Experiment 1, and particularly in the high-cooperation condition, the fast-tempo period in Experiment 2 did not reduce participants’ tendency to share resources despite the agent demanding much less often.
It is tempting to compare performance between the two experiments given their parallel design. However, plotting when agents released staff against when participants reached maximum queue length shows that even with earlier requests in Experiment 1, participants were still unable to treat the backlog of patients in their waiting room before the trial ended. Thus, the only conclusion we can make about joint performance in Experiment 2 is that it supported the findings from Experiment 1—cooperation affects joint performance. Joint performance was lower with the low-cooperation agent, F(1, 34) = 12.24, p < .01, and even lower when the participants initially interacted with the low-cooperation agent, F(1, 34) = 5.63, p = .02 (Figure 9).
Mean patients treated with 95% CIs (confidence intervals) show individual performance and joint performance, with joint performance lower in the low-cooperation condition compared to the high-cooperation condition.
To further investigate the interdependence of the agent and participant, a linear regression was calculated to predict joint performance based on participant performance. For Experiment 1, participants’ performance accounted for a much smaller proportion of the variance in the joint performance, F(1, 142) = 18.09, p < .01, R2 = 0.11, compared to Experiment 2, F(1, 142) = 273.00, p < .01, R2 = 0.66, supporting the idea that participants used a more autonomous strategy in Experiment 2.
General Discussion
Both experiments confirm that cooperation with automation is an important complement to reliance on automation. People and automated agents were intentionally placed in parallel roles to focus on the need to coordinate in a dynamic task environment through cooperation. While sharing resources potentially produces conflicting goals, in this study, demand was engineered to minimize conflict. From the researcher’s perspective, there was a relatively clear coordination solution—schedulers in the slow tempo would be able to share resources with their counterparts experiencing a fast-tempo period. This allowed us to test how participants responded to agents expressing different levels of cooperation and to motivate future studies on the social or affective influences of cooperation rather than only the information processing components of the task. Overall, both experiments showed agent cooperation affected participant cooperation, supporting our general hypothesis that agent cooperation could be an important construct in human-agent coordination beyond automation reliability and its effects on reliance and compliance.
Although the goal was to “treat as many patients as possible,” participants could take a defensive approach (Stephens et al., 2011), requesting resources without returning them, to maximize patients treated in their own hospital. However, such defensive behavior would undermine cooperation and reciprocity (Axelrod & Hamilton, 1981) and in the microworld lead to fewer patients treated. Though resource-sharing decisions may have been governed by the instrumental value of resources (Molm et al., 2007), the environment demand was not made explicit to participants. Therefore, the act of sharing resources demonstrates their trust in the agent. Accepting requests came at the cost of losing spare resources, reducing their margin of maneuver, and risking that the agent would not reciprocate and return resources when needed. In both experiments, less cooperative agents induced less cooperative behavior in the participants and more cooperative agents induced more cooperative behavior in the participants.
Our hypothesis on cooperative reciprocity for Experiment 1 was thus supported. The cooperative nature of agents affected both dimensions of participants’ cooperative behaviors—resource sharing and resource requesting. Participants engaged in more productive exchanges—successful requests—with the high-cooperation agent than with the low-cooperation agent despite the low-cooperation agent requesting resources from participants more often. This suggests people reciprocated the cooperative behavior of the agent rather than merely mirroring the number of requests. In addition, participants who previously experienced a low-cooperation agent and were working with the high-cooperation agent demonstrated even more productive exchanges than participants who experienced the high-cooperation agent first. It may be that participants were better at timing their requests in later trials to coincide with when the agent had available resources, but there was no evidence of learning effects. It may be that the high-cooperation agent surpassed low expectations formed with the low-cooperation agent, indicating the interaction history influenced decisions to cooperate. This latter explanation and the higher joint score with the high-cooperation agent support the idea that to maintain resilience, systems must continually invest in a renewal of shared goals and a willingness to accommodate (Woods, 2004).
The hypothesis for Experiment 2—that if people’s initial exposure to an agent is in a highly demanding situation, then the effects of agent cooperation level will be more prominent—was partially supported by participants’ mean acceptances, particularly in the fast-tempo, high-cooperation condition. However, the initial fast-tempo period may have overwhelmed participants and led to more autonomous requesting strategies. Rather than anticipating demand and making sufficient early requests, participants’ delay led to a backlog that undermined their ability to accept requests as agents transitioned into fast tempo (especially the low-cooperation agent). However, when prompted, people still responded in a cooperative way that contrasted their relatively autonomous requesting strategy. This suggests cooperation may have two dimensions—proactive and reactive actions (Hollnagel, 2012). Given that participants started with a medium-tempo margin in each trial and were free to request resources, it remains a question why participants did not anticipate demand in subsequent trials. Certainly expertise with such demanding situations in actual work environments might ameliorate the tendency toward an autonomous, reactive response. However, our results are consistent with the general error tendency of cognitive tunneling when people confront unexpected high-demand situations (Woods et al., 1994). In summary, this autonomous behavior was limited to reducing participants’ requests for resources and did not reduce their tendency to accept requests.
Previous work in human-automation cooperation has focused on performance, with adjustable autonomy typically referring to better coordination (Zieba et al., 2010) or better management of functional dependencies. That speed of assigning resources was an important factor for this particular task might make this study a strange example of a joint human-agent task; automation is known to be much better at this than people (Fitts et al., 1951). However, it is not always the case that automation should be used when the costs of a disengaged human counterpart would be greater than the efficiencies of implemented automation (Kirlik, 1993). Our study focuses on the social processes of coordination in resilience, in circumstances where reciprocity and goal tradeoffs feature more prominently than reliability and mode management. Doing so departs from the idea that teams should be formed purely based on complementary abilities rather than how those team members interact. It also challenges the tempting idea that if functions can be automated, then they should be automated.
Establishing trust between people and autonomous agents may be one of the most daunting problems for the success of human-automation teams (Groom & Nass, 2007), and is important for cooperation and maintaining margins of maneuver. To work to this ideal, we investigated a shared-resource activity between people and automation, where social processes could potentially limit joint performance. As Groom and Nass (2007) observe, future research intent on developing human-robot teams must go beyond technical performance to address the social and organizational qualities that make a successful teammate.
Limitations and Future Directions
Microworlds have the advantage of mimicking interactive situations observed in field studies but with a greater degree of experimental control (Gonzalez, Vanyukov, & Martin, 2005). This experimental control is particularly valuable when investigating the dynamics of interdependent behavior. Microworlds make it possible to manipulate the behavior and interdependencies of dyads, the simplest unit of agent networks in complex sociotechnical systems, but such interdependencies can also lead to challenges in the analysis.
In this study, the high-cooperation agent’s requesting behavior could be affected by participants who were slower or chose not to use their resources because the high-cooperation agent was designed to check if participants’ resources were unused prior to requesting them. The low-cooperation agent’s requesting behavior could be affected by how quickly it ran out of resources it needed, which could be influenced by resources it did or did not have due to the participant’s actions. Both high-cooperation and low-cooperation agents’ acceptances depended in part on when participants made requests. This led to challenges in interpreting the results. However, this interdependence approach echoes the challenge of designing strategies to combat conflict in such joint tasks, where a person’s decisions depend on the agent’s decisions, which depend on the person’s decisions, and so on (Schelling, 1960). Most studies seek to avoid interdependencies, but a systems perspective that considers these interdependencies is becoming more important with the changing role of increasingly autonomous automation.
In practice, it may be difficult to draw conclusions about the impact of cooperative behavior on performance outcomes; it is possible that as cooperative activity goes up, productivity decreases or remains unchanged or other important measures are affected, such as worker turnover and quality of work life. Cooperation may create value where there is none, and the quality of the output is not necessarily measured in terms of responding to demands more quickly (Wicks, Berman, & Jones, 1999), as represented in this microworld. In other words, increased cooperation might not increase performance in routine situations even if it enhances resilience in high-demand situations. As a consequence, organizations might neglect the importance of cooperative automation.
It is challenging but critical to identify performance measures that emphasize priorities consistent with cooperative control. In practice, these priorities can manifest as micro-decisions to cooperate that extend to macro outcomes for a system of interconnected agents. To avoid the tendency of reactive responses, future microworld studies could explore different social exchange structures, moving from negotiated exchange, where exchange decisions are made jointly, to reciprocal exchange, where decisions are more unilateral, like in altruism (Molm et al., 2000). Such a design could refocus human-agent interactions from interrupting to request resources, a structure with more reactive affordances, to interrupting with needed resources, a structure with more proactive affordances. A key question would be to understand how to foster proactive behavior that helps systems maintain margins of maneuver and to understand what causes people to ignore or avoid early opportunities that could enhance cooperation.
Conclusion
This study complements the supervisory control perspective by reframing human-automation interaction in terms of cooperative control for human-agent systems. Cooperation is central to resilience because it enables networks of interdependent agents to pool resources and accommodate a greater range of surprises. People can become less cooperative when interacting with less trusting and inconsiderate agents, more cooperative when interacting with more trusting and considerate agents, and cooperative strategies might be expressed in both proactive and reactive behaviors. These behaviors are qualitatively different than those observed in investigations of supervisory control automation. This study’s approach thus departs from the typical supervisory control approach to human-automation interaction and acts as a starting point for future studies to explore how cooperation in human-agent interactions can enhance system resilience.
Key Points
Rather than supervisory control situations where reliance and compliance are the focus, increased automation capability may produce cooperative control situations where reciprocity and proactive and reactive cooperation are the focus. People adopt cooperative, defensive, or autonomous strategies to maintain their margin of maneuver in dynamic exchange scenarios with an automated agent, and an automation agent’s strategies influence those adopted by people. Analyzing human-agent interaction in a dynamic system presents a challenge but is essential in revealing patterns of interactive behavior underlying decisions to cooperate.
Footnotes
Acknowledgements
This work was supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE-1256259. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of the National Science Foundation. The authors would like to thank Anousoune K. Bounket and Kim Le for their work on the microworld, and Ja Young Lee, David Woods, Vindhya Venkatraman, and three anonymous reviewers whose comments improved the manuscript.
Erin K. Chiou is a PhD student and National Science Foundation Graduate Research Fellow in the Department of Industrial and Systems Engineering at the University of Wisconsin-Madison. She received her master’s of science in industrial and systems engineering from the University of Wisconsin-Madison in 2013.
John D. Lee is Emerson Electric Quality and Productivity Professor in the Department of Industrial and Systems Engineering at the University of Wisconsin-Madison. He received his PhD in mechanical engineering from the University of Illinois, Urbana-Champaign in 1992.