Abstract
As machines increasingly behave more like active cognitive agents than passive tools, additional heuristics for supporting joint human-machine activity are urgently needed to complement existing usability heuristics. Despite the rich and extensive design guidance produced by forty years of cognitive systems engineering (CSE) and related fields, the lack of large-scale impact can be attributed, in part, to insufficient translation of CSE principles and guidelines to language and tools that are ready for designers and other decision-makers responsible for these automation-infused solutions. Towards this need, we synthesized a partial and preliminary list of ten machine requirements intended to capture some of the essentials of joint activity. We believe solidifying these essentials and their implications for machines is a first and necessary step towards deriving joint activity design heuristics that are valuable, practical, and sustainable for operational personnel. Through iterative refinement, we believe the combination of strong ideas and strong practicality in these tools can be the basis for a large-scale shift in the design and evaluation of human-machine teams.
Keywords
Introduction
The proliferation of artificial intelligence (AI) and machine learning (ML) available to organizations and the increasing variety of ways in which these technologies can interact with people (Amershi et al., 2014; Arrieta et al., 2020) emphasizes the pressing need for reliable, practical, and sustainable tools to help stakeholders quickly identify which technologies are likely to be successful for them. This is nontrivial because people overestimate machine capabilities and how they interact with people so frequently that it has been called a law (see “Robin Murphy’s Law” in Woods & Hollnagel, 2006), which could be feasibly measured as an “over-promise rate” (NASEM, 2021). Organizations need additional support to guide the development of AI/ML technologies early in the design process in order to evaluate an increasing volume of AI/ML options, counteract the tendency to overestimate capabilities, and quickly discard disadvantageous options before the cost of change becomes prohibitively expensive.
For this needed intervention to be impactful, its implementation must be grounded in strongly supported scientific ideas, aligned with stakeholders’ mental models about what is relevant and valuable, pragmatic relative to the costs of implementation, compatible with the resources and actors available, and sustainable as organizational goals and priorities continue to change over time (Fitzgerald, 2019). However, there are currently no available tools and methods for designing and evaluating how well technologies can integrate into a joint human-machine cognitive system that fully satisfy these requirements for impactful implementations.
Although empirical evaluations, either before (Roth et al., 1987) or after (Beede et al., 2020) full-scale implementation, are perhaps the most reliable methods to uncover the ways in which technologies can interact with people in surprising (and detrimental) ways, they are neither sufficiently pragmatic nor sustainable to have a large-scale impact on their own. High-fidelity testing is costly. It typically involves solutions late in their development lifecycle, reserved time from end-users, and months or more of planning and preparation from a dedicated study team. Because of these constraints, tests are often limited to a focused subset of candidate technologies. Therefore, empirical evaluations alone are usually too costly (in money, time, and human resources) to be pragmatic for a broader exploration of candidate technologies, or else too slow and limited to sustainably keep pace with changing organizational needs. Though some empirical testing is almost certainly necessary to mitigate the risks of implementation surprise, complementary analytical evaluations (e.g., heuristic reviews) will also be necessary to satisfy the pragmatic and sustainable requirements of organizational interventions.
Despite widespread adoption, usability heuristics alone are conceptually insufficient to reliably evaluate technologies that behave more like active cognitive agents than passive tools (Rayo, 2017). The prominence, popularity, and consistency of principles like Nielsen’s 10 usability heuristics (Nielsen, 1994, 2020) is a testament to their model alignment, pragmatism, and sustainability relative to the interface design community. Nevertheless, as technologies become capable of making inferences and interjections, there remains concern that usability alone does not sufficiently address the critical need to support macrocognitive and coordinative functions like event detection, sensemaking, replanning, forming a basic compact, and common grounding (Rayo, 2017). There is strong agreement among experts that conceptualizing AI/ML technologies like teammates or team players is valid and valuable beyond conceptualizing these technologies simply like tools (NASEM, 2021). Therefore, tool-like usability heuristics alone do not appear sufficiently grounded in the supported scientific ideas to be impactful interventions.
A growing body of literature from cognitive systems engineering and other related fields is providing guidance beyond usability alone for designing and evaluating AI/ML technologies for human-machine teaming (NASEM, 2021); however, despite forty years of research, this literature has yet to have the large-scale impact desired beyond the human factors community (Dominguez et al., 2021). This shortcoming may in part stem from the wide variety of perspectives, each with their own terminology, from which relevant AI/ML guidelines can be derived, including: joint activity (Klein et al., 2005), teamwork (Feigh & Pritchett, 2014), interdependence (Johnson et al., 2014), situation awareness (Endsley, 2017), macrocognition (Klein et al., 2003; Rayo, 2017), distributed problem solving (Smith, 2018), joint cognitive systems (Woods & Hollnagel, 2006), human-AI interaction (Amershi et al., 2019), and resilience (Woods, 2019). Furthermore, most of this guidance was written for human factors professionals, often too vague or inaccessible for a broader audience (e.g., Rayo, 2017). In either case, the inconsistency and inaccessibility of guidance suggests that this body of literature as a whole, regardless of its potential value, has insufficient alignment and/or pragmatics to engage the broader set of organizational actors (beyond the limited number of human factors professionals) required to keep pace with the scale and tempo of AI/ML proliferation.
The aim of this work is to begin to bridge the gap between the scientific literature on joint human-machine activity and the actors available to make large-scale impacts. Our end goal is to compile, synthesize, and translate guidance relevant to designing for and evaluating joint activity in a way that is understandable, pragmatic, and sustainable for operational personnel. We chose to specifically target operational personnel because they are likely to be the people in organizations who (1) are most familiar with the technologies and demands of work, (2) have a direct stake in the outcome, and (3) comprise a large workforce capable of a large-scale impact. However, before we can address this end, we first seek to solidify the essentials of joint activity from which subsequent design heuristics can be derived. We report our partial and preliminary findings, focused specifically on the subset of joint human-machine activity, as a set of ten machine requirements stemming from joint activity essentials. We expect this partial list to grow and change as we expand beyond human-machine activity to joint activity in general and as we shift from machine-focused requirements to system-focused essentials.
Methods
We began with a broad review of the literature relevant to human-machine joint activity, including but not limited to the literature bases mentioned previously. For each article, we compiled a list of all takeaways relevant to designing for and/or evaluating joint human-machine activity and categorized each takeaway as either a problem (to be avoided) or guideline (to be design for). We used these problems and guidelines as the basis for thematic analysis. In alignment with the constant comparative analysis method (Fram, 2013), we first constructed open codes to reflect the individual language of the authors of each article. We then proceeded through iterative axial coding to group our open codes into clusters of common concepts. Informed by what we were finding, we continued to review the literature until we had reached saturation.
From these axial codes, we next constructed a set of machine requirements for satisfying joint activity essentials. First, we composed one or more concise evocative statements to represent each of our axial codes. Importantly, we chose language intended to resonate with operational personnel, which was not necessarily the language found in the academic literature, to better align with their mental models. Second, we expounded upon each of these evocative statements with a short 1-3 sentence synopsis. Finally, we elaborated on each heuristic even more with 1-3 paragraphs of further details, a toy example to illustrate the core concept(s), and a list of tips and tricks for effective use. We modeled this progressive description of each heuristic after Nielsen (2020) and expected that this would help to engage operational personnel in a way that is meaningful, but not overwhelming.
Results
Our literature review covered a total of 113 articles. From these articles, we identified 590 problems and/or guidelines relevant to the design and evaluation of joint human-machine activity. From these open codes, we identified nine axial codes: jointness or teamwork, escalation and resilience, observability, directability, limits and boundaries, informativeness, redirecting attention, interjection strength, and diversity. The following section includes the partial and preliminary ten machine requirements we composed to convey to operational personnel the core concepts underlying these nine axial codes. Due to space constraints, we report only the tagline (in bold) and a consolidated synopsis of the remainder of related materials for each requirement.
Ten Machine Requirements To Satisfy Essentials Of Joint Activity
The machine changes and augments what people can do, rather than replaces them
Machines can never replace people, but they can change people’s roles and the kinds of the work people do (Woods & Hollnagel, 2006). No matter how capable machines become to take actions on their own, people will ultimately be held responsible for the actions of machines (Murphy & Woods, 2009). At minimum, this requires people to monitor or supervise machines, which often creates additional problems (Sarter et al., 1997). Not only do machines that attempt to replace people fail to do so, they also miss out on the benefits of being part of a team. Effective teams enable all members to contribute to the work so that the team accomplishes more than any of its members could accomplish alone. Likewise, machines should augment, amplify, or contribute in a way that aids, not replaces, the work of people (Woods, 1985).
The machine interacts with people: they cannotremain separate or invisible
Effective teams are well-coordinated, which requires team members to interact with others in ways that help the team align their activities. These interactions include making actions and intentions visible to others, monitoring others’ actions and intentions, synchronizing actions, aligning goals, shifting roles, and minimizing the burden that these interactions have on others in the team (Klein et al., 2005). Machines attempting to remain separate from or invisible to people do not eliminate the need for people to coordinate with machines, but rather make this coordination difficult or impossible to accomplish (Roth et al., 1987). Instead, machines should expand, rather than reduce, the ways in which they can interact with people to coordinate activities (Johnson et al., 2020).
The machine clearly indicates to people its past, present, and future status and intentions
To be well-coordinated, teams need to have a shared understanding of the overall task and the progress being made towards achieving it (Klein et al., 2005). Team members need to clearly communicate their activities, status, and intentions so that others can seamlessly and efficiently coordinate their current and future actions without surprises. For machines, this requires more than simply making actions or computations visible. People must understand what the machine has done, what it is doing, and what it is going to do next (Sarter et al., 1997).
The machine conveys to people the ways it is misaligned to the world, even when it is unaware that it is misaligned to the world
Machines are literal-minded, meaning that they will only function in accordance with their programming and resultant models of the world. However, a machine cannot tell if its model of the world is in fact the world it is in (Woods & Hollnagel, 2006). Consequently, machines will often take the right actions (according to their models of the world) in the wrong world. People need to help keep machines aligned to the current situation and make sure they are not operating outside their limits (Hoffman et al., 2002). To help people do this, machines must send signals or clues to people that convey when, how, and why they are operating outside their limits. However, like any team member, machines have limits to how well they can understand their own limits (Woods, 2018). Therefore, machines will be unreliable at communicating their own limits and need help from people. At minimum, this requires some way for people to simultaneously understand the world, how the machine is “seeing” the world through its model, and how well the machine’s view of the world is aligned to the world itself (Rayo et al., 2020).
The machine changes its actions, plans, and priorities based on people’s directions
Effective teammates are able to direct the actions of others and be directed by others (Klein et al., 2005). However, unlike human teams which share responsibility, people are always responsible for the outcomes of machine actions (Murphy & Woods, 2009). Therefore, people must always be in control. Otherwise, people will find ways to indirectly exert their influence, like turning off the machine (Christoffersen & Woods, 2002), or else machines can cause catastrophic accidents, like the two Boeing 737 MAX crashes which killed a total of 346 people (FAA, 2020). The ability to comply with people’s directions is a compulsory machine design requirement (Johnson et al., 2020).
The machine helps people adapt how it works to keep the system working
As situations escalate from normal to exceptional, machines often get in the way more than they help (Woods & Patterson, 2000). Machines stick to their set of rules, but successfully responding to these exceptional situations often requires changing procedures, sacrificing some goals, or otherwise breaking the normal rules (Chuang et al., 2019). Machines must allow (or even help) people break the machine’s rules during exceptional circumstances; otherwise, machines are likely to exacerbate the situation with additional burden (Woods & Patterson, 2000).
The machine helps guide, but does not force, people’s flow of attention to what is important
Attention is limited. Effective teams send signals to help each other shift and focus their attention on what is important; however, machines often struggle to understand what is important (Woods, 1995a). Machines that frequently send unhelpful alerts can annoy or even impair people’s ability to figure out what is important (Rayo & Moffatt-Bruce, 2015). Therefore, people must be able to determine whether the signals machines send are important without having to fully shift their attention from what they are currently doing (Woods, 1995a).
The machine conveys to people changes or events without disrupting their workflow
Effective teammates assess whether their message or action is important enough to interrupt what another is doing, which depends upon both the importance of the interruption and the importance of the other’s current actions (Klein et al., 2005). However, machines are often poor at gauging interruptibility, which can exacerbate periods of high workload with additional unhelpful disruptions (Woods, 1995a). Machines should alert people to important changes in the situation in the least disruptive manner possible, without loss of information or urgency, so that managing the interruptions themselves does not become an additional burden.
The machine simultaneously helps people understand the details and the bigger picture
Though machines can detect, process, and display massive volumes of data, they struggle to reliably make sense of what that data means, especially when something is novel or unexpected. For example, a substantial proportion of AI/ML research continues to focus on image classification (Arrieta et al., 2020), something so basic for people that we do not consider it to be a decision. Machines still largely rely upon people to understand the bigger picture, but data overload becomes a problem. The volume of data machines display can overwhelm and inhibit people from seeing the bigger picture, yet machines that reduce this volume risk removing crucial data, which also prevents people from seeing the bigger picture (Woods et al., 2002). Effective machines must organize, but not reduce, the data available in a way that helps people simultaneously see the bigger picture without getting lost in the details and see the details without losing sight of the bigger picture (Woods, 1995b).
The machine enables and encourages people to consider multiple different perspectives
Every perspective is limited, both revealing and hiding certain aspects of the current situation (Hoffman & Woods, 2011). Effective teams overcome the limits of each perspective by supporting people in seeing and contrasting multiple viewpoints. Machines need to be explicitly designed to support switching, comparing, and combining different viewpoints. Otherwise, switching between viewpoints can be too costly or disruptive (Woods, 1984). Machines must provide low-cost ways for people to shift perspectives and highlight when it is valuable to do so.
Discussion
Synthesizing the rich and extensive literature on joint human-machine cognitive systems in a way that is valuable, practical, and sustainable for operational personnel is ambitious. We chose colloquial (rather than academic) language and phrases to align and connect with practitioner mental models. We limited the scope and level of detail in communicating concepts to enhance the pragmatics and explainability of these requirements. This operationalization, like any, risks oversimplification of the true complexity underlying these concepts (Feltovich et al., 2004). However, we tried to mitigate this risk by composing statements that would evoke rather than mask the underlying complexity. Ultimately, we believe this operationalization of concepts specifically for operational personnel is a necessary endeavor if the literature on joint cognitive systems is to have the large-scale impact that is desired (Dominguez et al., 2021).
However, the utility of these machine requirements, essentials of joint activity, and eventual design heuristics can only be fully evaluated by the impact they have for operational personnel. To this end, each should be empirically evaluated to assess how and how well operational personnel can utilize them. This involves, at minimum, the reliability, validity, and pragmatism of using these tools to design and/or evaluate candidate technologies. An effective intervention like joint activity design heuristics will require both strong ideas and strong practicality to have a large-scale and beneficial impact.
Footnotes
Acknowledgements
This material is based upon work supported by the US Air Force Research Laboratory 711th Human Performance Wing (US AFRL 711 HPW) under Contract No. FA8650-22-C-6453. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of the US AFRL.
