Abstract
Transparency in automation and AI systems is the ability for the operator to know or see the agent’s working processes to be able to more accurately trust them. The problem is that automation frequently exists to offload an otherwise overly busy human operator. Requiring that operator to process, understand, and evaluate “transparency information” on the automation’s processing in the moment of execution is likely to be too big a task. This panel will explore the need, opportunity, methods and benefits stemming from spreading the transfer of “transparency information exchange” throughout the lifecycle of human–autonomy interaction, thereby taking advantage of best practices in training, pre-mission planning, explanation, and post-mission debriefing, after action reviews and even learning and initial design across organizational stakeholders to reduce the need and resources required to transfer transparency information “in the moment” in frequently high tempo periods of execution.
Keywords
Introduction
Transparency in automation and AI systems is the ability for the operator to know or see—as “through glass”—the workings of the machine. Bhaskara, Skinner and Loft’s recent (2020) review of the transparency literature says “In a transparent system, information regarding the agent’s actions, decisions, behavior, and intentions is communicated to the operator through an appropriate interface with the aim of improving trust in the system, performance, and operator situation awareness (SA).” (p. 216). Increasing system transparency has generally been shown to improve human SA, trust and, frequently, overall human–machine performance over systems that include less or no transparency (de Visser, Cohen, Freedy, & Parasuraman, 2014; Mercado, et al., 2016; Lyons & Havig, 2014; Ososky, Sanders, Jentsch, Hancock, & Chen, 2014).
How we commonly talk about and research the concept of transparency presents a problem, however. Automation is generally installed precisely because human operators do not have time, skill, attention, or adequate precision to perform the automated task and/or because those human resources must be devoted to some other tasks. Yet, for transparency to function and yield benefits, the human must be able to absorb the requisite “transparency information” and use it to first understand what the automation is doing and then determine whether and how to intervene to support or override it. That is, the human must not only perform the cognitive task automation was installed to alleviate the human from doing but must also evaluate automation’s performance of that task and decide whether it is acceptable. The result is conflicting goals—in precisely the contexts where the human is known to have insufficient capacity to perform the automated task, they are asked to absorb and evaluate more information about what the automation is doing.
One solution to this problem may be to take advantage of reduced human task load at other portions of the human-machine integration lifecycle to either: 1) transfer transparency information then, so that it does not have to be conveyed during the highly constrained period of execution, and/or 2) to create conditions that enable transfer at execution time to be more efficient and require fewer human resources. This might be called taking a LifeCycle Transparency (LCT) perspective (Miller, 2021) on the requirements and methods for transparency information exchange. In practice, LCT involves making use of pre- and post-mission phases to help ensure that the human operator understands and can affect how the automation intends to perform under a variety of conditions so that less transparency information will have to be transferred and processed in the moment of execution.
Training and even system design can serve to inculcate knowledge of how automation is likely to perform. Pre-execution planning and contingency planning, as well as rehearsals with simulation, can provide detailed and mission-specific knowledge about expected automation performance, as well as more general knowledge about its “style” of thought and behavior. Post-execution debriefing and explanation can be used to convey transparency information after the fact, though this may still be highly useful to the extent that the system will be used by the operator again in future similar conditions.
Panel Description and Objectives
The objectives of this panel will be to discuss, clarify and apply the concept of LifeCycle Transparency. Thoughts on what constitutes “transparency information” and how and when it might be, or be made to be, present in other portions of the system-usage lifecycle will be pertinent, as will evidence and anecdotes illustrating benefits accruing from transparency information exchange “spreading” across the lifecycle. Methodologies, tools and approaches from system design and from organizational practice for such spreading will be welcome as well. Finally, techniques for studying the effects of lifecycle transparency in the lab and for evaluating the effectiveness of LCT in real world environments will also be a focus for this panel.
Following a general introduction of the concept and definition of LCT, each individual panelist will be allotted approximately 6 to 8 minutes to describe their perspective, approach and, at a high level, any findings or lessons learned from their work. The remaining time in this session will be devoted to audience questions and panel discussion.
The discussion session will be largely audience-driven, but potential questions might include:
What is the nature of “transparency information” and what are the conditions or requirements that necessitate its transfer in or near the time of execution vs. at other times of the human-machine system lifecycle?
We have proposed training, pre-mission planning, post-mission debriefing and explanation as forms of temporally shifted transfer of transparency information. Are there other approaches? Are there characteristics which define when and how specific techniques work to displace transparency information transfer from the time of execution?
Is there evidence that temporal shifting of transparency information into non-execution portions of the life cycle can, in fact, reduce the need for in-mission information bandwidth? Can such transfer happen without loss of performance?
What human and/or machine interaction and performance phenomena are affected by LCT? What are good metrics for assessing impacts?
Why does LCT work (assuming it does)? Can such understanding be used to predict in what circumstances it will work better or worse?
Participant biographies and brief descriptions of the panelist’s research and perspectives are provided below.
Participant Biographies
Daniel Barber
SOAR Technology
Dr. Daniel Barber has over 15+ years of experience conducting and applying interdisciplinary research within the fields of human-machine interaction, robotics, machine learning, modeling and simulation, augmented cognition, physiological assessment, control systems, path planning, communication frameworks, and environment modeling. In the execution of these efforts, Dr. Barber has also developed multiple prototype live virtual and constructive (LVC) systems and autonomy that enable human–machine interaction in the domains of counter small–unmanned air systems (C-sUAS), mounted and dismounted human–robot teaming for the Department of Defense, driverless cars, and nuclear power plant main control room operations. For the U.S. Air Force Research Laboratory’s (AFRL) “Counter-sUAS C2 Automation-Autonomy and Human Machine Teaming” project he was the lead developer for the course of action recommendation service, and under the Army’s Robotics Collaborative Technology Alliance delivered a multi-modal interface using speech and gestures for squad level human–robot collaboration. He has extensive experience executing human-in-the-loop experiments both in the field and laboratory to evaluate human–machine team performance and cognitive demands and is currently the principal investigator for an effort with AFRL to evaluate the effects of transparency on a human autonomy team lifecycle within the domain of C-sUAS.
Eric Holder
Human Research and Engineering Directorate
Army Research Laboratory
Dr. Eric Holder received his Ph.D. in experimental psychology from Texas Tech University in 2003. He is a Human Factors professional and specializes in qualitative and quantitative data collection and analysis, user-centered design and evaluation, instructional design, human performance, and task and workload analysis. Dr. Holder has formal training in human factors engineering, usability, experimental psychology and statistical analysis and is cross-trained in cognitive, social, and industrial-organizational psychology and industrial engineering and has more than 18 years of experience in human-centered research, development and design. This research experience includes the government, industry, commercial and military sectors and both U.S. and international work across a wide range of projects and stages of the research and development lifecycle. His current position is with the U.S. Army Research Laboratory's Human Research and Engineering Directorate (HRED) stationed at Ft. Huachuca, AZ. Example projects include research on human autonomy teaming (HAT), explainable artificial intelligence, computer vision support to imagery analysis workflows, and advanced manufacturing of mission-adaptable 3D printed drones, among other themes.
Lixiao Huang
Arizona State University
Dr. Lixiao Huang is an Associate Research Scientist at the Center for Human, Artificial Intelligence, and Robot Teaming (CHART) within Global Security Initiative (GSI) at Arizona State University. She completed her Ph.D. in Human Factors and Applied Cognition from North Carolina State University in 2016 and Postdoc in the Humans and Autonomy Lab (HAL) at Duke University in 2018. She is the founding chair of the Human–AI–Robot Teaming (HART) technical group at the Human Factors and Ergonomics Society, advocating cutting-edge HART research, interdisciplinary collaboration, and advanced testbeds and analytics. She has worked on ARL, ONR, and DARPA research projects as a research lead. Dr. Huang's research interests include 1) Human–AI–Robot Teaming effectiveness; 2) Humans’ responses (i.e., emotional states, behavioral patterns, and cognitive processes) to robots and technologies, especially emotional attachment, motivation, problem–solving, coordination, and metacognition; 3) The design of human–robot systems using Human Factors methods to make AI and robots effective, safe, user-friendly, trustworthy, and engaging.
Joseph Lyons
Air Force Research Laboratory
Joseph B. Lyons is a Principal Research Psychologist within the 711 Human Performance Wing at Wright-Patterson AFB, OH. Dr. Lyons received his PhD in Industrial/Organizational Psychology from Wright State University in Dayton, OH, in 2005. Some of Dr. Lyons’ research interests include human–machine trust, interpersonal trust, human factors, and influence. Dr. Lyons has worked for the Air Force Research Laboratory as a civilian researcher since 2005, and between 2011-2013 he served as the Program Officer at the Air Force Office of Scientific Research where he created a basic research portfolio to study both interpersonal and human-machine trust as well as social influence. Dr. Lyons is an AFRL Fellow, a Fellow of the American Psychological Association and a Fellow of the Society for Military Psychologists.
Emilie Roth
Roth Cognitive Engineering
Dr. Emilie M. Roth is the owner and principal scientist of Roth Cognitive Engineering. She is a cognitive psychologist by training (Ph.D. University of Illinois, Urbana-Champaign), and has over 30 years’ experience in cognitive analysis and design in a variety of domains, including nuclear power plant operations, railroad operations, military command and control, and healthcare. She is on the advisory board of the Journal of Cognitive Engineering and Decision Making; a fellow of the Human Factors and Ergonomics Society; and recently served as a member of the Board on Human–Systems Integration at the National Academies.
Helen Wauck
Smart Information Flow Technologies
Dr. Helen Wauck has 8 years of experience in human-computer interaction research, particularly in user interface design, usability testing, and the development and evaluation of game-based spatial skill assessments. At Smart Information Flow Technologies, Dr. Wauck has worked on the design, implementation, and evaluation of a user-facing graphical IDE, immersive VR interfaces for swarm control and game-based assessment, and information analysis tools for novice to expert users under several DARPA, AFRL, and NIH research programs. She is currently leading the design and evaluation of user interface concepts for incorporating lifecycle transparency in multi-UAV human-automation teaming operations. Dr. Wauck has published and presented her work at prominent human-computer interaction conferences, including the ACM Conference on Human Factors in Computing Systems (CHI), the ACM Annual Symposium on Computer-Human Interaction in Play (CHI PLAY), and ACM Intelligent User Interfaces (IUI).
Participant Positions
Daniel Barber
SOAR Technology
Over the last decade, a significant amount of research in the area of transparency for autonomy has shown its value in improving human autonomy teaming (HAT). Attempts to define models and methods for transparency application to an existing or under-development system have helped to further guidelines and provide a means to a) quantify the amount and type of transparency information and b) align this information to human teammate mental models. Although significant in contribution, further work is needed to map these models to specific human–computer interaction methods such has graphical displays and to resolve ambiguity of transparency content across categories and mental models. In addition to these challenges, autonomy capability will or has surpassed humans in specific situations such that a human collaborator may interfere with overall team performance. In these situations, it is not possible to apply transparency directly as the time for cognitive processing will be insufficient, and as such we must look to other moments of interaction within a HAT lifecycle to apply transparency and measure if these moments are more or less effective than during primary task execution.
Eric Holder
Human Research and Engineering Directorate
Army Research Laboratory
As future technologies are conceived, designed, tested, implemented, updated, replaced, decommissioned, etc. it is critical to make sure the users and other stakeholders understand the impacts and any concerns and best practices to ensure smooth implementation and transitions. My goal in the panel is to talk about observations, best practices and concerns seen from a DoD perspective at the various stages of the life cycle in terms of transparency and understanding. Some discussion will also cover bi-directional aspects of this transparency and how the understanding of users and use cases can factor into successful outcomes.
Lixiao Huang
Arizona State University
As abundant research has focused on the end users of a system during a short window of targeted tasks, there are neglected research areas on related stakeholders of a system, such as people who make the decision to deploy or retire the system, who maintain and troubleshoot it, who train the operators on it, etc. These stakeholders all need the information to complete their tasks, and their decisions and information access impact the end operators’ perception, understanding, and usage of the system. Ho et al. (2017) used a case study of the Automatic Ground Collision Avoidance System (Auto-GCAS) to provide informative influences among managers, engineers, and testing pilots. Huang et al. (2021)’s Distributed Dynamic Team Trust (D2T2) framework explained the interpersonal relationships’ influence on human–machine relationships and how we should measure human–machine trust at the team level. However, establishing the best practices of system-related tasks and usage in terms of the right level of information needed at the right time to ensure users take the correct actions is essential yet often missing. This should determine and guide the design of making information available and the measurement of human reactions in the study of transparency-related issues. Holder et al. (2021) recommended some best practices in designing bi-directional transparency at the different stages of a system, from product concept design to field operation. Yet more detailed and refined work must be done to establish what information is needed and what action should be taken to achieve optimal outcomes for the multi-stakeholders in specific task domains. System developers, maintenance engineers, managers, experienced users, and trainers are all needed to set the correct practices of system usage and information requirements in the lifecycle of the system. The recommended best practices determine the design and evaluation of requisite information transparency. To achieve calibrated trust (Lee & See, 2004), establishing best practices and appropriate expectations must be done first, while considering distributed stakeholders’ long-term interactions with the system throughout its life cycle.
Joseph Lyons
Air Force Research Laboratory
Human–Autonomy Teaming (HAT) is an emerging domain for the Department of Defense (DoD). Research has noted that understanding intent within teaming relationships, effective communication, and having predictability in the face of uncertainty as key factors that shape HAT success. Lifecycle transparency approaches (i.e., those that attempt to shift the experience of key transparency elements to opportunities outside of the workload-saturated task context) are important for the DoD for three reasons. First, often advances in technology capability are accompanied by commensurate increases in opaqueness (as in the case with machine learning systems). Researchers must combat this opaqueness with richness in transparency opportunity. This requires deliberate efforts in design, test, and fielding for novel technologies. Second, DoD operations can be unstructured and are influenced by a variety of factors. Methods to train HATs within the military must contain richness in context to promote robust understanding and predictability of HAT behaviors, limitations, and assumptions. This necessitates moving beyond the task itself for providing transparency opportunities. Third, many military tasks are characterized by high workload coupled with high-consequence decisions. Therefore, operational task environments should not be the only opportunity for conveying transparency information.
Emilie Roth
Roth Cognitive Engineering
There is a growing body of evidence that ‘transparency’ contributes to making intelligent systems more trustworthy and effective (Chen et al., 2018). However, as Miller (2021) has argued, it may not be reasonable to expect a person to be able to process in real-time all the information that has traditionally been included under the transparency label, especially under high workload, dynamically changing conditions. While it may be possible to shift some aspects of transparency to before or after execution, there are other elements that are intended to provide
Helen Wauck
Smart Information Flow Technologies
Transparency in automation is critical for effective human-automation collaboration, yet the field of explainable AI (XAI) is replete with examples of "data dump" explanations that overwhelm the user with too much transparency information in the middle of a high tempo human-automation team operation. Lifecycle Transparency (LCT), provides a conceptual framework for moving some transparency to lower tempo phases of the operation – for example, pre-planning and debrief. However, the most basic questions about explanation presentation interfaces remain unexplored: What is the space of autonomous behavior users want to have explained? How do we select and structure explanations to give the user the information they actually want to know, given the universe of all possible technically valid explanations? What kinds of explanations are appropriate for each phase of an operation? Answering these questions requires a user-centered design approach. However, integrating transparency into the pre-planning and debrief phases of an operation poses certain unique challenges: end users are typically not used to thinking about automation transparency as a part of pre-planning or debrief phases of a human-automation teaming scenario, and pre-planning and debrief phases may be very informal and ill-defined. Overcoming these challenges to develop solutions users actually want requires more than just cursory usability evaluations during prototype development; rather, it demands involving users across the entire user-centered design lifecycle, from ethnography to contextual inquiry to prototype implementation and usability evaluation.