Combining Cognitive Task Analysis and Participatory Design Methods to Elicit and Represent Task Flows

Abstract

Representations of step-by-step procedures, such as task flows, are developed and used to support technology design and evaluation as well as for training purposes in complex work domains. However, task flows may not represent how people carry out work under uncertainty, time pressure, or high-risk conditions. We combined methods of cognitive task analysis and participatory design, resulting in a new approach for developing task flows. This approach accounts for both cognitive and behavioral work and explicitly represents its dynamic nature. Additional advantages of this approach include flexibility and adaptiveness to help overcome challenges of conducting research in real-world domains, including time constraints and access to subject matter experts. We demonstrate this approach in the context of developing a task flow for a submarine watch team’s use of an updated imaging system to maintain the ship’s safety by forming and maintaining a picture of the external environment. We provide a detailed description of each phase as well as a domain-neutral ready-to-use job aid.

Keywords

cognitive task analysis design methods team design cognitive systems engineering participatory design task flow knowledge elicitation

Background

Representations of step-by-step procedures are tools that systems engineers and work domain practitioners rely on to inform development and design of training, workflows, technology, interfaces, and software, as well as to support evaluation. A variety of terms are used for work representations including task flows, task diagrams, flow charts, algorithms, checklists, decision ladders, as well as multiple types of task analysis outputs (summarized in Crystal & Ellington, 2004). As guidelines, task flows may prescribe or guide what behavior should look like via a representation of a sequence of steps and decision points. However, as is frequently the case with guidelines, these steps may be developed with little to no examination of how people carry out their work in real-world complex domains under uncertainty, time pressure, or high-risk conditions. Task flows may represent behavior without consideration for the cognitive, perceptual, and communication processes underlying complex decision making. Also, they may fail to represent the contextual constraints and facilitators that play a role in performance, including other people and technologies. Some exceptions can be found in the development of work representations that account for these factors such as Rasmussen’s decision ladder, an approach from the Cognitive Work Analysis (CWA) tradition, aimed at eliciting decision making processes (Rasmussen & Rouse, 2013; Vicente, 1999).

Decision ladders have been applied by human factors scientists across domains to represent a sequence of activities underlying task performance. For instance, Bisantz et al. (2003) describe the application of decision ladders to represent the workflow of sonar operators on a navy ship. The authors highlight the variability in activities as a function of the operator and situation. Similarly, research modeling the work of cyber protection teams, illustrates the results of workflow examination via an iterative case study using CWA (Trent et al., 2019). Trent et al. (2019) developed a cognitive workflow model that deepens beyond traditional workflows. They note that the effort departs from doctrinal tasks and offers a descriptive representation of many layers of parallel and looping tasks, beyond a step-wise representation that fails to capture complexity of the dynamic, team-based execution of work. The Trent et al. (2019) study used interview-based methods to present interim versions of the work model, similar to the approach of this study. CWA methods and decision ladders are tools for representing the outputs of knowledge elicitation techniques, rather than specifications of how to elicit knowledge. Bisantz and Roth (2007) highlighted that the analysis of cognitive work entails both elicitation and representation methods. We introduce a new, pragmatic approach of knowledge elicitation that leverages asynchronous and synchronous techniques and participatory design methods.

Human factors professionals are frequently called upon to support the development of task flows. Drawing upon experiences in multiple complex sociotechnical domains (e.g., submarine, aviation, military command and control, and healthcare), we present a novel approach for developing task flows. Through an integration of cognitive task analysis (CTA; Crandall et al., 2006) and participatory design (PD; Sanders, 2002) traditions reviewed below, this approach triangulates across multiple participants and roles to elicit and represent an operationally authentic sequence of steps, along with cognitive and behavioral performance challenges and facilitators. In addition to the novel integration of CTA and PD, this approach offers a way to overcome the challenges inherent in eliciting subject matter expert (SME) input in real-world domains. These challenges include limited opportunities to elicit knowledge from SMEs (individually and/or in groups), the mission-driven mandate to be considerate of their time constraints. By introducing strategies for overcoming constraints of conducting real-world research and by combining methods from two different traditions, our approach increases mission impact while enabling efficient and flexible knowledge elicitation. The result is a process for real-time evolution of representations with SMEs as opportunities arise over the course of a development effort. We suggest this approach is applicable to other domains where access to SMEs may be limited due to time and logistical constraints. In the current paper, we provide a description of a detailed approach based on multiple knowledge elicitations from SMEs. However, as long as the resulting task flow is informed by the contribution of multiple SMEs, as little or as much data collection as needed or possible can be conducted. The dynamic nature of this approach allows the SMEs to make each contribution efficiently and opportunistically (when situation permits).

Domain Description

This approach was used in developing a task flow for a submarine watch team’s use of imaging technology, henceforth referred to as the imaging task flow. Submarine watch teams, comprised of watch standers, use imaging systems to maintain the ship’s safety by forming and maintaining the contact picture (a representation of the external environment including collision threats), both while on the surface and at periscope depth (Defense Systems, 2016; In-Depth Engineering, 2014). At periscope depth, the imaging system is essential to a ship’s safety as it is the only means of discerning visual contacts. New imaging technology (called Integrated Submarine Imaging System or ISIS) is replacing digital and visual imaging for submarine periscopes, providing high-resolution imagery to support effective watch team performance. The process presented here was developed and used during an effort to redesign the user interface for a submarine watch team’s use of new imaging technology; the resulting task flow formed the basis for software developers’ work. It is often the case that due to domain and data sensitivity, research conducted in the real world may not be disseminated in scientific outlets, thereby prohibiting lessons learned and sharing of adapted or novel approaches. Due to the nature of the submarine domain in which we worked, we are limited to sharing descriptions of the approach without delving into detail on content and findings.

Combination of Cognitive Task Analysis and Participatory Design Methods

Cognitive Task Analysis

CTA methods are used to elicit experts’ tacit knowledge in complex domains (Crandall et al., 2006; Hoffman et al., 1998). CTA methods primarily grew out of human factors, cognitive psychology, cognitive science, and systems engineering traditions (see Roth, 2008, for a discussion). SMEs are often unable to articulate their cognitive processes (Nisbett & Wilson, 1977). CTA methods attempt to overcome this challenge by using incident-based elicitation techniques. These methods enable understanding of problem spaces and inform products such as training, interface design concepts, and physical space layouts. A rich literature supports the use of CTA to understand high-risk, dynamic, real-world environments by applying methods flexibly and adaptively as the constraints of the domain or the research effort may require. One obstacle of CTA methods is the level of experience on the part of the researcher needed to carry out knowledge elicitation and analysis as well as the time commitment needed on the part of both the researcher and the participants. In 1998, Militello and Hutton (1998) streamlined CTA to create an Applied Cognitive Task Analysis (ACTA) toolkit accessible to researchers and industry practitioners alike, including predetermined questions and probes and efficient procedures for using them. Within this toolkit, the task diagram and knowledge audit methods support the elicitation of a sequence of steps comprising a task of interest along with example incidents to characterize steps that are particularly challenging (e.g., for imaging, they may include detecting and classifying contacts in nighttime conditions). In the current paper, we adopt these two ACTA approaches, in addition to elements of PD, to inform development of a task flow that captures cognitive work as underlying behavioral tasks, but impose limited time demand on SMEs.

Participatory Design

PD (also synonymous with cooperative design, co-design and co-creation) refers to a multitude of strategies and techniques that include the users of a system in the design, implementation, and/or evaluation of the new system (Bødker, 1996). PD methods enable knowledge elicitation and ideation through visual creation methods (Sanders, 2005) by actively including SMEs in the design of an artifact (product, interface, or system). The goal of these methods is to create artifacts that are useful, usable, and desirable to the end user (Sanders, 1992, 2002). PD can serve as an elicitation tool for past, present, and future aspects of work. Much of the value of PD lies in priming and eliciting SMEs lived experiences with the goal of supporting them in their future work. The advantage of this approach is that the SMEs’ tacit knowledge (including goals, challenges, and constraints) is elicited and explored jointly with design professionals, who co-create graphical descriptions of the tool capabilities and solutions. This immediate feedback, which often manifests itself as in-depth conversation, eliminates the need for time-consuming cycles of design, code, and evaluation. A skilled facilitator of PD empowers SMEs to create solutions that will work in their world. PD’s primary benefit is its ability to gather multiple solution alternatives early in the design process, quickly, and with less cost. Research has suggested that designers create more innovative concepts and ideas when working within a co-design environment with others than they do when creating ideas on their own (Asaro, 2000; Banathy, 1993).

Incorporating multiple users’, developers’, and designers’ direct participation in design is a tenet of the design thinking approach that emerged in the 1970s (Sanders & Stappers, 2008). Our team included one member with extensive experience in using these methods. Inspired by the benefits of engaging multiple SMEs concurrently and using tools such as post-its and prototyping supplies (Dominguez et al., 2015; Papautsky, Dominguez, et al., 2015), the authors adopted these methods of visual creation for knowledge elicitation and sharing of a knowledge representation to inform task flow development. Empirical research suggests that group sessions are effective at bringing together multiple knowledge bases and building on one another’s contributions (Paulus & Brown, 2007).

Using CTA to inform design concepts is not a novel approach, but one that is gaining traction. In a 2001 study, CTA was used to seed design concepts in the intelligence analysis domain (Patterson et al., 2001). More recently, Rayo et al. (2018) applied a combination of CTA and PD to study decision making of physicians ordering diagnostic imaging. CTA and PD are complementary methods to elicit expert and practitioner knowledge. Their objectives may differ in that CTA is implemented to elicit tacit knowledge from experts and PD aims to generate a diverse set of solutions, but the latter naturally captures the practical outcomes of the former in a concrete conceptual product. In a complementary manner, human factors literature reports on the use of visual artifacts as knowledge elicitation tools to facilitate triangulation of knowledge on an individual basis (Papautsky et al., 2015). Graell-Colas et al. (n.d.) explain the need for shared understanding and agreement on potential solutions among team members, highlighting the benefit of conducting knowledge elicitation in a casual group setting. We propose that by building flexibility into the task flow development process, through creating moveable or dynamic pieces (boxes and arrows) and fostering discussion among SMEs, we can leverage domain expertise to capture rich contextual information resulting from lived experiences around each step. The integration of the two, for research and design purposes, yields a novel approach to task flow development that we call a dynamic task flow approach. We envision it to be applicable across multiple work domains, including healthcare for the development of stepwise procedures like clinical algorithms and checklists. The following steps are illustrated by describing the application of the process (not the content) for an imaging task flow, while grounding each step in theory and potential applications:

Identify steps, their sequence, and challenges

Create the first iteration of the task flow

Conduct iterative rapid triangulation

Unpack challenges (decision points) through individual interviews

Create task flow overlays

Dynamic Task Flow Approach

Participants

As is consistent with other CTA methods, the dynamic task flow approach relies upon SMEs’ experience and understanding of practice across multiple situations within their domain. This approach is intended to be used with experienced participants. We developed the imaging task flow by eliciting knowledge from experienced (having at least 2 years of experience) submarine watch standers, from both active duty and retired populations. Researchers, designers, engineers, and project sponsors also participated in the process, but only watch standers contributed to the task flow with task-specific knowledge. All watch stander participants were male.

Phase 1: Identify Steps, Their Sequence, and Challenges

To overcome challenges of SME access and efficiently identify tasks, sequences, and challenges, we used the task diagram interview as detailed in Militello and Hutton (1998), tailored to be administered asynchronously via email if needed. The objective behind the task diagram interview is to elicit an overview of the tasks of interest and identify the steps that are cognitively and/or perceptually challenging. This overview is used to guide subsequent CTA interviews by revealing which steps the researcher needs to probe with questions to unpack the challenges. Although the questions can be asked of SMEs in face-to-face interviews as traditional in CTA, this may not be feasible due to time constraints. Another alternative is to administer questionnaires or interviews through email. Literature suggests that email interviews offer benefits that include access to geographically remote or distant populations and opportunities for participants to think through their responses on a longer time scale (James, 2016; McCoyd & Kerson, 2006). In the nursing domain, Hershberger and Kavanaugh (2017) conducted a study comparing phone and email interviews. Authors found that email interviews yielded insightful and rich data (2017). Participant responses are used to inform the critical steps that make up the task flow and the order of their execution, forming the basis for the initial version of the task flow. A unique contribution of this research’s approach is administration of the ACTA task diagram step via email.

For the imaging task flow, we asked five watch standers to individually (1) identify the overall goal of the imaging task, (2) list and describe the steps associated with accomplishing this goal, and (3) identify which of the steps are particularly challenging. Using Microsoft Word or PowerPoint, each participant created his own representation of his work as a watch stander comprised of boxes and arrows, with the challenging steps identified with an asterisk. Unlike in the ACTA approach, we did not limit respondents to three to six steps as we wanted to elicit an overview of all the steps they found relevant. All responses suggested that the overall goal was to ensure the ship’s safety through building, interrogating, and maintaining the contact picture, with challenges including the integration of information across multiple sensors. Although email administration of the task diagram precluded the back-and-forth deepening that face-to-face interviews afford, we elicited the initial version of the task flow to be deepened, expanded, and revised in subsequent phases, while mitigating challenges associated scheduling and logistics. To mitigate validity issues associated with email administration, we elicited initial task flows from multiple respondents and conducted multiple iterations of the task flow with additional SMEs, individually and in groups in subsequent phases. At the completion of Phase 1, we had responses from five watch standers to inform the first iteration of the task flow.

Phase 2: Create the First Iteration of the Task Flow

To create a flexible and dynamic representation, this phase entails integrating the responses from the previous phase and creating the first iteration of the task flow in a digital format that is large enough to be printed, posted, and manipulated on a wall. Every element depicted by each individual respondent is consolidated into this initial iteration to maintain the variability across respondents. In other words, all SME contributions are maintained in the initial iteration. This idea is an adaptation of PD with application in CTA-type knowledge elicitation. This phase is not intended to be an analysis of the data collected so far, but rather a starting point for the creation of an integrated representation of these data to be used and iterated upon in the next phase of the process. This common starting point, a digital all-encompassing representative task flow that lends itself to be easily updated in real-time via group facilitation, is a key beginning to the process.

For the submarine watch stander ISIS application, this all-encompassing task flow was created based on all of the watch standers’ responses. Using this amalgamated task flow as a guide, our team created laminated cards for each of the subtasks along with arrows represented by the SMEs in their individual task flows. The laminated cards and arrow enabled this now wall-sized task flow to be configured and reconfigured by an individual or a group in a dynamic manner on a large conference room wall. Additionally, we also included blank laminated cards and extra arrows to allow participants to generate additional steps. To retain the variabilities that existed across the respondents, we made cards for each step generated by SMEs in Phase 1. Figure 1 is a photograph of the laminated task flow components pile. At this point, the research team also created a digital task flow representation (using Microsoft PowerPoint) to be updated by the research team as a function of the dynamic task flow process. At the completion of Phase 2, we had an initial iteration of the task flow both in digital and analog formats.

Figure 1

Laminated task flow components.

Phase 3: Conduct Iterative Rapid Triangulation

Leveraging a modified PD approach that is efficient and flexible, this triangulation phase involves eliciting knowledge from multiple SMEs in a dynamic but systematic manner. This can be done concurrently by multiple SMEs or individually. The objective of this step is to create a prototype task flow and let it be challenged and refined in subsequent steps by other participants. This step enables triangulating SME perspectives toward a task flow representing multiple perspectives. This step ideally combines PD methods and a cognitive wall walk method, along with the use of artifacts as knowledge elicitation tools. A wall walk is a process of vetting information with SMEs. It involves putting a prototype’s components or features on a wall and guiding a discussion of how a user would accomplish tasks with this prototype, often with a use case scenario (Dominguez et al., 2011). A group of SMEs and the research team (or member) may be co-located in a large meeting space. The most recent version of the task flow is depicted on a large wall that every member of the group can see. A scenario, germane to the SMEs, is read aloud. As the group walks through a scenario as a collective, SMEs are encouraged to speak to each component. Often, SMEs will dominate the conversation for 10 min or more, as they negotiate the nature, priority, and structure of their task in that moment. During these discussions, the research team captures data in the form of notes and photographs. When the team reaches consensus, a member of the research team makes modifications to the large task flow on the wall and records with a photograph. Data are captured by the researchers and differences across individual operators are aggregated and integrated into a new task flow that represents these expert perspectives. In sum, this step is a researcher-guided individual or group real-time revision process combining concurrent knowledge elicitation (group wall walk CTA approach) and PD-style representation of a task flow that captures the dynamic and adaptable aspects of work.

For the imaging task flow, the research team facilitated approximately nine interactive sessions to iterate on the task flow, with about 4 weeks between them. These sessions ranged in duration from a few minutes to longer, depending on SME time constraints. Specifically, we leveraged opportunities at meetings, design, and testing events where participants were present and available to contribute to the task flow. For each wall walk, we arranged laminated task flow cards and arrows on a foam board (Figure 2), placed the foam board in front of the room, and facilitated a group discussion regarding the current version of the task flow (which represented domain performance knowledge elicited to date), encouraging mental simulations of the imaging task on both the individual and group levels. We started the discussion with the statement: “We asked you a lot of questions about your work and this is a representation [dynamic task flow on a foam board] of what we think we heard. All of these pieces are moveable – let’s think and revise as needed.” As a function of the discussion, we (1) moved, took away, and added cards (steps, arrows), (2) revised organization and sequence, and (3) added comments, questions, and descriptions on sticky notes. As the dynamic task flow for ISIS evolved, each change provided an opportunity for the rest of the group to ask questions, challenge assumptions, or contribute, thereby enriching the understanding of the group about the overall task. We also encouraged discussion among participants that involved negotiating discrepancies. In situations where discrepancies emerged, we facilitated guided discussion to reach consensus (based on seniority, level of experience, and role). If consensus was not reached, we flagged the components under discussion with a sticky note to be brought up in subsequent sessions and/or in individual interviews. We then photographed the modified task flow, along with documenting any details captured on the sticky notes, and revised the digital file task flow accordingly. For the subsequent wall walk, we presented an updated version of the dynamic task flow and repeated the process just described. The evolution of the task flow structure throughout this process is represented in Figure 3 (intentionally illegible). As a function of iteration through knowledge elicitation, later versions of the task flow comprised more steps (and descriptions) as we unpacked challenging aspects of the work (Phase 4). At the completion of Phase 3, we had an updated version of the task flow informed by multiple SMEs across approximately nine knowledge elicitation sessions.

Figure 2

(a) Laminated imaging task flow components pinned on a board before rapid triangulation. (b) Laminated imaging task flow components pinned on a board following rapid triangulation.

Figure 3

Multiple versions of the imaging task flow, demonstrating how additional task components and relationships are overlaid onto the task flow with progressive iterations. Due to domain sensitivity, the current graphic is intentionally illegible. ISIS = Integrated Submarine Imaging System.

Phase 4: Unpack Challenges (Decision Points) Through Individual Interviews

Phase 4 entails conducting targeted CTA interviews (Crandall et al., 2006), with the objective of unpacking steps or decision points that were identified to be challenging in previous phases by eliciting examples and incidents. In the interest of efficiency, the interviewer may focus on an individual step per participant rather than eliciting more detailed incidents associated with the entire task as one would when conducting traditional CTA (Crandall et al., 2006). Similarly, the focus should be on unpacking the cognitive and perceptual cues associated with performing each step. Given that each step may vary in complexity, it may take more or less time to unpack. In this method, unpacking individual steps supports a deep examination of these steps in turn.

For the imaging task flow, we interviewed nine officers (including several senior officers). We did some interviews ad hoc, as sidebars to existing meetings, and others scheduled separately. Mindful of time limitations constraining these interviews, we asked participants about one or several steps (rather than all of them). For instance, we would begin these interviews with the following prompt: “We understand that the step of detecting contacts might be challenging during poor weather due to visibility. Can you please describe how you go about detecting contacts in poor weather conditions, and what the main difficulties are?” This enabled us to gain an understanding of the cognitive and perceptual challenges of the imaging task, an aspect that is typically missing from behavioral task flows. We captured interview notes detailing why steps are challenging and ways of overcoming the challenges, along with incidents or examples elicited from each officer. As part of the analysis, two research team members developed concise summary statements of challenges and examples that we used to tag each challenging step in the task flow with supporting documentation. We also used the individual interviews as opportunities to vet iterations of the task flow. At the completion of Phase 4, we had an annotated task flow.

Phase 5: Create Task Flow Overlays

Again, this phase leverages modified CTA approach by targeting specific aspects of work. The final phase was focused on enriching the static task flow with dynamic aspects of work by creating overlays focused on different aspects of the task. Whereas, the previous phase describes how to conduct CTA interviews to understand operational context and challenges behind the task flow; this phase describes overlays, with the primary being cognitive. Two other useful overlays are part of this approach. The first concerns people and communication, capturing the team coordination and information sharing element of dynamic work. The second concerns capturing automation or other advanced technologies that are currently used or that would make work more efficient and effective if applied.

Cognitive Overlay: identified and tagged steps with cognitive and perceptual challenges based on the following probes:

Which steps are challenging?

What are examples of these challenges from your lived experiences?

[For each challenging step] How do you overcome these challenges?

People and Communication Overlay - identified and tagged each step based on the following probes:

Who do you seek information from?

Who do you provide information to?

Who supports you in other ways (besides information) and at which steps?

Automation Overlay: identified and tagged each step based on the following probes:

For each step, what technology or other support tools do you use and for what purposes?

How might automation, artificial intelligence, or other software innovations improve your success at this step?

In summary, including the above content overlaid onto a task flow is a new way to enable understanding and represention of tasks in complex real-world domains. At the completion of Phase 5, we had multiple task flow overlays as described above.

Resulting Dynamic Task Flow Package

By employing an integrated CTA and PD approach that is dynamic, we have extended the traditional task flow to one that represents multiple aspects of performance in addition to behavioral. Figure 4 provides an overview in the form of a job aid of the dynamic task flow approach that can be used by a researcher or a practitioner to carry out a similar process.

Figure 4

Overview of the task flow development process.

The process demonstrates flexibility, and potential efficiency and generalizability of the approach. The final imaging task flow package included a PowerPoint slide deck with the following task flow representations:

Behavioral task flow: included behavioral steps only

Behavioral task flow with a cognitive overlay: relevant steps were tagged as challenging with concise summary statements of the rationale

Behavioral task flow with people/communications overlay: each step was tagged with the roles of individuals involved in information exchange (if any)

Behavioral task flow with technology overlay: each step was tagged with whether and how technology was used

In the current paper, we focused on the dynamic task flow approach. However, the main objective of the use case described was to represent the imaging process as it actually occurs in practice to guide the development and evaluation of technology to support watch standers in keeping the ship safe from collision threats. There are two main ways in which the final task flow was used by the broader ISIS team we supported:

To ensure that each feature of the prototype technology supported each step. Specifically, we linked task flow steps to features of the technology intended to support these steps. Making the cognitive and behavioral tasks and challenges explicit, enabled evaluation of the technology provided to support them.

To assess current operational guidance with the purpose of identifying and addressing gaps and discrepancies as compared to elicited task flow steps.

Conclusion

The application of the described approach, a combination and adaptation of CTA and PD, can yield a task flow that reflects an authentic, fully-described perspective of how cognitive and behavioral work occurs. This approach can support research, development, design, and evaluation of solutions ranging from training to technology, as well as guideline enforcement. The main contribution of our approach to this application is that it afforded flexibility (ad hoc manner) and adaptiveness to collect as little or as much data as feasible and/or needed and have a version of a task flow representation at each phase. We leveraged both the strengths of CTA by eliciting SMEs knowledge and PD by developing the task flow in a collaborative and creative environment. In addition, each participant interaction can be efficient and the data collection can be scaled up or down (number of interactions and time with each SME) based on what the program requires or allows. In the current example, we conducted a more thorough process because our role was championed by program managers. We realize that not all programs afford this level of interaction with SMEs. Additional value of the described approach lies in side-stepping data collection challenges in work domains where SMEs may have little to no availability to participate in lengthy interviews. This process has the potential to be generalizable across any complex domains where access to SMEs’ time presents a challenge. One example of such a domain is healthcare given it is currently receiving a great deal of research attention to address workflows as relevant to patient safety, with human factors perspective and methods being recognized as critical. Given the ubiquitous use of task flows (e.g., clinical algorithms) within healthcare to train and guide the activities of clinicians and staff, we propose that the dynamic task flow approach is particularly applicable to the development and evaluation of clinical checklists, algorithms, and workflows.

One of the limitations of the described approach includes the need to have an internal champion to facilitate the logistics of SME interactions. In addition, a portion of our approach was time-consuming, such as creation of overlays. This was an optional phase that we pursued to capture a thorough representation of work as conducted. We did not have an opportunity to formally evaluate the quality of the resulting task flow. This limitation was due to the sensitive nature of the domain. However, we received positive sponsor feedback regarding its usefulness, value, and application in both, developing a new system, as well as evaluating pre-existing processes.

Authors’ Note

This manuscript was conceived under the authors’ current affiliations. The described process was applied under an effort conducted when all authors’ affiliation was Cognitive Solutions Division of Applied Research Associates, Inc. Cognitive Solutions Division is no longer in existence.

Footnotes

Acknowledgments

We appreciate the team leadership and guidance of Josh Smith, the contributions of multiple subject matter experts, and all of the collaborators on the Johns Hopkins University Applied Physics Lab and contractor team who contributed ideas and refinements to this process. This work was supported by PEO IWS 5A (Prime contract: N00024-03-D-6606; Subcontract: 111332 to Johns Hopkins Applied Physics Lab) awarded to the Cognitive Solutions Division of Applied Research Associates, Inc.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Statement A: Approved for Release; Distribution is unlimited – by NAVSEA Office of Corporate Communications. The third author’s affiliation with The MITRE Corporation is provided for identification purposes only and is not intended to convey or imply MITRE’s concurrence with, or support for, the positions, opinions or viewpoints expressed by the author.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Elizabeth Lerner Papautsky

Author Biographies

Elizabeth Lerner Papautsky, University of Illinois at Chicago, PhD (Human Factors Psychology, 2009, Wright State University).

Robert Strouse, The Ohio State University College of Medicine, MFA (Design, 2010, The Ohio State University)

Cindy Dominguez, The MITRE Corporation, PhD (Human Factors Psychology, 1997, Wright State University)

References

Asaro

P. M

. (2000). Transforming society by transforming technology: The science and politics of participatory design. Accounting, Management and Information Technologies, 10(4), 257–290.doi:10.1016/S0959-8022(00)00004-7

Banathy

B. H

. (1993). Comprehensive systems design in education: Selecting the scope of design inquiry. Educational Technology, 33(4), 33–35.

Bisantz

Roth

. (2007). Analysis of cognitive work. Reviews of Human Factors and Ergonomics, 3(1), 1–43.doi:10.1518/155723408X299825

Bisantz

A. M.

Roth

Brickman

Gosbee

L. L.

Hettinger

McKinney

. (2003). Integrating cognitive analyses in a large-scale system design process. International Journal of Human-Computer Studies, 58(2), 177–206.doi:10.1016/S1071-5819(02)00130-1

Bødker

. (1996). Creating conditions for participation: Conflicts and resources in systems development. Human–Computer Interaction, 11(3), 215–236.doi:10.1207/s15327051hci1103_2

Crandall

Klein

G. A.

Hoffman

R. R

. (2006). Working minds: A practitioner’s guide to cognitive task analysis. Mit Press.

Crystal

Ellington

. (2004). Task analysis and human-computer interaction: Approaches, techniques, and levels of analysis. AMCIS 2004 Proceedings, New York. 391.

Defense Systems . (2016). Navy adds to digital imaging deal for submarines. Defense Systems . https://defensesystems.com/articles/2016/08/25/navy-lockheed-submarine-periscope-imagery.aspx

Dominguez

Grome

Strouse

Crandall

Nemeth

O’Connor

. (2011, April). Linking cognitive data to design in US Navy command and control [Symposium]. Proceedings of the 13th Annual International Symposium. International Council on Systems Engineering., Denver (Also Invited Paper in INCOSE Special Session of IEEE Systems Symposium, Montreal.

10.

Dominguez

Strouse

Papautsky

Moon

. (2015). Cognitive design of an APP enabling remote bases to receive unmanned helicopter resupply. Journal of Human-Robot Interaction, 4(2), 50–60.doi:10.5898/JHRI.4.2.Dominguez

11.

Graell-Colas

Bernal

Sanders

E. B.-N.

Gill

Blake

Patterson

. (n.d.). Designing the visual palette: Emerging concepts in macrocognitive processes through visual generative techniques.

12.

Hershberger

P. E.

Kavanaugh

. (2017). Comparing appropriateness and equivalence of email interviews to phone interviews in qualitative research on reproductive decisions. Applied Nursing Research, 37, 50–54.doi:10.1016/j.apnr.2017.07.005

http://www.ncbi.nlm.nih.gov/pubmed/28985920

13.

Hoffman

R. R.

Crandall

Shadbolt

. (1998). Use of the critical decision method to elicit expert knowledge: A case study in the methodology of cognitive task analysis. Human Factors, 40(2), 254–276.doi:10.1518/001872098779480442

14.

In-Depth Engineering . (2014). AN/BVY-1 Integrated Submarine Imaging System (ISIS). In-Depth Engineering. http://www.indepth.com/programs04.html

15.

James

. (2016). Using email interviews in qualitative educational research: Creating space to think and time to talk. International Journal of Qualitative Studies in Education, 29(2), 150–163.doi:10.1080/09518398.2015.1017848

16.

McCoyd

J. L. M.

Kerson

T. S

. (2006). Conducting intensive interviews using email: A serendipitous comparative opportunity. Qualitative Social Work, 5(3), 389–406.

17.

Militello

L. G.

Hutton

R. J

. (1998). Applied cognitive task analysis (ACTA): A practitioner’s toolkit for understanding cognitive task demands. Ergonomics, 41(11), 1618–1641.doi:10.1080/001401398186108

http://www.ncbi.nlm.nih.gov/pubmed/9819578

18.

Nisbett

R. E.

Wilson

T. D

. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259.doi:10.1037/0033-295X.84.3.231

19.

Papautsky

E. L.

Crandall

Grome

Greenberg

J. M

. (2015). A case study of source triangulation: Using artifacts as knowledge elicitation tools in healthcare space design. Journal of Cognitive Engineering and Decision Making, 9(4), 347–358.

20.

Papautsky

E. L.

Dominguez

Strouse

Moon

. (2015). Integration of cognitive task analysis and design thinking for autonomous helicopter displays. Journal of Cognitive Engineering and Decision Making, 9(4), 283–294.doi:10.1177/1555343415602624

21.

Patterson

E. S.

Woods

D. D.

Tinapple

Roth

E. M

. (2001). Using cognitive task analysis (CTA) to seed design concepts for intelligence analysts under data overload. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 45(4), 439–443.doi:10.1177/154193120104500437

22.

Paulus

P. B.

Brown

V. R

. (2007). Toward more creative and innovative group idea generation: A cognitive-social-motivational perspective of brainstorming. Social and Personality Psychology Compass, 1(1), 248–265.doi:10.1111/j.1751-9004.2007.00006.x

23.

Rasmussen

Rouse

W. B

. (2013). Human detection and diagnosis of system failures. Springer Science & Business Media.

24.

Rayo

M. F.

Pawar

Sanders

E. B.-N.

Liston

B. W.

Patterson

E. S

. (2018). Participatory bullseye toolkit interview: Identifying physicians’ relative prioritization of decision factors when ordering radiologic imaging in a hospital setting. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, 7(1), 1–7.doi:10.1177/2327857918071001

25.

Roth

E. M

. (2008). Uncovering the requirements of cognitive work. Human Factors, 50(3), 475–480.doi:10.1518/001872008X288556

http://www.ncbi.nlm.nih.gov/pubmed/18689056

26.

Sanders

E. B.-N

. (1992). Converging perspectives: Product development research for the 1990s. Design Management Review, 3(4), 49–54.

27.

Sanders

E. B. -N

. (2002). From user-centered to participatory design approaches. In Frascara

(Ed.), Design and the social sciences (pp. 18–25). CRC Press.

28.

Sanders

E. B. -N

. (2005). Information, inspiration and co-creation. [Conference session]. Proceedings of the 6th International Conference of the European Academy of Design, University of the Arts, Bremen, Germany. 29-31 March 2005.

29.

Sanders

E. B.-N.

Stappers

P. J

. (2008). Co-creation and the new landscapes of design. CoDesign, 4(1), 5–18.doi:10.1080/15710880701875068

30.

Trent

Hoffman

R. R.

Merritt

Smith

. (2019). Modelling the cognitive work of cyber protection teams. The Cyber Defense Review, 4(1), 125–136.

31.

Vicente

K. J

. (1999). Cognitive work analysis: Toward safe, productive, and healthy computer-based work. CRC Press.