Let the Reader Decide

Introduction

Credit where it is due, Endsley’s original 1995 paper in Human Factors has done more to raise the general consciousness to the idea of situation awareness than any single article before or since. For that we are grateful. The world of research has not remained static, however, and its focus in human factors has shifted from the individual person to whole systems (Hutchins, 1995; Leveson, 2004; Rasmussen, 1997; Walker, Stanton, Salmon, & Jenkins, 2009a; Wilson, 2012). We think it is legitimate to raise concerns over the status quo and propose new paradigms (Salmon, Stanton, Walker, & Jenkins, 2008; Salmon, Stanton, Walker, & Jenkins, 2009; Salmon, Stanton, Walker, Jenkins, Ladva, et al., 2009; Sorensen, Stanton, & Banks, 2011; Stanton et al., 2006; Stanton, Salmon, Walker, & Jenkins, 2009a, 2009b, 2010). This is how science moves forward, but we do recognize that paradigm owners tend to resist change. We offer this commentary in the spirit of detached, calm, and reasoned academic debate. Distributed situation awareness (DSA) is presented as an alternative way of thinking about situation awareness (SA) in systems. As Hutchins (1995) advocated, the unit of analysis is not the individual person, or even teams of people (as presented with the three-level model), but the entire system under investigation. This notion has gained considerable credence within human factors, with Hollnagel (1993) even suggesting that given the complexity of modern day sociotechnical systems, the study of information processing in the minds of individuals has lost relevance.

Systems Thinking

It is easy for the research community to fall into the “fallacy” of the linear flow of the Endsley (1995) model based on the original paper. Italics have been added for emphasis:

The first step in achieving SA is to perceive the status, attributes, and dynamics of relevant elements in the environment. (Endsley, 1995, p. 36)

Comprehension of the situation is based on the synthesis of disjointed level one elements. (p. 36)

Based on knowledge of level one elements, particularly when put together to form patterns with the other elements (gestalt) the decision maker forms a holistic picture of the environment, comprehending the significance of objects and events. (Endsley, 1995, p. 37)

. . . the third and highest level of SA. This is achieved through knowledge of the status and dynamics of the elements and comprehension of the situation (both level 1 and level 2 SA). (Endsley, 1995, p. 37)

A later publication exacerbates this confusion (Endsley & Jones, 2012):

The first step in achieving SA is to perceive the status, attributes, and dynamics of relevant elements in the environment. (p. 14)

The second step in achieving good SA is understanding what the data and cues perceived mean in relation to relevant goals and objectives. Comprehension (level 2 SA) is based on a synthesis of disjointed level 1 elements. (p. 16)

And finally,

A person can only achieve level 3 SA by having a good understanding of the situation (Level 2 SA) and the functioning and dynamics of the system they are working with. (p. 18)

We agree with Endsley that there are subtleties; what we disagree about is the ability of the three-level model to cope with them (Salmon, Stanton, Walker, & Jenkins, 2009; Salmon, Stanton, Walker, Jenkins, Ladva, et al., 2009; Sorensen et al., 2011). On the one hand, Endsley (2015) puts forward seven fallacies, which it is felt other models fall into and promulgate through the literature, causing confusion. On the other hand is the much simpler idea that due to the complexity of the sociotechnical systems that form the subject of much contemporary analysis, the study of information processing in the mind of individuals has lost relevance (Hollnagel, 1993). Endsley and colleagues have made unquestionably good progress on numerous thorny psychological issues around their model, but we contend there is a much more elegant solution: Instead of looking at the information processing of a person (or persons) embedded in a situation, look instead at the interactions or transactions that take place between actors. This solution requires a shift of focus from nodes to links, which is the main essence of DSA and indeed other human factors systems thinking approaches (e.g., Rasmussen, 1997). This approach overcomes the much more fundamental fallacy that one cannot ever know completely what is going on in peoples’ minds (Dekker, 2013; Dekker, Hummerdal, & Smith, 2010). To paraphrase the “ecological” approach, one should try and look at not what is in the mind but, rather, what the minds are in. We therefore invite readers with an open mind on our journey of discovery and let them decide for themselves.

Going Out into the World: Observational Studies of Command and Control

Our interests in SA began with studies of teams in military (Stanton et al., 2006; Stanton, Salmon, et al., 2009b; Stewart et al., 2008) and civilian (Salmon, Stanton, Walker, & Jenkins, 2008; Stanton, Salmon, et al., 2009a; Walker, Stanton, Salmon, Jenkins, & Rafferty, 2010) command-and-control domains. We were faced with the task of collecting exhaustive data on how multiperson teams distributed across multiple locations performed in training and real operational settings but were unable to interfere with their tasks and certainly not allowed to interrupt their work. Thus our methods had to be nonintrusive and naturalistic. Despite initially wanting to apply the Situation Awareness Global Assessment Technique (SAGAT), these real-life demands meant that we could not use the approach and had to rely on other data collection methods, such as recording of conversations and communications (Rafferty, Stanton, & Walker, 2013) and post hoc interviews using the critical decision method (Klein, Calderwood, & McGregor, 1989). In the air traffic domain, for example, we recorded all of the communications from the controllers’ desks and video recorded their activity (Walker et al., 2010). In the naval domain, we had access to all of the voice communications in the command team as it performed training activities (Stanton et al., 2006). In the energy distribution domain, we had access to procedures, voice communications, and critical decision method interview transcripts (Salmon, Stanton, Walker, & Jenkins, 2008).

When it came to analyzing these data, it was clear we had direct access to exactly what was going on in the command-and-control teams and were able to represent the system’s awareness in its entirety using propositional networks. These networks can be used to understand the dynamics of awareness as it changes and propagates through a system. Through this analysis, the distributed nature of awareness became very apparent to us (Salmon, Stanton, Walker, & Jenkins, 2008, 2009; Salmon, Stanton, Walker, Jenkins, & Rafferty, 2010; Stanton et al., 2006; Stanton, Baber, & Harris, 2008; Stanton et al., 2010; Walker et al., 2006, 2010). It is clear that in these types of environments, each “agent in the system” has quite different goals and tasks and consequently has a very different understanding of the situations he or she was working in, even when presented with the some or all of the same data. This view offers compelling and useful insights into the distributed nature of awareness in complex sociotechnical systems.

Going Back into the Lab: Testing SAGAT

We have tried hard to be good three-level theorists in our experimental studies; we were concerned with the way in which media could be designed to keep distributed teams involved in a collaborative task (Walker, Stanton, Salmon, & Jenkins, 2009b). Different media were investigated to support the collaboration. There were four conditions: voice only (a telephone link between participants), voice and video (a live video link between participants), voice and data (an electronic shared workspace), and voice, video, and data (all three media). The participants undertook a simulated mission-planning task. At various stages in this task, the participants were stopped and were asked questions about the tasks consistent with Levels 1, 2, and 3 of Endsley’s model via the SAGAT method (the freeze, blank, and probe approach). The measure of SA had been expected to indicate better media for supporting the distributed team. In the event, SAGAT results showed best performance on the worst, voice-only condition. As the media became richer, the SAGAT scores became poorer.

Although it seems obvious now, the result was against the hypothesis that SA would be better in the media-rich condition (i.e., voice, video, and data). The explanation lies in that the greater the support from the environment, the less the person has to remember as the artifacts in the system hold the information (similar to the manner in which mobile phones hold contact numbers). In the same way that pilots use the speed bugs to remember for them (Hutchins, 1995), the participants were using the video and shared electronic workspace to remember. Similar findings are being reported in the wider literature (Sparrow, Liu, & Wegner, 2011). The awareness of the system was distributed across the agents and media, and therefore, the levels of SA held “in the heads” of participants, as scored by the SAGAT approach, were found to be poor. Only when deprived of the support were the human agents forced to remember the planning details. If this information had been taken at face value, it might have led to recommending the poorest medium for the design of the system (i.e., voice only). In the end, it was realized that it was necessary to consider the system as a whole. The sociotechnical view of DSA led to a different, and considerably richer, conclusion for system design. This theory has led us into many new domains, including road design (Walker, Stanton, & Chowdhury, 2013), evaluation of road systems and road user behavior (Salmon, Lenne, Walker, Stanton, & Filtness, 2014; Salmon, Stanton, & Young, 2012), advanced driver training (Walker, Stanton, Kazi, Salmon, & Jenkins, 2009), aviation accident investigation (Griffin, Young & Stanton, 2010), and submarine control rooms (Stanton, 2014).

DSA as the Alternative Paradigm

In the original paper specifying the DSA theory and approach, Stanton et al. (2006) indicate how the system can be viewed as a whole, by consideration of the information held by the artifacts and people and the way in which they interact. The dynamic nature of SA phenomena means they change moment by moment, in light of changes in the task, environment, and interactions (both social and technological). These changes need to be tracked in real time if the phenomena are to be understood (Patrick, James, Ahmed, & Halliday, 2006). DSA is considered to be activated knowledge for a specific task within a system at a specific time by specific agents, that is, the human and nonhuman actors in a system. Although this perspective can be challenging when viewed through a cognitive psychology lens, from a systems perspective it is not (e.g., Hutchins, 1995; Leveson, 2004; Rasmussen, 1997; Walker et al., 2009a; Wilson, 2012). Thus, one could imagine a network of information elements, linked by salience, being activated by a task and belonging to an agent—the “hive mind” of the system, if you will (Seeley et al., 2012).

To understand how this system might work, imagine a network where nodes are activated and deactivated as time passes in response to changes in the task, environment, and interactions (both social and technological). In regard to the system as a whole, it does not matter if humans or technology own this information, just that the right information is activated and passed to the right agent at the right time. This idea is founded on the theory of “transactional memory,” which involves the reliance that people have on other people (Wegner, 1986) and machines (Sparrow, Liu, & Wegner, 2011) to remember for them. It does not matter if the individual human agents do not know everything (indeed, it would be impossible for them to), provided that the system has the information, which enables the system to perform effectively (Hutchins, 1995). We know that agents are able to compensate for each other, enabling the system to maintain safe operation (i.e., there is no one best way, as described by the advocates of cognitive work analysis; see Vicente, 1999). This dynamism is impossible to model using reductionist, linear approaches. The systems thinking paradigm provides the necessary theoretical foundations and tools to explore the nonlinearity experienced in complex sociotechnical systems (Walker et al., 2010). For a more complete explanation of DSA theory and measurement, the interested reader is referred to the book by Salmon, Stanton, Walker, and Jenkins (2009).

One of the core misconceptions expressed by Endsley (2015) is that the DSA approach has no accompanying methodology to support the design of systems or to undertake analyses of DSA in the wild. This assumption is incorrect. The Event Analysis of Systemic Teamwork methodology (see Stanton et al., 2013), which incorporates the propositional network approach to describe DSA, has been applied proactively to model DSA across different system design concepts (Baber, Stanton, Atkinson, McMaster, & Houghton, 2013) and used to assess DSA in all manner of complex naturalistic settings (e.g., Salmon, Stanton, Walker, Jenkins, Baber, et al., 2008; Stanton, 2014; Walker et al., 2013). In addition, the authors have used EAST to generate DSA requirements specifications in systems design. For example, this approach was used by the authors to examine DSA during a large-scale U.K. Army field trial of a new £2.4 billion mission-planning and battlespace management system (Stanton, Walker, et al., 2009). For our purposes, the DSA approach was the most appropriate methodology that could be applied to assess SA in this complex naturalistic setting because of the dynamic nature of the activities. It would be impossible to “script” the system activities and metrics ahead of time, as the planning and operations teams had to adapt to the changing nature of the environment. Based on live observations, the DSA analysis identified design issues adversely affecting system performance (Salmon, Stanton, Walker, Jenkins, Ladva, et al., 2009). The outputs were used to generate explicit system redesign recommendations (Stanton, Walker, et al., 2009) that have been subsequently implemented. Consequential improvements in system performance were observed.

The relationship between SA and task performance has remained resolutely difficult to prove, with some research both proving and falsifying the link, even within the same study (Endsley, 1995), which begs the question, why bother with SA if it is not revealing anything about how teams actually perform on tasks? The systems view of SA is not as equivocal. We have conducted experimental research into the conversations teams have when performing tasks and found a very strong positive relationship between DSA and the teams’ performance on the task (Sorensen & Stanton, 2013). We have also shown the same effect in high-fidelity, predeployment training environments (Rafferty et al., 2013). DSA, therefore, does tell us how teams actually perform, making SA as a concept more, rather than less, useful. This is a key insight that has been supported by the research of others (Bleakley, Allard, & Hobbs, 2013; Golightly, Ryan, Dadashi, Pickup, & Wilson, 2013; Patrick & Morgan, 2010).

Revolutionary Newcomer

In science and engineering, there is a natural rise and fall of paradigms as progress is made. In this way, a new paradigm becomes more established until it gives way to new developments. We contend that DSA presents a new paradigm for analyzing and explaining SA in systems, and there is a groundswell of studies that are tipping the balance of evidence in that direction (Bourbousson, Poizat, Saury, & Seve, 2011; Fioratou, Flin, Glavin, & Patey, 2010; Golightly et al., 2013; Golightly, Wilson, Lowe, & Sharples, 2010; Macquet & Stanton, 2014; Patrick & Morgan, 2010; and others). We do not expect the debate to end here, but we do encourage readers to approach all of the ideas with an open mind, try out the approaches, and decide for themselves (e.g., Haavik, 2011; Schulz, Endsley, Kochs, Gelb, & Wagner, 2013).