Workload and Performance: Associations,Insensitivities,and Dissociations

Introduction

One of the more intriguing and important questions in the whole science of human factors/ergonomics (HF/E) concerns the role and utility of the subjective reactions of systems operators and systems users. Some have argued that it is only these affective dimensions that distinguish HF/E from pure engineering (Hancock, Weaver, & Parasuraman, 2002). As an accomplished psychophysicist (e.g., Halpern & Warm, 1980), professor Joel Warm was acutely aware of the complexities of the relationship between subjective experience and objective reality. His well-known work on vigilance was most influential in the identification of associations between the observer’s subjective state and objective measures of loss of alertness. For example, the vigilance decrement in performance is accompanied by both subjective fatigue and stress (Warm, Matthews, & Finomore, 2008) as well as changes in neurophysiological functioning (Warm, Tripp, Matthews, & Helton, 2012). But Warm’s vigilance studies also identified instances in which subjective and objective responses diverged. For example, a manipulation of temporal perception induced a subjective sense of time flying but had no corresponding impact on task performance (Finomore et al., 2016). In the present article, we examine another of the constructs central to Warm’s vigilance theory (Warm & Dember, 1998): mental workload. We examine the relationships between primary-task performance, subjective report, and neuroergonomic/neurophysiological measures of mental workload apparent in vigilance as well as the many other tasks facing the working individual.

If one adopts a radically behavioristic perspective, one might argue that effective and predictive modeling leaves such subjective experiences as interesting phenomenological by-products of the utilitarian process but not absolutely necessary to the practical enterprise of human–machine systems operations. In the present work we examine such a proposition, framed primarily in the domain of cognitive workload assessment, which was one area in which Professor Warm made so many extensive and influential contributions. Here, we can ask questions as to how a fuller understanding of an individual’s subjective state adds to systemic operational efficiency, at all levels of analysis. In its essence, this is a conceptual challenge as to how different reflections of cognitive workload mesh together because task performance itself is frequently considered the primary indicator of cognitive workload. It also highlights potential divergences among objective measures, those of physiological and performance-based workload indicators not being always interchangeable descriptions. A special focus in the present work centers on the question as to what interpretations are to be drawn when these differing reflections of cognitive workload agree, disagree, or are apparently indifferent one to the other. We designate such patterns here as the (a) association, (b) insensitivity, and (c) dissociation (AIDs) of workload (and see Hancock, 2017). Nor shall we neglect the dimension of physiological reactivity in this overall examination of AIDs patterns.

The structure of the present article is as follows. After a brief introduction to some of the underlying “big-picture” conceptual issues that frame this overall concern, we describe subjective and objective workload measures and their AIDs profile. Whereas Warm’s vigilance studies predominantly showed associations between differing workload measures, other task paradigms reveal rather more complex relationships between subjective and objective responses. We discuss some of the factors that may promote divergence, including issues of observational granularity, the mediating effects of self-regulation, the inherent issue of timing, and hysteretic characteristics exhibited by both operator and attendant system. We conclude with three contrasting perspectives on the AIDs of workload. First, the different types of measure may simply be incommensurate one with another at a very basic level of observation. Second, future research may identify neural networks that drive each and all of these various expressions of workload. Third, multiple forms of workload assessment may be pragmatically useful without a full conceptual understanding of specific workload AIDs patterns. We cannot currently define any one perspective as superior to the others, but Professor Warm’s workload studies provide a paradigm for converging operations that may eventually serve to resolve the difficult issues identified in this article.

The Bigger Picture

Before narrowing our focus to the specifics of cognitive workload, it is important to first consider some of the larger contextual dimensions of this whole area (and see Moray, 1979). HF/E has often referred to itself as engineering psychology and, over its existence, has represented, occasionally, a tempestuous partnership between engineering and psychology. This meeting of the purportedly “objective” expressions of engineering and the evidently more “subjective” focus of psychology has not always resulted in harmony. Even within psychology itself, the battle over objectivity and the associated aspiration for accompanying scientific respectability led to the rise, and temporary dominance, of quite radical forms of behaviorism (Watson, 1913). It is important, however, to note that the conflict between ratiocination and the strong behavioral emphasis was never quite as divisive as some contemporary sources now like to portray it (cf., Kimble, 1994). Similarly, many engineering stalwarts of HF/E have courted potential criticism by some of their colleagues by endeavoring explicitly to deal with the intangible and often indecipherable dimensions of human cognition (cf. Marras & Hancock, 2013). One would like to feel that these clashes are now in our past and the mutual respect between our contributing parent sciences has reached, at least in the HF/E arena, a productive level of accord. Certainly, this mutual appreciation for each of the component elements of HF/E was very evident in the work of Joel Warm himself, whose contributions to engineering psychology are preeminent.

Much progress toward such integration, theoretically, practically, and even politically, has been achieved by revolutions in the computer sciences as well as the cognitive neurosciences. HF/E has benefited, and continues to benefit extensively, from these multidisciplinary and interdisciplinary advances while itself looking to contribute to the fundamental knowledge of each. It was one of the main aims of Parasuraman’s neuroergonomic initiative to create such a fertile interchange of understanding (Hancock, Baldwin, Warm, & Szalma, 2017; Parasuraman & Rizzo, 2006). But we have yet to reach a stage of complete integration in HF/E. This may well be because we have still to identify reliable and principled ways to fully weld mind and machine together; always assuming that this intimate symbiosis is the state that we should aspire to (Hancock, 2009). It may be asserted that the evident materialism of technology can only ever sit uncomfortably, and even discordantly, alongside the immanence of the emergent properties of mind, contingent upon one’s operational theory of mind itself. Indeed, these two phenomena (mind and technology) may be cast as emanating from radically different and incommensurate domains of knowledge, or what the naturalist Stephen Gould (1997) called non-overlapping “magisteria.” If so, they may prove to represent fundamentally incompatible tracts of knowledge.

But we have reasons for hope that any such incompatibility can be resolved, and one of the major contributions of the science of professor Joel Warm was, through his systematic experimental program of research, to help set our feet on the path to resolution in this matter. Here, we take up the case of cognitive workload as a special and, indeed, specific instance of this more general concern of matching objective observation with subjective reaction. For, as Annett (2002) has noted, “workload . . . being in part a measure of the observer’s private sensation but in part an assessment of the task” (p. 968) serves to provide a fertile ground for such explorations. We seek here to articulate potential avenues of resolution that can point our way forward, not only in this specific instance of workload but perhaps in the more general case, also.

The Many Faces of Workload

In what follows, the terms cognitive workload and mental workload are used synonymously. Cognitive workload may be defined as “an emergent property of the active brain which is tasked with the demands of surviving and prospering in an incompletely specified and under-explained world” (Hancock, 2017, p. 3). The history of cognitive workload is extensive and detailed (see e.g., Annett, 2002; Hancock & Meshkati, 1988; Moray, 1979). From this wide corpus of extant knowledge, there have emerged numerous operational definitions of cognitive workload that revolve around the individual’s capacity to respond. These have often been couched in temporal terms, for example, the time needed for a task to be completed divided by the time available for that task. Definitions have also embraced the attentional dimension, most often formalized in the assumptions of resource theory (Norman & Bobrow, 1975). Specifically, performance is sometimes, but not always, limited by the availability of general-purpose processing resources. As a greater proportion of resources is allocated to processing, the person is assumed to experience increasing subjective workload, accompanied by various physiological concomitants of that level of mental work (Matthews, Wohleber, & Lin, in press). That is, all manifestations of workload reflect the level of allocation of the specific pool of attentional resources accessed in response to the incipient demands of the task set before them (see Hancock & Caird, 1993; Kahneman, 1973; Wickens, 1980, 2008). These are two, but by no means all, of the conceptual approaches that have been used to try to understand workload (Hancock & Meshkati, 1988).

To assess such cognitive workload, four principal methods have been employed. These are well known in the HF/E community, namely, (a) primary-task performance, (b) secondary task performance, (c) subjective response, and (d) physiological indicators (see, e.g., Meshkati, Hancock, Rahimi, & Dawes, 1995). It is not our purpose here to describe the varying structures of, nor assess the relative merits of, these respective methods. That has been accomplished on numerous previous occasions (see, e.g., Hancock, Meshakati, & Robertson, 1985; Matthews & Reinerman-Jones, 2017; Meshkati et al., 1995). It is, however, important to note that secondary-task techniques have been less prevalent and less popular in more recent times, perhaps as a result of both the methodological challenges involved in administering them in both the field and the laboratory and the debate over the underlying theoretical assumptions that are involved in their generation and interpretation. The latter have most evidently focused upon the relationship, writ large, between workload and attention and most specifically resource theory (Wickens, 2008). We make no determination here about these specific concerns, but in the present work, we largely confine our observations to the three other major techniques.

The problems emerge when these different reflections provide conflicting indications as to what is going on. Let us suppose, for example, that a research investigation has included assessments of primary-task performance while taking a number of physiological measures and at the same time recording the operator’s subjective response. Such studies are frequent and growing in the HF/E literature (see, e.g., Fallahi, Motamedzade, Heidarimoghadam, Soltanian, & Miyake, 2016; Matthews, Reinerman-Jones, Barber, & Abich, 2015; Teo, Reinerman-Jones, Matthews, & Szalma, 2015). How do we interpret the outcome when some if not all of these methods and their specific component techniques provide diverging information? This problem has been termed the “AIDs of workload” (Hancock, 2017). This issue, which is our focus here, is especially pertinent for a reflection upon the contributions of Joel Warm. He and his colleagues were intimately involved with this question, especially as it pertained to investigations of vigilance (and see Hancock, 2013). Warm and his colleagues had concluded that vigilance decrements did not emanate from extensive boredom and underload (Hancock & Warm, 1989; Warm, Parasuraman, & Matthews, 2008). Rather, the opposite was true. Vigilance deficiency is actually a result of stress and comes from its attention-demanding nature. This proposition was confirmed, in part, by the ubiquitously associated high scores on accompanying subjective workload scales (i.e., NASA Task Load Index [NASA-TLX] scores) in various vigilance evaluations. Importantly, Warm was at the forefront of efforts to establish this associational relationship between response efficiency and experienced workload and had made important progress toward this goal, as we explain next. But association is far from a ubiquitous result in the general run of workload investigations.

Associations, Insensitivities, and Dissociations: The AIDs of Workload

Starting from the era of time and motion studies and its engineering precursors, the most frequent workload assessment method has been to measure the physical output of the individual. Today, this approach has evolved to focus on primary-task performance in general. As work content changed across the 20th century, its form has also changed from mainly physical output to cognitive achievement, this metamorphosis being in answer to the demands of a modern information-based society. Recognition of this growing dominance of cognitive transformations was articulated, for example, in the aviation domain, where the Cooper-Harper approach to aircraft handling qualities extracted the expression of thoughts and perceptions of test pilots via a forced-choice sequence. In such cases, failed output (i.e., the crash of test pilot and test aircraft) was considered then, and remains, simply unacceptable. This lead was followed by others in aviation who formalized further measures, such as the now ubiquitous NASA-TLX (Hart & Staveland, 1988) and also the Air Force’s subjective workload assessment technique (SWAT; Reid & Nygren, 1988). Each of these served as foundational bases for what came to be known more generally as the subjective approach to workload evaluation (cf. Gopher & Donchin, 1986; O’Donnell & Eggemeier, 1986). There are those who express little sympathy with such amorphous and intangible perceptions. Yet, despite arguably being the phenomenon of primary concern (and see Gilbert, 2009; Hancock et al., 2002) in HF/E, perceptions, and especially feelings, are difficult to specify, and some, if not many, practical researchers are more comfortable with externally inspectable, and more readily quantifiable, reflections of operator/user response.

Figure 1 provides a descriptive foundation from which to consider the various patterns of AIDs of these cognitive workload methods. Here, each of the three respective methods can show patterns of increase (+), decrease (–), or zero change (o). Thus, primary-task performance can improve, it can stay the same, or it can degrade. Similarly, subjective workload can increase, decrease, or stay the same, as can patterns in associated physiological reflections. Figure 1 demonstrates that there are numerous (27) possible outcome patterns and that the nature of the outcome in any given study might remain problematically ambiguous unless all three types of measure are secured.

OPEN IN VIEWER

Figure 1.

An overarching workload matrix illustrating three disparate measures in which the respective methods can reflect either increasing, decreasing, or stable workload responses. Clearly, the opportunity exists for double associations (in which all methods agree, labeled A+ and A–). There can also be single associations (two methods agree, but one either is insensitive or dissociates from the others). Equally, there may be double dissociations (in which all methods disagree) or single dissociations (such that only two methods disagree; and see single associations). Insensitivities (methods show no change in relation to varying external task demand) can be plotted within each of the remaining spaces after the association/dissociation relationships have been established (after Hancock, 2017).

When the response pattern agrees between any two of these reflections, we can identify an instance of association, but only two outcomes represent perfect association between all three measures, and these are denoted as A– and A+ in Figure 1. We term these conditions double association. For example, the vigilance decrement may be accompanied by both subjective workload increase (Warm, Dember, & Hancock, 1996) and increases in cerebral oxygen saturation measured by functional near infrared spectroscopy (Funke et al., 2010). Similarly, when two methods directly disagree, we witness a dissociation. For example, Vidulich and Bortolussi (1988) showed that introduction of voice control for helicopter piloting improved pilot performance in a demanding control scenario but also elevated subjective workload due to the effort required to speak precisely and monitor associated feedback. When all three types of workload measure directly contradict each other, we encounter cases of double dissociation. The latter necessarily includes some degree of insensitivity, because if one method demonstrates an increasing trend and another shows concomitant decreasing trend, then to vary from these respective profiles, the third method must show neither of these patterns, and this implies insensitivity.

Single associations and dissociations are most commonly identified in experiments that use only two of the three major methods, although we could also find a single association between two measures together with insensitivity on the part of the third. As a consequence, identifying double associations and dissociations requires that a particular experiment or procedure has used all three of these major measurement approaches. It is important to note that the term double dissociation has been used in other domains of cognitive psychology and that such usages are not always directly overlapping or completely equivalent and consistent in meaning (and see Davies, 2010).

If any two methods prove indifferent to each other, we can identify insensitivities, and data can show double insensitivities in line with the observation on association and dissociations that we have made above. In earlier work (Hancock, 1996), which considered only primary-task performance and subjective responses, these single dissociative patterns were termed performance insensitivity and workload insensitivity, respectively (cf. Annett, 2002, Figure 1). Performance insensitivity occurs when circumstances or task factors serve to change the degree of subjective workload experienced but the performance itself remains stable across the implicit or explicit manipulations of task difficulty. Thus, dissociation occurs as a result of the output performance remaining unchanged while the experienced degree of workload reflected in subjective report either increases or decreases across conditions. We might well imagine that such insensitivity occurs, for example, in highly demanding, risky, and stressful environments in which any sort of performance degradation provides an existential threat. Here, behavioral compensation to increased task load is imperative (and see Hancock & Warm, 1989), and performance is preserved at the cost of ever-greater cognitive effort.

In contrast, workload insensitivity provides the opposite effect. That is, output performance is evidently changing but the performer remains subjectively oblivious to such variation in his or her overt behavior. For example, Horrey, Lesch, and Garabet (2009) had vehicle drivers perform an additional secondary task, either a guessing game or a mental arithmetic task. Performance was lower on the guessing game, but there was no subjective workload difference. This pattern might occur in states of denial, in which everyone surrounding an individual witnesses his or her evident output response failure but, for reasons of fatigue, indifference, and the like, the individual does not register his or her own incipient failure (or, indeed, success). In Horrey et al.’s study, drivers’ awareness of workload may have been dominated by the primary driving task, reducing sensitivity to the additional load imposed by the secondary task. Changes in task strategy may also influence performance without influencing overall subjective workload. Performance increase may be a consequence of “working smarter, not harder” (Kluger & DeNisi, 1996), whereas performance decrease may result from reallocating attention from the task to distracting activities. Insensitivities are forms of dissociation because one measure breaks with another, but they are not direct dissociations in which such multiple measures are in direct disagreement. However, as we have seen, primary-task performance and subjective response are only two of the three major ways in which we can register workload. Thus, we now add and explore the case of physiological insensitivity (Hancock, 2017).

As should be clear from the foregoing, beyond the double association and double dissociation, there are many partial patterns that may emerge. We may witness association between two of the methods but active dissociation on behalf of a third. Various patterns of insensitivity can also be evident, even when two of these reflections either directly agree or disagree with each other. All such instances can be plotted within the framework offered in Figure 1. Thus, the present illustration can become the foundation for a taxonomic differentiation of the extant and developing body of empirical findings. Such descriptive taxonomies are crucial to the development of subsequently emerging theoretical explanations for the patterns we observe therein.

To the present time, the AIDs of workload are best illustrated in studies comparing changes in primary-task performance and changes in concomitant subjective measures. Some studies draw on concepts from resource theory as the foundation for understanding how task demands influence workload (Matthews & Reinerman-Jones, 2017). Especially during (a) epochs of incipient demand overload, (b) events with significant time pressure, and (c) those with limited scope for strategic compensation, subjective workload and primary-task performance tend to align well and mostly provide cases of association (see Matthews et al., in press). Indeed, Warm’s vigilance studies represent an evident case study of such performance and workload convergence. We might speculate that this is perhaps so because the required responses in vigilance are discrete, punctate, and evidently memorable, meaning the granularity of the output performance and the summed nature of the memory of the associated subjective workload here are similar. In the vigilance case, loss of performance, the subjective sense of high workload, and physiological responses, such as slower cerebral blood-flow velocity, may all be attributed to resource insufficiency (Warm et al., 2012).

However, numerous studies show divergence of subjective and objective response. Wickens, Hollands, Banbury, and Parasuraman (2013) summarized several reasons for dissociation as attributable to the resource theory basis for the workload concept (and see also Yeh & Wickens, 1988). For example, changes in performance driven by data limitations may not be reflected in subjective experience. That is, performance may reflect the quality of data from perception or memory rather than resource variation. However, cases of this kind are most often really instances of insensitivity of one or the other method, rather than true dissociation. Thus, such divergent patterns could be attributed to measurement shortcomings rather than any fundamental underlying issue with the conceptualization of workload. Indeed, insensitivity might even prove a desirable side effect of diagnosticity, for an instrument diagnostic of some specific and identified factors that influence workload in a particular context should indeed be insensitive to other such sources in that context. It is only through a full understanding of this constellation of “driving” factors that we can build a fully comprehensive model of workload and the profiles of individual and specified “suites” of measures that can predict it.

The more problematic case is the one in which workload indices respond in opposite directions. For example, Abich, Reinerman-Jones, and Matthews (2017) required participants to detect human threat stimuli (i.e., possible terrorists) in a video feed from a simulated unmanned ground vehicle. They found that increasing event rate elevated both workload and detection efficiency at one and the same time. In this case, the demand manipulation may have made the task more gamelike and engaging, so that higher demands were experienced as a challenge, that is, as an affectively enjoyable form of workload. This facet of response may well link directly to Kalsbeek’s (1968) concept of “willing to spend,” in which an individual exercises a value judgment over his or her attentional resource application strategy (and see Hancock & Caird, 1993). Conversely, the person who simply gives up on an overly difficult task will experience low workload along with ever-poorer performance (Hancock & Caird, 1993). Thus, dissociations may be driven by the operator’s affective and strategic responses to changing task demands as reflected in his or her individual coping strategies (and see Sharples, 2018).

Such hedonic dimensions of experience, in the performance–workload domain, have relatively rarely been evaluated because, for one thing, such affective dimensions do not appear on the most commonly used subjective scales. This is an evident shortfall. It is clear that we react differently to imposed task demands when they carry different affective connotations (Hancock, Pepe, & Murphy, 2005). Putatively “difficult” tasks may actually be sought out by enthusiasts who see their demand as challenging and enjoyable. Thus, some throw themselves from aircraft not simply as antiseptic, sterile beings engaged in an obligatory “task” of skydiving but as persons actually embracing the challenging and hedonic dimensions of the experience. Others fish, perhaps as a result of these self-same motivations. It is thus our contention that in the search for greater generality of valid workload evaluation across the full spectrum of human pursuits, we need to augment standard subjective instruments with components that assess these active affective dimensions if a fuller and more explanatory picture is to emerge. That such multiple scale approaches are now beginning to be more prevalent, we fully acknowledge.

As with subjective measures, physiological indices of workload sometimes agree well with variations in performance efficiency. A case in point is the close parallel between vigilance decrement and declining cerebral blood flow in the middle cerebral arteries, which provides important evidence for the workload model of sustained attention (Warm, Tripp, Matthews, & Helton, 2012). Under other circumstances, physiological insensitivities and dissociations are seen. Indeed, different physiological responses themselves may diverge, one from another. Workload is measured via numerous neural systems reflecting different conceptions of how response to task demands might be expressed in physiological change (see Table 1). The logic is typically that greater demands lead to greater brain activity, but there are substantial differences in the more detailed rationales for each workload index (Matthews, Reinerman-Jones, Barber, et al., 2015). For example, electroencephalogram (EEG) is influenced by numerous brain regulatory systems, and to attribute increasing spectral power, even in selected frequency bands, to overall processing activity and mental workload is a considerable simplification at best. It is also important to reinforce that many brain systems act via inhibitory effects. Thus, even a priori, there are reasons to believe that some of the associations are of a negative relation anyway.

OPEN IN VIEWER

Table 1:

Summary of Leading Physiological Workload Indices and Their Conceptual Bases

Physiological Measure	Example Workload Index	Conceptual Assumption
Electroencephalogram	Frontal theta power	Processing activity is expressed in greater power
Electrocardiogram	Heart rate variability	Task demands influence cardiac baro-reflexive function
Functional near-infrared spectroscopy	Frontal blood oxygenation	Task demands elevate metabolic activity
Transcranial Doppler sonography	Cerebral blood-flow velocity	Task demands elevate metabolic activity
Eye tracking	Blink duration	Visual demands shorten blink duration

Multiple workload measures, including those listed in Table 1, have been validated through studies showing their sensitivity to task demand manipulations (Matthews & Reinerman-Jones, 2017; Vidulich & Tsang, 2012). Unfortunately, the use of physiology does not solve the dissociation problem. Studies using multiple physiological workload indices often show evident insensitivity, where different metrics each prove sensitive to different task factors (Matthews, Reinerman-Jones, Barber, et al., 2015; Wilson, 2002). Furthermore, psychometric analyses show that metrics from different physiological systems are only weakly correlated at best (Matthews, Reinerman-Jones, Barber, et al., 2015); thus, they index different responses. Individuals may also differ in relation to how discrete physiological systems respond to changing task demands (Teo et al., 2018). Lack of convergence is even more starkly apparent when considering relationships between subjective and physiological response. It is not unusual for the NASA-TLX and physiological measures to show differing responses to common task demand manipulations (Hancock et al., 1985; Luque-Casado, Perales, Cárdenas, & Sanabria, 2016).

Table 2 provides illustrative patterns from four studies in which task demands were manipulated in various ways within simulations of complex operational tasks. Here, multiple sensors were used to record differing physiological responses. Changes in subjective workload induced by a task manipulation were quantified in terms of the points change on the 0-to-100 NASA-TLX scale. Each study shows a unique pattern of physiological response, some of which conflict with conventional interpretations, so that disagreement is evident. NASA-TLX scores do not tell us much about such neurophysiological responses. Similarly, correlational studies in adequately large samples suggest that the NASA-TLX outcomes diverge from a range of physiological metrics (Matthews, Reinerman-Jones, Barber, et al., 2015). Nor can we simply jettison subjective assessments and work with physiological metrics alone. Studies of stressor effects utilizing both types of measure have shown that they predict independent parts of the variance in performance, that is, subjective assessments add to performance prediction over and above physiology (Matthews et al., 2010; Matthews, Reinerman-Jones, Abich, & Kustubayeva, 2017). Disagreements between measures are not simply a nuisance founded in the limitations of individual measurement instruments; they signal that a unitary workload construct is not adequate for capturing the manifold ways in which people react to and manage changing task demands (Matthews, Reinerman-Jones, Wohleber, et al., 2015).

OPEN IN VIEWER

Table 2:

Correspondences in Subjective and Physiological Workload Metrics in Four Studies

Study	Demand Manipulation	NASA-TLX Effect (Points)	Workload Response (Expected)	Workload Response (Unexpected)
NPP simulation (Reinerman-Jones, Matthews, & Mercado, 2016)	Gauge change detection (harder) vs. other tasks (easier)	4.9	Higher rSO2 (fNIR)	Lower beta and gamma, higher alpha (EEG) Lower heart rate
UGV simulation (Matthews, Reinerman-Jones, Barber, & Abich, 2015)	Change detection (harder) vs. threat detection (easier)	16.1	Lower HRV Higher rSO2 (fNIR) Higher frontal theta (EEG)
UGV simulation (Matthews, Reinerman-Jones, Barber, et al., 2015)	Dual task (harder) vs. single task (easier)	10.0	Shorter eye fixation duration
UAV simulation (Wohleber, Matthews, Reinerman-Jones, Panganiban, & Scribner, 2015)	High cognitive load (time pressure, number of UAVs) vs. lower cognitive load	16.7	Higher beta and gamma (EEG)	Higher HRV

Note. EEG = electroencephalogram; fNIR = functional near infrared spectroscopy; HRV = heart rate variability; NASA-TLX = NASA Task Load Index; NPP = nuclear power plant; rSO₂ = regional cerebral blood oxygenation saturation; UGV = unmanned ground vehicle; UAV = unmanned aerial vehicle.

Not So Alike After All: Problems With Convergence

Dynamic Issues: Workload History and Hysteresis

A further issue we can identify is that the prior profile of task demand affects the current level of experienced workload (and see Hancock, 1996; Prytz & Scerbo, 2015). Studies of workload history and workload transitions illustrate that change itself is often challenging and even stressful to operators (Helton et al., 2008). Relative to stable workload conditions, both sudden increases and sudden decreases in workload can be associated with decrements in performance (Cox-Fuenzalida, Beeler, & Sohl, 2006) together with subjective disturbance (Helton et al., 2008). From a theoretical standpoint, operators appear to be consciously aware of possible dynamic instability induced by both underload and overload, prior to any manifest physiological disturbance and performance impairment (Hancock & Warm, 1989). Indeed, people are sensitive to deviations in rates of change from expectation in making progress toward goals (Carver & Scheier, 2013). Subjective workload may thus be biased by deviations from expected rates of work.

There are numerous facets of the task demand profile that induce nonlinear reactions in many reflections of the associated workload level (and see Hancock, 2017). A specific dynamic process that has recently garnered empirical attention is the phenomenon of hysteresis (Jansen, Sawyer, van Egmond, de Ridder, & Hancock, 2016). Self-regulatory aftereffects expressed in hysteretic profiles are evident in a clear and consistent fashion (Hancock, Williams, Miyake, & Manning, 1995). As is shown in Figure 2, a previous epoch of low task demand tends to elevate perceived workload response when the individual returns to a prior baseline level of task demand. The converse is also true; an interval of high task demand tends to suppress perceived workload on a following interval of baseline task demand. This effect is most easily illustrated graphically; see Figure 2.

OPEN IN VIEWER

Figure 2.

The hysteretic effect of prior loading can be seen in the illustrated data. Participants experiencing a constant level of medium task difficulty (central three columns) express no change in workload across exposures. Participants encountering a low task demand as the middle of three performance intervals now rescale their workload as more intense on a final exposure to a common, medium level of task demand (left three columns). The opposite effect is also confirmed when the middle exposure is to a high demand (right three columns). The effect works for both the NASA Task Load Index and subjective workload assessment technique (after Hancock, Williams, Miyake, & Manning, 1995).

In essence, this pattern reflects a temporal lag in recovery to baseline. The central point in the present context is that differing reflections of cognitive workload exhibit different hysteretic latencies (and see Morgan & Hancock, 2011). Thus, transient effects in brain states may be dampened out in milliseconds; variations in heart rate and its linked variability may diminish within seconds. However, some effects in conscious experience may last a lifetime and never be effectively extinguished. Thus, where we seek patterns of association, it may well be that the workload response profile, in reaction to change in ongoing task demand, exhibits differential, inherent recovery latencies. These differing recovery latencies can then serve to mask what might be actual “locked” associations, giving a false impression of insensitivity or dissociation. Observations of hysteresis should encourage us much more to consider time series data in all of these task demands and comparative cognitive workload investigations.

Incommensurate Orders of Observation?

Added to these differences in hysteretic properties, at its heart, the practical aspirations for an associated performance–workload link lie in the practicalities of future prediction. That same predictive motivation energizes modeling efforts also (Longo, 2015). Much of this aspiration is founded upon the bedrock of a unitary and coherent theory of behavior. Some have argued that in a practical sense, if one is able to model behavior accurately, accompanying changes in physiological response or subjective experience could be considered relatively superfluous (and see Watson, 1913). This is not to say understanding of any mediating construct is unimportant or uninformative; such constructs can still be legitimate domains of study. They are more simply, in a world of such pragmatism, not essential components of the process. But, we have never reached this level of certainty in modeling, and it is doubtful whether even theoretically that a full robust and assuredly deterministic model is itself achievable. That being said, we now search in the physiological, the neurophysiological, and the subjective for the sources of some of the predictive shortfalls we necessarily experience. But can such diverse orders of response ever phenomenologically match to each other?

We take here conscious experience and differing forms of implicit processing as properties that emerge from the activities of the neurophysiological substrate systems that found them. Problematically, there is no assurance that any emergent properties will bear any homeomorphic mapping to those elements that serve to found them. The search for a singular neurophysiological seat of consciousness has persisted since the time of Descartes, and even before, but as yet no ultimate, single command neuron has been identified. In light of our current understanding of the brain, it is highly doubtful that there is such an entity. Similarly, task performance is dominated by forms of motor outflow (see Hancock & Newell, 1985). To a degree, and founded upon the coherent processing propositions (e.g., systematic information processing), such motor outflow patterns are, in large part, contingent upon the neurophysiological activities that precede them. However, motor responses serve to affect action in the world, whereas other associated cognitive processes that are assumed to be involved need not necessarily do so. Thus, contingent upon one’s theoretical assumptions, each of a performance response, a physiological reaction, and/or a subjective state need not partake of the same universe of discourse. Thus, we might argue that there is almost a necessary disconnect between these respective orders of measure given the way in which they are made manifest in the world. Although this conceptual argument remains subject to debate, the degree to which it is considered valid must temper any eventual aspiration that performance and workload can be deterministically locked together.

But on the Other Hand

To this point, we have emphasized reasons why the various aspects of workload may fail to associate together. However, it is important to consider the other side of the coin, that is, why they ought to associate. In response to contextual work demands, it remains the one, singular responsive individual who is emitting all of this pertinent and measurable information. Thus, we are measuring the performance efficiency, the physiological patterns, and the subjective appreciation of one particular person. A priori, it seems reasonable then that what is emitted from a single source should be correlated and related in certain cause-and-effect patterns. In theory, the enterprise from this perspective appears to be a signal detection challenge in which the various sources of noise simply have to be identified and suppressed. The prior observations concerning incommensurate temporal resonances is a case in point. McNaught and Callander (1965) observed that “every effector is potentially connected to every receptor” (p. 7), emphasizing the networked nature of human capacity. The underwritten belief is that such an associative pattern must exist, at least somewhere and at some level. It must, therefore, require a massive and even exhaustive search to identify it but predicated on the assurance that it must eventually be there to find. To date, our community has been motivated by the veracity of this belief being true. However, it remains an open question how much systematic associative patterns can be treated on a nomothetic basis, that is, generalizable across individuals, versus an idiographic perspective that recognizes the uniqueness of each individual’s own personal workload response. This represents a strong and continuing challenge to our science (and see Hancock, Hancock, & Warm, 2009).

Toward Pragmatic Solutions

Thus far, our argument has been that subjective workload is a slippery construct that is hard to anchor in the physical reality of brain functioning. Nevertheless, subjective assessments remain useful in human factors for systems evaluation and design. De Winter (2014) highlighted an important distinction between representational and operational measurement (Hand, 2004). The former refers to well-defined quantifiable attributes of objects, such as mass, velocity, and so on, whereas the latter refers to properties that may be ill defined but pragmatically useful. De Winter sees workload as falling toward the operational end of the spectrum of measurements, given that workload has utility beyond performance measurement for assessment of operator functioning. Indeed, dissociations between workload and performance may provide important, if not essential, insights into operator skills and strategies (Hancock, 1996).

Similarly, current perspectives on test validation emphasize that validity must be tied to specific purposes for testing, rather than trying to establish constructs as fixed personal attributes indifferent to context (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014). In this vein, Matthews and Reinerman-Jones (2017) have distinguished multiple purposes for workload assessment in which the usage and interpretation of subjective workload measures may differ. For example, from a pragmatic perspective, data may exist that allow the measure to be used to determine a redline for overload with a specified task and operator population. By contrast, workload scales can also be used as outcome measures in tests of cognitive and neurological theories of operator response to task demand manipulations. In this case, workload may be treated as more of a representative variable, that is, as a measurement that can be linked to an underlying latent construct via a statistical model. Other purposes include specifying individual differences in operator response and performance prediction. In each case, a contextualized understanding of the workload measure that may downplay general theory is needed.

It is also frequently desirable to measure workload as one component of a multivariate assessment strategy, which, depending on context, may assess related variables, such as stress, fatigue, trust, risk acceptance, and so forth to build up a detailed picture of operator response to task demands (Abich et al., 2017). Dissociations between workload measures are, from this perspective, a feature rather than an obstacle that specifies how the operator is interpreting and responding to events. That is, in some performance settings, workload may be too simplistic a construct. Operators may exhibit multiple, separable responses to task demands that should be assessed separately. For example, Guznov, Matthews, Funke, and Dukes (2011) measured workload, stress, and performance on a task requiring control of multiple robots in a “capture-the-flag” scenario, played against a computer-controlled robot team. Uncertainty over the location of the adversary’s robots was manipulated by varying the range of a radar that mapped the field surrounding each of the player’s robots. Paradoxically, higher uncertainty reduced mental workload and distress but also impaired performance, a workload–performance dissociation. In this case, high workload was a reflection of awareness of the enemy’s actions, which was beneficial to performance but not to the player’s sense of well-being. In complex task environments, multiple measures may be necessary to reveal the full range of subtleties of this kind.

A Summary and Brief Conclusions

The foregoing examination has looked to understand the varying relationships between different reflections of cognitive workload. To the first question, that is, is such a search ultimately beneficial, we have provided a positive response. We find there is value in understanding the intricacies of underlying cognitive processes, even if predictive models of behavior might appear to be very accurate and reliable without the inclusion of either of these affective or physiological dimensions. We should note, however, that current models of workload certainly do not yet reach this standard of satisfactory prediction, although important progress is being made in this area (see Young et al., 2015). In part, this whole issue is an “outer envelope” problem that centers its focus on extremes of conditions when the system and operator are tasked to their greatest limits (Hoffman & Hancock, 2017). By this we mean that insights from dimensions of workload beyond manifest performance itself can provide vital input in these off-nominal and emergency operations. Most importantly, they can provide warnings as to incipient operator failure. As these “extreme” situations more and more become the focus of certain HF/E sectors of special concern, we see value in pursuing the difficult, yet potentially highly rewarding, inquiries into the convergences and divergences of workload measures.

That being said, there remain both conceptual and methodological challenges to the clarification of a unified workload picture. We have articulated a number of these barriers but do not lay claim here to have explored exhaustively all of the different concerns. What we do consider the proximal problems for HF/E to solve are how to relate the emergent and unified experiences of subjective cognition to often discrete and quantized elements of ultimate outcome performance. Thus, do people especially notice, in a conscious sense, either their actual speed or their error in responding? Is it possible they code only outliers, which are connoted by especially slow or evidently erroneous responses, while still being “unaware” of their speed and accuracy within a general, “satisficed” range? Further, how do they comprehend multiple tasks, and do they impose a necessary implicit, or even explicit, hierarchy on all such amalgams? Further concerns attend the various physiological reflections of workload. After all, many of these physiological processes embrace their own primary function and only distally reflect cognitive workload as a by-product of these primary actions. Although we can view this latter concern as a signal-to-noise challenge, we must be very aware that in such cases, neither signal nor noise can be considered as simple unidimensional concepts (Parasuraman et al., 2000). Thus, heart rate may be the very pertinent signal in respect to physical exercise, exercise still being the noise factor in workload evaluation, and of course, vice versa. Thus, what is signal and what is noise here are contingent upon the context as well as the experimenters’ focus of concern. Consistent with this multidimensional signal-to-noise concern, there remains, as we have noted earlier, the persistent problem of intrinsic frequency and all the issues that attend when the temporal characteristics of these various measures of workload are incommensurate with each other.

Questions as to AIDs are thus both conceptual and methodological in nature. There are many methodological challenges that result from conceptual concerns. Thus, theory energizes methodological innovation in order to measure hypothetical relations of interest. Equally, conceptual challenges often derive from methodological innovations. For example, each new progressive window on the brain, most frequently developed in the realm of medical research, allows behavioral scientists important glimpses of subsuming neurophysiological activity. The latter link tells us what we can measure. The former link looks to specify what we should measure. As long as neither of these motive forces (theory or method) become overdominant, a productive synergy continues. In practical workload assessments, we do have to begin to specify much more explicit “stopping rules.” That is, the pluri-potentiality of all reflections has to be pragmatically bounded by collective agreement that is based on sound theoretical foundations. Otherwise, we can enact virtually an unlimited number of potential physiological reflections and an unending sequence of investigations finding scattered patterns in disparate systems as statistical chance dictates.

Both temporal and spatial resolution of workload responses continually advance. In the future, we will be able to measure more and more aspects of overall behavior down to, for example, single neuronal firings. But what principled reasons do we have for looking at that particular grain of analysis and any one particular neuron and its activity? Theoretically, specified boundary conditions help us navigate the ocean of data that promises to come our way. These are significant challenges to the science of HF/E. However, they are ones that the persuasive optimism and dedicated experimental efforts that characterized the life of Professor Warm encourage us to believe are amenable to resolution.

Key Points