Sustained Attention to Science: A Tribute to the Life and Scholarship of Joel Warm

The Career of Joel Warm

The life in science of Joel Warm can be readily recounted (see Hoffman, Boles, Szalma, & Hancock, 2018). Like many, his first interest in Human Factors emerged when he encountered problems and challenges associated with working with technology during his service in the U.S. Military. He was fortunate in being able to work alongside Dr. Glen Hawkes and Dr. Michel Loeb at the U.S. Army Medical Research Laboratory at Fort Knox, Kentucky. It proved to be a happy introduction to the application of basic psychophysics to very practical, real-world problems. Very early on, Joel was helping military personnel in their important struggle to achieve mission success, it was a theme that was also to characterize the last phase of his career. Such early experiences led to doctoral studies at the University of Alabama where Joel studied under the tutelage of Dr. George Passey. Like Joel’s previous mentors, Passey emphasized the value of theory as it applies to practice. It was after securing his PhD that, in 1966, he came within the sphere of influence of Dr. Earl Alluisi, one-time president of Human Factors and Ergonomics Society (HFES). Joel joined Alluisi as a postdoctoral researcher at the University of Louisville.

Upon leaving Louisville a year later, Joel moved to the Department of Psychology at the University of Cincinnati (UC). It was to be his professional home for the remainder of his academic career. Cincinnati proved a splendid opportunity because it afforded Joel the chance to work with William (Bill) Dember. This became a most fruitful collaboration that would be sustained across decades until Bill died in 2006 (Hoffman, Hancock, Parasuraman, Szalma, & Scerbo, 2015). An outstanding and eminent psychologist in his own right, Bill was ever supportive of Joel’s efforts and from their partnership others were also invited to participate in mutual collaboration (see, for example, Warm, Dember, & Hancock, 1996). It was there, at Cincinnati across the next four decades that Joel developed the Graduate Program in Human Factors and Applied Experimental Psychology. In all, Warm graduated more than 40 PhDs from that program including a future president of the American Psychological Association (APA) and many others whose names are well-known to the HF/E community both in the United States and across the globe. Under his guidance, Cincinnati became a major “name” in our field to rival any of the premier pedagogical programs.

During his time at Cincinnati, Joel secured a number of senior appointments commensurate with his standing. He was a senior postdoctoral fellow at NIOSH’s (National Institute for Occupational Safety and Health) Taft Laboratories in 1986 and was a Distinguished Summer Faculty at the Navy’s Warminster Facility in 1992. Of course, his experience with governmental agencies had begun much earlier when he was a research associate at Fort Knox in the late 1950s. Following his 41 years in the Department of Psychology at Cincinnati, Joel came home to the military, becoming an Air Force Senior Scientist at the 711 Human Performance Wing at Wright-Patterson AFB in Dayton, Ohio, in 2009. His outstanding service efforts matched his preeminence in research. For example, he was the president of the Tri-State Human Factors Chapter from 1988 to 1989 and shortly thereafter, he served as president of the Southern Society for Philosophy and Psychology in 1991. Both of these were groups that he supported for decades, being an honorary life member. Joel’s final designation was as Professor Emeritus of psychology at UC; Senior Scientist at the Warfighter Interface Division, Human Effectiveness Directorate, Air Force Research Laboratory (AFRL); and Distinguished Researcher in the Human Factors Group of the University of Dayton Research Institute (UDRI).

The present narrative is an appropriate place to recount Joel’s scholastic achievements. He was an associate editor of both the present journal and Theoretical Issues in Ergonomics Science; president of the Applied Experimental and Engineering Psychology Division (21) of APA; and he also served the National Academies on a number of task forces, as well as the standing Board on Human-Systems Integration. For such achievements he was made a Fellow of the Association for Psychological Science (APS), the APA, the American Association for the Advancement of Science (AAAS) as well as our own Society (HFES). He won numerous awards from different scientific societies including the Fitts and the Ely Awards of HFES as well as the Admiral Kollmorgen Award of the Augmented Cognition Society. Joel also won the Franklin V. Taylor Award of the Society for Applied Experimental and Engineering Psychology of APA. These many achievements were celebrated by numerous friends and colleagues at a Festschrift convened in his honor which was sponsored by the APA, the AFRL, and the Department of Psychology, at the UC. It brought together many of Joel’s collaborators, admirers, and ex-students and resulted in an especially poignant celebration summed up in an appropriately useful and insightful text (Hoffman et al., 2015).

Warm’s Leadership with Respect to Vigilance

Joel classified himself as a perception researcher and this was reflected in his major text Psychology of Perception that he co-wrote with Bill Dember (Dember & Warm, 1979). This classic text went through numerous editions. But Joel’s first love was always vigilance. His edited text on this topic in the early 1980s is the one against which all others are still measured (cf. Buckner & McGrath, 1963; Davies & Parasuraman, 1982; Davies & Tune, 1970; Warm, 1984). When Mackworth (1950) first identified the empirical pattern of attentional failure that typified what he called the “vigilance decrement,” the challenge was laid down to the community to explain precisely what was going on in such circumstances (Hancock, 2013, 2017). Almost all of the then leaders in practical behavioral research took up this gauntlet. Investigative works by the likes of Broadbent and Gregory (1965), and Wilkinson (1963) adorned many of the British research journals. In the United States the likes of Adams and Boulter (1962), and Frankmann and Adams (1962), Alluisi (who introduced Joel to vigilance; J. L. Szalma, personal communication, ca. 1997), Mackie (1977), Jerison (1963), Jerison and Pickett (1963), Wiener (1987), Buckner and McGrath (1963), and O’Hanlon (1965) all made important and original observations. It was into this milieu alongside the collaboration of his mentors that Joel first entered the vigilance arena. It was one which he would embrace and adorn for the rest of his life.

When giving career advice Joel observed, “find an interesting research topic and publish, publish, publish!” (J. L. Szalma, personal communication, ca. 1999). Joel followed this advice himself in sustaining his attention to the study of vigilance for more than five decades. He pursued the vigilance question across multiple sensory modalities (the visual, auditory, and tactile senses) and pioneered the effects within a fourth-olfaction (Warm, Dember, & Parasurman, 1991). He investigated every topic within the scope of vigilance, including its psychophysics, training for vigilance with knowledge of results, cueing, motivation, as well as physiological and neurological correlates of performance, stress and workload, individual differences (including special populations), and even the effect of transcendental meditation on vigilance (Warm, Seeman, Bean, Chin, & Wessling, 1977). There was no one single aspect of vigilance he did not consider and publish about during his career. In several instances, he extended the scope of the field by introducing entirely new dimensions. Most especially, as we examine here, he featured in helping pioneer the perceived cognitive workload associated with vigilance (Warm et al., 1996). With respect to the latter, Joel especially pioneered research on the relationship of perceived workload to the psychophysics of vigilance. In this area, he was unsurpassed.

Joel’s career ubiquitously featured vigilance, but in the early 1980s the explanation for, and the theoretical underpinnings of, vigilance and its associated decrement were still unclear (see Hancock, Volante, & Szalma, 2016). Although the taxonomy developed by Parasuraman and Davies (see Parasuraman, 1979) had codified the pattern of the extant experimental findings, the specific reason for the decrement proved somewhat more elusive. It was in collaboration that Joel and one of us (P.A.H.) established that vigilance was a high workload and stressfully demanding task (see Hancock & Warm, 1989; Warm, Parasuraman, & Matthews, 2008). This proposition was very much in contrast to the then held opinion that vigilance induced underload, boredom, lassitude, and disengagement. We had often discussed this conundrum of putative “underload” during the interval that one of us (P.A.H.) was writing a chapter for Warm’s (1984) edited text. This conjunction derived from the fact that one of us (P.A.H.) had been working with Sandy Hart at NASA to publish the now well-known NASA Task Load Index (NASA-TLX) workload questionnaire in an edited text on the issue (Hart & Staveland, 1988). Until this publication, the TLX, like the SWAT (Subjective Workload Assessment Technique), was only available in difficult to find Technical Reports. The TLX had been used on a number of tasks which concerned NASA such as complex motor movements and flight-representative multitasking. In conjunction with Joel we then examined the cognitive workload of vigilance. The a priori expectation was that vigilance would render low TLX scores. It did not. The mystery was why. Joel and his students set out on an empirical voyage, soon establishing that (i) vigilance rendered exceptionally high workload scores (compared with, for example, multitasking, driving, etc.) and (ii) parameters of vigilance (e.g., event rate time-on-task, etc.) were clearly reflected by changes in workload value (see, for example, Becker, Warm, Dember, & Hancock, 1995). Such findings helped to identify just how stressful and loading a demand vigilance imposes (Hancock & Warm, 1989; Warm et al., 2008). Joel went on to articulate how many of the variables “associated” with vigilance variation. It is an important challenge, to distill why vigilance here proved so “associative,” when other tasks illustrated either “dissociation” with, or “insensitivities” (indifference) to workload (see Hancock & Matthews, 2019). Part of this search was fueled by a series of experiments Joel had conducted with numerous colleagues examining how neurophysiological measures, particularly, transcranial Doppler sonography (TCDS) could reflect changes in vigilance capability. This work laid the foundation for more diagnostic methods to specify the underlying physiological activity at even greater levels of detail. Some of Joel’s most elegant empirical demonstrations served to support this renaissance in understanding to counter the putative “mindlessness” theories of sustained attention (cf. Robertson & Garavan, 2004; Robertson, Manly, Andrade, Baddeley, & Yiend, 1997; Thomson, Besner, & Smilek, 2016; and see also Fraulini, Hancock, Neigel, Claypoole, & Szalma, 2017).

Joel reasoned, based on arguments by Natsoulas (1967) regarding perceptual assessments, that if self-reports of workload were valid, then manipulations that modulated task load (which were manifest in performance differences) should be associated with corresponding changes in perceived workload. Joel always referred to this as the “Natsoulas Imperative,” by which he meant that if one is to believe self-reports one must clearly link them to variables that can be psychophysically manipulated. The idea here is that perceived workload scores may be considered valid if, and only if, they rest on the firm ground of psychophysics.

Whenever a core theoretical issue emerged in vigilance, Joel entered the fray with programmatic empirical attack organized around what he referred to as a “conceptual spool” (J. L. Szalma, personal communication, ca. 1995). He was a prolific experimentalist whose work was always grounded in theory (cf. Szalma, Hancock, Dember, & Warm, 2006; Szalma, Hancock, Warm, Dember, & Parsons, 2006; Szalma et al., 2004). Warm and his colleagues conducted much of what connotes the vital archive in this area (see Warm & Dember, 1998; Warm et al., 1996) to validate the Davies and Parasuraman vigilance taxonomy (Davies & Parasuraman, 1982; Parasuraman, 1979). He was also a major force behind rejecting the arousal explanation of vigilance in favor of a cognitive resource view (Hancock & Warm, 1989; Warm et al., 1996). Joel relentlessly pursued the question of the relationship between performance, perceived workload, and stress (Warm et al., 1996, 2008, 2015). Although the stress of vigilance had been studied previously (e.g., Frankenhaeuser, 1986; Frankenhaeuser, Nordheden, Myrsten, & Post, 1971; R. I. Thackray, 1981; J. Thackray, Bailey, & Touchstone, 1975; R. I. Thackray, Jones, & Touchstone, 1974; for summaries, see Warm et al., 2008; and for an early review see Hancock, 1984), Warm attacked the problem with his characteristic verve and thoroughness. With respect to perceived workload, Joel and his colleagues were the first then to examine this critical issue in a vigilance context.

The Mindlessness Theory of Vigilance and More

As Joel was sequentially investigating the workload and stress of vigilance, a new theory of sustained attention was emerging. The “Mindlessness Theory” proposed that the vigilance decrement resulted from a failure to regulate responding because of the repetitive nature of the task (Robertson et al., 1997). So observers respond inappropriately because they become “mindless,” that is, their attention is no longer directed toward the task. Joel argued that this theory was not supported by the evidence, but rather the data supported a capacity depletion view. For instance, if vigilance is due to mindlessness, he argued, then why was the associated workload so high? However, Joel did not rely only on the extant empirical research to support his claim. He set out to conduct a series of programmatic studies to clearly demonstrate that the evidence supports capacity depletion rather than the mindlessness account (Grier et al., 2003). Even after several such studies appeared to have unequivocally established this, Joel kept going, finding more and more ways to test these theories. The result is yet another body of elegant research that bears directly on theoretical accounts of sustained attention.

Although we have strongly featured Joel Warm’s centrality to vigilance, it would be wrong to leave the impression that this was the only area to which he made vital contributions; it was not. Most especially, he also presented important work and foundational experiments on the perception of time (Warm, Morris, & Kew, 1963). Although his investigations in this area are not as well-known in our community as are his vigilance studies, nevertheless they had important influences in the duration estimation world where such early works served to impact later research and researchers (Block, Hancock, & Zakay, 2010; Hancock, 1993). Like many productive scientists, Joel was never bound by any nominal limits as to the areas of concern, and his interest in all aspects of perception means that his reputation extends well beyond the applied aspects of experimental psychology alone. It is perhaps fitting in a memorial issue to allow the contributing authors to express their own particular debt and this is what is featured in the section which follows.

Contributions to this Special Issue

This commemorative issue for Professor Warm is composed of a number of papers, each of which in its own way derives either directly or indirectly from his inspiration. In the first of these works, Hancock and Matthews (2019) have looked to address an issue critical for HF/E as well as all of the psychological and behavioral sciences. It is an issue addressed by Warm but is one that to date can boast no definitive resolution. At once both a theoretical and practical challenge, their paper asks what happens when the varying methods of workload assessment contradict each other. For their specific discussion, Hancock and Matthews (2019) took the issue of cognitive workload assessment. As a staple of HF/E, workload evaluation is familiar to many in HF/E as it has an extensive history (Gopher & Donchin, 1986; Hancock & Meshkati, 1988; Moray, 1979; O’Donnell & Eggemeier, 1986), and has been the subject of much theoretical, as well as practical discussion (Hancock & Caird, 1993; Wickens, 2008), and in some ways still puzzles our community today (Longo & Chiara-Leva, 2017).

Hancock and Matthews seek to specify where the three major reflections of workload, namely, (i) primary task performance, (ii) subjective report, and (iii) physiological reflections may exhibit evident disagreement (dissociation), remain indifferent to each other (insensitivity), or provide concordant changes (association) (see Hancock, 1996). In raising and elaborating upon the conceptual and methodological reasons for these patterns, both of these authors have been profoundly influenced by Warm’s scientific body of work and their time as his direct collaborators. The aforementioned “Natsoulas’ Imperative” was a topic of frequent discussion between each of these individuals, being a foundation of their present report.

Hughes and her colleagues report a meta-analytic review concerning cardiac measures of cognitive workload. They quantify the sensitivity of such cardiac measures, and evaluate the circumstances in which different cardiac measures best reflect task workload. In doing so, they highlight Warm’s contributions to deepening our understanding across a wide swathe of human performance. Having mastered the various performance measures, especially signal detection theory and self-reports, Warm increasingly employed psychophysiological methods (Helton et al., 2010; Shaw et al., 2009) as he understood their value as tools for the objective assessment of performance. Hughes et al. call for practitioners to incorporate such measures whenever feasible and appropriate. To this end, their meta-analysis reports the validity and reliability of various cardiac measures that prove useful when assessing cognitive workload. These authors provide guidance as to which option is most appropriate given the nature of the experimental design to hand and the basic and/or applied contexts within which such research is set.

Satterfield and colleagues examine how self-control may contribute to the theoretical understanding of vigilance. Warm’s research identified many factors directly related to the vigilance decrement and provided much evidence to suggest that the decrement results from a depletion of information-processing resources over time. This contrast with underload theories, which posit that the decrement is a result of disengagement from the task, due to its monotonous nature. A newer concept, the resource-control theory of vigilance, seeks to bridge these two perspectives suggesting that an inherent bias toward self-generated mind-wandering draws attentional resources away from the task. Satterfield and colleagues tested this theory and demonstrated that depleting self-control does not result in a steeper vigilance decrement, providing evidence against resource-control theory and providing support for cognitive resource theory. The influence of Warm is evident in such work as it directly expands on his own. Furthermore, the authors here include his former students demonstrating that his academic legacy continues.

The abbreviated vigilance task (see Temple, Warm, Dember, Jones, LaGrange, & Matthews, 2000) was developed as a relatively fast, computerized method to elicit the vigilance decrement. The work of Craig and Klein describes the use of both behavioral and brain imaging methods to delineate the contribution of well-established attention types, with an emphasis on executive attention, to performance patterns in such abbreviated vigilance. The authors conclude that executive attention, as they operationalize it, does not play a significant role in the performance of that task. However, associated cognitive processes may play differing roles in vigilance across time. It is Warm’s work on the limited attentional resource model of vigilance, and its corresponding mental demands, which underlies and informs the approach taken by Craig and Klein. The lead author is especially indebted to Professor Warm for his encouragement on pursuing an ambitious research project for a (then) graduate student; Warm also being the research advisor to the second author on this work.

Claypoole and her colleagues provide a critical examination of the effects of event rate on cognitive vigilance. Their work demonstrates an inverse relationship between event rate and performance. As event rate increases, the proportion of correct detections decreases and, at the same time, false alarms increase. Thus, cognitive vigilance tasks tend to interact with the other domains of the classic vigilance taxonomy, especially in comparison with traditional sensory-based vigils. Most interestingly, their results indicate that mental demand and effort appear to be the main contributors to workload in cognitive vigilance, as opposed to the traditional finding of the primacy of mental demand and frustration that are associated with workload in sensory-based vigilance tasks.

As we have noted, it was Warm who helped promulgate the idea that failures in vigilance were due to task overload and cognitive resource depletion. Other researchers, however, have suggested that underload and mindlessness are the cause of such deficits, as we have discussed earlier. To reexamine these competing propositions, Epling and colleagues used a dual-task methodology, where participants were asked to complete a vigil simultaneously with a memory task. Each task was also completed by itself. Mindlessness proponents may suggest that the novel stimulation provided by the subjectively interesting memory task would prevent underload, and thus mitigate performance impairment on the vigilance task. In addition, if vigilance is mindless, memory task performance should also not suffer in the dual-task condition. However, it was found that vigilance and memory performance were both impaired in the dual, compared with single-tasks; thus providing evidentiary support for resource theory. The extent of dual-task interference was also found to be less than that of a prior experiment using a more demanding executive memory task paired with the same vigilance task. This further implicates resource limitations as the underlying factor in such performance degradation.

The study by Neigel and colleagues investigated the role of task engagement in semantic vigilance performance, and evaluated associated stress, and workload. Results here demonstrated that as task engagement increases over the course of the vigil, so does the proportion of correct detections across each period on watch. The authors also showed that task engagement type may be related to the distress associated with performing this semantic vigilance task. However, this differentiation of engagement type was not reflected in the assessed workload. This research has implications for a further understanding of cognitive resource theory, and how it then applies in real-world semantic vigilance performance. The authors are all academic descendants of Warm (the first three authors were the students of the fourth author). They are each especially grateful for Joel’s contributions to the field of Humans Factors and their own progress therein.

The work of Greenlee, DeLucia, and Newton (2019) demonstrates the importance of driver vigilance in partially automated vehicles. These authors showed that, as in other tasks that require humans to monitor automation, operating a simulated automated vehicle induces a vigilance decrement in target detection. This means that drivers become less likely to respond safely to potential failures of vehicle automation as a drive proceeds. Greenlee and colleagues also manipulated signal processing demands associated with monitoring by varying both event rate and spatial uncertainty. They found that increases in monitoring demand led to more severe impairment in driver vigilance. These findings align with the overload explanation for the vigilance decrement that Warm strongly advocated. Thus, vigilance is grueling, stressful, and results in depletion of limited attentional resources. The application of these observations to the design and operation of such semi-automated vehicles is examined, evaluated, and explored. Although Joel Warm was not directly involved in this project, he nevertheless helped shape this research through his published vigilance research and his mentorship and collegiality with the research team. Both Greenlee and DeLucia were brought into the world of vigilance through previous collaboration with him. Newton is Professor Warm’s academic grandchild (as he himself would have said).

Using a simulation, Wohleber et al. (2019) investigated vulnerability to vigilance decrement in operators of multiple Unmanned Autonomous Systems (UAS). Multi-UAS operation is possible only with support from automation. The proximal human factors concern here is that operators’ calibration of dependence on automation can become impaired as cognitive fatigue develops. The present study showed the vigilance decrement on a cognitively demanding surveillance task, consistent with the resource theory of vigilance. This was accompanied by a decreasing automation-dependence over time. Neglect of the automation may result from task-shedding strategies, pointing toward the need to modify resource theory to explain operator behavior in these complex task environments. The research was inspired by the resource theory of vigilance developed by Warm and his colleagues. Joel effortlessly bridged the twin worlds of basic and applied research. In this spirit, the study addressed the extent to which principles underlying the vigilance decrement, established in basic research, generalize to a complex operational setting. Joel was also one of the first to highlight the risks of vigilance decrement in the then novel context of automated systems. The present study’s findings confirm operator vulnerability—though strategy changes associated with fatigue may provide novel insight and avenues of remediation here.

Conclusion

It is very evident from this introductory article in just what esteem Joel Warm was held by so many individuals. For decades, he was our leader in all things vigilance and a mentor to so many in our science. We have lost yet another giant of our field and a great source of personal inspiration. As the new generation of HF/E scientists come to the fore, they can look in admiration on the achievements of their forebears (see Hancock, 2008; Hancock, Baldwin, Warm, & Szalma, 2017), and seek to replicate and improve on the significant scientific understanding that such individuals have passed-on.