Abstract
Objective
This study investigates the cost of detection response task performance on cognitive load.
Background
Measuring system operator’s cognitive load is a foremost challenge in human factors and ergonomics. The detection response task is a standardized measure of cognitive load. It is hypothesized that, given its simple reaction time structure, it has no cost on cognitive load. We set out to test this hypothesis by utilizing pupil diameter as an alternative metric of cognitive load.
Method
Twenty-eight volunteers completed one of four experimental tasks with increasing levels of cognitive demand (control, 0-back, 1-back, and 2-back) with or without concurrent DRT performance. Pupil diameter was selected as nonintrusive metric of cognitive load. Self-reported workload was also recorded.
Results
A significant main effect of DRT presence was found for pupil diameter and self-reported workload. Larger pupil diameter was found when the n-back task was performed concurrently with the DRT, compared to no-DRT conditions. Consistent results were found for mental workload ratings and n-back performance.
Conclusion
Results indicate that DRT performance produced an added cost on cognitive load. The magnitude of the change in pupil diameter was comparable to that observed when transitioning from a condition of low task load to one where the 2-back was performed. The significant increase in cognitive load accompanying DRT performance was also reflected in higher self-reported workload.
Application
DRT is a valuable tool to measure operator’s cognitive load. However, these results advise caution when discounting it as cost-free metric with no added burden on operator’s cognitive resources.
Introduction
Understanding system operator’s cognitive load is one of the foremost challenges in human factors and ergonomics (HF/E) research. There is in fact an abundance of evidence showing that nonoptimal levels of cognitive load, that is, the “demand for cognitive control imposed by a task” (ISO, 2016), is detrimental to human and task performance. Excessive cognitive load or overload is observed under conditions of multitasking or when task demand exceeds system operator’s cognitive capacity. In contrast, cognitive underload may result from low levels of task demand requiring minimal attentional control from the system operator. Either condition results in poor performance and potential safety risk, with overload being often associated with distraction, and underload with vigilance decrement (see Biondi et al., 2019, for a review of the effects of nonoptimal cognitive load on driving).
While the effects of underload and overload on performance are undoubted, debate is present on how to accurately measure cognitive load. Unlike visual and motor workload which are tangible and therefore easier to measure (Strayer & Cooper, 2015), HF/E research has for long aimed to develop metrics which accurately measure cognitive load and can afford meaningful cross-studies comparisons.
The detection response task (DRT; ISO, 2016) is a paradigm developed by the International Organization for Standardization that quantifies dynamic changes in operator’s cognitive load resulting from attentional allocation to one or more tasks at hand. In the ISO paradigm, participants press a button following the presentation of a visual or tactile stimulus every 3–5 s. As the cognitive demand imposed by the task (or tasks) at hand increases, this results in longer response (RT) times and lower accuracy (or hit rate).
The conceptualization of the DRT is grounded in resource theories of attention. Kahneman (1973) posits that attentional capacity is limited, and task performance in a given task depends on the relationship between task demand and resource capacity. Within this framework, as task demand increases or exceeds attentional capacity, cognitive load increases as a result. For example, relative to a condition of single task driving, performing a concurrent task like talking on a cellphone will increase the combined demand of the two activities, therefore increasing the cognitive load on the driver.
Given the relative simplicity of this task whereby a simple detection response is required to the presentation of an individual intermittent stimulus, it is argued that DRT is the source of a minimal or no interference with the primary task. In the original ISO document, it is discussed that perceptual interference may arise when assessing predominantly visual activities (like driving) by means of using the visual version of DRT (i.e., where the DRT stimulus is visual). However, based on multiple resource theories of attention (Wickens, 2002), such intramodal interference is bound to decrease or fully disappear when using the tactile version of the DRT (i.e., where the DRT stimulus is tactile), which does not add onto the visual load of the driver. The tactile DRT is also posited to minimize other possible sources of sensory interference, especially in conditions where the primary task does not involve the processing of vibrotactile stimuli.
Despite the general agreement that DRT, and its tactile version in particular, does not impose additional perceptual or sensory load, a growing number of studies investigate whether the presence of DRT could add onto the cognitive load in multitasking scenarios. Palada et al. (2019) had participants complete a classification task where they were required to distinguish between targets and nontargets presented on a computer screen. This task was performed with or without the tactile DRT. DRT was sensitive to classification task difficulty, with poorer DRT performance found with greater task demand. When comparing classification task performance with or without DRT, however, a minimal yet significant effect was found with lower target detection accuracy observed with DRT relative to the condition without DRT. In a similar experiment, Castro et al. (2019) had participants complete a pursuit tracking task consisting in maintaining a cursor as close as possible to a target moving horizontally across the screen. The tracking task was performed with or without the visual DRT. Tracking accuracy was calculated as the distance between the cursor and the target. Performance in the tracking task was minimally affected by the DRT presence.
In this study, we attempt to validate the hypothesis that DRT does not produce attentional cost on cognitive load by utilizing an additional, nonintrusive metric of cognitive load: pupil diameter. Pupil diameter is sensitive to changes in cognitive task demand. In their comprehensive review on physiological measures of cognitive load, Lohani et al. (2019) show evidence supporting the notion that higher cognitive task demand results in larger pupil diameter. Experimental studies by Krejtz et al. (2018), Chen and Epps (2014), and Wright et al. (2013) also show dynamic changes in pupil diameter under conditions of varying cognitive load.
Additional reasons for selecting pupil diameter are that it does not require voluntary control and it should not tap onto the same resource pool used by the DRT. Following the hypothesis by Miller et al. (2005) whereby changes in pupil diameter result from dynamic adjustments by the autonomic nervous system innervating the sphincter and dilator ocular muscles, motor resources required by DRT are not expected to interfere with those involved in ocular motion. Further, because in this study we adopted the tactile version of the DRT, this should further minimize the risk of sensory interference.
In summary, we have two main objectives.
Objective 1. Examine the effect of greater cognitive load on DRT performance and pupil dilation combined. Based on previous studies, we expect larger pupil diameter and slower DRT responses under conditions of greater cognitive task demand.
Objective 2. Validate the hypothesis that DRT does not produce additional cost on cognitive load. By comparing pupil diameter with and without DRT performance, we expect no significant effect of DRT presence on pupil diameter.
Method
Participants
Twenty-eight volunteers (16 males) were recruited from the student and staff population at the University of Windsor. Their age ranged between 18 and 30 years (average = 22, standard deviation = 3). All participants had normal or corrected-to-normal vision and hearing. They received a $20 gift-card compensation for their participation. This research complied with the American Psychological Association Code of Ethics and was approved by the Research Ethics Board at the University of Windsor (#19–045). Informed consent was obtained from each participant.
Equipment, Data Acquisition, and Analysis
Eye-tracker and pupil diameter
A desktop-mounted Gazepoint GP3 eye-tracker with data collection frequency of 60 Hz was used. Previous researches show this as a reliable tool for HF/E research (Coyne & Sibley, 2016; Mannaru et al., 2017). The eye-tracker was placed below an AOC 27-in. screen with a resolution of 1,920 × 1,080 connected to a PC running Windows 10. The eye-tracker has a graphical user interface which was used for the calibration process. Pupil diameter was calculated in pixels. Normalized pupil diameter was calculated as the ratio between recordings in the n-back condition (0-back, 1-back and 2-back) and the mean pupil diameter recorded in the control condition with no n-back in pixels. Pupil diameter and normalized pupil diameter were analyzed. Recordings above 30 pixels and below 5 pixels were considered outliers and removed from the analysis.
DRT
The vibrotactile version of the DRT was manufactured by Red Scientific Ltd (Salt Lake City, UT, USA) as per ISO 17488 (2016). A vibrotactile motor was placed on the participants’ left collarbone area and a microswitch was attached to either the index or middle finger of the left hand. The vibrotactile motor emitted a short stimulus (1 s in duration), similar to a phone vibration. Upon its presentation which occurred every 3–5 s, participants were instructed to press the microswitch as fast as possible. RT in milliseconds and hit rates were recorded. RT were recorded as the time interval between the onset of the vibrotactile stimulus and the depression of the microswitch. Accuracy was calculated as the ratio between the numbers of hits and the number of total presented stimuli. In accordance to ISO guidelines, responses faster than 100 ms or longer than 2,500 ms were eliminated from the calculation. Nonresponses or responses produced later than 2,500 ms were considered as misses.
n-Back
The n-back task (Mehler et al., 2011), a widely adopted paradigm in HF/E research, was chosen to manipulate cognitive task demand. We adopted three versions of the n-back: 0-back, 1-back, and 2-back. The audio version of the n-back was considered with audio files being presented from an external audio player device. Audio files were sourced from http://agelab.mit.edu/delayed-digit-recall-n-back-task.
Performance in the n-back task was calculated as the proportional number of correct responses. For example, if participants produced the correct response in 50% of the trials and produced the wrong or no response in the remainder trials, accuracy for that specific n-back condition would be 50%.
NASA Task Load Index (NASA-TLX)
Self-reported mental workload was measured using the mental demand subscale of the NASA-TLX (Hart & Staveland, 1988). In its original version, the questionnaire consists of six 100-point scales measuring different components of workload. In our study, we decided to focus on the mental demand subscale only. Participants rated mental demand using an online version of the questionnaire developed using Google Forms.
Procedure and Design
A factorial design with DRT (two levels: present, absent) and n-back (four levels: control, no n-back; 0-back; 1-back; 2-back) as independent measures was considered. DRT RT and accuracy, pupil diameter (PD), and self-reported mental workload were the dependent measures.
Upon entering the laboratory, participants completed an intake survey with demographics. The familiarization phase then began. For the n-back, participants listened to an audio file with series of digits and were instructed to repeat aloud the last digit (0-back), the penultimate digit (1-back), or the third to last digit presented in the series (2-back). This lasted until they felt comfortable completing all versions of the n-back. Later on, they were instructed on how to complete the DRT task. They were also given sufficient time to familiarize with it until they felt comfortable performing the task without the assistance from the research assistant. Lastly, they were provided details on the eye-tracker and the calibration process. The eye-tracker required a 9-point calibration phase in which the participant was required to focus on the markers displayed at various locations on the screen. The calibration process lasted 1 min approximately.
After familiarizing with the protocol, data collection commenced. This phase lasted approximately 40 min. When DRT was present, participants completed four conditions: control (only DRT, no n-back), DRT + 0-back, DRT + 1-back, DRT + 2-back. When DRT was absent, participants completed three conditions with no DRT: 0-back, 1-back, 2-back. We did not include a condition with no DRT and no n-back. In total we had seven conditions which were counterbalanced across participants using a Latin square table. During each condition, participants were instructed to keep their gaze fixed on a black fixation cross presented over a white background at the center of the display located approximately 30 cm away. A chinrest was used to minimize the occurrence of gaze shifting caused by head movements. Each of the seven experimental conditions lasted approximately 5 min. At the end of each condition, participants completed the NASA-TLX (for a total of seven times), after which the next condition commenced.
Data were analyzed using R project for statistical computing (R Team, 2008).
Results
This section presents the results for DRT RT and accuracy, pupil diameter, mental demand subscale of the NASA-TLX, and n-back task accuracy. DRT presence (present vs. absent) and n-back condition (control, 0-back, 1-back, 2-back) are treated as independent variables. Parametric tests were run if the assumption of sphericity was not violated.
DRT
Mauchly’s test showed that the assumption of sphericity was not violated for DRT RT, W = 0.76, p > .05. A repeated-measure ANOVA with n-back (four levels) as independent factor was conducted. A significant main effect was found for DRT RT, F(3,81) = 10.82, p < .001. Post hoc t-test analysis revealed significant differences between control and 1-back, t(27) = 3.77, p < .001, 95% CI [591.18, 822.67], and control and 2-back, t(27) = 3.62, p < .001, 95% CI [605.38, 831.79]. See Figure 1 for DRT RT across the four conditions.
DRT RT across four n-back conditions. Error bars represent standard error. DRT = detection response task; RT = response times.
Similar analyses were conducted on DRT misses (Table 1). Mauchly’s test showed that the assumption of sphericity was not violated for DRT accuracy, W = 0.54, p > .05. A significant main effect of condition was found, F(3,81) = 9.36, p < .001, with proportions of misses increasing in conditions of greater cognitive task demand. t-Tests revealed a significant difference between 0 and 1-back, t(27) = 2.13, p < .05. No significant difference was found between 1-back and 2-back, t(27) = 1.55, p = .06. Nonparametric tests yielded comparable results.
Average and Standard Error (SE) of Proportions of DRT Misses Across n-Back Conditions
Note. DRT = detection response task.
Pupil Diameter
Mauchly’s test showed that the assumption of sphericity was not violated for pupil diameter, W = 0.86, p > .05. A repeated-measure ANOVA with DRT (two levels) and n-back (three levels) independent factors was conducted. Nonnormalized pupil diameter was used as dependent measure. Significant main effects of DRT, F(1,27) = 42.59, p < .001, and n-back, F(2,54) = 28.78, p < .001, were found. No significant interaction was found.
Post hoc t-tests were conducted to reveal differences between the two conditions with and without DRT across n-back tasks. A significant difference was found for 0-back, t(27) = 4.71, p < .001, 95% CI [13.62, 15.38], with pupil diameter increasing in size with the addition of DRT (M noDRT = 13.95 pixels, M DRT = 15.04 pixels, Δ0-back = 1.09 pixels). Similar results were found for 1-back (M noDRT = 14.18 pixels, M DRT = 15.96 pixels, Δ1-back = 1.78 pixels), t(27) = 5.61, p < .001, 95% CI [14.15, 16.01], and 2-back (M noDRT = 15.73 pixels, M DRT = 16.93 pixels, Δ2-back = 1.20 pixels), t(27) = 3.69, p < .001, 95% CI [15.49, 17.16]. See Figure 2 for pupil diameter across n-back and DRT conditions.
Pupil diameter across three n-back conditions. Error bars represent standard error. DRT = detection response task.
Post hoc t-tests within the two DRT conditions revealed significant differences between 0-back and 2-back with DRT-present, t(27) = 5.17, p < .001, 95% CI [15.11, 16.87], ΔDRT = 1.88 pixels, and DRT-absent, t(27) = 7.21, p < .001, 95% CI [13.94, 15.73], Δno-DRT = 1.76 pixels.
Taken together, these data indicate that an increase in pupil diameter between 1.09 and 1.78 pixels was observed when participants completed the DRT relative to the no-DRT condition. A similar increase of approximately 1.77 pixels was found when transitioning from a relatively simple task like the 0-back to a taxing task like the 2-back.
Results found using normalized pupil diameter reflected that observed for nonnormalized data.
NASA-TLX
Mauchly’s test showed that the assumption of sphericity was not violated, W = 0.58, p > .05. A repeated-measure ANOVA with DRT (two levels) and n-back (three levels) as independent factors was conducted. Significant main effects of DRT, F(1,182) = 10.34, p < .001, and n-back, F(1,156) = 10.06, p < .001, were found. No significant interaction was found, F < 0. See Table 2 for mental demand ratings.
Average and Standard Error (SE) of Mental Demand Ratings Across n-Back With DRT-Present and -Absent
Note. DRT = detection response task.
Post hoc t-tests revealed significant differences between DRT and no-DRT for 0-back, t(27) = 5.15, p < .001, 95% CI [17.64, 31.10]; 1-back, t(27) = 4.92, p < .001, 95% CI [36.88, 52.04]; and 2-back, t(27) = 2.64, p < .05, 95% CI [62.37, 75.84].
n-Back Accuracy
Mauchly’s test showed that the assumption of sphericity was not violated, W = .88, p > .05. Repeated-measure ANOVA with DRT (two levels) and n-back (three levels) as independent factors were conducted for each individual scale. Significant main effects of DRT, F(1,27) = 28.03, p < .001, and n-back, F(2,54) = 90.72, p < .001, were found. A significant interaction was also found, F(2,54) = 18.98, p < .001. This indicates that n-back accuracy suffered under greater n-back task demand, and, more interestingly, in the presence of DRT (Table 3). Nonparametric tests yielded comparable results.
n-Back Accuracy Scale Across n-Back and DRT Conditions
Note. DRT = detection response task.
Discussion
In this study we set out to (1) examine the effect of greater cognitive task demand on DRT performance and pupil diameter and (2) validate the hypothesis that performing the DRT does not add to the overall cognitive load.
For objective 1, longer DRT RT found with greater cognitive task demand were associated with larger pupil diameter. Also, consistent with the studies by Harbluk et al. (2013) and Bruyas and Dumont (2013), average DRT RT increased by about 200 ms when performing the 2-back, relative to the control condition with no n-back task. Note, however, that average control RT appeared to be longer than in other studies. Despite differences in absolute RT being contemplated in ISO (2016), this is possibly the result of the different overall task demand experienced in our experiment.
Pupil diameter data showed an average increase of 1.88 pixels in the 2-back condition, relative to when performing the 0-back. This is consistent with current literature showing larger pupil diameter under conditions of greater cognitive task demand. Using similar equipment, Chen and Epps (2014), for instance, found changes in pupil diameter of about 1 pixel when transitioning from a task with low cognitive task demand to one requiring greater cognitive resources. Similar results were found by Wong and Epps (2016) where a 1- to 2-pixels difference in pupil diameter was found between tasks with different task demands. Greater cognitive task demand also increased self-reported mental workload and reduced DRT accuracy.
More interestingly, for objective 2, a significant main effect of DRT was found for pupil diameter, with a consistent increase in pupil diameter found with DRT-present relative to when no DRT was performed. The increase in cognitive load found with DRT was also supported by self-reported ratings of mental workload and n-back performance, with higher ratings and lower n-back performance found in the DRT-present condition relative to the one without DRT across the three n-back tasks. These results are inconsistent with that found in the study by Palada et al. (2019), in which authors found no practical difference in the primary classification task performance in the presence of DRT. Also, while the decrease in tracking task performance found with DRT was minimal in the study by Castro et al. (2019), here the increase in pupil diameter found with DRT (1.09 to 1.78 pixels) was comparable to that observed when transitioning from the relatively easy 0-back to the highly taxing 2-back task (1.77 pixels). This result is particularly interesting since it conflicts with the afore-discussed hypothesis that DRT has a negligible, if any, cost on cognitive load.
Support to the hypothesis that DRT has no cost on cognitive load comes from the studies by Stojmenova and Sodnik (2018a), Stojmenova & Sodnik, 2018b). In their comprehensive analysis on uses and limitations of DRT, Stojmenova and Sodnik (2018a) address the issue of DRT interference by discussing the results from their driving simulator study (2Stojmenova & Sodnik, 2018b. In this experiment, participants were asked to drive a simulated vehicle (primary task), while completing the n-back task (secondary task). DRT presence was also manipulated to examine its effect on pupil diameter and primary- and secondary-task performance. Completing the DRT resulted in worse speed maintenance and, similar to that found in our study, lower n-back performance. Larger pupil diameter was also found in the presence of DRT, but the difference was not significant.
The absence of a significant effect of high cognitive load on larger pupil diameter seems at odd with the results observed by Stojmenova and Sodnik (2018b) for primary and secondary-task performance and the broader literature on the effect of dynamic changes of high cognitive load on pupil diameter (Chen & Epps, 2014; Krejtz et al., 2018). In particular, if we consider the simulated highway task selected in their study (Stojmenova & Sodnik, 2018b) as a more automated activity requiring low attentional resources, and the n-back as a more demanding task requiring a deliberate attentional control, it is surprising how pupil diameter was immune to the purported increase in cognitive load brought upon by the concurrent DRT performance. A reconciling explanation could be found in Lavie’s load theory of attention (Lavie et al., 2004). In her treatise, Lavie argues that the magnitude of interference produced by task-irrelevant stimuli reduces with greater perceptual load. In other words, when comparing conditions of low and high perceptual load—completing a task in a controlled experimental environment versus performing the same task while driving, for example—the greater perceptual load experienced in the latter minimizes or eliminates the interference produced by task-irrelevant information on the primary task. This is due to the operational dynamics of the passive perceptual selection mechanism whose capacity to process less relevant information is exhausted in high load scenarios (but which still processes even secondary stimuli in conditions of low load). Such interpretation would help reconcile our results with the studies by Palada et al. (2019) and Castro et al. (2019) whereby a minimal although significant decline in primary task performance was found when DRT was present.
Taken together, the findings from our and other studies suggest that DRT performance might in fact have a cost on system operator’s cognitive load. While our intention is not to detract from its merit and value in HF/E research, we advise further investigation be conducted before discounting it as a cost-free measure of cognitive load. It is purported that cognitive cost of DRT might tack on operator’s resource capacity only in conditions of low perceptual load (Lavie et al., 2004), and decrease in more complex scenarios. This hypothesis will be the subject of future scrutiny.
Key Points
We tested the hypothesis that DRT has no cost on cognitive load by using pupil diameter as an alternative metric of cognitive load.
Larger pupil diameter was found in the presence of DRT, relative to the conditions with no DRT
Higher self-reported workload and worse n-back task performance were also found with DRT.
Results warn caution when discounting the DRT as cost-free metric of cognitive load
Footnotes
Acknowledgment
The authors thank the personnel at the Department of Kinesiology in the University of Windsor and acknowledge the financial support from the University of Windsor-SSHRC Explore grant, Mitacs, and Atlas Copco.
Author Biographies
Francesco N. Biondi is an assistant professor at the Department of Kinesiology in the University of Windsor. He received his master’s degree in experimental psychology in 2011, and his doctorate in experimental psychology in 2015 from the University of Padova, Italy. He holds academic appointments at the Department of Civil Engineering in the University of Windsor and at the Department of Psychology in the University of Utah.
Balakumar Balasingam is an assistant professor at the Department of Electrical and Computer Engineering in the University of Windsor. He received his PhD in electrical engineering from McMaster University in 2008.
Prathamesh Ayare received his MS in electrical engineering from the University of Windsor in 2019.