An Investigation of Speech Features,Plant System Alarms,and Operator–System Interaction for the Classification of Operator Cognitive Workload During Dynamic Work

Abstract

Objective

To investigate speech features, human–machine alarms, and operator–system interaction for the estimation of cognitive workload in full-scale realistic simulated scenarios.

Background

Theories and models of cognitive workload are critical for the design and evaluation of human–machine systems. Unfortunately, there are very few nonintrusive cognitive workload measures available for realistic dynamic human–machine interaction.

Method

The study was conducted in a full-scope control room research simulator of an advanced nuclear reactor. Six crews, each consisting of three operators, participated in 12 scenarios. The operators rated their workload every second minute. Machine learning algorithms were trained to estimate operators’ workload based on crew communication, operator–system interaction, and system alarms.

Results

Random Forest (RF) utilizing speech and system features achieved an accuracy of 67% on test data. Utilizing speech features only, the accuracy achieved was 63%. The most important speech features were pitch, amplitude, and articulation rate. A 61% accuracy was achieved when alarms and operator–system interaction features were used. The most important features were the number of alarms and amount of operator–system interaction. Accuracy for algorithms trained for each operator ranged from 39% to 98%, with an average of 72%. For a majority of analyses performed, RF and extreme gradient boosting (XGB) outperformed other algorithms.

Conclusion

The results demonstrate that the features investigated and machine learning models developed provide a potential for the dynamic nonintrusive measurement of cognitive workload.

Application

The approach presented can be developed for nonintrusive workload measurement in real-world human–machine applications, simulator-based training, and research.

Keywords

cognitive workload dynamic measurement nonintrusive machine learning simulator research

Introduction

The measurement of cognitive workload during human–machine interaction is a topic of considerable interest for design, training, and evaluation (Gopher & Donchin, 1986; Hart & Staveland, 1988; International Organization for Standardization [ISO], 2006; Reinerman-Jones et al., 2019; Young et al., 2015). Postsession assessment of workload has been popular due to the ease of application (Grier, 2015; Reid & Nygren, 1988). However, there is a need to understand the dynamic interaction between workload and performance, and to investigate how operators and teams manage and adapt to cognitive workload (Hancock, 2017; Moray, 1988; Rusnock & Borghetti, 2018; Vicente, 1997). For example, it is challenging to detect and understand tipping points where cognitive workload relates to performance impairment (Chen et al., 2012; Hancock, 2017). Research using relatively simple tasks and highly controlled settings has established the potential of estimating cognitive workload using psychophysiological and behavioral features (e.g., Humphrey & Kramer, 1994; Yap et al., 2010; Yin et al., 2007), for example, by training machine learning algorithms (Wilson & Russell, 2003). While these studies demonstrate a potential and provide important background for unobtrusive and continuous measures, the extent to which such approaches can be used to estimate operators’ cognitive workload during realistic human–machine work is unclear. Investigating the measurement of workload during realistic work is especially challenging because the team’s strategy, interactive behavior, and timing of critical actions might strongly influence the overall task load, the distribution of load within the team, and the data available for workload estimation (Vukovic et al., 2019).

Dynamic Estimation of Cognitive Workload

What approaches might be suitable for realistic human–machine studies and field applications? Studies on real-time workload measurement include the use of eye behavior (Ahlstrom & Friedman-Berg, 2006; Schulz et al., 2011), heart rate, and heart rate variability (Kennedy & Scholey, 2000), brain activity (Humphrey & Kramer, 1994; Wilson & Russell, 2003), galvanic skin response (Shi et al., 2007), electroencephalographic signals, and breathing rate (Ferreira, et al., 2014). Although recent research has investigated less intrusive methods for some of these metrics (e.g., Rahman et al., 2020), these methods tend to require equipment that may interfere with the operator’s tasks and controlled conditions not easily implemented for realistic human–machine work. Dynamic workload measurement approaches using speech features (Brenner, et al., 1985; Lively et al., 1993; Vukovic et al., 2019), system variables (Rahman et al., 2020), and user interaction (Chen et al., 2012) can be applied in a nonintrusive manner, and may be more suitable for workload measurement in realistic settings. We have therefore focused on the latter type of features in our study.

Several studies suggest that fundamental frequency, vocal intensity, and speaking rate are among the speech features most sensitive to cognitive load and are therefore appropriate candidates for workload estimation (Brenner et al., 1994; Huttunen, Keränen, Väyrynen, et al., 2011, Huttunen, Keränen, Pääkkönen, et al., 2011; Wittels et al., 2002). We have, therefore, included these features in our study. The fundamental frequency (f0), defined as the rate at which the vocal folds open and close (Lieberman, 2007), may be influenced by stress and level of arousal (Brenner et al., 1994; Lively et al., 1993; Luig & Sontacchi, 2010). Speaking rate has been associated with an acceleration of cognitive and motor processes, often emerging under stress (Brenner et al., 1994), and increased vocal intensity has been associated with stress (Brenner et al., 1994; Luig & Sontacchi, 2010). Also, structural aspects, such as time spent pausing, may reflect an increased level in cognitive processing (Esposito et al., 2007; Lin et al., 2013).

Research on user–system interaction has found correlations between workload and several behavioral features related to interaction with the user interface, for example, frequency of pauses and changes of direction in mouse movements (Arshad et al., 2013; Lin et al., 2013), number of air traffic controllers’ flight clearances (Tobaruela et al., 2014), and mouse and keyboard activities (Pimenta et al., 2016). Braarud and Kirwan (2011) reported information intensity in terms of number of alarms as a factor of human–machine task load. Alarms require operators’ attention and aid the team in initiating tasks in response (Wu & Li, 2013). To our knowledge, no research has been published on using alarms as an indicator of workload related to operators’ main tasks, although the related topic of alarm fatigue due to false or nuisance alarms has been researched within hospital settings (e.g., Cvach, 2012; Ruskin & Hueske-Kraus, 2015).

A question with no clear answer in the literature is the unique contribution from different types of workload indicators and the accuracy that can be achieved from combining these types. In general, combining different measurement methods may compensate for the limitations of each individual method (Cegarra & Chevalier, 2008). For example, Chen et al. (2012) combined speech features and user interaction to predict workload. Solovey, et al. (2014) found that including physiological data gave higher algorithm classification accuracy than vehicle sensor data alone. On the contrary, McDonald et al. (2020) reported that combining physiological data and driver behavior data did not improve classification accuracy above the accuracy achieved with driver behavior data alone.

Application of Machine Learning Algorithms

Training machine learning algorithms for dynamic workload measurement have been reported in the literature (Vukovic et al., 2019; Wilson & Russell, 2003). However, the literature provides limited support for selecting adequate machine learning algorithms for cognitive workload estimation; hence, we aimed to include a set of commonly applied algorithms to ensure that an adequate algorithm would be included in the study. Machine learning algorithms reported include k-nearest neighbor (KNN; Solovey, et al., 2014), Naïve Bayes (NB; Solovey, et al., 2014), neural networks (NNs; Wilson & Russell, 2003), Random Forest (RF; McDonald et al., 2020; Ragab et al., 2014), support vector machines (SVMs; McDonald et al., 2020; Vukovic et al., 2019), and extreme gradient boosting (XGB; Momeni et al., 2019). Results from comparing algorithms’ performance are scattered, but there is some evidence that RF is the highest performing algorithm for driver distraction classification (McDonald et al., 2020; Ragab et al., 2014). Other results include Momeni et al. (2019), who found XGB to outperform a set of algorithms in their classification of cognitive workload; Solovey, et al. (2014) found decision trees, logistic regression, multilayer perceptrons, and NB to be equally good algorithms for classifying driver workload, while KNN performed significantly worse.

An important question for algorithm performance is the relevance of the training data. Individual differences have been identified in the experience of cognitive workload (Braarud, 2020; Guastello et al., 2015; Matthews et al., 2015), and there are individual differences regarding most speech features. Algorithm training on data from the person to estimate workload for is referred to as person-dependent, while algorithm training on data from other individuals is referred to as person-independent (Chen et al., 2012). Earlier research suggests that person-dependent models should outperform person-independent models (van Segbroeck et al., 2014; Vukovic et al., 2019). In addition, in human–machine systems, the team is a relevant unit of analysis (Hutchins, 1995) and may also be a relevant unit for training algorithms. There might be patterns of speech within the team, and activities by one operator might impact the other team members’ workload.

Study Purpose

Existing literature provides limited insights into how to continuously measure cognitive workload in a way that is applicable to realistic human–machine work. The extent to which the techniques studied in relatively controlled settings can provide tenable results is unclear, as is the answer to the question “which are the most suitable machine learning techniques (algorithms) for this kind of estimation?” Furthermore, there is uncertainty about the combined effect of different types of features and their relative importance. To address this gap in the literature, this study investigated the potential of using nonintrusive features from speech, operator–system interaction, and system alarms for cognitive workload estimation in a full-scope research simulator. Teams of licensed control room operators, guided by authentic operating procedures, performed in realistic accident scenarios that were relatively demanding. The operators’ self-labeled cognitive workloads were used for supervised training of machine learning algorithms. We investigated the performance of six commonly applied algorithms in order to identify the best performing model. We applied the best performing model to provide insights into the approach’s efficiency for cognitive workload estimation, and we investigated the combined and relative importance of the different types of features. In addition, the study explored three different approaches to training data for machine learning estimation of cognitive workload.

Method

Participants

The Organisation for Economic Co-operation and Development (OECD) Halden Reactor Project (OECD NEA, 2020) established an agreement with a plant with a reactor design similar to the one simulated at the research site. Six control room crews from the plant volunteered for participation in the study, which lasted 3 working days for each crew. Each crew consisted of a supervisor, a reactor operator, and a turbine operator, all being male licensed operators. Their mean age was 39.2 years (SD = 11.9), ranging from 26 to 62 years, and their mean control room work experience was 10.6 years (SD = 10.8), ranging from 1 to 35 years. Five crews consisted of team members that work as a crew at their home plant, while one crew was assembled from different home plant crews. The study was reviewed and approved by the Halden Reactor Project Human Studies Review Committee, and was performed according to Halden Reactor Project’s human participant protection procedures.

Simulator and Scenarios

The study was performed at the Institute for Energy Technology’s (IFE’s) Halden Man-Machine Laboratory (HAMMLAB; Institute for Energy Technology [IFE], 2019), which is a full-scope research simulator based on an advanced nuclear power plant. The simulator has a fully computerized human–machine interface. Figure 1 shows the control room layout. The shift supervisor workstation is at the back, the reactor operator workstation is to the left, and the turbine operator workstation to the right. The large screen display at the front provides a plant overview.

Figure 1

The Halden Man–Machine Laboratory control room.

The simulation and test scenarios were controlled from an experimenters’ gallery adjacent to the control room. Two subject-matter experts implemented and controlled the scenarios according to a scenario script and role-played the plant staff external to the control room, including field operators, technical support, and plant management. The control room team’s orders to field operators were thus implemented in the simulator by the subject-matter experts during the scenario. The control room team communicated with external staff by telephone, as they would at their home plant. Both subject-matter experts were former shift supervisors and training instructors with extensive knowledge of the simulated plant and the study’s specific scenarios. The subject-matter experts were involved in the scenario design as members of the research team.

Each team participated in 12 scenarios (Supplemental Material provides an overview of each of the 12 scenarios). Scenarios 1 to 8 were designed to last for 15–25 min. Due to a second topic of the study regarding human factors validation methods (Braarud et al., 2020), Scenarios 9 to 12 were designed to last 30–40 min. Even though the duration of scenarios varied, the scenarios comprised similar types of operator behavior and thus provided a consistent set of data for the investigation of cognitive workload estimation. The general scenario structure consisted of initial work, comprising normal operation or periodic testing including minor plant system failures and aggravation deviations leading to actuation of plant safety functions (reactor protection systems). The scenarios included events commonly included in nuclear power plant safety analysis, such as loss of reactor coolant accidents, loss of all offsite power, loss of turbine condenser, and loss of main feedwater. The team applied event-based procedures to acquire an overview of the plant’s safety status and to develop a basis for selecting a strategy to mitigate the situation and control plant safety.

In order to make parts of scenarios complicated and cognitively demanding, malfunctions were simulated for a number of safety system components. In these scenarios, safety components did not take the correct position after automatic actuation. While individual component failures like these can occur several times during a plant’s lifetime, the combination of the main event and multiple safety component malfunctions makes the specific scenarios unlikely. Nevertheless, the general type of scenario was not unfamiliar to the operators because similar scenarios are included in their annual retraining. The subject-matter experts stated that the workload imposed by the scenarios were within the range experienced during simulator training. The teams were instructed to use work routines and procedures as they ordinarily would in simulator training at home. This included routines for communication, teamwork, and interacting with plant staff external to the control room.

Operator Workload Rating

Each operator was instructed to self-assess his cognitive load every second minute using a six-point rating scale. The scale applied in the study was developed specifically for the evaluation of control room work and was based on the concepts of spare capacity (Gopher & Donchin, 1986; Moray, 1967; Wickens & Tsang, 2015) and adaption to task demand (Hancock & Warm, 1989; Hockey, 1997; Moray, 1988; Rusnock & Borghetti, 2018; Vicente, 1997). A 2-min interval for periodic self-assessment was chosen based on the subject-matter experts’ evaluation of the structure and timing of the control room work. Also, Jordan (1992) suggested 2-min intervals for similar types of application. Table 1 shows a description of the scale values. This description was available at each control room workstation as a part of the operators’ instructions for workload rating.

Table 1

Description of Workload Scale Levels

Scale Value	Label	Spare CapacityAvailable for Additional Tasks, Additional Verification, and/or Discussions	Adaption
6	Extremely high	None	Although adaption is performed, important tasks cannot be performed properly or completed.
5	Very high	None	Manageable by postponing less important tasks or by adapting the “general” work routines and working methods.
4	High	Very little	Managed within the “general” work routines and procedures established.
3	Medium	Some	Managed within the “general” work routines and working methods.
2	Low	Much	Managed within the “general” work routines and working methods.
1	Very low	Very much	Managed within the “general” work routines and working methods.

The rating scale was implemented on tablet devices positioned on the participants’ desks. The tablets provided an audio signal every second minute for the participants to rate workload.

In addition to the workload rating during the scenario, the NASA Task Load Index (Hart & Staveland, 1988) was administered postsession for each scenario. The study utilized an unweighted version, often referred to as the raw TLX (Byers et al., 1989; Nygren, 1991).

Session Recording

The simulator sessions were recorded using the laboratory’s data collection systems. Each operator wore a headset with a microphone. The recordings included separate audio tracks for each operator, the simulated plant’s scenario history (alarms, process parameters, and process events), operator process commands, navigation and interfaces accessed by the operator, video of each operator workstation, and an overview video of the control room.

Speech Feature Extraction

Speech features were extracted from the session audio recordings using the Praat tool (Boersma & van Heuven, 2001). The Praat library was run from Python using the Parselmouth application programming interface (Jadoul et al., 2018). Praat was chosen as the accuracy is on par with other popular acoustic analysis software packages (Burris et al., 2014) and the application programming interface enables convenient use of the extraction algorithms from Python. The parameter settings used were: silence threshold, −20 dB; minimum dip between peaks, 2 dB; minimum pause duration, 0.3 s; and time step, 0.01 s. A small sample of audio recordings was manually checked to verify that the algorithm’s count of syllables and pauses was correct, and that noise or other disturbances were not influencing the feature extraction. The fundamental frequency range was set to exclude periods without speech when calculating mean values. Table 2 presents the features extracted. An aggregated, or mean, value for f0 was calculated for each operator for every 2-min segment of the scenario corresponding to the workload rating times.

Table 2

Speech Features Extracted

Speech Features	Description
Syllables	The number of syllables
Pauses	The number of pauses
Rate of speech	The number of syllables divided on the length of the segment
Articulation rate	The number of syllables divided on the speaking duration
Speaking duration	The speech duration
Amplitude mean	The mean amplitude of the segment
Amplitude max	The maximum amplitude of the segment
f0 mean	Mean fundamental frequency in the range 70–200
f0 Std Dev	Standard deviation of f0
f0 median	Median of f0
f0 minimum	Minimum vale of f0
f0 maximum	Maximum value of f0
f0 lower quartile	25% quartile of f0
f0 upper quartile	75% quartile of f0

Plant Alarms and Operator Activity Extraction

The identification of plant alarm and operator activity features was discussed with the two subject-matter experts on control room work and based on the available variables in the simulator logs as well as the design of the human–system interface (HSI). The initial discussions considered using the behavior of plant process parameters and the status of safety functions as task demand indicators. However, it was noted that plant alarms already cover status and encapsulate easily accessible variables covering the reason for requiring the operator’s attention. Alarms were therefore chosen as indicators of incoming task load. We identified three categories of features that we assumed to be related to the operators’ cognitive workload: incoming alarms and the acknowledgment of these, interface navigation, and commands directly manipulating plant systems. Most of the simulator’s alarms were specified as either belonging to the scope of the Reactor Operator or the scope of the Turbine Operator, while the remaining alarms were not specified for a specific operator. For our purpose, the former was defined as “operator alarms” and the latter as “crew alarms.” Regarding HSI activity, the calculation included all types of operator navigation recorded by the simulator’s logging system. Therefore, all plant alarms and operator activity features were extracted from the simulator logs and calculated as counts of occurrence during each scenario segment. Table 3 presents the features extracted from the simulator session recordings. A value was calculated for each operator for every 2-min scenario segment.

Table 3

System Features Extracted

System Features	Description
Operator alarms	The number of activated alarms, specifically for each control room position.
Crew alarms	The number of activated alarms common for all crew members.
Acknowledged operator alarms	The number of operator alarms acknowledged by an operator.
Acknowledged crew alarms	The number of crew alarms acknowledged by a crew member.
Human–system interface (HSI) activity	The number of operator interface manipulation such as navigating to process images, selecting objects, and entering values.
Commands	The number of commands to plant system objects and system functions, such as start, stop, open, and close.

Data Pre-Processing, Algorithm Training, and Evaluation

For the classification algorithms, the six-point workload scale was aggregated to three levels, representing low, medium, and high cognitive workload. Currently, much of the reported research on machine learning classification of cognitive workload seems to be limited to two (Vukovic et al., 2019) or three (Charfuelan & Kruijff, 2013; McDonald et al., 2020; Yap et al., 2010; Yin et al., 2007) levels. A six-level classification is more challenging than a three-level classification. To be able to compare the study’s results with results reported in the literature, we chose to aggregate the scale to three levels for the machine learning classification. To produce labeled speech and plant system data, each 2-min period of speech and plant system development prior to the workload rating was labeled with the corresponding self-assessed workload.

All algorithms were trained in R (R Core Team, 2019) using the “caret” package (Kuhn, 2008; Kuhn et al., 2019) as the framework. The R caret package was used to split data into training and test sets, normalizing all features and controlling the algorithm training. Table 4 presents the six machine learning algorithms that were selected based on approaches commonly reported in earlier literature and our awareness of commonly applied classification algorithms within data sciences. The NN algorithm applied was limited to one hidden layer. The table also shows the parameters tuned for each algorithm. The parameters are described in the references provided in Table 4.

Table 4

Machine Learning Algorithms Applied

Abbreviation	Machine Learning Algorithm	Reference	R Parameters Tuned	Parameter Description
KNN	K-nearest neighbors	Venables and Ripley (2002)	k	The number of neighbors
NB	Naïve Bayes	Majka (2019)	usekernel	Distribution type for numeric predictors
			laplace	Laplace smoothing
			adjust	Bandwidth adjustment
NN	Artificial Neural Network	Ripley and Venables (2016)	size	Number of units in the hidden layer
NN	Artificial Neural Network	Ripley and Venables (2016)	decay	Weight decay
RF	Random Forest	Liaw and Wiener (2018)	mtry	Number of randomly selected predictors
SVM	SVM with Linear kernel	Karatzoglou et al. (2019)	C	Cost
XGB	Extreme gradient boosting	Chen et al. (2019)	Gamma	Regularization
			nrounds	Number of boosting iterations
			max_depth	Maximum depth of the tree
			eta	Shrinkage
			colsample_bytree	Subsample ratio of columns
			min_child_weight	Minimum sum of instance weight
			subsample	Subsample percentage

The data were partitioned into algorithm training and algorithm test sets by the caret “createDataPartitionfunction.” This function performed random partitioning within the levels of the outcome variable, balancing the class distributions of the operators’ workload rating within the training and test sets. The split was set to 80% training data and 20% test data, except for the training and evaluation of person-dependent models. Person-dependent models utilized the data for a given operator, so in order to obtain a reasonable number of test cases, the split was set to 70% training and 30% test data. The data were centered and scaled using caret functionality. To identify the best model, models were trained using 10-fold cross-validation (James et al., 2013) repeated five times with accuracy as the training metric. However, that approach would inflate the degrees of freedom and increase type I error if used to compare models statistically. Therefore, for model comparison based on model variability, models were trained using twofold cross-validation repeated five times (Dietterich, 1998). Models were tested by estimating cognitive workload from the withheld test data. A model’s accuracy was determined by comparing the estimated classification with the operators’ actual workload classification. Accuracy ranged from 0% to 100% correct classification.

Results

As described in the previous sections, we used different algorithm training approaches for identifying the best model and for statistically comparing model performance. Please note, therefore, that the results from these two approaches are not directly comparable.

Overview of Operators’ Cognitive Workload Classification

The operator’s self-rating of cognitive workload during the 12 scenarios resulted in a total of 2780 labeled 2-min scenario segments: 1202 segments were labeled low, 1406 were medium, and 172 were high cognitive workload. The occurrences of workload categories were unbalanced with relatively few high workload segments.

Table 5 shows that the operators rated scenarios segments quite differently. For example, the Crew 1 Supervisor rated 20% of the segments as low workload and 70% of segments as medium, while the Crew 6 Supervisor rated 97% of segments as low and 3% as medium workload. The observed differences might result both from individual differences in the experience of cognitive workload and from the unique progression of the scenarios due to the crew’s strategy and timing of actions.

Table 5

Counts of Self-Rated Cognitive Workload Segments

	Segments(n = 2780)	Proportion of Categories
	Segments(n = 2780)	Low(n = 1202)	Medium(n = 1406)	High(n = 172)
Crew 1
Reactor Op.	131	0.50	0.47	0.03
Supervisor	131	0.20	0.70	0.10
Turbine Op.	131	0.97	0.03	0.00
Crew 2
Reactor Op.	152	0.31	0.43	0.26
Supervisor	152	0.32	0.68	0.00
Turbine Op.	152	0.63	0.37	0.00
Crew 3
Reactor Op.	144	0.18	0.32	0.50
Supervisor	143	0.24	0.73	0.02
Turbine Op.	144	0.13	0.78	0.09
Crew 4
Reactor Op.	153	0.57	0.43	0.00
Supervisor	153	0.70	0.30	0.00
Turbine Op.	153	0.36	0.64	0.00
Crew 5
Reactor Op.	164	0.16	0.79	0.05
Supervisor	170	0.35	0.65	0.00
Turbine Op.	167	0.35	0.53	0.12
Crew 6
Reactor Op.	180	0.57	0.43	0.00
Supervisor	180	0.97	0.03	0.00
Turbine Op.	180	0.24	0.76	0.00

Note. Op. = Operator.

To compare the operators’ classification of workload during the scenario with the postsession NASA TLX rating, we averaged the 2-min workload ratings to produce a scenario average score. For a simple comparison, we rescaled workload scores by dividing the score by the respective scale length, providing a proportional score ranging between 1/scale length to 1. For the TLX rating, we calculated the average of all six TLX items. Figure 2 shows the TLX average and the online average per scenario.

Figure 2

Operators’ TLX average and online workload rating average. Mean and standard deviation. TLX = Task Load Index.

Figure 2 shows that the mean ratings for the TLX and online ratings follow a similar pattern from Scenarios 1 to 8. The average online ratings were slightly above or at a similar level as the mean TLX rating. The standard deviation of both the TLX ratings and the online rating reveals substantial individual differences. This corresponds to the literature regarding individual difference in the experience of cognitive workload (Braarud, 2020; Guastello et al., 2015; Matthews et al., 2015). The correlation between the TLX average and the average online workload rating was r = .55 (p < .001). The NASA TLX is a postsession evaluation based on the operator’s perception and memory of the whole scenario while the online rating focuses on the immediate perception of 2-min scenario segments. Some level of disagreement between these two measures must therefore be expected. Based on the similar pattern of means and individual variation, along with the moderate correlation between the two measures, we find that the online rating measures captured an element of cognitive workload in the control room.

Overview of Extracted Features

This section provides the features’ distribution across the operator-rated cognitive workload levels. For brevity, we selected the five most important speech features and three most important system features identified in the “Variable Importance” section later. The uneven selection, five versus three, was based on the study including 14 speech features but only six system features. Figure 3 shows the feature distribution for the reactor operator, supervisor, and turbine operator positions, respectively. The features were scaled with a minimum-maximum range suitable for non-normal distribution.

Figure 3

Dispersion of eight features aggregated for each operator position.

Figure 3 suggests several hypotheses regarding how the features relate to levels of cognitive workload; however, the development of such hypotheses is outside the scope of this paper. Two caveats apply regarding the interpretation of the dispersions presented in Figure 3. The operators classified few segments as high cognitive workload and consequently the dispersion of features for high cognitive workload is relatively more uncertain than for low and medium cognitive workload. Due to substantial individual differences, the development of hypotheses about features should include an investigation of person-level data beyond the aggregated visualizations per position presented here.

Classification of Workload: Identification of Best Performing Algorithm

The six machine learning algorithms were trained using each of the three input data types (speech, system data, or combined), resulting in 18 model types. For each model type, the best model was selected. The accuracy of a given model was determined by using it to classify the 20% data withheld. Figure 4 shows the models’ accuracy grouped by data type along with 95% confidence intervals. The 95% interval is a binomial proportion confidence interval, which is based on the accuracy resulting from the binomial proportion of hits (correct prediction; Brown et al., 2001).

Figure 4

Machine learning models’ accuracy on test data. Black lines represent 95% confidence intervals.

The differences in accuracy on test data were small between the models trained on speech data and models trained on system data. For the best performing algorithm (RF), accuracy achieved with speech data were 63% while accuracy achieved with system data were 61%. Training the algorithms on both speech and system data (combined) resulted in increased accuracy; however ,the increase was small. The accuracy of the RF algorithm, combined data, was 67%. The models’ accuracies can be compared to a random classifier resulting in an accuracy of 44%.

Comparing Model Performance Within Each Data Type

For a closer investigation of algorithm differences, we analyzed the variability in accuracy during training of the models (Dietterich, 1998). Based on Dietterich (1998), the models were developed by splitting the training data in two folds and repeating this procedure five times—resulting in 10 models with an accompanying calculated training accuracy. A t-test was performed for pairwise comparison of models with a Bonferroni correction of p values (Kuhn et al., 2019). The Bonferroni correction of a .05 criterion within each data type resulted in a .0033 criterion. Table 6 shows the results.

Table 6

Pairwise Difference Between Models’ Accuracy and Statistical Significance

	NB	NN	RF	SVML	XGB
Speech data
KNN	.03*	−.01*	−.03*	.06*	−.02*
NB		−.04*	−.06*	.03*	−.05*
NN			−.02*	.07*	−.01*
RF				.09*	.01*
SVML					−.08*
System data
KNN	.05*	−.02*	−.02*	.07*	−.03*
NB		−.07*	−.07*	.01*	−.08*
NN			.00*	.08*	−.01*
RF				.09*	.00*
SVML					−.09*
Combined
KNN	.03*	−.01*	−.06*	.04*	−.06*
NB		−.04*	−.09*	.01*	−.09*
NN			−.05*	.05*	−.05*
RF				.10*	.00*
SVML					−.10*

Note. *p < .0033. KNN = K-nearest neighbors; NB = Naïve Bayes; NN = neural networks; RF = Random Forest; SVML= SVM with Linear kernel; XGB = extreme gradient boosting.

Overall, the RF and XGB algorithms performed the best. For speech data, RF outperformed all models except for XGB, while XGB outperformed all models except for RF and NN. For system data, the best performing algorithms were RF, XGB, and NN. The were no significant differences for pairwise differences involving RF, XGB, and NN. For combined data, both RF and XGB outperformed all algorithms.

Confusion Matrix Across Data Sources

To provide further insight into model accuracy, we developed a confusion matrix with algorithm estimation and the operators’ classification—incorporating the accuracy according to the data sources (McDonald et al., 2020). For brevity, we focus here on the RF algorithm only.

Figure 5 shows poor performance on the classification of high cognitive workload. This challenge persisted for both data sources as well as for the combined data. Nevertheless, although the overall accuracy of the speech and system variables RF algorithms was almost identical, the speech data algorithm classified a higher proportion of high workload correctly, while the system variables algorithm more frequently misclassified high cognitive workload as low. Figure 5 also shows that the algorithm trained on all data improved the classification of both low and medium workload in comparison with the algorithms trained only on speech or system data; however, the combined data algorithm performed worse than the speech data algorithm at classifying high cognitive workload. Although high workload classification was challenging, the RF algorithm did not misclassify the operators’ low or medium workload as high workload. An investigation of the result of each of the six algorithms showed that they were all weaker at classifying high cognitive workload than lower workload.

Figure 5

Confusion matrix: data sources.

Variable Importance

Estimation of feature importance provides information on the relative importance of the variables for the models. Feature importance was estimated using the mean decrease in accuracy when removing a variable from the model (Kuhn et al., 2019). Figures 6 and 7 show the RF model features’ relative importance for the 10 most important speech features and the system features respectively in estimating cognitive workload. The most important feature is set to 100% and the other features importance are scaled accordingly.

Figure 6

Relative importance of speech features for estimation of cognitive workload measured by mean decrease in accuracy.

Figure 7

Relative importance of operator–system interaction features for estimation of cognitive workload measured by mean decrease in accuracy.

The results show that the operator’s pitch (f0), amplitude, and articulation rate are the most important features for estimating cognitive workload. Both mean level and variability were important for pitch and amplitude. The top three features—pitch, articulation rate, and amplitude—suggest that raising the pitch, speaking faster, and speaking louder are relevant indicators of cognitive workload.

The system data included only six features. The results show that plant system alarms and human–system interaction are the most important features. Also, acknowledging operator alarms contributed substantially. The low contribution of commands probably relates to the relatively few plant system and plant object commands actuated by the operators during the scenarios. The advanced and highly automated plant simulated implies a high proportion of cognitive work and that relatively few, but important, commands are executed.

When combining speech and system data for algorithm training, the results show that the five most important variables were all speech features. The top five features were identical to those identified as most important for the speech only data. The most important system variable was operator alarms, which was the sixth most important variable.

Person-Dependent and Person-Independent Algorithm Training

We trained three model types applicable to the estimation of workload of each individual operator. One “generic” model based on training data from all other operators expect the one in question, a “team” model for each of the six teams based on the training data from all three operators within a team, and a “person” model for each of the 18 operators based on training data from the given operator only. From “generic,” “team,” to “person” models, the relevance of the training data increases while the number of training cases reduces—gradually more relevant but less data available. All three model types were tested on each operator’s test data—the 30% withheld data for each operator. Thus, within each model type, the analysis resulted in 18 model accuracies.

Figure 8 shows that the person and team models performed better overall than the generic person-independent models despite having less data available for model training. The mean accuracy of the generic model was .58; the team model, .71; and the person model, .72. A nonparametric Friedman test showed that accuracy was significantly affected by the training approach, χ²(2) = 9.24, p < .01. Post-hoc comparisons using a Wilcoxon test with a Bonferroni-adjusted α level (.017) showed that both the person model and the team model were significantly different from the generic model.

Figure 8

Accuracy of cognitive workload estimation from generic, team, and person models.

Discussion

An operator’s cognitive workload during full-scope simulation scenarios could be classified from combining speech features and system features with an accuracy of 67%. Speech features and system features achieved similar accuracy, 63% versus 61%. The most important speech variables were pitch (f0), articulation rate, and amplitude, and the most important system variables were alarms and operator–system activities. Training algorithms specifically for a given person or for a given team of three operators improved mean accuracy to 72% and 71%, respectively.

Machine Learning Algorithm Performance

The differences in accuracy between algorithms were moderate and consistent across all three data types. RF and XGB achieved the highest classification accuracy. However, KNN and NN achieved similar levels of accuracy for the system data. The relatively good performance of RF corresponds with results reported by McDonald et al. (2020), while the relatively good performance of XGB corresponds with results reported by Momeni et al. (2019).

The 67% accuracy of RF trained on the combined data, and the 71% and 72% accuracy of the team and person models, respectively, are a promising result for the classification of cognitive workload in realistic accident scenarios. The accuracy achieved is below the accuracy of workload classification reported for simple individual tasks but similar to several results reported from realistic settings. For example, Yin et al. (2007) reported an accuracy of 78.9% for the relatively simple Stroop task, and Yap et al. (2010) reported an accuracy 84.4%. Charfuelan and Kruijff (2013) achieved a classification accuracy of 74% for realistic urban search and rescue training. McDonald et al. (2020) reported accuracy ranging from 24% to 73% for various machine learning algorithms used to classify driver distraction. However, Chen et al. (2012) demonstrated that relatively high accuracy can be achieved in realistic settings. They reported an accuracy of 82.2% over three levels of workload for operators of an emergency communication center. Compared to our study, Chen et al. (2012) identified and annotated shorter workload segments—10-s segments compared to our 2-min segments. This might help to determine a segment’s cognitive workload level. Longer segments pose a higher chance of including varying workload levels, so they are potentially more challenging to label. Further, Chen et al. (2012) utilized postsession segment coding by subject-matter experts. This might result in more consistent labeling compared to participant self-labeling.

The challenge of correctly classifying high cognitive workload was seen for all algorithms. We interpret this challenge as representing questions regarding measurement rather than about the type of algorithm or type of data. The proportion of high workload observed was low and concentrated on a few individual operators. Professional operators are trained to cope with high workload, so more extreme cases of cognitive workload might be necessary to classify workload as high. Related to this issue is the procedure for subjective labeling of workload. Even though the procedure was extensively explained to the operators before the data collection, future studies could verify the comprehension of the procedure during the data collection. The latter conflicts with the conventional experimental procedure of avoiding influence on participants’ responses during data collection but might be a measure needed to increase the quality of training data for machine learning algorithms.

Data Types, Important Speech, and System Variables in Cognitive Workload Estimation

The accuracies achieved with speech data and system data individually provide convergent evidence for speech features and system features being relevant for cognitive workload estimation. However, the lack of improved accuracy when combining these two relatively different sources of data is a surprising result. From a theoretical viewpoint, we expected speech features to indicate the effort of the operator in managing workload and to be highly influenced by individual differences between operators. On the other hand, alarms represent the task load from the plant process perspective and most alarms were common across crews.

That surprising result aside, we propose that the study demonstrates a potential for both speech data and system data to be used for estimating cognitive workload during realistic human–machine work. The top three speech features indicate that raising pitch, speaking faster, and speaking louder are relevant indicators of cognitive workload. This finding aligns with prior research (Brenner et al., 1994; Luig & Sontacchi, 2010). Alarms and operator–system interaction were found to be important system features, indicating a potential for using this type of feature to classify and measure cognitive workload.

Training Approaches and Accuracy Achieved

The study found that person-dependent and team-dependent models resulted in higher classification accuracy, with 72% and 71%, respectively, compared to a generic person-independent model (58% accuracy). The improved accuracy of person models corresponds with earlier findings (van Segbroeck et al., 2014; Vukovic et al., 2019) and may be related to both operator differences regarding features extracted and the operators’ self-rating of cognitive workload. Interestingly, the team models resulting from training the algorithm on a given team’s three operators performed on average as well as the person models. The results suggest that developing person models, whenever feasible, is better than using generic models for use in dynamic, realistic, human–machine research. The accuracy of the team models is interesting since this option provides training data from all team members as opposed to the person model involving a single operator.

Study Limitations and Future Research

An inherent challenge with simulator studies of realistically demanding human–machine work is limited controllable scenario progression. Increased testing and fine-tuning of scenarios with the participation of actual operators could provide scenario segments that professional operators would classify more consistently as high workload. In addition, scales for cognitive workload classification could be calibrated with each participant, both before and during data collection.

The relevance of system variables for estimating cognitive workload might depend on the context. For example, the utilization of alarms assumes that operators attend to activated alarms and utilize alarm information in their control of the system. This would be expected for most nuclear control room work where alarms are frequently relevant to the crew’s main tasks. However, one can imagine a context where operators perceive alarms as less relevant, either due to the alarm content or due to poorly designed alarm systems. Reviewing work procedures and/or discussing with users and subject-matter experts would be important to identify to what extent system variables potentially can serve as workload indicators.

The choice of ground truth measure guiding the classification is an important assumption for modeling. We chose a subjective scale because it is currently the most widely used (Matthews et al., 2015) and was feasible to obtain during the simulation. However, subjective scales have been criticized for lack of correspondence with performance-based and physiological workload measures (Matthews et al., 2020). Primary task measures (Hancock & Matthews, 2019; Young et al., 2015) remain challenging to define and implement for complex systems due to limited sensitivity to levels of workload (Hancock & Matthews, 2019; Wu & Li, 2013), while secondary task measures have been successfully applied to realistic process control (Wu & Li, 2013). Secondary task measures, such as the Detection-Response Task (International Organization for Standardization [ISO], 2016), have been extensively applied for cognitive workload measurement in driving environments (Strayer et al., 2019), military (Hollands et al., 2019), and laboratory settings (Castro et al., 2019). Physiological measures, for example, heart rate and heart rate variability (Mulder, 1992), have been found to be sensitive to cognitive workload in human–machine control room settings (Gao et al., 2013; Wulvik et al., 2020). In addition to ground truth, several of the above measures represent nonintrusive approaches for cognitive workload estimation. Future research could compare the approach presented here with these approaches to investigate the merits of this study’s approach relative to other approaches.

The study included male operators only, which might question the generalizability across gender. This limitation could be more relevant for the estimation of workload from speech features than for the estimation from system variables.

This study demonstrates the potential of utilizing speech features, user interaction, and alarms to estimate cognitive workload in realistic human–machine settings. However, if considering a fully developed approach, the accuracy achieved can be classified as moderate only. For practical reasons, this study covered a subset of the speech and user interaction features presented in the literature. Examples of relatively comprehensive feature extraction include Charfuelan and Kruijff’s (2013) extraction of speech features and Arshad et al. (2013) extraction of user interaction features. We believe that further development of the specification and extraction of features specific to complex human–machine interaction could improve the features’ sensitivity to cognitive workload, and thereby increase estimation accuracy.

Application Potential

The approach presented is applicable to many domains. System states, user interaction, and data from human sensors are becoming easily available, and high-quality verbal communication can be recorded even in team settings. Applications include cognitive load indication in complex training situations (van Merriënboer & Sweller, 2005) and workload assessment in human–machine design and evaluation (Reinerman-Jones et al., 2019). The approach investigated is nonintrusive, so it does not affect team tasks and can be applied to complex research and applied settings without requiring cumbersome, dedicated, data registration equipment (Chen et al., 2012). The approach can potentially be applied for real-time estimation of workload and can be suitable for user-adapted design.

Conclusion

This study investigated the classification of operators’ cognitive workload through machine learning by combining speech and system features. The achieved classification accuracies demonstrate the approach’s potential for dynamic nonintrusive cognitive workload estimation in realistic settings. However, we believe future studies should advance the specification and extraction of speech and system features to potentially increase classification accuracy. Future studies should investigate improved ground truth measures for classification, for example, by utilizing secondary tasks or physiological measures.

The findings suggest that the most promising cognitive workload indicators are the speech features, pitch (fundamental frequency), articulation rate, and amplitude, and the system feature of number of activated alarms. RF and XGB outperformed other algorithms in terms of classification accuracy, and training using person-dependent classification models improved accuracy over generic models. The nonintrusive approach presented is widely applicable to human–machine research and applied human factors.

Key Points

There are few nonintrusive measures available for evaluating cognitive workload in realistic dynamic human–machine interaction.

Earlier research on simple tasks has established a relationship between speech features, user–system interaction, and system variables with cognitive workload.

This study found, through the use of machine learning algorithms, that speech features and system variables occurring as part of a control room team’s natural work in a realistic full-scope simulator could classify the cognitive workload experienced by operators.

The types of variables used to classify cognitive workload are available and easily accessible in many research and applied settings. The approach is, therefore, feasible for many domains for the measurement and evaluation of cognitive workload.

Supplemental Material

Supplementary Material 1 - Supplemental material for An Investigation of Speech Features, Plant System Alarms, and Operator–System Interaction for the Classification of Operator Cognitive Workload During Dynamic Work

Supplemental material, Supplementary Material 1, for An Investigation of Speech Features, Plant System Alarms, and Operator–System Interaction for the Classification of Operator Cognitive Workload During Dynamic Work by Per Ø. Braarud, Terje Bodal, John E. Hulsund, Michael N. Louka, Christer Nihlwing, Espen Nystad, Håkan Svengren and Emil Wingstedt in Human Factors: The Journal of Human Factors and Ergonomics Society

Footnotes

Acknowledgment

The simulator data utilized in this study were originally collected for work funded by the OECD Halden Reactor Project, Norway.

ORCID iD

Michael N. Louka

Supplemental Material

The online supplemental material is available with the manuscript on the HF website.

Author Biographies

Per Ø. Braarud is a senior scientist in the Human-Centered Digitalization department at Institute for Energy Technology (IFE) in Halden, Norway. He received his MSc in cognitive psychology from the Norwegian University of Science and Technology in 1994.

Terje Bodal is a senior engineer in the Applied Data Science department at Institute for Energy Technology (IFE) in Halden, Norway. He received his BSc in informatics from Østfold University College, Norway, in 1994.

John E. Hulsund is a senior scientist in the Applied Data Science department at Institute for Energy Technology (IFE) in Halden, Norway. He received his MSc in experimental particle physics from the University of Bergen, Norway, in 1993.

Michael N. Louka is the department head of the Virtual and Augmented Reality department at Institute for Energy Technology (IFE) in Halden, Norway. He received his BSc(hons) and Associateship in computing and IT from University of Surrey, UK, in 1994.

Christer Nihlwing is a principal engineer in the Control Room and Interaction Design department at Institute for Energy Technology (IFE) in Halden, Norway.

Espen Nystad is a senior scientist in the Human-Centered Digitalization department at Institute for Energy Technology (IFE) in Halden, Norway. He received his MSc in psychology from the Norwegian University of Science and Technology in 2002.

Håkan Svengren is a principal engineer in the Control Room and Interaction Design department at Institute for Energy Technology (IFE) in Halden, Norway.

Emil Wingstedt is a research scientist in the Applied Data Science department at Institute for Energy Technology (IFE) in Halden, Norway. He received his MSc in theoretical physics from the Norwegian University of Science and Technology in 2002.

References

Ahlstrom

Friedman-Berg

F. J

. (2006). Using eye movement activity as a correlate of cognitive workload. International Journal of Industrial Ergonomics, 36, 623–636.doi:10.1016/j.ergon.2006.04.002

Arshad

Wang

Chen

. (2013). Analysing mouse activity for cognitive load detection [Conference session]. Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, ACM, Adelaide, Australia. 115–118.

Boersma

van Heuven

. (2001). Praat, a system for doing phonetics by computer. Glot International, 5, 341–345.

Braarud

P. Ø

. (2020). An efficient screening technique for acceptable mental workload based on the NASA task load Index—development and application to control room validation. International Journal of Industrial Ergonomics, 76, 102904.doi:10.1016/j.ergon.2019.102904

Braarud

P. Ø.

Kirwan

. (2011). Task complexity: What challenges the crew and how do they cope? In Skjerve

A. B.

Bye

(Eds.), Simulator-based human factors studies across 25 years: The history of the Halden Man-Machine Laboratory (pp. 233–251). Springer-Verlag.

Braarud

P. Ø.

Svengren

Nihlwing

. (2020). Human-system validation experiment 2018: Multistage assessment, operators self-evaluation, and mental workload assessment (Halden Work Report 1261). OECD Halden Reactor Project.

Brenner

Doherty

E. T.

Shipp

. (1994). Speech measures indicating workload demand. Aviation, Space, and Environmental Medicine, 65, 21–26.8117221

Brenner

Shipp

Doherty

Morrissey

. (1985). Voice measures of psychological stress: Laboratory and field data. In Titze

Scherer

(Eds.), Vocal fold physiology, biomechanics, acoustics, and phonatory control (pp. 239–248). The Denver Center for the Performing Arts.

Brown

L. D.

Cai

T. T.

Dasgupta

. (2001). Interval estimation for a binomial proportion. Statistical Science, 16, 101–133.

10.

Burris

Vorperian

H. K.

Fourakis

Kent

R. D.

Bolt

D. M

. (2014). Quantitative and descriptive comparison of four acoustic analysis systems: Vowel measurements. Journal of Speech, Language, and Hearing Research, 57, 26–45.doi:10.1044/1092-4388(2013/12-0103)

24687465

11.

Byers

J. C.

Bittner Jr.

A. C.

Hill

S. G

. (1989). Traditional and raw task load index (TLX) correlations: Are paired comparisons necessary? In Mital

(Ed.), Advances in industrial ergonomics and safety I (pp. 481–485). Taylor & Francis.

12.

Castro

S. C.

Strayer

D. L.

Matzke

Heathcote

. (2019). Cognitive workload measurement and modeling under divided attention. Journal of Experimental Psychology: Human Perception and Performance, 45, 826–839.doi:10.1037/xhp0000638

30998070

13.

Cegarra

Chevalier

. (2008). The use of tholos software for combining measures of mental workload: Toward theoretical and methodological improvements. Behavior Research Methods, 40, 988–1000.doi:10.3758/BRM.40.4.988

19001390

14.

Charfuelan

Kruijff

G. J

. (2013). Classification of speech under stress and cognitive load in USAR operations [Conference session]. Acoustics, Speech and Signal Processing (ICASSP), IEEE International Conference, 508–512.

15.

Chen

Ruiz

Choi

Epps

Khawaja

M. A.

Taib

Yin

Wang

. (2012). Multimodal behavior and interaction as indicators of cognitive load. ACM Transactions on Interactive Intelligent Systems, 2, 1–36.doi:10.1145/2395123.2395127

16.

Chen

Benesty

Khotilovich

Tang

Cho

Chen

Mitchell

Cano

Zhou

Xie

Lin

Geng

. (2019). xgboost: Extreme gradient boosting. R package version 0.90.0.2. https://CRAN.R-project.org/package=xgboost

17.

Cvach

. (2012). Monitor alarm fatigue: An integrative review. Biomedical Instrumentation & Technology, 46, 268–277.doi:10.2345/0899-8205-46.4.268

22839984

18.

Dietterich

T. G

. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10, 1895–1923.doi:10.1162/089976698300017197

19.

Esposito

Stejskal

Smékal

Bourbakis

N. G

. (2007). The significance of empty speech pauses: Cognitive and algorithmic issues. In Faundez-Zanuy

, et al. (Eds.), Brain vision artificial intelligence, LNCS, 4729 (pp. 542–554). Springer-Verlag.

20.

Ferreira

Kim

Siirtola

Röning

Forlizzi

J. F.

Dey

A. K

. (2014). Assessing real-time cognitive load based on psycho-physiological measures for younger and older adults [Symposium]. 2014 IEEE Symposium of Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB), Orlando, FL, 2014, 39–48.doi:doi:10.1109/CCMB.2014.7020692

21.

Gao

Wang

Song

Dong

. (2013). Mental workload measurement for emergency operating procedures in digital nuclear power plants. Ergonomics, 56, 1070–1085.doi:10.1080/00140139.2013.790483

23654299

22.

Gopher

Donchin

. (1986). Workload: An examination of the concept. In Boff

K. R.

Kaufman

Thomas

J. P.

(Eds.), Handbook of perception and human performance, vol II, cognitive processes and performance. Wiley & Sons.

23.

Grier

R. A

. (2015). How high is high? A meta-analysis of NASA-TLX global workload scores. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 59, 1727–1731.doi:10.1177/1541931215591373

24.

Guastello

S. J.

Shircel

Malon

Timm

. (2015). Individual differences in the experience of cognitive workload. Theoretical Issues in Ergonomics Science, 16, 20–52.doi:10.1080/1463922X.2013.869371

25.

Hancock

. (2017). Whither workload? Mapping a path for its future development. In Longo

Leva

M. C.

(Eds.), H-Workload (pp. 1–15). Springer.

26.

Hancock

P. A.

Matthews

. (2019). Workload and performance: Associations, Insensitivities, and dissociations. Human Factors: The Journal of the Human Factors and Ergonomics Society, 61, 374–392.

27.

Hancock

P. A.

Warm

J. S

. (1989). A dynamic model of stress and sustained attention. Human Factors: The Journal of the Human Factors and Ergonomics Society, 31, 519–537.

28.

Hart

S. G.

Staveland

. (1988). Development of the NASA task load index (TLX): Results of empirical and theoretical research. In Hancock

P. A.

Meshkati

(Eds.), Human mental workload (pp. 139–183). North-Holland.

29.

Hockey

G. R

. (1997). Compensatory control in the regulation of human performance under stress and high workload; a cognitive-energetical framework. Biological Psychology, 45, 73–93.

30.

Hollands

J. G.

Spivak

Kramkowski

E. W

. (2019). Cognitive load and situation awareness for soldiers: Effects of message presentation rate and sensory modality. Human Factors, 61, 763–773.doi:10.1177/0018720819825803

30768371

31.

Humphrey

D. G.

Kramer

A. F

. (1994). Toward a psychophysiological assessment of dynamic changes in mental workload. Human Factors, 36, 3–26.doi:10.1177/001872089403600101

8026842

32.

Hutchins

. (1995). Cognition in the wild. MIT Press.

33.

Huttunen

Keränen

Väyrynen

Pääkkönen

Leino

. (2011). Effect of cognitive load on speech prosody in aviation: Evidence from military simulator flights. Applied Ergonomics, 42, 348–357.doi:10.1016/j.apergo.2010.08.005

20832770

34.

Huttunen

K. H.

Keränen

H. I.

Pääkkönen

R. J.

Päivikki Eskelinen-Rönkä

Leino

T. K

. (2011). Effect of cognitive load on articulation rate and formant frequencies during simulator flights. The Journal of the Acoustical Society of America, 129, 1580–1593.doi:10.1121/1.3543948

21428521

35.

Institute for Energy Technology (IFE) . (2019). HAMMLAB. Institute for Energy Technology. https://ife.no/en/laboratory/hammlab/

36.

International Organization for Standardization (ISO) . (2006). ISO 11064-7, 2006. Ergonomic design of control centres part 7: Principles for the evaluation of control centres. International Organization for Standardization.

37.

International Organization for Standardization (ISO) . (2016). Road vehicles—Transport information and control systems—Detection-response task (DRT) for assessing attentional effects of cognitive load in driving. ISO.

38.

Jadoul

Thompson

de Boer

. (2018). Introducing parselmouth: A python interface to praat. Journal of Phonetics, 71, 1–15.doi:10.1016/j.wocn.2018.07.001

39.

James

Witten

Hastie

Tibshirani

. (2013). An introduction to statistical learning: With applications in R (Corrected ed.). Springer. https://link-springer-com-443.web.bisu.edu.cn/book/10.1007/978-1-4614-7138-7.

40.

Jordan

C. S

. (1992). Experimental study of the effect of an instantaneous self-assessment workload recorder on task performance. DRA Technical Memorandum (CAD5) 92011. DRA Maritime Command Control Division, Portsmouth.

41.

Karatzoglou

Smola

Hornik

Maniscalco

M. A.

Teo

C. H

. (2019). Kernlab: Kernel-based machine learning lab. R package version 0.9-29. https://CRAN.R-project.org/package=kernlab

42.

Kennedy

D. O.

Scholey

A. B

. (2000). Glucose administration, heart rate and cognitive performance: Effects of increasing mental effort. Psychopharmacology, 149, 63–71.doi:10.1007/s002139900335

10789884

43.

Kuhn

. (2008). Building predictive models in R using the caret package. Journal of Statistical Software, 28, 1–26.doi:10.18637/jss.v028.i05

44.

Kuhn

Wing

Weston

Williams

Keefer

Engelhardt

Cooper

Mayer

Kenkel

Benesty

Lescarbeau

Ziem

Scrucca

Tang

Candan

Hunt

R Core Team . (2019). Caret: classification and regression training. R package version 6.0-84. https://CRAN.R-project.org/package=caret

45.

Liaw

Wiener

. (2018). randomForest: Breiman and Cutler’s random forests for classification and regression. R package version 4.6-14. https://CRAN.R-project.org/package=randomForest

46.

Lieberman

. (2007). The evolution of human speech: Its anatomical and neural bases. Current anthropology, 48, 39–66.

47.

Lin

Tang

. (2013). Automatic cognitive load classification using high-frequency interaction events. International Journal of Technology and Human Interaction, 9, 73–88.doi:10.4018/jthi.2013070106

48.

Lively

S. E.

Pisoni

D. B.

Van Summers

Bernacki

R. H

. (1993). Effects of cognitive workload on speech production: Acoustic analyses and perceptual consequences. The Journal of the Acoustical Society of America, 93, 2962–2973.doi:10.1121/1.405815

8315159

49.

Luig

Sontacchi

. (2010). Workload monitoring through speech analysis: Towards a system for air traffic control [Conference session]. Proceedings of the 27th International Congress on the Aeronautical Sciences, Nice, France.

50.

Majka

. (2019). naivebayes: High performance implementation of the naive Bayes algorithm. R package version 0.9.6. https://CRAN.R-project.org/package=naivebayes

51.

Matthews

Reinerman-Jones

L. E.

Barber

D. J.

Abich

. (2015). The psychometrics of mental workload: Multiple measures are sensitive but divergent. Human Factors, 57, 125–143.doi:10.1177/0018720814539505

25790574

52.

Matthews

Winter

J. D.

Hancock

P. A

. (2020). What do subjective workload scales really measure? Operational and representational solutions to divergence of workload measures. Theoretical Issues in Ergonomics Science, 4.

53.

McDonald

A. D.

Ferris

T. K.

Wiener

T. A

. (2020). Classification of driver distraction: A comprehensive analysis of feature generation, machine learning, and input measures. Human Factors, 62, 1019–1035.doi:10.1177/0018720819856454

31237788

54.

Momeni

Dell’ Agnola

Arza

Atienza

. (2019). Real-time cognitive workload monitoring based on machine learning using physiological signals in rescue missions [Conference session]. Proceedings of 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany.

55.

Moray

. (1967). Where is capacity limited? A survey and a model. Acta Psychologica, 27, 84–92.doi:10.1016/0001-6918(67)90048-0

6062244

56.

Moray

. (1988). Mental workload since 1979. International Review of Ergonomics, 2, 123–150.

57.

Mulder

L. J

. (1992). Measurement and analysis methods of heart rate and respiration for use in applied environments. Biological Psychology, 34, 205–236.doi:10.1016/0301-0511(92)90016-N

1467394

58.

Nygren

T. E

. (1991). Psychometric properties of subjective workload measurement techniques: Implications for their use in the assessment of perceived mental workload. Human Factors, 33, 17–33.doi:10.1177/001872089103300102

59.

OECD NEA . (2020). OECD Nuclear Energy Agency. https://www.oecd-nea.org/jointproj/halden.html

60.

Pimenta

Carneiro

Neves

Novais

. (2016). A neural network to classify fatigue from human–computer interaction. Neurocomputing, 172, 413–426.doi:10.1016/j.neucom.2015.03.105

61.

R Core Team . (2019). R: A language and environment for statistical computing. R foundation for Statistical Computing. http://www.R-project.org/.

62.

Ragab

Craye

Kamel

M. S.

Karray

. (2014). A visual-based driver distraction recognition and detection using random forest. Proceedings of the International Conference on Image Analysis and Recognition, 8814, 256–265.

63.

Rahman

Ahmed

M. U.

Barua

Begum

. (2020). Non-contact-based driver’s cognitive load classification using physiological and vehicular parameters. Biomedical Signal Processing and Control, 55, 1–13.doi:10.1016/j.bspc.2019.101634

64.

Reid

G. B.

Nygren

T. E

. (1988). The subjective workload assessment technique: A scaling procedure for measuring mental workload. In Hancock

P. A.

Meshkati

(Eds.), Advances in psychology, 52. Human mental workload (pp. 185–218). North-Holland.

65.

Reinerman-Jones

L. E.

Hughes

D’Agostino

Matthews

. (2019). Human performance metrics for the nuclear domain: A tool for evaluating measures of workload, situation awareness and teamwork. International Journal of Industrial Ergonomics, 69, 217–227.doi:10.1016/j.ergon.2018.12.001

66.

Ripley

Venables

. (2016). nnet: Feed-forward neural networks and multinomial log-linear models. R package version 7.3-12. https://CRAN.R-project.org/package=nnet

67.

Ruskin

K. J.

Hueske-Kraus

. (2015). Alarm fatigue: Impacts on patient safety. Current Opinion in Anesthesiology, 28, 685–690.doi:10.1097/ACO.0000000000000260

26539788

68.

Rusnock

C. F.

Borghetti

B. J

. (2018). Workload profiles: A continuous measure of mental workload. International Journal of Industrial Ergonomics, 63, 49–64.doi:10.1016/j.ergon.2016.09.003

69.

Schulz

C. M.

Schneider

Fritz

Vockeroth

Hapfelmeier

Wasmaier

Kochs

E. F.

Schneider

. (2011). Eye tracking for assessment of workload: A pilot study in an anaesthesia simulator environment. British Journal of Anaesthesia, 106, 44–50.doi:10.1093/bja/aeq307

21037266

70.

Shi

Ruiz

Taib

Choi

E. H.

Chen

. (2007). Galvanic skin response (GSR) as an index of cognitive load [Conference session]. Proceedings of the 2007 Conference on Human Factors in Computing Systems, ACM Press, San Jose, California.

71.

Solovey

E. T.

Zec

Perez

E. A. G.

Reimer

Mehler

. (2014). Classifying driver workload using physiological and driving performance data: Two field studies [Conference session]. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, ACM, 4057–4066.

72.

Strayer

D. L.

Cooper

J. M.

Goethe

R. M.

McCarty

M. M.

Getty

D. J.

Biondi

. (2019). Assessing the visual and cognitive demands of in-vehicle information systems. Cognitive Research: Principles and Implications, 4, 18.doi:10.1186/s41235-019-0166-3

31227955

73.

Tobaruela

Schuster

Majumdar

Ochieng

W. Y.

Martinez

Hendrickx

. (2014). A method to estimate air traffic controller mental workload based on traffic clearances. Journal of Air Transport Management, 39, 59–71.doi:10.1016/j.jairtraman.2014.04.002

74.

van Merriënboer

J. J. G.

Sweller

. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17, 147–177.doi:10.1007/s10648-005-3951-0

75.

van Segbroeck

Travadi

Vaz

Kim

Black

M. P.

Potamianos

Narayanan

S. S

. (2014). Classification of cognitive load from speech using an I-Vector framework. Interspeech.

76.

Venables

W. N.

Ripley

B. D

. (2002). Modern applied statistics with S (4th ed.). Springer.

77.

Vicente

K. J

. (1997). Operator adaptation in process control: A three-year research program. Control Engineering Practice, 5, 407–416.doi:10.1016/S0967-0661(97)00018-X

78.

Vukovic

Sethu

Parker

Cavedon

Lech

Thangarajah

. (2019). Estimating cognitive load from speech gathered in a complex real-life training exercise. International Journal of Human-Computer Studies, 124, 116–133.doi:10.1016/j.ijhcs.2018.12.003

79.

Wickens

Tsang

P. S

. (2015). Workload. In Boehm-Davis

D. A.

Durso

F. T.

Lee

J. D.

(Eds.), APA handbooks in psychology: APA Handbook of human systems integration (pp. 277–292). American Psychological Association.

80.

Wilson

G. F.

Russell

C. A

. (2003). Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Human Factors, 45, 635–644.doi:10.1518/hfes.45.4.635.27088

15055460

81.

Wittels

Johannes

Enne

Kirsch

Gunga

H.-C

. (2002). Voice monitoring to measure emotional load during short-term stress. European Journal of Applied Physiology, 87, 278–282.doi:10.1007/s00421-002-0625-1

12111290

82.

. (2013). Secondary task method for workload measurement in alarm monitoring and identification tasks [Conference session]. International Conference on Cross-Cultural Design, Las Vegas, NV: Springer Berlin Heidelberg, 346–354.

83.

Wulvik

A. S.

Dybvik

Steinert

. (2020). Investigating the relationship between mental state (workload and affect) and physiology in a control room setting (SHIP bridge simulator). Cognition, Technology & Work, 22, 95–108.doi:10.1007/s10111-019-00553-8

84.

Yap

T. F.

Epps

Choi

E. H. C.

Ambikairajah

. (2010). Glottal feature for speech-based cognitive load classification [Conference session]. Proceedings of the IEEE International Conference on Acoustic, Speech and Signal Processing (ICASSP’10), Dallas, USA, 5234–5237.

85.

Yin

Ruiz

Ambikairajah

. (2007). Speech-based cognitive load monitoring system [Conference session]. Proceedings IEEE ICASSP, 2041–2044.

86.

Young

M. S.

Brookhuis

K. A.

Wickens

C. D.

Hancock

P. A

. (2015). State of science: Mental workload in ergonomics. Ergonomics, 58, 1–17.doi:10.1080/00140139.2014.956151

25442818

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.10 MB