A Multilevel Approach to Relating Subjective Workload to Performance After Shifts in Task Demand

Abstract

Objective:

The aim of this laboratory experiment was to demonstrate how taking a longitudinal, multilevel approach can be used to examine the dynamic relationship between subjective workload and performance over a given period of activity involving shifts in task demand.

Background:

Subjective workload and conditions of the performance environment are oftentimes examined via cross-sectional designs without distinguishing within- from between-person effects. Given the dynamic nature of performance phenomena, multilevel designs coupled with manipulations of task demand shifts are needed to better model the dynamic relationships between state and trait components of subjective workload and performance.

Method:

With a sample of 75 college students and a computer game representing a complex decision-making environment, increases and decreases in task demand were counterbalanced and subjective workload and performance were measured concurrently in regular intervals within performance episodes. Data were analyzed using hierarchical linear modeling.

Results:

Both between- and especially within-person effects were dynamic. Nevertheless, at both levels of analysis, higher subjective workload reflected performance problems, especially more downstream from increases in task demand.

Conclusion:

As a function of cognitive-energetic processes, shifts in task demand are associated with changes in how subjective workload is related to performance over a given period of activity. Multilevel, longitudinal approaches are useful for distinguishing and examining the dynamic relationships between state and trait components of subjective workload and performance.

Application:

The findings of this research help to improve the understanding of how a sequence of demands can exceed a performer’s capability to respond to further demands.

Keywords

workload workload history overload human performance modeling shifts in task demand multilevel analyses dynamic relationships

Introduction

Stress, as a consequence of sustained attention or fluctuations in the information-processing demands of the task, requires adaptation for successful performance to be achieved (Hancock & Warm, 1989; Hockey, 1986). Adaptation is a function of variability in the process of self-regulation, within and between persons, and is not solely determined by the external environment (Hockey, 1986). Perceptions of one’s capacity to meet task demands, referred to as subjective workload, are typically assessed in retrospect to provide an indication of how individuals handled the conditions of stress. However, subjective workload’s relationship with performance is characterized by inconsistent results (i.e., dissociation; Yeh & Wickens, 1988), showing a range of negative, positive, and null effects (Cumming & Croft, 1973; Goldberg & Stewart, 1980; Krulewitz, Warm, & Wohl, 1975; Matthews, 1986; Moroney, Warm, & Dember, 1995).

This inconsistency in observed relationships challenges whether it is reasonable to examine subjective workload utilizing cross-sectional designs in which within- and between-person effects are aggregated across individuals (Helton, Funke, & Knott, 2014; Nygren, 1991). Accordingly, longitudinal, multilevel approaches (i.e., disaggregation; Curran & Bauer, 2011) can be used to distinguish different conceptualizations of subjective workload and better link conditions of stress to performance. Specifically, we propose that when individuals encounter potentially stressful conditions, within-person differences (i.e., intraindividual fluctuations) in subjective workload provide a better indication of an individual’s state condition (Helton et al., 2014), whereas between-person differences (i.e., interindividual differences) in subjective workload reflect trait-like differences in the ability to control attention over a performance episode (Hockey, 1986).

Therefore, the general purpose of the present study was to demonstrate how taking a longitudinal, multilevel approach can be used to examine the dynamic relationship between subjective workload and performance over a given period of activity involving shifts in task demand. Using a computer game representing a complex decision-making environment, we concurrently measured subjective workload and performance at regular intervals preceding and following shifts in task demand. A single-item, global measure of subjective workload was used because (a) such measures are equally if not more sensitive to different levels of subjective workload when compared to multi-item measures (Eggemeier, Shingledecker, & Crabtree, 1985; Hill, Zaklad, Bittner, Byers, & Christ, 1988; Nataupsky & Abbott, 1987; Verwey & Veltman, 1996; Vidulich & Bortolussi, 1988; Vidulich & Tsang, 1987), and (b) they are less disruptive to performance when administered during task execution (Tsang & Vidulich, 1994). On the basis of the proposition that individuals experience different control states depending on shifts in task demand and workload history, we expected subjective workload’s relationship with performance to be more dynamic, changing in both magnitude and direction, at the within- versus the between-person level.

Subjective Workload and Shifts in Task Demand

Self-regulation is necessary to adapt to task demands in order to maintain optimal information-processing states (Hockey, 1986). In this way, the ability to control attention is especially important when a change in the task environment (i.e., stress) has occurred. More specifically, in theory, with respect to task complexity, a shift in task demand changes the nature of how one needs to organize and execute the actions necessary for success (Wood, 1986). A consideration of the aftereffects of stress on performance (Cohen, 1980) highlights the importance of distinguishing immediate versus downstream effects and also the need to better account for changes in subjective states when adapting to stress. That is, although changes in subjective workload have been implicated as an important part of the adaptation process, few researchers have directly examined how variations in subjective experiences relate to performance. Indeed, effects can differ as a function of the level of analysis and the degree of task demand involved (Yeo & Neal, 2008).

From a theoretical perspective consistent with explanations of the aftereffects of stress (Cohen, 1980), individuals can tolerate a range of task demand by way of effortful coping strategies and still maintain an effective performance level (Hancock & Warm, 1989). However, individuals have a region of maximal adaptability, which when superseded as a function of either underload (i.e., hypostress) or overload (i.e., hyperstress) results in a transition from a comfort zone characterized by a state of dynamic stability to a state of dynamic instability. A lack of stability is associated with a reduction of available attentional capacity, higher levels of subjective workload, and subsequent reductions in performance (Hancock & Warm, 1989). Similarly, the role of different stress states has been emphasized such that deviations from one’s comfort zone are consistent with a discrepancy between actual and desired states and increased control activity is needed for performance to be stabilized (Hockey, 1986; Hockey & Hamilton, 1983). This state of active control requires effort as reflected in reports of subjective workload. Deviations in subjective workload reflect changes occurring within the individual (i.e., within-person effect); however, when repeated measurements are aggregated during stressful conditions, comparisons examined across persons reflect general attentional resource utilization capabilities (i.e., between-person effect).

Between-Person Subjective Workload

Subjective workload at the between-person level represents a trait characteristic of the individual, reflecting how some individuals self-regulate better than others when encountering adversity (Hancock & Warm, 1989; Hancock, Williams, & Manning, 1995; Hockey, 1997; Humphreys & Revelle, 1984; Matthews, 1986). At this level, subjective workload can better capture stable differences in characteristic patterns of cognitive activity when responding to stress (Hockey, 1986), controlling for within-person differences. These differences can be attributed to (a) control strategies, (b) aggregate processes involved in general states, or (c) styles of information processing (Hockey, 1986). In theory, modes of control (e.g., inappropriate, overload) that correspond with higher levels of subjective workload (i.e., experiencing high attentional costs) are associated with reduced performance. By contrast, modes of control (e.g., appropriate, target) that correspond with lower levels of subjective workload (i.e., experiencing low attentional costs) are associated with more variable performance (Hockey, 1986; Hockey & Hamilton, 1983). Put another way, a higher average level of subjective workload aggregated across measurement occasions reflects one’s capability is more likely to have been exceeded, such that an individual is less likely to have kept up with task demands. Accordingly, the following hypothesis was examined.

Hypothesis 1: Subjective workload at the between-person level will be negatively related to performance.

Although we propose subjective workload at the between-person level will be negatively related to performance, we do not put forth a specific hypothesis regarding the extent to which the between-person effect is dynamic in terms of the magnitude of the effect significantly increasing or decreasing in performance episodes. Accordingly, the following research question was examined.

Research Question 1: Does the magnitude of the negative effect of subjective workload at the between-person level change across a period of performance?

Within-Person Subjective Workload

The relationship between state conditions—represented by deviations from an individual’s typical level of meeting task demands—and performance is more dynamic with respect to both the magnitude and the direction of effects. That is, a deviation of subjective workload at the within-person level, controlling for the between-person effect, can represent different control states, depending on the shifts in task demand and workload history (Hancock & Warm, 1989; Hockey, 1984, 1986). Despite the distinction between different control states, subjective workload in general represents the extent to which individuals perceive they can meet task demands. In this way, across a potentially stressful period of performance, a single-item assessment can capture different modes of control (e.g., appropriate vs. overload and dynamic stability vs. dynamic instability; Hockey, 1986). Accordingly, the following hypothesis was examined.

Hypothesis 2: The relationship between subjective workload at the within-person level and performance will vary in terms of both magnitude and direction across a period of performance.

Immediate effects

An elevated level of subjective workload immediately after a shift (increase or decrease) in task demand reflects active coping by way of an increase in the allocation of cognitive resources (Hockey, 1997). As such, higher-than-typical levels of subjective workload at this point are more likely to reflect an appropriate state of control (Hockey, 1986). More specific to an increase in task demand, an increase in energetic arousal, indicating an increase in cognitive resources, can result from a shift to higher levels of task demand (Helton, Shaw, Warm, Matthews, & Hancock, 2008). Self-regulation theory suggests individuals will increase the allocation of cognitive resources to bridge current and desired levels of performance (Yeo & Neal, 2008). Accordingly, the following hypotheses were examined.

Hypothesis 3: Subjective workload at the within-person level will be positively related to performance immediately after an increase in task demand.

Hypothesis 4: Subjective workload at the within-person level will be positively related to performance immediately after a decrease in task demand.

Downstream effects

In regard to increases in task demand, we propose the assumed negative relationship between subjective workload and performance at the within-person level will be delayed from an increase in task demand (Cox-Fuenzalida, 2007). That is, changes in subjective workload taking place more downstream from increases in task demand provide an indication of how well individuals are keeping up with the consequences of earlier experiences, as the effects of stressors are more likely to appear after the individual has encountered the increase in task demand for some time (Hockey, 1984). From a cognitive-energetic perspective, resource capacity is conceptualized as finite processing units utilized for performance (Gopher & Donchin, 1986; Kahneman, 1973; Wickens, 1984). This definition implies scarcity such that during an increase in task demand, performance can be maintained but at a cost. Over a period of performance, costs can accumulate, resulting in an unfavorable transition state marked by a depletion of resources (Hockey, 1997). Consequently, a transitional state, such as overload, reflects dynamic instability, which is manifested in performance vis-à-vis (a) increases in response times and errors; (b) greater variability in performance; (c) fewer tasks completed per unit time; (d) high-risk, low-effort decision strategies; and (e) rule-based control (Hockey, 1993; Holding, 1983; Messick Huey & Wickens, 1993; Rasmussen, 1986).

Alternatively, decreases in task demand or more modest periods of task demand should not adversely affect the level of resources available, such that subjective workload would not represent capabilities being exceeded (Cox-Fuenzalida, 2007). That is, individuals are not likely to transition from a comfort zone characterized by a state of dynamic stability to a state of dynamic instability (Hancock & Warm, 1989). Accordingly, the following hypotheses were examined.

Hypothesis 5: Subjective workload at the within-person level will be negatively related to performance downstream after an increase in task demand.

Hypothesis 6: Subjective workload at the within-person level will be positively related to performance downstream after a decrease in task demand.

Method

Participants

Eighty-seven undergraduates (mean age = 18.62, SD = 0.81; 47% male) from the University of Oklahoma participated for credit toward a psychology course research requirement. Data for 12 participants were removed from analysis due to hardware problems or participants not following instructions.

Performance Task

Participants were decision makers in a computer-based command-and-control peacekeeping environment created using the distributed dynamic decision-making DDD simulation software package (Aptima, 2007). A two-dimensional map was displayed on the monitor with an information panel on the left side. Participants engaged the environment using both buttons of a two-button mouse, controlling three types of units to maintain “influence” in a fictional foreign region of responsibility populated with locals that saw the participant units as either friendly or hostile. Participant units and locals were depicted on the map with different icons (e.g., soldier, medic, tech support). By offering different kinds of aid, participant units could persuade hostiles to become friendly. Participant units differed in terms of their movement speed, fuel capacity, and detection range and how effective they were at persuading different types of hostile locals. A participant’s level of influence increased over the region of responsibility by keeping a restricted zone free (shaded, central region of the map) of hostile locals. Locals appeared in random locations on the perimeter of the map and then moved toward the restricted zone. If hostile locals reached the restricted zone, a participant’s level of influence would decrease (1 point per second per hostile local). The left-side panel displayed information regarding the capabilities and status of selected units and locals as well as the participant’s score.

The performance environment reflected an open-loop system involving continuous changes in stimuli with no definitive endpoint signaling task completion. Four interdependent subtasks composed the peacekeeping game: (a) detecting (searching for) locals, (b) distinguishing between friendly and hostile locals, (c) arranging units to persuade hostile locals, and (d) persuading hostile locals. Participants could select their units and identify locals using the left mouse button and could arrange their units and persuade locals using the right mouse button. In general, the task was designed to be fairly overwhelming as the number of locals appearing on the screen steadily increased over the course of a mission. In this way, scores tended to decrease over the course of a mission.

Procedures

At the onset of their participation, participants were told that the purpose of this study was to examine how different people learn to perform new and challenging tasks. After a training presentation, participants performed a 15-min practice mission. Following the practice mission, participants performed five actual missions (each 15 min). All missions were paused every 2 min, and the participants would indicate the level of workload they were experiencing right before the mission was paused.

Manipulation of Task Demand

Missions 1, 3, and 5 were similar to the practice mission in terms of the behavior of the locals and were used to determine the typical level of subjective workload for person-mean centering within-person effects. Missions 2 and 4 were two counterbalanced missions involving shifts in task demand. Separate analyses were conducted by shift condition for Missions 2 and 4, such that assessments from Missions 1 through 3 were used for analyses involving Mission 2 and Missions 3 through 5 for analyses involving Mission 4. Between-person subjective workload was grand-mean centered. During Missions 2 and 4, task demands were increased for 1 min and decreased for 1 min. Specifically, 4 min into the mission, task demands either increased or decreased, and at 8 min into the mission, task demands either decreased or increased (opposite the direction of the shift at Minute 4). During a 1-min interval, the level of task demand was determined by the number of locals entering the region of responsibility, depending on whether there was an increase (i.e., seven new friendly and seven hostile locals) or decrease (i.e., zero and zero) as compared to the normal level of task demand (i.e., three and three). No special instructions regarding these shifts were provided at any time before or during the missions. The effects of the change could last up to 3 min, which was the time it took a local to move from the perimeter of the region to the center of the restricted zone.

Subjective Workload

A single-item subjective workload measure was adapted from previous research (Grech, Neal, Yeo, Humphreys, & Smith, 2009; Yeo & Neal, 2008). The item read, “Indicate on the line below, the level of workload you were experiencing just before the screen froze.” Participants responded to this item using a 5-point scale: 1 (little to do; little demands), 2 (active involvement required, but easy to keep up), 3 (challenging, but manageable), 4 (extremely busy, barely able to keep up), and 5 (too much to do; overloaded; postponing some tasks). Correlations between the scores obtained from this single-item measure with scores from the NASA Task Load Index (Hart & Staveland, 1988) taken immediately after each mission supported the validity of the scores for the one-item measure. Specifically, correlations were .36, .43, .64, .54, and .48 (ps < .01) for Missions 1 to 5, respectively.

Results

Table 1 shows the means, standard deviations, and correlations between scores of subjective workload and performance averaged across intervals for each mission.

Table 1:

Means, Standard Deviations, and Correlations for Study Variables at the Between-Person Level

Variable	1	2	3	4	5	6	7	8	9	10
1. SWL 1^a	—
2. SWL 2	.73	—
3. SWL 3	.67	.80	—
4. SWL 4	.55	.64	.81	—
5. SWL 5	.54	.54	.75	.90	—
6. Perf. 1	−.52	−.34	−.41	−.30	−.27	—
7. Perf. 2	−.31	−.43	−.44	−.32	−.27	.70	—
8. Perf. 3	−.41	−.39	−.56	−.47	−.43	.65	.76	—
9. Perf. 4	−.22	−.17	−.39	−.51	−.46	.58	.65	.77	—
10. Perf. 5	−.22	−.09	−.31	−.44	−.48	.52	.55	.70	.84	—
M	3.21	2.94	2.71	2.73	2.56	101.99	201.29	556.23	494.81	753.11
SD	0.88	0.86	0.91	0.89	0.98	763.22	645.38	746.01	707.48	694.40

Note. N = 75. SWL = subjective workload; Perf. = performance. r > |.23| = p < .05. r > |.30| = p < .01.

Mission number.

Consistent with Hypothesis 1, higher levels of subjective workload were associated with lower levels of performance (rs from −.43 to −.56, p < .01) for every mission. Hypotheses concerning how subjective workload is related to performance were examined using hierarchical linear modeling (HLM), using the performance scores across 2-min intervals for each mission as the dependent variable. We followed the guidelines for growth curve modeling put forth by Bliese and Ployhart (2002). Separate HLM analyses were conducted using maximum likelihood estimation and an unconstrained covariance structure by shift condition for Missions 2 and 4 in which the shifts in task demand occurred. The intercept represents performance at Minute 2, and the Level 1 predictors included fixed linear and quadratic trajectory parameters, within-person subjective workload, and the cross-level interactions. As commonly recommended, the criterion for testing between-person interactions with linear and quadratic performance was set to p < .10 to compensate for low power due to reductions in parameter reliability (Mathieu, Aguinis, Culpepper, & Chen, 2012). Table 2 summarizes the HLM results.

Table 2:

Summary of Hierarchical Linear Modeling Analyses Predicting Performance for Missions Involving Shifts in Task Demand

	Shifts in Task Demand
	Early Increase/Late Decrease		Early Decrease/Late Increase
Fixed Effects	Mission 2	Mission 4	Mission 2	Mission 4
Intercept^a (β₀₀)	1.12 (.10)**	1.04 (0.13)**	0.50 (0.11)**	0.32 (0.13)*
Level 1
Linear performance (β₁₀)	−0.54 (0.05)**	−0.49 (0.07)**	0.25 (0.06)**	0.24 (0.06)**
Quadratic performance (β₂₀)	0.03 (0.01)**	0.04 (0.01)**	−0.07 (0.01)**	−0.07 (0.01)**
WP SWL^a (β₃₀)	0.24 (0.06)**	0.39 (0.08)**	−0.17 (0.07)*	−0.12 (0.07)
Linear Performance × WP SWL (β₄₀)	−0.17 (0.04)**	−0.22 (0.06)**	0.12 (0.05)*	0.10 (0.05)*
Quadratic Performance × WP SWL (β₅₀)	0.02 (0.01)**	0.02 (0.01)*	−0.02 (0.01)*	−0.02 (0.01)*
Linear Performance × BP SWL (β₁₁)	−0.20 (0.06)**	−0.14 (0.06)*	−0.10 (0.06)†	−0.01 (0.07)
Quadratic Performance × BP SWL (β₂₁)	0.01 (0.01)	0.00 (0.01)	0.01 (0.01)	−0.03 (0.01)
Level 2
BP SWL^a (β₀₁)	0.07 (.12)	0.08 (.14)	0.03 (.12)	−0.04 (0.14)
Level 1 df	239	197	197	239
Level 2 df	39	31	31	39

Note. Values represent standardized parameter estimates. Parenthetical values indicate standard errors. The intercept reflects performance at Minute 2. SWL = subjective workload; BP = between-person; WP = within-person.

Effects are represented at Minute 2.

†

p < .10. *p < .05. **p < .01 (two tailed).

Between-Person Effects of Subjective Workload

As shown in Table 2, the between-person main effect of subjective workload early in each mission (Minute 2, β₀₁) was nonsignificant, but there were significant interactions involving between-person subjective workload and the linear performance trend (β₁₁). These interactions speak to Research Question 1 regarding dynamic between-person effects and specifically reflect how negative between-person effects emerged and became stronger in later performance intervals. Specifically, although between-person effects did not interact with the quadratic performance parameter, between-person effects accentuated the negative linear performance trajectory. However, this interaction was stronger and more consistently observed in missions involving early increases rather than early decreases in task demand. The significant interaction with the linear trend coupled with nonsignificant between-person main effects at Minute 2 and nonsignificant interactions with the quadratic trend reflect dynamic effects in terms of magnitude but not direction.

These dynamic effects coupled with the aforementioned nonsignificant between-person main effects provide mixed support for Hypothesis 1 in that negative between-person effects were observed but not until later performance intervals. In fact, and in support of Hypothesis 1, tests of the between-person main effect at Minute 14 in each mission showed that individuals who experienced more subjective workload on average performed worse than their counterparts who experienced less subjective workload (βs ranged from −0.35 to −1.06, ps < .01).

Within-Person Effects of Subjective Workload

Consistent with our general prediction that within- versus between-person effects would be more dynamic, the results showed that linear and quadratic performance trajectories were more consistently accentuated by within-person compared to between-person subjective workload. As shown in Table 2 and consistent with Hypothesis 2, within-person effects interacted with both the linear (Linear Performance × Within-Person Subjective Workload) and quadratic performance trajectories (Quadratic Performance × Within-Person Subjective Workload) in both shift conditions. Figure 1 illustrates how the within-person effects accentuated the performance trajectories as a function of the shifts in task demand.

Figure 1.

Interactions between linear performance, quadratic performance, and within-person subjective workload (SWL). Performance and subjective workload reflect standardized scores.

Table 3 shows the within-person relationship at each 2-min interval by shift condition.

Table 3:

Summary of Within-Person Effects of Subjective Workload at Each Interval

Shifts in Task Demand
Early Increase/Late Decrease			Early Decrease/Late Increase
Minute	Mission 2	Mission 4	Minute	Mission 2	Mission 4
2	.24 (.06)**	.39 (.08)**	2	−.17 (.07)*	−.12 (.07)
4	.10 (.04)*	.19 (.05)*	4	−.07 (.05)	−.04 (.06)
6^a	−.01 (.04)	.02 (.05)	6^b	.00 (.05)	.01 (.05)
8^c	−.07 (.04)†	−.10 (.05)*	8^d	.03 (.05)	.03 (.05)
10^b	−.09 (.04)*	−.19 (.05)*	10^a	.02 (.04)	.02 (.04)
12^d	−.06 (.04)	−.24 (.05)*	12^c	−.02 (.04)	−.02 (.04)
14	.00 (.07)	−.26 (.09)*	14	−.10 (.07)	−.10 (.08)

Note. Values represent the standardized relationship between within-person subjective workload (β₃₀) and performance at each 2-min interval. Parenthetical values indicate standard errors.

Immediate performance following increases in task demand.

Immediate performance following decreases in task demand.

Downstream performance following increases in task demand.

Downstream performance following decreases in task demand.

†

p < .10. *p < .05. **p < .01 (two tailed).

In support of Hypothesis 2, within-person subjective workload demonstrated both positive and negative relationships with performance. However, both positive and negative effects were observed only in missions involving early increases in task demand.

Immediate effects

To test Hypotheses 3 and 4, immediate performance was operationalized as the 2-min mark immediately following a shift, and the effect of within-person subjective workload (β₃₀) at this particular interval was examined. Specifically, this was Minute 6 when a shift occurred earlier in a performance period and Minute 10 when a shift occurred later in a performance period. As shown in Table 3, the results did not support Hypothesis 3. Specifically, when increases occurred early in a performance period, at Minute 6, within-person subjective workload was not related to performance (Mission 2, β₃₀ = −.01, p > .05; Mission 4, β₃₀ = .02, p > .05). Similarly, when increases occurred later in a performance period following earlier decreases in task demand, at Minute 10, within-person subjective workload was not related to performance (Missions 2 and 4, β₃₀ = .02, p > .05). The results also did not support Hypothesis 4. When decreases occurred early in a performance period before any increases occurred, at Minute 6, within-person subjective workload was not related to performance (Mission 2, β₃₀ = .00; Mission 4, β₃₀ = .01, p > .05). Opposite to Hypothesis 4, when decreases occurred later in a performance period following increases in task demand, at Minute 10, within-person subjective workload was negatively related to performance (Mission 2, β₃₀ = −.09, p < .05; Mission 4, β₃₀ = −.19, p < .01).

Downstream effects

To test Hypotheses 5 and 6, downstream performance was operationalized as the second 2-min mark (i.e., 4 min later) following when a shift in task demand began, and the effect of within-person subjective workload at this particular mark was examined. Specifically, this was Minute 8 when a shift occurred early in a performance period and Minute 12 when a shift occurred later in a performance period following an earlier shift. As shown in Table 3, the results offered mixed support for Hypothesis 5. When increases in task demand occurred early in the performance episode, the results supported Hypothesis 5. As shown in Table 3, at Minute 8, within-person subjective workload was negatively related to performance (Mission 2, β₃₀ = −.07, p < .05, one tailed; Mission 4, β₃₀ = −.10, p < .05). Furthermore, these downstream negative effects occurred even more downstream following later decreases in task demand at Minute 10 for both Mission 2 (β₃₀ = −.09, p < .05) and Mission 4 (β₃₀ = −.19, p < .01) and for Mission 4 at Minute 12 (β₃₀ = −.24, p < .01) and Minute 14 (β₃₀ = −.26, p < .01). When increases in task demand occurred later in a performance period, a similar pattern of effects occurred such that the negative effects were stronger more downstream; however, the magnitudes of the effects were not significant. The results did not support Hypothesis 6. Specifically, when decreases occurred early in a performance period, at Minute 8, within-person subjective workload was not related to performance (Missions 2 and 4, β₃₀ = .03, p > .05).

Discussion

By taking a longitudinal, multilevel approach coupled with manipulations of task demand, this study showed that it is important to disaggregate between- and within-person effects to model and distinguish the dynamic relationships between trait and state components of subjective workload and performance. The between-person effects, representing differences in the relatively stable capability to self-regulate, were negatively related to performance in general. However, this effect became more pronounced throughout a period of performance, especially following early increases in task demand. The within-person effects, representing different control states (e.g., appropriate vs. overload) in relation to the shifts in task demand and workload history, yielded dynamic effects that changed in both magnitude and direction. In this way, state and trait-like components of subjective workload yield different relationships with performance. Between-person relationships underscore the importance of selective attention over a period of performance, as better self-regulators are less likely to experience the potentially negative aftereffects of stress. Despite a lack of support for three of our hypotheses, within-person relationships changed in a manner that was largely consistent within a cognitive-energetic framework. That is, within-person fluctuations can represent either an overload or an appropriate state of control depending on shifts in task demand and workload history. Despite these differences in effects across levels of analyses, the pattern of effects converged such that negative effects across both levels were evident downstream from increases in task demand.

In this way, it is important to distinguish multilevel effects to better understand immediate versus downstream effects in response to shifts in task demand. As a function of different control states, subjective workload–performance relationships observed prior to or immediately after an increase in task demand are different compared to those more downstream. If an early increase in task demand is experienced, effortful processing becomes more challenging, which would be characteristic of an unfavorable transition state, such as overload, which is captured by positive within-person fluctuations in subjective workload. However, such increases are negatively related to performance only when assessed downstream from the increase in task demand. The present findings showed that when fluctuations in subjective workload were examined closer to when increases in task demand occurred, a positive or nonsignificant relationship, reflecting a stable control state, was observed. Similarly, the between-person effect becomes stronger in magnitude throughout a performance episode particularly when individuals experience increases in task demand early in a performance episode. Finally, given that effects were more dynamic when early increases were followed by later decreases rather than vice versa, it is important to study dynamic multilevel effects in relation to shifts in task demand. Taken together, the nature of these dynamic multilevel effects accounts for why sometimes an observed dissociation between subjective workload and performance is demonstrated in cross-sectional research that does not account for within-person effects.

By looking at changes within performance episodes, insight has been gained into how the state component of subjective workload relates to the dynamics of human performance. The cognitive-energetic perspective theorizes that both lower (i.e., underload) and higher levels of subjective workload (i.e., overload) are negatively related to performance (Hockey, Briner, Tattersall, & Wiethoff, 1989). Contrary to underload, the present study suggests even when subjective workload is positively related to performance earlier in a performance episode, the consequences of the early allocation of resources is manifested more downstream, adversely affecting performance, which is captured well with repeated measurements. Similarly, the present findings support the principle of scarcity, such that as the absolute levels of task demands individuals experience increase, cognitive costs increase and it becomes more challenging for individuals to keep up with task demands.

Limitations and Directions for Future Research

There are several limitations to the present study. First, when shifts in task demand are counterbalanced within a performance episode, the dynamic, within-person main effects are difficult to discern. For example, it is likely the downstream effects of an increase were affecting the assessments of subjective workload when the subsequent decrease in task demand occurred. Likewise, when increases occurred later in the performance episode, it is likely the mission ended before participants experienced the downstream effects of the increase. In this way, the present study potentially underestimated the influence of downstream effects of stress on performance. Second, it is possible the operationalizations of immediate and downstream effects were not sensitive enough to the hypothesized dynamic within-person effects when considering the frequency of the subjective workload assessments in relation to the duration of task demand shifts. Because subjective workload was measured every 2 min rather than more frequently, like every minute or 30 s, immediate within-person effects may have been underestimated, which could explain why support was not found for the hypotheses concerning immediate effects although the overall pattern of within-person effects followed the predicted dynamic trend. Similarly, although within-person subjective workload was negatively related to performance downstream from increases in task demand, it is uncertain when such negative effects first emerge and for how long they occur.

In trying to address the issue of the timing of the aftereffects of stress on performance, catastrophe models have recently been applied and hold considerable promise (Guastello et al., 2013; Guastello, Boeh, Schimmels, et al., 2012; Guastello, Boeh, Shumaker, & Schimmels, 2012). Specifically, this approach models nonlinear dynamical processes to predict future performance as a function of past performance. In this way, catastrophe models can be potentially useful for separating workload effects from fatigue effects vis-à-vis nonlinear performance trajectory parameters. Moreover, similar to time-invariant variables from an HLM perspective, catastrophe models can incorporate individual differences (i.e., time-invariant variables) to account for variability in the nonlinear trajectory of performance. However, these models have yet to account for subjective experiences (e.g., subjective workload), especially in regard to time-varying influences (i.e., within-person effects) when responding to stress. In this way, state components of subjective experiences have yet to be linked to performance. Future research should address the role that workload history plays, not only in regard to distinguishing state and trait-like components of subjective workload but also with respect to disentangling workload effects from fatigue effects.

Third, single-item assessments are limited with respect to indicating the source of variation and quantifying the contribution by type of resource demand (i.e., diagnosticity; O’Donnell & Eggemeier, 1986). In the present laboratory study, due to the limited sources of task variation and task demand, one’s experience of subjective workload is relatively narrow and unambiguous; in such cases, single-item measures are suitable for discriminating between levels of subjective workload (Sackett & Larson, 1990). Nevertheless, future research is needed to examine if there are differences between dimensions of subjective workload in terms of their state and trait components and the degree to which these components for the different dimensions are differentially related to performance in relation to shifts in task demand and workload history (cf. Helton et al., 2014). Accordingly, multidimensional assessments may help isolate workload effects from fatigue effects. However, researchers should be mindful of how concurrent assessments using multi-item scales are likely more disruptive to task performance.

Fourth, individual differences variables—in addition to the between-person effect of subjective workload (i.e., stable differences in characteristic patterns of cognitive activity when responding to stress)—need to be investigated. Future investigations should explore this stable difference with other between-person variables related to cognitive resources (e.g., general mental ability, expertise) or personality, such as neuroticism, extraversion, or conscientiousness (Carver & Connor-Smith, 2010). With respect to resource allocation, future research should speak more directly to the level of processing occurring in performance episodes. The relationships between shifts in task demand, stress, the components of subjective workload, and performance could be different if participants received enough practice to where performance started to asymptote and the level of processing could be considered “automatic” versus “controlled.” In the present study, participants were given a brief period of training and practice on a complex task, and performance scores showed improvement across missions. As such, it can be inferred that many of the participants were in earlier stages of skill acquisition characterized by more controlled processing. Accounting for the nature of cognitive processing (i.e., automatic vs. controlled) may help to better explain the manner in which state and trait components of subjective workload relate to performance.

Finally, given the student sample and lab context of this study, it is important to extend this research to more real-world environments that represent contexts in which errors result in potentially serious consequences. Related, the relationship between subjective workload and performance during complex decision-making and problem-solving tasks may be different when compared to more sensory or vigilance tasks, especially with respect to underload (See, Howe, Warm, & Dember, 1995).

Key Points

With the use of a longitudinal, multilevel framework, state (i.e., within-person differences) and trait (i.e., between-person differences) components of subjective workload can be distinguished and related to performance over a given period of activity.

In relation to shifts in task demand and workload history, state and trait components yield different dynamic relationships with performance.

Although both state and trait relationships with performance can be dynamic, state relationships are more dynamic, potentially yielding positive and negative effects depending on shifts in task demand and workload history.

Despite differences in their patterns, state and trait effects can converge such that negative relationships with performance for both components are likely to occur downstream from increases in task demand.

Footnotes

Derek L. Mracek is in the industrial and organizational psychology doctoral program at the University of Oklahoma. He earned an MA in industrial and organizational psychology from East Carolina University in 2011 and a BS in psychology from the University of Minnesota Duluth in 2006.

Matthew L. Arsenault is in the industrial and organizational psychology doctoral program at the University of Oklahoma. He served 4 years in the U.S. Air Force before earning his BS in psychology from the University of North Dakota in 2007 and his MS in industrial and organizational psychology from the University of Oklahoma in 2011.

Eric Anthony Day is an associate professor of psychology and faculty member in the industrial and organizational psychology doctoral program and a research associate for the Center for Applied Social Research at the University of Oklahoma. He earned his PhD in industrial and organizational psychology from Texas A&M University in 1998.

Jay H. Hardy III is in the industrial and organizational psychology doctoral program at the University of Oklahoma. He earned his MS in industrial and organizational psychology from the University of Oklahoma in 2012 and a BS in psychology from Colorado State University in 2009.

Robert A. Terry is an associate professor in the Department of Psychology at the University of Oklahoma. He received his PhD in quantitative psychology in 1989 from the University of North Carolina at Chapel Hill. He specializes in the areas of quantitative modeling, measurement, and the analysis of very large and complex data sets.

References

Aptima. (2007). DDD v4.0 scenario writing guide: Revision 2. Woburn, MA: Author.

Bliese

P. D.

Ployhart

R. E.

(2002). Growth modeling using random coefficient models: Model building, testing, and illustrations. Organizational Research Methods, 5, 362–387. doi:10.1177/109442802237116

Carver

C. S.

Connor-Smith

(2010). Personality and coping. Annual Review of Psychology, 61, 679–704. doi:10.1146/annurev.psych.093008.100352

Cohen

(1980). Aftereffects of stress on human performance and social behavior: A review of research and theory. Psychological Bulletin, 88(1), 82–108. doi:10.1037/0033-2909.88.1.82

Cox-Fuenzalida

L.-E.

(2007). Effect of workload history on task performance. Human Factors, 49, 277–292. doi:10.1518/001872007X312496

Cumming

R. W.

Croft

P. G.

(1973). Human information processing under varying task demand. Ergonomics, 16, 581–586. doi:10.1080/00140137308924548

Curran

P. J.

Bauer

D. J.

(2011). The disaggregation of within-person and between-person effects in longitudinal models of change. Annual Review of Psychology, 62, 583–619. doi:10.1146/annurev.psych.093008.100356

Eggemeier

F. T.

Shingledecker

C. A.

Crabtree

M. S.

(1985). Workload measurement in system design and evaluation. In Proceedings of the Human Factors and Ergonomics Society 29th Annual Meeting (pp. 215–219). Santa Monica, CA: Human Factors and Ergonomics Society.

Goldberg

R. A.

Stewart

M. R.

(1980). Memory overload or expectancy effect? “Hysteresis” revisited. Ergonomics, 23, 1173–1178. doi:10.1080/00140138008924824

10.

Gopher

Donchin

(1986). Workload: An examination of the concept. In Boff

K. R.

Kaufman

Thomas

J. P.

(Eds.), Handbook of perception and human performance (pp. 1–49). New York, NY: Wiley.

11.

Grech

M. R.

Neal

Yeo

Humphreys

Smith

(2009). An examination of the relationship between workload and fatigue within and across consecutive days of work: Is the relationship static or dynamic? Journal of Occupational Health Psychology, 14, 231–242. doi:10.1037/a0014952

12.

Guastello

S. J.

Boeh

Gorin

Huschen

Peters

N. E.

Fabisch

Poston

(2013). Cusp catastrophe models for cognitive workload and fatigue: A comparison of seven task types. Nonlinear Dynamics, Psychology, and Life Sciences, 17, 23–47.

13.

Guastello

S. J.

Boeh

Schimmels

Gorin

Huschen

Davis

Peters

N. E.

Fabisch

Poston

(2012). Cusp catastrophe models for cognitive workload and fatigue in a verbally cued pictorial memory task. Human Factors, 54, 811–825. doi:10.1177/0018720812442537

14.

Guastello

S. J.

Boeh

Shumaker

Schimmels

(2012). Catastrophe models for cognitive workload and fatigue. Theoretical Issues in Ergonomics Science, 13, 586–602. doi:10.1080/1463922X.2011.552131

15.

Hancock

P. A.

Warm

J. S.

(1989). A dynamic model of stress and sustained attention. Human Factors, 31, 519–537. doi:10.1177/001872088903100503

16.

Hancock

P. A.

Williams

Manning

C. M.

(1995). Influence of task demand characteristics on workload and performance. International Journal of Aviation Psychology, 5, 63–86. doi:10.1207/s15327108ijap0501_5

17.

Hart

S. G.

Staveland

L. E.

(1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human Mental Workload, 1, 139–183. doi:10.1016/S0166-4115(08)62386-9

18.

Helton

W. S.

Funke

G. J.

Knott

B. A.

(2014). Measuring workload in collaborative contexts trait versus state perspectives. Human Factors, 56, 322–332. doi:10.1177/0018720813490727

19.

Helton

W. S.

Shaw

Warm

J. S.

Matthews

Hancock

(2008). Effects of warned and unwarned demand transitions on vigilance performance and stress. Anxiety, Stress, & Coping, 21, 173–184. doi:10.1080/10615800801911305

20.

Hill

S. G.

Zaklad

A. L.

Bittner

A. C.

Byers

J. C.

Christ

R. E.

(1988). Workload assessment of a mobile air defense missile system. In Proceedings of the Human Factors and Ergonomics Society 32nd Annual Meeting (pp. 1068–1072). Santa Monica, CA: Human Factors and Ergonomics Society.

21.

Hockey

G. R. J.

(1984). Varieties of attentional state: The effects of environment. In Parasuraman

Davies

D. R.

(Eds.), Varieties of attention (pp. 449–483). New York, NY: Academic Press.

22.

Hockey

G. R. J.

(1986). A state control theory of adaptation and individual differences in stress management. In Hockey

G. R. J.

Gaillard

A. W. K

Coles

M. G. H.

(Eds.), Energetics and human information processing (pp. 285–298). Leiden, Netherlands: Martinus Nijhoff. doi:10.1007/978-94-009-4448-0_19

23.

Hockey

G. R. J.

(1993). Cognitive-energetical control mechanisms in the management of work demands and psychological health. In Baddeley

A. D.

Weiskrantz

(Eds.), Attention, selection, awareness and control: A tribute to Donald Broadbent (pp. 328–345). Oxford, UK: Oxford University Press.

24.

Hockey

G. R. J.

(1997). Compensatory control in the regulation of human performance under stress and high workload: A cognitive-energetical framework. Biological Psychology, 45, 73–93. doi:10.1016/S0301-0511(96)05223-4

25.

Hockey

G. R. J.

Briner

R. B.

Tattersall

A. J.

Wiethoff

(1989). Assessing the impact of computer workload on operator stress: The role of system controllability. Ergonomics, 32, 1401–1418. doi:10.1080/00140138908966914

26.

Hockey

G. R. J.

Hamilton

(1983). The cognitive patterning of stress states. In Hockey

G. R. J.

(Ed.), Stress and fatigue in human performance (pp. 331–362). New York, NY: Wiley.

27.

Holding

(1983). Fatigue. In Hockey

G. R. J.

(Ed.), Stress and fatigue in human performance (pp. 145–164). New York, NY: Wiley.

28.

Humphreys

M. S.

Revelle

(1984). Personality, motivation, and performance: A theory of the relationship between individual differences and information processing. Psychological Review, 91, 153–184. doi:10.1037/0033-295X.91.2.153

29.

Kahneman

(1973). Attention and effort. Englewood Cliffs, NJ: Prentice Hall.

30.

Krulewitz

J. E.

Warm

J. S.

Wohl

T. H.

(1975). Effects of shifts in the rate of repetitive stimulation on sustained attention. Perception & Psychophysics, 18, 245–249. doi:10.3758/BF03199369

31.

Mathieu

J. E.

Aguinis

Culpepper

S. A.

Chen

(2012). Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling. Journal of Applied Psychology, 97, 951–966. doi:10.1037/a0028380

32.

Matthews

(1986). The influence of visual workload history on visual performance. Human Factors, 28, 623–632. doi:10.1177/001872088602800601

33.

Messick Huey

Wickens

C. D.

(Eds.). (1993). Workload transition: Implications for individual and team performance. Washington, DC: National Academy Press.

34.

Moroney

B. W.

Warm

J. S.

Dember

W. N.

(1995). Effects of demand transitions on vigilance performance and perceived workload. In Proceedings of the Human Factors and Ergonomics Society 39th Annual Meeting (pp. 1375–1379). Santa Monica, CA: Human Factors and Ergonomics Society.

35.

Nataupsky

Abbott

T. S.

(1987). Comparison of workload measures on computer-generated primary flight displays. In Proceedings of the Human Factors and Ergonomics Society 31st Annual Meeting (pp. 548–552), Santa Monica, CA: Human Factors and Ergonomics Society.

36.

Nygren

T. E.

(1991). Psychometric properties of subjective workload measurement techniques: Implications for their use in the assessment of perceived mental workload. Human Factors, 33, 17–33. doi:10.1177/001872089103300102

37.

O’Donnell

R. D.

Eggemeier

F. T.

(1986). Workload assessment methodology. In Boff

K. R.

Kaufman

Thomas

J. P.

(Eds.), Handbook of perception and human performance: Vol. 2. Cognitive processes and performance (pp. 42.41–42.49). New York, NY: Wiley.

38.

Rasmussen

(1986). Human information processing and human machine interaction. Amsterdam, Netherlands: North Holland.

39.

Sackett

P. R.

Larson

J. R.

(1990). Research strategies and tactics in industrial and organizational psychology. In Dunnette

M. D.

Hough

L. M.

(Eds.), Handbook of industrial and organizational psychology (pp. 419–489). Palo Alto, CA: Consulting Psychologists Press.

40.

See

J. E.

Howe

S. R.

Warm

J. S.

Dember

W. N.

(1995). Meta-analysis of the sensitivity decrement in vigilance. Psychological Bulletin, 117, 230–249. doi:10.1037/0033-2909.117.2.230

41.

Tsang

P. S.

Vidulich

M. A.

(1994). The roles of immediacy and redundancy relative subjective workload assessment. Human Factors, 36, 503–513. doi:10.1177/001872089403600307

42.

Verwey

W. B.

Veltman

H. A.

(1996). Detecting short periods of elevated workload: A comparison of nine workload assessment techniques. Journal of Experimental Psychology: Applied, 2, 270–285. doi:10.1037/1076-898X.2.3.270

43.

Vidulich

M. A.

Bortolussi

M. R.

(1988, April). Speech recognition in advanced rotorcraft: Using speech controls to reduce manual control overload. Paper presented at the Proceedings of the American Helicopter Society National Specialists’ Meeting, Atlanta, GA.

44.

Vidulich

M. A.

Tsang

P. S.

(1987). Absolute magnitude estimation and relative judgement approaches to subjective workload assessment. In Proceedings of the Human Factors and Ergonomics Society 31st Annual Meeting (pp. 1057–1061). Santa Monica, CA: Human Factors and Ergonomics Society.

45.

Wickens

C. D.

(1984). Processing resources in attention. In Parasuraman

Davies

D. R.

(Eds.), Varieties of attention (pp. 63–102). New York, NY: Academic Press.

46.

Wood

R. E.

(1986). Task complexity: Definition of the construct. Organizational Behavior and Human Decision Processes, 37, 60–82. doi:10.1016/0749-5978(86)90044-0

47.

Yeh

Wickens

C. D.

(1988). Dissociation of performance and subjective measures of workload. Human Factors, 30, 111–120. doi:10.1177/001872088803000110

48.

Yeo

Neal

(2008). Subjective cognitive effort: A model of states, traits, and time. Journal of Applied Psychology, 93, 617–631. doi:10.1037/0021-9010.93.3.617