Abstract
Objective:
Our objective was to explore the value of considering the number of tasks that use a piece of information when calculating the relevance information has to an operator.
Background:
Whereas frequency and criticality of information are often identified as information attributes, the number of tasks that use the information is rarely considered.
Method:
We calculated the relevance of pieces of information in air traffic control using criticality and frequency, and compared it to a formula that also considered the number of tasks.
Results:
Including number of tasks resulted in information ranking that better accounted for aircraft relevant information, and better supported the information needs of air traffic controllers as determined by judgments of controllers.
Conclusion:
The attribute of number of tasks is valuable in calculating the relevance of information.
Application:
Interface designers should consider the number of tasks that use a particular piece of information when determining the placement of information within a display.
Introduction
Operators in safety-critical environments make substantial use of visual displays to extract important information to achieve task goals. For example, air traffic controllers rely on monitoring radar screens to identify information about the route, speed, and altitude of aircraft to maintain safe separation distances between aircraft and efficiently guide the flow of traffic. Task analyses can be used to identify the specific pieces of information (hereafter referred to as information objects) that an operator requires at each step of a task to achieve an overall task goal.
Information objects have particular attributes associated with their use in specific tasks. Certain attributes may be more influential than others in informing design decisions regarding the placement of information objects within a display. Frequency and criticality are two examples of such information attributes that are commonly identified in the literature (Caffiau, Scapin, Girard, Baron, & Jambon, 2010; Hamilton, 2000; Wickens, Lee, Liu, & Gordon Becker, 2004). Specifically, frequently used information should be placed in or near to the operator’s primary visual field so that the objects can be accessed quickly and efficiently (Wickens et al., 2004). Similarly, critical information should be placed within the operator’s primary visual field to increase the likelihood of capturing the operator’s attention. An information object that is used in a frequently performed critical task is likely to be particularly relevant to achieving task goals. Information that is relevant improves the operator’s representation of the task environment and reduces the cognitive effort associated with information processing. For example, Wickens, Goh, Helleberg, Horrey, and Talleur (2003) in their SEEV model consider Value of the information as one of the important attributes in directing attention.
Typically, the literature often focuses on the characteristics of frequency and criticality with reference to the tasks being performed. However, the utility of the attribute of the total number of tasks needing an information object is rarely mentioned (cf. Durso, Sethumadhavan, & Crutchfield, 2008; Durso, Sethumadhavan, Crutchfield, & Morris, 2011). Using a task analysis, the number-of-tasks attribute is associated with an information object by finding all of the tasks wherein that information object is identified as information received by the operator. For example, if the information object primary target is found as information received in 21 different tasks, then the number of tasks attribute will be substantiated with a value of 21. Durso et al. (2008) calculated the relevance of information displayed on radar screens of air traffic controllers operating at tower facilities by standardizing the value of three attributes in the task analyses: frequency, criticality, and number of tasks. However, because previous literature has largely focused on frequency and criticality as information attributes, it is unclear whether number of tasks provided additional value in determining the relevance of information.
By including the number of tasks in which an information object is used, it might be possible to develop a more precise relevance formula. If number of tasks proves to be an influential information attribute, then including it in relevance formulas could prove beneficial for modeling how an optimal scanner should attend to information in his or her workspace or predicting which pieces of information an operator would prefer.
One reason that the concept of number of tasks may not be perceived as an information attribute is because of its association with the concept of frequency. Both number of tasks and frequency provide information about the rate of using an information object. Whereas number of tasks provides a value corresponding to the total number of activities that use an information object, frequency provides information about the rate of performing individual activities. This suggests an inherent difference between number of tasks and frequency, thereby allowing for each attribute to contribute to calculating information relevance. The number-of-tasks attribute might also play an important role in such models. Frequency and number of tasks may operate through different mechanisms. In SEEV, for example, visual attention to a display accumulates across tasks. If a display is relevant, for example, to both aviating and navigating, the probability of attending to the display is increased (Wickens et al., 2003).
The aim of the current study was to find out whether number of tasks provided added value in calculating information relevance. We used task analyses conducted by Alexander, Ammerman, Fairhurst, Hostetler, and Jones (1989, 1990) on En Route and Terminal Radar Control (TRACON) air traffic controllers to compute relevance values of information objects (Durso et al., 2008). In addition, because the Federal Aviation Administration (FAA) has expressed interest in constructing a common workstation capable of being used by both En Route and TRACON controllers (Nextgen; Joint Planning and Development Office, 2011), we explored whether number of tasks similarly influenced relevance calculations when structuring the display of information for a common workstation; global relevance provides such an estimate (see Durso et al., 2008). Thus, we calculated global relevance using both the frequency, criticality, and number of tasks formula, and frequency and criticality formula.
Although we conduct this test of the value added by number of tasks in determining relevance using air traffic control (ATC), we believe task analyses of most domains would reveal that information objects vary in the number of tasks that require them. For example, in health care, some information about the patient and his or her vitals is used in tasks ranging from medication dosage to comfort measures. Even while driving, some information such as the speed of the car is consulted in a variety of tasks such as determining following distance, passing, initiating braking, gear selection, and obeying legal constraints, compared with the information object of revolutions per minute, which is consulted in fewer tasks.
Method
Information Objects
We examined task analyses of En Route and TRACON ATC to derive information objects (Alexander et al., 1989, 1990, Appendix D, Information Received column). Information objects constituted information that was received by a controller while performing tasks. For example, “Review plan view display for potential violation of flow restrictions” was a task, and one of the information objects needed for the task was primary target. Each task could be associated with multiple information objects, and each information object could be required to complete multiple tasks.
Information Attributes
Each task was associated with a frequency and criticality value. Frequency ranged on a scale of 1 to 3 (1 = low frequency, 3 = high frequency), and criticality ranged on a scale of 1 to 4 (1 = low criticality, 4 = extreme criticality). Alexander and colleagues, including two subject matter experts (SMEs), collected the information documented in the task analysis. They reviewed ATC manuals, observed operations at air traffic facilities, and interviewed controllers at a number of facilities (J. Alexander, personal communication, July 12, 2007). Based on the information gathered from the literature, observations, and interviews, they derived task sequences, task frequency, and task criticality ratings leveraged in the current study. The draft task analysis and task criticality and frequency ratings were then reviewed by other air traffic controllers from different facilities before release.
We associated these attributes with information objects by computing the mean frequency and criticality of the tasks corresponding with each object. In addition, we considered the total number of tasks in which each object was used; this constituted the third attribute of the information objects. The value of each information attribute was standardized. The resulting z scores were converted to positive values (see Durso et al., 2008, for details regarding this calculation). In brief, we computed a composite score that compensated for differences in variance and that equally weighted either frequency and criticality or frequency, criticality, and number of tasks. We then converted the scores (to T-scores) such that relevance scores that differed by 10 units were one standard deviation apart.
Procedure
We computed two relevance values for each information object for both TRACON and En Route and also across the two facilities (i.e., common workstation). One value was computed using Durso et al.’s (2008) relevance calculation of frequency, criticality, and number of tasks (FCN formula); the second value only used frequency and criticality (FC formula). The information objects from each formula were rank ordered by their relevance score.
Results and Discussion
The task analyses listed 92 information objects used by controllers in the En Route facility and 89 information objects in the TRACON facility; 24 information objects appeared in both task analyses. Each information object was associated with three attributes—frequency, criticality, and number of tasks (Alexander et al., 1989, 1990). Table 1 lists the descriptive statistics associated with the three attributes.
Descriptive Statistics of Information Attributes in En Route and Terminal Radar Control (TRACON)
Of course, the relevance scores from the two formulas would be expected to correlate, but the magnitude of the correlations indicates substantial differences. For En Route, scores from FC and FCN correlated, r(90) = .583, p < .001, and for TRACON, r(87) = .579, p < .001. Thus, in both cases only about 33% of the variance in one formula can be accounted by the other.
Some information objects were highly relevant according to both the formulas. To distinguish between the two formulas, we deliberately created distinct lists of relevant objects. Thus, for each formula for each facility we began the list from the most relevant object and proceeded downward in relevance. We tried to choose information objects from the alternative formula that were not similar in rank (i.e., not within 5 rank-ordered positions) and that were not highly ranked in the other formula, but which still allowed us to focus on the top objects. Table 2 shows information ranking in En Route and Table 3 in TRACON facilities produced by the FC and FCN relevance formulas. Ranks for each object for both formulas are shown in the tables.
En Route Information Objects Based on Relevance and Percentage of Controllers Choosing It in a Forced Choice
Participants did not recognize this nomenclature from the task analyses and thus did not complete this forced choice.
Terminal Radar Control (TRACON) Information Objects Based on Relevance and Percentage of Controllers Choosing It in a Forced Choice
We then asked controllers to tell us which of the paired concepts they would prefer to have available at their workstation. We were fortunate to obtain judgments from six controllers who were working at either the Civil Aerospace Medical Institute in Oklahoma City or the FAA Technical Center in Atlantic City. They were given 15 pairs of information objects, 5 pairs from the relevance analyses conducted for each workstation: En Route, TRACON, and common. One member of each pair was ranked highly by the FCN formula and one by the FC formula. The six SME participants made a forced choice for each pair. A majority of the information objects chosen by the SMEs were based on the FCN formula (92% En Route, 73% TRACON). Table 2 and 3 present a breakdown of the controller judgments.
We also were able to combine information relevance from the two types of facilities to compute relevance in a common workstation. Again, the correlation between formulas revealed the shared components, but also a substantial amount of difference, r(155) = .77, p < .001. Table 4 depicts the global information rankings using the two formulas of calculating relevance for the common workstation. When SMEs made forced-choice judgments, as before, a majority of the information preferred by the SMEs was produced from the FCN formula (80%). The most relevant objects produced using the FC formula, such as message waiting alarm and airport altimeter setting, seem especially distant from the task of separating aircraft.
Combined En Route/Terminal Radar Control (TRACON) Information Objects Based on Relevance and Percentage of Controllers Choosing It in a Forced Choice
For the two pairs that did not favor the FCN formula—blinking full data block in TRACON and Mode C altitude in the common workstation—the reversal from the usual pattern was agreed on by all of the controllers. Thus, it seems unlikely these were merely statistical perturbations, but rather these reversals might suggest that additional factors besides the addition of number of tasks may be worth exploring in future research to obtain a complete formula for information relevance.
Conclusion
Number of tasks plays an important role in determining the relevance of information objects. Using a relatively simple method of calculating relevance and using expert judgments, we demonstrated that considering number of tasks along with the traditional attributes of frequency and criticality would improve assessments of information relevance, which in turn would improve display and workstation design.
Key Points
Frequency and criticality are widely recognized as valuable features of information; however, number of tasks may add value when assessing information.
To test the contribution of number of tasks to information relevance, we compared the relevance ranking of information with and without the attribute of number of tasks.
Including number of tasks in the relevance formula resulted in information that was more likely to be chosen by expert air traffic controllers.
This suggests that number of tasks adds value to information relevance calculations that can in turn be used to design a workstation that is more likely to be used effectively.
Footnotes
Acknowledgements
Thanks to our subject matter experts and to Jerry Crutchfield and Todd Truitt. Special thanks to Larry Cole for consulting with us about the task analyses. We gratefully acknowledge support of this project from AJP-61 of the Federal Aviation Administration.
Francis T. Durso is a professor of psychology at Georgia Institute of Technology. He earned his PhD in experimental psychology from the State University of New York at Stony Brook in 1980.
Sadaf Kazi is a PhD student of psychology at Georgia Institute of Technology. She earned her MS in engineering psychology from Georgia Institute of Technology in 2013.
Cale M. Darling is a PhD student of psychology at Georgia Institute of Technology. He earned his BS in psychology from Oklahoma State University in 2011.