The Application of Reduced-Processing Decision Support Systems to Facilitate the Acquisition of Decision-Making Skills

Abstract

Objective:

The study was designed to examine whether the availability of reduced-processing decision support system interfaces could improve the decision making of inexperienced personnel in the context of firefighting.

Background:

Although research into reduced-processing decision support systems has demonstrated benefits in minimizing cognitive load, these benefits have not typically translated into direct improvements in decision accuracy because of the tendency for inexperienced personnel to focus on less-critical information. The authors investigated whether reduced-processing interfaces that direct users’ attention toward the most critical cues for decision making can produce improvements in decision-making performance.

Method:

Novice participants made incident command-related decisions in experimental conditions that differed according to the amount of information that was available within the interface, the level of control that they could exert over the presentation of information, and whether they had received decision training.

Results:

The results revealed that despite receiving training, participants improved in decision accuracy only when they were provided with an interface that restricted information access to the most critical cues.

Conclusion:

It was concluded that an interface that restricts information access to only the most critical cues in the scenario can facilitate improvements in decision performance.

Application:

Decision support system interfaces that encourage the processing of the most critical cues have the potential to improve the accuracy and timeliness of decisions made by inexperienced personnel.

Keywords

decision making expertise cognitive load interface skill acquisition

Introduction

Recent research has highlighted the potential threat to many high-reliability systems as a consequence of the impending exodus of highly skilled members of the workforce as they reach retirement age (Perry, Wiggins, Childs, & Fogarty, 2012). The potential threat to industrialized systems as a result of inexperienced personnel filling the vacated positions has instigated research into the design of systems to support these personnel as they acquire expertise. One approach has involved investigations into decision support systems (DSSs) that are designed to reduce the demands on working memory during the decision-making process, so-called reduced-processing DSSs (Morrison, Wiggins, & Porter, 2010; Perry et al., 2012). However, the value of reduced-processing DSSs remains unclear, particularly in terms of their utility in improving the performance of inexperienced practitioners.

In the context of time-critical environments, accurate and efficient performance is dependent on operators’ capacity to formulate rapid decisions using relatively limited information (Anderson, 1993; Gonzalez, Lerch, & Lebiere, 2003; Rasmussen, 1990; Schriver, Morrow, Wickens, & Taileur, 2008; Wiggins, 2006). For instance, expert fire ground commanders (FGCs) use a relatively small number of key cues in both identifying the nature of the fire and generating an appropriate response (Klein, 1989, 1997). However, the capacity for efficient information acquisition and interpretation appears to be a skill that requires significant task-related experience to develop.

When decision makers are unaware of the ecological validity of cues (Brunswik, 1956), the decision-making process tends to be less efficient, characterized by the processing of information that, although relevant, may not be critical in formulating an accurate response (Lipshitz, Omodei, McLennan, & Wearing, 2007; O’Hare, 1992; Ward, Suss, Eccles, Williams, & Harris, 2011; Wiggins, Stevens, Howard, Henley, & O’Hare, 2002). Efficient information processing is especially significant in time-constrained environments, in which the capacity to identify critical cues is associated with an increased likelihood of an appropriate response and a capacity to avoid miscues (Ward et al., 2011; Ward, Williams, & Bennett, 2002).

The intention of reduced-processing DSSs is to ensure that attentional resources are directed toward processing those cues that are most relevant to a task (Paas, Renkl, & Sweller, 2004). This objective is achieved by restricting access to only those cues that hold the greatest ecological validity. Directing information processing toward critical information not only ensures that this information is acquired, but it also reduces the potential for interference in the retrieval of information from memory. The inability to acquire less-relevant information ensures the presence of fewer memory retrieval cues, resulting in a reduced likelihood of recalling nonessential information that could negatively affect decision accuracy (Ericsson & Kintsch, 1995).

In past research into reduced-processing DSSs, restrictions to the cues that can be accessed have been determined by the user, whereby the cues that are considered most important during the decision-making process are selected prior to engaging the system. Only the cues that the user selected for acquisition could be accessed, thereby reducing the possibility that information would be processed that is less relevant when formulating a response (Morrison et al., 2010; Perry et al., 2012; Wiggins & Bollwerk, 2006).

Both Wiggins and Bollwerk (2006) and Morrison et al. (2010) have investigated the utility of reduced-processing DSSs during decision making. The outcomes of these studies indicate that among experienced operators, restricting the amount of information that participants can acquire results in more rapid decision making with a minimal loss in accuracy. However, neither Wiggins and Bollwerk nor Morrison et al. examined the utility of reduced-processing DSSs as a potential training tool, particularly among less experienced operators.

Restricting access to only critical cues during training has the potential to minimize inefficiencies in the acquisition, evaluation, retention, and subsequent elimination of superfluous information. This is a strategy that has been adopted successfully as part of cue-based learning initiatives, whereby cognitive interviews are used to identify key cues and learners are then able to observe the application of the key cues in the context of the operational environment (Wiggins & O’Hare, 2003).

Despite the potential opportunities afforded by reduced-processing DSSs, their capacity to facilitate skill acquisition is significantly dependent on the availability of highly ecologically valid cues. For example, Perry et al. (2012) were able to demonstrate that when inexperienced FGCs self-selected cues in a reduced-processing DSS, their performance failed to improve in comparison to more experienced participants. Moreover, the cues selected by experienced and inexperienced FGCs differed, presumably because more experienced practitioners were able to identify cues that held greater ecological validity.

Given the risks of self-selected reduced-processing DSSs among inexperienced operators, the alternative is the preselection of cues with a high degree of ecological validity, indicated by those cues that are engaged by highly experienced operators (Shanteau, 1992; Wiggins et al., 2002). This approach to reduced-processing DSSs should provide advantages both in the reduction in cognitive demands and in the capacity for inexperienced operators to attend to cues that embody the greatest ecological validity. Potentially, it optimizes the process of skill acquisition, albeit within relatively specific contexts.

Aim, Design, and Hypotheses

The aim of the present study was to examine the impact on decision accuracy of different approaches to the delivery of reduced-processing DSS interfaces. To maintain a level of consistency with previous research, we conducted the study within the context of firefighting, incorporating both the decision task and the reduced-processing interfaces developed by Perry et al. (2012). The experimental design incorporated interface configuration (preconfigured vs. self-configured), training (training vs. control), and DSS interfaces (intuitive vs. quasi-analytical) as between-groups variables and mean decision accuracy across two test trials as the dependent variable.

The DSS interfaces were based on the quasi-analytical and intuitive interfaces adopted by Perry et al. (2012). They differed in the amount of information that could be accessed prior to making a decision. The quasi-analytical interface permitted participants to acquire information pertaining to eight cues. In contrast, the intuitive interface enabled access to only three cues.

To determine whether improvements in decision accuracy were associated with the interface or its impact on the learning process, participants were allocated to one of two training conditions. Participants in the training condition made a series of decisions with a DSS interface, receiving feedback following each decision. Participants in the control condition did not complete the training phase. We hypothesized the following:

Participants who engaged the intuitive interface would record significantly greater decision accuracy scores than would participants who engaged the quasi-analytical interface.

Participants in the training condition would record significantly greater decision accuracy scores than would participants in the control condition.

Participants who received training with a preconfigured interface would record significantly greater decision accuracy scores than would participants who received training with a self-configured interface.

Participants who received training with the preconfigured intuitive interface would record significantly greater decision accuracy scores than would participants who received training with the preconfigured quasi-analytical interface.

Method

Participants

The participants consisted of 128 undergraduate psychology students (33 male, 95 female) with a mean age of 21.67 (SD = 6.40) who participated in the study for course credit. To further encourage participants to invest in the training process, they were informed that they could earn $25 for exceptional performance on the task. However, all participants received $25 at the completion of the experiment, irrespective of their performance. Each of the eight experimental groups consisted of 16 participants.

Stimuli

DSS interfaces

Each interface displayed a building overview that depicted the location of three possible entry points, with the assumption that a fire was in progress (see Figures 1 and 2). To the right of the building overview, the interface displayed the list of cues that participants could access to make their decision. On clicking on a particular cue, the corresponding values for each of the three points of entry would be displayed in the interface.

Figure 1.

Example of the quasi-analytical interface. Participants could access all eight cues (listed to the right of the figure). In the self-configuration condition, participants first ranked the cues in order of importance for making a decision. These rankings determined the order that cues were presented in the menus. In the preconfiguration condition, the cues were presented in order from the most important to the least important and were not controlled by participants.

Figure 2.

Example of the intuitive interface. The cues available to participants (listed to the right of the figure) were limited to three. In the self-configuration condition, participants selected the three cues that they considered most significant for the decision-making process. In the preconfiguration condition, participants could access only the three most critical cues derived from subject matter experts.

In the self-configuration condition, the quasi-analytical and the intuitive interfaces differed according to both the number of cues that were available and the manner in which the cues could be configured. As illustrated in Figure 1, the quasi-analytical interface enabled access to all eight cues. Participants first ranked the cues in order from the most important to the least important on the basis of their perception of the relative significance of the cues in formulating a decision. These rankings determined the order in which cues were subsequently presented in the interface (see Perry et al., 2012, for a more detailed description). The intuitive interface limited to three cues the information that participants could access during the decision-making process (see Figure 2). These cues were selected by participants during the “cue set-up phase” that preceded the scenarios.

In the preconfiguration condition, the cue set-up phase was unavailable to participants. Instead, the interfaces were preconfigured to present those cues that were most frequently used by the experienced FGCs in Perry et al. (2012). In the quasi-analytical interface, the cues were presented, in order, from the cue most frequently accessed by experienced FGCs to the cue least frequently accessed. Similarly, in the intuitive interface, the three cues available were the three cues most frequently accessed by the experienced FGCs when using the intuitive interface.

Decision scenarios

Consistent with Perry et al. (2012), eight firefighting-related scenarios were developed through consultation with five subject matter experts (SMEs). Each of the SMEs had obtained the rank of inspector with a mean of 24.26 (SD = 5.78) total years of experience as a firefighter and a mean of 6.34 (SD = 5.66) years of command experience.

Each scenario required participants to determine which of three potential points of entry to a burning building was most appropriate for rescuing a victim trapped inside. The scenarios consisted of a brief paragraph describing the time of day, the type of building, and confirmation that a person was trapped inside. The appropriateness of each point of entry was determined from the ratings of the SMEs, who demonstrated moderate agreement in the ranking of decision options, Kendall’s W = .67. The rankings of the SMEs determined the decision accuracy score assigned to each point of entry, scored on a scale of 1 to 3. Participants who selected the point of entry that was rated by the SMEs as the most appropriate obtained a score of 3, whereas participants who selected the point of entry that was rated as the least appropriate by the SMEs obtained a score of 1. Thus, higher decision accuracy scores were considered an indication of more accurate decisions.

Of the eight scenarios, two were used as test scenarios. The remaining six scenarios were presented only during the training phase. Both the presentation order of the training scenarios and the presentation order of the test scenarios were counterbalanced with an incomplete Latin squares design (Bradley, 1958).

Cues

The DSS interfaces were populated with a series of eight text-based cues relating to point-of-entry decisions in the context of firefighting. The eight cues used in the interfaces were derived from Perry et al. (2012), who assessed the cue use of expert FGCs when formulating point-of-entry decisions. Reported in the order that experts used the cues in Perry et al., the cues were structural damage, the distance from the victim, the level of heat, smoke, obstructions to the entry, the distance from the seat of the fire, the fuel load, and the entry construction. Considering that experts come to learn the diagnostic value of cues for decision making on the basis of their experience, they will likely give greater attention to the most valid cues for a given situation (Schriver et al., 2008). Therefore, for the purposes of this study, an assumption was made that insights into the ecological validity of each cue could be derived from the degree to which experts use cues (Shanteau, 1992).

In the preconfiguration condition, the interfaces were constructed to emphasize the cues that were most frequently used by experts in Perry et al. (2012). For example, in the quasi-analytical interface, the most frequently accessed cue was presented at the top of the list, and the least frequently accessed cue was presented at the end of the list. In the intuitive interface, access to cues was restricted to the three most frequently accessed cues: structural damage, the distance from the victim, and the level of heat.

Decision training

Having made a decision in each of the six training scenarios, participants in the decision training condition were provided feedback regarding the correct ordering of decision options, as rated by SMEs, from the entrance that was most appropriate to the entrance that was least appropriate.

Procedure

Participants first completed a demographic questionnaire before progressing to the training phase, in which the type of training was manipulated. Participants in the training condition used either the intuitive or the quasi-analytical interface to make decisions in a series of six decision scenarios as to the most appropriate point of entry to a burning building. After making a decision in each scenario, participants were provided with feedback regarding the accuracy of their decision. Participants in the control condition were not exposed to the DSS interfaces during the training phase and instead completed an unrelated rail control task. On completion of the training phase, participants completed the test phase, which consisted of two decision scenarios in which participants used the interface corresponding to their experimental condition, either the quasi-analytical or intuitive interface.

Results

The data consisted of participants’ mean decision accuracy score across the two test trials. We analyzed the data using planned comparisons with Bonferroni adjustment to directly target the specific hypotheses. A separate contrast was conducted for each hypothesis. Descriptive statistics for the mean decision accuracy scores of each group are provided in Table 1.

Table 1:

Descriptive Statistics for the Decision Accuracy Scores During the Test Scenarios for Each Experimental Condition

	Training Condition
	Control		Training		Total
Configuration	M	SD	M	SD	M	SD
Intuitive interface
Self-configuration	2.00	.73	1.88	.72	1.94	.73
Preconfiguration	1.94	.68	2.69	.48	2.32	.58
Total	1.97	.71	2.29	.60	2.13	.66
Quasi-analytical interface
Self-configuration	1.94	.77	1.94	.77	1.94	.77
Preconfiguration	2.06	.77	2.00	.73	2.03	.75
Total	2.00	.77	1.97	.75	1.99	.76

Hypothesis 1: Intuitive Versus Quasi-Analytical Interface

It was hypothesized that participants who engaged the intuitive interface would record significantly greater decision accuracy scores than participants who engaged the quasi-analytical interface. No significant difference was evident between the decision accuracy scores for participants who engaged the intuitive interface (M = 2.13, SD = 0.66) and participants who engaged the quasi-analytical interface (M = 1.99, SD = 0.76), F(1, 120) = 1.25, p = .27, d = .21. Therefore, the use of the intuitive DSS interface alone was not sufficient to improve decision accuracy. The intuitive interface condition is represented by Groups 1, 2, 3, and 4 in Figure 3, and the quasi-analytical interface condition is represented by Groups 5, 6, 7, and 8.

Figure 3.

Mean decision accuracy scores for each experimental condition. The error bars represent the 95% confidence interval.

Hypothesis 2: Training Versus Control

It was hypothesized that participants who received training would record higher decision accuracy scores than would participants who did not receive training. No significant difference was evident between the decision accuracy scores for participants who received training (M = 2.13, SD = 0.68) and the scores for those participants who did not receive training (M = 1.98, SD = 0.74), F(1, 120) = 1.25, p = .27, d = .21. Therefore, the mere provision of training did not lead to improvements in decision accuracy. The control condition is represented by Groups 1, 3, 5, and 7 in Figure 3, and the training condition is represented by Groups 2, 4, 6, and 8.

Hypothesis 3: Training With Preconfigured Versus Self-Configured Interfaces

It was hypothesized that the provision of decision training would be associated with higher decision accuracy scores for participants in the preconfiguration condition in comparison to participants in the self-configuration condition. Consistent with this hypothesis, a significant difference was evident between the preconfiguration and self-configuration conditions. Participants who received training in the preconfiguration condition (M = 2.34, SD = 0.60) recorded significantly higher mean decision accuracy scores than did participants who received training in the self-configuration condition (M = 1.91, SD = 0.75), F(1, 120) = 6.03, p = .02, d = .63. The combined training and self-configuration conditions are represented by Groups 2 and 6 in Figure 3, and the combined training and preconfiguration conditions are represented by Groups 4 and 8.

Hypothesis 4: Training With Preconfigured Interfaces, Intuitive Versus Quasi-Analytical

It was hypothesized that for participants in the preconfiguration condition, those who received training with the intuitive interface would record higher decision accuracy scores than would participants who received training with the quasi-analytical interface. Consistent with the hypothesis, a significant difference was evident between the two groups. Participants who received training and preconfiguration with the intuitive interface (M = 2.69, SD = 0.48) recorded significantly higher decision accuracy scores than did participants who received training and preconfiguration with the quasi-analytical interface (M = 2.00, SD = 0.73), F(1, 120) = 7.45, p = .01, d = 1.12. Participants in the preconfiguration condition who received training with the intuitive interface are represented by Group 4 in Figure 3, and participants in the preconfiguration condition who received training with the quasi-analytical interface are represented by Group 8.

Discussion

The aim of the present study was to examine the impact on decision accuracy of different approaches to the delivery of reduced-processing DSS interfaces among inexperienced learners in the context of fire control. To establish the utility of reduced processing within the context of DSSs, performance was compared across two interfaces that differed in the frequency of cues available. Consistent with Hypothesis 4, participants who received training with the preconfigured intuitive interface recorded significantly greater decision accuracy scores than did participants who received training with the preconfigured quasi-analytical interface.

The enhanced decision-making performance for participants who received training with the preconfigured intuitive interface highlights one of the benefits afforded by the reduced-processing approach to decision support. By reducing the amount of information that the learner can process to the most critical cues for decision making, the intuitive interface potentially enabled inexperienced decision makers to invest a greater proportion of cognitive resources in understanding the utility of those cues for making accurate decisions (Chandler & Sweller, 1996; Paas et al., 2004; Sweller, 1988). The result is a potentially more rapid uptake of the cue-based decision-making skills that are characteristic of expert decision making (Klein, 1989, 1997; Schriver et al., 2008; Shanteau, 1992).

Although decision accuracy was enhanced when participants were provided with training and preconfiguration with the intuitive interface, neither of these variables in isolation appeared to facilitate performance. Contrary to Hypotheses 1 and 2, neither the mere use of the intuitive interface nor the provision of decision training resulted in improvements in decision accuracy. Given that participants were naive to the domain of firefighting, they would have had little prior knowledge of the ecological validity of the available cues, so they did not make more accurate decisions through the mere provision of the intuitive interface. To make accurate decisions, participants would have had to invest cognitive resources in both learning to identify the important cues and learning how to appropriately use those cues within the limited number of training trials that were available. These demands may have detracted from learners’ capacity to recognize and retain the ecological validity of cues and thereby improve their subsequent performance.

It appears that participants learned to appropriately use the cues to make accurate decisions only when their attention was directed toward the relatively limited number of cues afforded by the reduced-processing DSS. As training was effective in improving decision accuracy only when it was used in combination with the intuitive interface, it could be argued that directing learners toward critical cues facilitated cue acquisition (Chandler & Sweller, 1996; Paas et al., 2004).

Consistent with Hypothesis 3, participants who received training with a preconfigured interface recorded significantly higher decision accuracy scores than did participants who received training with a self-configured interface. Although the difference between the preconfiguration and self-configuration conditions was statistically significant, the effect appears to be driven by the enhanced performance of participants who received both training and preconfiguration with the intuitive interface. Therefore, the preconfiguration of cues was necessary for participants to learn to use the intuitive interface to make accurate decisions.

Preconfiguring the interfaces in a manner consistent with the configuration used by expert personnel eliminated the acquisition of less-relevant information during decision making. From a theoretical perspective, the ability to acquire only the most critical information likely aided the indexing of this information in memory, facilitating the processes of encoding and retrieval. Restricting access to only critical cues ensured that participants were focusing on task-relevant information when encoding the associations between the cues and the outcomes. Similarly, the information acquisition restrictions in the preconfigured interfaces reduced the potential for nonrelevant cues to interfere with the memory retrieval process when participants were attempting to recall relevant information from memory (Ericsson & Kintsch, 1995). Therefore, it appears that the interfaces that ensured that participants focused on more-relevant information during encoding and retrieval were associated with greater accuracy in subsequent decision making.

Overall, the results indicated that among inexperienced participants, relative improvements in decision accuracy can be achieved through the provision of training following reduced-processing decision support scenarios that incorporate a limited number of key cues used by expert personnel in similar situations.

Implications

This study extends research into the capacity for reduced-processing DSSs to improve the decisions made by inexperienced personnel. Previous research established that although reduced-processing interfaces enable inexperienced operators to process information in a manner consistent with experienced operators, difficulties in discriminating relevant from less-relevant information compromised decision accuracy (Perry et al., 2012). On the basis of the results of this study, it can be concluded that reduced-processing DSSs have the potential to indirectly improve the decisions made by inexperienced personnel by directing their attention toward the most relevant cues for decision making.

Although interface preconfiguration can potentially reduce the acquisition of less-relevant information among inexperienced personnel, such an approach should be applied with caution. It is extremely difficult to anticipate the full range of decisions that will confront operators in applied environments. Consequently, the information that operators will need to access cannot always be foreseen, and it would not be advisable to advocate a system that completely restricts access to information. Rather, it is envisaged that reduced-processing applications would offer information as a system advisory, either highlighting or recommending the most critical cues to consider for a particular situation. Such an approach could assist operators in identifying the most critical cues for decision making but also enable some flexibility to access additional information, should it be necessary. The development of these systems could be supported through future research to identify both the decisions with which inexperienced personnel are likely to require support and the stage during the operational process at which the application of the DSS is likely to be most advantageous.

The outcomes of this study have important implications for process control environments, such as electrical power control and rail control, where there is an increasing reliance on relatively inexperienced operators. Potentially, reduced-processing DSSs could be implemented as a tool to support inexperienced operators’ acquisition of effective and efficient decision-making skills while simultaneously safeguarding the integrity of the operational systems. Provided that the information relevant to decision making can be represented appropriately, the capacity exists to present the information to users through a reduced-processing interface that restricts access to the most relevant cues. When used as a tool to support real-time decision making, reduced-processing DSSs offer the potential to assist the operator to acquire the cognitive skills necessary for accurate and efficient decision making.

Key Points

Inexperienced personnel are more likely to process extraneous information, which can affect the accuracy and efficiency of decision making, particularly in time-constrained environments.

Decision support systems that encourage users to process only critical information can improve efficiency and accuracy during point-of-entry decisions in the context of firefighting.

Reduced-processing decision support systems offer an opportunity to both facilitate the acquisition of skilled performance and safeguard the integrity of the system.

Footnotes

Acknowledgements

The authors acknowledge the funding support provided by the Australian Research Council (DP0664862). The authors would like to thank the anonymous reviewers for their insightful comments on earlier versions of this article.

Nathan C. Perry received a PhD in psychology from the University of Western Sydney in 2011. He is a lecturer in the School of Social Sciences and Psychology at the University of Western Sydney.

Mark W. Wiggins received a PhD in psychology from the University of Otago in 2001. He is an associate professor of psychology at Macquarie University.

Merilyn Childs received a PhD in social sciences from the University of Western Sydney in 2001. She is an associate professor of higher education and deputy director of the Flexible Learning Institute at Charles Sturt University, New South Wales, Australia.

Gerard Fogarty received a PhD from the University of Sydney in 1984. He is a professorial research fellow in the Department of Psychology at the University of Southern Queensland.

References

Anderson

J. R.

(1993). Rules of the mind. Hillsdale, NJ: Lawrence Erlbaum.

Bradley

J. V.

(1958). Complete counterbalancing of immediate sequential effects in a Latin square design. Journal of the American Statistical Association, 53, 525–528.

Brunswik

(1956). Perception and the representative design of psychological experiments. Berkeley: University of California Press.

Chandler

Sweller

(1996). Cognitive load while learning to use a computer program. Applied Cognitive Psychology, 10, 151–170.

Ericsson

K. A.

Kintsch

(1995). Long-term working memory. Psychological Review, 102, 211–245.

Gonzalez

Lerch

J. F.

Lebiere

(2003). Instance-based learning in dynamic decision making. Cognitive Science, 27, 591–635.

Klein

G. A.

(1989). Recognition-primed decisions. In Rouse

W. B.

(Ed.), Advances in man-machine systems research (Vol. 5, pp. 47–92). Greenwich, CT: JAI Press.

Klein

G. A.

(1997). The recognition-primed decision (RPD) model: Looking back, looking forward. In Zsambok

C. E.

Klein

(Eds.), Naturalistic decision making (pp. 285–292). Mahwah, NJ: Lawrence Erlbaum.

Lipshitz

Omodei

McLennan

Wearing

(2007) What’s burning? The RAWFS heuristic on the fire ground. In Hoffman

(Ed.), Expertise out of context: Proceedings of the sixth international conference on naturalistic decision making (pp. 97–111). New York, NY: Lawrence Erlbaum.

10.

Morrison

B. W.

Wiggins

M. W.

Porter

(2010). User preference for a control-based reduced processing decision support interface. International Journal of Human–Computer Interaction, 26, 297–316.

11.

O’Hare

(1992). The “artful” decision maker: A framework model for aeronautical decision making. International Journal of Aviation Psychology, 2, 175–191.

12.

Paas

Renkl

Sweller

(2004). Cognitive load theory: Instruction implication of the interaction between information structures and cognitive architecture. Instructional Science, 32, 1–8.

13.

Perry

N. C.

Wiggins

M. W.

Childs

Fogarty

(2012). Can reduced processing decision support interfaces improve the decision-making of less-experienced incident commanders? Decision Support Systems, 52, 497–504.

14.

Rasmussen

(1990). Mental models and the control of action in complex environments. In Ackermann

Tauber

M. J.

(Eds.), Mental models and human-computer interaction (pp. 41–46). Amsterdam, Netherlands: North-Holland/Elsevier Science.

15.

Schriver

A. T.

Morrow

D. G.

Wickens

C. D.

Taileur

D. A.

(2008). Expertise differences in attentional strategies related to pilot decision making. Human Factors, 50, 864–878.

16.

Shanteau

(1992). How much information does an expert use? Is it relevant? Acta Psychologica, 81, 75–86.

17.

Sweller

(1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257–285.

18.

Ward

Suss

Eccles

D. W.

Williams

M. A.

Harris

K. R.

(2011). Skill-based differences in option generation in a complex task: A verbal protocol analysis. Cognitive Processing, 12, 289–300.

19.

Ward

Williams

A. M.

Bennett

S. J.

(2002). Visual search and biological motion perception in tennis. Research Quarterly for Exercise and Sport, 73, 107–112.

20.

Wiggins

Stevens

Howard

Henley

O’Hare

(2002). Expert, intermediate and novice performance during simulated pre-flight decision-making. Australian Journal of Psychology, 54, 162–167.

21.

Wiggins

M. W.

(2006). Cue-based processing and human performance. In Karwowski

(Ed.), International encyclopedia of ergonomics and human factors (2nd ed., pp. 641–645). London, UK: Taylor and Francis.

22.

Wiggins

M. W.

Bollwerk

(2006). Heuristic-based information acquisition and decision making among pilots. Human Factors, 48, 734–746.

23.

Wiggins

M. W.

O’Hare

(2003). Weatherwise: Evaluation of a cue-based training approach for the recognition of deteriorating weather conditions during flight. Human Factors, 45, 337–345.