Information Search and Decision Making in Computer-Based Wildfire Simulations

Abstract

Decision making in complex environments has been investigated in many domains, including medicine, aviation, business, and police operations. However, how incident commanders (ICs) make protective-action recommendations (PARs) to populations exposed to wildfire risks is underinvestigated. In this study we examined the effect of expertise on IC non-time-limited information search and decision making and how ICs update information during the evolution of complex, computer-simulated wildfire scenarios. The results indicate that higher expertise reduces the overall amount of information being searched for, without affecting the quality of PAR decisions. In addition, a statistical trend suggests that information updating during the progression of the scenario involves disproportionate less-static information versus dynamic information. Finally, ICs demonstrated a strong preference for evacuation recommendations over alternative recommendations, even when an evacuation may result in less optimal outcomes.

Keywords

wildfire information search decision making simulation information board

Introduction

Decision making in complex environments involves a series of real-time decisions in a highly dynamic environment (Brehmer, 1990; Gonzalez, Vanyukov, & Martin, 2005) where the time dimension is of critical importance (Brehmer, 1992) and decision makers respond to changes in their environment by adjusting their decision timing (Kerstholt & Raaijmakers, 1997). This type of decision making is sometimes referred to as dynamic decision making (DDM; Brehmer, 1992; Busemeyer, 2002), naturalistic decision-making task (Lipshitz et al., 2001), or complex dynamic control task (Osman, 2010).

The dynamic nature of the decision-making environment is a result of closely interconnected elements that change dynamically over time and that can leave the decision maker with outdated or incomplete information. It has been argued that system thinking is essential for learning how to make decisions in dynamic systems (Doerner, 1997; Sterman, 1994; Sterman & Booth-Sweeney, 2002). Decision making in dynamic systems has been studied outside the laboratory in aviation, weather forecasting, military operations, and other domains (see Busemeyer, 2002).

One important contributor to the cognitive complexity in DDM is the context or environment in which decision making occurs (Knauff & Wolf, 2010; Osman, 2010). Among the challenges for decision makers are the linear, and often nonlinear, relationships between variables required to predict future states; the fact that some variables may change rapidly whereas others do not change at all; and the uncertainty inherent to the often limited knowledge about the system dynamics. Durso and Drews (2010) introduced the distinction between control of natural (e.g., wildfire) versus technical systems (e.g., airplane). The argument is that the system attributes differ between natural and technical systems and that these differences have important implications for system complexity, decision making, and control.

Wildland Fire Management

The present study focuses on expert decision makers whose responsibility is the management of the complex, natural system of a wildfire that threatens inhabitants and structures of a community. In the United States and throughout the world, the threat of wildfire to people and structures is rapidly increasing as population growth in fire-prone areas, known as the wildland urban interface (WUI), expands (Cortner, 1991; Theobald, 2007). Events resulting in loss of life, such as the 2012 Colorado wildfires (e.g., City of Colorado Springs, 2012), the 2010 wildfires in Russia, the 2009 Victoria Bushfires in Australia, and the 2007 fires in southern Greece, highlight the dangers of living in the WUI as well as the need for improved planning and decision making related to protecting people during wildfires.

Given the increased necessity of managing fires in the WUI, it is important to better understand the factors that lead to effective protective-action decision making of incident commanders (ICs). Unfortunately, at this point, little empirical work exists in this context (although see Brehmer & Allard, 1991). Cova, Drews, Siebeneck, and Musters (2009) describe the options and complex trade-offs ICs face when making protective-action recommendations (PARs). Increasing the understanding of factors that distinguish poor from good PAR decision making has several benefits: On the applied side, it may help minimize the loss of life and property (Cruz et al., 2012) and support the development of better training programs for junior ICs by identifying important factors that require consideration in decision making. On the basic research side, this work provides additional insights into human cognition when facing management, control, or decision tasks in complex, dynamic environments.

PAR Decision Making

Lindell and Perry (1992) define three factors that contribute to the success of community response in emergencies: (a) the nature of the hazard and the characteristics of its community impact, (b) the existing resource base and organization of the community, and (c) the community preparedness to select and implement responses to emergency conditions. The operation of these three factors is embodied in two event chains: a hazard chain (i.e., hazard generation, hazard transmission, community impact, and hazard exposure) and a response chain (i.e., emergency assessment, protective response, hazard mitigation, emergency management, and exposure reduction) (Figure 1).

Figure 1.

The chain of events for hazards and community response (Source: Lindell & Perry, 1992).

When investigating emergency responses, which are part of the response chain, the key concepts requiring consideration are time, anticipation, uncertainty, complexity, criticality, and assumptions (Lindell & Perry, 1992, p. 55). In their model, time refers to the available time before a hazard’s impact, which is one of the least researched areas in evacuation in particular and protective actions in general and which tends to focus on the time to carry out an action. Anticipation refers to maximizing efficient resource use. Uncertainty refers to the uncertainty of information about the hazard assessment (i.e., what is happening at any one point in time?). Complexity arises from managing limited resources and uncertainties under time constraints. Criticality refers to deciding which tasks to perform when there is insufficient time. Assumptions refers to accurate knowledge of the behavior of the hazard, public, and emergency organizations, which aids decision making. The variables and relationships constitute a model of the expert’s protective action decision-making process at a macrolevel.

Information that is being used as a basis for decision making is a critical component of PAR decision making. Consequently, certain information needs to be available and constantly updated allowing the IC to anticipate effectively, to manage the complexity, and to identify the criticality of certain actions when involved in DDM during the evolution of a community threatening wildfire.

One challenge when dealing with WUI fires is that ICs often face choices on how to deal with threats to life and property. Among these choices are (a) to evacuate the inhabitants of threatened structures to safer places, (b) to shelter inhabitants in safer places near the fire, (c) to allow residents to stay and defend their homes (Cova et al., 2009; Handmer & Tibbits, 2005). In addition, in all cases, the IC may, prior to implementing any of the choices, take a wait-and-see approach. When considering these choices, complex trade-offs between different goals and management strategies are involved (Wilson, Winter, Maguire, & Ascher, 2011) that require information about the wildfire situation and the projection of its development into the future.

Given the complex nature of a wildfire, it comes as no surprise that during the management of a wildfire, an IC actively seeks some information and passively receives other, unsolicited information. For example, an IC may receive a report of current wind speed (e.g., 5 mph) or inquire about the distance between the wildfire and the closest community (e.g., 7 miles). The dynamic nature of a wildfire implies that dynamic information might become outdated quickly (e.g., wind speed), whereas static information will not change over time (e.g., legal context). Although initially both types of information are important to establish the basis for the wildfire situation assessment, later, as the wildfire progresses, only changing information needs to be monitored.

Information Search and Decision Making

Previous work on information search shows differences in search strategies as a result of age, with older adults requesting less information than younger adults in the context of medical decision tasks (Ende, Kazis, Ash, & Moskowitz, 1989; Leventhal, Leventhal, Schaefer, & Easterling, 1993; Meyer, Russo, & Talbot, 1995), everyday problem-solving tasks, and decisions about car purchasing and apartment renting (Johnson, 1990, 1993).

Other studies on information search behavior focused on search behavior in web searches. The results of this work are contradictory, with experts compared to novices performing in some studies more queries (Aula & Kaeki, 2003; Hsieh-Yee, 1993; Sutcliffe, Ennis, & Watkinson, 2000; White & Morris, 2007) or fewer queries (Hölscher & Strube, 2000). Similarly, some work indicates that experts open more items during in a search (Hölscher & Strube, 2000), whereas others find that experts open fewer items (White & Morris, 2007).

However, as Dhami and Harries (2010) point out, although there are substantial bodies of research on information search behavior and judgment and decision making, few researchers have studied both combined (Dhami & Harries, 2010). In addition, the work on information search in judgment and decision making often uses simple information and nonexperts (e.g., Payne, Bettman, & Johnson, 1993).

Given the relatively little research, it comes at no surprise that only few theories describe how expert decision makers search for information in complex and dynamic environments. Klein’s (1997, 2008) recognition primed decision making (RPD), developed by studying firefighters, hypothesizes that a decision maker initially performs a situation assessment that involves plausible goals, critical cues, expectancies, and one candidate course of action. When facing familiar situations, the decision maker recognizes the situation and implements a solution quickly, whereas when facing less familiar situations, the decision maker performs an evaluation of the action (involving mental simulation) to identify potential problems. Finally, unfamiliar situations may require the generation of an alternative action, once the previously evaluated action has been identified as flawed.

Klein (1997, 2008) proposes that information search is triggered by the violation of expectations after an initial situation assessment. When facing an evolving situation, the decision maker updates information until expectations are matched, which allows the identification of likely successful action. One plausible interpretation of RPD is that the decision maker, during the initial assessment, collects information broadly, whereas subsequently, information search becomes more selective. However, RPD does not make specific predictions of how domain expertise affects information search.

A second theory is the instance-based learning theory (IBLT; Gonzalez, Lerch & Lebiere, 2003). IBLT proposed as a mechanism for decision making the learning and cue-based retrieval of instances that are based on situational elements, potential actions, and the utility associated with the outcome of each action. According to IBLT, effective decision makers focus more on task-relevant information while simultaneously ignoring task-irrelevant information. This assumption encapsulates the information reduction hypothesis developed by Haider and Frensch (1996, 1999a, 1999b). As Haider and Frensch (1999a) state, “improvements in task performance . . . reflect an increase in knowledge which information has to and which information does not have to be processed” (p. 172).

Support for this hypothesis comes from research using domain novices and experts (Dillon & Song, 1997; Hölscher & Strube, 2000; Lazonder, Harm, & Woperies, 2000; Wilson et al., 2011). This and other work suggest that domain expertise enhances search performance, resulting in experts being faster and more targeted (Patel, Drury, & Shalin, 1998). However, relatively little work has focused on expertise differences in information-updating behavior in general and with regard to the distinction between dynamic and static information in particular.

Other empirical work on contextual factors in the context of information search suggests that time pressure increases the amount of information searched for and negatively affects the speed of decision making (Kerstholt, 1996). In addition, when one is facing high information-access costs, time pressure leads to delayed decisions, whereas low access costs result in a quicker decision and the use of a judgment-oriented strategy (Kerstholt, 1996).

In the current study we pursued the following objectives while focusing on non-time-limited protective-action decision making: (a) to identify if expertise differences affect information search and PAR decision making and (b) to examine information-updating behavior of ICs during the evolution of a wildfire scenario.

To pursue these goals, we used a simulated and controlled environment, since in naturalistic settings the study of what information ICs actively seek out and what information they passively receive is methodologically challenging and does not allow for control of information access.

Method

Participants

Twenty-nine ICs with different experience levels in WUI fire management participated in this institutional review board–approved study. The types of ICs range on a scale of 1 to 5, with Level 5 ICs managing small wildfires with low levels of complexity and Level 1 ICs managing large and complex incidents that require national resources and multiple agencies (National Parks Service, 2014). For this study, three levels of experience were identified and aggregated into three expert groups: Group 1 (n = 10) consisted of ICs at Levels 1 and 2 (IC1-2), Group 2 (n = 9) included ICs at Level 3 with more than 10 years of experience of work at this level (IC3_high), and Group 3 (n = 10) included ICs at Levels 5 and 4 and those ICs with less than 10 years of experience at Level 3 (IC3_low). All participants reported a minimum of 3 years of WUI fire management experience. The participants’ average age was 49 years (Mdn = 50 years, range = 28–65 years) with average age for the three groups of IC1-2, IC3_high, and IC3_low being 55.9 (Mdn = 55) years, 49 (Mdn = 50) years, and 43 (Mdn = 42) years, respectively. Although not statistically different, the older age of the IC1-2 group reflects the longer time required to work at this level.

Participants answered a question regarding their comfort level of working at their current incident command type. Confidence on a 1-to-10 point Likert-type scale (10 = highest level of comfort) was 7.8 (Mdn = 8) for the IC1-2 group, 7.5 (Mdn = 7) for the IC3_high group, and an average of 6.8 (Mdn = 7.5) for the IC3_low group. In addition to comfort level, we also asked how adequate the ICs felt working in their current position (Likert-type scale ranging from 1 = lowest to 10 = highest). Consistent with the idea that the classification into the three groups expresses different levels of expertise, there was a significant difference between the groups overall, F(2, 25) = 5.8, p = .01. The highest level of subjective adequacy was found in the IC1-2 group (M = 7.7; Mdn = 8), next was the IC3_high group (M = 6.7; Mdn = 7), and lowest ratings were present in the IC3_low group (M = 5.4; Mdn = 6). An alternative measure to assess skills is to identify perceived peer opinion of participants. Here we asked participants how their peers would assess their ability to manage a fire at their current level. Ratings on a Likert-type scale ranged from 1 = low to 10 = high. For the three groups, there was a trend in the differences between ratings, F(2, 25) = 3.127, p = .06. The highest ratings were found in the IC1-2 group, with an average of 7.5 (Mdn = 7.5). Next was the IC3_high group, with an average of 6.8 (Mdn = 7), followed by the IC3_low group (M = 5.4; Mdn = 5). Finally, participation in formal exercises over the past 5 years differed between the three groups as well. The combined IC1-2 group participated in an average of 7.1 exercises (Mdn = 7.8), the IC3_high group reported participation in an average of 7.7 exercises (Mdn = 5), and the IC3_low group expressed the lowest exercise participation (M = 3.6 times; Mdn = 3).

Materials

Participants interacted with five microworld-based, computer-simulated wildfire scenarios by requesting information and implementing PARs over the evolution of the scenarios. The scenarios presented the background of each community and the initial situation concerning the wildfire. Once started, participants were able to select repeatedly information about 31 variables that were previously identified as relevant to WUI fire management (Drews, Musters, Siebeneck, & Cova, 2014). Similar to other microworld experiments, this simulation consisted of a complex simulation of a wildfire allowing participants to interact with the simulation to receive information under conditions that are similar to problems they would encounter in the real world (Drews & Bakdash, 2013; Gonzales et al., 2005; Turkle, 1984).

Scenarios

The test scenarios were developed in collaboration with two expert ICs with more than 20 years of experience in the field. Out of the five scenarios, three scenarios were developed such that an optimal PAR decision should result in an evacuation (evacuation scenario), a shelter-in-place recommendation (shelter-in-place scenario), and the omission of an action (no-action scenario). The two additional scenarios were designed to have no optimal PAR decision, creating a more ambiguous and uncertain situation.

Each scenario included five phases for assessment and decision making. After each phase, the scenario progressed by one step, or 1 hr of scenario time. Thus, each scenario progressed over a total of 5 hr of scenario time, if not terminated earlier by the participant making a PAR decision. Participants were encouraged to make a decision as soon as possible, based on their assessment of the scenario, and were instructed to explore scenario-specific information as efficiently and carefully as possible. No time or information selection constraints were imposed on participants since they may lead to inferior performance because of the potential challenges associated with dealing with relatively new situations and the requirement to adapt to changing conditions (Gonzalez, 2004).

The presentation sequence of the five scenarios was counterbalanced across participants to control for potential sequence effects. In addition, the location of information cells on the information board was randomized for each participant to control for potential presentation effects.

Information Presentation

To assess the scenario, participants explored an information board (Payne et al., 1993) that provided scenario-specific information (Figure 2). The information board consisted of 31 information cells that were labeled with variable names (e.g., wind speed) to allow participants the selection of specific information. Once participants selected a variable name on the board, a pop-up window revealed the specific information associated with this variable at this time step (e.g., 5–7 mph). Depending on the variable selected, the variable-specific information was presented in either numerical or alphanumerical form.

Figure 2.

PARSE: Protective Action Recommendation Scenario Environment.

Once participants felt confident that they had explored all information they deemed necessary, they terminated the information search and were provided with the option of selecting a PAR decision. The PAR decision options available were “evacuation,” “shelter,” and “do nothing/wait” (Figure 2). A stay-and-defend option was not included since this PAR is not commonly used in the United States as opposed to other countries (e.g., Australia). In order to ensure the participants interpreted the protective-action options correctly, definitions of the three options were provided as part of the instructions. In this study, evacuation was defined as a recommendation that the threatened population leave the area and move to a safer location, shelter was the recommendation that the threatened population shelter in place in a designated shelter that has been deemed suitable, and do nothing/wait was defined as allowing the participant to proceed to the next time step of the scenario and to continue to monitor the situation.

Study Design

The study used the between-subjects factor expertise (three levels) to identify how expertise affects information search and decision making in WUI fire scenarios. More specifically, the first question this research addresses is “Does expertise affect information search and PAR decision making?” and the second question is “How do ICs update information during the evolution of a wildfire scenario?”

Overall, we evaluated search strategies and decision quality in a microworld wildfire simulation to assess which variables are used by the decision maker to reach an assessment of a wildfire and what PAR is to be issued, if any.

Results

Overall Amount of Information Searched

To identify the number of unique information items that participants explored, we performed an ANOVA with the between-subjects factor of expertise as the independent variable and the number of unique information items selected by participants as the dependent variable. The results indicate that there was a significant difference between the three expert groups based on the overall number of information items searched, F(2, 29) = 3.9, p = .03, with high experts (IC1-2) selecting an average of 9.4 (SD = 1.5), medium experts (IC3_high) selecting an average of 9.9 (SD = 1.6) and low experts (IC3_low) selecting an average of 14.8 (SD = 1.5) unique information items (Figure 3). Post hoc least significant difference tests revealed that the differences between the high expert group (IC1-2) and the low (IC3_low) expert group (p = .017) and difference between the medium (IC3_high) and the low (IC3_low) expert groups (p = .035) were significant. Thus, low experts (IC3_low) selected more information items across all scenarios than the other two groups. Because this analysis does not allow identification of which information items the groups investigated, the next analysis will focus on this issue.

Figure 3.

Information items searched.

Specific Information Items Searched

Authors of previous work identified subjective importance ratings of ICs for information items in the context of PARs (Drews et al., 2014). In the present analysis we aim at replicating and extending our knowledge of information selection during the exploration of information items when dealing with a simulated wildfire scenario. For this purpose, we analyzed the frequency with which the three groups selected specific information items across all scenarios using a Kruskall-Wallis test. The analyses revealed that for five information items, there were significant differences between the expert groups. Significant differences were identified for fire intensity, χ² = 7.1, p = .023; fire spread rate, χ² = 7.1, p = .029; number of structures, χ² = 11.99, p = .002; time to warn, χ² = 6.59, p = .037; and percentage of the wildfire contained, χ² = 6.44, p = .04. In all cases, the low (IC3_low) expert group selected these information items more frequently than (roughly twice as much as) the high (IC1-2) expert group (see Table 1). Thus, there are group-based differences in the selection of specific information items, with the low (IC3_low) expert group searching specific information items more frequently over the evolution of the test scenarios.

Table 1:

Frequency of Access of Specific Information Items Across All Scenarios Between Groups for Selected Variables

Information Item	Group
Information Item	IC1-2	IC3_high	IC3_low
Fire intensity	10.8	13.2	20.8
Fire spread	10.9	13.4	20.6
Number of structures	11.4	10.8	22.4
Percent contained	13.3	11.1	20.3
Time to warn	11.8	12.5	20.5

Note. IC1-2 = high expert; IC3_high = medium expert; IC3_low = low expert.

Number of Steps Until PAR Decision

To identify if there was a difference between the expert groups in terms of the number of steps until reaching a PAR decision, we performed a two-factorial ANOVA with expertise (three levels) as a between-subjects variable and scenario (five scenarios) as a within-subjects variable. As a dependent variable, the aggregated number of steps across all scenarios entered the analysis. Overall, there was no difference in the number of steps each group took to make a recommendation, F(2, 18) < 1.0. The average number of steps for the three groups was 8.1 (SD = 1.2) for the high (IC1-2) expert group, 7.55 (SD = 1.24) steps for the medium (IC3_high) expert group, and 7.7 (SD = 1.2) for the low (IC3_low) expert group. However, there was a significant effect of scenario, F(4, 16) = 4.6, p < .01, but no significant interaction between scenario and expertise. For the specific scenarios, the average number of time steps until decision was 1.5 (SE = 0.22) steps for the evacuation scenario, 2.8 (SE = 0.41) steps for the no-action scenario, 1.3 (SE = 0.17) steps for the shelter scenario, and 1.6 (SE = 0.17) and 1.2 (SE = 0.17) steps for the two ambiguous scenarios. Thus, the data indicate that the no-action scenario required the largest number of time steps to reach a PAR.

PAR Decision Quality

In the case of the normative scenarios (evacuation scenario, shelter scenario, no-action scenario), the scenario design implied one optimal option (i.e., no negative consequences as a result of the decision). These PAR options were considered optimal for each specific scenario, and all other options were considered nonoptimal. To analyze participants’ decision making, three analyses with a focus on optimal versus nonoptimal decision were performed. The results of these analyses suggest that there was no difference based on expertise in decision quality. The numbers and percentages of decisions for each scenario and group are shown in Table 2.

Table 2:

Numbers and Percentages of Optimal and Nonoptimal PAR Decisions by Group and Scenario

	Scenario
	Evacuation		Shelter		No Action		Combined
Group	Optimal	Nonoptimal	Optimal	Nonoptimal	Optimal	Nonoptimal	Optimal	Nonoptimal
IC1-2 (n = 10)	7 (77%)	2 (22%)	4 (44%)	5 (55%)	4 (40%)	6 (60%)	15 (54%)	13 (46%)
IC3_high (n = 9)	5 (63%)	3 (37%)	4 (50%)	4 (50%)	2 (25%)	6 (75%)	11 (46%)	13 (54%)
IC3_low (n = 10)	7 (77%)	2 (22%)	3 (30%)	7 (70%)	5 (63%)	3 (37%)	15 (58%)	11 (42%)
Total (%)	21 (73%)	7 (27%)	11 (36%)	16 (64%)	11 (41%)	15 (59%)	41 (53%)	37 (47%)

Note. Values not adding up to 100% are a result of participants not issuing a PAR. PAR = protective-action recommendation; IC1-2 = high expert; IC3_high = medium expert; IC3_low = low expert.

To explore if there were differences in the number of optimal decisions between scenarios, a related-samples Wilcoxon signed rank test was performed, which indicated a statistical difference between the evacuation and the shelter scenario (p = .035) as well as a significant difference between the evacuation and the no-action scenario (p = .033). Overall, these results illustrate the existence of a preference toward evacuation PARs, even when facing scenarios where non–evacuation PARs lead to better outcomes. For example, in the shelter scenario, an evacuation would have sent the evacuees into the fire due to a blocked egress route. Overall, the results indicate a potential expertise-independent limitation in exploring the complete action space available to the decision maker.

Information Updating: Dynamic and Static Information

Dynamic (i.e., rapidly changing information) versus static (i.e., not changing information) information was analyzed to identify what information participants selected over the first two steps of an evolving scenario. The analysis was limited to the first two scenario time steps because the number of observations dropped with every additional time step, precluding additional analyses.

A three-factorial ANOVA with the between-subjects variable expertise (three levels) and the within-subject variables information type (dynamic vs. static) and time step (Step 1 vs. Step 2) was used to analyze information search (i.e., information items selected). The ANOVA revealed significant main effects of time step, F(1, 11) = 20.2, p = .001; information type, F(2, 11) = 155.7, p = .001; and expertise, F(2, 11) = 4.9, p = .031; and a statistical trend of the interaction between time step and information type, F(1, 11) = 3.4, p = .09. None of the other interactions was significant.

A post hoc comparison qualified the main effect of expertise, indicating that the number of information items selected by the high (IC1-2) and low (IC3_low) expert groups (p = .03), and the medium (IC3_high) and low (IC3_low) expert groups (p = .016), differed statistically, with the low (IC3_low) expert group selecting the largest number of information items (M = 10.3, SE = 0.9), followed by the high (IC1-2) expert (M = 6.9, SE = 0.9) and medium (IC3_high) expert groups (M = 6.1, SE = 1.1). The analysis of time step indicated that participants selected less information at Time Step 2 (M = 5.1, SE = 0.5) compared to Time Step 1 (M = 10.4, SE = 1.1). In addition, investigating the impact of information type shows that participants selected more dynamic information items than static information items during Time Step 1 (M_dyn = 14.3, SE_dyn = 1.1; M_stat = 6.5, SE_stat = 1.1) and Time Step 2 (M_dyn = 7.8, SE_dyn = 0.95; M_stat = 2.5, SE_stat = 0.2). Finally, the statistical trend of the interaction between time step and information type suggests that less static information is searched with the progression of the scenario.

Discussion

In the current study we pursued the following goals: to identify if IC expertise affects information search and PAR decision making and to examine information-updating behavior during the evolution of a wildfire scenario.

Information Search

The results of this work indicate that low-expertise (IC3_low) ICs investigate more variables when dealing with a simulated wildfire scenario than do higher-level experts. Low-expertise ICs appear to require more information to reach a point that allows them to either advance the scenario time (i.e., move the scenario forward by one time step) or end the scenario by making a PAR decision. This finding of increased information need is consistent with previous work by Patel et al. (1998) and supports the IBLT (Gonzalez et al., 2003). Specifically, the mechanism for information search as stated by IBLT, that is, the information reduction hypothesis (Haider & Frensch, 1999a, 1999b) appears to drive the identified differences between expert groups. Although RPD (Klein, 1997) does not account for search strategies of nonexperts, the current results concerning expert search behavior could be included in future revisions of RPD to describe search behavior.

It is important to evaluate the expertise differences in information search in the context of the absence of qualitative differences of PAR decisions, since PAR decision making performance of low (IC3_low) experts was similar to the other groups, although this group explored almost 50% more information.

A more detailed analysis identified that a specific subset of information items contributed to the overall difference in information search between expert groups. Among these variables were fire intensity, fire spread rate, number of structures, percentage of fire containment, and time to warn the community. The fact that low-expertise (IC3_low) ICs examined these variables twice as frequently suggests that the expectations about dynamic changes in the scenario may have differed between groups. This finding could be an indication that this group is still developing an implicit theory about the required frequency of information updating, whereas the other groups already acquired this knowledge.

PAR Decision Making

The number of scenario time steps to reach a PAR decision did not differ between expert groups: Participants reached a decision at approximately the same point during the evolution of the scenarios. However, in the no-action scenario, participants took more steps until a decision was reached, which is reflective of a wait-and-see attitude in this scenario. In other words, ICs seemed to be more prepared to issue a PAR in this study, whereas identifying the absence of a threat to a community required longer and more careful monitoring of the scenario. Recent work on meta(re)cognition in decision making (see Cohen, Freeman, & Wolf, 1996; Osman, 2010) suggests that unfamiliar situations may require additional time until a decision is made. It is possible that the no-action scenario is representative of a class of scenarios ICs encounter less frequently.

In addition, the quality of PAR decision making did not differ between expert groups. However, the results indicate that especially in the nonevacuation scenarios, there is need for improvement (only 42% of the PARs were optimal). The absence of a difference in the quality of PAR decisions between groups allows for at least two explanations. First, it is possible that for the low-expertise (IC3_low) ICs, the additional information resulted in an improved comprehension of the situation compared to the higher-expertise ICs, leading to comparable performance in PAR decisions overall. Second, when high-expertise ICs did not request additional information, their decision quality might have been relatively impaired, leading to lower PAR decision quality. Additional work is required to examine these explanations.

An unexpected result of this study was that ICs favored the evacuation option in their PAR decisions. The extent of this preference may be explained by the fact that current policies in the United States favor evacuations. However, such strong preference has the potential to lead to cognitive tunnel vision (Sheridan, 1981), leading to suboptimal PAR decisions. The presence of such tunnel vision may have been present in one wildfire scenario, whereby the decision to evacuate led evacuees directly into the wildfire with a blocked egress road. It is possible that current policies and training practices of ICs create a preference toward evacuation that, in some cases, could lead to suboptimal outcomes. To address this issue, IC training should emphasize the consideration of all PAR options and not only a subset.

Information Updating

Overall, all participants explored dynamic information more than static information items. Clearly, when dealing with a dynamic environment, like wildfires, this strategy is effective. However, a statistical trend of the interaction between type of information and scenario time (time step) was found indicating that participants may selectively reduce the amount of static variables viewed over time relative to the dynamic variables. This trend indicates that participants are somewhat sensitive to the two types of information and adapt over time to take this difference into account in their search.

Taken together, the results of this work have implications for at least three areas: First, for training of ICs, the results suggest the value of (a) teaching the importance of specific information to inform selective information search and (b) improving the understanding of the nature of dynamic as opposed to static information. Second, for human factors in information system design, the results suggest that information presentation in these systems needs to flexibly adapt over time as a function of users’ needs for dynamic information. Finally, for human factors theory development, the results suggest important expertise-related differences in information search behavior that should be addressed in theories of expertise and expertise acquisition.

Limitations

There are several limitations of this study. One limitation is that participants in this study were recruited from a specific region in the United States (the western states), and their behavior may be a result of regional practices due to geographical and other factors. Thus, participants from other geographical regions or from different countries may display different information search behavior and preferences.

Two potential limitations are of a methodological nature: In the present study, no information-access costs were imposed on participants, allowing them to select any information deemed necessary. This method is in contrast to naturalistic contexts where information may come with information-access costs, which may affect information search behavior (see Drews & Doig, 2014; Kerstholt, 1996). Thus, additional work in which information-access costs are determined and information search is examined in a naturalistic setting is needed to complement the results of the present study.

Similarly, it is possible that mode of the information presentation in this study may have evoked problem representations that differ from representations evoked in the field. With the current work being a single, first step toward improving the understanding of information selection and information search of ICs in a microworld, authors of future studies will need to complement the present findings by using high-fidelity simulation or field studies.

Footnotes

Author(s) Note

The author(s) of this article are U.S. government employees and created the article within the scope of their employment. As a work of the U.S. federal government, the content of the article is in the public domain.

Frank A. Drews is a professor at the University of Utah and the director of the VA Center for Human Factors in Patient Safety at the VA MC Salt Lake City. He received his PhD in psychology at the Technical University Berlin, Germany, in 1999.

Laura Siebeneck is an assistant professor in the Department of Public Administration and the Emergency Administration and Planning program at the University of North Texas. Her research interests include hazards, disaster evacuation and return-entry, emergency management, and resilience. She teaches courses in hazard mitigation and preparedness, research methodology, and emergency management.

Thomas J. Cova is a professor of geography and director of the Center for Natural and Technological Hazards at the University of Utah in Salt Lake City. His research and teaching interests are environmental hazards, emergency management, transportation, and geographic information systems (GIS) with a particular focus on wildfire planning and analysis. He has published on a variety of topics in many leading hazards, transportation, and geographic information science journals and is most known for work on evacuation vulnerability and routing in fire-prone communities. He teaches courses on environmental hazards, emergency management, and GIS.

References

Aula

Käki

(2003). Understanding expert search strategies for designing user-friendly search interfaces. In Isaas

P. T.

(Ed.), Proceedings of the IADIS International Conference WWW/Internet 2003 (ICWI2003) (pp. 759–762). Algarve, Portugal: IADIS.

Brehmer

(1990). Strategies in real-time, dynamic decision making. In Hogarth

R. M.

(Ed.), Insights in decision making (pp. 262–279). Chicago, IL: University of Chicago Press.

Brehmer

(1992). Dynamic decision making: Human control of complex systems. Acta Psychologica, 81, 211–241.

Brehmer

Allard

(1991). Dynamic decision making: The effects of task complexity and feedback delay. In Rasmussen

Brehmer

Leplat

(Eds.), Distributed decision making: Cognitive models for cooperative work. New technologies and work (pp. 319–334). Oxford, UK: Wiley.

Busemeyer

J. R.

(2002). Dynamic decision making. In Smelser

N. J.

Bates

P. B.

(Eds.), International encyclopedia of the social and behavioral sciences: Methodology, mathematics and computer science (pp. 3903–3908). Oxford, UK: Elsevier.

City of Colorado Springs. (2012). Waldo Canyon Fire. Retrieved from http://www.springsgov.com/units/communications/Colo%20Springs%20Waldo%20AAR%20Final_10_23_12.pdf. Retrieved: 2/12/2014

Cohen

M. S.

Freeman

J. T.

Wolf

(1996). Metarecognition in time-stressed decision making: Recognizing, critiquing, and correcting. Human Factors, 38, 206–219.

Cortner

H. J.

(1991). Interface policy offers opportunities and challenges: USDA Forest Service strategies and constraints. Journal of Forestry, 89(6), 31–34.

Cova

T. J.

Drews

F. A.

Siebeneck

L. K.

Musters

(2009) Protective actions in wildfires: Evacuate or shelter-in-place? Natural Hazards Review, 10, 151–162.

10.

Cruz

M. G.

Sullivan

A. L.

Gould

J. S.

Sims

N. C.

Bannister

A. J.

Hollis

J. J.

Hurley

R. J.

(2012). Anatomy of a catastrophic wildfire: The Black Saturday Kilmore East Fire in Victoria, Australia. Forest Ecology and Management, 284, 269–285.

11.

Dhami

M. K.

Harries

(2010). Information search in heuristic decision making. Applied Cognitive Psychology, 24, 571–586.

12.

Dillon

Song

(1997). An empirical comparison of the usability for novice and expert searchers of a textual and a graphic interface to an art-resource database. Journal of Digital Information, 1(1), 33–42.

13.

Doerner

(1997). The logic of failure: Recognizing and avoiding error in complex situations. New York, NY: Basic Books.

14.

Drews

F. A.

Bakdash

J. Z.

(2013). Simulation training in health care. Reviews of Human Factors and Ergonomics, 8, 23–85.

15.

Drews

F. A.

Doig

(2014). Evaluation of a Configural vital signs display for intensive care unit nurses. Human Factors, 56, 569–580.

16.

Drews

F. A.

Musters

Siebeneck

L. K.

Cova

T. J.

(2014). Environmental factors that influence wildfire protective-action recommendations. International Journal of Emergency Management, 10, 153–168.

17.

Durso

F. T.

Drews

F. A.

(2010). Health care, aviation, and ecosystems: A socio-natural systems perspective. Current Directions in Psychological Science, 19(2), 71–75.

18.

Ende

Kazis

Ash

Moskowitz

M. A.

(1989). Measuring patients’ desire for autonomy: Decision-making and information-seeking preferences among medical patients. Journal of General Internal Medicine, 4, 23–30.

19.

Gonzalez

(2004). Learning to make decisions in dynamic environments: Effects of time constraints and cognitive abilities. Human Factors, 46, 449–460.

20.

Gonzalez

Lerch

F. J.

Labiere

(2003). Instance-based learning in dynamic decision making. Cognitive Science, 27, 591–635.

21.

Gonzalez

Vanyukov

Martin

M. K.

(2005). The use of microworlds to study dynamic decision making. Computers in Human Behavior, 21, 273–286.

22.

Haider

Frensch

P. A.

(1996). The role of information reduction in skill acquisition. Cognitive Psychology, 30, 304–337.

23.

Haider

Frensch

P. A.

(1999a). Eye movement during skill acquisition: More evidence for the information-reduction hypothesis. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 172–190.

24.

Haider

Frensch

P. A.

(1999b). Information reduction during skill acquisition: The influence of task instruction. Journal of Experimental Psychology: Applied, 5, 129–151.

25.

Handmer

Tibbits

(2005). Is staying at home the safest option during bushfires? Historical evidence for an Australian approach. Global Environmental Change Part B: Environmental Hazards, 6(2), 81–91.

26.

Hölscher

Strube

(2000). Web search behavior of Internet experts and newbies. Computer Networks, 33, 337–346.

27.

Hsieh-Yee

(1993). Effects of search experience and subject knowledge on the search tactics of novice and experienced searchers. Journal of the American Society for Information Science, 44, 161–174.

28.

Johnson

M. M. S.

(1990). Age differences in decision making: A process methodology for examining strategic information processing. Journal of Gerontology: Psychological Sciences, 45(2), P75–P78.

29.

Johnson

M. M. S.

(1993). Thinking about strategies during, before, and after making a decision. Psychology and Aging, 8, 231–241.

30.

Kerstholt

(1996). The effect of information costs on strategy selection in dynamic tasks. Acta Psychologica, 94, 273–290.

31.

Kerstholt

J. H.

Raaijmakers

J. G.

(1997). Decision making in dynamic task environments. In Ranyard

Crozier

W. R.

Svenson

(Eds.), Decision making: Cognitive models and explanations (pp. 205–217). London, UK: Routledge.

32.

Klein

(1997). The recognition-primed decision (RPD) model: Looking back, looking forward. In Zsambok

C. E.

Klein

(Eds.), Naturalistic decision making (pp. 285–292). New York, NY: Psychology Press.

33.

Klein

(2008). Naturalistic decision making. Human Factors, 50, 456–460.

34.

Knauff

Wolf

A. G.

(2010). Complex cognition: the science of human reasoning, problem-solving, and decision-making. Cognitive Processing, 11, 99–102.

35.

Lazonder

A. W.

Harm

J. A.

Woperies

I. J.

(2000). Differences between novice and expert users in searching information on the World Wide Web. Journal of the American Society for Information Science, 51, 576–581.

36.

Leventhal

E. A.

Leventhal

Schaefer

Easterling

(1993). Conservation of energy, uncertainty reduction, and swift utilization of medical care among the elderly. Journal of Gerontology: Psychological Sciences, 48, 78–86.

37.

Lindell

M. K.

Perry

R. W.

(1992). Behavioral foundations of community emergency planning. Washington, DC: Hemisphere Press.

38.

Lipshitz

Klein

Orasanu

Salas

(2001). Taking stock of naturalistic decision making. Journal of Behavioral Decision Making, 14, 331–352.

39.

Meyer

B. J. F.

Russo

Talbot

(1995). Discourse comprehension and problem solving: Decisions about the treatment of breast cancer by women across the lifespan. Psychology and Aging, 10, 84–103.

40.

National Parks Service. (2014). Levels and types of ICS management. Retrieved from http://www.nps.gov/fire/wildland-fire/learning-center/fire-in-depth/incident-command-system-levels.cfm

41.

Osman

(2010). Controlling uncertainty: A review of human behavior in complex dynamic environments. Psychological Bulletin, 136, 65–86.

42.

Patel

S. C.

Drury

C. G.

Shalin

V. L.

(1998). Effectiveness of expert semantic knowledge as a navigational aid within hypertext. Behavior and Information Technology, 17, 313–324.

43.

Payne

J. W.

Bettman

J. R.

Johnson

E. J.

(1993). The adaptive decision maker. Cambridge, UK: Cambridge University Press.

44.

Sheridan

(1981). Understanding human error and aiding human diagnosis behavior in nuclear power plants. In Rasmussen

Rouse

W. B.

(Eds.), Human detection and diagnosis of system failure (pp. 19–35). New York, NY: Plenum Press.

45.

Sterman

J. D.

(1994). Learning in and about complex systems. System Dynamics Review, 10, 291–330.

46.

Sterman

J. D.

Booth-Sweeney

(2002). Cloudy skies: Assessing public understanding of global warming. System Dynamics Review, 18, 207–240.

47.

Sutcliffe

Ennis

Watkinson

(2000). Empirical studies of end-user information searching. Journal of the American Society for Information Science, 51, 1211–1231.

48.

Theobald

D. M.

(2007). Expansion of the U.S. wildland-urban interface. Landscape and Urban Planning, 83, 340–354.

49.

Turkle

(1984). The second self: Computers and the human spirit. London, UK: Granada.

50.

White

Morris

(2007). Investigating the querying and browsing behavior of advanced search engine users. In Kraaij

de Vries

A. P.

(Eds.), Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007) (pp. 255–262). New York, NY: ACM.

51.

Wilson

R. S.

Winter

P. L.

Maguire

L. A.

Ascher

(2011). Managing wildfire events: Risk-based decision making among a group of federal fire managers. Risk Analysis, 31, 805–818.