Comparison of Naïve Decision Aides for Time-Pressured Degraded Environments

Abstract

Decision accuracy often suffers when missing information and time-pressure are introduced, and there is little consensus on how best to support decision making in these conditions. In this study we compare two naïve decision aides which aim to improve decision accuracy in two environments biasing towards Take-the-Best and Weighted Additive decision strategies. The first support aide slowly acquires missing information for the participants and the second aide provides option suggestions based on estimates of missing information. We found that while both decision aides were able to significantly improve decision accuracy, the aide which provided option suggestions outperformed the decision aide which acquired missing information. We also find that both decision aides have unique mediating effects on the presence of information imbalance.

Keywords

Decision Making Decision Support Systems Missing Information

Introduction

Numerous studies over the past three decades have been aimed at exploring not only the underlying processes of decision making, but also the reshaping of these processes under uncertainty (Garcia-Retamero & Rieskamp, 2009, 2008; Rieskamp & Hoffrage, 1999). These efforts are well-supported by the reality that inferences in everyday life are often made with uncertain or missing information and a limited time-frame. For instance, when emergency personnel engage in triage operations, it is imperative that accurate and timely judgements are made about the severity of the patients’ conditions. Physicians will never find themselves in possession of all information concerning their patients however, and it may become necessary to make decisions in lieu of perfect knowledge. In these situations, numerous decision support techniques can be employed to ensure more informed and accurate decisions are made.

Design and application of these decision support systems (DSS) is a widespread practice, and falls into many theoretical definitions largely depending on the application area of the system itself and the nature of the decision making process they seek to improve (Silver, 1990; Gonzalez, 2005; Silver, 1990). The approach to these systems range from complex and do main specific to fairly simplistic information managers or recommendation aides (Bennett et al., 2023; Deo, 2015; Strickland et al., 2022). However, the vast majority of the research on DSS ignores those scenarios in which the decision maker is forced to make choices with limited time and missing information.

In this study we aim to begin closing that research gap by first comparing two simplistic DSS applied to a multi-option, multi-attribute task characterized by missing information and time pressure. The first decision support proposed is an Acquisition (ACQ) agent which reveals missing information at discrete time steps during the task. In doing so, this agent externalizes an information search process the participants may have been able to do in real-world scenarios, which comes at the price of increased time pressure. The second proposed support method is an Estimates (EST) aide which presents three option suggestions to the participants, with each suggestion being informed by a separate estimation strategy for the missing information in the task. The DSS were tested in two environments favoring Take the-Best (TTB) and Weighted Additive (WADD) decision strategies separately. TTB, as introduced by the ABC Research Group, is a hallmark fast-and-frugal heuristic and has been proven independently and in comparison to WADD in numerous decision do mains(Gigerenzeretal.,1999;Garcia-Retamero&Dhami,2009; Rieskamp & Hoffrage, 2008). WADD is similarly prototypical in the decision making field and has been considered a gold standard for modeling rational choice and common decision rules (Payne et al., 1993).

We first hypothesize that our ACQ support method will outperform the EST support method in our environment biased towards TTB heuristics. We theorize this because the ACQ support prioritizes acquiring information rank-ordered on the predictive power of the cues, meaning that participants in the TTB environment can expect all information on the most predictive cue to eventually be acquired. We conversely hypothesized that the EST methods will better support our WADD environment, because the aide offloads much of the cue-summation analytic strategies require, or at least provides a reasonable reinforcing opinion on calculated option scores.

Method

A 2 x 2 factorial design (Decision Environment x Support Aide) was constructed with separate participants being assigned to each of the TTB_ACQ, TTB_EST, WADD_ACQ, and WADD_EST conditions.

The study features a gamified lane-defense task built in the Unity engine. Tasks in the game are structured around an OptionCue framework with each task contrasting 3 options characterized by 5 attributes (cues) which could be combined to determine option criterion values. Here the options take the form of lanes along which an enemy sprite could progress, and attributes which described the enemy which would traverse each lane. Participants are given a set of data containing missing information which describes the enemies in each lane, see Fig. 1. The goal of the game was to choose the lane with the strongest enemy in which to place a defense tower, and to do so within 60 seconds.

Figure 1.

Armor cue page in the general Intel menu showing missing cue for Lane 1, positive cue for Lane 2, and negative cue for Lane 3.

Task Design

Option attributes for the experiment are encoded as binary cue arrays (e.g. [1, 0, 1, 1, 0] for 5 attributes) where a 1 indicates a threat and 0 indicates a non-threat. The criterion values for each option are computed as the dot product of that option’s cue array and the cue weights of the environment. Another binary array encodes the missing information for each option in the task. Correct options were positively conditioned on the likelihood of missing information. In every 3-option task, this was done so that the correct option would have a 20% higher chance than the incorrect options for any of its attributes to be missing. Then, tasks were chosen so that a participant’s decision would reasonably reveal their decision-making process. For instance, the collection of tasks for the WADD environment only included tasks where TTB would consistently choose erroneously and a properly implemented WADD decision strategy would choose the right alternative. Additionally, tasks were designed so that the first, second, and third enemy were all viable choices in one-third of the rounds.

Participants

An a priori power analysis assuming a Fischer’s exact test and Type I error rate of 0.05 recommended a population size of 18 given a medium effect size (0.7). Consequently, a total of 60 participants were recruited for this study using the online recruitment platform Prolific. Participants were pre-screened to ensure users were fluent in English, over the age of 18, and had not participated in previous studies involving the same decision environment. Participants were awarded $7.50 for completion of the study with a potential bonus $2 for performing above 70% accuracy. Attention checks were implemented in the interface to ensure participants meaningfully interacted with the information displays. This study was approved by the Georgia Tech IRB committee.

Decision Interface and Support Methods

Within the experiment, participants completed a brief training period followed by 39 experiment tasks. The first 20 tasks involve no decision support and serve as a control period for the study. Upon completion of the 20^th task, participants were introduced to the support method they were assigned, and have access to that support for the remaining 19 rounds, which we denote as the “Intervention” period.

The participants in the decision-making process have access to a tab-based Intel menu, shown in Figure 1, which showcases the values for the five attributes. In the menu, threats that are minimal are depicted by green downward pointing arrowheads, while significant threats are indicated by red upward pointing arrowheads. The degradation of the environment is achieved through pieces of missing information, which are represented as a “?” in the Intel menu.

A feedback screen, as seen in Figure 2, was shown after each task completion. This screen informed the participants about the accuracy of their choice and revealed the cue values for each enemy (with previously missing information coded in blue) along with the true criterion value of the enemies. Participants were allowed to examine the feedback screen for as long as they wanted before proceeding to the next task.

Figure 2.

Feedback screen shown following every study round.

Decision Aides

As previously discussed, this experiment directly compares an information acquisition and an information estimation support aide. The information acquisition aide (ACQ) reveals a piece of missing information every 10 seconds from the start of each Intervention task and prioritizes gathering information by order of attribute rank (cue weight). Each time the ACQ aide acquired a piece of information, the Intel tab containing that new information was highlighted to notify the participants that the tab should be revisited (Tables 1 and 2).

Table 1.

Control and Intervention Accuracy: ACQ condition.

	Performance Group	Control	Intervention	Change
WADD	Low	39%	50.53%	+11.53
	Mid	50%	71.93%	+21.93*
	High	70%	85.53%	+35.53*
TTB	Low	36.67%	52.63%	+15.96
	Mid	60%	57.89%	-2.11
	High	70.91%	96.17%	+25.26*

Table 2.

Control and Intervention Accuracy: EST condition.

	Performance Group	Control	Intervention	Change
WADD	Low	40%	55.26%	+15.26*
	Mid	57.50%	72.11%	+14.61*
	High	75%	78.95%	+3.95
TTB	Low	27.86%	67.67%	+39.81*
	Mid	40%	87.72%	+47.72*
	High	68%	86.32%	+18.32*

The information estimation aide (EST) featured a “teammate” panel for the participants, which is shown in Figure 3. The panel displays three option choices and the calculated criterion values which led to the choices. The three choices displayed correspond to three information estimation strategies: optimistic, pessimistic, and average. The optimistic strategy replaces missing information with non-hostile guesses (0), the pessimistic with hostile guesses (1), and the average with middle-of the-road guesses (0.5).

Figure 3.

Teammate screen for the EST DSS showing option choices for three estimation strategies.

Results

Accuracy and decision speed were tracked as dependent variables, with accuracy being a binary (incorrect/correct) measurement. Analyses with accuracy as the dependent variable are reported with 1-tailed Fischer exact tests and analyses with the response time as the dependent variable used repeated measures ANOVA. Significant results are reported at the α = 0.05 level. Each participant was shown a randomized task set to minimize ordering effects.

Performance Groups

Participants were sorted into performance groups based on accuracy by taking the number of correct responses over the total number of trials and examining the modalities in the resulting groups of accuracy percentages. Natural groupings within the data emerged, which we have labeled as Low/Mid/High. Low, Mid, and High groupings correspond to an accuracy of 0-55%, 56-70%, and 71-100% and contained 19, 20, and 21 participants respectively. These groups are not evenly distributed within the decision environments or decision support aides, which influenced the use of a Fischer’s Exact test rather than a Chi-Square or G-Test of independence. For the sake of clarity, the TTB_ACQ contained no Mid-performers and the WADD_EST condition contained no High-performers. The TTB_EST and WADD_ACQ conditions were more balanced between the three groups.

Learning Effects

Participants in the study completed a brief set of training tasks before completing 20 Control tasks without a decision support tool and 19 Intervention tasks with their assigned support. Because these splits are not particularly large, learning effects may have played a significant role in Control to Intervention accuracy effects. To address this possibility, we have provided WADD and TTB condition data from a previous study (Sealy & Feigh, 2021) in Figures 4 and 5. The data was collected from the same environments and the same set of randomized experiment tasks and represents 20 participants in both the TTB and WADD condition completing 1,560 total tasks. In both conditions, there was no significant accuracy increase from the first to the second half of experimentation, suggesting that adaptation to the task environment has concluded by the midpoint of the study.

Figure 4.

Comparison of WADD results: (A) previous study (Sealy & Feigh, 2021), (B) ACQ, and (C) EST support aides in the current study. Blue (dotted) bars represent the first 20 decision tasks in all plots. Orange (filled) bars represent the latter 19 tasks which comprised the support aide-assisted tasks in the second and third plot. Error bars represent 95% confidence intervals of the mean.

Figure 5.

Comparison of TTB results: (A) previous study (Sealy & Feigh, 2021), (B) ACQ, and (C) EST support aides in the current study. Blue (dotted) bars represent the first 20 decision tasks in all plots. Orange (filled) bars represent the latter 19 tasks which comprised the support aide-assisted tasks in the second and third plot. Error bars represent 95% confidence intervals of the mean.

Information Acquisition Support Aide

The ACQ decision support aide significantly affected accuracy in both the WADD_ACQ (χ²(1, N = 585) = 17.04) and TTB_ACQ (χ²(1, N = 585) = 35.18) participants. This effect was more prominent in the WADD condition, with both Mid (χ²(1, N = 234) = 11.73) and High (χ²(1, N = 156) = 5.36) performers seeing significant increases in accuracy post-Intervention. A similar significant effect was only seen in the High performance group (χ²(1, N = 429) = 48.86) for the TTB condition.

Significant increase was observed in the response times for both the TTB_ACQ Low performers (F(1, 117) = 10.61), and the WADD_ACQ Low performers (F(1, 195) = 17.14). No significant effects were found for either Mid or High performance groups, with response times generally falling for these groups.

Information Estimation Support Aide

Results from the EST aide introduction were more promising overall for both environment conditions. Significant effects on accuracy were found in both the WADD_EST (χ²(1, N = 585) = 12.13) and TTB_EST (χ²(1, N = 585) = 71.43) groups, as well as within the Low (χ²(1, N = 156) = 4.12) and Mid performance groups (χ²(1, N = 390) = 9.07) of the WADD condition and all performance groups of the TTB condition.

Response times in the TTB_EST Low (F(1, 273) = 8.67) High performance groups (F(1, 195) = 4.56) were significantly reduced by the support aide. No significant effects were found for the WADD_EST subgroups.

Mediating Missing Information

Studies have indicated that the way missing information is distributed across options, which defines the degraded nature of the study, plays an important role in task difficulty and information processing (Rieskamp et al., 2003; Garcia-Retamero & Rieskamp, 2009; Körner et al., 2007; Canellas et al., 2015; Sealy & Feigh, 2021). Consequently, we were also interested in the ability of our decision aides to lessen the impact of missing information.

To observe these mediating effects, we quantify the number of cues in which at least one option had missing information. We refer to this metric as Information Imbalance, which can, in the case of a 5-cue decision task, range from 0 (all information is available) to 5 (every cue is missing a value for at least one option). We varied this in our tasks from 0-4. Figure 6 shows the difference in accuracy (Intervention - Control) for these levels of imbalance in each condition.

Figure 6.

Intervention-Control accuracy difference for TTB (green-solid) and WADD (purple-dashed) conditions with both support aides.

Discussion

Impacts on Decision Accuracy

Although both decision aides produced significant increases in accuracy across our analytic and heuristic decision environments, the evidence specifically examining the effect on performance groups somewhat contradicts our initial hypothesis for the Take the-Best environment. Weights for the TTB environment were calculated using the geometric series proposed by Martignon and Hoffrage, which guarantees TTB decisions can be represented as a linear model (Martignon & Hoffrage, 2002). This construction assures that any cue weight w_j is larger than the sum of all lower ranked cue weights and that a properly applied TTB strategy will be able to effectively discriminate on cues in rank-order given there is information to make said discrimination. Missing information is especially problematic for these non-compensatory environments due to the possibility of highly-ranked missing cues being necessary for successful application of the heuristic’s stopping rule. Knowing this, we expected that a method which revealed missing information would provide the best support for this environment assuming that participants were willing to wait for the information to be revealed. This was true for the High TTB_ACQ group who saw significant improvement in accuracy post-intervention, but not for participants who fell in the lower performance groups. Instead, the Low and Mid performers using the Estimates aide (TTB_EST) saw very significant increases in accuracy when using that decision aide instead. However, it should be noted that the majority (11 out of 15) of TTB_ACQ participants fell into this High performance group. The success of the Estimates aide in contradiction to the earlier hypothesis is reassuring however, because its application in real-world scenarios is not dependent on an ability to perform information retrieval, whereas an Acquisition aide may simply not be feasible to implement.

Impacts on Decision Speed

Changes in response times within the TTB_ACQ and WADD_ACQ groups were particularly interesting in the Low performance groups. While response times decreased slightly for other groups (with exception of the WADD_ACQ High performers), the Low performers saw significant increases of above in response times for both decision environments (over 8 seconds in each group). This suggests that these groups were spending greater amounts of time waiting for information to be revealed by the agent. Knowing this, we would expect more significant increases in accuracy across both groups, and especially in the TTB_ACQ Low performers, who would have seen highly-ranked discriminating cues being revealed most often.

Mediating Effect of Information Imbalance

The data suggest that both decision support methods were able to increase accuracy in cases where information imbalance was present. This trend is predictably more linear in the Acquisition aide conditions, where the imbalance is directly targeted by the mechanisms of the aide. Despite this linearity, the TTB environment was better served on average by the Estimates aide, which follows our findings of the significant overall accuracy increases within perform groups for the TTB_EST condition. Interestingly, the WADD environment reaches an inflection point at the case of 2 cues containing missing information, before which participants saw greater accuracy in the Estimates case, and after which saw a 5% accuracy increase in the Acquisition case.

Conclusions

Results from this experiment have shown significant promise of two naive decision aides in the support of both analytic and heuristic decision strategies. The support methods we tested constitute a rudimentary first pass at improving decision outcomes in environments with imperfect information.

Footnotes

ORCID iD

William I.N. Sealy

References

Bennett

K. B.

Edman

Cravens

Jackson

(2023). Decision support for flexible manufacturing systems: Application of the cognitive systems engineering and ecological interface design approach. Journal of Cognitive Engineering and Decision Making, 15553434221118976.

Canellas

M. C.

Feigh

K. M.

Chua

Z. K.

(2015, Dec). Accuracy and effort of decision-making strategies with incomplete information: Implications for decision support system design. IEEE Transactions on Human-Machine Systems, 45(6), 686-701. doi: 10.1109/THMS.2015.2420575

Deo

R. C.

(2015). Machine learning in medicine. Circulation, 132(20), 1920–1930.

Garcia-Retamero

Dhami

M. K.

(2009). Take-the-best in expert novice decision strategies for residential burglary. Psychonomic Bulletin & Review, 16(1), 163–169.

Garcia-Retamero

Rieskamp

(2008). Adaptive mechanisms for treating missing information: A simulation study. The Psychological Record, 58.

Garcia-Retamero

Rieskamp

(2009). Do people treat missing information adaptively when making inferences? The Quarterly Journal of Experimental Psychology, 62(10), 1991–2013. doi: 10.1080/17470210802602615

Gigerenzer

Todd

P. M.

Group

A. R.

, et al. (1999). Simple heuristics that make us smart. Oxford University Press New York.

Gonzalez

(2005). Decision support for real-time, dynamic decision-making tasks. Organizational Behavior and Human Decision Processes, 96(2), 142-154. doi: https://dx-doi-org.web.bisu.edu.cn/10.1016/j.obhdp.2004.11.002

Körner

Gertzen

Bettinger

Albert

(2007). Comparative judgments with missing information: A regression and process tracing analysis. Acta psychologica, 125(1), 66–84.

10.

Martignon

Hoffrage

(2002). Fast, frugal, and fit: Simple heuristics for paired comparison. Theory and Decision, 52(1), 29–71.

11.

Payne

J. W.

Bettman

J. R.

Johnson

E. J.

(1993). The adaptive decision maker. Cambridge University Press.

12.

Rieskamp

Busemeyer

J. R.

Laine

(2003). How do people learn to allocate resources? comparing two learning theories. Journal of Experimental Psychology: Learning, Memory & Cognition, 29, 1066–1081.

13.

Rieskamp

Hoffrage

(1999). When do people use simpleheuristics, and how can we tell? In Gigerenzer

Todd

P. M.

(Eds.), (pp. 141–167). New York: Oxford University Press.

14.

Rieskamp

Hoffrage

(2008). Inferences under time pressure: How opportunity costs affect strategy selection. Acta psychologica, 127(2), 258–276.

15.

Sealy

W. I.

Feigh

K. M.

(2021). Impact of missing information and strategy on decision making performance. In 2021 ieee international conference on systems, man, and cybernetics (smc) (pp. 3140–3145).

16.

Silver

M. S.

(1990). Decision support systems: directed and nondirected change. Information Systems Research, 1(1), 47–70.

17.

Strickland

Boag

R. J.

Heathcote

Bowden

Loft

(2022). Automated decision aids: When are they advisors and when do they take control of human decision making? Journal Of Experimental Psychology-Applied.