Abstract
Objective
The goal of the current study was to compare two types of shooting simulators to determine which is best suited for assessing different aspects of lethal force performance.
Background
Military and law enforcement personnel are often required to make decisions regarding the use of lethal force. A critical goal of both training and research endeavors surrounding lethal force is to find ways to simulate lethal force encounters to better understand behavior in those scenarios.
Method
Participants of varying degrees of experience completed both marksmanship and shoot/don’t shoot scenarios on both a video game and a military-grade shooting simulator. Using signal detection theory, we assessed sensitivity as a measure of lethal force performance overall. We used hit rate to assess shooting accuracy and false alarm rate to assess decision making.
Results
Results demonstrated that performance was correlated across simulators. Results supported the notion that shooting accuracy and decision making are independent components of performance. Individuals with firearms expertise outperformed novices on the military-grade simulator, but only with respect to shooting accuracy, not unintended casualties. Individuals with video game experience outperformed novices in the video game simulator, but again only on shooting accuracy.
Conclusion
Experience played a crucial role in the assessment of shooting accuracy on a given simulator platform; decision-making performance remained unaffected by experience level or type of simulator.
Application
We recommend that in expert populations or when assessing shooting accuracy, a military-grade shooting simulator be used. However, with a novice population and/or when interested in decision making in lethal force, a video game simulator is appropriate.
The use of lethal force carries severe consequences, which necessitates effective training. Numerous technological advances have been developed to improve training techniques for lethal force encounters. Still, significant opportunities exist for empirical research to help lethal force training move forward. A major hurdle in lethal force training and research is the need to simulate encounters under realistic, safe, and repeatable conditions. Many law enforcement and military instructional facilities use shooting simulators to achieve this objective. Applied researchers are beginning to use these simulators to measure lethal force performance. The literature is small, but growing, and made more relevant as simulators become available or researchers become more integrated into training regimens.
There is an open question regarding the fidelity of various types of shooting simulators and how simulator type may interact with participants’ level of experience and training—especially when considering the differences between equipment found in military facilities versus academic laboratories. For research to advance lethal force training, the technology and practices used in research should reflect those used in real-world operations. However, professional shooting simulators are not designed with research in mind, so a balance is needed between utility for researchers and ecological validity of a given simulator. This investigation was conducted to address these concerns by comparing a common military-grade simulator versus a laboratory-type simulator with emphases on decision making, shooting performance, and participant experience.
Two major constructs of interest in studying lethal force performance are (1) the decision to shoot (or not) and (2) the accuracy of a fired shot. Here, we will refer to the decision to shoot (or not) as “lethal force decision making” or LFDM. We measured LFDM as the rate of false alarms, or instances where a participant fired a shot either too soon or upon a nonhostile person. LFDM can only be measured in dynamic, shoot/don’t shoot scenarios. Second, the accuracy of a fired shot is referred to here as “shooting accuracy.” Shooting accuracy is measured as the rate of targets hit. Shooting accuracy, unlike LFDM, can be measured in dynamic shoot/don’t shoot scenarios, as well as in static marksmanship drills. In the current study, we refer to marksmanship as instances of measuring shooting accuracy that do not involve a judgment or decision component. In practice, these two constructs of LFDM and shooting accuracy are often thought of as interchangeable. Much lethal force training revolves around proper weapon use, and performance is assessed via marksmanship drills. However, marksmanship drills eliminate the decision-making aspect. We distinguish between these two aspects of performance because it is hypothesized that these two constructs are orthogonal. Successful execution of the motor responses necessary for accuracy does not mean successful identification of a threat. A poor decision combined with lethal accuracy likely describes an unintended casualty.
These considerations also need to be factored into research if laboratory-based simulations are going to have real-world corollaries. Several previous laboratory studies have examined the interaction between firearms and cognition using a variety of simulated weapons. For example, participants have been asked to respond using a gaming-style gun (Taylor et al., 2017; Witt & Brockmole, 2012; Xu & Inzlicht, 2015), a nonfunctional BB gun (Biggs et al., 2013), machined metal with a laser, handles, and a trigger (Wilson, Head, Helton et al., 2013, 2015), or a keyboard (Brewer et al., 2016; Correll et al., 2007). Gaming platforms are sometimes used along with these gaming-style guns to create a simulation of shoot/don’t shoot scenarios. For example, Biggs et al. (2015) used the Nintendo Wii and found that response inhibition training reduced the number of civilian casualties during game play.
The study by Biggs et al. (2015) provides examples of the advantages and limitations of video game-based simulators. An advantage is the ability to assess LFDM using an inexpensive and easily accessible platform. Another advantage is the ability to use this type of simulator with virtually any population due to its user-friendliness. However, one important feature of video game-based simulators is the use of a constructed aim point. With video games, participants move a controller around in real space but are not aiming through the weapon sights. Use of a constructed aim point may be functional in an untrained population because it allows for understanding where a shot will land. Even so, a constructed aim point may not resemble realistic marksmanship scenarios. Use of a constructed aim point could therefore limit the study of shooting accuracy.
In contrast, high-fidelity shooting simulators utilize a realistic weapon and a true aim point. Use of a true aim point provides the ability to investigate not only LFDM, but also shooting accuracy. While some previous studies have used realistic simulators to assess marksmanship (e.g., Goldberg et al., 2017; Yates, 2004), here we are interested in simulation used to measure both aspects of lethal force performance. Two previous studies have used military-grade simulators to assess electroencephalography measures during a shoot/don’t shoot task (Johnson et al., 2014; Kerick et al., 2007). Johnson et al. (2014) also compared novices and experts, but the scenarios were only scored as pass/fail, and the results showed that experts had significantly higher pass rates than novices. Finally, another study used a high-fidelity simulator to investigate the influence of suspect race on lethal force performance in both trained and untrained populations (James et al., 2013). However, James et al. (2013) did not explicitly compare performance between their groups of participants. This previous work represents a first step in understanding differences in lethal force performance in different populations. However, it remains unclear whether the type of simulator used interacts with the experience level of participants.
In the current study, we sought to compare performance within the same group of individuals on a shooting simulator that is representative of that available to a typical academic laboratory (i.e., a video game platform) and a shooting simulator that is representative of what is used by military and law enforcement for training purposes and is sometimes available to certain research laboratories (i.e., DoD labs). We took every action to equate these two simulators, but there are clear discrepancies, which will be elucidated in the “Method” section. The goal of the current study was not to equate the two simulators, but instead to assess performance on both and compare measures of lethal force performance. We sought to aid in the interpretation of past research using these types of simulators as well as aid development of new research into lethal force performance by determining appropriate boundaries for use of different types of simulators, for assessing the various components of lethal force performance in a variety of populations. Our goal was to assess the following hypotheses/predictions: (1) we predicted that there would be a significant positive relationship between lethal force performance across our two simulators; (2) we hypothesized that shooting accuracy would be unrelated to LFDM supporting the notion that these are indeed orthogonal constructs; (3) we also anticipated that the fidelity of the simulator would not influence LFDM performance because the realism of the weapon and a true aim point should not change the decision-making aspect; and (4) we wanted to investigate whether each simulator was sensitive to specific types of experience (i.e., firearms experience for the high-fidelity simulator and video game experience for the low-fidelity simulator).
Method
Participants
Seventy-two healthy adults (age: M = 30.25, SD = 7.67; 53 males) participated on a volunteer basis. Participants self-reported normal or corrected-to-normal vision. Participants gave written informed consent. The study protocol was approved by the Naval Medical Research Unit—Dayton Institutional Review Board in compliance with all applicable federal regulations governing the protection of human participants.
Materials
The video game simulator used was the Nintendo Wii game Reload: Target Down (Mastiff, 2013; Figure 1A, D). This game has been used previously in shooting research (Biggs et al., 2015) and chosen because it is not widely known and is a “rail shooter” game. This type of game requires the shooter to direct the weapon, but provides a consistent path through the game, which allows for performance comparison within and between participants. The Wii remote was placed in a black plastic holder (0.2 kg) to simulate a firearm (Figure 1A). The Wii remote allowed individuals to aim and pull the trigger button to shoot, as well as provided haptic feedback to simulate recoil. The game was displayed on a 60″ television screen and participants stood 8′ from the screen.
Panel A: Gun holster with Wii remote. Panel B: Game instructions showing targets to shoot and (Panel C) civilians to avoid shooting. Panel D: Example rooms from shoot/don’t shoot scenarios. Reprinted with permission.
The high-fidelity simulator was the Indoor Simulated Marksmanship Trainer (ISMT; Meggitt Training Systems, Suwanee, GA), which is used by the U.S. Marine Corps for training purposes. The system setup consisted of an operator computer station connected to a hit camera and projector (Figure 2A, C). The projection screen size was 75″ w × 57″ h and participants stood 15′ from the projection screen. The Bluefire® weaponry used were real M9 pistols (0.9 kg; Beretta 92FS; Figure 2A) with inner components replaced with an infrared laser and Bluetooth transmitter. The refillable magazines housed compressed air to simulate recoil. Speakers in the system provided auditory shot feedback. The hit camera detected shots fired and relayed the timing and accuracy information to the synced operator station.
Study Procedures
The study consisted of two sessions in counterbalanced order of simulator exposure (i.e., Wii and ISMT). Each study session was approximately 90 min, with 45–60 min spent on the shooting simulator. The Wii session included marksmanship and shoot/don’t shoot scenarios, surveys, and cognitive tasks. The ISMT session included marksmanship and shoot/don’t shoot scenarios and cognitive tasks. The results of the cognitive tasks will be reported elsewhere. The average number of days between sessions was 19.4 (SD = 30.7).
Questionnaires
In addition to a demographics questionnaire, a video game experience questionnaire (Blacker & Curby, 2013) was administered. Participants reported how many hours/week they spent playing each genre (i.e., first/third person shooter, fighting, strategy, fantasy, sports, and other) of game over the past year. Based on previous work, individuals who reported >5 hr/week were considered “gamers.” Individuals who reported <1 hr/week were considered “nongamers” and anyone who reported 1–5 hr/week were considered “intermediate” (e.g., Green & Bavelier, 2003).
Participants completed a firearms experience questionnaire that was developed specifically for this study. The questionnaire contained 12 questions (e.g., history of use, formal and informal training). The full questionnaire is available in the supplemental Online Information. Given the novelty of this questionnaire, seven independent raters assessed the questionnaire data for all participants with respect to expertise. Expertise was defined as “both experience (i.e., time spent practicing) and skill” and participants were categorized as novice, intermediate, or expert and assigned a 0–10 rating (0 = novice, e.g., never shot a firearm; 10 = expert, e.g., grand master competitive shooter, experienced law enforcement or military).
Shooting Simulator Tasks
Wii. For the Wii session, participants completed two baseline marksmanship courses of fire followed by shoot/don’t shoot room clearing scenarios. Prior to each task, an instruction screen presented images of shoot targets (Figure 1B) or civilians to avoid shooting (Figure 1C).
The initial marksmanship courses introduced stationary and moving targets, time limits, ammunition restrictions, and varying target distances. Once each course was complete, the experimenter recorded the scores including total score, highest combination, accuracy, number of headshots, number of targets shot, and number of civilian casualties. For the dynamic scenarios, two shoot house settings were used, where the shooter moved through a series of seven rooms with stationary and moving targets and civilians to avoid shooting (Figure 1D). The movement throughout the shoot house was preprogrammed so that movement of the shooters’ character was automatic and had a specific time limit. Both shoot houses were completed three times in an interleaved order with the first shoot house counterbalanced across participants.
ISMT. For the ISMT session, participants began with marksmanship courses to familiarize themselves with the system and to assess baseline marksmanship skill (see Table 1 for details). Next, participants were given instructions for the shoot/don’t shoot video scenarios as follows: “Now you will complete a series of video based scenarios. These scenarios will have ‘bad guys’ or hostiles that you are supposed to shoot. The scenarios may also have civilians or non-hostiles that you are not supposed to shoot. In these scenarios, your goal is to shoot hostile targets as quickly and accurately as possible. You want to be quick so that you shoot them before they can shoot at you. We will be keeping track of how many shots a hostile fires at you before you get a shot off. But you also want to be accurate. When you shoot, a circle will display on the screen: green means you missed the target completely, yellow means you landed a non-lethal hit, and red means you landed a lethal hit.”
ISMT Marksmanship Course of Fire Details
Note. aOnly the first three shots were scored.
Participants completed 33 unique video scenarios in one of two pseudorandom orders. Each video lasted between 20 and 180 s. Nine scenarios were “don’t shoot” scenarios that included no hostiles. Twenty-four scenarios were “shoot” scenarios that included between one and five hostiles. For each scenario, there were zero to five hostile targets presented and zero to five nonhostile individuals presented (i.e., nonhostiles could be civilians or potential hostiles who surrendered). The following data were recorded from each scenario: number of shots fired, targets hit, targets missed, lethal hits, nonlethal hits, false alarms (FA; i.e., shot too soon or shot a nonhostile/civilian), and first shot reaction time (RT). First shot was defined as the first instance where the participant fired a shot within a given scenario and the RT was automatically time-locked in the ISMT to the onset of the first target making a hostile action. All of these dependent variables were extracted from the video replay after data collection by one of the experimenters (and quality checked by a second experimenter).
Simulator Similarities and Differences
Participants completed the same basic set of tasks on both simulators, beginning with a set of marksmanship tasks that increased in difficulty. Participants then completed shoot/don’t shoot scenarios. For marksmanship, the Wii courses were scored as accuracy or a percentage of possible points and the ISMT courses were also scored as a percentage of possible points. In the shoot/don’t shoot scenarios, the proportion of shoot to don’t shoot scenarios in the ISMT (27%) was selected to equate the proportion of shoot to don’t shoot targets in the Wii (25%). While this is not a perfect comparison, the Wii scenarios were preset and the only variable we could control was how many times they completed each shoot house; however, each shoot house contained seven rooms, which can be considered separate scenarios. Thus, the ISMT had 33 scenarios, whereas the Wii had 42. We calculated hit rate, FA rate, and d′ for both simulators in order to compare performance in the shoot/don’t shoot scenarios. Further, speed and accuracy were stressed in both simulators for the shoot/don’t shoot scenarios. The Wii scenarios automatically included time pressure because the player only had a set amount of time to clear a room and participants were given more points for clearing the room faster. Participants were instructed to score as many points as possible in the Wii scenarios. Similarly, in the ISMT scenarios, we applied a time pressure by instructing participants to shoot hostile targets before the targets shot at them. Immediate shot accuracy feedback was also provided in both simulators and participants were stationary for both simulators.
There were some inherent differences between the simulators that were outside of our experimental control. First, the Wii scenarios included a “ground truth,” meaning that prior to each scenario they saw a preview of which characters were hostile and which were nonhostile (Figure 1B, C). Conversely, in the ISMT scenarios, participants had to make the decision to shoot based on the actions of the characters. Second, the Wii controller utilized a constructed aim point, whereas the ISMT used a true aim point, as discussed above. Finally, because of the lack of unique scenarios in the Wii, participants completed the two different shoot houses three times each. Some ISMT scenarios took place in the same environment (e.g., same building), but the actions of the characters were never repeated in the same environment.
Statistical Analyses
Currently, there is no consensus about the best approach for quantifying performance in shoot/don’t shoot scenarios. For example, some previous work has used a binary pass/fail system (Johnson et al., 2014), which is often used in real training courses, but bypasses much quantitative data that is available. As such, the methodology used here represents a novel contribution and an effort to approach greater standardization. We utilized a sensitivity index (d′) derived from signal detection theory (Green & Swets, 1966) and its components. Sensitivity considers both hit rate, which was used to measure shooting accuracy, and FA rate, which was used to measure LFDM. For both the Wii and the ISMT, hit rate was calculated as the number of hostile targets hit divided by the total number of hostile targets presented, whereas FA rate was calculated as the number of FAs divided by the number of nonhostile targets presented. In the subsequent analyses, we will consider d′ as an overall measure of performance encompassing both constructs, but will independently consider hit rate and FA rate.
Results
Comparing Lethal Force Performance Across Simulators
ISMT data for ten participants was lost due to error and excluded from subsequent analyses. The first question of interest was how performance on the two simulators compared to one another. We examined correlations between shoot/don’t shoot sensitivity (d′) on the simulators. Any participants with a d′ < 0 on either simulator was excluded (n = 1). As shown in Figure 3A, significant correlation between d′ on the Wii and the ISMT emerged, R = .390, p < .005. Further, we were interested in whether this effect was driven by hit rate or FA rate. Analyses revealed that both hit rate, R = .476, p < .001, and FA rate, R = .358, p < .01, were significantly positively correlated across simulators (Figure 3B, C).
Scatterplots illustrating a significant positive correlation between the Wii and Indoor Simulated Marksmanship Trainer (ISMT) for (Panel A) d′, (Panel B) hit rate, and (Panel C) false alarm rate. As seen in Figure 3B, there is a statistical outlier for ISMT hit rate; however, this participant’s d′ value was not a statistical outlier and excluding this participant did not change the significance or direction of any reported effects.
Relationship Between Marksmanship and Lethal Force Performance
The next question was how marksmanship performance related to shoot/don’t shoot performance. For the ISMT, marksmanship was significantly positively correlated with hit rate, R = .68, p < .001 (Figure 4A), but showed no relationship with FA rate, R = −.12, p = .358 (Figure 4B). For the Wii, accuracy on the marksmanship course was significantly correlated with hit rate, R = .413, p < .001 (Figure 5A), and FA rate, R = −.324, p < .05 (Figure 5B). Interestingly, this marksmanship performance on the Wii was also significantly correlated with marksmanship on the ISMT, R = .455, p < .001.
Scatterplots showing a significant positive correlation between marksmanship on the Indoor Simulated Marksmanship Trainer and (Panel A) hit rate, but no relationship with (Panel B) false alarm rate.
Scatterplots showing a significant positive correlation between baseline accuracy on the Wii and (Panel A) hit rate, and a significant negative correlation with (Panel B) false alarm rate.
While marksmanship is one way to measure our construct of shooting accuracy, hit rate in the shoot/don’t shoot scenarios represents another critical measure of shooting accuracy. As predicted, hit rate and FA rate were not significantly correlated with one another on the ISMT, R = −.017, p = .899, or the Wii, R = −.151, p = .247.
First Shot RT Analyses
One of the unique dependent variables that the ISMT provided is first shot RT. First shot RT was standardized via z-score per scenario, for shoot scenarios only. Then an average first shot RT for each participant was created by averaging their standardized values across all scenarios. This value therefore indicates whether an individual decided to shoot faster (negative values) or slower (positive values) compared to other participants in our sample. We found that first shot RT was significantly negatively correlated with both hit rate, R = −.424, p < .005 (Figure 6A), and FA rate, R = −.411, p < .005 (Figure 6B). These relationships indicate that the faster an individual decided to shoot, the higher their hit rate was, but also the higher their FA rate. The latter relationship is particularly relevant as it suggests that the faster an individual decided to shoot, the more likely they were to inflict an unintended casualty.
Scatterplots showing significant negative correlations between first shot reaction time (RT) and (Panel A) hit rate and (Panel B) false alarm rate. First shot RT is shown as a z-score.
Firearms Training and Expertise
To assess how experience influenced performance on the two simulators, we examined our sample with respect to firearms expertise, as well as video game expertise. Interrater reliability was found to be very high for expertise ratings (Cronbach’s alpha = .972). Further, each rater’s data was highly correlated with every other rater with all Rs ≥ .75, all ps < .001. For the categorization, the modal category amongst raters was used to group individuals. Table 2 shows the breakdown of the number of raters that chose the modal category. Individuals in each category where only three raters (i.e., not a majority) chose the modal category were included in the following analyses; however, excluding those individuals did not change the significance or the direction of any of the reported effects. Two approaches were used to examine differences in simulator performance based on expertise: (1) an extreme group’s approach comparing novice versus expert participants and (2) a correlational approach examining ratings as a continuous variable.
Rater Agreement for Categorization of Participants as Expert/Intermediate/Novice
Experts (n = 18) outperformed novices (n = 33) on the ISMT, as measured by d′’ t(49) = 2.436, p < .05, d = 0.711 (Figure 7A), which was driven by hit rate, t(49) = 3.017, p < .005, d = 0.959, not FA rate, t(49) = 0.579, p = .565, d = 0.171 (Figure 7B). The experts also outperformed the novices on ISMT marksmanship, t(49) = 2.895, p = .006, d = 0.874 (Figure 7C), but there was no difference on first shot RT, t(49) = 0.733, p = .467, d = 0.208 (Figure 7D). Importantly, there were no differences in d′ or hit/FA rate for the Wii between the two groups, all ts ≤ 0.751, all ps ≥ .457. These group results were paralleled in the continuous ratings data whereby more expertise was associated with higher d′ (ρ = 0.418, p < .005), hit rate (ρ = 0.515, p < .001), and marksmanship (ρ = 0.497, p < .001) on the ISMT, but not with any Wii metrics (all ps ≥ .404). Notably, there was no relationship between FAs or first shot RT and amount of expertise (both ps ≥ .094).
Comparison of novices versus experts on both simulators for (Panel A) d′, (Panel B) hit rate and false alarm (FA) rate, (Panel C) Indoor Simulated Marksmanship Trainer (ISMT) marksmanship, and (Panel D) first shot reaction time (RT) on the ISMT. Error bars represent standard error of the mean, *p < .05.
We examined the same simulator comparisons as described above, but for our extreme groups separately. For novices, d′, hit/FA rates, and marksmanship were all significantly positively correlated for the two simulators, all Rs ≥ .400, all ps < .05. However, for experts, these same correlations all yielded nonsignificant results, all Rs ≤ .331, all ps ≥ .180. These results suggest that while performance on the two simulators was comparable for novice individuals, this was not the case for experts.
Video Game Experience
We anticipated that firearms experience may influence performance on the ISMT. Similarly, individuals with more experience playing video games may be at an advantage in the Wii simulator. After accounting for our lost ISMT data (n = 10) and one outlier, our sample contained 25 “gamers,” 21 “nongamers,” and 16 “intermediate” participants. The intermediate group was not included in the following extreme group’s analysis to parallel the firearms experience approach.
Gamers outperformed nongamers on the Wii, as measured by d′, t(44) = 2.533, p < .05, d = 0.746 (Figure 8A), which was driven by hit rate, t(44) = 2.830, p < .01, d = 0.830, not FA rate, t(44) = 0.126, p = .900, d = 0.038 (Figure 8B). However, gamers and nongamers did not significantly differ in their performance on the ISMT, with respect to d′, t(44) = 0.236, p = .815, d = 0.069, hit rate, t(44) = 1.975, p = .055, d = 0.572, or FA rate, t(44) = 0.769, p = .446, d = 0.225. Interestingly, gamers were significantly faster than nongamers on first shot RT in the ISMT, t(44) = 2.830, p < .01, d = 0.812 (Figure 8D), and showed a marginally significant advantage in ISMT marksmanship performance, t(44) = 2.065, p = .045, d = 0.604 (Figure 8C). Gamers’ faster RT is consistent with previous research showing an RT advantage for individuals with video game experience (Dye et al., 2009). Similar to the firearms expertise data, correlations were tested amongst number of video game hours reported per week and the relevant shooting metrics. Any individuals who reported 0 hr/week were excluded from these analyses, to prevent skewing the data. The only significant correlation found was a negative relationship between hours/week of video game play and first shot RT, whereby more video game experience was associated with faster RTs, R = −.408, p < .01.
Comparison of gamers versus nongamers on both simulators for (Panel A) d′, (Panel B) hit rate and false alarm (FA) rate, (Panel C) Indoor Simulated Marksmanship Trainer (ISMT) marksmanship, and (Panel D) first shot RT on the ISMT. Error bars represent standard error of the mean, *p < .05. Note: One additional outlier that reported >60 hr/week of video game play was removed from these analyses; however, including the outlier did not change the direction or significance of the effect.
Discussion
Here, we set out to compare a video game and a military-grade shooting simulator. We had participants of varying degrees of experience complete both marksmanship and shoot/don’t shoot scenarios on both platforms. To compare performance across simulators, we used sensitivity, hits, and FAs from signal detection theory because they represent established metrics that can be used across different labs, groups, and simulators in future work. We found that performance on the shoot don’t/shoot scenarios was significantly correlated between the two simulators in terms of sensitivity, hit, and FA rate. Second, in line with our hypothesis, we demonstrated that while marksmanship and hit rate are correlated, shooting accuracy (as measured by marksmanship and hit rate in shoot/don’t shoot scenarios) on the ISMT was not related to FAs; and on the Wii, hit rate was not related to FAs, which supports the notion that LFDM and shooting accuracy are disparate constructs. Finally, by utilizing a diverse sample, we showed that firearms expertise influenced shooting accuracy performance on the ISMT simulator and video game experience influenced hit rate on the Wii, but experience was not related to LFDM on either simulator.
These findings suggest that both simulators can sufficiently assess LFDM in a diverse sample of participants and demonstrate that the rate of incorrect decisions is significantly correlated across both simulators. Furthermore, marksmanship had no relationship with this decision component in the high-fidelity simulator with a true aim point, which suggests that the ability to hit a target is not related to the decision to correctly shoot. This finding is particularly relevant for informing military and law enforcement training practices, as it implies that specific experience with shoot/don’t shoot scenarios is needed. A unique measurement yielded by the ISMT was first shot RT. This RT was negatively correlated with FA rate, which suggests that the faster an individual decided to shoot, the more likely they were to inflict an unintended casualty. While decades of research have demonstrated a speed–accuracy tradeoff in decision-making tasks (e.g., Fitts, 1966; Wickelgren, 1977), this is the first demonstration of this finding in the context of lethal force performance to the best of our knowledge.
Another of our interests in the current study was whether these two simulators are appropriate for different populations based on firearms expertise. We found that the correlations between simulators on our measures of interest held up for novices, but not experts. This effect was highlighted by the fact that experts outperformed novices on the ISMT with respect to shooting accuracy. While firearms expertise had no impact on performance on the video game simulator, the military-grade simulator was able to discriminate between these different populations. These findings are particularly relevant for research that examines specific facets of decision making in expert populations, such as how option generation influences outcome in lethal force encounters (Suss & Ward, 2018; Ward et al., 2011). Many of these studies use a delayed retrospective report method (Harris et al., 2017), but given the current findings, a high-fidelity simulator may provide an additional tool for expert populations in dissecting the decision-making process.
An important implication of these findings is that shooting accuracy and LFDM may be orthogonal constructs. This finding is novel and warrants further replication, but has significant translational potential. Specifically, LFDM training could potentially be conducted separately on a computer-based platform while marksmanship training is conducted on a range or with a simulator. This approach would create two distinct paths to training that could be integrated into carefully controlled scenarios, but to enhance performance, they may need to be trained and enhanced separately. Novel training to address LFDM is further supported by the failure to find differences between experts and novices in the rate of FAs. There were significant differences in marksmanship capability specific to the military-grade simulator with a true aim point, yet these training differences were not evident in decision making.
The current work has the potential to validate existing research in academia. It is a critical question whether low-fidelity simulators such as gaming systems or computer-based tasks can readily replicate realistic differences in LFDM. The answer appears to be a qualified yes. Specifically, marksmanship capabilities are not readily preserved unless the simulation possesses a true aim point, although there do not appear to be any decision-making differences between a novice and expert population. This lack of difference creates the potential for computer-based experiments to assess LFDM in both an expert and novice population. The present results thus suggest that even mock guns, video games, or other response input devices commonly found in laboratory-based settings could sufficiently replicate decision-making differences. Granted, there remain additional factors such as stress and experience that could impact LFDM based upon the population or weapon realism, and these factors need further exploration.
Based on these findings, we would recommend the following for future research into lethal force performance. First, we recommend the use of sensitivity as a standardized measure of LFDM and shooting accuracy performance to help investigators compare results across studies, simulators, and groups of participants. Second, we would argue that LFDM and shooting accuracy can be measured with either type of simulator used here when studying a novice population. However, with an expert population, a realistic, military-grade simulator is preferred. It follows that any comparison between groups is also preferred on a more realistic simulator. Third, while we found good reliability between simulators for marksmanship and hit rate, we would argue that a military-grade simulator is preferred when the measure of interest is shooting accuracy. The use of a true aim point inherently provides a more accurate measure of where an individual is shooting compared to a constructed aim point. It is possible that the relationship between marksmanship on the ISMT and Wii can be explained by variables that we did not account for such as visuomotor coordination or dexterity (e.g., Deeny et al., 2009; Sheeran, 1985). Finally, we would recommend that participant experience with firearms be captured in as much detail as possible and any study using a video game simulator would be advised to assess video game experience.
The current study had a few limitations that are worth detailing. First, we made every effort to equate the shoot/don’t shoot scenarios across the Wii and the ISMT. However, one difference that we could not avoid was the presence of a “ground truth” in the Wii scenarios. For the Wii, participants were explicitly shown which characters were hostile or nonhostile, and these characters were easily differentiated by their clothing, whereas the ISMT hostiles/nonhostiles had to be differentiated based on their actions alone. This difference likely accounts for the overall higher d′ for the Wii compared to the ISMT. Second, we attempted to measure shooting accuracy in both simulators by using hit rate in the shoot/don’t shoot scenarios, as well as an individual’s baseline marksmanship performance; however, only the ISMT tracked lethal and nonlethal hits, so no comparison across simulators could be made. Similarly, no RT measure can be extracted from the Wii game, which is a particularly important variable when discussing the temporal resolution of a lethal force decision. The additional measures that the ISMT provides represent limiting factors in using video game-based simulators. Another limitation concerns an alternative explanation that shots fired at civilians could be either a bad decision or simply poor aim. It is a possible criticism of LFDM research that there are limited theoretical positions to explain lethal force errors with some models based entirely upon police procedure rather than distinguishable cognitive or motor factors. However, in the current study, any sizeable percentage of LFDM errors attributable to shooting accuracy should have produced at least a minor correlation between marksmanship and FAs or hit rate and FAs in the shoot/don’t shoot scenarios. The lack of any such relationship suggests that, while it is possible some lethal force errors were due to poor accuracy, they are not a primary factor in lethal force errors. Finally, while our sample size here was moderate and adequate for the presented analyses, a larger sample size may have afforded additional statistical approaches such as factor analysis to confirm that LFDM and shooting accuracy are indeed orthogonal constructs. Future work should address this important topic.
Both simulators used here have advantages and disadvantages that include cost, access, user-friendliness, realism, flexibility, and so on. Thus, there is an increased need for cost-effective, flexible, and realistic simulators (Soetedjo et al., 2011) that can be used in both research and training. One additional approach to studying lethal force performance that is worth noting is the use of a constructed shoot house type of scenario that uses a mock projectile weapon (Wilson, Head, Helton et al., 2013, 2015). This approach introduces realism by having participants move in real space, identify actors as friend or foe, and aim a mock weapon in real space. Another area of exploration is the use of mixed reality in shooting simulations, but virtual and augmented reality, like video games, utilize a constructed aim point, which may limit conclusions about shooting accuracy. Although any virtual or augmented reality trainer may be effectively used for training that involves LFDM, there are significant concerns about a simulation that uses a constructed aim point if shooting accuracy is a key variable of interest. Emerging virtual and augmented reality systems are enhancing realism and may in the future be able to provide simulation that utilizes a true aim point.
A related point specifically involves conclusions that can be made regarding lethality, or the ability of an individual shooter to effectively engage a target with lethal fire. Specifically, different components of a reaction are more relevant at different engagement distances. Speed matters more in close quarters scenarios where a shooter is likely to be accurate, but accuracy matters more at greater distances where speed is a nominal variable unless the shooter can fire an accurate shot. This discrepancy could interact with the shot placement issues that occur when using a constructed aim point versus a true aim point. With regard to the simulation system, a constructed aim point could differentially impact either speed or accuracy and therefore have an undue influence on lethality interpretations. While this issue is outside the scope of the current study, this issue is worth considering in future research.
In conclusion, our results suggest similar utility of both video game and realistic shooting simulators in LFDM assessment. However, assessment of shooting accuracy, specifically in an experienced population, is superior in a realistic simulator with a true aim point. Finally, when comparing lethal force performance, particularly shooting accuracy, across populations that differ on expertise, we recommend the use of a military-grade simulator with a true aim point. Together, basic and applied research into lethal force performance has the potential to improve training practices for law enforcement and military personnel, as well as our understanding of how individuals make and execute lethal force judgments.
Key Points
Shoot/don’t shoot performance was significantly correlated between a video game and a military-grade shooting simulator.
Individuals with firearms expertise had better accuracy (i.e., shooting accuracy) on the military-grade simulator with a true aim point, compared to novices.
No difference was found between experts and novices on LFDM performance on either simulator.
Individuals with video game experience had better accuracy (i.e., shooting accuracy) on the video game simulator with a constructed aim point, compared to novices.
Supplemental Material
Supplementary Material 1 - Supplemental material for Measuring Lethal Force Performance in the Lab: The Effects of Simulator Realism and Participant Experience
Supplemental material, Supplementary Material 1, for Measuring Lethal Force Performance in the Lab: The Effects of Simulator Realism and Participant Experience by Kara J. Blacker, Kyle A. Pettijohn, Grant Roush and Adam T. Biggs in Human Factors: The Journal of Human Factors and Ergonomics Society
Footnotes
Acknowledgments
The views expressed in this article reflect the results of research conducted by the authors and do not necessarily reflect the official policy or position of the Department of the Navy, Department of Defense, or the U.S. Government. We would like to thank Andrew Warner, Dominick Pistone, and Mackenzie Riggenbach for assistance with data collection. This work was supported by the Office of Naval Research award H1602 to ATB. ATB generated the idea for the study. KJB, KAP, GR, and ATB designed the experiment. KJB categorized and chose the marksmanship and shoot/don’t shoot scenarios from both simulators. KJB, KAP, and GR collected the data. KJB analyzed the data. KJB wrote the first draft of the manuscript and KAP, GR, and ATB critically edited it. All authors approved the final submitted version of the manuscript.
Author(s) Note
The author(s) of this article are U.S. government employees and created the article within the scope of their employment. As a work of the U.S. federal government, the content of the article is in the public domain.
Supplemental Material
The online supplemental material is available with the manuscript on the HF website.
Author Biographies
Kara J. Blacker obtained her PhD in psychology from Temple University in 2013 and is currently affiliated with the Naval Medical Research Unit—Dayton and the Henry M. Jackson Foundation for the Advancement of Military Medicine, Inc.
Kyle A. Pettijohn obtained his PhD in psychology from University of Notre Dame in 2016 and is currently affiliated with the Naval Medical Research Unit—Dayton and the Henry M. Jackson Foundation for the Advancement of Military Medicine, Inc.
Grant Roush obtained his BA in psychology from California State University, Los Angeles, in 2010 and is currently affiliated with the Naval Medical Research Unit—Dayton and the Henry M. Jackson Foundation for the Advancement of Military Medicine, Inc.
Adam T. Biggs obtained his PhD in psychology from University of Notre Dame in 2011 and is currently a commissioned officer in the U.S. Navy.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
