The Accuracy of the Violent Offender Identification Directive Tool to Predict Future Gun Violence

Introduction

Gun violence presents a growing concern to scholars and practitioners alike (Rozel & Mulvey, 2017). Research indicates that a number of factors are associated with involvement in criminal gun violence as a perpetrator or victim, including characteristics such as age, sex, race, economic and social disadvantage, area of residence, criminal history, social network characteristics, and previous exposure to violence, among others (Braga, Papachristos, & Hureau, 2010; Fowler, Dahlberg, Haileyesus, & Annest, 2015; Nofziger & Kurtz, 2005; Papachristos, Braga, & Hureau, 2012; Pear, Castillo-Carniglia, Kagawa, Cerdá, & Wintemute, 2018). Moreover, like many other aspects of criminal offending and victimization (Spelman, 1995), gun violence concentrates among a small number of individuals (Papachristos et al., 2012). Thus, in line with the Risk–Needs–Responsivity model of intervention (Andrews & Bonta, 2006; Andrews, Bonta, & Wormith, 2006), focusing interventions on those individuals who are most at risk may be a useful approach to attempt to reduce gun violence. Predicting future involvement in gun violence, however, is a challenging problem. Shootings, and violence in general, are relatively rare occurrences.

Notably, although risk assessment tools are commonly used to predict recidivism in sentencing contexts (Monahan & Skeem, 2016), validated instruments have not typically been used by police departments to predict future violence—particularly gun violence—among large populations of known individuals. In the current study, we apply a novel risk prediction tool—the Violent Offender Identification Directive (VOID)—in predicting future gun violence from information available in police records. This evaluation of the VOID tool in this study has two benefits to the wider criminal justice research and practitioner community. First, we quantify how well one can predict future involvement in gun violence in a large sample of individuals known to the police. Second, we use items typically available in police record management systems, meaning that much of the instrument can be replicated in other jurisdictions.

Violence Risk Assessment

There are three basic approaches to risk assessment: actuarial, structured professional judgment, and unstructured clinical judgment (Dolan & Doyle, 2000; Meehl, 1954; Singh, Desmarais, & Van Dorn, 2013). Actuarial tools assign weights to research-based risk factors to produce a weighted combination of risk factors known as a risk score. Structured professional judgment directs the users’ attention to specific, empirically validated risk factors. Unstructured clinical judgment, on the contrary, rests on more subjective clinical evaluations. In general, actuarial tools and structured clinical judgments both provide reasonably accurate risk assessments, and both perform better than unstructured clinical judgments (Dawes, 1979; Douglas, Yeomans, & Boer, 2005; Grove, Zald, Lebow, Snitz, & Nelson, 2000; Hanson, 2009; Kahneman, 2011; Meehl, 1954; Tetlock & Gardner, 2015).

Many risk assessment tools, using a variety of both unchanging or “static” risk factors (i.e., age at first arrest, gender) and changing or “dynamic” risk factors (e.g., residential stability, relationship status), have been found to predict violent behavior with some accuracy (Campbell, French, & Gendreau, 2009; Douglas et al., 2005), and no single tool emerges as the best predictor across multiple studies and meta-analyses (Campbell et al., 2009; Douglas et al., 2005; Fazel, Singh, Doll, & Grann, 2012; Glover, Nicholson, Hemmati, Bernfeld, & Quinsey, 2002; Kroner, Mills, & Reddon, 2005; Singh et al., 2013; Yang, Wong, & Coid, 2010). Common violence risk assessment tools (e.g., Historical, Clinical, Risk Management [HCR-20; C. D. Webster, Eaves, Douglas, & Wintrup, 1995]; Level of Service Inventory–Revised [LSI-R; Andrews & Bonta, 1995]; Structured Assessment of Violence Risk in Youth [SAVRY; Borum, Bartel, & Forth, 2003]; Violence Risk Appraisal Guide [VRAG; Quinsey, Harris, Rice, & Cormier, 2006]) typically include different combinations of static and dynamic criminal history/involvement items, measures of individuals’ stability and resources, contextual factors such as neighborhood disadvantage, and psychological characteristics.

Risk assessment tools are commonly used in corrections—particularly probation and parole—to determine which individuals are likely to recidivate (Miller & Maloney, 2013; Monahan & Skeem, 2016). Thus, risk assessment tools have generally been developed to follow a known set of individuals about whom a large amount of information can be gathered. Police, on the contrary, are interested in predicting future violence among a much larger set of individuals. Moreover, police departments are often limited to whatever information is available in their record management systems: Whereas police may have data on dynamic factors such as individuals’ gang membership or employment status, their information is frequently limited to prior criminal history and victimizations, along with basic demographic information (Jennings, 2006). While the lack of dynamic risk factors available in police record management systems prevents police departments from matching specific treatments to those predicted as being of high risk (a common goal of the majority of risk assessment instruments), the static characteristics available still tend to be predictive of involvement in future violence (Berk, Sherman, Barnes, Kurtz, & Ahlman, 2009; Neuilly, Zgoba, Tita, & Lee, 2011).

Early case studies have suggested that police departments traditionally have not relied on quantitative indicators of prior criminal history to identify chronic offenders, but rather target individuals who are simply believed to be currently active in committing crime based on officer intelligence (Martin & Sherman, 1986; Spelman, 1990). Although risk assessment tools for use by police officers have become increasingly popular in recent years (Messing, Campbell, Wilson, Brown, & Patchell, 2017; Storey, Kropp, Hart, Belfrage, & Strand, 2014), most of these focus on predicting future violence involvement of victims or perpetrators who are identified by police and “screened” via questionnaires to develop risk scores. In addition, common police risk assessment methods (e.g., the Lethality Screen used as part of the widely adopted Lethality Assessment Program [Messing et al., 2017]) tend to focus on specific contexts for violence, particularly domestic or intimate partner violence. Although several instruments used by police, including the Ontario Domestic Assault Risk Assessment (ODARA; Hilton et al., 2004) and Brief Spousal Assault Form for the Evaluation of Risk (B-SAFER; Kropp & Hart, 2004), have been found to effectively predict domestic or intimate partner violence (Hilton, Harris, Rice, Houghton, & Eke, 2008; Messing & Thaller, 2013; Storey et al., 2014), all focus exclusively on domestic violence in this context and remain limited in their ability to predict gun violence among individuals who have not been involved in previous incidents. A main reason for this is that collecting dynamic indicators, or otherwise conducting in-depth “screening,” for an entire set of individuals known to a police department is simply not feasible. However, the success of these instruments in predicting domestic or intimate partner recidivism does demonstrate the utility of risk assessment for police.

Given that a variety of static risk factors appear to be efficacious in predicting violence, however (Mills, Kroner, & Hemmati, 2003), it may be that police departments can achieve accurate predictions of future gun violence among a large population of individuals using only the data that they have available (e.g., prior arrests, victimizations, and other types of contact with the police). In what follows, we evaluate the predictive power of the VOID, a risk prediction instrument that is currently being used to this end by an upstate New York police department to identify people who may be involved in future gun violence, either as victims or as perpetrators, with a goal of providing interventions for those individuals prior to any involvement in violence.

The VOID

In brief, the VOID is a risk prediction instrument that is used in conjunction with two coordinated interventions, one that is enforcement-based and another that is prevention-based (i.e., social services). First, the police department calculates risk scores from the VOID tool. Next, supplementing this information with additional intelligence, the department identifies the 10 offenders at highest risk—the “top ten in crisis.” These 10 people are then prioritized for enforcement and preventive attention, and these interventions comprise the VOID.

The identification of the “top 10 in crisis” list occurs in two stages. In the first stage, numeric scores are calculated for persons known to the police department via prior police contacts. These are known as the VOID scores, based mainly on information about individuals’ prior criminal histories, and the algorithm by which the scores are generated is known as the VOID tool. These scores are then ranked, and analysts from the department’s Crime Analysis Center (CAC) review a large number of the highly ranked individuals, from among whom CAC personnel select the 10 individuals to be placed on the VOID list. Analysts use not only prior criminal history but also current human intelligence to determine which individuals to place on the list (Abrahamse, Ebener, Greenwood, Fitzgerald, & Kosin, 1991)—so the 10 individuals placed on the list do not necessarily have the 10 highest VOID scores.

Examples of intelligence used by analysts may include whether an individual is regularly out in public (so at higher risk) or “laying low” (at lower risk). Some pieces of information that obviously influence an individual’s risk are not captured in the police database (such as whether the individual is deceased, has moved outside of the department’s jurisdiction, or is currently incarcerated). These pieces of information can be used to eliminate individuals who may have high risk according to their VOID score, but should not be the target of any police intervention. Thus, although the initial construction of the scores is purely actuarial, officers’ subjective judgment is used to formulate the smaller pool of targeted individuals.

The VOID list is reconstructed on a monthly basis, and two scores are calculated for all known offenders. One, a historical score, takes into account all of an offender’s previous involvement in the various kinds of incidents. The other, a 90-day score, includes only the last 90 days’ involvement in those incidents. The department places more emphasis on the 90-day score, inasmuch as the object of VOID is to identify the people at high risk of near-term violence. Additional intelligence is gathered by the crime analysis unit on a day-to-day basis and at VOID meetings with Enhanced Supervision Unit (ESU) officers, as well as through debriefings of offenders and conferences with arresting officers.

Once placed on the VOID list, an individual is subject to a two-pronged effort to prevent his or her involvement in gun violence. Detectives initiate (or continue) investigations to build prosecutable cases against VOID list members, thereby removing them from the street and from further risk of involvement in gun violence. Such long-term investigations typically focus solely on drug offenders, so VOID helps to prioritize resources on individuals who are at high risk of violence offending and victimization.

At the same time, officers assigned to the ESU perform service-based outreach to individuals on the list, as well as to other parties who might exercise prosocial influences over the individuals. The ESU’s main function is to act as a liaison between the individuals and different social service agencies, as well as to collect intelligence on the current behavior of the top 10 in crisis. The current status of the individuals on the VOID list is reviewed at biweekly meetings that include detectives, ESU, crime analysts, command staff, and probation and parole representatives. Although the scores are reconstructed and reviewed every month, turnover of individuals on the list is not that frequent. Once individuals are on the list they tend to stay on the list until all parties involved deem it appropriate to remove an individual. To date, the majority of individuals placed on the list tend to typically stay on the list for around 6 to 12 months.

Development of the VOID Tool

The VOID tool was developed in 2012 by members of the police department’s analysis unit. Tasked in 2012 with formulating a mechanism for identifying people at very high risk of near-term involvement in gun violence (particularly a fatal or nonfatal shooting or a “shots fired” incident, and not merely a robbery with a gun or a menacing), staff identified a number of people who appeared from street intelligence to be at high risk. Nominations were sought from sworn personnel whose assignments (or whose approaches to their jobs) made them knowledgeable sources, such as community policing officers, detectives, and other patrol officers. The analysis unit proceeded to examine the backgrounds of these nominated high-risk individuals for markers that would substantiate the intel-based designation as high risk. The markers included, for example, weapons possession, truancy, and individual or family involvement in a street gang. This analysis of individual backgrounds confirmed that the identified individuals were people whose histories and circumstances were those that are plausibly associated with gun violence.

With a view toward making the risk scoring more systematic and broadly applicable, data were assembled on all shooters and shooting victims since 2009. Information from the record management system on their involvement in various kinds of incidents was collated and examined to identify those that were most prevalent, including but not limited to criminal possession of marijuana, involvement in shootings and shots fired incidents, resisting arrest, gang involvement, field interviews, a history as a missing person, domestic assault, and disorderly conduct. Each of these was used as a risk factor for constructing the VOID scores. Weights that were commensurate with the aggregate frequencies of the various incidents were formulated for each of the risk factors. Some types of incidents are also given greater weight based on context. For example, field interviews conducted in a known gang location are given additional weight above other field interviews.

Method

VOID Scores

Table 1 lists the VOID tool risk factors and the weights used for each individual item. Specifically, the VOID risk scoring tool incorporates information on offenders’

arrests (aggravated assaults, arson, burglary, etc.)

field interviews (with additional weight if a field interview is in a gang area)

victimizations (assault, aggravated assault, robbery, menacing, and reckless endangerment)

status as a suspect in a case (same offense types as victimizations)

appearance as a subject of a crime analyst bulletin (a “be on the lookout” bulletin disseminated from the CAC)

involvement in a juvenile offending incident (obtained from a specific database of juvenile incidents)

involvement in a runaway incident from a local facility

prior involvement in shots fired (as either victim, offender, or witness)

truancy

involvement in jail incidents (incidents of misconduct at a local jail)

known gang membership

OPEN IN VIEWER

Table 1:

VOID Items and Weights

Item	Weight
Arrest history
Aggravated assault	2
Assault	2
Arson	1
Burglary	1
Criminal mischief	1
Firearm possession	3
Larceny	2
Loitering/trespassing	3
Marijuana possession	4
Marijuana sale	4
Menacing	2
Murder	3
Possession of controlled substance	4
Reckless endangerment	2
Robbery	2
Sale of controlled substance	4
Sale of imitation substance	4
Violation of order of protection	2
Weapon possession	3
Disorderly conduct	5
Other
Field intelligence card	2
Field intelligence card in gang location	3
Gang member	5
Crime analyst bulletin	5
Jail incidents	2
Runaway	3
Runaway from facility	3
Shots fired involvement	10
Truant	3
Juvenile history
Aggravated assault	2
Arson	1
Assault	2
Burglary	1
Criminal mischief	1
Field intelligence card with gang member	3
Larceny	2
Loitering/trespassing	3
Marijuana possession	4
Marijuana sale	4
Menacing	2
Murder	3
Possession of controlled substance	4
Reckless endangerment	2
Disorderly conduct	5
Robbery	2
Sale of controlled substance	4
Sale of imitation substance	4
Weapon possession	3
Suspect history
Aggravated assault	2
Assault	2
Menacing	2
Reckless endangerment	2
Robbery	2
Victim history
Aggravated assault	2
Assault	2
Menacing	2
Reckless endangerment	2
Robbery	2

All of these factors are cumulative (with the exception of gang membership), and each is given a weight. Weights are applied to specific incidents, such that an offender’s score increases with each additional incident. For example, an arrest for robbery or aggravated assault carries a weight of 2. So, if a known offender had three previous robbery arrests and two previous aggravated assault arrests (and no points for anything else), that individual’s VOID score would be 3 × 2 + 2 × 2 = 10. Online supplementary analysis provides descriptive statistics of the VOID tool, as well as bivariate associations between each risk factor and involvement in gun violence (see Supplemental Material, available in the online version of this article).

Results

We assess the predictive accuracy of the VOID tool in absolute terms, in that we show the tool can effectively discriminate those at low and high risk effectively enough to be useful in practice. We also assess the relative accuracy of the VOID tool compared with five other optimized models. Four of them based on logistic regression models (King & Zeng, 2001; Tollenaar & van der Heijden, 2013) and the fifth based on a machine learning technique known as generalized boosted regression models or GBM (Ridgeway, 1999). ² These are meant to test whether we can use the same items in the VOID tool, but provide more accurate predictions using optimized regression results.

As several of the indicators are correlated, we fit two separate logistic regression models: the first includes each item individually as a predictor, eliminating predictors with zero results in the outcome; the second uses simple sum scores of violent arrests, nonviolent arrests, violent juvenile arrests, nonviolent juvenile arrests, victimizations, suspected offenses, and each of the remaining items individually in the model. We then fit each of these models using traditional logistic regression, as well as using a rare events logistic model (King & Zeng, 2001; Piquero, MacDonald, Dobrin, Daigle, & Cullen, 2005), which corrects for potential small sample bias in logistic regression given there are only 103 instances of the violent outcome in this sample. The GBM model includes all of the original items, allowed up to three-way interactions and a depth of the tree of up to 100 leaves. For each of these new models, we used the same pre-2013 data to predict gun violence in 2013 (either as a victim, offender, or combining either status). Because there is no “hold-out sample” for the regression models (i.e., cases that are not used to generate estimates of model parameters), we compare the model predictions to the 2013 data. This should provide a strong advantage to the new regression models compared with the VOID risk tool, as they are validated on the same sample that was used to generate model parameters. However, subsequent results show that both the logistic regressions and the generalized boosted predictions perform similarly to the original VOID tool.

The predictive accuracy of diagnostic tests is often evaluated by way of a receiver operating characteristic (ROC) curve, which plots the rate of true positives—called “sensitivity”—against the rate of false positives—called “specificity” (Davis & Goadrich, 2006). The ROC curve predicting involvement in either offending or victimization for the VOID scores, as well as the other optimized model predictions, is shown in Figure 1 (ROC graphs for the individual outcomes of specifically offending or victimization show very similar patterns). We would prefer high rates of true positives (correctly predicting individuals associated with gun violence) and low rates of false positives (falsely predicting someone will become involved in gun violence); though at any level of predictive accuracy, a higher rate of true positives can be achieved only by accepting a higher rate of false positives.

OPEN IN VIEWER

Figure 1:

ROC Curve for the Different Classification Models Predicting Either a Suspect in a Shooting or a Victim of a Shooting

Table 2 displays the area under the curve (AUC) statistics, their standard error, and the 95% confidence interval, lower and upper bound, for the VOID scores, as well as the five other models considered. It conducts this breakdown for the outcome of suspected involvement in gun violence, the victim of gun violence, or the union of the two categories (either a victim or suspect).

OPEN IN VIEWER

Table 2:

AUC Statistics for Each Model

Method	AUC	SE	95% CI
			Lower	Upper
Predicting suspected shooter
VOID	0.939	0.020	0.901	0.978
Logit 1	0.874	0.037	0.802	0.947
Logit 2	0.856	0.041	0.775	0.936
Rare Events Logit 1	0.938	0.019	0.900	0.976
Rare Events Logit 2	0.936	0.020	0.897	0.976
GBM	0.865	0.036	0.794	0.935
Predicting victimized by shooting
VOID	0.870	0.029	0.812	0.927
Logit 1	0.867	0.031	0.806	0.928
Logit 2	0.877	0.028	0.822	0.932
Rare Events Logit 1	0.852	0.032	0.789	0.914
Rare Events Logit 2	0.863	0.029	0.805	0.920
GBM	0.786	0.041	0.705	0.866
Predicting either victim of shooting or suspected shooter
VOID	0.898	0.019	0.860	0.935
Logit 1	0.862	0.026	0.812	0.912
Logit 2	0.858	0.026	0.807	0.909
Rare Events Logit 1	0.887	0.020	0.847	0.927
Rare Events Logit 2	0.892	0.019	0.854	0.930
GBM	0.813	0.029	0.755	0.870

Note. AUC = area under the curve; CI = confidence interval; VOID = Violent Offender Identification Directive; GBM = generalized boosted regression models.

For each outcome, VOID does quite well compared with each of the other models in terms of the AUC statistic, although when examining the confidence intervals for the different models they all substantively overlap. As each score is well above 0.5, most have an AUC of over 0.85, all of the models predict each outcome at a much better chance than random. An AUC of over 0.85 means that there is over an 85% likelihood of a gun violence case scoring higher on VOID than a non-gun violence case. Comparing the different outcomes of being a victim or a suspect, each prediction method (including VOID) does slightly better predicting suspects, as opposed to victims of shootings. Although, clearly each is better than random.

With an offender population of nearly 240,000, though, one should be more concerned with how well the highest scores predict involvement in gun violence. The crime analysts who use the tool do not have the capability to examine a large proportion of the population to make additional intelligence-based assessments, and so in practice there are limits to the threshold the analysts can set to further investigate whether an individual should be considered for placement on the VOID list. The ROC curves show that the top few percent of each prediction identifies about 60% of those who subsequently become involved in shootings as either a suspect or victim. However, just 1% of the population is still nearly 2,400 individuals.

Tables 3 and 4 present the classification rates, sensitivity (the proportion of true positives captured), the PPV (the positive predictive value, also sometimes known as precision), and the diagnostic odds ratio (Singh, 2013). ³ These statistics are provided for the top 1,000 scores in Table 3 (VOID only includes the top 959 because there are tied scores after that), as well as the top 5,000 scores in Table 4 for the VOID tool as well as the other models. The cut-offs were based on current practice (assessing the top 1,000 scores), as well as the authors’ estimates of how many more scores the analysts can reasonably be expected to examine based on personal observations of the process (likely somewhere around 5,000 scores). Different ways to select an optimal cut-off, such as Youden’s index (Krzanowski & Hand, 2009), select over 37,000 cases at a VOID score of over two in this sample, which is an infeasible number of individuals for the analysts to effectively review on a regular basis. ⁴

OPEN IN VIEWER

Table 3:

Classification Statistics of Top 1,000 Scores by Different Methods

Method	Number top scores	Total gun violence	Sensitivity	PPV	Diagnostic odds ratio
Predicting suspected shooter (outcome n = 56)
VOID	959	21	0.389	0.022	160.3
Logit 1	1,000	28	0.519	0.028	261.7
Logit 2	1,000	27	0.500	0.027	242.8
Rare Events Logit 1	1,000	24	0.444	0.024	193.6
Rare Events Logit 2	1,000	23	0.426	0.023	179.4
GBM	1,000	29	0.537	0.029	282.2
Predicting victimized by shooting (outcome n = 54)
VOID	959	16	0.286	0.017	100.2
Logit 1	1,000	17	0.304	0.017	104.7
Logit 2	1,000	16	0.286	0.016	96.0
Rare Events Logit 1	1,000	18	0.321	0.018	113.9
Rare Events Logit 2	1,000	19	0.339	0.019	123.6
GBM	1,000	18	0.321	0.018	113.9
Predicting either victim of shooting or suspected shooter (outcome n = 103)
VOID	959	32	0.311	0.033	114.8
Logit 1	1,000	40	0.388	0.040	156.2
Logit 2	1,000	37	0.359	0.037	137.5
Rare Events Logit 1	1,000	37	0.359	0.037	137.5
Rare Events Logit 2	1,000	36	0.350	0.036	131.6
GBM	1,000	41	0.398	0.041	162.9

Note. Total observations are 237,232. PPV = positive predictive value; VOID = Violent Offender Identification Directive; GBM = generalized boosted regression models.

OPEN IN VIEWER

Table 4:

Classification Statistics of Top 5,000 Scores by Different Methods

Method	Number top scores	Total gun violence	Sensitivity	PPV	Diagnostic odds ratio
Predicting suspected shooter (outcome n = 56)
VOID	4,773	39	0.722	0.008	127.7
Logit 1	4,829	39	0.722	0.008	126.1
Logit 2	4,747	34	0.630	0.007	83.9
Rare Events Logit 1	5,000	39	0.722	0.008	121.7
Rare Events Logit 2	4,999	37	0.685	0.007	101.9
GBM	4,837	40	0.741	0.008	138.4
Predicting victimized by shooting (outcome n = 54)
VOID	4,773	24	0.429	0.005	36.7
Logit 1	4,829	28	0.500	0.006	48.4
Logit 2	4,747	25	0.446	0.005	39.7
Rare Events Logit 1	5,000	26	0.464	0.005	40.5
Rare Events Logit 2	4,999	24	0.429	0.005	35.0
GBM	4,837	30	0.536	0.006	55.8
Predicting either victim of shooting or suspected shooter (outcome n = 103)
VOID	4,773	56	0.544	0.012	58.7
Logit 1	4,829	60	0.583	0.012	68.0
Logit 2	4,747	53	0.515	0.011	52.5
Rare Events Logit 1	5,000	58	0.563	0.012	60.6
Rare Events Logit 2	4,999	54	0.524	0.011	51.7
GBM	4,837	63	0.612	0.013	76.7

Note. Total observations are 237,232. PPV = positive predictive value; VOID = Violent Offender Identification Directive; GBM = generalized boosted regression models.

For the top 1,000 scores in Table 3, The VOID tool does a very good job relative to the rate of gun violence in the population, capturing 32 individuals who later become involved in gun violence as either a suspect or a victim. It is important to note, though, that predicting gun violence is difficult, as the number of people associated with gun violence remains low relative to the number of people identified by the tool. The PPV is only slightly over 3%, that is only around 3% of those within the top 1,000 scores will go on to actually be involved in gun violence. When breaking down by either victims or suspects the results are similar, as a ratio measure the VOID tool does quite well in capturing individuals involved in gun violence, but the PPV is smaller still for each individual outcome.

Widening the net to examine the top 5,000 scores (displayed in Table 4) results in a larger capture rate for VOID as well as the other predictive models examined. For the combined outcome, the different models result in capturing around 60 individuals, just slightly over half of the sample. This widening of the net, though, increases sensitivity but decreases PPV; when examining the top 5,000 scores, only slightly over 1% of the sample end up subsequently being involved in gun violence as either a victim or a suspect.

Given how the VOID tool is currently used in practice (i.e., analysts use field intelligence to decide which of the 1,000 or so individuals with the highest scores to place on the top-10 list), it is unlikely that the alternative models presented here would greatly outperform the VOID tool as it is currently applied.

Figure 2 shows how many of the individuals with the top N scores (shown on the x axis) became involved in gun violence (as either a victim or suspect). The figure is, then, essentially a continuous graphical representation of the number of true positives (the Total Gun Violence column) in Table 4. The VOID score ranking is displayed as a darker and thicker line, and rankings based on the other models are displayed as thinner gray lines. For the top 1,000 scores, the line for each model is between 30 and 40 individuals, and for the top 5,000 scores, each tool successfully predicts involvement in gun violence for 60 individuals. While the predictions from the other regression models slightly outperform the VOID scores at the lowest rankings, they each have similar performance by the time one gets to around the top 5,000 scores. VOID outperforms the other models once one gets past the top 11,000 scores, and results in a much larger AUC, but this is largely irrelevant to the way the tool is currently used. The model prediction that initially captures the most violent individuals until around the 10,000 mark in Figure 2 is the GBM prediction.

OPEN IN VIEWER

Figure 2:

Number of Gun Violent Cases (Either as a Victim or Suspect) Within Top Rankings of Each Model

This suggest that widening the net of initial VOID scores to examine from 1,000 to around 5,000 would potentially be beneficial, depending on the capacity of the analysts to evaluate that many cases. Even so, the VOID scores perform reasonably well compared with the optimal regression models, even when only considering the top 1,000 scores. There is a strong limitation, though, in predicting such a rare outcome of gun violence—one must be willing to tolerate a large proportion of false positives to capture a substantive number of future individuals likely to be involved with gun violence (Berk, 2011).

Discussion

The analyses reported here indicate that the VOID tool is predictive of future violence, and that optimized models using the same information are unlikely to appreciably improve upon the VOID tool. Moreover, we believe the ad hoc tool created by analysts performed as well as optimized regression models for several reasons, and that it would be practical for other police departments to implement similar risk prediction tools.

First, because many of the individual items used to signal risk within police databases are correlated, their contributions to predicting violence are likely in large part redundant. The work of Dawes (1979) is particularly applicable to the situation of generating risk scores from police records. In that article, Dawes showed that choosing ad hoc weights for regression equations only reduced prediction errors by a slight amount when the predictor variables have a positive effect on the outcome and the predictor variables all have positive inter-item correlations. This is a situation that readily applies to information available in police databases. Chronic offenders tend to be crime generalists, and any police contact is likely associated with a higher propensity for future criminal behavior. Any one particular act is only a small signal, but aggregated over many events produces a stronger signal that the offender is at high risk.

This finding that simple tools based on static risk factors can produce near similar prediction accuracy compared with more complicated instruments is a long-standing finding in recidivism research (Dressel & Farid, 2018; Harcourt, 2007; Tollenaar & van der Heijden, 2013). As such, it is not surprising that we found the same thing in predicting future involvement in violence based on police records. In fact, recent methods capitalizing on this fact take complicated models and reduce them to simpler instruments (Goel, Rao, & Shroff, 2016; Jung, Concannon, Shroff, Goel, & Goldstein, 2017; Zeng, Ustun, & Rudin, 2017). All of these factors make us confident that an instrument like VOID could be applied by police departments beyond the one we studied. Even if a department does not have access to the specific individual items used in the VOID instrument, it is still likely that combining scores from past criminal incidents that are available in police records could predict future violence reasonably well.

Conclusion

The analysis in the current study provides two major contributions to the wider criminal justice community. The first is illustrating the potential to predict future violence based on data often available to police departments. The second is the utility of predictive modeling for focusing police resources on high-risk gun offenders.

There are several limitations of the findings though. First, we only evaluate the efficacy of the general VOID tool to predict violence in a historical sample. We did not evaluate the prediction accuracy of the officers’ subjective judgments nor did we evaluate the effectiveness of the more dynamic 90-day VOID scores. Identifying whether these terms or other factors predict near-term risk of gun violence (as opposed to risk of gun violence in the next year) would provide more actionable information to a police department. Accomplishing such an assessment was not possible with the current agency, as shootings are too rare, and historical data could not be compiled from all of the databases further into the past. However, an analysis to determine whether very recent contacts with the police are diagnostic enough to use in making policing decisions could be accomplished using data from an agency with a larger sample size of shootings.

Evaluating the ability of the subjective judgment of intelligence officers provides an additional potential way to reduce the number of false positives. Even when limiting to the top 1,000 scores, the probability of a high-risk individual being subsequently involved in gun violence (either as a victim or suspect) was only around 3%. When widening the tool to examine the top 5,000 scores, its sensitivity increased, capturing around 60% of those involved in gun violence in the next year, but the PPV decreased to around 1%. When predicting a rare outcome one needs to balance capturing a small number of true positives by accepting a larger number of false positives (Berk, 2011); it may be the case that incorporating officer judgment can further improve the accuracy of the VOID instrument—either within the top scores or by incorporating further information to identify high-risk individuals not flagged by their prior police contacts.

A second limitation of the article is that we did not evaluate the effectiveness of the VOID program in reducing gun violence. Simply identifying the individuals at the most risk does not necessarily mean the police department will be effective in curbing their future gun violence. It is no doubt a difficult population to intervene with, and similar programs have not been found to be effective (Saunders, Hunt, & Hollywood, 2016). We have, however, validated that the tool has diagnostic utility in predicting those likely to be involved with future gun violence, which is presumably a first step before identifying whether a program effectively reduces future involvement with gun violence. The subjective judgment component of the VOID program in practice makes conducting outcome evaluations difficult, as those identified through intelligence will by design differ from those who were not chosen to be on the final focused list.

A final limitation is that we do not tackle ethical issues regarding the implementation of such a tool to identify chronic offenders. Although the police department under study does not use any demographic characteristics to determine VOID scores—such as age, race, or gender—that does not mean the instrument does not produce disparate outcomes among disadvantaged groups. Using prior criminal history creates a large disparity in identifying young male minorities with high VOID scores in this jurisdiction, and is likely to do so in other jurisdictions if police departments adopt similar instruments (Lum & Isaac, 2016). Because of the small number of individuals associated with gun violence, and the fact that many encounters do not collect this information, it is not possible to conduct additional analyses on these person-based characteristics in this sample. Future research, though, should consider assessing the fairness of different risk prediction instruments for cognizable groups (in particular minorities) by the police along with their accuracy (Berk, 2016; Corbett-Davies, Pierson, Feller, Goel, & Huq, 2017).

Relatedly, the VOID tool incorporates aspects that can reasonably be deemed out of the control of the individual, such as prior victimizations. The majority of police departments implementing chronic offender lists will likely use arrest histories readily available in police records, but these will often include instances in which no charges were ultimately found against the individual. Although arrest and victimization histories may be predictive of future involvement in gun violence, it is an ethical question whether such markers should be used by the police department to intervene with such individuals, as any intervention may infringe on an individual’s rights. This critique equally applies to the use of field stops as a component in the predictive system, which can create a feedback loop where individuals are continually targeted as high risk based solely on prior discretionary police behavior (Ferguson, 2017).

As the tool predicts violent offending as well as victimization, it brings up additional questions as to what the police should do with the predictions. This department has service-based aspects as well as traditional law enforcement tactics. Because the approaches are used simultaneously in this jurisdiction, we would not be able to say whether service-based approaches (which mitigate most of the ethical critiques of using the predictions) would work independent of traditional policing tactics (Papachristos, 2016). We do, however, demonstrate that the tool is effective in predicting who is at high risk of future violent gun offending or gun victimization, and as such it could be used for either strategy.

Finally, it is also the case that the subjective decision-making component of the VOID program, which identifies the top 10 “in crisis,” could add an additional stage that could result in the disproportionate targeting of vulnerable groups. Although there are still current debates about what exactly it means to be fair when making such predictions (Berk, 2016), police departments will need to address this problem alongside the question of whether such chronic offender programs are ultimately effective in reducing future violence.

Supplemental Material