Studying Terrorism Empirically: What We Know About What We Don’t Know

Abstract

Although the empirical and analytical study of terrorism has grown dramatically in the past decade and a half to incorporate more sophisticated statistical and econometric methods, data validity is still an open, first-order question. Specifically, methods for treating missing data often rely on strong, untestable, and often implicit assumptions about the nature of the missing values. We draw on Manski’s idea of no-assumption bounds to demonstrate the vulnerability of empirical results to different tactics for treating missing cases. Using a recently available open-source database on political extremists who radicalized in the United States, we show how point estimates of basic conditional probabilities can vary dramatically depending on the amount of missing data in certain variables and the methods used to address this issue. We conclude by advocating for researchers to be transparent when building analytical models about the assumptions they are making about the nature of the data and their implications for the analysis and its interpretation.

Keywords

point identification no assumption bounds data validity missing data domestic extremism terrorism data

It is always better to have no ideas than false ones; to believe nothing, than to believe what is wrong.

—Thomas Jefferson

The empirical and analytical study of terrorism has grown dramatically in the past decade and a half since the devastating attacks on September 11, 2001 (LaFree & Freilich, 2017; Ranstorp, 2009; Silke, 2007, 2009; Young & Findley, 2011). The surge in research has led to an assortment of increasingly sophisticated empirical approaches to studying correlates of terrorist activity. For instance, a variety of studies use time-series analysis (Apel & Hsu, 2017; Dugan & Chenoweth, 2012; S. Johnson & Braithwaite, 2017), cost–benefit analysis (Frey, Luechinger, & Stutzer, 2007, 2009), group-based mixture modeling (LaFree, Dugan, & Miller, 2015; Morris, 2017), hierarchical linear modeling (B. D. Johnson, 2017), and geospatial analysis (Behlendorf, LaFree, & Legault, 2012; LaFree, Dugan, Xie, & Singh, 2012).

Despite advances in methodological rigor, a more fundamental, and necessary, consideration for the proliferation of increasingly sophisticated analytic techniques is the validity of the data. Problems with data validity manifest in many forms, including ambiguity from open-source data (unclassified information from print and electronic media and other publicly available sources), unreliability of estimates, and measurement error, which may affect results in non-random ways. Of these concerns, limitations of open-source data have been the most directly discussed in existing terrorism literature, with scholars noting especially how biases in media reporting practices might prejudice data in particular directions (Dugan, 2011; Jongman, 1993; LaFree & Dugan, 2007). This line of research parallels criminological literature that analyzes differences between self-reported crime data and official arrest statistics, which may be biased by missing crimes not known to the police (Hindelang, Hirschi, & Weis, 1979; Menard, 1987; Pollock, Hill, Menard, & Elliott, 2016).

However, as Freilich and LaFree (2016) noted, while scholars studying terrorism have recognized the need for increased methodological rigor, few studies have systematically addressed first-order questions about issues of reliability and validity. In particular, recent quantitative research on terrorism has relied especially on open-source data, which often includes a good deal of missing information. Although researchers are generally forthright about their treatment of missing data, few prior studies have specifically examined the consequences of analysis with large amounts of missing data. In most instances, after indicating the amount of missing data (sometimes only in a footnote), point estimates are offered without further explanation.

However, arriving at point estimates necessarily requires researchers to make assumptions—often very strong ones—about the data being analyzed, including the extent and consequences of missing data. For instance, many analyses use multiple imputation techniques (e.g., Gruenewald & Pridemore, 2012; Jasko, LaFree, & Kruglanski, 2016; Mullins & Young, 2010), which assume that uncaptured data are missing at random (MAR; Rubin, 1976, 1987). Even stronger assumptions are required to drop certain cases or mean-substitute values, which assumes that cases are missing completely at random (MCAR; Rubin, 1976). Generally, to justify assumptions like these, researchers should be thoroughly transparent about the credibility of the assumptions in the study’s context. If the assumptions are unfounded, the estimates may be biased. Too often in empirical work, these assumptions are not explicitly stated and justified.

Much of the empirical research on terrorism (Asal & Rethemeyer, 2008; Cronin, 2006; Miller, 2012) has occurred at the group or event level of analysis, but missing data issues are also important for studies relying on individual-level terrorism or radicalization data (Gruenewald, Chermak, & Freilich, 2013; Pape, 2003; Smith, Damphousse, Jackson, & Sellers, 2002). As with data containing macro-level units of analysis, individual-level datasets are not immune to missing data concerns. Although researchers guided by certain theoretical traditions might be interested in including specific variables (like criminal record or mental illness history) in multivariate analysis, variables like these may be frequently missing in available open sources. In situations where data are missing for variables with strong theoretical relevance, the challenges may be especially great.

We hasten to add that we realize missing data is most often an unavoidable problem in the study of terrorism and violent extremism and we present no universal solution. However, with few exceptions (Chermak, Freilich, Parkin, & Lynch, 2012; Dugan & Distler, 2016), the existing literature on terrorism has not considered the repercussions of assumptions about missing data and evaluated the impact of these assumptions on conclusions about relationships between two variables. The current study illustrates the primacy of missing data issues in empirical terrorism research and the implications for ignoring them. Specifically, we proceed as follows: First, we briefly review individual-level studies in terrorism research, noting in particular the pervasiveness of missing data and the various techniques used to address the problem. Second, we provide a detailed explanation of the problems that threaten our ability to make statistical inferences, more specifically strict point estimates, with a high degree of non-random missingness. In particular, we draw on the ideas of Manski (2011, 2015) to guide our argument that making point estimates in these instances requires what are often untenable assumptions on the part of researchers. Third, using data from Profiles of Individual Radicalization in the United States (PIRUS), a relatively new individual-level terrorism dataset, we demonstrate the tenuous nature of certain point estimates by constructing “no-assumption” bounds around each estimate (Manski, 1990). The goal in our analysis is not to test a specific theoretical framework or empirical model, but rather to demonstrate the vulnerability of particular theoretically important variables to missing data concerns, which might affect the conclusions reached and their implications. Finally, we conclude by reiterating the need for transparency when making data assumptions. We do not argue that scholars should not make such assumptions, as this is impractical in an environment where data are never perfect. Rather, we stress that scholars should be clear and up-front about what assumptions they are making and both policy makers and researchers should be aware of the limitations these assumptions pose. Moreover, we stress that researchers studying terrorism exercise caution when relying on methods with unfounded or non-credible assumptions strictly for the utility of providing point estimates.

Open Source Terrorism Literature and Missing Data

Although much of the growing body of research using open-source terrorism data has not dealt extensively with the issue of missing data, there have been attempts to consider the quality of data. For example, Gassebner and Luechinger (2011) pooled a wide variety of correlates of terrorism and conducted multivariate analysis on terrorist attacks in three group-level databases using combinations of these correlates. Then, using extreme bounds analysis, they conducted a meta-analysis of all the results using bounds that range from the lowest to the highest coefficient. The authors were then able to determine which correlates were the most robust by singling out those variables whose bounds most consistently did not cross zero. Although this study was an important step in understanding consistent correlates across many analyses, missing data, though acknowledged (Gassebner & Luechinger, 2011, Footnote 5), was not directly addressed as the authors built their simulation models.

Gill and Horgan (2013) analyzed profiles of 1,240 former members of the Provision Irish Republican Army (PIRA) and compared the members across a number of pertinent defining demographic characteristics using sample means and cross-tabulations. Although the authors provide a useful descriptive analysis, they confronted a good deal of missing data on some of their measures. For example, the researchers report that nearly 60% of respondents’ birth counties were missing (Gill & Horgan, 2013, p. 445). As a result, while the authors express confidence in the findings based on the large sample available for the study, they also concede (Gill & Horgan, 2013, p. 438) that there is no completely satisfactory way of estimating the effects that missing observations have on the analysis.

Another prominent dataset containing information on individual extremists is the American Terrorism Study (ATS), collected by Smith and his colleagues (Smith & Damphousse, 1996, 1998; Smith et al., 2002; Smith & Orvis, 1993). The ATS draws data from Federal Bureau of Investigation (FBI) lists of individuals indicted after federal terrorism investigations and includes information on individual cases, counts per indictment and indictments per individual, prosecutor and defense data, outcome data, and group affiliation data. As with other open-source data based on terrorism and extremist crime, some variables in the ATS have a high percentage of missing data. For instance, Smith et al.’s (2002) descriptive statistics on charges levied against international terrorists show that data are missing for nearly 43% of cases (Smith et al., 2002, pp. 325-326). As such, the percentages they discuss only account for non-missing cases. Although the authors acknowledge that no conclusions are possible without data and analyses that are more complete (Smith et al., 2002, p. 329), the fact is that missing data may bias results, especially if missing data on charges is not randomly dispersed among the individuals in the dataset. In other statistical applications using the ATS, researchers have used listwise deletion (i.e., an entire record is excluded from the analysis if any single value is missing; Shields, Damphousse, & Smith, 2006) or simply dropped variables with large amounts of missing data (e.g., B. D. Johnson, 2012, p. 184). However, both listwise deletion and omitting variables with lots of missing cases has the potential to result in biased coefficients (Rubin, 1987); as such, this approach is tantamount to making a heroic MCAR assumption.

A third prominent database of radicalized individuals is the Extremist Crime Database (ECDB; Freilich & Chermak, 2009; Freilich, Chermak, Belli, Gruenewald, & Parkin, 2014). The database culls information from print and electronic media, court records and existing databases related to extremist actions in the United States, including both violent and non-violent crimes, and ideological and non-ideological crimes. As with previous examples, despite the unique contributions of the dataset, missing data remains an issue. For instance, Gruenewald and Pridemore (2012, p. 150) note that nearly half the available cases in their analysis using the ECDB contained missing data on at least one of the variables of interest. Several of the variables were missing more than a third of the time (p. 150). To compensate, the authors used multiple imputation by chained equations (MICE; see Royston, 2004; White, Royston, & Wood, 2011), which carried out a total of 50 imputed datasets without assuming that the included variables follow a multivariate normal distribution, as is typical of normal imputation methods. As noted by several other researchers who used the ECDB (Chermak et al., 2012; Kerodal, Freilich, & Chermak, 2016), and as with other data that rely heavily on open media sources, the nature of the missing data is probably not random and likely correlates with incidents and offender information deemed noteworthy by media sources. Given this, even multiple imputation techniques, which require a weaker MAR assumption, may not account for potential biases.

Finally, Jasko et al. (2016) use PIRUS data to model individuals’ decisions to commit acts of violent extremism within the quest for significance framework derived from psychological literature (Kruglanski, Chen, Dechesne, Fishman, & Orehek, 2009; Kruglanski & Orehek, 2011). As we also note in the current article, while the PIRUS data contain a rich set of demographics, characteristics, and measures related to individual radicalization and extremist behavior, many variables of interest in the dataset have high levels of missing information. Jasko et al. (2016) manage the missing data problem by performing multiple imputation analysis and conducting multivariate analysis across the pooled datasets. They also drop variables where more than 80% of the data are missing.

To be clear, if researchers simply drop cases with missing data from the analysis, it amounts to making an MCAR assumption. In other words, researchers are making the assumption that the point estimate derived from complete cases is, in expectation, not statistically different from the point estimate obtained using all cases. If this assumption is wrong, it is a problem of either selection bias or external validity. Moreover, it is important to note that the problem is not that the analysis is not sophisticated enough—indeed using a more sophisticated econometric model may exacerbate the problem; rather, the problem is with the amount of missing information in the data. Importantly, neither MCAR or the weaker MAR assumption is without costs and how we proceed in invoking them has important implications for our conclusions. This is critical because providing point estimates for a set of multivariate coefficients without understanding or explicitly discussing the merit of their underlying assumptions may produce misleading results. Given the potential implications, we next explain the fundamental problem with the common methods researchers use for handling missing data in the terrorism and extremist political violence literature.

Problems With Missing Data for Inference

As researchers who study terrorism, we have a goal to try to learn, or infer, some features of the population (e.g., the use of violence given certain characteristics or behaviors) from a random sample taken from this population. Generally, inference about these population parameters requires assumptions (e.g., random sampling), which are incorporated along with the data. As Manski (1995) describes, this structure of data plus assumptions has allowed researchers to deconstruct the inference problem into two unique components—problems of statistical inference and problems of identification. Generally, social science research is quite familiar with the former. Statistical inference problems deal with the issue of what we can learn from a sample of finite size. For instance, given our data, can we reject the null hypothesis that some parameter is equal to zero? Questions like these involve calculating quantities such as T-scores (i.e., the ratio of the parameter estimate to its standard error), p values (i.e., the smallest level at which we can set the alpha level and still reject the null hypothesis), and confidence intervals (i.e., the range in which the value of the parameter lies with some probability C).

Ignoring or systematically dropping missing data can have serious consequences for inference and policy analysis. According to Manski (2015), policy analysis requires a combination of data plus assumptions. Often, the assumptions which are implicit in an empirical model or set of conclusions arrived at by a researcher are either not well understood or not well articulated. In certain cases, assumptions can be overly strong, unbelievable, or even heroic. For instance, Manski (2011, p. 261) refers to the concept of incredible certitude, that is, “conclusions resting on critical unsupported assumptions or leaps of logic.” In other words, any conclusions derived from models using assumptions so strong that they are unbelievable are problematic for effective policy analysis. A necessary condition, then, for credible empirical conclusions is complete transparency on the part of researchers as to what assumptions they are making in the estimation of any empirical model.

Partial Identification and Nonparametric Bounds

As Manski (1995) noted, the inference problem also entails problems of identification, which the social sciences address in a less satisfactory manner. Manski (1995) argues that while identification has been traditionally thought of as a yes-no question—that is, is a parameter identified or not?—this thinking is actually incorrect. Instances in which we can definitely answer yes to a question result in point identification. If a parameter is point identified, then we are able to determine a single point estimate of it (albeit with usual sampling variability). Yet, as we have argued above, there are many instances where point identification requires making some necessary assumptions which are simply too strong to be credible.¹ Instead, we might be more comfortable with imposing weaker assumptions that are more credible, yet, alone, are incapable of justifying point identification. This leads to a solution of partial identification, where instead of a single point estimate of the treatment effect, we are able to derive bounds between which the true effect must lie.

Consider the MCAR assumption frequently made with missing data. This assumption is violated if cases with complete data are different from those with missing data. In such instances, and as illustrated in more detail below, Manski and Nagin (1998) recommend to begin with “no-assumption” bounds; that is, determine what we can learn from the data alone. No-assumption bounds in the case of prediction often still lead to ambiguous conclusions, though they are undisputable in that all researchers can agree on what is contained in them, because they do not require arguable assumptions. Researchers can then layer on increasingly strong behavioral assumptions, the validity of which they can address individually. A key element of this process is the transparency of assumptions that lead to the collection of findings, as opposed to findings that may be based instead on tenuous assumptions.

In sum, any analysis involving missing data necessarily requires assumptions, some of which may not be tenable. Again, we do not wish to undermine the importance of prior empirical research on terrorism; indeed, we hold the opposite view. Rather, we believe that researchers should be systematically transparent about the nature of these assumptions.

Data and Measures

The current study uses data from the PIRUS project (Jensen et al., 2016). To be eligible for inclusion in PIRUS, individuals must have radicalized within the United States to such a degree that they committed ideologically motivated illegal violent or non-violent acts, joined a designated terrorist organization, or associated with an organization whose leader(s) has/have been indicted of an ideologically motivated violent offense. Furthermore, each individual had to meet at least one of the following criteria: (a) arrested or indicted for illegal ideologically motivated behavior, (b) killed in the process of engaging in illegal ideologically motivated behavior, (c) an avowed member of a designated terrorist organization, or (d) associated with an organization whose leader(s) or founder(s) has/have been indicted for an ideologically motivated violent offense. Finally, to warrant inclusion, there must be evidence that all included individuals have engaged in illegal behavior because of their extremist political ideologies.

PIRUS draws from publicly available print and electronic media, court documents, and published sources on 1,473 individuals from 1948 to 2014. For all individuals, most or all of their radicalization must have occurred while they were residing in the United States. Hence, we consider all of these cases to be domestic. PIRUS includes individuals who left the United States to attend training camps abroad if there is evidence that they were already radicalized when they made the decision to attend. Researchers collected all data used in this study between July 1, 2013, and June 3, 2014. To check the reliability of the coding, researchers took a 10% random sample of cases and coded each case twice using separate individuals. They used several interrater reliability measures, prominently Krippendorff’s (2011) alpha procedure, to assess reliability of coding practices, which resulted in an alpha score of .76, which is above the common standard for acceptable reliability (>.70).

Dependent Variable

For the purpose of this study, the dependent variable is whether an individual’s exposure was for participation in, or planning of, a violent act.² Examples of violent acts include murder, robbery, kidnapping, and arson with the intent of causing bodily harm as well as conspiring or plotting to commit any such crime. Examples of non-violent acts include providing financial assistance to a terrorist group, protesting, committing arson with a specific goal of avoiding human injury, weapons possession, tax fraud, and recruiting for a terrorist organization. To clarify, an individual is only included in the database if they were identified as engaged in a politically motivated extremist act. As such, this variable distinguishes between violent extremists and non-violent extremists—a variable for which we have no missing information.

Independent Variables

Because our focus in this article is on methods, we specifically chose independent variables that were both common in social science theories of terrorism and violent extremism and that had a range of missing values so that we could demonstrate the varying impact degrees of missing values have on point estimates for the probability of violence. First, we include unemployment, measured “0” for unemployed and “1” for employed. Unemployment is missing for 61.2% of the cases. Second, we include perpetrators’ prior criminal history apart from the activity that brought them into the political extremism database. We code no criminal history “0” and any prior criminal history “1.” Criminal history data were missing in 53.97% of the cases. Third, we include a measure of whether individuals ever participated in the military. We code no military participation as “0” and any military enlistment as “1.” The military measure was missing in 41.89% of the PIRUS cases. Finally, we include a measure of whether perpetrators were part of an extremist political movement or group with “0” indicating no membership in a political movement or group and “1” indicating prior membership in an extremist political movement or group. Group membership was missing in only 1.5% of the cases.

We do not argue that these four included independent variables provide an exhaustive test of social science theories or are even the correlates most robustly related with outcomes of interest. Rather, we picked four variables that are of general interest to researchers and policy makers interested in political extremists and that exhibit varying amounts of missing data.³ We present the basic descriptive statistics for each variable, including percent missing, in Table 1.

Table 1.

Descriptive Statistics.

Variables	M ^a	SD	n	% missing
Dependent variable
Violent act	.626	.484	1,473	0.00
Independent variables (0 = no, 1 = yes)
Work history	.929	.255	571	61.24
Criminal history	.541	.499	678	53.97
Military history	.188	.391	856	41.89
Group membership	.856	.351	1,451	1.50

For non-missing cases only.

As shown in Table 1, over 62% of the 1,473 respondents had committed violent acts. A large majority (92.9%) reported some type of employment. Over 50% had a prior criminal record. Nearly 19% served in the military at some time. Finally, over 85% reported membership in some terrorist or extremist movement or group. The amount of missing data on these four variables ranges from a high of nearly 62% (employment) to a low of 1.5% (group membership). For our purposes, the variables are ideal to include in the model not only because they represent typically theorized correlates of violent extremism but also to demonstrate the impact of estimating no-assumption bounds given that the variables differ in amount of missingness.

The Current Study

In this article, we consider the implications of ignoring missing data in a simple prediction problem typical of quantitative terrorism research. Specifically, we illustrate the impact of missing data in an analysis of four variables that may be related to whether individuals with an ideological commitment to extremist causes have engaged in violence. Following Manski (2015), we construct no-assumption bounds for each of these variables and then compare them to naïve cases where missing data are ignored.⁴ Our results reveal that even small amounts of missing data can lead to large uncertainty in prediction. Importantly, we note that even though we use a relatively simple example, issues with big assumptions about missing data are likely even more important in complex multivariate models.

Our goal here is not to recommend specific methods for imputing missing data or specific assumptions about how researchers should treat missing data. Instead, we are concerned with showing the seriousness of prediction challenges in terrorism research in instances where no assumptions are made and how attempts to refine estimates naturally require some major and perhaps unwarranted assumptions.

The Prediction Problem With Missing Data

Given its importance in understanding criminal activity, suppose we are interested in the probability that an individual included in the PIRUS data commits a violent act (V) given their employment or work status. That is, we are interested in the following two conditional probabilities: P(V|W) and P(V|NW), where W denotes someone who works and NW denotes someone who does not work. The problem is that we do not observe work status for some portion on the sample, that is, some data are missing on P(W) and P(NW), which implies that for this subset of individuals, we also do not know P(V|W) or P(V|NW).

To see how we can put bounds on these estimates, we can also define a variable if work status is missing (M) or not missing (NM). Here, the notations M and NM refer to work status but this can generalize to any predictor or set of predictors. Using this, we can also define the following quantities: P(V|W, NM) and P(V|NW, NM), which are directly observable. These are the conditional probabilities of becoming violent, given work status, for those individuals for whom we observe work status. Similarly, P(W|NM) and P(NW|NM) are definable from observable data. Mathematically, the assumption P(V|W, NM) = P(V|W) holds only if either (a) P(W|M) = 0 or (b) P(V|W, M) = P(V|W, NM).⁵ In other words, if we wish to estimate the probability of being violent, given some form of employment using only those individuals for whom we observe work status, then we must be willing to assume that either (a) none of the individuals who are missing are employed or (b) the proportion of violent individuals who are employed in the missing data is the same as the proportion in the observable data. Although neither of these assumptions is directly testable, it seems unlikely that either is true given that each requires major and likely untenable assumptions. Rather, a more defensible starting point is to ask what we can definitely say about this quantity without making any assumptions about the missing values. That is, what can we learn from the data alone? This simple example illustrates the potential consequences of missing data on a single variable but problems will be necessarily exacerbated in more complicated models which themselves rely on more assumptions.

Deriving No-assumption Bounds

We continue our basic illustration where we want to know the likelihood of violent behavior given that someone was working in some form (W). We already know from Table 1 that in this case, there is a considerable amount of data missing for individual’s work status, P(M) = .612. To account for missing data without making any assumptions, we begin by using Bayes Rule to write P(V|W) = P(V ∩ W) / P(W). This allows us to separate the quantity into two constituent parts. For the denominator, we can use the law of total probability to write P(W) = P(W|NM) × P(NM) + P(W|M) × P(M). From the data alone, we can directly observe three of these quantities: P(NM) = .388, P(M) = 1-.388 = .612, and P(W|NM) = .930. The fourth quantity P(W|M), though unobservable, must fall between [0, 1] as it is a probability. This means that P(W) is strictly bounded to fall between a lower bound of (.930)(.388) + (0)(.612) = .361 and an upper bound of (.930)(.388) + (1)(.612) = .973.

Similarly, for the numerator, we write P(V ∩ W) = P(V ∩ W|NM) × P(NM) + P(V ∩ W|M) × P(M). From the data, we directly observe P(V ∩ W|NM) = .529, along with P(NM) and P(M). Again realizing that P(V ∩ W|M) must fall between [0, 1], this means P(V ∩ W) has a lower bound of (.529)(.388) + (0)(.612) = .205 and an upper bound of (.529)(.388) + (1)(.612) = .817. That is even making no assumptions about the missing data, we know P(W) must fall in the range [.361, .973] and P(V ∩ W) must fall in the range [.205, .817].

Although we calculated a lower bound for P(W), note that if the P(W|M) = 0, this means that none of the missing data would affect the probability of violence conditional on employment, and therefore P(V|W, NM) = P(V|W). As such, the more important boundary for P(W) is the upper bound, which describes the case where all missing data are for those individuals who were employed. Dividing either of the bounds on the numerator by the upper bound of the denominator will yield the bounds on the P(V|W), the key quantity of interest. Dividing the lower bound of P(V ∩ W) by the upper bound of P(W) yields (.205) / (.973) = .211, while dividing the upper bound of P(V ∩ W) by the upper bound of P(W) yields (.817) / (.973) = .840. This means that without making any assumptions as to the amount of missing data, we know that P(V|W) must fall in the interval [.211, .840]. To put this into perspective, this range is far less precise than the simple point estimate derived from only complete cases. This uncertainty is important to consider. Although the range is large, we can begin to constrict it by making progressively stronger assumptions. For instance, Manski and Nagin (1998) show how the treatment effect of sentencing decisions in criminal cases can be narrowed by making behavioral assumptions about the judge’s behavior.

Using this same logic, we can compute the no-assumption bounds on the probability of violence conditional on not working (NW). This yields no-assumption bounds of [.034, .992] for P(V|NW). Were we not to attempt to account for the missing data and just use the point estimates from the observable data, we might conclude that not working is a strong predictor of violent behavior, as P(V|NW, NM) = .800, which is approximately 40% higher than P(V|W, NM) = .561. However, that the no-assumption bounds overlap so considerably means that we are not able to say that not working is more predictive of violence unless we are willing to make additional, stronger, and likely untestable assumptions.

The second independent variable we identified above, prior criminal history (CH), like work history, was missing at a fairly substantial rate (54%). As such, we might anticipate that the no-assumption bounds will yield comparable results in terms of how informative the predictive probabilities they yield are going to be. Like work status, we can compare the initial P(V|CH, NM) = .686 and compare it with the predicted probability of violence, given no criminal history (NCH) among non-missing data, P(V|NCH, NM) = .629. The predicted probability of violence, given a criminal history, among the non-missing data, is approximately 9% higher than the predicted probability of violence, given no criminal background. However, we can make the same no-assumption bounds for previous criminal history as we did for work history, which will reveal more about what the data can tell us when we make no distributional assumptions. Using the same mathematical processes as we demonstrated using work history, we calculate the no-assumption bounds for P(V|CH) to be [.217, .901] and for P(V|NCH) = [.176, .895]. The no-assumption bounds overlap almost entirely, showing that we are unable to conclude whether criminal history has any impact on violent extremism without making stronger assumptions about the data.

Next, consider military experience (MIL) as a predictor, where we are interested in P(V|MIL). The military experience variable is missing at 41.89%, which is a lesser rate than the previous two independent variables. First, we note that using only cases for which MIL is observable, the probability of becoming violent is almost identical for those individuals who have a history of military service and those who do not, .634 versus .643. However, because there is an extremely high amount of missing data for this variable, the no-assumption bounds are also extremely wide for P(V|MIL = 0) of [.342, .812] and for P(V|MIL = 1) of [.131, .924]. In this case, the high rate of missing values for MIL makes prediction using this variable very difficult, and narrowing these bounds requires strong additional assumptions on the part of the researcher.

Finally, as a contrast to the first three variables, which had high degrees of missing data, consider using group membership (GM) as a predictor. In this case, P(M) = .015, considerably smaller than the rate of missing data with employment or criminal history. For individuals belonging to a group (GM = 1), we can calculate P(V|GM = 1, NM) = .607. The no-assumption bounds for P(V|GM = 1) are [.593, .611]. These bounds are considerably tighter than in the previous case using employment as a predictor.

Thus, the width of the bounds is a direct function of the probability of missing data for the variables examined. The P(M) for work status is very large compared to the P(M) for group membership, hence the larger bounds for work status. For those individuals who do not belong to a group, P(V|GM = 0, NM) = .727, and the no-assumption bounds for P(V|GM = 0) are [.656, .752]. In this case, the two no-assumption bounds do not overlap. As with work history, the point estimates for P(V|GM = 1, NM) and P(V|GM = 0, NM) are considerably different, ~20% higher for those who are not part of a group. However, in this case, we can definitively say that the true probability of violence is higher for those who do not join a group than those who do, without making additional assumptions.

Taken as a whole, these examples illustrate why constructing no-assumption bounds can usefully depict the relative strength of relationships between variables. In Table 2, we summarize the steps we have just taken for each of the four independent variables (and two additional iterations of the group membership variable). Column 1 depicts the conditional probabilities of violence, given non-missing data, for each of the two possible values of the independent variables. Column 2 depicts the percentage difference between the conditional probabilities. Finally, columns 3 and 4 present the calculated lower and upper ends of the no-assumption bounds. The table re-emphasizes how important missing data assumptions are in drawing conclusions about predicted probabilities. Only in the case of the original group membership variable do the no-assumption bounds for group membership = 1 and group membership = 0 not overlap, indicating that in this case, missing data would not greatly affect inferences made about the relationship between the variable and participation in violent extremism. Conversely, the fact that each of the other three variables is missing at over 40% means that the bounds for each independent variable condition overlapped with the bounds for its converse. Despite some point estimates being numerically quite different from their counterparts, without additional information it would be unwise to draw strong conclusions.

Table 2.

Estimates of Conditional Probabilities and No-Assumption Bounds for Independent Variables.

		P(V\|X, NM)	% difference between P(V\|X, NM) for X = 1 and X = 0	Lower no-assumption bound for P(V\|X)	Higher no-assumption bound for P(V\|X)
Independent variables (X)
Work history	=1	0.570	20%	0.211	0.840
Work history	=0	0.800	20%	0.034	0.992
Criminal history	=1	0.686	9%	0.217	0.901
Criminal history	=0	0.629	9%	0.176	0.895
Military service	=1	0.634	1%	0.131	0.924
Military service	=0	0.643	1%	0.342	0.812
Group membership (GM)	=1	0.607	17%	0.593	0.611
Group membership (GM)	=0	0.727	17%	0.656	0.752

Discussion and Conclusion

The paths individuals take to radicalization has generated a good deal of research interest in recent years (Borum, 2011a, 2011b; Dalgaard-Nielsen, 2010; Jasko et al., 2016). A growing pool of open-source data is becoming available for researchers to unpack interesting and practically useful relationships among sets of variables. However, researchers using open sources to study terrorism and political extremism invariably confront non-trivial amounts of missing data. We intend for this article to provide caution in interpreting data reliant on variables with large amounts of missing data. To communicate our emphasis on recognizing data limitations, we conducted a demonstrative example using the PIRUS data to calculate the relationship between four criminologically relevant variables and the likelihood that an individual extremist would commit an act of violence. Each variable was missing for a non-zero percentage of individuals. The first main conclusion from our findings is that the widths of no-assumption bounds vary enormously depending on the amount of missing data for particular variables. A second and related conclusion is that when there is a great deal of missing data, in order to have informative results, the researcher is required to make sometimes very strong, and often untenable, assumptions about the nature of the missing data.

We want to stress that our emphasis in this article is not that a great deal of missing data make analysis impossible. On the contrary, we recognize that policy makers must still make important decisions based on analysis drawn from imperfect information. However, we also emphasize that it is critical to communicate to both policy makers and researchers that calculating point estimates with data that are missing for a significant portion of cases is not an assumption-free process. Furthermore, making different assumptions about the distribution of the missing data can have profound effects on the recommendations that ensue following analysis.

Let us reconsider the exercise we conducted above using unemployment as the independent variable and the likelihood of using violence as the dependent variable. As we noted, if we were to use listwise deletion for individuals whose value for work history was missing, the predicted probability of violence for individuals without employment is approximately 40% higher than the predicted probability of violence for employed individuals. If we take these findings as the true value of the population parameter probability, reasonable counterterrorism recommendations might emphasize providing policies such as unemployment benefits, employment services, or job training.

However, as we demonstrated, providing such a point estimate without fully explaining the constraints of missing data would be injudicious. Unless we are certain that missing data are MCAR, it is difficult to conduct listwise deletion and justify presenting point estimates as if they are parameter values. Indeed, after calculating no-assumption bounds for employment, given the prevalence of missing data for this variable, we demonstrate that the initial probabilities are potentially misleading. The no-assumption bounds, whose lower end represents the probability of violence if the missing data are treated as zero and whose upper end represents the probability of violence if the individuals with missing data are as likely as the individuals without missing data to be violent, revealed that the true relationship between employment status and engaging in violent extremism can only be estimated by making strong assumptions about missing data on employment status. The bounds’ uninformative ranges demonstrate the need for caution in interpreting the importance of this variable. Many researchers combat missing data with multiple imputation methods. However, as we have seen, this strategy is also not assumption-free and researchers often fail to be explicit about the nature of the assumptions made.

We suspect that some readers might be uncomfortable with the unsatisfying nature of the no-assumption bounds, and in some ways, we agree. However, the width of these bounds can often be narrowed by making reasonable and credible behavioral assumptions. For instance, Manski and Nagin (1998) show how such a process can work in a criminal sentencing–recidivism context by making reasonable assumptions about the behavior of judges. The authors note that disagreement can then reflect the nature of these behavioral assumptions. Future research would do well to take into account the important limitations of missing data in their analyses. Terrorism researchers still know relatively little about correlates of individual-level extremism; specifically, we know little regarding what behavioral assumptions we can make about individuals that would help to narrow the bounds of missing data in the variables we analyze. We believe that as researchers begin to understand political extremism better, a pool of literature will develop that can help inform behavioral assumptions. Such a literature could help researchers layer their assumptions about specific variables in the face of missing data concerns.

If researchers choose to impute data, then they must be clear about the benefits and drawbacks of using an imputation technique. A recent work (LaFree, Jensen, James, & Safer-Lichtenstein, 2016) accounts for missing PIRUS data by conducting analyses using four different imputation techniques, each discussed with its own assumptions, pros, and cons. By checking for robust results across four analyses, which each account for missing data in different ways, the authors can be more confident about the nature of their results. Again, we note that it is incumbent upon the researcher to be as transparent and thorough as possible regarding the assumptions they are making, and then allow for reasonable debate as to the validity of such assumptions.

Despite the abundance of caution we preach as necessary in treating missing data, we are not blind to the practical limitations of such an approach. Indeed, to advocate for completely eschewing the use of data with missing values to make policy recommendations would render the purpose of many studies moot. Instead, we argue in favor of an abundance of caution in reporting results and categorizing the assumptions made to obtain those results. If we are transparent with respect to the assumptions made and the justifications for these assumptions in given situations, the obligation then falls on consumers of the research to make their own judgments about whether justifications are convincing. This conclusion is supported by Crenshaw and LaFree (2017) who conclude an examination of counterterrorism strategies with the advice that “a necessary condition for credible empirical conclusions requires transparency on the part of researchers and policymakers as to what assumptions they are making in evaluating available data and reaching conclusions” (p. 210). Our point is that any one decision about how to treat missing data might have a drastic impact on analytical results; as such, to contextualize the findings and their implications for policy, it is important to be able to catalog them according to the treatment of missing data and the assumptions being made in the analysis.

Footnotes

Authors’ Note

The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Department of Justice or the U.S. Department of Homeland Security.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the National Institute of Justice through Award 2012-ZA-BX-0005 and the Science and Technology Directorate of the U.S. Department of Homeland Security through Award 2012-ST-061-CS-0001.

Notes

Author Biographies

Aaron Safer-Lichtenstein is a PhD candidate in the department of Criminology and Criminal Justice at the University of Maryland. In addition to this, he works as a research analyst at the Study of Terrorism and Responses to Terrorism (START) consortium.

Gary LaFree is director of the National Consortium for the Study of Terrorism and Responses to Terrorism (START) and distinguished scholar and professor of Criminology and Criminal Justice at the University of Maryland. His research is on the causes and consequences of violent crime and terrorism.

Thomas Loughran is an associate professor in the department of Criminology and Criminal Justice at the University of Maryland, College Park. His research interests include offender decision making and deterrence, illegal market participation, and public policy.

References

Apel

Hsu

H. Y.

(2017). Interrupted time series analysis in the study of terrorism. In LaFree

Freilich

(Eds.), The handbook of the criminology of terrorism (pp. 276-294). West Sussex, UK: Wiley.

Asal

Rethemeyer

R. K.

(2008). The nature of the beast: Organizational structures and the lethality of terrorist attacks. The Journal of Politics, 70, 437-449.

Behlendorf

LaFree

Legault

(2012). Microcycles of violence: Evidence from terrorist attacks by ETA and the FMLN. Journal of Quantitative Criminology, 28, 49-75.

Borum

(2011a). Radicalization into violent extremism I: A review of definitions and applications of social science theories. Journal of Strategic Security, 4, 7-36.

Borum

(2011b). Radicalization into violent extremism II: A review of conceptual models and empirical research. Journal of Strategic Security, 4, 37-62.

Chermak

S. M.

Freilich

J. D.

Parkin

W. S.

Lynch

J. P.

(2012). American terrorism and extremist crime data sources and selectivity bias: An investigation focusing on homicide events committed by far-right extremists. Journal of Quantitative Criminology, 28, 191-218.

Crenshaw

LaFree

(2017). Countering terrorism: No simple solutions. Washington, DC: Brookings.

Cronin

A. K.

(2006). How al-Qaida ends: The decline and demise of terrorist groups. International Security, 31, 7-48.

Dalgaard-Nielsen

(2010). Violent radicalization in Europe: What we know and what we do not know. Studies in Conflict & Terrorism, 33, 797-814.

10.

Dugan

(2011). The making of the Global Terrorism Database and its applicability to studying the lifecycles of terrorist organizations. In Gadd

Karstedt

Messner

(Eds.), The SAGE handbook of criminological research methods (pp. 175-198). Los Angeles, CA: Sage.

11.

Dugan

Chenoweth

(2012). Moving beyond deterrence: The effectiveness of raising the expected utility of abstaining from terrorism in Israel. American Sociological Review, 77(4), 597-624.

12.

Dugan

Distler

(2016). Measuring terrorism. In LaFree

Freilich

(Eds.), Handbook on the criminology of terrorism (pp. 189-206). Hoboken, NJ: Wiley-Blackwell.

13.

Freilich

J. D.

Chermak

S. M.

(2009). United States extremist crime database (ECDB), 1990-2008: Preliminary results. Washington, DC: Department of Homeland Security University Network Research and Education Summit.

14.

Freilich

J. D.

Chermak

S. M.

Belli

Gruenewald

Parkin

W. S.

(2014). Introducing the United States extremist crime database (ECDB). Terrorism and Political Violence, 26, 372-384.

15.

Freilich

J. D.

LaFree

(2016). Measurement issues in the study of terrorism: Introducing the special issue. Studies in Conflict & Terrorism, 39, 569-579.

16.

Frey

B. S.

Luechinger

Stutzer

(2007). Calculating tragedy: Assessing the costs of terrorism. Journal of Economic Surveys, 21, 1-24.

17.

Frey

B. S.

Luechinger

Stutzer

(2009). The life satisfaction approach to valuing public goods: The case of terrorism. Public Choice, 138, 317-345.

18.

Gassebner

Luechinger

(2011). Lock, stock, and barrel: A comprehensive assessment of the determinants of terror. Public Choice, 149, 235-261.

19.

Gill

Horgan

(2013). Who were the volunteers? The shifting sociological and operational profile of 1240 Provisional Irish Republican Army members. Terrorism and Political Violence, 25, 435-456.

20.

Gruenewald

Chermak

Freilich

J. D.

(2013). Distinguishing “loner” attacks from other domestic extremist violence. Criminology & Public Policy, 12, 65-91.

21.

Gruenewald

Pridemore

W. A.

(2012). A comparison of ideologically-motivated homicides from the new extremist crime database and homicides from the supplementary homicide reports using multiple imputation by chained equations to handle missing values. Journal of Quantitative Criminology, 28, 141-162.

22.

Hindelang

M. J.

Hirschi

Weis

J. G.

(1979). Correlates of delinquency: The illusion of discrepancy between self-report and official measures. American Sociological Review, 44, 995-1014.

23.

Horowitz

Manski

(1998). Censoring of outcomes and regressors due to survey nonresponse: Identification and estimation using weights and imputations. Journal of Econometrics, 84, 37-58.

24.

Horowitz

Manski

(2000). Nonparametric analysis of randomized experiments with missing covariate and outcome data. Journal of the American Statistical Association, 95, 77-84.

25.

Jasko

LaFree

Kruglanski

(2016). Quest for significance and violent extremism: The case of domestic radicalization. Political Psychology. Advance online publication. doi:10.1111/pops.12376

26.

Jensen

LaFree

James

Atwell-Seate

Pisoiu

Stevenson

Tinsley

(2016). Empirical assessment of domestic radicalization (EADR) (NIJ Special Report). Washington, DC: National Institute of Justice.

27.

Johnson

B. D.

(2012). Cross-classified multilevel models: An application to the criminal case processing of indicted terrorists. Journal of Quantitative Criminology, 28, 163-189.

28.

Johnson

B. D.

(2017). Applying multilevel models to terrorism research. In LaFree

Freilich

(Eds.), The handbook of the criminology of terrorism (pp. 244-259). West Sussex, UK: Wiley.

29.

Johnson

Braithwaite

(2017). Spatial and temporal analysis of terrorism and insurgency. In LaFree

Freilich

(Eds.), The handbook of the criminology of terrorism (pp. 232-243). West Sussex, UK: Wiley.

30.

Jongman

A. J.

(1993). Trends in international and domestic terrorism in Western Europe, 1968-88. In Schmid

A. P.

Crelinsten

R. D.

(Eds.), Western responses to terrorism (pp. 26-76). London, England: Frank Cass.

31.

Kerodal

A. G.

Freilich

J. D.

Chermak

S. M.

(2016). Commitment to extremist ideology: Using factor analysis to move beyond binary measures of extremism. Studies in Conflict & Terrorism, 39, 687-711.

32.

Krippendorff

(2011). Computing Krippendorff’s alpha-reliability (Departmental papers [ASC], 43). Retrieved from http://repository.upenn.edu/ascpapers/43

33.

Kruglanski

A. W.

Chen

Dechesne

Fishman

Orehek

(2009). Fully committed: Suicide bombers’ motivation and the quest for personal significance. Political Psychology, 30, 331-357.

34.

Kruglanski

A. W.

Orehek

(2011). The role of quest for significance in motivating terrorism. In Forgas

Kruglanski

Williams

(Eds.), Social conflict and aggression (pp. 153-164). New York, NY: Psychology Press.

35.

LaFree

Dugan

(2007). Introducing the global terrorism database. Terrorism and Political Violence, 19, 181-204.

36.

LaFree

Dugan

Miller

(2015). Putting terrorism in context: Lessons from the Global Terrorism Database. London, England: Routledge.

37.

LaFree

Dugan

Xie

Singh

(2012). Spatial and temporal patterns of terrorist attacks by ETA 1970 to 2007. Journal of Quantitative Criminology, 28, 7-29.

38.

LaFree

Freilich

(2017). Bringing criminology into the study of terrorism. In LaFree

Freilich

(Eds.), The handbook of the criminology of terrorism (pp. 3-14). West Sussex, UK: Wiley.

39.

LaFree

Jensen

James

Safer-Lichtenstein

(2016, December 5-6). Correlates of violent political extremism in the United States. Paper presented at National Institute of Justice meeting on the Domestic Radicalization to Violent Extremism Program, Washington, DC.

40.

Manski

C. F.

(1990). Nonparametric bounds on treatment effects. The American Economic Review, 80, 319-323.

41.

Manski

C. F.

(1995). Identification problems in the social sciences. Cambridge, MA: Harvard University Press.

42.

Manski

C. F.

(2011). Policy analysis with incredible certitude. The Economic Journal, 121, F261-F289.

43.

Manski

C. F.

(2015). Communicating uncertainty in official economic statistics: An appraisal fifty years after Morgenstern. Journal of Economic Literature, 53, 631-653.

44.

Manski

C. F.

Nagin

D. S.

(1998). Bounding disagreements about treatment effects: A case study of sentencing and recidivism. Sociological Methodology, 28, 99-137.

45.

Menard

(1987). Short-term trends in crime and delinquency: A comparison of UCR, NCS, and self-report data. Justice Quarterly, 4, 455-474.

46.

Miller

(2012). Patterns of onset and decline among terrorist organizations. Journal of Quantitative Criminology, 28, 77-101.

47.

Morris

N. A.

(2017). Methodological advances in the study of terrorism: Using latent class growth analysis to estimate terrorism trends. In LaFree

Freilich

(Eds.), The handbook of the criminology of terrorism (pp. 244-259). West Sussex, UK: Wiley.

48.

Mullins

Young

(2010). Cultures of violence and acts of terror: Applying a legitimation–habituation model to terrorism. Crime & Delinquency, 58, 28-56.

49.

Pape

R. A.

(2003). The strategic logic of suicide terrorism. American Political Science Review, 97, 343-361.

50.

Pollock

Hill

M. C.

Menard

Elliott

D. S.

(2016). Predicting consistency between officially recorded and self-reported records of arrest. American Journal of Criminal Justice, 41, 623-644.

51.

Ranstorp

(2009). Mapping terrorism studies after 9/11: An academic field. In Jackson

Smyth

M. B.

Gunning

(Eds.), Critical terrorism studies: A new research agenda (pp. 16-22). London, England: Routledge.

52.

Royston

(2004). Multiple imputation of missing values. Stata Journal, 4, 227-241.

53.

Rubin

D. B.

(1976). Inference and missing data. Biometrika, 63, 581-592.

54.

Rubin

D. B.

(1987). Multiple imputation for nonresponse in surveys. New York, NY: John Wiley.

55.

Shields

C. A.

Damphousse

K. R.

Smith

B. L.

(2006). Their day in court: Assessing guilty plea rates among terrorists. Journal of Contemporary Criminal Justice, 22, 261-276.

56.

Silke

(2007). The impact of 9/11 on research on terrorism. In Ranstorp

(Ed.), Mapping terrorism research (pp. 175-193). Oxford, UK: Routledge.

57.

Silke

(2009). Contemporary terrorism studies: Issues in research. In Jackson

Smyth

Gunning

(Eds.), Critical terrorism studies: A new research agenda (pp. 34-48). New York, NY: Routledge.

58.

Smith

B. L.

Damphousse

K. R.

(1996). Punishing political offenders: The effect of political motive on federal sentencing decisions. Criminology, 34, 289-322.

59.

Smith

B. L.

Damphousse

K. R.

(1998). Terrorism, politics, and punishment: A test of structural-contextual theory and the “liberation hypothesis.” Criminology, 36, 67-92.

60.

Smith

B. L.

Damphousse

K. R.

Jackson

Sellers

(2002). The prosecution and punishment of international terrorists in federal courts: 1980-1998. Criminology & Public Policy, 1, 311-338.

61.

Smith

B. L.

Orvis

G. P.

(1993). America’s response to terrorism: An empirical analysis of federal intervention strategies during the 1980s. Justice Quarterly, 10, 661-681.

62.

White

I. R.

Royston

Wood

A. M.

(2011). Multiple imputation using chained equations: Issues and guidance for practice. Statistics in Medicine, 30, 377-399.

63.

Young

J. K.

Findley

M. G.

(2011). Promise and pitfalls of terrorism research. International Studies Review, 13, 411-431.