Advantages of Matched Over Unmatched Opt-in Samples for Studying Criminal Justice Attitudes: A Research Note

Abstract

Despite the growing popularity of online opt-in samples in criminology, recent work shows that resultant findings often do not generalize. Not all opt-in samples are alike, however, and matching may improve data quality. Replicating and extending prior work, we compare the generalizability of relational inferences from unmatched and matched opt-in samples. Estimating identical models for four criminal justice outcomes, we compare multivariate regression results from national matched (YouGov) and unmatched (MTurk) opt-in samples to those from the General Social Survey (GSS). YouGov coefficients are almost always in the same direction as GSS coefficients, especially when statistically significant, and are mostly of a similar magnitude; less than 10% of the YouGov and GSS coefficients differ significantly. By contrast, MTurk coefficients are more likely to be in the wrong direction, more likely to be much larger or smaller, and are about three times as likely to differ significantly from GSS coefficients. Matched opt-in samples provide a relatively inexpensive data source for criminal justice researchers, compared to probability samples, and also appear to carry a smaller generalizability penalty than unmatched samples. Our study suggests relational inferences from matched opt-in samples are more likely to generalize than those from unmatched samples.

Keywords

web survey public opinion Amazon Mechanical Turk sample matching

In an increasingly web-based, low-response-rate world, the use of online surveying has dramatically increased in criminological research (Thompson & Pickett, 2019). Some studies have used unmatched opt-in samples, such as Amazon Mechanical Turk (MTurk), which allow nearly any of a platform’s users to answer the online questionnaire (e.g., Barnum & Solomon, 2019; Herman & Pogarksy, 2020; Miethe et al., 2019; Pickett et al., 2013; Seigfried-Spellar et al., 2017). These are convenience samples (Landers & Behrend, 2015), but they provide a relatively inexpensive method of exploring many theories and ideas and have the potential to yield a diverse, nationwide group of respondents (Weinberg et al., 2014). The risk with surveying opt-in respondents, of course, is that the resultant findings may not generalize to any broader population. Recent research suggests that observational findings from such samples often fail to generalize (Thompson & Pickett, 2019), even though experimental results tend to be externally valid (Coppock, 2019; Mullinix et al., 2015).

Another option, however, and one that is quickly becoming the most commonly utilized online sampling methodology in both criminological and sociological studies (e.g., Enns and Ramirez, 2018; Filindra & Kaplan, 2017; Haner, Cullen, et al., 2019; Lee et al., 2020; Lehmann & Pickett, 2017; Schutten et al., 2020; Simmons, 2017; Socia et al., 2019), is matched opt-in samples, such as those from YouGov. The objective of sample matching is to increase generalizability by adjusting for variables that affect selection—a model-based instead of a design-based approach to inference (Ansolabehere & Rivers, 2013). Model-based inference depends on a host of assumptions, which, if met, allow researchers to render selection ignorable through modeling adjustments (Mercer et al., 2018).

Recently, Thompson and Pickett (2019) examined the generalizability of both univariate estimates and relational inferences from online crowdsourced samples, such as MTurk, and unmatched online opt-in samples, such as SurveyMonkey Audience. As expected, they found significant differences in sociodemographic characteristics and attitude prevalence between the online nonprobability samples and those of the GSS sample. They also found that the regression coefficients for predictors of four criminal justice attitudes—views on the death penalty, acceptance of police use of force, fear of crime, and spending on law enforcement—often differed significantly. Ultimately, Thompson and Pickett (2019) concluded that these types of nonprobability online samples produce regression results that are “normally in the same direction as the GSS coefficients, especially when they are statistically significant, but [that] differ considerably in magnitude” (Thompson & Pickett, 2019, p. 1).

Thompson and Pickett’s (2019) findings are elucidating, but they provide no information about matched opt-in samples, such as YouGov. One major drawback to obtaining matched samples is the larger price tag that accompanies them, relative to unmatched opt-in samples. Therefore, an important question is whether matched opt-in samples are more generalizable than unmatched online samples. Specifically, are matched opt-in samples more likely than unmatched samples to produce relational inferences comparable to high-response rate, probability-based surveys, such as the General Social Survey (GSS)? We address this research question in the current research note using a matched YouGov sample and an unmatched MTurk sample. Using the same methodology and framework as Thompson and Pickett (2019), our study examines how regression results in the YouGov and MTurk samples compare to those in the General Social Survey (GSS) for four criminal justice outcomes. In so doing, our analysis both replicates and extends Thompson and Pickett’s (2019) findings.

The Matched Sampling Design

To rely on sampling theory alone to make generalizations from a sample to a population, the sample must be drawn probabilistically from a sampling frame that has complete coverage (Groves et al., 2009). However, online opt-in sampling (e.g., MTurk, SurveyMonkey) is normally not random (Mercer et al., 2018), and never has complete or even near complete coverage—it only includes Internet users who are signed up on a specific survey platform (Baker et al., 2013; Callegaro et al., 2015). Thus, it risks bias in relationships if selection (S) into the opt-in sample is a collider variable between unmeasured (U) predictors of the dependent variable (Y) and the included regressors (X) (i.e., X → S ← U → Y), a situation Thompson and Pickett (2019, p. 5) term “confounded sampling.” (It also risks bias if the dependent variable causes selection.) The matched sample strategy seeks to eliminate bias due to confounded sampling by adjusting—through matching and weighting—for variables that influence selection, thus yielding a sample that approximates a simple random sample of the population (Ansolabehere & Rivers, 2013; Mercer et al., 2018).

In general, matched opt-in surveys (e.g., YouGov) use a multistage sampling process that involves: (1) building a synthetic sampling frame (SSF) from large, high-quality, probability-based samples (e.g., American Community Survey), which is intended to represent the target population, (2) using this SSF in conjunction with matching to select respondents from an existing pool of opt-in panelists so that the sampled panelists are similar to those in the SSF on a number of specific covariates (matching variables), and (3) using weighting (propensity scoring and poststratification) to adjust for imperfect matching (Ansolabehere & Rivers, 2013). In doing so, theoretically, sample selection bias is reduced or eliminated.

The specific assumption underlying matched sampling is one of conditional exchangeability (or ignorability)—specifically, that sample selection is unrelated to the outcome variable after conditioning on observables (Baker et al., 2013; Mercer et al., 2018). In other words, the assumption is that any relationship between sample selection and the outcome variable(s) is spurious, due to common causes of both, such as respondent demographics, and that adjusting for these confounders through matching and weighting renders selection independent of the outcome (Thompson & Pickett, 2019). The reasonableness of this assumption will vary across different outcomes, for two reasons. First, different outcomes will share different common causes with selection, and not all of those confounders may be observed. Second, some outcomes, especially those pertaining to altruism and social support, may have a causal effect on selection. Even if the assumption is correct, the adjustment (matching and weighting) variables still “must be the correct variables for ensuring conditional exchangeability, and the panel must be able to supply respondents that are close matches to each case in the SSF” (Mercer et al., 2018, p. 261). When the assumption is wrong, and the outcome causes selection, or when it is correct, but the adjustment process (matching, weighting, or statistical control) fails to condition on the correct confounders, survey estimates will be biased, because sample selection will be related to the outcome variable (Mercer et al., 2018; Thompson & Pickett, 2019).

The good news is that the matching methodology produces election polling estimates that are similar to other large probability-based samples (e.g., American National Election Survey [ANES]) and random digit dial phone surveys, which suggests that this methodology may address the ignorability concern sufficiently for those outcomes–and at a fraction of the cost of large probability-based surveys (Ansolabehere & Rivers, 2013; Vavreck & Rivers, 2008). Additionally, the PEW Research Center’s (Kennedy et al., 2016) analysis of eight online vendors finds that sophisticated sampling and weighting methods, such as those used by YouGov (Rivers, 2016), result in the “smallest average estimated bias” of the samples that were analyzed. However, as Baker et al. (2013, p. 95) caution, “no single set of covariates can be expected to correct all the bias in a full range of survey topics, and the number of covariates needed for a given survey may be quite large.” The issue is that the specific confounders—the variables related both to selection and to the outcome of interest—will vary depending on the survey topic (Thompson & Pickett, 2019). Simmons and Bobo (2015), for example, found evidence suggesting YouGov’s sample matching method eliminated bias in regression models predicting egalitarianism, but not in those predicting racial resentment.

Notably, researchers have used matched samples to explore a range of criminal justice attitudes, including, but not limited to, attitudes toward the death penalty and other punitive policies (Lehmann & Pickett, 2017; Norris & Mullinix, 2019; Simmons, 2017), gun control (Filindra & Kaplan, 2017; Haner, Cullen, et al., 2019), sex offenders (Social & Harris, 2016; Socia et al., 2019), specialty courts (Thielo et al., 2019), prison privatization (Enns & Ramirez, 2018), terrorism (Haner, Sloan, et al., 2019; Haner et al., 2020), school shootings (Lee et al., 2020; Schutten et al., 2020), police-community relations (McManus et al., 2019), police brutality (Graham et al., 2020), body-worn cameras (Graham et al., 2019), and offender redeemability (Burton et al., 2020). Still lacking, however, is evidence about whether sample matching effectively eliminates selection bias in opt-in samples for criminal justice attitudes. The current study addresses this research void. To our knowledge, our study is the first to examine the generalizability of regression results from matched opt-in samples in relation to criminal justice attitudes.

Methodology

Data

The GSS data in this study come from the 2018 sample because it was the most recent available. In order to collect a nationally representative sample of respondents, the GSS uses a multistage cluster sampling design, which selects participants from primary sampling units. These respondents complete face-to-face interviews, which are conducted by the National Opinion Research Center (NORC). In the 2018 GSS sample, 2,348 respondents were surveyed; however, the full analytic sample in this study ranges from N = 1,070 to N = 1,496, both because the GSS only asks roughly two-thirds of the sample the 200 “core” questions and because of item nonresponse.

The YouGov data were collected between March 7th and 16th, 2020. YouGov initially surveyed 1,023 respondents, who were matched to an SSR based on gender, age, race, and education. This SSR was constructed through stratified sampling from the 1-year American Community Survey sample (2017). Matching yielded a sample of 1,000 respondents, which was then weighted to the SSR using propensity scores, after which the weights were post-stratified. Overall, the analytic sample ranges between N = 599 to N = 719 due to item nonresponse and “don’t know” responses.

The MTurk data were collected between March 20th and 21st, 2020. Respondents were limited to those who had completed over 500 HITs (human intelligence tasks), had a 95% or higher approval rating, and were aged 18 or older in order to capture high-quality respondents (Peer et al., 2014). In total, 845 respondents completed the survey; however, the analytic sample ranges between N = 663 to N = 772 due to item nonresponse and “don’t know” responses.

Dependent Variables

In this study, we analyzed four dependent variables, which used the exact same wording and response options across all three samples. They measure (1) fear of crime, (2) law enforcement spending preferences, (3) global attitudes towards police use of force, and (4) death penalty support. The exact item wording and response options can be found in Table 1.

Table 1.

Dependent Variable Item Wording and Response Options.

GSS variable label	GSS item and response options
CAPPUN	Do you favor or oppose the death penalty for persons convicted of murder? • Favor • Oppose • Don’t know
FEAR	Is there any area right around here–that is, within a mile–where you would be afraid to walk alone at night? • No • Yes • Don’t know
POLHITOK	Are there any situations you can imagine in which you would approve of a policeman striking an adult male citizen? • No • Yes • Don’t know
NATCRIMY	We are faced with many problems in this country, none of which can be solved easily or inexpensively. Are we spending too much, too little, or about the right amount on law enforcement? • Too much • Too little • About right • Don’t know

Independent Variables

Based on previous public opinion research on crime and justice (Brown & Socia, 2017; Cochran & Sanders, 2009; Unnever & Cullen, 2010), sociodemographic and attitudinal predictors were included in these models. In accordance with Thompson and Pickett’s (2019) predictors, race (0 = Non-White, 1 = White), ethnicity (0 = Non-Hispanic/Latino, 1 = Hispanic/Latino), sex (0 = female, 1 = male), age (in years)¹, education (0 = less than a Bachelor’s degree, 1 = Bachelor’s degree or higher), and income were included. Income was measured on a 16-point scale ranging from 1 = “less than $10,000” to 16 = “$500,000 or more” and was treated continuously. Respondents’ political ideology was measured categorically (1 = Liberal to 3 = Conservative), and “Liberal” is the reference group in the models.

Finally, as well as being used as a dependent variable, fear of crime is used as an independent variable in the models examining policy attitudes, for two reasons. First, since at least the 1980s, theoretical and empirical scholarship on public opinion has considered fear of crime—under a utilitarian model of attitudes—to be a potential cause of punitiveness and support for aggressive policing (Baker et al., 2016; Kleck & Jackson, 2017; Langworthy & Whitehead, 1986; Silver & Pickett, 2015). Second, including fear as a predictor of policy attitudes replicates Thompson and Pickett (2019), increasing the comparability of our findings to theirs, and produces identical model specifications across the remaining dependent variables.

Analytic Strategy

For the analysis, the GSS and YouGov samples are weighted using the provided weights.² The MTurk sample is not weighted, because MTurk samples are rarely weighted in studies that use them (Mullinix et al., 2015) and because Thompson and Pickett (2019) found that weighting MTurk samples did not help. In line with Thompson and Pickett’s (2019) analysis, we examine the univariate statistics for the demographics and outcome variables of these three samples. Next, we estimate a total of four comparable models (three logistic regression models for each FEAR, POLHITOK, and CAPPUN, and one ordinal regression model for NATCRIMY) between the samples using all available predictors for the relevant dependent variable. Following Thompson and Pickett (2019), we treated “Don’t know” responses as missing values. Furthermore, in line with Thompson and Pickett’s (2019) methodology, and with methods used in other studies (Open Science Collaboration, 2015), we compared coefficients’ directions, whether the opt-in sample’s coefficient falls within the GSS’s 95% confidence interval, and vice versa, and whether the coefficients in the samples differ significantly from each other. To test whether coefficients differ significantly, we used the Paternoster et al. (1998) method.³

Results

The descriptive statistics for the samples can be seen in Table 2. Like Thompson and Pickett (2019), we find that the demographic characteristics of the opt-in samples differ from those of the probability sample. However, our study shows that the differences are much smaller for the matched sample. The YouGov sample is significantly older (+2.32 mean years), has a higher income (+.26 mean), is more conservative (+5.71%), and has fewer Whites (−4.82%), Hispanics (−3.08%), and college graduates (−4.68%) than the GSS sample. But these differences pale in comparison to those between the MTurk and GSS samples, which are both more numerous and far larger in size. Compared to the GSS, the MTurk respondents are much younger (−8.77 mean years), much more likely to be male (+12.67%), much more likely to be college graduates (+24.08%), much less likely to be Hispanic (−10.74%), and much more liberal (+21.91%). They also have a significantly a lower average income (−1.17 mean) and are more likely to be White (+3.83%).

Table 2.

Descriptive Statistics.

	GSS^a		YOUGOV^a		MTURK
	Mean or %	SD	Mean or %	SD	Mean or %	SD
LE spending	2.43	0.68	2.36**	0.72	2.15***	0.76
Less (%)	10.74	–	14.16**	–	22.44***	–
Same (%)	35.22	–	35.68	–	40.49**	–
More (%)	54.04	–	50.16*	–	37.07***	–
Fear of crime (%)	33.28	–	40.84***	–	37.29*	–
Use of force (%)	64.01	–	66.44	–	68.99**	–
Death penalty (%)	63.56	–	65.21	–	48.05***	–
White (%)	72.38	–	67.56**	–	76.21**	–
Hispanic/Latino (%)	15.83	–	12.75*	–	5.09***	–
Male (%)	45.54	–	48.53	–	58.21***	–
Age	46.62	17.71	48.93***	18.17	37.85***	10.62
Bachelor’s degree (%)	33.08	–	28.40**	–	57.16***	–
Income	5.44	3.44	5.70*	3.53	4.27***	1.60
Ideology	2.03	0.79	2.07	0.83	1.74***	0.84
Liberal (%)	29.28	–	31.41	–	51.19***	–
Moderate (%)	38.24	–	30.39***	–	23.15***	–
Conservative (%)	32.48	–	38.19**	–	25.66***	–

SD = standard deviation.

Weighted data.

p < .05. **p < .01. ***p < .001 (two-tailed) for difference from GSS.

Similar to Thompson and Pickett (2019), we also find significant univariate differences in criminal justice attitudes between the GSS and opt-in samples. Once again, however, the differences are generally larger for the unmatched than the matched sample. The opt-in respondents wanted to spend less on law enforcement (YouGov, +3.42%; MTurk, +11.7%), were more afraid of crime (YouGov, +7.56%; MTurk, +4.01%), were more supportive of police use of force (YouGov, +2.44%; MTurk, +4.89), and diverged in death penalty support (YouGov, +1.65%; MTurk, −15.51%). With one exception (fear of crime), the differences for the MTurk sample are more than twice as large as for the YouGov sample.

Turning to the multivariate analyses, Figure 1 provides the odds ratios (OR) and confidence intervals for the models estimating fear of crime in the GSS, YouGov, and MTurk samples. Compared to the GSS sample, the YouGov sample has six coefficients in the same direction, and two in the opposite direction; the MTurk sample has seven coefficients in the same direction and one in the opposite direction. Two of the YouGov coefficients (for Hispanic and Male), but three of the MTurk coefficients (for Hispanic, Male, and Moderate), fall outside of the respective GSS confidence interval. Only one YouGov confidence interval (for Male) fails to capture the respective GSS coefficient, but the MTurk sample has two failures to capture (for Male and Moderate). None of the YouGov and GSS coefficients differ significantly, per the Paternoster et al. (1998) test, but one MTurk coefficient (for Male) is significantly different from its GSS counterpart. Thus, the YouGov sample slightly outperforms the MTurk sample for this particular outcome variable, fear of crime.

Figure 1.

Odds ratios and 95% confidence intervals for logistic regression: FEAR.

Figure 2 presents the OR results for support for law enforcement spending. Compared to the GSS results, seven of the YouGov coefficients are in the same direction, but only six are in the MTurk sample. Only one YouGov coefficient (for Income) falls outside the GSS confidence interval, but three do in the MTurk sample (for White, Male, and Education). None of the YouGov confidence intervals exclude the respective GSS coefficient, but three of the MTurk confidence intervals do. Per the Paternoster et al. (1998) test, none of the coefficients differ significantly between the YouGov and GSS samples. Between the MTurk and GSS samples, however, two coefficients (for White and Male) differ significantly. Thus, the YouGov sample outperforms the MTurk sample for this outcome variable, spending preferences.

Figure 2.

Odds ratios and 95% confidence intervals for ordinal regression: NATCRIMY.

Figure 3 provides the OR estimates for acceptance of police use of force. Compared to the GSS results, eight of the YouGov coefficients, but only four of the MTurk coefficients, are in the same direction. Only three YouGov coefficients (for White, Age, and Income) fall outside the GSS confidence intervals, but six MTurk coefficients do (for White, Hispanic, Age, Education, Income, and Fear). Similarly, twice as many MTurk confidence intervals fail to capture the respective GSS coefficient compared to the YouGov confidence intervals. The difference in coefficients is statistically significant for three variables (White, Age, and Income) in the YouGov sample, but is significant for five variables (White, Hispanic, Education Income, and Fear) in the MTurk sample. For this outcome, then, the YouGov sample does almost twice as well as the MTurk sample.

Figure 3.

Odds ratios and 95% confidence intervals for logistic regression: POLHITOK.

Figure 4 provides the OR estimates for capital punishment attitudes. Compared to the GSS sample, the YouGov sample has five coefficients in the same direction and four in the opposite direction, whereas the MTurk sample has four coefficients in the same direction and five in the opposite direction. Four of the YouGov coefficients (White, Hispanic, Age, and Conservative) fall outside of the GSS confidence interval, whereas six of the MTurk coefficients do (White, Hispanic, Male, Education, Conservative, and Fear). In the YouGov sample, only one confidence interval (for Conservative) fails to capture the respective GSS coefficient, whereas there are five failures to capture in the MTurk sample (for White, Male, Education, Conservative, and Fear). Per the Paternoster et al. (1998) test, none of the YouGov coefficients differ significantly from its GSS counterpart, but one of the MTurk coefficients did (Education). Overall, though, the YouGov sample slightly outperforms the MTurk sample for this specific outcome variable, death penalty support.

Figure 4.

Odds ratios and 95% confidence intervals for logistic regression: CAPPUN.

Table 3 summarizes all of the regression results from all of the models. The results are quite clear about which opt-in sample is preferable and are consistent no matter which comparison we focus on. Let us start with the directional findings. The YouGov coefficients are more likely than the MTurk coefficients (71.43% vs. 60.00%) to be in the same direction as the GSS coefficients. The YouGov advantage remains when we just look at the GSS coefficients that are statistically significant: 86.67% of the YouGov coefficients are in the same direction, compared to 66.67% of MTurk coefficients. And it remains when we just look at the coefficients that are statistically significant in the opt-in samples: 100% of the significant YouGov coefficients are in the same direction as the GSS coefficients, compared to 80% of the MTurk coefficients.

Table 3.

Summary of Findings.

All estimates	Matched (YouGov) sample	Unmatched (MTurk) sample
Opt-in and GSS coefficients are in the same direction	71.43% (25/35)	60.00% (21/35)
Opt-in coefficient is outside GSS confidence interval	28.57% (10/35)	51.43% (18/35)
Opt-in confidence interval excludes the GSS coefficient	17.14% (6/35)	42.86% (15/35)
Confidence interval discrepancy AND coefficients differ significantly	8.57% (3/35)	25.71% (9/35)
Statistically significant estimates
GSS coefficient is significant AND Opt-in coefficient is in same direction	86.67% (13/15)	66.67% (10/15)
Opt-in coefficient is significant AND the GSS coefficient is in same direction	100% (10/10)	80.00% (8/10)

What about the magnitude of the coefficients? Which opt-in sample produces coefficients that are more similar in size to those in the probability sample? The advantages of the matched YouGov sample are even larger here. The YouGov coefficients are more likely than the MTurk coefficients (71.43% vs. 51.43%) to fall within the respective GSS confidence interval. The YouGov confidence intervals are also more likely than the MTurk confidence intervals (82.86% vs. 42.86%) to capture the respective GSS coefficient. Perhaps most notably, the YouGov coefficients are less likely than the MTurk coefficients (8.57% vs. 25.74%) to differ significantly from the GSS coefficients. Put another way, the MTurk coefficients are more than three times as likely as the YouGov coefficients to be significantly larger or smaller than the GSS coefficients. We also compared the absolute differences between the GSS coefficients and those in the two opt-in samples. Once again, the matched YouGov sample did better than the unmatched MTurk sample. The average absolute difference was 36% larger in MTurk than YouGov, indicating that the matched YouGov sample produced coefficients more similar in size to those in the GSS.

Finally, we examined how the absolute differences varied across the dependent and independent variables. Two notable findings emerged. First, among the dependent variables, the absolute differences were largest by far for POLHITOK. This was true in both MTurk and YouGov, and suggests that opt-online samples may provide particularly inaccurate estimates of the correlates of attitudes toward police use of force. Second, in both opt-in samples, the absolute differences in coefficients across samples were especially large for race and/or ethnicity. This is highly consistent with past research. Kennedy et al. (2016, p. 4), for example, compared findings across nine opt-in panels to those from a probability sample. They found that: (1) univariate “estimates for Hispanics and blacks show the largest biases of all subgroups” (p. 15), and (2) “marginal effects [multivariate estimates] associated with race and ethnicity are rarely correct” (p. 22). The explanation is not so much that Blacks and Hispanics are underrepresented in opt-in panels, although they are, but that the Blacks and Hispanics who do join these panels are different in unknown ways from those who do not. The strong implication is that “researchers using online nonprobability samples are at risk of drawing erroneous conclusions about the effects associated with race and ethnicity” (Kennedy et al., 2016, p. 4).⁴

Conclusion

This study examined the generalizability of criminal justice attitudes from matched and unmatched opt-in samples. Compared to the GSS, a high-response rate probability sample, the descriptive statistics and the prevalence of attitudes were more similar in the matched opt-in sample (YouGov) than in the unmatched sample (MTurk). The regression coefficients in the matched opt-in sample (YouGov) were almost always in the same direction as the GSS coefficients, especially when statistically significant, and were often also of a similar magnitude. Less than 10% of the coefficients in the YouGov sample differed significantly from those in the GSS sample. By contrast, over 25% of the MTurk coefficients differed significantly from the GSS coefficients. Additionally, in a variable-by-variable comparison of coefficients, the matched sample’s coefficients were closer than the unmatched sample’s to the GSS’s about 63% of the time. The key takeaway is that criminologists should use matched rather than unmatched opt-in samples whenever probability samples are not feasible.

Our study is not without limitations. To assess generalizability, we compared our opt-in samples to the GSS, but research shows that the GSS itself differs from the U.S. Census’s Current Population Survey (Simmons & Bobo, 2015).⁵ Other studies find that when compared against each other, there often are as many inconsistencies in results between different probability samples as between opt-in samples and probability samples (Ansolabehere & Schaffner, 2014; Bhutta, 2012). Thus, future research should replicate our study using multiple probability samples from different sources, not just from the GSS. Additionally, we only explored four criminal justice attitudes. It is possible that our findings would differ if we focused on other types of criminal justice attitudes, such as attitudes toward offender redeemabilty, body-worn cameras, or gun control. Future research should thus replicate our analysis using other attitudinal measures to further examine the utility of matched opt-in samples in criminal justice research. Finally, we focused on observational relational inferences, but research shows that opt-in samples perform better in experiments (Coppock, 2019; Mullinix et al., 2015), because of low levels of effect heterogeneity (Coppock et al., 2018). Although opt-in samples as a whole may be better suited for experiments, we would still expect matched opt-in samples to outperform unmatched samples in experimental studies as well, given their greater descriptive and attitudinal similarity to the general population. Future research should test this possibility.

In conclusion, our study replicates Thompson and Pickett’s (2019) by showing that the magnitude of coefficients in unmatched opt-in samples is often off, and extends their analysis by showing that findings from matched opt-in samples are more generalizable. In every comparison (Table 3), we found that the matched sample outperformed the unmatched sample. The coefficients in the matched sample were more likely to be in the same direction as the probability sample and were also more likely to be of a similar size. In fact, our findings suggest that researchers using matched opt-in samples to explore criminal justice attitudes would obtain coefficients that differ significantly from those in probability samples less than 10% of the time. Thus, matched opt-in samples provide a relatively inexpensive data source for criminal justice researchers, compared to probability samples, and also appear to carry a smaller generalizability penalty than unmatched samples.

Supplemental Material

sj-pdf-1-cad-10.1177_0011128720977439 – Supplemental material for Advantages of Matched Over Unmatched Opt-in Samples for Studying Criminal Justice Attitudes: A Research Note

Supplemental material, sj-pdf-1-cad-10.1177_0011128720977439 for Advantages of Matched Over Unmatched Opt-in Samples for Studying Criminal Justice Attitudes: A Research Note by Amanda Graham, Justin T. Pickett and Francis T. Cullen in Crime & Delinquency

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Amanda Graham

Supplemental Material

Supplemental material for this article is available online.

Notes

Author Biographies

Amanda Graham is an assistant professor in the Department of Criminal Justice and Criminology at Georgia Southern University. Her research interests focus on policing, fear of police brutality, the impact of race in police-community relationships, police legitimacy, scale development, and measurement.

Justin T. Pickett is an associate professor in the School of Criminal Justice at SUNY-Albany. He is the 2015 recipient of the American Society of Criminology’s Ruth Shonle Cavan Young Scholar Award. His research interests include survey research methods, public opinion, police-community relations, and theories of punishment.

Francis T. Cullen is a Distinguished Research Professor Emeritus and senior research associate in the School of Criminal Justice at the University of Cincinnati. His current research interests include redemption as a correctional theory, the impact of racial beliefs on public policy opinion, and the criminology of Donald Trump.

References

Ansolabehere

Rivers

(2013). Cooperative survey research. Annual Review of Political Science, 16, 307–329.

Ansolabehere

Schaffner

B. F.

(2014). Does survey mode still matter? Findings from a 2010 multi-mode comparison. Political Analysis, 22(3), 285–303.

Baker

Brick

J. M.

Bates

N. A.

Battaglia

Couper

M. P.

Dever

J. A.

Tourangeau

(2013). Summary report of the AAPOR task force on non-probability sampling. Journal of Survey Statistics and Methodology, 1(2), 90–143.

Baker

Cleary

H. M. D.

Pickett

J. T.

Gertz

M. G.

(2016). Crime salience and public willingness to pay for child saving and juvenile punishment. Crime & Delinquency, 62(5), 645–668.

Barnum

T. C.

Solomon

S. J.

(2019). Fight or flight: Integral emotions and violent intentions. Criminology, 57(4), 659–686.

Bhutta

C. B.

(2012). Not by the book: Facebook as a sampling frame. Sociological Methods & Research, 41(1), 57–88.

Brame

Paternoster

Mazerolle

Piquero

(1998). Testing the equality of maximum-likelihood regression coefficients between two independent equations. Journal of Quantitative Criminology, 14(3), 245–261.

Brown

E. K.

Socia

K. M.

(2017). Twenty-first century punitiveness: Social sources of punitive American views reconsidered. Journal of Quantitative Criminology, 33(4), 935–959.

Burton

A. L.

Cullen

F. T.

Burton

V. S.

Jr. Graham

Butler

L. C.

Thielo

A. J.

(2020). Belief in redeemability and punitive public opinion: “Once a criminal, always a criminal” revisited. Criminal Justice and Behavior, 47(6), 712–732.

10.

Callegaro

Manfreda

K. L.

Vehovar

(2015). Web survey methodology. Sage.

11.

Cochran

J. K.

Sanders

B. A.

(2009). The gender gap in death penalty support: An exploratory study. Journal of Criminal Justice, 37(6), 525–533.

12.

Coppock

(2019). Generalizing from survey experiments conducted on mechanical turk: A replication approach. Political Science Research Methods, 7(3), 613–628.

13.

Coppock

Leeper

T. J.

Mullinix

K. J.

(2018). Generalizability of heterogenous treatment effect estimates across samples. PNAS, 115(49), 12441–12446.

14.

Enns

P. K.

Ramirez

M. D.

(2018). Privatizing punishment: Testing theories of public support for private prison and immigration detention facilities. Criminology, 56(3), 546–573.

15.

Filindra

Kaplan

(2017). Testing theories of gun policy preferences among Blacks, Latinos, and Whites in America. Social Science Quarterly, 98(2), 413–428.

16.

Graham

Haner

Sloan

M. M.

Cullen

F. T.

Kulig

T. C.

Jonson

C. L.

(2020). Race and worrying about police brutality: The hidden injuries of minority status in America. Victims & Offenders, 15(5), 549–573.

17.

Graham

McManus

H. D.

Cullen

F. T.

Burton Jr Jonson

C. L.

(2019). Videos don’t lie: African Americans’ support for body-worn cameras. Criminal Justice Review, 44(3), 284–303.

18.

Groves

R. M.

Fowler

F. J.

Couper

M. P.

Lepkowski

Singer

Tourangeau

(2009). Survey methodology (2nd ed.). Wiley.

19.

Haner

Cullen

F. T.

Lero Jonson

Burton

A. L.

Kulig

T. C.

(2019). Price of liberty or never again: Americans’ views on preventing mass murder. Justice Evaluation Journal, 2(1), 50–72.

20.

Haner

Sloan

M. M.

Cullen

F. T.

Graham

Lero Jonson

Kulig

T. C.

Aydın

. (2020). Making America safe again: Public support for policies to reduce terrorism. Deviant Behavior. Advance online publication. https://doi.org/10.1080/01639625.2020.1738638

21.

Haner

Sloan

M. M.

Cullen

F. T.

Kulig

T. C.

Lero Jonson

(2019). Public concern about terrorism: Fear, worry, and support for anti-Muslim policies. Socius, 5, 1–16.

22.

Herman

Pogarsky

(2020). Morality, deterrability, and offender decision making. Justice Quarterly, 2020, 1–25, Advance online publication. https://doi.org/10.1080/07418825.2019.1709884

23.

Kennedy

Mercer

Keeter

Hatley

McGeeney

Gimenez

(2016, May 2). Evaluating online nonprobability surveys. Pew Research Center. https://www.pewresearch.org/methods/2016/05/02/evaluating-online-nonprobability-surveys/

24.

Kleck

Jackson

D. B.

(2017). Does crime cause punitiveness? Crime & Delinquency, 63(12), 1572–1599.

25.

Landers

R. N.

Behrend

T. S.

(2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology, 8(2), 142–164.

26.

Langworthy

R. H.

Whitehead

J. T.

(1986). Liberalism and fear as explanations of punitiveness. Criminology, 24(3), 575–591.

27.

Lee

Pickett

J. T.

Burton

A. L.

Cullen

F. T.

Johnson

C. L.

Burton

V. S.

(2020). Attributions as anchors: How the public explains school shootings and why it matters. Justice Quarterly. Advance online publication. https://doi.org/10.1080/07418825.2020.1769710

28.

Lehmann

P. S.

Pickett

J. T.

(2017). Experience versus expectation: Economic insecurity, the Great Recession, and support for the death penalty. Justice Quarterly, 34(5), 873–902.

29.

McManus

H. D.

Graham

Cullen

F. T.

Burton

V. S.

Jr. Jonson

C. L.

(2019). Friend not foe? Reconsidering race, the police, and community relations. Race and Justice. Advance online publication. https://doi.org/10.1177/2153368719849486

30.

Mercer

Lau

Kennedy

(2018). For weighting online opt-in samples, what matters most. Pew Research Center.

31.

Miethe

T. D.

Venger

Lieberman

J. D.

(2019). Police use of force and its video coverage: An experimental study of the impact of media source and content on public perceptions. Journal of Criminal Justice, 60, 35–46.

32.

Mullinix

K. J.

Leeper

T. J.

Druckman

J. N.

Freese

(2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2(2), 109–138.

33.

Norris

R. J.

Mullinix

K. J.

(2019). Framing innocence: An experimental test of the effects of wrongful convictions on public opinion. Journal of Experimental Criminology. Advance online publication. https://doi.org/10.1007/s11292-019-09360-7

34.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

35.

Paternoster

Brame

Mazerolle

Piquero

(1998). Using the correct statistical test for the equality of regression coefficients. Criminology, 36(4), 859–866.

36.

Peer

Vosgerau

Acquisti

(2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46(4), 1023–1031.

37.

Pickett

J. T.

Mancini

Mears

D. P.

(2013). Vulnerable victims, monstrous offenders, and unmanageable risk: Explaining public opinion on the social control of sex crime. Criminology, 51(3), 729–759.

38.

Rivers

(2016, May 13). Pew research: YouGov consistently outperform competitors on accuracy. YouGov. Retrieved from https://today.yougov.com/topics/finance/articles-reports/2016/05/13/pew-research-yougov

39.

Schutten

N. M.

Pickett

J. T.

Burton

A. L.

Cullen

F. T.

Jonson

C. L.

Burton

V. S.

Jr. (2020). Punishing rampage: Public opinion on sanctions for school shooters. Justice Quarterly. Advance online publication. https://doi.org/10.1080/07418825.2019.1707857

40.

Seigfried-Spellar

K. C.

Villací-Vukadinović

Lynam

D. R.

(2017). Computer criminal behavior is related to psychopathy and other antisocial behavior. Journal of Criminal Justice, 51, 67–73.

41.

Simmons

A. D.

(2017) Cultivating support for punitive criminal justice policies: News sectors and the moderating effects of audience characteristics. Social Forces, 96(1), 299–328

42.

Simmons

A. D.

Bobo

L. D.

(2015). Can non-full-probability internet surveys yield useful data? A comparison with full-probability face-to-face surveys in the domain of race and social inequality attitudes. Sociological Methodology, 45(1), 357–387

43.

Silver

J. R.

Pickett

J. T.

(2015). Toward a better understanding of politicized policing attitudes: Conflicted conservatism and support for police use of force. Criminology, 53(4), 650–676.

44.

Social

K. M.

Harris

A. J.

(2016). Evaluating public perceptions of the risk presented by registered sex offenders: Evidence of crime control theater? Psychology, Public Policy, and Law, 32(4), 375–385.

45.

Socia

K. M.

Rydberg

Dum

C. P.

(2019). Punitive attitudes toward individuals convicted of sex offenses: A vignette study. Justice Quarterly. Advance online publication. https://doi.org/10.1080/07418825.2019.1683218

46.

Thielo

A. J.

Cullen

F. T.

Burton

A. L.

Moon

M. M.

Burton

V. S.

Jr. (2019). Prisons or problem-solving: Does the public support specialty courts? Victims & Offenders, 14(3), 267–282.

47.

Thompson

A. J.

Pickett

J. T.

(2019). Are relational inferences from crowdsourced and opt-in samples generalizable? Comparing criminal justice attitudes in the GSS and five online samples. Journal of Quantitative Criminology. Advance online publication. https://doi.org/10.1007/s10940-019-09436-7

48.

Unnever

J. D.

Cullen

F. T.

(2010). The social sources of Americans’ punitiveness: A test of three competing models. Criminology, 48(1), 99–129.

49.

Vavreck

Rivers

(2008). The 2006 cooperative congressional election study. Journal of Elections, Public Opinion and Parties, 18(4), 355–366.

50.

Weinberg

J. D.

Freese

McElhattan

(2014). Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociological Science, 1, 292–360.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.14 MB