Why Getting Inside the “Black Box” Is Important

Abstract

We propose that the causes of the effectiveness (or ineffectiveness) of police interventions can be better understood with an increased focus on the measurement of treatment implementation and outputs, as opposed to the more common “black box” conceptualizations of police interventions and outcome-only evaluations used in most experimental studies. We present findings from a randomized, experimental evaluation of broken windows policing at hotspots in three California cities. Our analysis suggests that variation in the treatment delivered to target street segments within and between cities limited the ability of the study to detect potential treatment impacts and was due in part to the failure of the police agencies to take ownership of the science of the intervention.

Keywords

broken windows experimental methods evaluation research hot spots policing

There is now ample evidence that increased police patrol directed at hotspots is more effective in reducing crime and disorder than random, undirected (routine preventive) patrol (Braga, Papachristos, & Hureau, 2014; Weisburd & Eck, 2004). However, two issues remain less clear. First, we know little about the causes of effectiveness, or conversely ineffectiveness. Aside from increased police presence (dosage and frequency) or the use of problem-solving approaches, which specific police actions are most effective in reducing crime and disorder and under what conditions? Second, what actions should police leaders take to “create and maintain the causes” of successful strategies (Sherman et al., 2014, p. 96)?

In this article, we address two obstacles that have impeded the progress being made in understanding both of these issues: (a) the research and evaluation methodologies being used to study police interventions and (b) the lack of “ownership of police science” in police agencies (Neyroud & Weisburd, 2014; Weisburd & Neyroud, 2011). We suggest that through addressing the first obstacle, police academics and researchers can better ascertain the extent to which the second may have impacted the effectiveness of an evaluated strategy. In turn, an increased focus on eliminating these obstacles is necessary to increase our knowledge about both how police strategies are effective or ineffective, and how police can implement interventions in ways that they are likely to have impact.

We begin by briefly discussing a common shortcoming of police evaluation research. We then present findings from a randomized, experimental evaluation of broken windows policing at hotspots (Weisburd, Hinkle, Famega, & Ready, 2011) to illustrate how the research methodology for evaluations of police interventions is important in assessing whether the police really took ownership of an intervention.

Getting Inside the “Black Box” in Policing Experiments

In 1998, Lawrence Sherman advocated for evidence-based policing—that police practices should be based on scientific evidence about what works best. He proposed that more randomized experiments examining the outcomes of police work (e.g., crime rates, repeat victimization, etc.) were needed, as well as ongoing outcomes research about the results achieved by applying (or ignoring) basic research in practice (Sherman, 1998). Today, randomized, experimental designs have generally become accepted as the best method for generating valid evidence about the effectiveness of criminal justice programs, practices, and strategies (Weisburd & Hinkle, 2012). The statistical benefits of randomizing subjects or places into treatment and control groups eliminates the possibility of confounding variables, as everything affecting the outcomes of interest aside from the intervention should vary randomly across the two groups (Boruch, McSweeny, & Soderstom, 1978; Campbell & Stanley, 1963; Riecken et al., 1974; Weisburd, 2003).

However, a challenge of both experimental and non-experimental designs, is ensuring the treatment is delivered as designed in terms of dosage (e.g., total time of police presence), frequency (e.g., number of times police come and go), and outputs (the achievement of specific treatment activities; see Sherman et al., 2014). Clearly, police failure to take ownership of the science of an intervention poses a significant threat to the internal validity of evaluations. This may manifest through implementation failure or treatment decay. Key personnel may be shifted, or not shifted as anticipated, and personnel involved in treatment delivery may resist or lose interest in the intervention (Copes & Vieraitis, 2005). A more complex threat is treatment variability across subjects or areas (e.g., beats, hotspots, etc.). Police interventions frequently proscribe a combination of multiple tactics and activities. The personnel administering the treatment may employ these tactics in varying degrees across areas, or not at all in some areas. The potential for treatment variability increases when multiple sites are used to evaluate an intervention (e.g., more than one police department). Despite extensive literature recognizing the critical importance of assessment, monitoring, and enhancement of treatment fidelity in other fields (e.g., health behavior intervention research, see Bellg et al., 2004); and even the development of a measure to evaluate treatment fidelity (for health behavior interventions, see Borrelli et al., 2005) it has been somewhat overlooked in the policing research field.

Treatment decay and variability are difficult to adjust for statistically even in experimental evaluations, as using multivariate models to account for variation in treatment reintroduces the possibility of omitting confounding variables and thus loses the benefits gained from randomization. Such models can be used as sensitivity analyses, as will be done in the current study (see also Hinkle, Weisburd, Famega, & Ready, 2014), but much caution is needed if attempting to inform policy from such analyses. In the best case scenario, a watered down intervention will still have some impact on the outcome measures and suggest the strategy is promising. But if no treatment effect is found, little is gained if there is no way to examine if it is a result of the designed treatment being ineffective (theory failure), or the failure to implement the treatment as designed (Lipsey, Petrie, Weisburd, & Gottfredson, 2006). Therefore, while randomized experiments are important to examine the impact of police interventions on outcomes, evidence-based policing also necessitates evaluation of treatment implementation and outputs in all research designs to avoid erroneously concluding a strategy did not work (Weisburd, Lum, & Yang, 2003). Despite this, researchers evaluating police interventions have frequently conceptualized the intervention as a “black box” with little examination of exactly what the police treatment entailed (Lipsey et al., 2006) or the outputs that were delivered.

Even randomized policing experiments with perfectly implemented interventions can only tell us whether the overall strategy had an impact on the outcome measures when the treatment is conceptualized as a black box in analyses. As others have noted, criminology is largely past only wanting to know whether a strategy works in reducing crime (Pawson & Tilley, 1997; Rosenbaum, 1988); the more functional focus is on how interventions do or do not work. This is to say that research needs to focus on actual actions taken by police, rather than just assuming the intervention went as planned and fit the style of policing being evaluated. This can be an involved question to address, as innovations in policing over the last several decades (e.g., community, problem-oriented, broken windows, zero-tolerance, and pulling levers policing) have not been easily defined, simple strategies with straightforward instructions for implementation. To the contrary, several have eluded concise, consistent definitions, and are more accurately philosophies than strategies, and thus many different types of police activities have been evaluated under the same black box umbrellas.

Whether necessitated by vague treatment design, lack of output measures, or manuscript word limits, the analyses presented are often simple models with variables for treatment, any blocking factors (for block-randomized studies) and crime or disorder as the outcome. Although methodologically appropriate (since nothing else needs to be controlled for in randomized experiments as outlined earlier), with complex innovations that prescribe multiple police actions, it is difficult to assess how the outcomes were produced (see Boruch, 2010; Lipsey et al., 2006) without data on the outputs delivered. This in turn makes it difficult for police leaders to assess what actions need to be taken to create and maintain the causes of successful strategies. External validity is also limited when treatment is conceptualized as a black box in design or analyses. The generalizability of results cannot be reasonably ascertained without detailed explanations of treatment and measures of outputs. Such a description is essential if a program is to be replicated at other sites or implemented more broadly (Lipsey et al., 2006).

The Current Study

The current study uses data from an experimental evaluation of broken windows policing at hotspots (Weisburd et al., 2011, 2012) to illustrate the importance of examining detailed data on treatment implementation and outputs. The experiment was conducted for approximately 6 months, in three southern California cities all located between 55 and 65 miles east of Los Angeles. Based on population estimates from the 2008 Uniform Crime Report, City A is the largest of the cities with an estimated population of 172,543 over 50 square miles. At the time of the experiment, the police department employed 218 sworn officers. City B’s population of 70,730 is spread over 36 square miles, with a police force of 78 sworn officers. The smallest of the cities, City C, covers 16 square miles with a population of 51,177, and a police force of 74 sworn officers. According to the 2010 Census, the majority populations of both City A and City C are Hispanic or Latino (69% and 71%, respectively); while City B’s population is only 30% Hispanic or Latino (73% White). The median household incomes, 2008 to 2012, vary considerably across the three cities: City C-$41,496, City A-$54,994, and City B-$66,901 (United States Census Bureau, 2010).

Selection of Random Assignment Hotspots

For this study, hotspots were defined as street segments (the two block faces on both sides of a street, including both of the intersections that are connected to the street block) with 10 or more disorder calls for service¹ and 3 or more Uniform Crime Reporting Part 1 crime calls for service the year before the experiment began. Disorder calls for service included all calls regarding prostitution, drug possession, disturbing the peace, vandalism, public drinking, misdemeanor DUI, noise complaints, fights, and thefts from automobiles. Part 1 crime calls for service included the FBI defined Part 1 offenses, excluding thefts from autos, which were included as a disorder for site selection purposes. Study segments also had to be isolated from one another by at least one full segment in every direction. These criteria, and consulting with police leaders about how many street segments they could adequately treat, returned a total of 110 street segments across the three cities—60 in City A, 30 in City B, and 20 in City C. A block-randomized design approach was used to randomly allocate street segments in equal numbers to the experimental and control conditions within each city (see Bloom, 2005; Sherman & Weisburd, 1995). Both the treatment and control groups consisted of 55 street segments—30 street segments in City A, 15 in City B, and 10 street segments in City C. Table 1 presents the average disorder calls for service on the treatment and control segments for a year prior to the intervention. Two segments were dropped due to being extreme outliers, leaving a sample of 108 street segments.

Table 1.

Preintervention Disorder Calls for Service on Study Street Segments.

	Mean disorder	SD
Treatment (n = 54)	14.67	9.19
Control (n = 54)	15.30	13.63

Treatment

A central concern was to develop a police intervention that was “true” to the notion of broken windows policing as developed by Wilson and Kelling in 1982. They suggested that broken windows policing should lead to crime prevention through the following mechanism. Police-led reductions in disorder should reduce fear of crime and community withdrawal among residents, and in time, help restore community controls. This should leave the community less vulnerable to crime, as an orderly community is less inviting to criminals than a disorderly area where offenders may perceive that their chances of arrest are lower. Conceptualizing a police intervention true to the broken windows thesis was a more difficult task than with many other policing strategies as Wilson and Kelling never provided detailed guidelines regarding what broken windows policing should look like in practice. Past studies of broken windows policing have used misdemeanor arrests (see e.g., Kelling & Sousa, 2001) or misdemeanor convictions (see Worrall, 2002) as proxies of police activity in relation to the broken windows theory. However Wilson and Kelling (1982) spent a great deal of time discussing that police must work with people in the community to negotiate consensus on behavior, and issuing citations and making arrests for every law violation were not suggested as the methods to achieve that result. In particular, Wilson and Kelling discussed that police should talk to people they find involved in activities such as public drinking about why the behavior is not allowed, and rely on warnings (or asking individuals to keep the behavior confined to a back alley out of site, etc.) rather than relying on law enforcement for every minor crime or disorder they encounter. In a later work with Catherine Coles, Kelling further illustrated that his conception of broken windows policing was not equivalent to zero tolerance policing, stating that,

in fact, the ideas presented in “Broken Windows” were antithetical to the use of “streetsweeping” tactics targeted on “undesirables”; rather, they advocated close collaboration between the police and citizens, including street people, in the development of neighborhood standards. Moreover, neighborhood rules were to be enforced for the most part through non-arrest approaches—education, persuasion, counseling, and ordering—so that arrest would only be resorted to when other approaches failed. (Kelling & Coles, 1996, pp. 22–23)

As such, the “broken windows” policing style that officers were trained in emphasized three key aspects. First, that officers pay attention to and consistently address social and physical disorder problems, no matter how small the violation. Second, that officers deal with social disorders using an order maintenance approach or through informal police actions such as polite reminders, mediating, counseling, verbal warnings or advising or a firm “knock it off,” unless persistent or defiant conduct or a preexisting arrest warrant justify an escalated response of citation or arrest. Third, that officers address physical disorders by notifying the appropriate agency for cleanup or repair as quickly as possible, and following up to make sure the problem was dealt with in a timely manner. Officers were also provided with a short treatment protocol book which outlined the designed policing style in detail, including several example scenarios.²

Dosage and Frequency

The research and police intervention teams agreed upon a treatment dosage of 180 minutes of broken windows style policing at each of the 55 treatment segments for each week of the approximately 30 week experiment. The research team emphasized in the initial training session, and multiple refresher trainings conducted during the intervention period, that officers should make an effort to visit each segment multiple times each week, for short periods of time, and especially at times of the day or night when the likelihood of observing social disorders would be higher.

To monitor the dosage, frequency, and nature of the treatment being delivered, officers were instructed to complete a log form³ for each visit to a treatment segment, regardless of how long they were present on the street segment, and even if they did not observe any disorders nor take any actions beyond police presence. The log forms recorded the following: the date of the visit, name of the street segment, city, officer ID number(s), the time of officer arrival and departure from the segment, social and physical disorders observed by the officer, and actions taken by the officer. Officers in two-officer units were required to submit only one log form with both officer IDs to prevent duplication and artificially inflating the intervention dosage. The log forms were scheduled to be picked up from the departments on a biweekly basis, and the data were subsequently entered into a database in order for the research team to provide departments with regular summaries of dosage levels for each segment.

Results

It is important to note that the original study (Weisburd et al., 2011, 2012) was designed to test the impact of broken windows policing at hotspots on individual-level outcomes such as fear of crime, police legitimacy, and perceptions of collective efficacy, crime, social, and physical disorder. Despite statistical power for these outcomes being very high,⁴ the results indicated that the treatment did not have a significant effect on these outcomes. The lone exception was perceptions of physical disorder, which modestly increased in the treatment areas. Examining the impact of the intervention on the street-segment level outcomes of crime and disorder was not a specific goal of the original study design. The mean change in disorder calls for service (CFS) from pre- to postintervention showed declines in both the treatment and control segments,⁵ but the ANOVA results (presented in the technical report by Weisburd et al., 2012) did not indicate a statistically significant impact of the intervention on disorder CFS. A follow-up study (Hinkle et al., 2014) provided support for the argument that this was due to a lack of statistical power for street-level outcomes due to relatively low-base rates of disorder at the study segments (see Table 1).⁶

As discussed in Hinkle et al. (2014), low-base rates and statistical power present a challenge to detecting treatment effects when evaluating microplace police interventions in smaller cities. Although a sample size of 110 hotspots is not a small sample, it took three cities to yield enough street segments with 10 or more disorder CFS and 3 or more Uniform Crime Reporting Part 1 crime CFS the year before the experiment began. If additional cities could have been recruited for the experiment, there would have been more hotspot street segments, and likely a higher preintervention base rate of calls for service to obtain more power. However, multisite evaluations also create additional challenges for researchers once studies are underway, particularly treatment implementation and variability across sites, which may also wash out potential treatment impacts.⁷ Later, we examine treatment implementation and output measures to assess the extent to which police created and maintained the conditions proscribed, and whether they took ownership of the experiment.

Examining the Delivered Dosage and Frequency of Visits and Activity

The log forms submitted by officers for each visit to treatment segments were used to calculate the treatment dosage and frequency variables, and the results discussed here.⁸ Figure 1 illustrates that in two of the three cities, the designed dosage of 180 minutes of broken windows style policing per treatment segment per week, was met or exceeded through most of the intervention period, but all three cities initially had problems meeting the dosage goal. This was due to police agencies underestimating the number of officers that would need to be assigned to the experiment and failing to allocate the agreed upon resources. During the planning stage of the study, each of the three agencies agreed to form a Broken Windows Policing Unit comprised of 8 to 12 officers, and to assign a supervisor responsible for overseeing members of the unit and managing their operations. City B did not achieve the minimum agreed upon staffing until Week 4, City C until Week 8, and City A until Week 9. Although collectively, the designed dosage was met or exceeded after the initial weeks, only City B was able to maintain (and usually exceed) the minimum personnel allocation throughout the rest of the intervention period; moreover, they had to supplement the unit officers with patrol officers in order to achieve this.

Figure 1.

Implementation of police intervention.

When dosage is examined by city for the entire experimental period, significant differences in delivery are apparent. The second column in Table 2 shows the goal for hours of police presence during the experiment for each city (the number of segments × 3 (hrs/week) × number of weeks of participation in the experiment). Overall, City C officers surpassed the goal for presence on treatment segments, spending a total of 944 hours over the 30 weeks they participated in the experiment. They also spent significantly more time on treatment segments (X = 94 hours per segment), than City A officers (X = 48 hours per segment). City B officers’ presence on treatment segments (1,363 hours) came in just under the goal of 1,395 hours. In City A, officers achieved only 55% of the goal in dosage during the experiment. As the unit of analysis is street segments, and 54% of all treatment segments were in City A, there is clearly a risk that the weaker implementation in City A impacted the overall findings of the experiment.

Table 2.

Dosage—Number of Hours Police Presence on Treatment Segments.

City	Goal hours	Sum hours	Minimum hours per segment	Maximum hours per segment	Mean hours per segment	SD
City A (n = 30)	2,610	1,450	32	64	48	6.33
City B (n = 15)	1,395	1,363	78	109	91	8.43
City C (n = 10)	900	944	73	121	94	14.62

F = 168.222. p = .000.

Multiple indicators suggest that both police supervisors and officers in City A did not take ownership of the experiment. During Weeks 5 to 7, there was almost complete treatment decay, with either one, or no officers delivering treatment during these 3 weeks. During Weeks 9 to 17, the department assigned patrol officers to the experiment to supplement the original unit officers and increase dosage. A member of the research team conducted four additional training sessions during patrol shift briefings.⁹ During this 9-week period, the number of officers participating in the experiment ranged from 16 to 27, yet the sum dosage at treatment hotpots was less than during weeks when as few as six officers were participating in the experiment. Due to an internal dispute over resources, by Week 18, all but two of the original unit officers and five patrol officers were removed from the experiment.

Table 3 presents the number of visits that officers made to treatment segments during the experiment. City A police made a total of 1,643 visits to segments, City B police made 2,405 visits, and City C police made 1,224 visits during the intervention period. The mean visits per segment column in Table 4 shows the average number of visits per treatment segment. Again, ANOVA test confirmed that there were significant differences by city. On average, City B officers made the most visits to each treatment segment (X = 160 visits per segment); City A officers made the fewest visits to each treatment segment (X = 55 visits per segment).

Table 3.

Frequency—Number of Police Visits to Treatment Segments.

City	Number of visits	Minimum visits per segment	Maximum visitsper segment	Mean visits per segment	SD
City A (n = 30)	1,643	34	74	54.77	9.77
City B (n = 15)	2,405	134	197	160.33	19.71
City C (n = 10)	1,224	98	146	122.40	14.45

F = 309.57, p = .000.

Table 4.

Dosage—Average Minutes per Police Visit to Treatment Segments.

City	Minimum minutes per visit	Maximum minutes per visit	Mean minutes per visit	SD
City A (n = 30)	39.19	61.62	53.55	5.09
City B (n = 15)	29.21	37.67	34.25	3.22
City C (n = 10)	39.27	50.48	46.20	3.34

F = 97.323; = .000.

The average minutes per police visit to treatment segments are displayed in Table 4. Repeated training sessions stressed that officers should make an effort to visit each segment multiple times each week, for short periods of time, yet in City A, officers spent on average 54 minutes per visit to a treatment segment. This is much higher than what both prior research and common sense would suggest is effective in terms of the chance of observing social disorder, as activity on a street cools down upon the arrival of a squad car and will remain lowered for a good while after they leave. Koper (1995) found that the ideal dosage for police presence to deter disorderly and criminal behavior was 10 to 15 minutes, and that longer presences had diminishing effects (see also, Telep, Mitchell, & Weisburd, 2014). With just this data, it is not clear whether there was a failure to implement treatment as designed, or whether the duration of visits is long because officers had several disorder problems to address during visits; which is why measures of outputs are also important (and will be discussed later).

Examining the Outputs

Data from the officer log forms were also used to examine the nature of treatment that was delivered. Officers were instructed to record both the physical and social disorders they observed during each visit to a street segment, and to indicate the number of specific actions they took to address the disorders on the log form. The data collected regarding the physical disorders will not be discussed here rather we will focus on the treatment of social disorders as police had a wider array of options for dealing with social problems. Log forms included a list of 16 social disorders¹⁰ along with an “other” category and also provided a checklist for actions the officers took to address these problems.¹¹ Officers were asked to indicate the number of times each action was taken to address a specific type of disorder (e.g., an officer may have conducted two stop-and-frisks (N = 2), made an arrest (N = 1), and issued a citation (N = 1) in the course of addressing one type of disorder. However, officers did not identify the number of incidents of a type of disorder during each visit on the forms. For example, officers in City C observed traffic disorder(s) during 427 visits to treatment segments; however, they recorded talking to 649 “suspects” in the process of addressing traffic disorders (conducting traffic stops; see Appendix 1). Therefore, during some of the 427 visits during which traffic disorders were observed, multiple incidents of traffic disorders were observed.

Table 5 displays the total social disorders observed by officers during visits to treatment segments and measures of officer activity level in responding to the disorders (officers could take multiple actions in response to a disorder incident).¹² The data show tremendous variability across cities in how active the police were, both in terms of the total number of actions taken, and the averages per segment, visit, and hour. In City A—which was the largest city and had the most treatment segments (n = 30)—officers observed the fewest social disorders and, compared with the other cities, officers were the least active. While in City C, the smallest city with the fewest segments (n = 10), officers were the most active. The difference is particularly striking when looking at the average number of actions per segment. City A officers performed an average of 39 actions per segment over the 6-month intervention period, while City B officers performed 201 and City C officers 408. Thus, there was clearly a large variation in intensity of police action across the three cities in the study. While this is accounted for in the original study analyses by the block-randomized design, it is still important to understand the scope of treatment variability. These output data, when considered in conjunction with the frequency data, suggest that the long duration of the average visit to treatment segments in City A (54 minutes) is more likely reflective of a failure to implement treatment as designed, as opposed to officers having several disorder problems to address during segment visits.

Table 5.

Overall Activity on Treatment Street Segments by City.

	City A (n = 30)	City B (n = 15)	City C (n = 10)
Total social disorders observed during segment visits	488	849	688
Total number of actions taken	1,171	3,007	4,075
Average actions per segment	39.03	200.47	407.50
Average actions per visit	0.71	1.25	3.33
Average actions per hour	0.81	2.21	4.32

We examined the disorder problems that were most commonly observed and addressed on the treatment segments to understand the nature of disorder that was treated. Table 6 lists, in descending order, the five most prevalent social disorders that officers responded to by city. The types of social disorders addressed across the three cities show both some similarities and differences. Pedestrian checks and traffic violations (stops) were the two most frequent disorders dealt with in each city, and loitering and public drinking also made the top five in every city.

Table 6.

Five Most Prevalent Social Disorders Addressed by City.

City A		City B		City C
Type of disorder	N	Type of disorder	N	Type of disorder	N
Pedestrian checks	154	Traffic stops	231	Traffic stops	427
Traffic stops	81	Pedestrian checks	208	Pedestrian checks	170
Loitering	86	Loitering	63	Public drinking	12
Noise	18	Fist fights	36	Loitering	7
Public drinking	15	Public drinking	32	Verbal dispute	7

The broken windows policing style emphasized that officers use discretion while attempting to respond to disorders, but ideally to use citations and arrests only with people who had been warned before, or when aggravating factors were present. We examined the five most frequent actions taken by officers in response to social disorders to see if their responses were consistent with the designed treatment. The data summarized in Table 7 show that to a large degree the policing style was implemented as designed, with one exception. City C officers issued 414 citations (the majority for traffic offenses), which was more than any single type of action taken in City A—which had three times as many treatment street segments. Citations were among the least used actions in City A and City B. Otherwise, the common actions fit with the intervention that was to focus on talking to people, stopping suspicious people, and using warnings to negotiate norms of conduct as suggested by Wilson and Kelling (1982). Arrests, while used more than citations in City A and City B, were fairly low compared with informal actions in all three cities (43 arrests in City A, 91 in City B, and 116 in City C). As a whole, these data lend credence to the experiment as they show that the policing style was mostly delivered as intended, with the exception of the possible over-reliance on traffic citations in one city.

Table 7.

Five Most Frequent Actions Taken by Police by City.

City A		City B		City C
Action taken	N	Action taken	N	Action taken	N
Warning or advise	393	Warning or advise	832	Talk to suspect	888
Talk to citizen or victim	214	Talk to citizen or victim	620	Warning or advise	671
Field interview	180	Stop and frisk	398	Talk to citizen/victim	555
Talk to suspect	117	Field interview	322	Citation	414
Stop and frisk	89	Talk to suspect	329	Field Interview	174

Outcome Measures

To explore the possibility that variation in treatment dosage and outputs (the specific actions taken by police to address disorder) within and across cities may have masked the ability of the study to detect treatment impacts, and keeping in mind the low-base rates of crime and disorder CFS on the street segments, we used Poisson regression¹³ to examine the effects of treatment variation on postintervention disorder CFS for the 30 segments with the highest preintervention CFS.^14,15 We used two measures to examine this: the total number of hours of police presence on each segment and the total number of actions taken (by police) at each segment.¹⁶ This allows us to look at two ways the treatment varied across segments and whether these variations potentially affected the ability of the experimental analyses to detect impacts on disorder. Moreover, it allows for examining whether any potential impact was actually due to the mechanisms of broken windows policing (the specific police actions taken to reduce disorder, see Weisburd, Hinkle, Braga, & Wooditch, 2015), or another mechanism such as deterrence from simply more police presence on the segments. It is important to note at the outset that this analysis cannot be used to argue the effectiveness or ineffectiveness of the treatment. As noted earlier, performing simple observational analyses on experimental data loses all benefits of randomization and cannot rule out potential confounding factors. This is simply an exploratory approach to further examine whether variation in treatment dosage and activity may have further limited the ability of the original study to detect a treatment effect in street segment level analyses of disorder and crime CFS.

Table 8 shows the model for the total number of hours of police presence on segment variable, controlling for the number of preintervention CFS on the segment and dummy variables for City B and City C (with City A as the reference category) to control for the earlier noted variation in treatment across city.¹⁷ This analysis shows that the sum of hours spent on treatment segments was a significant predictor of postintervention disorder CFS (B = −0.028, p = .000). However, the magnitude of the relationship is modest. The expected percent change in count shows that each additional hour of police presence at a street segment should drop disorder CFS by 2.8%. For current purposes, this still suggests that it is possible that the failure of the experimental analyses (see Weisburd et al., 2012) to find impacts on disorder may not solely have been due to low power from low-base rates (see Hinkle et al., 2014), but also due to the variation in dosage at the treatment segments. Specifically, these results show that for high-preintervention disorder segments, increased police presence may have more potential impact on disorder reduction.

Table 8.

Poisson Regression: Impact of Total Number of Hours of Police Presence on Disorder CFS.

Variable	B	SE	Sig.	%	% Std X	SD of X
Constant	3.784	.372	.000	–	–	–
Preintervention disorder calls for service	.028**	.011	.008	2.9	26.3	8.249
City C	1.110***	.224	.000	203.3	52.3	0.379
City B	0.721**	.248	.004	105.6	39.9	0.466
Sum hours of police presence at segment	−.028***	.005	.000	−2.8	−51.0	25.294

% = percent change in expected count for unit increase in X; % Std X = percent change in expected count for SD increase in X; SD of X = standard deviation of X; N = 30; Pseudo R²= .358.

p ≤ .05. **p ≤ .01. ***p ≤ .001.

Again, this cannot be interpreted as evidence of the effectiveness of broken windows policing as the effect size is modest, and there could be confounding factors as the benefits of the experimental design are lost in this methodological exploration. Also, any benefits could be due to simple deterrence from the increased police presence rather than the actual actions taken by police as part of the broken windows intervention (Weisburd et al., 2015). To explore this, we look at the impact of the total number of actions targeted at social disorder on postintervention disorder CFS. These results are shown in Table 9.

Table 9.

Poisson Regression: Impact of Total Number Actions Taken to Address Disorder on Disorder CFS.

Variable	B	SE	Sig.	%	% Std X	SD of X
Constant	2.417	.280	.000	–	–	–
Preintervention disorder calls for service	.036***	.001	.000	3.7	35.0	8.249
City C	.735	.440	.095	108.6	32.1	0.379
City B	−.004	.159	.981	−0.4	−0.2	0.466
Sum hours of police presence at segment	−.007**	.002	.002	−0.7	−34.3	60.763

% = percent change in expected count for unit increase in X; % Std X = percent change in expected count for SD increase in X; SD of X = standard deviation of X; N = 30; Pseudo R²= .327.

p ≤ .05. **p ≤ .01. ***p ≤ .001.

These findings show that the total number of actions police took against social disorder was significantly associated with postintervention disorder CFS (B = −0.007, p = .002). This suggests the intervention was more likely to have an impact on segments where police took more specific actions to address social disorder. However, the effect size is again modest, with each additional action leading to an expected 0.7% decline in disorder CFS. Nonetheless, the findings again offer support for the argument that the variability in treatment across the study street segments and cities could account in part for the failure of the experimental analyses to find impacts on disorder. As with the time measure, these should not be interpreted as evidence of treatment impact, but rather simply as evidence that treatment variation may have affected the ability of the original study to test impacts on street-segment disorder or crime outcomes.

Discussion

The overall broken windows policing intervention did not have a significant impact on crime and disorder calls for service in the experimental findings (Weisburd et al., 2012). This is likely due to the research design which was focused on impacts on perceptions of citizens rather than crime and disorder outcomes. Indeed, with the low-base rates of crime and disorder CFS, statistical power for examining impacts on crime and disorder CFS was quite low. Specifically, the intervention needed to produce crime declines of over 50% to reach the .80 power threshold (see Hinkle et al., 2014). This of course reinforces the importance of careful statistical power analyses before a study begins. This is not a criticism of the original study, as the authors noted at the outset the modest statistical power for examining these outcomes (as contrasted with the main outcomes of the study). But these analyses suggest that traditional estimates of statistical power would have in any event been overstated.

Our analyses of dosage, frequency, and output measures reveals that in all three cities police presence and visits to hotspots increased during the experimental period (arguably beyond the level these locations would generally receive), and outputs consistent with our operationalization of broken windows policing were delivered. In all three cities, informal actions like talking to suspects or issuing warnings were more common police actions than citations or arrests. However, these data also show variation in treatment between sites, and suggest that in varying degrees, police failed to take ownership of the science of the experiment (Neyroud & Weisburd, 2014; Weisburd & Neyroud, 2011). In one city, there was both a period of treatment decay, and overall much weaker implementation of all aspects of treatment. In all three cities, there was deviation from the treatment design in the desired frequency of visits to hotspots. After the experiment concluded, the research team conducted focus groups with officers who had participated in the experiment. Their responses to several questions provide additional evidence of a lack of ownership.

When asked the question: “How was Broken Windows Policing carried out in the field? What happened during a typical shift?” Officers from all departments made statements which suggest that the deviation from the treatment design in the desired frequency of visits to hotspots was a result of their commanding officers being more concerned with simply meeting the treatment dosage of 180 minutes per segment per week, than adhering to the way in which this dosage was designed to be delivered. Some supervisors “scheduled” officers to spend time at hotspots:

The way we had it set up was you just go to a segment for an hour block. They modified my patrol . . . working half of my shift on normal patrol and half of it I was assigned to segments for hour blocks. (City C officer)

We never sat a solid hour. I know other units in the police department did, like Sgt [name removed] and his unit—he made them sit there for an hour. We didn’t do that. I know we were supposed to, but we just couldn’t. (City B officer on night shift, emphasis added)

When I was there in a black and white police car there was actually no criminal activity going on the entire time I was there. It’s obvious, if a police car is going to be there for two hours or an hour nothing is going to be happening in that particular hundred block. (City A officer, emphasis added)

I think the feeling is that you could do in 10 minutes what some of us wasted an hour in doing . . . You could take a car, put a dummy in it and sit it on the block and get the same results. (City A officer)

These quotes also suggest that line officers generally did not take ownership for delivering the dosage as the experiment intended, and largely just stayed on the study blocks for an hour at a time as they were directed by their supervisors—despite a clear lack of belief in the approach as the latter quotes illustrate. A quote from one officer also suggests the possibility that at least he (but possibly others¹⁸) did not always report the time spent at hotspots on log forms accurately, but manipulated the time to meet the weekly dosage goals:

A lot of us don’t think we collected that data how you think we should have. And I’ll be frank, we use a term called “pencil f*ck” and it means if you asked for six hours I may have actually put in four and a quarter. (Participating officer, department withheld)

With regard to frequency of treatment and outputs delivered, our analyses show that officers in City B conducted on average, the most visits per segment, for the shortest duration of time. They also observed the most social disorders on their segment visits. Whereas, officers in City A conducted on average, the fewest visits per segment, for the longest duration of time, and observed the fewest social disorders on their segment visits. It is logical that the frequency and duration of visits would limit the number of disorders to observe and thus affect the action that officers could take or outputs delivered. “A lot of the time there isn’t anything, but you’re just trying to get in your time you’re supposed to do” (City B officer on day shift). In all three cities, traffic stops and pedestrian checks were the most frequent disorders addressed.

I’ll give an example, we go on (specific treatment segment). I go up there, and you know, with just your presence there people really aren’t doing a whole lot. If you want to find something you really got to search for it. If nothing was going on when I got there I’d pretty much just do traffic stops or ped [sic] checks. (City C officer)

Initial failure to allocate the agreed upon number of personnel, multiple changes to the personnel assigned to the experiment, and deviation from the designed treatment frequency, suggest that supervisors did not really become invested in the experiment but just went through the motions necessary to meet the goals that were set. Yet, the simple deviation from the designed treatment frequency had spiraling effects, by limiting the number of social disorders that could have potentially been observed and the opportunities for officers to employ the treatment, ultimately decreasing the intensity of treatment. The lack of ownership was also apparent in some line officers who, rather than question the utility of being directed to remain at locations when there was nothing to do, fell back to their usual enforcement activities like traffic stops, or possibly manipulated their log sheets to reflect their presence for the required time.

Arguably the most significant challenge facing evaluation researchers in policing is getting the agencies to take ownership of the science necessary to produce valid evidence on program effectiveness. When police fail to take ownership, implementation tends to be shallow at best. This clearly creates a challenging environment for producing reliable evidence on what works in policing. Over the last few decades, evaluations of police agencies’ attempts to implement new innovations have revealed that police agencies are notoriously resistant to change (Weisburd & Braga, 2006). Police have limited resources and are unlikely to fully deliver programs in which they see little utility (Sherman et al., 2014). A strategy that is of interest to police scholars may nonetheless be a hard sell to police leaders and the line officers who are asked to carry it out in the field. Yet, only recently has a discussion commenced about the need for police agencies to step up their use and ownership of science in order for policing to become an arena of evidence-based policies (Neyroud & Weisburd, 2014; Weisburd & Neyroud, 2011).

Many of the changes that are necessary for this to succeed must be borne by police agencies (i.e., changes in police education and training, leadership priorities, and measures of agency success). All of these need to reflect that police take science seriously (Neyroud & Weisburd, 2014; Weisburd & Neyroud, 2011), and until they do, it is unlikely that evidence-based policing will become the norm rather than the exception in police agencies across the country. However, police scholars and researchers have also contributed to the current state of affairs. Police research has been instrumental in the tremendous progress that has been made over the last 40 years in our understanding of what works. But it is also restricting our understanding of how innovations work (see Pawson & Tilley, 1997, 2009; Weisburd et al., 2015). As Weisburd and Neyroud (2011) have noted, “most police practices are not systematically evaluated, and we still know too little about what works and under what conditions in policing” (National Research Council, 2004; Weisburd & Eck, 2004). There have also been few efforts to develop management strategies to guide police in implementing and sustaining innovations in ways that they will have impact (although see Sherman et al., 2014).

Several decades ago, Goldstein (1979) proposed that police had become susceptible to “the ‘means over ends’ syndrome, placing more emphasis in their improvement efforts on organization and operating methods than on the substantive outcome of their work” (p. 236). In the quest to build knowledge on the causes of effectiveness, it is important that policing scholars not fall susceptible to an “ends-over-means” syndrome, where more emphasis is placed on examining changes in outcome measures than on improving research design, evaluation, and the dissemination of findings, in order to not limit the substantive outcomes of their work. If evaluation research is not directly designed to assess the extent to which police created and maintained the conditions proscribed, then regardless of whether an intervention is found to be effective or not, we are still no closer to identifying the causes of effectiveness or ineffectiveness.

In order for police researchers to make progress here, we propose that first, research and evaluation methodologies for police innovations must be better able to address whether interventions that do not produce expected outcomes are due to research design, implementation failure or variability and police lack of ownership, or treatment ineffectiveness. The use of randomized controlled trials in policing evaluations is a good start, but there are still implementation issues that need to be addressed. Intensive efforts in measuring and monitoring interventions throughout study periods, and regular meetings with the involved agencies are necessary to achieve and maintain proper treatment. This is especially important and challenging in multisite evaluations. Second, research and evaluation methodologies need to focus more on identifying the specific police actions or outputs of treatments that are effective, so that police agencies can make more efficient use of their limited resources and implement innovations in ways that are most likely to have impact. Third, researchers must follow the advice of Weisburd and Neyroud (2011; see also, Neyroud & Weisburd, 2014) and pursue and design studies that have the best chances of getting the involved agencies invested so that they take ownership of the science.

Conclusion

Our aim with this article was to illustrate how greater use of research and evaluation methodologies that focus not only on the outcomes but also the treatment and outputs of police interventions can improve our understanding of the causes of the effectiveness (or ineffectiveness) of police strategies. In monitoring and reporting findings on treatment dosage, frequency, and outputs, the broken windows experiment was able to offer more guidance and pathways for future studies in this area, than if it had taken a more typical “black box” approach to program evaluation. Thus, rather than being a study that could erroneously lead to “nothing works” type claims, ala many of the weak evaluations included in Martinson’s (1974) famous review (see Palmer, 1975), it simply shows that a specific level of dosage, frequency, and outputs failed to produce the expected outcomes. Moreover, those results, as well as the focus group interviews with project officers, suggested that at least part of the reason behind the variation in treatment was the failure of the agencies to take ownership of the science of the interventions. Through this type of research, we can understand more about how strategies are effective (or ineffective), and more progress can be made in developing police management strategies outlining how police leaders can create and maintain the causes of successful strategies to increase the ownership of science in police agencies.

Appendix 1: Summary of social disorders and actions taken by city

City A: Social disorders and actions taken

Social disorder type	Number of visits Disorder observed	Number of citizens/victims talked to	Number of suspects talked to	Number of suspects mediated	Number of field interviews conducted	Number of stop and frisks	Number of warnings	Number of citations	Number of arrests	Number of other actions	Total actions
Drinking in public	15	1	5	–	9	12	24	2	–	2	55
Drug activity	2		2	–	–		2	–	–	–	4
Fist fight	5	2	6	–	4	–	–	–	5	–	17
Littering	5	1	–	–	2	–	3	–	–	1	7
Loitering	86	50	35	7	43	11	127	1	10	7	291
Noise	19	2	7	–	–	–	23	–	–	3	35
Pedestrian(s) (check)	154	38	22	7	91	53	106	2	12	6	337
Prostitution	3	–	–	–	2	–	2	–	–	–	4
Public urination	2	–	–	–	2	1	1	2	–	–	6
Soliciting for money	4	1	1	–	1	–	4	1	–	–	8
Suspicious/erratic behavior	8	3	3	1	1	–	9	1	1	1	20
Theft from vehicle	3	2	1	–	1	–		–	–	–	4
Traffic (stops)	81	8	18	1	20	11	67	25	10	10	170
Vandalism	1	5	3	3		–	3	–	–	–	14
Verbal dispute	5	3	3	2	1	–	3	1	1	–	14
Other	95	98	11	16	3	1	19	3	4	30	185
Total	488	214	117	37	180	89	393	38	43	60	1,171

City B: Social Disorders and Actions Taken

Social disorder type	Number of visits disorder observed	Number of citizens/victims talked to	Number of suspects talked to	Number of suspects mediated	Number of field interviews conducted	Number of stop and frisks	Number of warnings	Number of citations	Number of arrests	Number of other actions	Total actions
Destruction of property	5	1	3	1	6	20	10	1	–	–	42
Drinking in public	32	21	14	6	9	13	50	2	3	2	120
Drug activity	11	5	8	3	10	10	8	3	4	3	54
Fist fight	36	20	2	2	36	33	28	3	8	5	137
Littering	11	–	6	2	1	–	5	–	–	3	17
Loitering	63	123	64	43	29	41	142	1	1	8	452
Noise	16	6	11	32	9	14	10	–	–	1	83
Pedestrian(s) (check)	208	181	16	13	123	107	156	4	16	39	655
Public urination	10	–	–	1	–	–	11	–	–	–	12
Soliciting for money	9	2	4	2	3	–	8	–	–	1	20
Suspicious/erratic behavior	26	5	14	1	24	18	16	5	4	5	92
Theft from vehicle	5	9	–	–	4	3	–	–	3	6	25
Traffic (stops)	231	36	165	18	105	109	255	11	27	50	776
Vandalism	4	–	–	–	–	2	–	–	2	–	4
Verbal dispute	25	11	12	12	2	3	34	–	4	–	78
Other	157	200	10	41	11	25	99	4	19	31	440
Total	849	620	329	177	372	398	832	34	91	154	3,007

City C: Social Disorders and Actions Taken

Social disorder type	Number of visits disorder observed	Number of citizens/victims talked to	Number of suspects talked to	Number of suspects mediated	Number of field interviews conducted	Number of stop and frisks	Number of warnings	Number of citations	Number of arrests	Number of other actions	Total actions
Destruction of property	1	–	–	–	–	2	–	2	2	–	6
Drinking in public	12	1	8	–	1	5	13	2	2	1	33
Drug activity	6	–	5	3	4	5	3	2	5	–	27
Fist fight	1	–	–	–	–	–	–	–	1	–	1
Littering	2	1	–	1	–	–	1	–	–	–	3
Loitering	7	1	13	–	7	4	12	–	1	–	38
Noise	3		1	–	–	–	–	–	–	–	1
Pedestrian(s) (check)	170	8	185	19	84	88	163	11	20	–	578
Public urination	2	–	–	–	–		1	–	–	–	1
Soliciting for money	–	–	–	–	–	–	–	–	–	–	–
Suspicious/erratic behavior	5	4	6	4	4	2	–	2	2	–	24
Theft from vehicle	4		2	–	1	–	2	1	1	–	7
Traffic (stops)	427	13	649	27	71	58	462	310	68	27	1685
Vandalism	–	–	–	–	–	–	–	–	–	–	–
Verbal dispute	7	1	9	4	–	–	7	–	1		22
Other	41	526	10	5	2	–	7	84	13	70	717
Total	688	555	888	63	174	164	671	414	116	98	4,075

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by National Institute of Justice (grant number 2007-IJ-CX-0047).

Notes

Author Biographies

Christine Famega is an associate professor in the Department of Criminal Justice at California State University – San Bernardino. Her research interests include evidence-based policing, police technology, and police operations.

Joshua C. Hinkle is an associate professor in the Department of Criminal Justice and Criminology at Georgia State University. His research interests include broken windows theory, police effectiveness, the impact of police tactics on targeted communities, and fear of crime.

David Weisburd is a distinguished professor of Criminology, Law and Society at George Mason University. He also serves as Walter E. Meyer Professor of Law and Criminal Justice at the Hebrew University in Jerusalem and is Chief Science Adviser for the Police Foundation in Washington, DC. For his research on policing, Professor Weisburd received the Stockholm Prize in Criminology in 2010, the Sutherland Award from the ASC in 2014, and the Israel Prize in 2015. He is presently the chair of the National Academy of Sciences Panel on Proactive Policing.

References

Bellg

A. J.

Borrelli

Resnick

Hecht

Minicucci

D. S.

Ory

Czajkowski

(2004) Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH behavior change consortium. Health Psychology 23(5): 443–451.

Bloom, H. (Ed.). (2005). Learning more from social experiments: Evolving analytic approaches. New York, NY: Russell Sage Foundation.

Borrelli

Sepinwall

Ernst

Bellg

A. J.

Czajkowski

Breger

Orwig

(2005) A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. Journal of Consulting and Clinical Psychology 73(5): 852–860.

Boruch, R. (2010). To flop is human: Can we (should we) invent orderly/disciplined/scientific approaches to anticipating failure to meet expectations in testing innovations and to learn from such failure? Paper presented at the Second Annual Symposium of the Center for Evidence-Based Crime Policy at George Mason University, Fairfax, VA, August 9–10.

Boruch

R. F.

McSweeny

A. J.

Soderstrom

E. J.

(1978) Randomized field experiments for program planning, development, and evaluation. Evaluation Quarterly 2: 655–95.

Braga

A. A.

Papachristos

A. V.

Hureau

D. M.

(2014) The effects of hot spots policing on crime: An updated systematic review and meta-analysis. Justice Quarterly 31(4): 633–663.

Cameron

Trivedi

P. K.

(2013) Regression analysis of count data, 2nd ed, Econometric Society Monograph No.53. New York, NY: Cambridge University Press.

Campbell

D. T.

Stanley

J. S.

(1963) Experimental and quasi-experimental designs for research, Boston, MA: Houghton Mifflin.

Copes

Vieraitis

L. M.

(2005) Evaluation research in the social sciences. In: Kerley

K. R.

(ed.) Policing & program evaluation, Chap 1. Upper Saddle River, NJ: Prentice Hall, pp. 1–13.

10.

Goldstein

(1979) Improving policing: A problem-oriented approach. Crime & Delinquency 25(2): 236–258.

11.

Hinkle

J. C.

Weisburd

Famega

Ready

(2014) The problem is not just sample size: The consequences of low base rates in policing experiments in smaller cities. Evaluation Review. 1–26. doi:10.1177/0193841X13519799.

12.

Kelling

G. L.

Coles

(1996) Fixing broken windows: Restoring order and reducing crime in American cities, New York, NY: Free Press.

13.

Kelling

G. L.

Sousa

W. H.

(2001) Do police matter? An analysis of the impact of New York City’s police reforms. (Civic Report No., 22). New York, NY: Manhattan Institute for Policy Research.

14.

Koper

C. S.

(1995) Just enough police presence: Reducing crime and disorderly behavior by optimizing patrol time in crime hot spots. Justice Quarterly 12: 649–672.

15.

Kreft

de Leeuw

(1998) Introducing multilevel modeling, London, England: Sage Publications.

16.

Lipsey

M. W.

Petrie

Weisburd

Gottfredson

(2006) Improving evaluation of anti-crime programs: Summary of a national research council report. Journal of Experimental Criminology 2: 271–307.

17.

Martinson

(1974) What works? Questions and answers about prison reform. The Public Interest 35: 22–54.

18.

Miller

D. E.

Kunce

J. T.

(1973) Prediction and statistical overkill revisited. Measurement Evaluation in Guidance 6: 157–163.

19.

National Research Council, 2004 National Research Council. (2004). Effectiveness of police activity in reducing crime, disorder and fear. In W. Skogan & K. Frydl (Eds.), Fairness and effectiveness in policing: The evidence (pp. 217–251). Committee to Review Research on Police Policy and Practices. Committee on Law and Justice, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

20.

Neyroud

Weisburd

(2014) Transforming the police through science: The challenge of ownership. Policing 8: 287–293.

21.

Palmer

(1975) Martinson revisited. Journal of Research in Crime and Delinquency 12: 133–152.

22.

Pawson

Tilley

(1997) Realistic evaluation, London, England: Sage.

23.

Riecken

H. W.

Boruch

R. F.

Campbell

D. T.

Caplan

Glennan

T. K.

Pratt

J. W.

Williams

(1974) Social experimentation: A method for planning and evaluating social programs, New York, NY: Academic Press.

24.

Rosenbaum

(1988) Community crime prevention: A review and synthesis of the literature. Justice Quarterly 5: 325–395.

25.

Sherman

L. W.

(1998) Evidence-based policing. Ideas in American policing, Washington, DC: Police Foundation.

26.

Sherman

L. W.

Weisburd

(1995) General deterrent effects of police patrol in crime “hot-spots”: A randomized controlled trial. Justice Quarterly 12: 755–782.

27.

Sherman

L. W.

Williams

Ariel

Strang

L. R.

Wain

Slothower

Norton

(2014) An integrated theory of hot spots patrol strategy: Implementing prevention by scaling up and feeding back. Journal of Contemporary Criminal Justice 30: 95–122.

28.

Telep

C. W.

Mitchell

R. J.

Weisburd

(2014) How much time should the police spend at crime hot spots? Answers from a police agency directed randomized field trial in Sacramento, California. Justice Quarterly 31(5): 905–933.

29.

Tilley

(2009) Sherman vs Sherman: Realism vs rhetoric. Criminology & Criminal Justice: An International Journal 9(2): 135–144.

30.

U.S. Census Bureau. (2010). United States Census 2010. Retrieved from http://www.census.gov/2010census/.

31.

Weisburd

(2003) Ethical practice and evaluation of interventions in crime and justice: The moral imperative for randomized trials. Evaluation Review 27(3): 336–354.

32.

Weisburd

Braga

A. A.

(2006) Police innovation: Contrasting perspectives, New York, NY: Cambridge University Press.

33.

Weisburd

Eck

J. E.

(2004) What can police do to reduce crime, disorder and fear? The Annals of the American Academy of Political and Social Science 593: 42–65.

34.

Weisburd

Hinkle

J. C.

(2012) The importance of randomized experiments in evaluating crime prevention. In: Welsh

B. C.

Farrington

D. P.

(eds) The Oxford handbook on crime prevention, New York, NY: Oxford University Press, pp. 446–465.

35.

Weisburd

Hinkle

J. C.

Braga

A. A.

Wooditch

(2015) Understanding the mechanisms underlying broken windows policing: The need for evaluation evidence. Journal of Research in Crime and Delinquency 52: 589–608.

36.

Weisburd

Hinkle

J. C.

Famega

Ready

(2012) Legitimacy, fear and collective efficacy in crime hot spots: Assessing the impacts of broken windows policing strategies on citizen attitudes, Washington, DC: U.S. Department of Justice, National Institute of Justice.

37.

Weisburd

Hinkle

J. C.

Famega

Ready

(2011) The possible “backfire” effects of hot spots policing: An experimental assessment of impacts on legitimacy, fear and collective efficacy. Journal of Experimental Criminology 7: 297–320.

38.

Weisburd

Lum

C. M.

Yang

S. M.

(2003) When can we conclude that treatments or programs “don’t work”? The Annals of the American Academy of Political and Social Science 587(1): 31–48.

39.

Weisburd, D., & Neyroud, P. (2011). Police science: Toward a new paradigm. New Perspectives in Policing. Harvard University, John F. Kennedy School of Government, Program in Criminal Justice Policy Management, United States of America. Washington, D.C.: U.S. Department of Justice, Office of Justice Programs, National Institute of Justice.

40.

Wilson

J. Q.

Kelling

G. L.

(1982) Broken windows: The police and neighborhood safety. Atlantic Monthly 211: 29–38.

41.

Worrall

J. L.

(2002) Does “broken windows” law enforcement reduce serious crime (CICG research brief), Sacramento: California Institute for County Government.