The Effectiveness of a Cognitive Skills Training Program for Adult Offenders

Abstract

This article provides an evaluation of the effectiveness of the “CoVa” cognitive skills training program. A quasi-experimental design was used to compare the known reoffending behavior of 2,229 CoVa participants with that of two control groups of offenders: (1) one formed using the inclusion criteria of the program, and (2) one formed using propensity score matching (PSM). Outcome measures included the prevalence, frequency, and impact of new, adjudicated crimes, with the impact defined as the combined severity of the offenses. Results indicated that the participants in the CoVa group were reconvicted less frequently than those in Control Group 1. Moreover, the impact of their recidivism was lower than the impact of those of Control Group 2. However, the effect sizes were very small, and no significant differences were found regarding reconviction prevalence. Explanations for the absence of (large) effects are discussed.

Keywords

adult offenders cognitive skills behavior program reconviction intervention effectiveness

To reduce reoffending rates, various countries in Europe and North America invest in offending behavior programs within the prison and probation services (cf. Redondo, Sánchez-Meca, & Garrido, 1999; Tong & Farrington, 2006). The risk–need–responsivity (RNR) model as proposed by Andrews, Bonta, and Gendreau (e.g., Andrews & Bonta, 2003; Andrews et al., 1990; Gendreau & Andrews, 1990) is one of the more influential models for the treatment of offenders (Ward, Melser, & Yates, 2007). The RNR model suggests that reoffending is less likely to occur when offenders participate in programs that (a) match the level of program intensity to the participants’ risk of reoffending, (b) target known criminogenic needs of the offender, and (c) tailor the intervention to the learning style, motivation, abilities, and strengths of the offender (specific responsivity) using cognitive behavioral or social learning principles (general responsivity).

Following the RNR principles, the Dutch Probation Service has developed several behavioral intervention programs for offenders (Ministry of Justice, 2005). One of the most widely implemented programs in the Netherlands is the cognitive skills training CoVa (an acronym of cognitieve vaardigheden [i.e., cognitive skills]). Since 2004, it is possible to administer the CoVa training to adjudicated adult offenders, both inside and outside the prison. The training itself is an adapted version of the Enhanced Thinking Skills (ETS) program¹ (Clarke, Simmonds, & Wydall, 2004), which has been used in England and Wales for over 20 years, although from 2009 the program is gradually replaced with the Thinking Skills Program (TSP; Gobett & Sellen, 2014).

The CoVa program is based on the idea that cognitive deficits are important criminogenic factors that can be remedied or improved (Buysse & Loef, 2012). Deficits in cognitive skills have an effect on both the onset and the continuation of offending behavior (Tong & Farrington, 2006). Various empirical studies provide support for this relationship (Antonowicz & Ross, 1994; Farrington, 2001; Ross & Fabiano, 1985). Conversely, the cognitive model suggests that if offenders manage to improve their cognitive skills, the risk of (re)offending reduces correspondingly (Ross & Fabiano, 1985). Basically, the program theory of CoVa (i.e., the description of how the program can achieve its goals) stipulates that the training will reduce cognitive deficits, which will in turn lead to lower reconviction rates.

Previous Research on CoVa and ETS

Whether CoVa actually reduces reoffending has not yet been thoroughly investigated. The training is only researched as part of a larger investigation into the effectiveness of the Prevention of Recidivism Program in detention (Bosma, 2017). A small part of the offenders in this evaluation participated in CoVa. The reconviction prevalence of 93 offenders who completed CoVa was compared to offenders who did not participate. After controlling for differences in sentence length, prior convictions, and accommodation, no statistically significant effect was found on the reconviction rates in the 6 and 24 months following release. However, there are also studies about the effectiveness of CoVa on other outcomes. Two studies examined the results of preintervention and postintervention psychometric assessments completed by CoVa participants (Buysse & Loef, 2012; Tierolf, 2007). Both studies showed that the CoVa training has a significant positive effect on impulsiveness reduction, problem solving, and perspective taking. However, most of the reported effect sizes were small, and these studies did not include a control group. A third study used neurocognitive tasks to examine the effect of CoVa among prisoners (Cornet, Van der Laan, Nijman, Tollenaar, & De Kogel, 2016). This study did include a control group, but the results did not provide any evidence of a favorable change in neurobiological functioning in the intervention group.

Evaluation research on the predecessor of CoVa―ETS―does include various studies involving reoffending data, though the results of these studies are mixed. Several studies on ETS did not find significant differences in the 1- or 2-year reconviction rates of ETS participants and matched controls (Cann, 2006; Falshaw, Friendship, Travers, & Nugent, 2003; Palmer, Hatcher, McGuire, & Hollin, 2015). Other studies only showed a significant and positive effect when the comparison was limited to those who fully completed the intervention, excluding subjects who dropped out prematurely (Cann, Falshaw, Nugent, & Friendship, 2003; Hollin et al., 2008). Several explanations had been put forward for the fact that no effects were found, including the rapid expansion of the program, the lack of motivation on the part of the participants, and low program completion rates (Falshaw et al., 2003; Hollin et al., 2008). Moreover, the authors acknowledged a methodological limitation common to each of these studies: the inability to control for potential selection bias due to differences in dynamic risk factors for reconviction.

There are, however, also studies that showed significant positive effects of ETS. Reconviction rate reductions of 6% to 14% points were found (Friendship, Blud, Erikson, & Travers, 2002; Sadlier, 2010; Travers, Mann, & Hollin, 2014; Travers, Wakeling, Mann, & Hollin, 2011). In contrast to the other studies, Sadlier (2010) did use both static and dynamic risk factors to compose a control group and showed that the proportion of ETS participants who were reconvicted within 1 year was 6.2% points lower than that of the control group. Moreover, ETS participants were convicted for 60 fewer recordable offenses within 1 year per 100 released offenders than control group participants. Nevertheless, no statistically significant effect was found on the rate of reconviction for severe offenses (for instance, violence against persons, sexual offenses, and vehicular manslaughter).

The Current Study

This article evaluates the CoVa program in terms of reconviction rates.² We will examine the prevalence, frequency, and impact of reoffending by the CoVa participants and compare them with a similar group of offenders who did not participate in CoVa.³ On the basis of the program theory, we expect that the CoVa participants have lower reconviction rates on these three outcome measures than the two control groups, which consist of offenders who have an RISc (Recidive Inschattings Schalen; Reclassering Nederland, 2010) assessment but did not participate in the program.

Although CoVa is one of the most widely implemented behavior programs in the Netherlands, the reconviction rates of the participants of CoVa are largely unknown. They are known only for a small proportion of participants, namely, those who had completed the CoVa training in prison and entered one of the Dutch remand centers between October 2010 and April 2011 (Bosma, 2017). The current study is a large-scale evaluation which also includes participants who follow the training outside of prison or who did not fully complete the training. Therefore, the current study contributes to the research into the effectiveness of CoVa. In contrast to the majority of earlier evaluations of ETS, this evaluation includes both static and dynamic risk factors in the composition of the control groups. Moreover, this study includes three outcome measures instead of only reconviction prevalence. The reconviction prevalence is an extreme dichotomous outcome in terms of human behavior because an offender either stops reoffending or continues (Bushway, Piquero, Broidy, Cauffman, & Mazerolle, 2001). However, if the process of desistance is actually more gradual and dynamic, then it may be of interest to examine the frequency as well as the impact of the repeat crimes to codetermine the success of the program.

Method

In this research, we used a quasi-experimental research design with observational data. The reconviction rates of CoVa participants who followed the training between 2008 and 2011 (n = 2,229) were compared to the reconviction rates in two control groups of offenders (n = 506 and n = 2,229) who did not participate in the CoVa program.

Program Description

The CoVa program consists of 20 group sessions of 2.5 hr, typically held twice a week, with a group consisting of 12 to 14 participants (Buysse & Loef, 2012). Every session is conducted by two regular trainers, with a third trainer on standby. During the training sessions, participants are instructed about values, made to practice new skills, learn to think creatively and rationally, and to consider both the short-term and the long-term consequences of their actions. Between sessions, participants are obliged to complete homework. The total amount of time invested by each participant is approximately 80 hr. When a participant is unable to attend a session, they have to make up for it later. Failure to appear for the second time—without a valid reason—leads to exclusion from the entire course.

Offenders have to meet certain selection criteria to participate in the CoVa program. Whether or not offenders meet the criteria is determined by a screening instrument called “RISc” (Reclassering Nederland, 2010), which closely resembles the OASys (Howard, Clark, & Garnham, 2003). The RISc is used by the Probation Service on a national scale to investigate problems on 12 life domains (e.g., accommodation, drug misuse, and education, work, and training). Together, the individual sections assessing these domains form an overall score indicating the risk of reoffending. To complete a RISc assessment, probation officers use criminal record information, interviews with the offenders and offender’s references, and their own professional judgment. The RISc was evaluated with regard to the interrater reliability, factorial structure, internal consistency, and concurrent validity (Van der Knaap, Leenarts, & Nijssen, 2007). The general conclusion of the evaluation was that the RISc has favorable psychometric qualities. Further research showed that the predictive validity of the RISc was “sufficient” with respect to the prediction of general, serious, and very serious recidivism in the population of probation and aftercare clients (Van der Knaap & Alberda, 2009).

CoVa is developed for offenders who show a moderate to high risk of reoffending as measured by the RISc assessment (i.e., a total score of 32 points or more; RISc version 3.2.5). In addition, offenders should show cognitive deficits on the thinking and behavior scale of the RISc. This scale consists of eight items: level of interpersonal skills, aggressive/controlling behavior, temper control, impulsivity, ability to recognize problems, problem solving skills, achieves goals, and mindset and learning ability (range = 0-2 points for each item). The offender should score at least 4 points on this scale. With regard to the five last mentioned items, the offender should have at least 1 score of 2 points (which indicates a severe problem) on any of these five items or a score of 1 (which indicates a problem) on each. If an offender does not meet the abovementioned selection criteria but is nevertheless considered a suitable candidate for the program by the probation officer, the offender is allowed to participate if the intervention coach agrees with the probation officer’s judgment (Ferwerda, Van Wijk, Arts, & Kuppens, 2009). Other treatment inclusion criteria are as follows: being an adult (i.e., 18 years or over), not being under special psychiatric care, sufficient proficiency in the Dutch language, a valid residence permit, and, if applicable, having to serve at least another 4 months of detention at the start of the training program. Contraindications for participation include addictions or mental health problems that obstruct group functioning.⁴ Furthermore, the CoVa training is also not suitable for offenders who either show little willingness to change or are presumed to have a low IQ.⁵ Offenders with lower intellectual abilities are generally assigned to an adjusted version of the training named “CoVa plus.” Fischer, Captein, and Zwirs (2012) estimated that 18% to 30% of the offenders who meet the selection criteria for CoVa exhibit one or more contraindications.

Data

We combined different data sources. Information gathered in the earlier mentioned research by Buysse and Loef (2012) was reused. Specifically, we reused data about the persons who participated in the CoVa program, their start and end date in the program, the setting in which the program was applied (inside or outside prison), and whether or not participants dropped out of the program prematurely. On the individual level, we linked the data of each CoVa participant to three databases:

Probation data: We used the RISc database to get more information about the psychosocial functioning of the CoVa participants before they entered the program. Probation officers apply this instrument to assess the risk of reoffending, obtain presentence reports, and to develop plans concerning probation and aftercare service supervision. The RISc database contains information about the prevalence of criminogenic needs in 12 different life areas.

Prison data: The Dutch prison registration system (TenUitvoerleggingprogramma Gevangeniswezen, abbreviated to “TULP-GW”) contains information about prison admission and release dates of all detainees in the Netherlands.

Conviction data: The Dutch Offender Index (Onderzoek- en Beleidsdatabase Justitiële Documentatie, abbreviated to “OBJD”) is an anonymized database. It contains information about all criminal cases of individuals prosecuted in the Netherlands for criminal offenses. Criminal cases are only registered for persons who are 12 years and older.

The resulting combined data set suffered from missing data. Around 12% of the records contained at least one missing value. In most cases, it was unclear whether a person had problems in one or more life areas of the RISc. Occasionally, the offense type or the country of birth of a participant was missing. The absence of data was found to be selective, so if we simply would have discarded the individuals with missing data, a biased estimate could be the result. Therefore, we used switching regression (Van Buuren, Boshuizen, & Knook, 1999) to iteratively impute the missing data. For continuous variables, we applied predictive mean matching (Heitjan & Little, 1991; Schenker & Taylor, 1996) to obtain plausible covariate values.

Participants and Control Groups

This study examined all offenders who had followed the CoVa training at least once somewhere between 2008 and 2011. As referred to above, this information was obtained from the study of Buysse and Loef (2012). If an offender followed CoVa more than once, only the last instance of participation was included. In addition, we excluded participants with missing end dates, intramural participants without a matching detention episode in the prison databases, and those whose records could not be linked to the other data sources. Furthermore, we excluded participants who left detention after 2011, ensuring that every participant was observed for a period of at least 2.5 years. The final treatment group, hereafter referred to as the “CoVa group,” includes 2,229 participants. This group consists of both completers and noncompleters (around one fifth of the participants dropped out prematurely).

To estimate the effect of CoVa on post-training recidivism, the CoVa participants were compared to offenders in two separate control groups. Both comparison groups were selected from the total population of offenders with a valid RISc assessment but who had not followed CoVa between 2008 and 2013.⁶ Next, assessments of forensic psychiatric patients and juvenile offenders were removed, as they were not allowed to participate in this particular training, thus making them inadequate controls. The resulting pre-matching group consisted of 52,242 offenders.

The first comparison group was composed by Buysse and Loef (2012). Buysse and Loef selected all RISc assessments of persons who fitted the inclusion criteria for CoVa. Offenders with a presumed low IQ as well as those with a very low score on items measuring motivation to change were removed. From the remaining pool of offenders, 300 offenders with an extramural sanction and 300 detained offenders were randomly selected. Subsequently, Buysse and Loef manually checked in the RISc data for any contraindications (e.g., severe psychiatric disorder or addiction). If none were found, the offender was permitted in the comparison group. From this comparison group, we excluded those offenders in the intramural group who could not be linked to the other data sources and who left detention or completed the RISc assessment after 2011. The resulting first comparison group consists of 506 offenders.

In addition to this first control group, we used the pre-matching group of 52,242 offenders to compose a second control group using propensity score matching (PSM; Rosenbaum & Rubin, 1983). Offenders were matched on the probability to be allocated to the CoVa group on basis of their characteristics. The independent variables used as covariates can be grouped into four categories: offender background characteristics, features of the index case, criminal career characteristics, and the RISc problem scores in 12 life areas. All covariates have shown associations with reconvictions (cf. Wartna, 2009; Zebel, Alberda, & Wartna, 2014). There are several approaches to applying PSM within a multiple imputation framework. We used the method of combining PSM and imputations of missing data that has shown the least bias in simulation studies (Hill, Reiter, & Zanutoo, 2004). This method consists of the following steps:

Estimate propensity score models on each separate imputed data set and find a match for each intervention subject;

Calculate the covariate balance statistics and the treatment effect for each separate matched sample;

Combine the effect estimates and the balance statistics over imputations using the rules for combining statistics established by Rubin (1978).

Outcome Measures

To estimate reconviction, we used conviction data from the criminal justice system. A conviction was defined as a valid disposal by the court or penalty order issued by the public prosecutor (i.e., decisions that found the defendant guilty). Acquittals and technical dismissals were excluded. Furthermore, subdistrict court cases (i.e., minor offenses) were not counted as valid reconviction events. However, criminal justice data do not necessarily display the actual number of committed offenses but only those that were detected by the justice system (Biderman & Reiss, 1967). Nevertheless, as this is a comparison study, the estimates served as lower bounds of the actual number of new offenses for both the treatment and control groups.

Three outcome measures of reoffending were considered: the prevalence of reconvictions, the number of reconvictions, and the impact of all crimes involved in the new penal cases. The prevalence was defined as the proportion of offenders who are convicted of an offense within a certain follow-up time. The observation period with respect to the CoVa participants started at the last day they attended the program or, if applicable, the day that they were released from prison. For the offenders in the comparison groups, the follow-up period started on the day they were released from prison or, if no prison sentence was served, on the day when the RISc questionnaire was completed. Reconviction frequency was defined as the number of new convictions. As some convictions lead to new imprisonments, the time at risk for reoffending was not similar for everyone. For that reason, we corrected the frequency for the duration of the subsequent detentions and rescaled the frequency as the number of new convictions per year of “not being detained.” The impact of the reoffenses that were adjudicated is a measure that weighs the frequency of registered offenses against their severity. Offense severity was measured with the MSI (in Dutch, the Misdaad Straf Index; Beerthuizen, Wartna, Verweij, & Tollenaar, 2015). This scale quantifies the severity of the committed offenses in a penal case on basis of the sentence imposed in that case (refer to the appendix for more information on this process). In this study, we determined a certain amount of “sanction points” for each reconviction within a 4-year observation period. Next, we calculated an overall MSI mean score per year with no detention for each person.

Analysis Strategy

The reconviction prevalence rates were estimated using survival analysis (Kaplan & Meier, 1958). To test whether there were differences in survival between the CoVa group and comparison groups, we used log-rate, Wilcoxon, and Tarone–Ware tests. Differences in frequency and impact of the reoffenses were tested with t tests. We used Cohen’s d to portray the effect sizes of CoVa (Cohen, 1988).

The quasi-experimental design of this study required that we statistically control for any differences found in relevant background characteristics. If there were differences between the participants and a comparison group, we used multiple regression models. These models allowed us to determine the effect of CoVa participation while accounting for group differences in various variables that are typically associated with reoffending (e.g., offense history or drug abuse). As this study used three different kinds of outcome measures, different multiple regression models were deemed appropriate. First, we applied a Cox regression analysis on the reconviction prevalence. The effect sizes of the covariates and this outcome variable were expressed in the exponent beta (e^β) or hazard ratio (Cox, 1972). Second, to analyze the number of reconvictions, we used a negative binomial regression model. This technique is particularly suited for the analyses of “count” variables with a variance in the outcome measure that is larger than the average (Long & Freese, 2001). Third, we used a multiple regression model when analyzing the impact of the reoffenses. All regression models were computed in Stata version 12.1 using the “mi estimate” procedure. This procedure combined multiple imputations and estimated adjusted coefficients.

Results

Outcome of the Matching Procedure

As the effectiveness of CoVa is determined by a comparison of reconviction rates of the treatment and control groups, it was important that these groups were comparable on various background characteristics that (co)influence the risk of recidivism. To test whether there were differences, we performed imbalance tests. Table 1 displays the results of the pre-matching and post-matching t tests. Significant differences between the CoVa group and the pre-matching group existed on every covariate, which underlines the need for matching the treatment and the comparison groups.

Table 1:

Background Characteristics of the CoVa Group and the Comparison Groups

	CoVa group	Pre-matching			Post-matching (selection inclusion criteria)			Post-matching (selection with PSM)
	CoVa group	RISc population			Comparison Group 1			Comparison Group 2
Characteristics	n = 2,229	n = 52,242	t value	Significance	n = 506	t value	Significance	n = 2,229	t value	Significance
Demographic characteristics
Male (%)	94.7	88.6	−12.4	***	93.5	−1.0		94.6	−0.1
Age	30.0	34.8	23.3	***	33.3	6.3	***	30.0	−0.1
Country of birth (%)
The Netherlands	69.3	72.3	3.0	**	67.8	−0.7		69.4	0.0
Morocco	4.7	3.4	−2.8	**	6.7	1.7	*	4.5	−0.4
Former Neth. Antilles, and Aruba	8.1	4.5	−6.2	***	6.1	−1.6		8.2	0.1
Surinam	7.3	5.6	−3.0	**	6.7	−0.5		7.3	0.0
Turkey	3.0	2.9	−0.3		3.0	0.0		2.9	−0.2
Other Western	2.3	4.7	7.3	***	3.6	1.4		2.2	−0.1
Other non-Western	5.2	6.5	2.6	**	6.1	0.8		5.6	0.4
Characteristics index case
Offense type (%)
Violent	20.1	42.2	25.2	***	30.0	4.5	***	19.6	−0.4
Sexual	1.3	4.4	12.2	***	4.7	3.4	***	1.3	−0.3
Violent property	27.5	8.1	−20.3	***	17.9	−4.9	***	26.6	−0.6
Property	22.8	19.4	−3.8	***	25.2	1.1		24.2	0.6
Public order	6.8	9.6	5.2	***	6.1	−0.5		6.5	−0.1
Drug	16.5	8.9	−9.6	***	11.1	−3.3	***	16.8	0.3
Traffic	1.1	2.0	3.9	***	0.0	−4.9	***	1.2	0.0
Other	3.9	5.4	3.3	***	5.0	1.0		3.9	0.1
Mean severity	69.2	53.7	−44.2	***	65.7	−4.2	***	69.2	0.0
Setting intramural (%)	42.7	9.8	−31.1	***	42.7	0.0		41.8	−0.5
Mean days in prison	779.6	545.7	−13.6	***	503.6	−6.9	***	746.9	−1.2
Criminal career characteristics
Mean age at first conviction	18.4	24.1	42.6	***	19.5	3.2	***	18.2	−0.8
Mean previous convictions	10.8	8.0	−7.8	***	13.5	2.9	**	11.2	0.6
Mean conviction density	0.8	0.6	−18.7	***	0.8	3.0	**	0.8	1.1
Mean frequency last 4 years	1.2	0.8	−15.7	***	1.2	0.3		1.2	1.0
Mean severity last 4 years	48.0	30.3	−23.0	***	45.1	−1.6		49.7	0.8
Problem areas (RISc; %)
Criminal history
Problem	61.4	40.4	−19.9	***	63.3	0.8		61.0	−0.3
Severe problem	9.9	3.5	−10.1	***	8.9	−0.7		10.2	0.3
Housing
Problem	24.4	23.0	−1.5		32.6	3.6	***	26.7	1.4
Severe problem	10.2	8.7	−2.3	**	14.2	2.4	**	10.3	0.2
Education and work
Problem	53.6	39.3	−13.3	***	62.9	3.8	***	52.7	−0.6
Severe problem	12.7	13.4	1.0		13.6	0.5		13.8	0.8
Income
Problem	45.0	33.1	−11.1	***	53.1	3.3	***	45.4	0.2
Severe problem	6.8	3.7	−5.6	***	7.1	0.3		7.3	0.5
Relations with family
Problem	36.0	39.2	3.1	***	45.8	4.0	***	36.6	0.2
Severe problem	16.6	22.4	7.1	***	26.8	4.8	***	16.8	0.0
Relations with friends
Problem	49.3	27.1	−20.6	***	53.1	1.6		49.9	0.5
Severe problem	11.4	5.1	−9.3	***	9.0	−1.7	*	12.2	0.5
Drug abuse
Problem	38.9	31.3	−7.3	***	46.2	2.9	**	37.8	−0.4
Severe problem	12.7	11.0	−2.3	*	19.1	3.4	***	13.9	0.8
Alcohol abuse
Problem	16.6	16.9	0.4		17.8	0.7		17.0	−0.4
Severe problem	17.3	23.5	7.5	***	32.1	6.6	***	17.4	0.8
Emotional well-being
Problem	11.0	16.5	8.0	***	24.0	6.4	***	11.0	−0.1
Severe problem	3.7	14.7	25.4	***	15.1	6.9	***	4.2	0.8
Thinking patterns
Problem	83.4	64.9	−22.7	***	86.4	1.7	*	83.0	−0.3
Severe problem	11.7	12.5	1.2		13.6	1.2		12.5	0.6
Attitude toward crime
Problem	40.5	30.2	−9.7	***	48.5	3.2	***	40.1	0.0
Severe problem	13.1	10.9	−3.1	***	10.7	−1.6		13.9	0.5
Total number of problems
0-2	6.8	24.2	30.6	***	0.3	−11.0	***	6.4	−0.6
3-5	38.9	34.4	−4.2	***	24.9	−6.4	***	38.0	−0.4
6-7	27.7	19.4	−8.6	***	31.3	1.6		28.2	0.3
8-11	26.6	22.0	−4.8	***	43.5	7.0	***	27.4	0.4

Note. CoVa = cognitieve vaardigheden; PSM = propensity score matching; RISc = Recidive Inschattings Schalen.

p < .05. **p < .01. ***p < .001.

Although the background characteristics of the first comparison group were more similar to the CoVa group than the pre-matching group, many characteristics were still statistically significantly different. The CoVa participants were younger and less often born in Morocco. Furthermore, the index case of the participants more often constituted a violent property crime or a drugs or traffic offense, while the offenders in Comparison Group 1 were more often convicted for violent or sexual offenses. Looking at their criminal careers, the CoVa participants had fewer previous contacts with the justice system. The CoVa participants were, however, younger at the time of their first conviction and spent a longer term in detention. In contrast, Comparison Group 1 showed more problems in several areas of functioning than the CoVa group. In general terms, the first comparison group seemed to consist of offenders with a higher average risk of reoffending than those in the CoVa group. We controlled for these differences when comparing the reconviction rates of both groups.

For the composition of Control Group 2, we used PSM based on entering the covariates in Table 1 into a logit model using CoVa treatment participation as the outcome variable. Figure 1 displays the distribution of the log of the propensity scores in a single imputation for the total pre-matching group and the CoVa group. It shows complete overlap of the propensity scores of the pre-matching group and the CoVa group, which means that using PSM allowed us to find a reasonable match for each CoVa participant.

Figure 1:

Common Support of Propensity Scores of CoVa and Pre-matching Group

The fourth column of Table 1 shows that, following matching, there were no significant differences left between the CoVa participants and Comparison Group 2. We therefore conclude that these groups were comparable after matching.

Although Comparison Groups 1 and 2 were extracted from the same population, the overlap was limited. A maximum of 71 individuals was included in both groups. The core strength of Comparison Group 2 is that this group was matched to the CoVa group on various important covariates in advance, while Comparison Group 1 and the CoVa group still differ on many of the background characteristics. Multiple regression models were thus needed to statistically control for these differences. However, the core positive feature of Comparison Group 1 is that persons with contraindications were removed from the sample. This was not feasible for Comparison Group 2, so, on this point, Comparison Group 1 is preferable. In sum, both the comparison groups have their own strengths which is why we decided to use both groups. Moreover, if we were to find similar results when comparing the treatment group to Comparison Groups 1 and 2, this would substantiate the findings on the effectiveness of CoVa. In that case, the outcome seems insensitive to the selection procedure of the comparison group.

Outcome Results

The outcome of the CoVa training was determined in three ways: by measuring the prevalence of reconvictions, the number of reconvictions, and the impact of the adjudicated reoffenses.

Reconviction Prevalence

Figure 2 depicts the failure curves of reconviction prevalence at different survival times. After 1 year, 32.1% of CoVa group participants had been reconvicted. This percentage had risen to 47.9 % and 56.3% after 2 and 3 years, respectively. The reconviction rates of Comparison Group 1 were a bit higher. After 1 year, 35.8% had been reconvicted, after 2 years, 49.5%, and after 3 years, 57.7%. In Comparison Group 2, the prevalence of reconviction was lower. Of the persons in this group, 31.3% were reconvicted within 1 year, 46.3 % within 2 years, and 55.8% within 3 years. These differences between the CoVa group and the comparison groups were not statistically significant. This did not change when we used a Cox regression analysis to control for differences between the CoVa group and Comparison Group 1 (see Table 2). The Exp(B) for program participation remained statistically nonsignificant.

Figure 2:

Reconviction Prevalence of the CoVa Group Versus Comparison Groups 1 and 2. The F Statistics for the Differences Between the CoVa Group and Comparison Group 2 Are Obtained by Combining the χ² From the Imputed Datasets

Table 2:

Multiple Regression Analyses CoVa Group Versus Comparison Group 1

	Reconviction prevalence			Reconviction frequency			Impact adjudicated offenses
	Cox regression			Negative binomial regression			Linear regression
Characteristics	Exp(B)	95% CI for Exp(B)	Significance	Exp(B)	95% CI for Exp(B)	Significance	Exp(B)	95% CI for Exp(B)	Significance
Demographic characteristics
Gender (male)	2.05	[1.43, 2.93]	***	1.57	[1.20, 2.05]	**	1.50	[1.15, 1.97]	**
Age	0.95	[0.94, 0.96]	***	0.96	[0.96, 0.97]	***	0.96	[0.95, 0.97]	***
Country of birth
The Netherlands	ref			ref			ref
Morocco	0.90	[0.66, 1.23]		0.85	[0.69, 1.05]		0.85	[0.65, 1.12]
Former Neth. Antilles. and Aruba	1.76	[1.35, 2.28]	***	1.42	[1.19, 1.69]	***	1.64	[1.31, 2.06]	***
Surinam	1.15	[0.86, 1.53]		1.05	[0.87, 1.28]		1.20	[0.95, 1.53]
Turkey	0.59	[0.39, 0.90]	*	0.69	[0.51, 0.94]	*	0.66	[0.47, 0.94]	*
Other Western	0.96	[0.61, 1.50]		0.81	[0.60, 1.11]		0.89	[0.61, 1.29]
Other non-Western	0.90	[0.66, 1.24]		1.01	[0.81, 1.24]		0.96	[0.74, 1.25]
Characteristics index case
Offense type
Violent	ref			ref			ref
Sexual	0.93	[0.52, 1.66]		0.75	[0.50, 1.13]		0.98	[0.63, 1.53]
Violent property	1.17	[0.94, 1.46]		1.13	[0.97, 1.31]		1.07	[0.89, 1.29]
Property	1.14	[0.91, 1.43]		1.16	[1.00, 1.34]		1.24	[1.01, 1.51]	*
Public order	0.76	[0.55, 1.04]		0.90	[0.71, 1.15]		0.82	[0.63, 1.07]
Drug	1.03	[0.79, 1.33]		0.92	[0.76, 1.10]		0.95	[0.76, 1.19]
Traffic	0.85	[0.40, 1.81]		0.79	[0.47, 1.33]		0.98	[0.51, 1.91]
Other	0.82	[0.54, 1.25]		0.84	[0.62, 1.12]		0.82	[0.58, 1.16]
Mean severity	0.99	[0.99, 1.00]	*	0.99	[0.99, 1.00]	**	1.00	[0.99, 1.00]
Setting (intramural)	1.16	[0.98, 1.37]		1.18	[1.05, 1.33]	**	1.21	[1.04, 1.39]	*
Criminal career characteristics
Conviction before age 18	0.90	[0.76, 1.07]		0.92	[0.82, 1.04]		1.00	[0.86, 1.16]
Mean previous convictions	1.94	[1.70, 2.21]	***	1.58	[1.44, 1.72]	***	1.64	[1.47, 1.82]	***
Mean severity last 4 years	1.00	[1.00, 1.01]	**	1.11	[1.05, 1.17]	***	1.01	[1.00, 1.01]	***
Problem areas (RISc)
Criminal history	1.02	[1.00, 1.05]		1.03	[1.01, 1.05]	**	1.03	[1.00, 1.05]	*
Housing	0.99	[0.96, 1.03]		1.00	[0.98, 1.03]		0.99	[0.96, 1.03]
Education and work	1.01	[0.99, 1.04]		0.99	[0.98, 1.01]		1.02	[1.00, 1.04]
Income	0.98	[0.95, 1.02]		1.02	[0.99, 1.04]		0.99	[0.95, 1.02]
Relations with family	1.01	[0.98, 1.05]		1.01	[0.98, 1.03]		1.02	[0.99, 1.05]
Relations with friends	1.01	[0.96, 1.05]		1.01	[0.98, 1.04]		1.02	[0.98, 1.05]
Drug abuse	1.00	[0.98, 1.03]		1.02	[1.00, 1.03]		1.01	[0.99, 1.03]
Alcohol abuse	1.05	[1.02, 1.08]	**	1.03	[1.01, 1.05]	**	1.05	[1.02, 1.07]	**
Emotional well-being	0.99	[0.95, 1.03]		0.99	[0.97, 1.02]		0.98	[0.95, 1.02]
Thinking patterns	1.02	[0.98, 1.06]		1.02	[0.99, 1.04]		1.02	[0.98, 1.05]
Attitude toward crime	1.01	[0.97, 1.05]		1.01	[0.98, 1.04]		1.01	[0.98, 1.05]
Program
CoVa participation	0.95	[0.79, 1.14]		0.85	[0.75, 0.96]	**	0.91	[0.77, 1.07]

Note. CoVa = cognitieve vaardigheden; CI = confidence interval; RISc = Recidive Inschattings Schalen.

p < .05. **p < .01. ***p < .001.

Reconviction Frequency

The CoVa group displayed a lower average frequency of reconvictions per year of no detention compared with both comparison groups. On an average, the CoVa participants were convicted for 0.51 new criminal cases per year not in detention, whereas Comparison Group 1 recidivated with 0.62 criminal cases and Comparison Group 2 with 0.54 criminal cases. The difference in frequency of the CoVa group and Comparison Group 2 was not statistically significant—t(df) = 35.45 = −1.12, p = .135—but the difference between the CoVa group and Comparison Group 1 was significant—t(df) = 684.772 = −2.75, p = .012. Since the CoVa group and Comparison Group 1 differed on background characteristics, this last result might be biased. For this reason, we next performed a negative binomial regression analysis. From the results in Table 2, it follows that, while controlling for differences in background characteristics, the effect of CoVa program participation was still statistically significant. This effect is comparable to a Cohen’s d of 0.10. According to Cohen (1988), this is a “very small” or “negligible” effect.

Impact of Adjudicated Repeat Offenses

The severity of the offenses was articulated in terms of sanction points and combined with the frequency to determine the impact (refer to the appendix). The average impact of repeat offenses per year of no detention for the CoVa group was 21.2 sanction points. Comparison Group 1 had an average of 23.6 sanction points and Comparison Group 2 an average of 24.3 sanction points. The difference between the CoVa group and Comparison Group 1 was not statistically significant—t(df) = 2,733 = −1.487, p = .137. In the additional multiple regression analysis, a similar result was found (see Table 2). However, the difference between the CoVa group and Comparison Group 2 was significant—t(df) = 332 = −2.30, p = .011. This effect is comparable with a Cohen’s d of 0.07.

Discussion

CoVa is a training program intended to improve cognitive skills and subsequently reduce recidivism among its participants. To evaluate the effectiveness of CoVa, we used a retrospective quasi-experimental design with two control groups and three outcome measures. Contrary to our expectations, the analyses showed no effects on the first outcome measure, the reconviction prevalence. However, statistically significant effects were found on the other outcome measures. The participants were reconvicted less frequently than the offenders in Comparison Group 1, and the impact score of their reoffenses was on average lower than that of Comparison Group 2. Although the comparison with two control groups led to a slightly different outcome, no contradictory results were found. The reconviction frequency in Comparison Group 2 and the impact of the reoffenses of Comparison Group 1 were also higher than in the CoVa group, although those differences were not statistically significant.

Our results are consistent with the analysis of Bosma (2017), which also did not show a significant treatment effect of CoVa based on the reconviction prevalence. Moreover, our results on the reconviction prevalence are consistent with the outcome of a substantial part of the research on ETS (Cann, 2006; Cann et al., 2003; Falshaw et al., 2003; Hollin et al., 2008; Palmer et al., 2015). These studies also did not find significant differences between all offenders allocated to a program and the comparison groups.

Although the nonsignificant results on the reconviction prevalence suggest that CoVa did not reduce the amount of persons with at least one reconviction, it does not necessarily mean that the program has no effect at all. If the process of desisting offending behavior is gradual, then the reconviction prevalence, an outcome measure which focuses only on the first reconviction, is too crude to measure program effects. For that reason, we also evaluated the post-treatment reconviction frequency and the impact of the adjudicated repeat offenses. These measures showed some significant positive effects (with regard to one control group), but the corresponding effect sizes are very small (Cohen, 1988). In other words, the current study did not reveal any large positive effects, thus suggesting that CoVa did not substantially reduce recidivism.

When interpreting the results of this research, it is important to consider some limitations. First, selection bias might have affected the outcome of this study. Because there was no random assignment to the treatment conditions, we cannot rule out unmeasured differences between treatment and control groups that may have hampered the comparison on the outcome variables. Nevertheless, in the current study, we used two different selection procedures for the composition of the control groups and more than 25 static and dynamic risk factors to match the offenders in the treatment and control groups. Hence, the risk of potential selection bias was reduced to a minimum.

Second, the use of convictions as an outcome measure is a limitation as well. It is well known that not every new crime leads to a conviction (e.g., Farrington, Auty, Coid, & Turner, 2013). Hence, it is interesting to see whether the same effects would be found if, for instance, self-report or police data are used in an evaluation about CoVa.

Third, a methodological limitation is the lack of information on the exact nature of the control conditions. To determine the effect of CoVa on reconviction, we contrasted the results on the outcome variables for the participants with those of offenders who received “treatment as usual” (TAU). A TAU design allows for only finding a relative treatment effect (Löfholm, Brännström, Olsson, & Hansson, 2013). Hence, the size of an effect depends on the level of treatment in the control groups, which is information we do not have. For instance, we are unaware of the proportion of persons who also participate in some sort of other treatment program and, if they do, the dosage of that program. In general, though, the level of TAU in the Netherlands can be considered quite high. The CoVa training might be the behavior modification program most frequently applied, but it is definitely not the only one. Other nationwide behavior programs for adults focus on employability or budgeting skills, housing, and the prevention of alcohol abuse and violence, including domestic violence (Fischer et al., 2012). The Probation Service also implements a great variety of locally organized programs that focus on offender rehabilitation and the prevention of recidivism. Moreover, every Dutch prison offers reintegration activities which aim at providing a regular income, proper housing, a valid ID card, health care, and guidance to help relieve financial debts (Beerthuizen, Beijersbergen, Noordhuizen, & Weijters, 2015). Following their release, exprisoners can receive further guidance and help with their reintegration in society through living in halfway houses (Van Wingerden, Alberda, Moerings, Wartna, & Van Wilsem, 2010).

As mentioned above, we do not have detailed information on interventions received by the offenders in the two comparison groups. However, as the default treatment level in the Netherlands is high, the TAU design used in this study may be an important reason for not finding large positive treatment effects for the CoVa training program. A previous meta-analysis of recidivism research on penal interventions in the Netherlands produced an average effect that is even smaller than the significant effects found in the current study (Wartna, Alberda, & Verweij, 2013). Although the effect sizes found in the current study may be very small in terms of the well-known classification of Cohen (1988), they can be termed “normal” in relation to the outcome of other Dutch evaluation studies, which in general could suffer from the fact that there is no such thing as a “nontreatment” condition in the Netherlands.

Conclusion

The current study adds substance to the body of literature regarding evaluation research of the CoVa treatment program. This article represents the first large-scale study examining the effectiveness of CoVa in terms of reconviction. Furthermore, the current study used both static and dynamic risk factors to adequately compose control groups, and did not focus solely on reconviction prevalence, as is often the case in effect studies on treatment programs regarding criminal behavior.

The results showed that there were no differences in reconviction prevalence, but participants in the CoVa group were less frequently reconvicted than the offenders in one comparison group, whereas the average impact of their reoffenses was lower than those of the other comparison group. Although the effects were classified as very small, they do not differ much from the average effects found in other research into penal measures in the Netherlands. Our overall conclusion is that, although there are weak indications of the program being effective in some areas of recidivism, it has yet to be proven that the CoVa program substantially reduces recidivism among adult offenders with established deficits on cognitive skills.

Footnotes

Appendix

Acknowledgements

The authors would like to thank K. Beijersbergen for her helpful comments and suggestions during the writing process. Parts of this paper have previously been published in a Research and Documentation Centre (WODC) report, available in Dutch at , which was financed by the Dutch Ministry of Security and Justice. The current paper does not necessarily reflect any views, opinions, or policies of either the WODC or the Ministry of Security and Justice. The authors are solely responsible for the contents of the current paper.

Notes

Suzan Verweij, MSc, works as a researcher at the Research and Documentation Centre (WODC) of the Dutch Ministry of Security and Justice. Her research interests include prisoner reintegration, desistance from crime, methods in recidivism research, and the effectiveness of judicial interventions and sanctions.

Bouke S. J. Wartna, PhD, works as senior researcher at the Research and Documentation Centre (WODC) of the Dutch Ministry of Security and Justice. He is program manager of the Dutch Recidivism Monitor. His main research has been into the effectiveness of penal interventions.

Nikolaj Tollenaar, PhD, works as a researcher at the Research and Documentation Centre (WODC) of the Dutch Ministry of Security and Justice. His research interests include research methodology, recidivism prediction modeling, chronic offenders, and quasi-experiments in the criminal justice domain.

Marinus G. C. J. Beerthuizen, PhD, works as a researcher at the Research and Documentation Centre (WODC) of the Dutch Ministry of Security and Justice. His research interests include moral development, videogame playing behavior in relation to juvenile delinquency, and methods to operationalize criminal severity.

References

Andrews

D. A.

Bonta

(2003). The psychology of criminal conduct (3rd ed.). Cincinnati, OH: Anderson.

Andrews

D. A.

Zinger

Hoge

R. D.

Bonta

Gendreau

Cullen

F. T.

(1990). Does correctional treatment work? A clinically relevant and psychologically informed meta-analysis. Criminology, 28, 369-404.

Antonowicz

D. H.

Ross

R. R.

(1994). Essential components of successful rehabilitation programs for offenders. International Journal of Offender Therapy and Comparative Criminology, 38, 97-104.

Beerthuizen

M. G. C. J.

Beijersbergen

K. A.

Noordhuizen

Weijters

(2015). Fourth measurement of the monitor of aftercare for former prisoners. Den Haag, The Netherlands: Research and Documentation Centre.

Beerthuizen

M. G. C. J.

Wartna

B. S. J.

Verweij

Tollenaar

(2015). De misdrijf-straf index: Op weg naar een maat voor de ernst van delicten afgeleid van de afdoening van strafzaken [The crime-penalty index: Towards a measure for the seriousness of offenses derived from sentences of criminal cases]. Den Haag, The Netherlands: Research and Documentation Centre.

Biderman

A. D.

Reiss

A. J.

(1967). On exploring the “dark figure” of crime. The Annals of the American Academy of Political and Social Science, 374, 1-15.

Bosma

A. Q.

(2017). Targeting recidivism: An evaluation study into the functioning and effectiveness of a prison-based treatment program. Leiden, The Netherlands: Meijers Research Institute.

Bushway

Piquero

A. R.

Broidy

L. M.

Cauffman

Mazerolle

(2001). An empirical framework for studying desistance as a process. Criminology, 39, 491-516.

Buysse

Loef

(2012). Effectiveness of the cognitive skills treatment (CoVa) for offenders. Amsterdam, The Netherlands: DSP-groep.

10.

Cann

(2006). Cognitive skills programs: Impact on reducing reconviction among a sample of female prisoners (Home Office Research Findings No. 276). London, England: Home Office.

11.

Cann

Falshaw

Nugent

Friendship

(2003). Understanding what works: Accredited cognitive skills programs for adult men and young offenders (Home Office Research Findings No. 226). London, England: Home Office.

12.

Clarke

Simmonds

Wydall

(2004). Delivering cognitive skill programs in prison: A qualitative study (Home Office Research Findings No. 242). London, England: Home Office.

13.

Cohen

(1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.

14.

Cornet

L. J. M.

Van der Laan

P. H.

Nijman

H. L. I.

Tollenaar

De Kogel

C. H.

(2016). Does a cognitive skills training program for prisoners affect neurocognitive functioning and heart rate activity? Criminal Justice and Behavior, 43, 1481-1504.

15.

Cox

D. R.

(1972). Regression models and life-tables (with discussion). Journal of the Royal Statistical Society, Series B: Statistical Methodology, 34, 187-220.

16.

Daly

Bordt

R. L.

(1995). Sex effects and sentencing: An analysis of the statistical literature. Justice Quarterly, 12, 141-175.

17.

Falshaw

Friendship

Travers

Nugent

(2003). Searching for “what works”: An evaluation of cognitive skills programs (Home Office Findings No. 206). London, England: Home Office.

18.

Farrington

D. P.

(2001). The causes and prevention of violence. In Shepherd

J. P.

(Ed.), Violence in health care (2nd ed., pp. 1-27). Oxford, UK: Oxford University Press.

19.

Farrington

D. P.

Auty

K. M.

Coid

J. W.

Turner

R. E.

(2013). Self-reported and official offending from age 10 to age 56. European Journal on Criminal Policy and Research, 19, 135-151.

20.

Ferwerda

Van Wijk

Arts

Kuppens

(2009). CoVa volgens plan? Een vooronderzoek naar de mogelijkheden en reikwijdte van een effectonderzoek van de cognitieve vaardigheden [CoVa according to plan? A preliminary effect study on the possibilities and scope of the cognitive skills training program]. Arnhem, The Netherlands: Bureau Beke.

21.

Fischer

T. F. C.

Captein

W. J. M.

Zwirs

B. W. C.

(2012). Correctional treatment programs for adult offenders: Current state of affairs and possibilities for innovation. Meppel, The Netherlands: Boom Lemma uitgevers.

22.

Friendship

Blud

Erikson

Travers

(2002). An evaluation of cognitive behavioural treatment for prisoners (Home Office Findings No. 161). London, England: Home Office.

23.

Gendreau

Andrews

D. A.

(1990). Tertiary prevention: What a meta-analysis of the offender treatment literature tells us about “what works.” Canadian Journal of Criminology, 32, 173-184.

24.

Gobett

M. J.

Sellen

J. L.

(2014). An evaluation of the HM prison service “thinking skills program” using psychometric assessments. International Journal of Offender Therapy and Comparative Criminology, 58, 454-473.

25.

Heitjan

D. F.

Little

R. J.

(1991). Multiple imputation for the fatal accident reporting system. Applied Statistics, 40, 13-29.

26.

Hill

J. L.

Reiter

J. P.

Zanutoo

E. L.

(2004). A comparison of experimental and observational data analyses. In Gelman

Meng

X. L.

(Eds.), Applied Bayesian modelling and causal inference from incomplete data perspectives (pp. 51-60). New York, NY: John Wiley.

27.

Hollin

C. R.

McGuire

Hounsome

J. C.

Hatcher

R. M.

Bilby

C. A. L.

Palmer

(2008). Cognitive skills behavior programs for offenders in the community: A reconviction analysis. Criminal Justice and Behavior, 35, 269-283.

28.

Howard

Clark

Garnham

(2003). Evaluation and validation of the Offender Assessment System (OASys). London, England: OASys Central Research Unit.

29.

Kaplan

E. L.

Meier

(1958). Nonparametric estimation from incomplete observations. Journal of the American Statistical Association, 53, 457-481.

30.

Löfholm

C. A.

Brännström

Olsson

Hansson

(2013). Treatment-as-usual in effectiveness studies: What is it and does it matter? International Journal of Social Welfare, 22, 25-34.

31.

Long

J. S.

Freese

(2001). Regression models for categorical dependent variables using Stata. College Station, TX: Stata Press.

32.

Ministry of Justice. (2005). Gedragsinterventies: Programma terugdringen recidive [Behavioral interventions: Prevention of Recidivism Program]. Den Haag, The Netherlands: Author.

33.

Palmer

E. J.

Hatcher

R. M.

McGuire

Hollin

C. R.

(2015). Cognitive skills programs for female offenders in the community: Effects on reconviction. Criminal Justice and Behavior, 42, 345-360.

34.

Poort

Bosker

Agema

(2011). De reclassering en licht verstandelijk beperkte cliënten [The Probation Service probationers with intellectual disabilities]. Proces, 90, 262-273.

35.

Reclassering Nederland. (2010). Handleiding RISc versie 3, deel 2: Indicatiestelling [Manual RISc version 3, part 2: Indication]. Utrecht, The Netherlands: 3 Reclasseringsorganisaties.

36.

Redondo

Sánchez-Meca

Garrido

(1999). The influence of treatment programs on the recidivism of juvenile and adult offenders: An European meta-analytic review. Psychology, Crime & Law, 5, 251-278.

37.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41-55.

38.

Ross

R. R.

Fabiano

E. A.

(1985). Time to think: A cognitive model of delinquency prevention and offender rehabilitation. Johnson City, TN: Institute of Social Sciences and Arts.

39.

Rubin

D. B.

(1978). Multiple imputations in sample surveys: A phenomenological Bayesian approach to nonresponse. Proceedings of the Survey Research Methods Section, American Statistical Association, 20-34. Retrieved from https://ww2.amstat.org/sections/srms/Proceedings/papers/1978_004.pdf

40.

Sadlier

(2010). Evaluation of the impact of the HM prison service enhanced thinking skills program on reoffending: Outcomes of the Surveying Prisoner Crime Reduction (SPCR) sample. London, England: Ministry of Justice.

41.

Schenker

Taylor

J. M. G.

(1996). Partially parametric techniques for multiple imputation. Computational Statistics & Data Analysis, 22, 425-446.

42.

Steffensmeier

Ulmer

Kramer

(1998). The interaction of race, gender, and age in criminal sentencing: The punishment of being young, black and male. Criminology, 36, 763-798.

43.

Tierolf

(2007). Evaluation report of the effects of the CoVa training course of the Dutch Probation and After-care Service. Utrecht, The Netherlands: Verwey Jonker instituut.

44.

Tong

L. S. Y.

Farrington

D. P.

(2006). How effective is the “reasoning and rehabilitation” program in reducing reoffending? A meta-analysis of evaluations in four countries. Psychology, Crime & Law, 12, 3-44.

45.

Travers

Mann

R. E.

Hollin

C. R.

(2014). Who benefits from cognitive skills programs? Differential impact by risk and offense type. Criminal Justice and Behavior, 41, 1103-1129.

46.

Travers

Wakeling

H. C.

Mann

R. E.

Hollin

C. R.

(2011). Reconviction following a cognitive skills intervention: An alternative quasi-experimental methodology. Legal and Criminological Psychology, 18, 48-65.

47.

Van Buuren

Boshuizen

H. C.

Knook

D. L

. (1999). Multiple imputation of missing blood pressure covariates in survival analysis. Statistics in Medicine, 18, 681-694.

48.

Van der Knaap

L. M.

Alberda

D. L

. (2009). Predictive validity of the Recidivism Risk Assessment Scales (RISc). Den Haag, The Netherlands: Research and Documentation Centre.

49.

Van der Knaap

L. M.

Leenarts

L. E. W.

Nijssen

L. T. J

. (2007). Psychometric qualities of the Dutch Risk Assessment Scales (RISc): Inter-rater reliability, internal consistency and concurrent validity. Den Haag, The Netherlands: Research and Documentation Centre.

50.

Van Wingerden

Alberda

D. L.

Moerings

Wartna

B. S. J.

Van Wilsem

. (2010). Recidive en nazorg: Onderzoek onder oud-bewoners van Exodus, DOOR, Moria & Ontmoeting [Recidivism and after-care: Research among old habitants of Exodus, Door, Moria and Ontmoeting]. Den Haag, The Netherlands: Boom Juridische uitgevers.

51.

Ward

Melser

Yates

P. M.

(2007). Reconstructing the Risk–Need–Responsivity model: A theoretical elaboration and evaluation. Aggression and Violent Behavior, 12, 208-228.

52.

Wartna

B. S. J.

(2009). Lapse into old mistakes: Measuring recidivism to help determine the rehabilitative effect of penal interventions. Den Haag, The Netherlands: Boom Juridische uitgevers.

53.

Wartna

B. S. J.

Alberda

D. L.

Verweij

(2013). What works in the Netherlands and what doesn’t? A meta-analysis of Dutch recidivism research into the effects of criminal justice interventions. Den Haag, The Netherlands: Boom Lemma uitgevers.

54.

Zebel

Alberda

D. L.

Wartna

B. S. J.

(2014). Recidivism following probation or a community service order: Overview of the relapse into crime of adult offenders dealt with by the Dutch Probation Services in 2002-2009. Den Haag, The Netherlands: Boom Lemma uitgevers.