Evaluation of European Domestic Violence Perpetrator Programmes: Toward a Model for Designing and Reporting Evaluations Related to Perpetrator Treatment Interventions

Existing Evidence From Europe

Arias et al. (2013) examined 19 Spanish- and English-language studies measuring recidivism rates of programme completers and found that while perpetrator intervention can have a positive (but nonsignificant) effect on recidivism, some treatments may actually have considerably negative effects. Feder et al. (2008) reviewed 31 experimental or quasi-experimental outcome studies and found no differences in effectiveness between Duluth-based and other cognitive-behavioural interventions, suggesting that such interventions had minimal impact beyond the effect of being arrested. Hence, evidence from reviews that include European evaluations supports the findings reported elsewhere, that is, that evaluations of domestic abuse perpetrator programmes are methodologically inconsistent, and thus, the evidence remains inconclusive. Focussing on only European evaluations, the recent review by Akoensi et al. (2013) suggests that while evaluations showed various positive changes (e.g., reductions in abusive behaviour and psychological improvements among perpetrators), the methodological quality of European studies was insufficient to develop strong conclusions or estimate an effect size, concluding that the evaluation of DVPPs in Europe must be improved.

Existing reviews and meta-analyses, also those involving European studies, mainly include only experimental or quasi-experimental studies measuring attitudinal and behavioural change (in particular recidivism related to physical abuse; Akoensi et al., 2013; Arias et al., 2013; Feder et al., 2008), and include mainly English-language studies (Feder et al., 2008; NICE, 2014) and evaluations published in peer reviewed journals, thus excluding studies that would be classed as “grey” and/or other nonpublished material (Feder et al., 2008). Existing reviews and meta-analyses have also left a number of questions unanswered, for example, what are the broader impacts of perpetrator interventions for women/victims and their children?; what are the motivations of both completers and of dropouts and how does this affect behavioural and/or attitudinal change?; and what elements or type of intervention affect positive change or “success”? (e.g., Bowen, Gilchrist, & Beech, 2004; Feder et al., 2008). Thus, in this study, we started out by wondering what the published evidence base might be missing by ignoring the larger body of research relating to European DVPPs, often in the form of grey literature, studies not published in English, or evaluations using different designations to measure a wider range of potential outcomes.

The Study

This article is based on the findings from the European Commission–funded project “IMPACT: Evaluation of European Perpetrator Programmes” (Daphne III Programme), which primarily aimed to fill the existing knowledge gap about the evaluation of European DVPPs with a view to identifying the possibilities and challenges of a harmonised, multicountry evaluation methodology that could be used by European perpetrator programmes in future. One of the main objectives of the project was to provide an overview and analysis of all evaluations relating to European DVPPs, examining the range of studies, with particular emphasis on the methods, inputs, outputs, and outcome measures used.

Study Identification and Selection

To overcome issues associated with publication bias, and to capture as many of the European evaluation studies as possible, we employed much wider inclusion criteria than used for existing reviews/meta-analyses. Thus, our review included evaluations of perpetrator intervention in Europe, both published (formally issued or controlled by a commercial publisher) and “grey” (reports not widely distributed or commonly used in abstracts or indexes, for example, reports produced or published by universities or academic research units, government reports, programme/funder reports, and PhD studies) produced between January 1999 and May 2014. The studies could be written in any European Union (EU) or EU accession country language, apply any outcome measures, and be of any type and design (including process and/or implementation; experimental, quasi-experimental, nonexperimental, quantitative, and qualitative). Eligible studies were identified via the following: searches of existing published reviews/meta-analyses; an updated systematic search to 2015 of the same electronic databases used in the review by Akoensi et al. (using the same search strings ¹ ); two separate European-wide surveys of perpetrator programmes carried out by the European “Work With Perpetrators” Network in 2007/2008 and by the IMPACT project in 2013 (Geldschläger et al., 2014); further direct contact with European perpetrator programme networks, study authors, and experts; and additional searches of specialist domestic violence websites (see Figure 1 for details of the study identification process).

OPEN IN VIEWER

Figure 1.

Study identification process.

Data Extraction and Analysis

The evaluations were divided into five “regions” of origin (Central, Eastern, Northern, Southern, and Western Europe) so that searches, translation, and data extraction could be conducted by a core review team from the IMPACT project (Hester, Lilley, O’Prey, & Budde, 2014) aided by members of the wider (multicountry and multilanguage) project team. Each study was assessed by at least two members of the team. The extraction process was systematized using a specifically designed template to capture detailed information from each of the studies (translated and recorded in English), including details of the intervention (e.g., theoretical paradigm, structure, and the wider context within which it is set), the evaluation (e.g., type/purpose, design, focus, limitations, and results), and the sample profile at different stages of the evaluation process (e.g., what data were being collected, when, how, and from who). Analysis explored a number of avenues, including relationships within and between studies of different designs, the extent to which “‘regionality” was relevant, and the different ways that domestic violence (and therefore perpetrator intervention) was conceptualised across Europe as indicated by the evaluation design and primary focus.

Experimental/Quasi-Experimental Studies

Only two of the 60 evaluations employed a randomised controlled trial (RCT) design, one conducted in a prison setting and one in a substance misuse clinic. The prison-based RCT, originating from Spain and conducted by Rodríguez-Espartal and Lopez-Zafra (2013), randomly assigned 36 male prisoners convicted for domestic violence–related crimes into two treatment groups - cognitive-behavioural therapy (n = 11) and, emotional therapy (n = 13) - and a control (non-treatment) group (n = 12). Self-reported change postintervention, collected via a battery of psychometrics, showed a greater decrease in distorted thoughts about women and the use of violence and an increase in the expectations about change in inmates who received emotional treatment (no change was found in other variables among those receiving treatment although there was an increase in negative results in the control group). However, no follow-up and no attrition were reported, and the study excluded inmates with psychopathology or physical disability, those receiving treatment for alcohol/substance misuse, or those with prison sentences shorter than 12 months.

The other RCT, an evaluation of the Dutch “Integrated treatment for substance abuse and partner violence” (Kraanen, Vedel, Scholing, & Emmelkamp, 2013), compared two individual treatments attended by patients at a substance misuse treatment clinic (as opposed to a specific perpetrator programme) who reported repeated intimate partner violence (IPV): the I-StoP (concurrently addressing substance misuse and IPV) and CBT-SUD+ (a manualised CBT treatment usually used for substance misuse in the Netherlands). Self-reported substance use and IPV perpetration measured before, during, and after treatment (completers and the intention-to-treat [ITT]) showed significant pre–post improvements in substance use and IPV perpetration. There were no differences in outcome between conditions. As completers of both treatments almost fully abstained from IPV in the 8 weeks before the end of treatment, and as it is more cost- and time-effective to implement CBT-SUD+ than I-StoP, the authors suggested IPV perpetrators should be treated in substance abuse treatment with CBT-SUD+. The study, however, suffered a high attrition rate, and results were largely based on self-reported perpetration among a small sample of completers (no follow-up or analysis of dropouts was conducted).

Fourteen studies (outcome = 13, process = 1) employed various quasi-experimental designs, comparing intervention outcomes between either different sites (e.g., Quintas, Fonseca, Sousa, & Serra, 2012), different settings (e.g., Novo, Fariña, Seijo, & Arce, 2012), different interventions (e.g., Boira, Lόpez del Hoyo, Tomás-Aragonés, & Gaspar, 2013), different cohorts of men (e.g., Bowen, Gilchrist, & Beech, 2005), or different offender populations (e.g., Echauri, Fernández-Montalvo, Martínez, & Azkarate, 2013). These studies (10 of which originated from Spain) used a battery of psychometric instruments to measure pre–post changes in psychopathological and psychosocial characteristics, such as hostility, anger, depression, anxiety, self-esteem, persecutory ideas, attitudes toward women and the use of violence, and levels of maladjustment to assess the extent to which the participants’ current problems affect other areas of their life (e.g., Echauri Tijeras, 2010; Echeburúa, Fernández-Montalvo, & Amor, 2006; Novo et al., 2012; Quintas et al., 2012). Providing some of the most statistically significant results, these studies suggest that domestic violence perpetration can be successfully “treated,” showing significant improvements in irrational beliefs about women and violence or significant decreases in psychopathological symptomatology (e.g., Diranzo, Murillo, & Minana, 2012; Echauri et al., 2013; Echeburúa & Fernández-Montalvo, 2009). However, their focus on obtaining men’s self-reports (in part as a result of Spanish evaluations not being permitted, by law, to validate any outcome measures with data from women/partners) and criminal justice data means they may suffer from inherent biases, including perpetrator denial, minimisation, and desired responding (e.g., Gadd, 2004; Gondolf, 2002). Also, basing “success” on levels of officially reported/recorded incidents of physical violence is problematic not only because police-recorded incidents may actually increase in the immediate term as women/victims feel more empowered to report (Gondolf, 2002; Hester & Westmarland, 2005) but also because emotionally controlling behaviours of the perpetrator may continue—or even increase—alongside a reduction in physical violence (e.g., Dobash, Dobash, Cavanagh, & Lewis, 1999), and thus, a reduction in physical violence is often insufficient for some victimised women to feel at ease and restore the freedom that living with coercive control involves (Kelly & Westmarland, 2015). These studies therefore have limitations in accurately reflecting changes in any controlling/coercive behaviours, repeat victimisation, or whether women/partners or their children feel safe/safer. Study samples tended to consist of participants mandated by the courts (n = 2,892) with strict selection criteria excluding perpetrators presenting with more complex problems, including mental health and substance misuse. This raises issues with generalisability of the results as it is suggested that men assigned to court-mandated programmes present with little or no motivation to change their abusive behaviour and 30% to 40% will just “go through the motions” while on the programme (Eckhardt, Holtzworth-Munroe, Norlander, Sibley, & Cahill, 2008).

Nonexperimental Studies

The majority of studies identified (k = 31) were of a nonexperimental design (outcome = 14, process = 2, both outcome and process = 15) involving 3,283 programme participants. Most employed a pre–post design (k = 25) with only seven studies using a follow-up period of more than 12 months (Calvo, Lecumberri, & Burset, 2011; Diranzo et al., 2012; Gabriel, von Wolffersdorff & für Soziales, 2006; Hofinger & Neumann, 2008; Lila, Oliver, Galiana, & Gracia, 2013; Perez-Ramírez, Giménez-Salinas Framis, & de Juan Espinosa, 2013; Power & Clarke, n.d.). Studies in this group tended to use a combination of quantitative and qualitative design with 13 studies triangulating outcome measures using data from women/partners and/or their support workers and referring professionals. With regard to the source of data, “who says” may be important for understanding how intervention “success” (perpetrator change) is being determined and how victims/survivors may actually benefit from their abusive partner’s participation in treatment.

Studies (particularly those originating from the United Kingdom) also tended to employ a wider range of outcome measures than used in the experimental studies, collecting data from both within the programme and across the potential community response. This included data on social-level changes in attitudes toward women and violence against women, levels of and resilience to repeat victimisation, quality of life (of both the perpetrator and the victim/partner), feelings of safety and well-being of women/partners (and their children), and levels of parenting stress. Evaluating the efficacy of the U.K. community–based Domestic Violence Intervention Project (DVIP) (Price, Rajagopalan, & Donaghy, 2009) measured recidivism, repeat victimisation and feelings of safety using police data, internal programme data, and women/partner reports at 3, 6, and 18 months’ follow-up (men’s self-reported levels of violence were not deemed robust to evidence a reduction in repeat victimisation). Results showed that perpetrators’ involvement with the programme led to 70% of women/partners reporting no further violence (and the remainder reporting less severe or less frequent violence), 65% felt safer or much safer, 69% reported that their children were safer, and 93% reported an improvement in their quality of life. Another U.K. evaluation, of the community-based “Repair” programme -a project of the Against Domestic Violence and Abuse in Devon (ADVA) partnership (ADVA & Sue Penna Associates, 2009), found a strong decrease in risk of reabuse among programme completers (corroborated by women/partner reports) and significant psychological improvement among perpetrators and among women/partners and children. The combined results of this body of nonexperimental research indicate the potential for largely positive outcomes for women/partners and their children (improvements in well-being, quality of life, and resilience to repeat victimisation), supporting the argument for a more nuanced definition of intervention “success” (see also Kelly et al., 2015; Westmarland, Kelly, & Chalder-Mills, 2010) and for the use of women/partner reports in evaluation. However, while the use of women/partner reports has long been proposed as a valid and reliable measure of outcome or ”success” (e.g., Gondolf, 2002; Mullender & Burton, 2000) it is important to point out that women/partner accounts of perpetrator behaviour or attitudinal change can only be seen as reliable and/or valid if those women/partners asked are actually in a position to reliably assess change, that is, are still in a relationship or have regular contact with the perpetrator (Kelly et al., 2015). Therefore, on balance, a combination and comparison of reports from both men and women/partners will yield new insights, including on how some men change and others do not (Kelly et al., 2015). This group of studies did, however, lack the use of control groups and/or adequate follow-up and were generally based on small sample sizes, restricting wider application of their findings due to the associated issues of generalisability, validity, and reliability.

Qualitative Studies

Twelve of the 60 evaluations employed an entirely qualitative design (outcome = 2, process = 4, both outcome and process = 6) involving 411 programme participants. Two of these 12 studies focussed on investigating the therapeutic process and role of facilitation as effective vehicles for change. Recognising that producing and facilitating change is a complex therapeutic task, evaluation of the Finnish Jyväskylä Model for Male Batterers took a constructionist and narrative approach to investigating treatment outcomes and what counted as ”success.” This addressed the different therapeutic strategies used to deal with perpetrators’ construction of violence, the discourses used by perpetrators within group therapy to explain or justify their abusive behaviour, and how both therapists and perpetrator participants deal with or negotiate over issues of violence in the context of therapeutic intervention. Measuring facilitator–participant interaction, design, and use of different therapeutic strategies, evaluators found that participants often constructed themselves as a victim, and thus, a central area of conversation between the therapist and the perpetrator was negotiating the issue of victimhood (Holma, Partanen, Wahlstrom, Laitila, & Seikkula, 2006). The results of this study suggested a need for different narratives to emerge within the therapeutic process to increase responsibility assumption while, at the same time, enabling a sense of agency among perpetrators and the possibility of acting nonviolently. The U.K. study by Garfield (2005) investigated the therapeutic process across three perpetrator intervention groups each with differing approaches, to explore the impact of programme length, duration, and facilitation quality on therapy outcomes. Results suggested that the quality of therapeutic alliance, whether or not deliberately facilitated, and the maintenance of that alliance in terms of group health and duration of the group were together predictive of integration of learning from group work into participants’ lives. Combined, the studies by Holma et al. and Garfield suggest that the role and quality of facilitation is a “powerful catalyst” for positive change among participants. These findings are further supported by the recent multisite evaluation of DVPPs in the United Kingdom by Kelly and Westmarland (2015) which found it was the input from facilitators that made the group context one that was conducive to change. Despite lacking the strengths of experimental evaluation methodology, and while largely excluded from published systematic evidence reviews, the qualitative studies we reviewed indicate the potential for therapeutic intervention to create positive change. They highlight the importance of facilitation quality in programme success (Garfield, 2005) and illustrate how therapists must consider new or different therapeutic discourses regarding masculinities to help perpetrators think about how they behave, and thus facilitate change by encouraging men to take responsibility for their violence while sensitively introducing the possibility of learning new identities (Holma et al., 2006; Partanen, 2008). A focus on the role and quality of programme facilitation would thus contribute to a deeper and more meaningful understanding of how DVPPs work in terms of creating change.

Who Is Participating?

Existing evidence suggests that men who are resistant to change tend to make up the majority of programme clients (Eckhardt et al., 2008) and that treatment noncompliance is associated with recidivism (Bennett & Williams, 2011). But domestically violent men will enter intervention programmes with different motivations, or at different stages of change, and are thus not uniform in their readiness to change (e.g., Daniels & Murphy, 1997; Eckhardt et al., 2008; Simmons, Lehmann, & Cobb, 2008; Kelly et al., 2015; Murphy & Maiuro, 2009). When evaluating DVPPs, attention therefore must be paid to motivation and what stage of change the participants under investigation are at. Sociodemographics, mental health, and substance misuse may also have a role to play both in treatment compliance and in change (e.g., Aldarondo & Sugarman, 1996). Of the 60 European evaluations we reviewed, only six had a particular focus on investigating what subgroups of men might have higher success in changing their behaviour (based on motivation and their socioeconomic and other characteristics) and 10 of the 60 studies reported some level of comparative analysis of completers and noncompleters. Evaluation of a (both voluntary and mandated) community-based DVPP in Austria (Kraus, 2012) identified four different subgroups of participants, comparing programme “completers” with those “not-admitted,” “dropouts,” and those “excluded.” The evaluation compared men who continued their violent behaviour with those who ceased, and compared men with clinically significant personality profiles against those presenting with “normal” personality profiles (with regard to violent behaviour). Overall, programme completers had the lowest rates of recidivism—and tended to be court-referred, and socially more adapted with “normal” psychological profiles compared with men who dropped out. A pre–post reduction in violence and increase in quality of life were reported by men and their women/partners. Those who continued their violent behaviour were more likely to be married and to have experienced childhood violence at the hands of their parents. Results were based on a mix of self-report, official data, and female/partner reports, although the study reported high attrition of female partners by the end of the treatment period. In a Spanish evaluation, Lila et al. (2013) used a battery of psychometric instruments to measure pre–post change among 212 male domestic violence offenders court-ordered to attend a CBT programme. Self-reported data showed that those most likely to experience change in recidivism risk, perceived severity, and responsibility assumption for their abusive behaviour were younger and had lower alcohol consumption, shorter sentences, lower impulsivity, and a higher degree of life satisfaction, community participation, and self-esteem. The study reported no attrition.

Subirana-Malaret and Andres-Pueyo (2013) conducted an ex post facto analysis of motivations and perceptions of men attending a voluntary, community-based programme in Spain between January 2001 and April 2008. Case files and interviews with participants explored a range of issues with regard to motivation and desire to continue treatment across six time points during the intervention. Results suggested that proactive measures improved retention and delayed dropout, but all observations were statistically insignificant. The authors concluded that socioeconomic factors were not a good predictor of adherence, and participants with “external” motivation, that is, with court or other mandate, were most likely to drop out of intervention. However, these findings contradict other studies that suggest that criminal justice sanctions can act as a lever or “incentive” to participate and positively affect adherence and longer term change in motivation (Dobash et al., 1999; Beckmann & Hagemann-White, 2004) or that found no difference in outcomes between those with “internal” and “external” motivation (e.g., Barz & Helfferich, 2006). Without data on motivation or stage of change, it is difficult to understand these contradictions (Hester et al., 2006; Sheehan, Thakor, & Stewart, 2013).

While the studies outlined above provide crucial information regarding who is actually participating/receiving treatment and who is not, who is completing, and who may be more “treatment resistant,” they tell us more about adherence to treatment than the actual situational factors underlying behavioural/attitudinal change and, as such, do not contribute directly to the evidence base about which elements or types of treatment are more successful at creating change, which is obviously also key to understanding programme effectiveness.

Limitations and Challenges of European Evaluations

Our review of European evaluations of DVPPs highlighted a number of methodological issues that transcended the different studies. In addition to design limitations (e.g., the general lack of control group design), other key problems were found relating to the reporting of information about the sample, attrition, and points of time used to collect data. In terms of reporting who the participants were and referral pathways, again the type of information collected and/or reported varied greatly across the studies. Sociodemographic data and referral route were collected/reported at intake or programme start in 24 and 20 studies, respectively, with only four studies reporting the same information for those who completed (Bowen et al., 2008; Beckmann & Hagemann-White, 2004; Lorenz & Bigler, 2013; Power & Clarke, n.d.). Who dropped out and why was reported in only one in 10 studies (k = 6; for example, Echeburúa et al., 2006; Milner & Singleton, 2008; Tejerina & Martínez, 2011). Accurate reporting of attrition is important to enable inferences about statistical power and the ability to generalise findings to wider populations. However, across the European evaluations, attrition rates were often unclear, or it was unclear as to which point in the evaluation process attrition occurred. Information on sample size and attrition at every stage of the intervention process was missing in most cases, and only two studies reported information about the sample size throughout the intervention—at intake, during intervention, upon completion, and at follow-up (e.g., Dobash et al., 1999; Echeburúa & Fernández-Montalvo, 2009). Where attrition was reported, it was most likely to occur in the transition stage between pretreatment/individual sessions and the “core” intervention group sessions, but such attrition was rarely investigated further as it was often not within the scope of the evaluation to do so (e.g., ADVA et al., 2009). We also found a general lack of clarity or consistency as to whether the attrition reported was from the programme itself or from the evaluation (if they were different).

Toward a Model for Conducting and Reporting Evaluations of DVPPs

Accurate and robust sample profiling is important to fully understand the effectiveness of DVPPs. In addition to information regarding the nature of the intervention approach, we need to understand who is participating and why; who is dropping out, when and why; who is completing; and who is changing, when, why, and how? Our review of all European evaluations highlighted that evaluation research did indeed address these questions/aspects but not all of these within any one evaluation. Different constituents were addressed by different evaluations, using different methods, based on different participant samples. Thus, if we are to better understand how perpetrator programmes may work to create positive change, and be able to compare programmes, the information gathered during the evaluation process needs to be harmonised/standardised to address not only the methodological challenges highlighted by previous research but also the additional areas highlighted in this article.

We propose a model that should be used and promoted in this field for a common understanding, concerning points of time of observation in evaluation studies (see Figure 2).

OPEN IN VIEWER

Figure 2.

Evaluation data and time points (for collection and reporting).

The model presented in Figure 2 divides evaluation into five time points (which should be clearly defined/reported), at which specific information should be collected and reported. The purpose is to guide evaluations so that reports are clearer about what data were collected, about who and at what stage (i.e., at intake/preintervention, start of intervention, during intervention, at the end of intervention, and during follow-up), about who dropped out/was excluded and why, and the source of the outcome data at each point. This will help reviewers to understand exactly who is participating/receiving “treatment” and—perhaps more importantly—who is not (e.g., those not admitted, excluded, or dropping out because of more complex issues such as substance misuse or mental health problems) and why, and who exactly is defining “success.”