Abstract
Internationally, interpersonal violence places huge burdens on the health, wellbeing and prosperity of society. In response to a notable increase in serious knife crime, in 2019, the UK Government awarded £35 million for the establishment of 18 Violence Reduction Units (VRUs) across England and Wales. There has been limited evaluation of community-level approaches for violence, with almost no published literature on the impact of VRUs. The article presents the approaches and experiences of two interdisciplinary teams of researchers from public health, psychology, criminology, and systems change, working as evaluators of four VRUs in England and Wales. The article describes the value of adopting a whole-system approach to evaluations, outlines good practice in evaluating VRUs, and elicits challenges to developing and embedding evaluation within complex systems.
Introduction and context
Internationally, interpersonal violence places huge burdens on the health, wellbeing, and prosperity of society. It presents a critical public health, human rights, and gender equality issue and for many countries is a leading cause of death for young people (United Nations, 2019). In 1996, the World Health Assembly declared violence a major and growing public health problem across the world (Resolution WHA49.25) and in 2002, the World Health Organization (WHO) launched its first World Report on Violence and Health (Krug et al., 2002). The report aimed to raise awareness of the extent and impact of violence globally, factors that increase risks of violence or protect people from harm, and effective prevention and response strategies. Critically, it called on countries to adopt a public health approach to violence prevention.
Public health takes a collective, evidence-based, multi-disciplinary approach, and is implemented with and for communities. It is grounded in four key steps of enquiry: understand the size and nature of the problem, identify risk and protective factors, develop and evaluate interventions, and widely disseminate effective practice (Krug et al., 2002). Globally, substantive efforts and action have been taken to advocate for the adoption and implementation of a public health approach to violence prevention. Increasingly countries have developed whole system frameworks and policies for the implementation of this approach (e.g. Finland – Korpilahti et al., 2020; Scotland – Arnot and Mackie, 2019; Australia – Our Watch, 2021; United States – David-Ferson et al., 2016), and emerging evidence of community public health level approaches suggests it is associated with reductions in violence and cost savings for communities (e.g. Cure Violence Global, 2022; Cardiff Model – Florence et al., 2011, 2014). Despite the progress in advancing a public health approach to violence prevention globally, for many countries implementation of the approach remains in its infancy, with significant gaps in understanding how the approach can best be implemented, and what impact this can have for communities and wider society. Understanding the implementation of a public health approach to violence prevention, and evidencing its impact, is critical for embedding the approach at local, national, and international level.
In response to a notable increase in serious knife crime, in 2019, the UK Government awarded £35 million for the establishment of 18 Violence Reduction Units (VRUs) across England and Wales. The VRU model was based on the Scottish VRU, which was rolled out nationally in 2006 after its inception by Strathclyde Police in 2005. While the operation of the Scottish VRU has been politically neutral, the VRUs in England and Wales are coordinated through regional Offices of the Police and Crime Commissioner (OPCC). The VRUs were tasked with bringing together a range of organisations, including the police, local government, health, and community groups to tackle violent crime through the adoption of a public health approach (Home Office, 2020). Layered onto the public health approach adopted by the VRUs is a ‘whole system multi-agency approach that is place-based’ (Public Health England, 2019). This approach incorporates five principles of: Collaboration, Co-production, Co-operation in data and intelligence sharing, Counter narrative development, and Community consensus (PHE, 2019). Guidance issued by the UK Home Office (2020) summarised a whole system approach to violence reduction as: focused on a defined population, with and for communities, not constrained by organisational or professional boundaries, focused on generating long-term as well as short-term solutions, based on data and intelligence to identify the burden on the population, including any inequalities, and rooted in evidence of effectiveness to tackle the problem.
The authors of this article have between them undertaken evaluation of four VRUs in England and Wales during their first 2 years of development and implementation. While our experience reflects only 4 out of the 18 VRUs, this represents a range of VRU types, including those based at a national, regional, and core city level. While funding levels reflect the scale of each VRU, there are structures and components of each that are fundamental to the VRU model; each of the four VRUs comprise core members responsible for identifying the overarching strategic themes and work-streams for the VRU. These members represent organisations including (but not limited to) the police, public health, local/national government, youth and community engagement, education, Youth Offending Service, probation services, fire and rescue services, Home Office, and a range of local organisations. Each theme/work-stream ensures that violence prevention is included in all policies, and pathways and multi-agency working, commissioning interventions for prevention, sharing best practice, and scaling up interventions. All four sought to adopt whole system approaches to address violence at all stages of the public health model and across the life course. This involved partnerships across health, police, local authority, and the third sector developing, for example, multi-agency data sharing systems (Quigg et al., 2021); implementation of programmes aiming to protect against violence and reduce risks (Butler et al., 2021; Caulfield et al., 2022; Fox et al, 2020; Kazlowska et al, 2021; Wilkinson et al., 2022), and responses to reduce the impacts of violence and further harm (Adams-Quackenbush, 2022; Quigg et al., 2022).
Key to each evaluation was to explore the implementation and effectiveness of the VRU as a system, rather than to only evidence the effectiveness of VRU-commissioned interventions. Such an ambitious, multi-agency, whole systems approach to violence reduction requires an equally ambitious approach to evaluation: one that can capture change at different levels, in different places, for different stakeholders, and which recognises that it might not be possible to identify single causal factors but instead seeks to identify plausible connections between interventions and outcomes. There has been limited evaluation of community-level approaches for violence, with almost no published peer-reviewed literature on the impact of VRUs. In Scotland, for example, while reductions in violent convictions have largely been attributed to the Scottish VRU (Coid et al., 2021), evidence concerning what is effective, how and why it is effective, and for whom, is limited.
In Australia, a review of a whole system approach to preventing violence against women and girls shows how a common framework can enhance the implementation of primary prevention, while also highlighting the complexities of implementing such an approach and evaluating impacts over the long-term (particularly when constrained by evaluation funding and time period; Our Watch, 2019). The Home Office commissioned a national-level process evaluation to investigate the early implementation of the VRUs (Craston et al., 2020) and assess the amenability of VRUs to an impact evaluation in the future (MacLeod et al., 2020). These evaluations document overarching approaches and early indicators of success across the VRUs. Consistent with the public health approach (Krug et al., 2002), individual VRUs also appointed evaluators to understand progress and investigate impact at a local level.
This article presents the combined experience and approach of two separate teams of researchers evaluating four VRUs in England and Wales during the first 2 years of implementation. The evaluation teams are experts in public health, psychology, criminology, and systems change. This article stems from conversations between the two teams, which led to a realisation that their shared knowledge is important to capture given the dearth of literature in this area. The process by which the two evaluation teams produced this current article followed an approach set out by Jolly et al. (2021). Following the initial idea for the collaborative article, the entire discussion and writing process took place online. An initial session, led by the first author, focused on agreeing the process by which the article would be written and the core lines of argument. The authors had already engaged with the work of the other team and were able to have initial discussions in the first sessions to discuss: ‘the core over-lapping elements/principles of our approaches to evaluating VRUs?’, ‘the key challenges in evaluating VRUs’, and ‘What learning do we want to share about these challenges?’. After agreeing core elements and challenges, the authors moved to real-time writing used a shared online document. Timed ‘writing sprints’ were undertaken, with one author drafting a section, and another author adding to and revising a section in subsequent live ‘sprints’. This process was new for most of the authors, and so it was important to set out key principles out early on around setting aside perfectionism and writing quickly/refining later. The authors engaged in three of these live writing sessions over a number of months. Following this, refining and redrafting the article happened online but asynchronously.
This article aims to exemplify emerging approaches to evaluating whole-system public health approaches to violence prevention across England and Wales. The article describes the value of a whole-system approach to evaluation of VRUs, outlines good practice in evaluating VRUs, and elicits challenges to developing and embedding evaluation.
Evaluating Violence Reductions Units in England and Wales
Approaching and designing VRU evaluation
The expectation was for VRUs to address ‘‘the root causes’ of serious violence through inter-agency working and public health focused interventions’ (Hopkins & Floyd, 2022: 359). Overlaying the public health approach, the four VRUs that the evaluation teams worked with also adopt whole system approaches. By treating violence not as the sole responsibility of one single agency, but as a public health issue, the VRUs recognise entrenched problems are best addressed through multidisciplinary and multiagency working and collaboration. Actors in the system include community residents and professionals working across criminal justice, policing, public health, youth services, adult social care, healthcare, education, housing, and the voluntary and community sector. These agencies were already operating across local areas within VRU regions as part of Community Safety Partnerships and other local initiatives.
The evaluation design of a VRU required consideration of the following key factors. First, the organisational intricacies of VRUs, and their parent organisations were an important consideration. VRUs often comprise of many hierarchical levels, engaged with multiple initiatives, and various interventions. Moreover, VRUs do not function in isolation of the regions and communities they serve. This can make it difficult to ascertain whether changes to regional violence are a direct result of the VRU activities, or whether there are other factors at play (e.g. independent community factors, changes to social funding, or political, environmental, and global factors). When an organisation has multiple moving parts and objectives it is vital to plan the evaluation carefully. The activities covered by an evaluation may be constrained by time or funding. In two of the areas, the evaluation teams worked within these constraints by dividing the workload into workstreams with dedicated teams focused on a specific aspect of the evaluation. This allowed for evaluative activities to occur concurrently. The teams also took a phased approach to meet requirements for the contracted deliverables.
Second, consideration of the principles that underpin a public health approach is critical. This requires the systematic collection of information around the types of violence, consequences, and root causes as well as factors that can be modified through intervention. The socioecological model (SEM) is recommended as a framework to understand the factors that affect and influence violence across the individual, relationship, community, and societal levels of the environment (Krug et al., 2002). This could include individual factors that lead to violence; relationship factors including parents, families exposure to adverse childhood experiences (ACEs); awareness of the impact of violence and feelings of safety across communities; and policies and structures that promote awareness, education and action to prevent serious violence. Considering outcomes and impacts across each of these levels is critical to understanding what can be measured through evaluation.
Finally, it is important to consider at what stage of implementation changes are best measured and to identify when, where and how these changes can occur. Below the key elements of robust VRU evaluation are outlined (see Figure 1).
Key elements of robust VRU evaluation outlined in this article.
Being clear from the start: Developing a theory of change and outcomes map
It is important for VRUs to have a clear Theory of Change (ToC) 1 to identify how programme activities lead to goals being achieved. This involves identifying the desired long-term goals and then working back to identify all the conditions (outcomes) that must be in place to achieve the goals (see Supplementary Figure 1). The ToC provides the narrative to explain the processes and mechanisms of change that should occur as a result of the VRU strategy. The ToC should illustrate how the VRU activity is anticipated to achieve a range of short-, medium-, and long-term outcomes. These conditions should then all be mapped out in an Outcomes Framework (see Supplementary Figure 2).
The majority of VRUs nationally did not have a ToC prior to commencing their activity in 2019/2020. Of the four included here, two developed their own ToC, one subsequently working with the evaluation team to refine and update it. The other two worked with the evaluation teams to develop their ToC, and to continually refine it as the evaluation progressed.
Developing a ToC should ideally involve stakeholders involved in the design and delivery of the programme to ensure that the objectives for evaluation are accurately defined. Stakeholder engagement activities, such as workshops, were used to identify inputs, activities, outputs, and intended goals. This helps stakeholders to understand the overall structure and function of a programme and build a strong consensus it is working towards. This enables stakeholders to identify long-term outcomes and changes in capacity building, relationships, and behaviour. This engagement activity is particularly useful when evaluating the impact of different activities that are funded under a single programme and/or when evaluating place-based approaches, where partners might have differing objectives and competing evaluation interests.
Where possible, the evaluation team engaged with stakeholders to identify and refine programme, place and intervention level inputs, outputs, outcomes and impacts, 2 the mechanisms that linked them and the appropriate measures and methods to assess their achievement. For example, one stakeholder engagement event was held with ~160 stakeholders (core VRU members, associate members, and wider partners) across a VRU region to share their views on the outcomes and impact that they anticipated their VRU could achieve, before providing details about the activities/interventions they contributed towards (the inputs) and the data they collect (the outputs). This information was used to develop a ToC that was reviewed and refined through further discussion with stakeholders.
The ToCs provided a framework to support the development and focus for evaluation, particularly in terms of identifying gaps in data required to evidence outcomes, the development of outcome and impact indicators and embedding an evidence-based approach to violence prevention across the system. The ToCs were viewed as ‘living documents’, adapted through the life of the evaluation, providing evidence about what works, for whom and why. In our most successful example, ongoing stakeholder engagement contributed to the development of the ToC, which was also refined to identify whether other, unexpected, short-term outcomes had occurred. An iterative approach was required to ensure that the ToC was useful and informative for stakeholders. For example, feedback from stakeholders in one VRU was that the ToC was challenging to understand. Subsequent engagement was carried out to simplify the ToC in response to this feedback.
In one area, the evaluation teams undertook a detailed and systemic approach to interrogate why interventions were commissioned and how they were intended to contribute to violence prevention, reduction, or desistence – as documented in the overarching ToC. The mapping was undertaken by reviewing documentation provided by the organisations when applying to the VRU funding call. Next, meetings with providers gave clarity about the objectives of each intervention, as well as insight into potential monitoring and outcome data. This was mapped against the ToC as shown in Supplementary Figure 2 (see Adams-Quackenbush and Caulfield, 2021 for a detailed account of the mapping process). This mapping activity was important for two key reasons. First, it allowed gaps in delivery – and therefore gaps in potential outcomes mapped against the ToC – to be identified. This informed future commissioning decisions. Second, meetings with providers allowed the evaluation team to fully understand the data collection processes of each intervention. This information was used to provide recommendations for data collection.
Evaluating complex system change
While in the above example the focus was on mapping interventions commissioned by the VRUs, it is vital to acknowledge within evaluation that VRUs are complex programmes involving multiple partners and stakeholders. The academic literature on whole systems approaches (French and Lowe, 2018; Ghate et al., 2013; Grint, 2008; Lowe and Plimmer, 2019; McGuire, 2006; Obolensky, 2010) highlight that the inherent challenges of ‘wicked issues’ (Grint, 2008) such as violence, combined with decreasing resource (Ghate et al., 2013) require a different public sector response to cope with increasing levels of ‘volatility, uncertainty, chaos and ambiguity’ (Ghate et al, 2013: 6). Systems are described as complex, adaptive, fluid and dynamic (Obolensky, 2010), and this approach recognises that outcomes will be produced by the whole-system rather than individuals or single organisations (Lowe and Plimmer, 2019). While there is a growing body of academic literature concerning the ideals of whole-systems approaches, there is limited evidence that chronicles its application in the real world.
The evaluation teams sought to explore the VRUs as whole-systems. The evaluations focused on documenting: how the VRUs were developed and implemented, how decisions are made, governance processes, flagship interventions and strengths/gaps. Where relevant, frameworks were used to structure the analysis and findings with reference to strategy. For example, the five principles set out by Public Health England (PHE) in their whole-system multi-agency approach to serious violence prevention (PHE, 2019) (collaboration; co-production; cooperation in data and intelligence sharing; counter-narrative development; and community consensus) and the WHO public health approach were used to structure evaluation findings.
Our VRU evaluations gathered evidence from across the ‘system’, which involved consulting with VRU core partner agencies, stakeholders, community members, and providers of commissioned interventions involved in supporting and delivering violence reduction/prevention activities. We drew on established methodologies such as systems evaluation approaches (measuring the system against performance goals; Egan et al., 2019) and appreciative inquiry (identifying what works within organisational change; Whitney et al., 2001) to understand the wider factors that influence change. These consultations were executed using a variety of qualitative activities such as:
Stakeholder engagement workshops/events, used to bring together a range of stakeholders from across the ‘system’ to gather views/perceptions about the context of the problem. These allowed evaluators to understand the perceived expectations and ambitions of the VRUs, and to explore stakeholder views on their understanding and experiences of VRU programme delivery.
Qualitative engagement/data collection, conducted via interviews and focus groups with stakeholders who represented aspects of the ‘system’ (e.g. core partners and wider members). This activity allowed further understanding of the process involved in the development of the VRU model, as well as the expectations, experiences, and challenges involved in that development. Qualitative engagement was also conducted with representatives from organisations commissioned to deliver VRU interventions to explore their expectations about the VRU, how they became involved, and their views on the whole-system, public health, approach.
Observations, as well as whole system activities (e.g. attendance at VRU steering group meetings and other key meetings) and intervention delivery to add context and meaning to the evaluations.
Smaller core team workshops, to share key findings and facilitate the development of future programme delivery and strategy development.
The early findings demonstrated how stakeholders perceived the VRUs to be umbrella organisations that bring individuals and agencies together and provided examples of where this had improved partnership working and pathways (Caulfield et al., 2020; 2021; Quigg et al, 2021; Timpson et al, 2020). This element of the evaluations highlighted that where there was evidence of co-production, that this could be strengthened. This was particularly so in the case of community engagement and while the VRUs acknowledged the importance of working within communities, there was still work to be done to develop trust and engage with wider partners. This has also proved to be a challenge in other VRUs (Hopkins and Floyd, 2022).
The next phase moved towards challenging the VRUs and stakeholders to critique their delivery model, identify strengths and weaknesses and consider areas for improvement. Here, findings were mapped against the key principles of a whole systems approach to tackle public health problems to assess progress and identify gaps and recommendations. VRU progress in was evident in this area, and it was clear that VRUs had started to develop strong relationships across the system and build capacity. However, in some VRUs, the need for a longer-term strategy was an area for improvement. Issues around sustainability and continuity were echoed in Years 1 and 2, with some lack of clarity between partners regarding how they would fund their activity, if not funded by VRUs. These are examples of how early stages of evaluation highlighted the potential of the whole-system approach and areas requiring further attention.
Understanding what works in what circumstances: Process and impact evaluation
Central to the teams’ approach to evaluating VRUs is exploring what works, for who and under what circumstances, and considering change over the long-term. From the perspective of the VRUs in commissioning interventions, it is vital to understand the impact of individual interventions. From the perspective of evaluators, it is vital to situate intervention evaluations within the overarching ToC, the system-level evaluation, and the outcomes mapping discussed earlier in this article.
Fundamental to the requirement of the evaluations was the ability to embed a ‘what works’ approach into the intervention delivery while informing future commissioning processes. To achieve this, several evaluation questions formed the framework for intervention process and impact evaluations (Table 1). In addition to the example evaluation questions outlined in Table 1, the UK Home Office also suggested outcomes and key performance indicators (KPI) for the VRUs throughout England and Wales. 3
Example framework for VRU process and impact evaluation.
VRU: Violence Reduction Unit.
The evaluation team worked across several areas to assess how the existing monitoring data 4 items could measure the desired outcomes 5 and where the gaps in this information remained. This involved working with data already collected by intervention providers, restructuring the VRU monitoring data templates in consultation with the service providers, and in one area training and supporting them in collecting the necessary information. This was deemed necessary as some of the data collected were not intended to measure outcomes (e.g. demographics) and other items measured superficial delivery targets/outputs (e.g. number of referrals or attendees). Most interventions were not initially set up with built-in evaluation metrics and very little data being collected were suitable to assess outcomes or the implementation and delivery. This challenge highlights the importance for VRUs to understand the service provider processes and level of data collection prior to commissioning and delivery. This step can help the VRUs understand what the service provider needs in terms of resources and training to obtain the data suited to quantify outcomes.
The detailed intervention evaluations can be accessed elsewhere (Caulfield, et al., 2020, 2021; Quigg et al., 2020, 2021, 2022; Timpson et al., 2020). The overall approach to impact evaluation outlined in this article was implemented for all intervention providers. Some VRUs commissioned more extensive outcomes, impact, and process evaluations for larger interventions that are expensive to fund, or for newer interventions (Adams-Quackenbush et al., 2022; Badger et al., 2022; Butler et al., 2021; Caulfield et al., 2022). Given the relative infancy of the VRUs the evaluations provide ‘an indication of the “direction of travel” (the trend in benefits or harms across impacts at specific time-points) rather than a final verdict about an intervention’s effects’. (Egan et al., 2019: 12) and process evaluation is a key part of this. The evaluation teams undertook process evaluations to help identify what is working well about intervention delivery and what is not. Gathering evidence about the strengths and weaknesses of an intervention helps determine whether it is being delivered as expected, understand barriers and challenges and put processes in place to ensure that quality can be enhanced in a timely way.
Once the intervention structure and processes were understood, the evaluation team had an understanding of how interventions are managed and implemented to support outcomes and identify any potential barriers to success. The evaluation teams were then able to support the continued development of interventions – through focused recommendations based upon the process evaluation findings – (PHE, 2018b).
It should be noted that all of the VRUs discussed within this article went beyond the Home Office guidance and identified outcomes that included measuring and monitoring risk and protective factors (where appropriate) for the individuals who engaged with the interventions. This action demonstrated an understanding that meeting the overall Home Office objectives required components such as risk, need, and protective factors to be considered. In the context of this article, it also shows the layers and complexities that can be involved when assessing organisations with multiple evaluation questions that also feed into the objectives of governing agencies. Therefore, the evaluation teams were not only attempting to answer the questions set out by the VRUs, but they were also required to evaluate the ability of the VRUs to contribute to overarching national objectives.
Challenges and reflections
Evaluating complex systems, particularly those that are developing rapidly and where limited evidence exists, unsurprisingly brings challenges. The first part of this article documented the approaches used by two evaluation teams across four VRUs. This second section highlights: the challenges and importance of evaluating the whole system and including multiple stakeholder perspectives, challenges to data collection, challenges brought about by short-term funding, and finally, the thorny issue of responses to evaluation recommendations and ensuring findings reach their intended audiences.
Focusing on whole systems
As highlighted throughout this article and wider literature, evaluating a whole system approach to violence prevention is challenging (Our Watch, 2021). Capturing the impact of VRUs across different levels (i.e. from specific interventions to whole system change) is vital to understand if and how the whole system public health approach to violence prevention can reduce violence. A key challenge is teasing out the impact of VRUs in a broader, ever-changing landscape. While national VRU evaluation aims to examine the impact of VRUs through comparison to other similar areas, VRU-level evaluation of the whole system ensures that local stakeholders can understand the mechanisms by which change, or a lack of change, has been made for their communities, providing learning to continually refine the system and enhance sustainability. Mapping the whole system, and if and how that system changes over time (including VRU activity and wider activities), can help us identify plausible connections between whole system changes and levels of violence (including risk and protective factors) across a community. Mapping and evaluation of specific interventions can identify how they are contributing to whole system changes.
However, there is a potential tension where the VRU is seen foremost as a commissioner of programmes and wishes to understand the impact of the individual programmes and activities it funds. This logic is clear: if expensive interventions and programmes with limited existing evidence are commissioned, understanding how these programmes operate and the impact they have are important questions to ask. However, evaluation budgets are not limitless, and achieving the right balance of evaluation focus is complex. The evaluation teams had to present a challenge to VRUs at times, continually emphasising the need to consider the whole system, and not to focus only on individual programmes at the expense of this. Some of this tension arises from different perspectives on the role of evaluation teams: are they there to evaluate the impact and effectiveness of the VRU, or are they there to evaluate the outcomes and effectiveness of interventions funded by the VRU? Our position has always been that both are important, as set out in this article, but that considering the whole system is vital if the whole system public health approach is to be effectively implemented and sustained.
Understanding the system from multiple stakeholder perspectives
Stakeholder engagement has been a key theme across the VRU evaluations undertaken by the authors of this article. As noted above, it is important that any evaluation of a VRU begins with an understanding of the system within which it is operating, including the key stakeholders involved. It is equally important that the evaluation encompasses the views of a wide range of stakeholders and explores the shared (or otherwise) understanding of change objectives across the system. Successful evaluation of a whole system must therefore include meaningful input and engagement from a range of stakeholders.
To gather evidence from across the system a range of approaches and methods are required. With statutory and voluntary sector VRU partners the evaluation teams focused on engagement workshops/events and qualitative data collection with stakeholders representing aspects of the ‘system’. Qualitative work with stakeholders can usefully focus on: the evolution of the approach being adopted by the VRU and how the whole system is pieced together, analysis of the roles and responsibilities of key actors within and beyond the VRU, how and where the VRU sits and engages with key strategic partners, the relationship between the VRU with Statutory and Local Authorities, and how the ToC fits in with the programme. This allows analysis of the following: Contextual Challenge (including Systems and Stakeholders, Fundamental Tensions and Crisis Context), Vision & Purpose (Aims and Approach; Continuity and Change), and Communication (Outcomes and Impact, Making Progress).
At a community level, understanding of the challenges and assets that exist and create positive and supportive environments is needed and this can only be understood by working with those communities. It is important to explore if and how members of the public, particularly programme target groups, can contribute to the design, delivery, and production of the evaluation. This can ensure that the evaluation and the outcomes measured are meaningful to those who should benefit most from the programme, and that the data collection tools are appropriate for the target group. Engaging the public in the interpretation and dissemination of findings also helps ensure that findings are interpreted appropriately and shared in meaningful ways. A key challenge is ensuring that all community voices can contribute to our understanding of the whole system and VRU impact and viewing communities as key stakeholders. We acknowledge that some of our evaluation work has been limited by a failure to adequately engage community voices (Badger et al., 2023): there is much learning to take from this, and also from our successful examples of community and peer research (see Rees et al., in preparation; Kazlowska et al., 2021; Wilkinson et al., 2022). The NIHR INVOLVE provides guidance on co-producing research and how to involve members of the public in research and evaluation and the evaluation teams approached this in several ways, most notably in one area by involving community researchers from the outset of the evaluation. Community researchers, who were already trained through an initiative at one of the partner Universities, were recruited to work with communities but also with statutory stakeholders. Co-producing data collection approaches and tools, and working collaboratively on participant recruitment, data analysis, and dissemination, brought a huge amount to the evaluation. The community researchers were able to build trust with communities (e.g., through evaluation of place-based pilots: Kazlowska et al., 2021; Wilkinson et al., 2022), identify new lines of enquiry that academic researchers might not otherwise uncover, and provide challenge to the wider evaluation team. In another area, the evaluation team worked closely with service providers to ensure children and young people had opportunity to contribute to the development of the research. Inclusion of this group was paramount to co-production, and the use of gatekeepers was central to ensuring safe and meaningful contribution. Here, children and young people were consulted on the development of research tools with adaptations made to ensure these were appropriate for the target audience. Furthermore, the VRU independently commissioned media outputs (e.g. short-videos) where children who had participated in an intervention accompanied the delivery partner, wider stakeholders, and the evaluation team to share experiences and impacts of the intervention, in a more publicly accessible manner.
Challenges to data collection
There are several challenges to account for when working to evaluate the processes and outcomes of interventions situated within a complex system (i.e. changing goals, unclear objectives, motivation, generalisability, evidence of impact). These challenges are well documented within the evaluation literature (e.g. Van der Loo, 1995). However, they continue to arise within evaluation practice despite the presence of systematic study designs created by experienced evaluation teams. There are a variety of reasons these challenges arise within all types of evaluations, but the absence of planning the intervention initiative with evaluation in mind seems to be the underpinning factor (Ammenwerth et al., 2003). Indeed, when objectives and outcomes are set by the organisation, perhaps before evaluators are commissioned, figuring out how to measure and evidence the outcomes is tasked retrospectively. In terms of data collection, inadequate intervention planning can mean that service providers are not collecting the necessary information in a measurable manner. It can also mean a shortage of interested or motivated participants to help co-create the data (e.g. employees, stakeholders, service users).
Some of these challenges were unavoidable in the VRU evaluations (e.g. halted intervention delivery due to a global pandemic), whereas others could have been avoided with collaborative planning prior to VRU implementation and commissioning of interventions (e.g. reaching adequate sample size of participants and obtaining suitable data for quantitative analyses). Ideally, the VRUs would have embedded evaluation into the regional intervention strategy, involving experienced researchers and evaluators at the pre-implementation stage, and ensured all outputs were measurable and aligned with their objectives and KPIs. However, the VRUs were tasked with setting up and delivering activity very quickly, which is a problem with the system rather than the individual VRUs (see below, ‘Short-term nature of funding’), and equally there were pressures to fund interventions within short-time periods (within the first 3 years, VRUs were funded year-on-year, often with only a 6- to 9-month period to identify, fund and implement/complete interventions). This may have also set better expectations of participation with the evaluation component for the service providers, thus enhancing the quantity and quality of the information collected and provided to the VRUs.
Some of the very practical challenges that the evaluation teams were faced with included: significant delays in Data Sharing Agreements being signed by partners; data being provided at the aggregate rather than individual level, despite intervention providers collecting this; case study data only being provided; issues with the completion of monitoring and outcome templates, which in one area came to light very late due to delays with Data Sharing Agreements meaning the evaluators could only review the data at a late stage; changes to monitoring and outcomes templates being made by internal data teams without discussion with the evaluators; no medium or long term outcomes/impact data collected. The ways that these challenges were tackled included one of the VRUs commissioning a solicitor to draw up Data Sharing Agreements, although having structures in place from the start would be preferable. The potentially sensitive nature of the data and the need to ensure anonymity and compliance with General Data Protection Regulation (GDPR) meant this was a lengthy process but one that could have been foreseen. Even with this legal input, some providers were reluctant to agree to the data sharing arrangements that differed from their own internal policies and processes, resulting in lengthy discussion between the providers, VRU, and evaluation team to try and reach a resolution. In two areas the evaluation teams developed tailored training and support for organisations commissioned by the VRUs to ensure that expectations were clear, with consideration of what is appropriate in terms of resourcing and practicalities for smaller organisations (e.g. Wales VPU evaluation toolkit; Quigg et al., 2020). The evaluation teams also recommended clearer direction from the VRUs for intervention providers to ensure that it is understood engaging with monitoring and evaluation and providing data is required.
Short-term nature of funding
A common theme emerging from evaluations across the four VRUs, both at a system and intervention level, was the short-term nature of the funding. Public health approaches require long term investment, which is often contradicted by the politically driven nature of government funding cycles. Services commissioned on an annual (or shorter) basis pose a significant risk to the potential impact of investment into reducing youth violence. The individual VRU’s approach to violence reduction is based on unique needs and regional longitudinal trends in violence, but the speed at which they were set-up required fast commissioning with limited time for planning and mapping. The VRUs developed objectives and outcomes to measure their efforts and assess local impact (see Theory of Change section, above). Ideally, VRUs will have considered their complex environments when planning the approach and designing measurable outcomes for the interventions; however, this is not always the case. The challenges of funding timelines along with the demand for quantifiable and tangible outputs often take precedent over whole process planning and systematic methodical approaches to implementation. That is, individual interventions are often applied without a full vision of how their outcomes will be measured to align with the VRU objectives and outcomes that will eventually feed into Home Office KPIs. For evaluators, this is where detailed outcomes mapping is a critical part of the evaluation process: linking the overarching ToC to the potential contribution of individual interventions and providing this information to commissioners to support their decision making in commissioning future VRU activities.
Responding to evaluation findings and recommendations, and closing the feedback loop
As noted at the beginning of this article, a public health approach includes the wide dissemination of effective practice (Krug et al., 2002). A challenge for the evaluation teams has been a lack of control and ownership in determining how and when findings and recommendations are shared with key stakeholders, including the communities for whom the work concerns. Delays in sharing findings can damage relationships with providers and stakeholders and should be guarded against, but was unfortunately the experience in some VRU areas. It is understood that evaluation reports may be contentious or challenging, and that the VRU will inevitably want to have control of messages arising from reports, however it is important that there is clarity about how this will be managed on completion.
Consideration also needs to be given about how information is disseminated for different audiences. Audiences include key stakeholders from across the system (partner organisations, including criminal justice and health agencies), delivery partner organisations (particularly those whose intervention has been subject to process and/or impact evaluation), other organisations working in this space, regionally and nationally, and communities affected by youth violence. The evaluation teams focused considerable efforts on keeping dissemination high on the VRU agendas using a range of methods for different audiences. For the central VRU staff and leadership this involved verbally presenting findings to inform commissioning decisions, providing over-arching summary reports, and making clear recommendations for the VRU to build an action plan around. A further challenge concerned ownership of the action plans in response to evaluation recommendations. Where key staff moved on from the VRUs, findings, recommendations and action plans were not always communicated to new staff and the same issues then occurred again. This particularly relates to recommendations to intervention providers around data collection and reporting.
Where delays did occur in sharing findings with other stakeholders, processes were eventually agreed to ensure clear lines of dissemination and responsibility for action plans but unfortunately in one area damage to some relationships had already been done. For intervention providers, summary and full technical reports were made available by the evaluation teams, plus workshops to discuss and address recommendations. For communities, the evaluation teams pushed for events and workshops, particularly focused on engaging those who gave their time for evaluation data collection, and for some VRUs outputs and key findings were shared by the VRU in different formats for various audiences (e.g. press releases; videos). These new processes involved sign-off by the VRU evaluation board, which consisted of representatives from across health, criminal justice, and local government, rather than residing with any one body or individual. While this does not prevent contentious findings being kept out of the public domain, since implementation of this new formalised process in one area the collective decision has been to publish and act on all reports. The evaluation teams also actively engaged in national practitioner and academic networks to share the learning. In other VRUs, there seems to have been some success in appointing communications officers, although building effective links with communities remains a concern (Hopkins and Floyd, 2022).
Summary and conclusions
Understanding the implementation of a public health approach to violence prevention, and evidencing its impact, is critical but at present limited evidence exists. Within a public health approach, evaluating interventions and widely disseminating effective practice are key (Krug et al., 2002) and for all these reasons it is vital that well designed evaluations of violence prevention and reduction initiatives are undertaken and the learning shared. Stemming from conversations between two teams evaluating VRUs across England and Wales, this article has documented a shared approach to evaluating these new, complex, changing initiatives. This article presents good practice in approaches to evaluating whole-system public health approaches to violence prevention and highlights challenges to developing and embedding evaluation in a complex system. This article contributes to an emerging area of practice in evaluating complex social programmes.
The VRUs, in treating the entrenched problem of violence as a public health issue, brought together multidisciplinary and multiagency partners across criminal justice, policing, public health, youth services, adult social care, healthcare, education, housing authorities, communities, and the voluntary and community sector. The roles and relationships of the various partners are complex, as are the organisational intricacies of VRUs. The VRUs commissioned substantial programmes of work and interventions across regions, from preventive community and education projects through to criminal justice resettlement interventions. Underpinning all of the learning in this article is a conclusion that, in promoting a public health approach, the Home Office should implement systems, policies, and processes that disincentivise short-term, reactionary activity. Long-term funding, of at least 3 to 5 years, would address many of the concerns about delivery and evaluation outlined above. In 2022, the Home Office confirmed VRUs would be funded for 3 more years, with funding provided annual basis.
The key message from this article is the value of a whole-system approach to evaluation of VRUs to understand the role of individuals and organisations and their relationship to impact. Following a socioecological model in evaluating these complex social programmes provides a framework to understand the factors that affect and influence violence across the individual, relationship, community, and societal level. We recommend that evaluators, VRUs, and those involved in similar initiatives, implement the overarching approach outlined in this article, beginning by developing a clear Theory of Change and undertaking detailed mapping of activities and interventions against this to understand potential impact and inform commissioning decisions. Focusing evaluation resources on understanding the whole-system, in order to support the on-going development of the VRUs, is crucial as part of a public health approach. The nature of a public health model to tackling violence, with focus across the life course, means that the full impact of the approach will only be visible in the medium- to long-term. The value of the approach to evaluation set out in this article is that it helps ensure that the right conditions are in place, with VRUs placing resources in the areas most likely to help them achieve their long-term goals.
Evaluators and VRUs – and those working on similarly complex social programmes – could consider agreeing a clear statement of approach and requirements early on, focused on the key learning set out in this article. This could usefully include statements about the value of the approach, but also work to engage key staff and stakeholders, repeated annually to account for staffing changes. Evaluators and VRUs should think through the life-cycle of evaluation, from initial approach to sharing key learning. For example, agreeing a process for review, sign-off, dissemination, and responsibility for actions arising from evaluation reports. Key learning for evaluators is the tension around the role of evaluators in focusing on whole system versus the individual interventions commissioned by the VRUs, and the importance of both. There remain challenges for the VRUs in building effective links with communities and this extends to research and evaluation, with key learning for evaluators to carve out clearer routes for community engagement with the evaluation findings that do not solely rely on the VRU communication channels. The challenges presented within this article were not new to the evaluation teams, but the rapid set-up and complexity of the VRUs did present conditions for considerable challenge and learning in some areas. Capturing and sharing the learning around these new and as yet relatively under-evidenced public health approaches to violence prevention is crucial.
Supplemental Material
sj-pptx-1-evi-10.1177_13563890231183993 – Supplemental material for Reflections on good practice in evaluating Violence Reduction Units: Experiences from across England and Wales
Supplemental material, sj-pptx-1-evi-10.1177_13563890231183993 for Reflections on good practice in evaluating Violence Reduction Units: Experiences from across England and Wales by Laura Caulfield, Zara Quigg, Nicole Adams-Quackenbush, Hannah Timpson and Sophie Wilson in Evaluation
Footnotes
Acknowledgements
We wish to thank all the researchers involved in our VRU evaluations: Matthew Millings, Rebecca Bates, Ellie McCoy, Charlotte Bigland, Sally-Ann Ashton, Leanne Burton, Jennifer Hough, Rebecca Harrison, Nadia Butler, Beccy Harrison, Carly Lightowlers, Zoe Bell, Martha Clarke, Chloe Booth, Clare Harewood, Mark Monaghan, Alison Thompson, Debbie Kerslake, Aleksandra Kazlowska, John McDaniel, Andre Castro-Bilborough, Nettes Derbyshire, Katarzyna Gut, Shazia Baig, Wasim Abbas, James Rees, Bozena, Sojka, Andy Jolly, Jane Booth, Sarah Brooks-Wilson, Dean Wilkinson, Isha Chopra, Sophie Badger, Olivia Mee, Angela Ellis Paine, Rachel Hopley, Zena Lynch, Matthew Johnson, Mary-Rose Puttick.
We are particularly grateful to communities, individuals, organisations, and commissioners who have participated in our evaluation activities.
Authors’ note
Sophie Wilson is also affiliated to PhD Student with Institute for Community Research and Development, University of Wolverhampton, Wolverhampton, UK.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work underpinning the experiences underpinning this article was supported by funding from four Violence Reduction Partnerships, although the development of this article received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Supplemental material
Supplemental material for this article is available online.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
