Abstract
Improving outcomes for looked after children and young people has been a longstanding concern in Wales. This article reports the findings of a scoping study which sought to identify interventions aimed at improving outcomes for looked after children that are effective or promising. The study was commissioned by an independent funding body to inform a £5 million investment programme for Wales. It comprised a rapid review of literature, informed through consultation with an expert advisory panel and groups of young people who had been in care. The article outlines the rapid review method, provides details of shortlisted interventions and describes the interventions subsequently approved for investment. It concludes that although there are many promising interventions which address the factors associated with poor outcomes for looked after children, the evidence base is weak. It is argued that decision-making on interventions should be informed by appraisal of the empirical evidence available, but should also be guided by professional judgement that considers the needs, priorities and preferences of service users, carers, practitioners and policy-makers.
Introduction
Improved outcomes for looked after children have long been an aim of the UK and Welsh Governments (Berridge, 2012; DCSF, 2009a; 2009b; Welsh Assembly Government, 2004). In Wales, a raft of legislation, statutory guidance and policy directives has been designed to address this issue (see, for example, Children Act 2004, Parts 3–5; Welsh Assembly Government, 2007a; 2007b; 2011). Despite some positive change, improved outcomes have generally remained elusive (CSSIW, 2015; Holland, 2015; Welsh Audit Office, 2012), as is the case elsewhere in the UK (see Coman and Devaney, 2011; Scottish Government, 2015). The extent to which obdurately poor outcomes are due to children’s pre-care experiences, the care system or other factors remains unclear (Forrester, et al., 2009). What is clear, however, is that in Wales the scale of the challenge is growing as the number of children in care has been rising, currently to 91 children in care per 10,000 of the 0–18 population, a ratio that is some one-third higher than that found in England and Northern Ireland (StatsWales, 2014).
In seeking to understand the underlying factors explaining this difference, it is helpful to reflect on the practice context in Wales. The country has been engaged in a process of devolution since the Government of Wales Act (1998) and the transfer of legislative power has created a highly complex practice landscape. Whereas the oversight of education and health has been devolved and led to increasingly idiosyncratic practice compared with the rest of the UK, family and youth justice remain the preserve of the UK Government. Practitioners in Wales are, therefore, frequently faced with having to attend to directives from two governments that place differing emphasis on, for example, respect for the United Nations Convention on the Rights of the Child (see Welsh Assembly Government, 2011). The recent Social Services and Well-being (Wales) Act (2014) has further increased this divergence.
The context of practice in Wales, especially for a mobile population such as looked after children, is further complicated by the relatively large number of municipal authorities, many of which are small. Some argue that this cultivates insular practice which can, in turn, lead to the fragmentation of core services and discontinuity in the provision of care (Williams, 2014). Wales is also a remarkably diverse nation in respect of language, economics and transport networks. Consequently, any strategic attempt to improve outcomes for looked after children must take into account a unique and complex mix of legislative, geographical, economic and cultural factors.
This article details the findings of a commissioned scoping study which sought to identify effective or promising interventions specifically for use with looked after children in Wales. The funding was supplied by the Big Lottery Fund with the expressed purpose of informing decisions about its £5 million investment to improve their life chances.
Initially, the ‘rapid review’ method employed for the study is discussed and details of the shortlisted interventions provided. This is followed by some reflections on the deliberations of the review team in recommending interventions for investment. It will be seen that appraisal of the evidence featured heavily in the process, but that broader issues, such as potential impact and effectiveness with looked after children, were also considered.
Among the questions asked were:
Who should be the target of interventions? Some may be suitable for the majority of looked after children while others may be targeted at particular age groups or those with specialist needs. Which needs should be targeted? Improved outcomes in certain areas may be particularly desirable as they are likely to have positive ‘knock on’ effects for other areas of the child’s life. How should interventions be delivered? Interventions may be delivered via a number of agencies and individuals, but the project team was especially interested in the involvement of children.
The needs of looked after children
An understanding of the needs of looked after children is essential in order to carry out a meaningful appraisal of intervention programmes. However, this makes the exercise complex as looked after children are known to experience greater difficulties in a number of areas compared with the general population. These include academic achievement, physical and mental health, behaviour and socialisation. For example, the educational attainment of looked after children in Wales is notably low. In the academic year 2014–2015 only 11% of them obtained five or more GCSEs (A*–C) compared to 58% of the general population (StatsWales, 2015). Two-thirds are reported as having special educational needs (Meltzer, et al., 2004) and they are more likely to be excluded from school (Axford, 2008). Similar disadvantages have been noted for mental health issues (Meltzer, et al., 2004; Sempik, 2010), risks of self-harm and suicide (Furnivall, 2013) and physical needs that require health service support (DCSF, 2009a; 2009b; Meltzer, et al., 2004) due to early pregnancy or substance misuse (Craine, et al., 2014; Panwar and Wilson, 2011). The same situation applies to children’s socialisation, relationship formation and maintenance (Guishard-Pine, McCall and Hamilton, 2007) and subsequent social exclusion (Axford, 2008; Hicks, et al., 2012; NICE, 2010).
It is important to note, however, that many looked after children perform relatively well in key domains and that there is no single difficulty or constellation of presenting problems that is common to every child (Rees, 2013). Similarly, the profile of individual need is always complex and specific to the individual child and his or her circumstances.
Although many of these difficulties affect the wider population, the underlying mechanisms that give rise to difficulties may differ for looked after children as many of them have experienced considerable disruption and trauma at a time when formative attachments were developing. When Jones and colleagues (2011: 621) reviewed 92 studies that mapped intervention outcomes, certain factors stood out as especially significant for such children – namely, placement stability and emotional and behavioural problems, which the authors perceived as ‘key mediators between underlying risk factors and outcomes’. The significance of placement stability has also been stressed by Leve and colleagues (2012) while Rees (2013) has emphasised the importance of regular birth family contact for positive performance in key domains. In the light of this evidence, interventions designed to boost placement stability which are sensitive to fundamental underlying issues of attachment, separation, loss, trauma and birth family relationships seemed to be good candidates for investment.
Reviewing the literature
Having recognised the variety of needs that may be experienced by looked after children, the research team sought answers to the following questions:
What is the range of interventions that meet the needs and enhance the well-being and outcomes of looked after children and which of these are targeted at specific areas, such as education or health, and which are more holistic? Which interventions have the strongest evidence base and are most likely to be applicable to the Welsh context? Do interventions which have strong evidence of effectiveness have common elements? If so, what are they?
To get things going, a briefing paper on the study and an invitation to participate as members of an expert advisory group were forwarded to a cross-section of 22 stakeholder organisations involved with looked after children. These organisations were identified by Children in Wales and included representatives from the Welsh Government, statutory services, independent agencies and third sector organisations. Seven young people with experience of being in the looked after system were also invited, and agreed to participate as consultants.
Given the restricted timescale and the success of this method in health and social work evaluations, a review of the literature was undertaken in the autumn of 2013 using a ‘rapid review’ methodology (Shaw and Holland, 2014). Figure 1 gives an overview of the process, decision points and outcomes. This method streamlines the processes of a full systematic review and takes less time, but importantly, shares many of its rigorous procedures by setting inclusion and exclusion criteria and a traceable methodology for database searching and analysing findings (Collins and Fauser, 2005; Shaw and Holland, 2014).
Chart showing rapid review process, decision points and outcomes.
Search terms and criteria.
The search was supplemented by internet trawls, hand searches of journals and reference lists together with suggestions from the expert advisory group, interested organisations and academics. In total, 340 publications were identified. Due to resource constraints, those published before 2000 and/or not in English were filtered out, as were interventions with care leavers and those with a very narrow focus, such as repetitive self-harm. As funders had stipulated that interventions must be additional to or an enhancement of services routinely provided by Welsh local authorities and/or health boards, those patently similar to statutory services were also omitted. In the end, 60 interventions were retained for further examination. The second stage involved reviewing the abstracts and/or summaries of the 60 interventions. Reviewers considered whether they were supported by robust empirical evidence, showed an awareness of underlying issues such as separation or trauma, and appeared suited to the Welsh context. Thirty-two interventions out of the 60 were retained for further consideration.
The 32 studies were then assigned to individual team members for quality appraisal. The National Institute for Health and Social Care Excellence’s (NICE) Quantitative and Qualitative appraisal checklists (NICE, 2012a; 2012b) were used to rate each intervention’s methodology as strong, moderate or weak (see Appendix A) and to scrutinise the underlying theory, study design, measures used, target population, use of intervention/control groups, the analysis and findings, especially those that showed a long-term effect on children’s functioning. Studies that concentrated on community settings, relationships and problem-solving approaches were rated more highly, as were those that directly impacted on the children.
Once this rating exercise was complete, the individual appraisals were shared among the team, who reviewed each summary and agreed the studies to be retained. At the end of this stage, 18 interventions were included (see Appendix B).
Interventions shortlisted for recommendation.
Shortlisted intervention consultation
Having selected the five most robust interventions, the next decision was where to target the investment. Final recommendations about this were informed by the preferences of the expert advisory panel and young people’s group. The former stated a preference for interventions that: target more than one aspect of children’s well-being; include direct work with young people; have originated in or are transferrable to the UK; and have been designed for looked after children and carers. The young people liked interventions that: involve direct work with young people; have a broad focus; can be used with those living in residential and foster care; and include some follow-up element to ensure longer-term sustainability. An additional issue was the extent to which the interventions were compatible with the legislative, political, social, economic and geographical context of Wales, especially the impact on children’s overall well-being – a key principle of the Social Services and Wellbeing (Wales) Act 2014.
Results: Investment for Wales
Following presentation of the findings of the scoping study to the funder, the following selection was approved for investment:
Fostering Changes (revised programme); follow-up Support Group for Carers; Fostering Healthy Futures (feasibility study); piloting of the Fostering Healthy Futures or an adaptation of this programme (determined by the recommendations of the feasibility study).
This selection will involve a five-year investment in Fostering Changes (revised programme) rolled out on a phased basis across Wales for foster, residential and kinship carers, and feasibility studies to explore the process for implementing or adapting the Fostering Healthy Futures programme. The evidence base for both of these programmes is strong. Fostering Changes (Pallett, et al., 2005) has been successfully implemented in parts of England since 2009, is age and placement inclusive and focuses on the key priorities for enhancing the well-being of looked after children. Likewise, Fostering Healthy Futures works directly with looked after children. Both initiatives include provision for ongoing evaluation.
Discussion
It is encouraging that this review was able to quickly identify a large number of interventions specifically aimed at improving the outcomes of looked after children and acknowledge the efforts of practice-oriented researchers from a range of disciplines, who have highlighted the exceptional needs of such children and sought to fashion effective interventions. But despite this welcome increase in attention, this study has indicated that there is some way to go in establishing a sizeable bank of interventions based on robust evidence of efficacy. Many of the studies deselected reported promising results, but had too few participants and/or were focused on too narrow a sub-group to afford confidence in the generalisability of their findings.
Another common reason for deselection was the lack of evidence of positive impact in key areas of children’s functioning. The majority of interventions provided indirect input which was mediated through carers, social workers, care workers and teachers. While positive change on indirect outcome measures such as carers’ knowledge and satisfaction is pleasing, the inability of many studies to demonstrate any significant effect in key areas of the child’s functioning is disappointing.
In recent times the social work profession has been increasingly challenged to demonstrate that its practice is based on sound evidence. Some within the profession have turned, therefore, to evidence-based practice, a scientific model for the appraisal of intervention efficacy that originated in the field of medicine (Straus, et al., 2011). An allure of evidence-based practice is that it offers a purportedly objective appraisal of evidence that may safeguard against criticism of subjectivity and bias. Integral to it is a predetermined hierarchical system of evaluation with meta-analyses of randomised controlled trials (RCTs) and individual RCTs considered the ‘gold standard’ and qualitative case studies regarded as inherently weak. Unsurprisingly, evidence-based practice is among the most contentious issues in contemporary social work (Mullen and Streiner, 2004). Critics argue that it is too mechanistic, narrow in focus, impractical and of low ecological validity and restricts clinical judgement or creative interventions, preferring terms such as ‘evidence-informed practice’ (Nevo and Slonim-Nevo, 2011) or ‘evidence-guided practice’ (Gitterman and Knight, 2013).
This review has shown that this for and against debate is an oversimplification as in reality there is no substantial body of research evidence available to practitioners (Mullen and Streiner, 2004). Few of the interventions scrutinised in this exercise were highly rated against orthodox evidence-based practice criteria, suggesting that empirical research on efficacy in child care is still very much in its infancy, making the current utility of evidence-based practice frameworks, at best, restricted. In addition, as the looked after children population is immensely heterogeneous and the needs of some children can only be met by interventions tailored to meet a child’s specific needs, highly structured interventions have to have some flexibility. Users’ perceptions and wishes can also be important and have to be incorporated somehow.
All of this made the task of the review team especially difficult; it was charged with recommending intervention(s) for funding that required maintaining the integrity of decision-making while employing an eclectic and consultative approach. In one case, an intervention was considered weak scientifically, but some members of the expert advisory group clearly liked it and it was retained until the latter stages of the exercise for the purpose of highlighting a novel and promising intervention, rather than being nominated as a good investment.
This review also reinforced what practitioners know by experience; there is unlikely to be an intervention that will serve as a panacea for all the challenges faced by the looked after population, so it is no surprise that a wide range of outcomes were targeted using a myriad of approaches. Nevertheless, it is remarkable given the context of practice within Wales and the emphasis on reducing the care population (ADSS Cymru, 2013; Welsh Assembly Government, 2016) that so few studies specifically targeted rehabilitation. It is probable that the authors of many small-scale interventions consider such an objective to be beyond the scope of a ‘bolt-on’ intervention and the preserve of social services. Paradoxically, however, any intervention that is not closely aligned with the primary objectives of social services, Child and Adolescent Mental Health Services (CAMHS) and adoption support services may struggle to see meaningful success.
Limitations
This review was, of necessity, undertaken within a short timescale. The use of rapid review methodology provided a helpful framework, but it is acknowledged that the project team could not apply the same degrees of scrutiny to studies as would typically occur during a full systematic review. In addition, the development of evidence-based clearing houses and repositories in social work have clearly helped to promote evidence-based practice, and quality ratings of some programmes considered in the review are readily accessible therein. Although these were consulted informally as part of the initial trawl of literature, a structured search strategy was not applied.
Footnotes
Note
Appendix A. Quality appraisal checklist and rating scale
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
