Abstract
The international evidence base on factors that most influence outcomes in mental health care finds that matching therapeutic intervention to diagnosis has a clinically insignificant impact on outcomes. Decades of outcome research into treatment of psychiatric disorders shows that, despite the development of many new techniques, the outcomes being achieved in studies 30 years ago are similar to those being achieved now. In the last few years, new service models that incorporate systems of feedback on progress and alliance have emerged and show promise with regards improving overall outcomes for mental health service users. Growing familiarity with this outcome literature, together with a desire to be part of a service that can continue to improve patient outcomes, led a small community Child and Adolescent Mental Health Services team to develop a new whole service model – Outcome Orientated Child and Adolescent Mental Health Services (OO-CAMHS). OO-CAMHS incorporates key aspects of the evidence base on what could make a differential positive impact on outcomes and relinquishes those aspects that do not. In this paper, we outline the evidence base on which OO-CAMHS is built, describe the key features of the approach and present some of the early findings on its impact.
Introduction
The search for what works in mental health treatments has caused fractious debates for decades. New schools of therapy arrive with regularity, each claiming to be the ‘best and brightest yet’ and to have special insights into the causes of psychological dysfunction that other models fail to appreciate. Clinical trials were set up as each school attempted to demonstrate its efficacy. As Bergin and Lambert (1978) described about this time, ‘Presumably, the one shown to be most effective will prove that position to be correct and will serve as a demonstration that the ‘losers’ should be persuaded to give up their views’ (p. 162). The result was that ‘behavioural, psychoanalytic, humanistic, rational-emotive, cognitive, time-limited, time-unlimited, and other therapies were pitted against each other in a great battle of the brands’ (Duncan, 2002, p. 35). However, it was becoming increasingly evident by the late 1980s/early 1990s that the belief that one (or more) therapy would prove superior to others had little support (Norcross & Goldfried, 1992). Besides an occasional significant finding for a particular therapy, the critical mass of data and its meta-analysis was revealing no clinically significant differences in effectiveness between the various ‘bona fide’ treatment models for psychological distress (Wampold, 2001). Despite the development of many new techniques, the outcomes being achieved in studies 30 years ago remain similar to those being achieved now (Smith, Glass, & Miller, 1980; Wampold, 2001). Furthermore, even within treatment models, there is little evidence to support that any components of a model considered crucial are in fact so. For example, Ahm and Wampold (2001) undertook a meta-analysis of 27 component studies and concluded that the effect size for the difference between a package with or without components considered critical was not significantly different from zero.
If specific models could not explain why therapy works, what could? As far back as the 1930s Saul Rosenzweig (1936) concluded that, since no form of psychotherapy or healing is without cures to its credit, its success is not reliable proof of the validity of its theory. Instead, he suggested that some implicit ‘common factors’ may be more important than the particular techniques employed. Rosenzweig’s insight turned out to be more empirically supportable than the specific techniques of each model (Duncan, Miller, Wampold, & Hubble, 2010). The organising question then became, if therapies work but not as the result of their specific techniques, what are the common therapeutic factors?
The good news is that for the treatment of common psychiatric disorders, research finds that the average treated person is better off than about 50–80% of the untreated sample (Duncan et al., 2010). With regards to what ‘common factors’ are particularly influential in the likelihood of a positive outcome (or not), it seems that it is factors outside of therapy (such as socio-economic status and the availability of social support) that have the largest impact on outcomes and recovery rates. Within treatment, the factor that has the greatest impact on outcomes is the therapeutic alliance (as rated by the patient) with matching treatment model to diagnosis having an insignificant impact (Duncan et al., 2010; Wampold, 2001). This relationship between the alliance and outcome seems remarkably robust across treatment modalities and clinical presentations (Castonguay & Beutler, 2005). Furthermore, many of the ‘technologies’ (such as specific psychotherapy models) have been developed in a Western cultural context and researched in predominantly Western societies, raising questions about their suitability when working with communities who do not share similar beliefs and practices. Taking this evidence into account when designing and delivering mental health services, means that we have to revisit some of the core assumptions on which services are designed, if we are to succeed in making meaningful differences to the probability of more of the patients we see improving and recovering.
Although the randomised controlled trial (RCT) literature shows good positive results overall, this does not always translate to the clinical setting. Drop-out rates from treatment are a significant problem, averaging nearly 50% for many mental health services (Wierzbicki & Pekarik, 1993). In addition, despite the fact that the general efficacy is good in research settings, not everyone benefits. Hansen, Lambert, and Forman (2002), using a national data base of over 6000 patients in the US, reported a sobering picture of routine clinical care in which only 20% of clients improved as compared to the 50–80% rates typical of RCTs. Whichever rate is accepted as more representative of actual practice, it seems a substantial portion of patients do not get the benefit they would have hoped for. There are number of reasons why patients in clinical practice do not do as well as those in research, including the fact that exclusion criteria for research tends to keep out patients who are more difficult to treat (for example, those with co-morbidity), and that variability among therapists is the rule rather than the exception. Not surprisingly, although rarely discussed, some therapists are much better at securing positive results than others. Therapist effectiveness varies, but clinicians are not very good at judging their own. Clinical decision making, it seems, relies on the clinician’s personal judgement and experience and they often continue to be confident in their decision making whatever the outcome. For example, a study by Sapyta, Reimer, and Bickman (2005) asked 143 clinicians to rate their performance in comparison to other clinicians from A+ to F. Two-thirds considered themselves A or better; 90% considered themselves in the top 25%; not one therapist rated him or herself as below average.
So, despite there being overall efficacy in the treatment of common mental health problems, drop outs are a substantial problem, many patients do not benefit and therapists vary significantly in effectiveness and are often poor judges of their ability.
Howard, Moras, Brill, Martinovich, and Lutz (1996) were the first to advocate for the systematic evaluation of patient response to treatment during the course of therapy, but it was feedback pioneer Michael Lambert who brought this idea to fruition. Using the Outcome Questionnaire 45, Lambert has conducted five RCTs and all five have demonstrated significant gains for feedback groups over treatment as usual (TAU) for patients rated as being ‘at-risk for a negative outcome’. From those deemed at risk of a poor outcome, 22% of TAU cases reached reliable improvement and clinically significant change compared with 33% for feedback to therapist groups, 39% for feedback to therapists and patients and 45% when feedback was supplemented with support tools, such as measures of the alliance (Lambert, 2010). The addition of patient feedback, without new techniques or models of treatment and leaving therapists to practice as they saw fit, enabled two times the amount of patients at risk of not improving from treatment to benefit.
Partners for Change Outcome Management System
Lambert’s pioneering work has led to a growing interest in the merits of using feedback measures to highlight problems such as little or no clinical improvement or a poor therapeutic alliance. The challenge, as always, has been how to turn findings and processes used in a research context into one feasible for the realities of busy clinical practice. Poor clinician uptake of any approach is usually a rate-limiting factor for the uptake of any new idea. A number of groups have been attempting to make this leap from research into clinical practice, with perhaps the best-known and most successful example being the Partners for Change Outcome Management System (PCOMS).
The PCOMS is a project developed in America and devoted to empirically derived clinical practices, through incorporating predictors of therapeutic success into an outcome management system that includes simple, easy-to-use ratings of both therapeutic progress and alliance (Duncan & Sparks, 2010). The PCOMS uses two brief scales, the Outcome Rating Scale (ORS) and the Session Rating Scale (SRS), to measure the patient’s perspective of benefit and the alliance, respectively. The development of the PCOMS has included research and publications in peer-reviewed journals to establish psychometric validation of its instruments (Bringhurst, Watson, Miller, & Duncan, 2006; Campbell & Hemsley, 2009; Duncan et al., 2003; Gillaspy & Murphy, 2011; Miller, Duncan, Brown, Sorrell, & Chalk, 2006; Miller, Duncan, Brown, Sparks, & Claud, 2003). The ORS and SRS scores exhibit good internal consistency and test–retest reliability, despite the ultra-brief nature (four items) of these measures. The PCOMS has been shown in a number of randomised clinical trials to significantly improve effectiveness in clinical settings (Anker, Duncan, & Sparks, 2009; Anker, Owen, Duncan, & Sparks, 2010; Reese, Norsworthy, & Rowlands, 2009; Reese, Toland, Slone, & Norsworthy, 2010).
The ORS and the SRS are both four-item measures designed to track outcome and the therapeutic alliance, respectively. They were based on Lambert’s continuous assessment model and the associated questionnaires he developed. In RCTs of the PCOMS, use of the ORS/SRS (as part of PCOMS) resulted in improved treatment outcomes when compared to TAU (Anker et al., 2009, 2010; Reese et al., 2009, 2010). Although three of the studies focused on individual therapy, Anker et al. (2009) and Reese et al. (2010) extended evaluation of the PCOMS to couples therapy. Patients in the PCOMS achieved reliable change in significantly fewer sessions than TAU. A recent met-analysis of feedback studies (Lambert & Shimokawa, 2011) found that those in feedback groups had 3.5 higher odds of experiencing reliable change and less than half the odds of experiencing deterioration.
Common factors in Child and Adolescent Mental Health Services
The common factors perspective seems to hold in marriage and family approaches (Shadish & Baldwin, 2002) and child and adolescent therapies (Miller, Wampold, & Varhely, 2008; Spielmans, Pasek, & McFall, 2007).
As with adults, extra-therapeutic factors have the biggest impact on outcomes. For example, few studies find that a diagnosis of attention deficit hyperactivity disorder (ADHD) is independently associated with continuing impairments, whereas most studies on outcomes for those with a diagnosis of ADHD find that co-morbidity with conduct disorder, together with adverse environmental conditions (such as socio-economic disadvantage, maternal depression and marital discord), rather than ADHD severity per se, are associated with the most adverse outcomes (Barkley, Fischer, Smallish, & Fletcher, 2004; Biederman, Faraone, & Milberger, 1996; Fergusson, Horwood, & Ridder, 2007; Lee & Hinshaw, 2004). A diagnosis of ADHD is thus by itself not a good predictor of future outcomes, but extra-therapeutic factors are.
As with the adult outcome literature, within treatment there is little evidence to support that matching a treatment model to a diagnosis differentiates which treatment is more likely to work and which is not. Miller et al. (2008) conducted a meta-analysis to determine whether differences in efficacy exist among treatment approaches applied to therapies for youth. Included were all studies published between 1980 and 2005 involving participants 18 years of age or younger with diagnoses of depression, anxiety, conduct disorder and ADHD that contained direct comparisons among two or more treatment methods intended to be therapeutic. Effect sizes were found to vary significantly, providing some evidence that differences in efficacy exist among treatments for these disorders. However, researcher allegiance was found to be strongly associated with the difference in effect sizes, so that when allegiance was controlled evidence of differences among the treatments disappeared. This result is consistent with the finding that evidence-based treatments for youth are superior to usual care, only if the ‘evidence-based’ treatment was developed by the researcher (Weisz, McCarty, & Valeri, 2006).
Karver, Handelsman, Fields, and Bickman (2005) examined common process factors in youth and family therapy and concluded that the success of ‘empirically supported’ treatments is likely to depend on the presence of these common process factors. In a meta-analysis of treatments for depression in children, Weisz et al. (2006) found that, although various treatments were more effective than no treatment, no difference in outcome was found between cognitive and non-cognitive approaches. Spielmans et al. (2007), in a meta-analysis of component studies, found that the theoretically purported critical ingredients of cognitive–behavioural therapy (CBT) are not specifically ameliorative for child and adolescent depression and anxiety, as full CBT treatments offered no significant benefit over treatments with only components of the full model.
Treatments involving medication have found similar results. For example, the Treatment of Adolescent Depression Study (TADS), which is the largest trial ever conducted for childhood depression, claimed to show an advantage for fluoxetine, especially when combined with CBT (Treatment for Adolescents with Depression Study Team, 2004). The TADS consisted of two separate randomised studies: a double-blind comparison of fluoxetine (109 participants) and placebo (112 participants) and an unmasked comparison between CBT alone (111 participants) and fluoxetine plus CBT (107 participants). A ‘common factors’ perspective would predict that once participants are free from allegiance effects (which favoured medication in this study), any group differences would disappear. A follow-up study indeed found that the outcomes for all groups converged by week 30 (March & the TADS team, 2007). A similar process happened in the largest study of treatment for children with ADHD, the Multimodal Treatment Study of ADHD (MTA; MTA Co-operative Group, 1999). At 14 months the authors concluded that medication was superior to behaviour therapy; however, by 3 years all the advantages for those on medication had been lost, resulting in all group differences in terms of efficacy disappearing, although those exposed to medication experienced more adverse effects (Timimi, 2008).
Despite this growing literature that consistently points towards the importance of the ‘common factors’ perspective, most mainstream Child and Adolescent Mental Health Services (CAMHS) continue to apply a model of matching treatments to diagnosis.
The picture is even more concerning when real-life clinical practice is examined. For example, Kazdin (2004) reports that 40–60% of youth who begin treatment drop out against advice. Furthermore, according to the results of one meta-analysis, although the effect size for outcomes in controlled studies is large, the authors reported that for traditional treatment in the community, the effect size was close to zero (Weisz, Donenberg, & Weiss, 1995). In another randomised trial, Weiss and colleagues (Weiss, Catron, & Harris, 1999; Weiss, Catron, & Harris, 2000) found equal outcomes among treated and untreated children. These authors concluded that their data does not support the effectiveness of traditional child mental health treatments as currently practiced.
Other evidence finds that changes to service configuration, including allocating extra resources, have little impact on outcomes. The Fort Bragg evaluation described the implementation, quality, costs and outcomes of a US$94 million demonstration project designed to improve mental health outcomes for children and adolescents who were referred for mental health treatment. The experimental service provided a full continuum of mental health care, including outpatient therapy, day treatment, in-home counselling, therapeutic foster homes, specialised group homes, 24-hour crisis management services and acute hospitalisation. Extensive mental health data were collected on children and their families and evaluations continued for several years. Outcomes in the experimental service were no better than those in the TAU group, despite the considerable extra costs incurred (Bickman, Guthrie, & Foster, 1995; Bickman, Lambert, Andrade, & Penaloza, 2000). This finding was then replicated in the Stark County evaluation study, which examined a new system of care designed to provide comprehensive mental health services to children and adolescents. Again there were no differences in outcomes when compared with care received outside the new system, despite the extra costs. In addition, children who did not receive any services, regardless of experimental condition, improved at the same rate as treated children (Bickman, Summerfelt, Firth, & Douglas, 1997).
Findings on alliance also demonstrate their importance in work with children and families. A meta-analysis of methodologically robust studies of alliance in CAMHS reviewed 23 studies of different kinds of treatment and found a positive overall therapeutic alliance–outcome relationship across studies that was modest but robust (Shirk & Karver, 2003) and of a similar degree to that found in adults. Externalising disorders (i.e. behavioural) showed more variation in alliance, making alliance possibly of greater salience than for internalising disorders. Much of the alliance literature is focused on individual treatment relationships. However, in work with children and youth, alliances usually need to be made with parents and possibly others (such as other family members or professionals) too. In studies of alliance in child and adolescent in-patient units, parental alliance and child alliance were often not associated. Positive child alliance was independent of presenting diagnosis and symptom severity and parental alliance was related to pre-treatment family functioning (Green et al., 2001). According to Garcia and Weisz (2002), premature dropout from treatment at a community CAMHS was associated more with problems in the therapeutic relationship than other issues (such as lack of family motivation or practical difficulties). These all reflect not only the importance of alliance, but also the greater complexity and added systemic issues that arise in treatments for children (who still dependent on adults for much of the decision making) as compared to individual treatments (Green, 2006).
Peabody and MATCH
Given the disappointing results achieved by traditional diagnostic-based approaches to child and adolescent mental health problems, particularly as applied in real-life clinical settings, clinical models for CAMHS that incorporate patient feedback are beginning to emerge.
Bickman, who led the evaluations of the Fort Bragg and Stark County initiatives discussed above, concluded that improved outcomes would only occur if a ‘common factors’ perspective were used. For him, key is building in processes that improve the amount and quality of the information that clinicians receive about the effects of their intervention. Bickman and colleagues have recently developed ‘The Peabody Treatment Progress Manual’ (Bickman et al., 2007), which includes 11 measures of treatment progress and process that are generic in nature and so can be adapted to most types of treatment.
Weisz and colleagues have also developed an approach that includes patient feedback on progress. The MATCH platform prompts and guides therapists to systematically implement ‘evidence-based’ practices for patients with anxiety, depression, conduct problems and traumatic experiences. The system includes modules with patient feedback on progress helping practitioners choose which module or aspects of a module to use, particularly when no progress is being made (Chorpita & Weisz, 2009).
Neither of the above approaches have any published literature on their efficacy; however, studies are in progress. Because the PCOMS had an existing evidence base and had also developed a version of their rating scales for use with children and young people – the Child Outcome Rating Scale (CORS) and Child Session Rating Scale (CSRS) (Duncan, Sparks, Miller, Bohanske, & Claud, 2006) – it is to the PCOMS approach that Outcome Orientated Child and Adolescent Mental Health Services (OO-CAMHS) turned.
Outcome Orientated Child and Adolescent Mental Health Services
Working with children and adolescents has its own unique challenges. Contextual factors are crucial, as adults in caring relationships with them make important decisions about their lives. This meant that adapting PCOMS approaches to working with children and families needed to also incorporate the evidence related to the role of wider contextual factors (the extra-therapeutic factors). The approach that lies at the heart of OO-CAMHS is known as ‘Client Directed Outcome Informed’ (CDOI) (Duncan et al., 2010). Any interaction with patients can be patient-directed and outcome-informed when the patient’s voice is privileged, social justice is embraced, recovery is expected and helpers purposefully form partnerships to: (1) enhance the factors across theories that account for success – the so-called common factors of change; (2) use patient’s ideas and preferences (theories) to guide choice of technique and model; and (3) inform the work with reliable and valid measures of the patient’s experience of the alliance and outcome (Duncan & Sparks, 2010).
The OO-CAMHS model has been developed and implemented in a community CAMHS team in Lincolnshire. It won an East Midlands Regional Innovation Fund award in November 2010 to help develop the model and implement it across the Lincolnshire CAMHS. Part of the model includes obtaining session-by-session ratings of the young person’s progress (as perceived by the young person themselves and/or their parents/carers). This is done on the ORS and the CORS. OO-CAMHS also includes the young person and/or their parent or carer giving session-by-session ratings of their experience of treatment. This is done at the end of each session using the SRS or the CSRS. Each ORS or SRS rating takes only a minute or two to complete and so is relatively easy to incorporate into clinical sessions.
However, the model is not limited to or by the importance of obtaining good outcome and alliance data. Because of the salience of context, exclusive individual work is the exception rather than the rule and most clinicians also have to deal with pressures from the ‘system’ around the ‘identified’ patient. This can lead to blaming the system and/or blaming the family, as the system starts to mirror the family dynamics. Furthermore, we are concerned that collapsing the complex process of treatment into needing a few simple tools, risks replicating the mistakes of believing that treatment is a process of matching a standardised treatment to a simplistic diagnosis. The lead author has written extensively about the dangers this type of ‘dumbing down’ in CAMHS work (Timimi, 2009, 2010, 2011), and we did not wish to replace one ‘dumbed down’ system with another. So, in addition to session-by-session measurement of outcome and alliance, OO-CAMHS involves examining the system around the young person and team dynamics resulting in the ‘CORE’ guiding principles of: Consultation, Outcome, Relationship and Ethics of care.
Consultation
A total of 40–85% of variance of outcome is accounted for by extra-therapeutic factors, such as social support, parental mental health, socio-economic status and motivation (Duncan et al., 2010; Wampold, 2001). Understanding the broad context and real-life experience of our patients is therefore crucial. This should also make us take seriously the de-centring of our importance (and our techniques) to the process of change.
It is not uncommon for a young person with problems, and their family, to have a variety of different organisations involved (such as school, specialist educational support, social services, parenting advisor, etc.) by the time that person is referred to CAMHS. Without proper consultation with the other agencies involved, subsequent intervention by CAMHS may be compromised. Not only may clinicians be providing similar interventions, therefore unnecessarily duplicating clinical work, but also the young person and their family may become bamboozled by conflicting advice. Either way, therapeutic relationships will be diluted and disorganised as patients and clinicians try to establish who is doing what.
Thus, before embarking on treatment and during treatment we actively think about the external factors/system around the patient. Complex cases can often be created by over-intervention, which distances people from their existing strengths, abilities and resilience and instead re-enforces feelings of vulnerability and lack of coping. We try to avoid more than one agency working on any one problem at any one time. We use professionals’ meetings when one agency or more are already involved with the problem/issue the patient has been referred for. Sometimes it is better to delay becoming involved or indeed decline becoming involved until the other agency/worker has finished their therapeutic input. Duplication of therapeutic input is usually confusing for the patient, despite the best intentions of professionals. This problem seems particularly prevalent when working with children in care who may undergo many assessments and multiple treatments.
It is not uncommon for agencies to imagine that CAMHS has ‘magical’ powers and that a diagnosis will lead to a particular effective treatment, with the broader context being irrelevant to this process. Explanation that diagnoses in psychiatry simply describes sets of observed behaviours and reported experiences that often go together, but does not explain the cause or what treatment to use (this can include reference to the ‘common factors’ literature cited above) is useful. This can open up opportunities to build on existing relationships with, for example, care staff and/or foster parents that may have been missed because of the mistaken belief that only specially trained professionals know how to deal with the young person’s problems. Reducing to the minimum the number of professionals involved is often more empowering than increasing them. The patient (depending upon age and/or when appropriate) and their family can be at the heart of discussions to inform the major stakeholders of their current needs and to direct future input from agencies.
Outcomes
Session-by-session monitoring of outcomes with regular feedback to the patient of how they are progressing can by itself improve the outcome. If the outcome is showing no signs of improvement after 3–6 sessions, there is a high risk of no improvement from treatment or even a worse outcome (Lutz, Stulz, & Köck, 2009). Monitoring outcomes keeps the clinician focused on whether what they and their patients are doing together is making a meaningful difference from the patient’s point of view and doing something different if it is not.
OO-CAMHS uses the ORS and CORS (for 8–12 year olds). The young person and/or their parent/carer at the start of every session fill these in. Because of its brevity, the ORS/CORS is more clinician friendly and patient centred than most ratings scales. If session-by-session measures do not meet the time demands of real clinical practice, clinicians and patients alike may use them with reluctance at best, and resistance at worse. Much of the fear and loathing involved doing session-by-session measures is not as strong with the ORS and SRS, as they often take only a minute or so for administration and scoring. Not only is it ultra-brief (four questions), but, in addition, and unlike standard rating scales, it does not ask the equivalent of leading questions that then shape what should be considered a healthy/unhealthy, normal/abnormal behaviour or experience. The ORS/CORS asks the person to put a mark on a 10 cm line that best approximates to their current experience (if it is their first session) or their experience since the last session with regards how they feel intra-personally, in close relationships (family), socially (school, friendships, work) and overall. The person then explains what each mark means, thus they set the parameters for what is meaningful to them. As there are four 10 cm lines, measuring where the person has put their marks means you can derive a total score out of 40. Scores go up if the patient is experiencing progress, unlike many diagnostic-based rating scales where scores go down as a result of improvement. This was felt to be better for OO-CAMHS, which wishes to focus more on increasing the young person’s sense of ‘wellness’, resources and resilience rather than the focus being on getting rid of ‘symptoms’.
A record is then kept of the outcome score session by session and it is plotted onto a graph. If no improvement has occurred after five sessions we discuss with the patient and/or his/her carers/significant others and/or the multidisciplinary team. At this point we consider a change of therapeutic approach, or a change of therapist, or agreeing a deadline with the patient after which a change in approach/therapist will be tried if there continues to be no change. We work hard to avoid getting stuck in long-term treatment with no accompanying improvement.
Doing sessional ratings provides an opportunity to track the patient’s data and share this with them. Due to the simplicity of the questionnaire design and scoring method (using a ruler to measure how far along a 10 cm line they are for each area), scores can be quickly gathered and plotted onto a basic graph (for example, see Figure 1).
An example of a graph from the session-by-session Outcome Rating Scale (ORS).
Emma’s data in Figure 1 shows she has rated herself as making good improvements by coming for treatment, but that the gains may be tailing off now. Presenting this data visually for each patient and/or their parent/carer provides a useful tool for monitoring and discussing what is happening in their treatment. In child and adolescent work, we often have two or more ratings, usually one by the young person and one by a parent (or one from each parent if parents are separated or disagree about their child’s problems). Different perceptions in the ratings can then be further discussed in sessions.
As the ORS is completed at the start of each session, the information is up to date, incorporates any recent improvement or deterioration and, crucially, is based on the patient and/or their parent/carer’s perceptions. It is always the patient who defines what is and is not a meaningful sign of progress or deterioration.
Relationship
Alliance, as rated by the patient, is by far the strongest factor from those within treatment, associated with improved outcomes. From the first session we strive to create a culture of strong interest in patient feedback. We seek honest feedback from the start and look for help from the patient to point out what is and is not making sense or proving helpful/unhelpful to them. Of course, honesty can never be guaranteed and much depends on the capability of the clinician to create a sense of safety to invite such feedback. Using a tool helps create a ritual, with the accompanying message (including at the symbolic level) that ‘we are interested in your feedback’.
We measure the alliance regularly, preferably at the end of each session. OO-CAMHS uses the SRS and CSRS. These only take a minute or two for the patient to complete. Patients are often reluctant to highlight problems or issues they have if simply invited to comment on how they experience the session, hence having a tool to rate alliance is useful at both a symbolic level and to institute a ‘ritual’ into treatment sessions that formally invites feedback. The score the patient and/or their carer give on the alliance measure is not as important as having the conversation with them about what can be done differently to improve the treatment from their perspective.
The SRS rates four areas of the treatment experience: being listening to and respected; the importance/relevance of the session content; the method/approach being employed; and the session overall. Once completed the ratings can be quickly scored and graphed to aid tracking (for example, see Figure 2).
An example of a graph from the session-by-session Session Rating Scale (SRS).
In the example in Figure 2, the clinician was getting positive feedback regarding all areas of the therapeutic relationship and treatment, resulting in high total scores initially; however, the dip in the most recent session highlights issues that need addressing before the patient (Emma) leaves. In particular, Emma has rated ‘the approach’ lower than others. The resulting brief discussion reveals that Emma is not keen to use a thought diary as recommended by the clinician. The clinician thanks Emma for her feedback and praises her for being able to be honest. Emma and the clinician agree that she will not use a diary and they will look at alternatives when they next meet the following week.
Like the ORS, the SRS is often used with both the young person and their parent or carer, as keeping both engaged in the therapeutic process can be crucial for securing a positive outcome. Issues of power are omnipresent when working with families. Not only are there power dynamics in any family, but also between patients and clinicians. This is particularly so when a child is put in a position of rating the adult ‘expert’ sitting in front of them. This can prove tricky for children, who may not feel that comfortable with giving criticism for elements they may not like about the treatment session. Learning to use the SRS as a therapeutic tool means being alert to such issues and learning to look for small differences that may prove to be helpful in opening up difficult conversations that truly give our patients a voice in shaping the therapeutic process.
Ethics of care
Building a team culture that is supportive to teams trying to put patient engagement and voice at the heart of their approach can be anxiety provoking. Support by senior clinicians and management is needed for the OO-CAMHS approach and use of rating scales to become embedded in team culture. It is hard for a lone clinician to continue using this approach if it is not part of their supervision and others in the team are not using it. Using rating scales for all patients means that a rich database of outcomes can be built, not just for each patient, but also for each clinician and the team as a whole. If this is not handled with clarity about what data is included in reports, it can become a source of anxiety for clinicians rather than a source of empowerment.
The OO-CAMHS approach involves building strong relationships with patients, which is mirrored by building strong relationships between clinicians. Clinicians are trained, compassionate and skilled people and team attitudes should reflect this. In the same way we wish to think about young people and families as being capable of sincere, reflective and creative thinking, so we should think this about clinicians. Strong team relationships makes it easier to ‘fail successfully’ as a clinician, and pass the patient who is not improving in treatment with one member of the team to another clinician in that team, without feeling this reflects badly on their clinical ability. To protect the integrity of this model in supporting clinicians, we advise that the database of outcomes should never be used to compare clinicians. League tables of performance (as we find in education at present) can create unhealthy paranoia and competitiveness instead of trust and co-operation, which is what the OO-CAMHS approach tries to foster in team relationships and clinical services.
Outcome measures have become commonplace in CAMHS teams but rarely do team members discuss the measures or use them to orientate care planning in supervision and team discussions. Using simple Patient Rated Outcome Measures (PROMs) such as the ORS gives team members (regardless of professional discipline or level of training) a means to track patients’ view of progress, which can form a good basis for discussion of case material with the patient’s view being central to the clinical formulation.
Becoming an outcome-orientated team can be especially useful when the patient is not improving or deteriorating. Clinicians will be able to spot these issues early, from the feedback they get in the sessions and the graphs of progress that result. Any case that is not improving (according to the patient’s rating on the PROM) can be brought for advice and support from their team before the problems escalate.
The OO-CAMHS model is designed not just to support young patients and their families by putting them at the centre of their own care, but by encouraging team members to support each other in using this approach. Clinicians can then also focus on building good relationships in their working lives. Good therapy sees positive value, strengths, acceptance and abilities in their patients. Good teams see positive value, strengths, acceptance and abilities in their clinicians.
Evaluation
We carried out an evaluation involving all cases seen in the team who have implemented the OO-CAMHS model and a comparable community CAMHS team with a similar number of annual referrals and staffing, for the six months between 1 April and 30 September 2011 (Table 1). This data is not research and there are multiple confounds (including staff leaving or being off sick) and so should be viewed with caution. They do, however, give some encouraging indication of the potential capacity of an OO-CAMHS team to improve the efficiency of the service. No reliable or comparable outcome data was available for the non-OOCAMHS team. However, using the database as available at the 1 October, and which included nearly all patients who had attended the OO-CAMHS team in the previous six months and many more over the six months prior to April, an average effect size of change on the ORS of 1.3 (very large) was achieved for a total of 306 ratings (to include young person and parents/carers ratings) for discharged cases and an average effect size of change of 0.7 (medium to large) was being achieved with a total of 326 ratings (to include young person and parents/carers) for currently open cases.
Comparison between an Outcome Orientated Child and Adolescent Mental Health Services (OO-CAMHS) team and a comparable non-OO-CAMHS service for cases seen between 1 April and 30 September 2011.
As can be seen in Table 1, one of the main outcomes seems to be improved therapeutic efficiency. Although largely speculative at present, we believe this improvement to have occurred primarily for three reasons. Firstly, the ‘Consultation’ principles means that multi-agency involvement in cases that could easily go on to long-term work are dealt with in a different fashion, often by activating the resources already present in the system rather than undertaking potentially unproductive work by CAMHS. Secondly, a move away from starting with assessments where the goal is diagnosis helps shift the service away from a discourse of chronicity and deficit that diagnosis can lead to. Our starting question is ‘Do we think our service can make a positive difference to this young person’s life?’ and not ‘What is wrong with this young person?’ (although this is also a question we ponder but it is not the primary one). Finally, the simple process of viewing as being potentially problematic when no improvement has occurred after five sessions creates a culture of active curiosity and change orientation, reducing the chances of cases ‘drifting’ in the system.
Conclusion
In this paper we have presented an overview of a project developed in a UK community CAMHS team. We examined the theory and evidence base on which the project was developed, and provided a summary of the model and some early comparative data. The project is now receiving national attention in the UK, including a growing collaboration with the CAMHS Outcome Research Consortium (CORC), a national UK group, supported by the Department of Health, which aims to foster routine use of outcome measures in work with children and young people who experience mental health and emotional wellbeing difficulties. Our on-going concern is the potential for projects such as ours to fall victim to the ‘model fidelity’ problem, whereby the model is imposed in a rigid manner that alienates clinicians, causing understandable resentments and resistance, which in turn can lead to poor outcomes. Given that clinician engagement is often the rate-limiting factor in the uptake of new ideas, we believe that care needs to be taken in the manner of implementation and the use of data. The data is there to act as a clinical tool, not to be put into comparative league tables. We hope that services that take up and develop an OO-CAMHS-style model will approach this task in the same collaborative spirit the approach aims to foster in clinical practice.
Footnotes
Declaration of Conflicting Interests
None declared.
Funding
The development of OO-CAMHS was supported by a grant from the East Midlands Regional Innovation Fund (MH 30011).