Abstract
The aim of the present research was to explore clinician attitudes to outcome measures and, in particular, the facilitators and barriers to implementing outcome measures. An up-to-date exploration of clinician attitudes is especially needed in the context of recent policies on the implementation of outcome measures in child and adolescent mental health services (CAMHS), and because evidence suggests that there is a disparity between policy recommendations and the use of outcome measures in clinical practice. Semi-structured interviews were conducted with nine CAMHS clinicians from a Mental Health Trust in South London. Two levels of implementation emerged from the analysis: (1) the service level, regarding the implementation of outcome measures across a service to inform service improvement and (2) the session level, regarding the implementation of outcome measures within individual clinical sessions. The present research described training and ongoing support as a crucial facilitator of use at both service and session levels. This included help overcoming local contextual barriers, such as resources, information systems and administrative processes. The research showed that a balance is needed between a mandatory and uniform approach across a service and providing clinicians with support to use outcome measures with all service users for whom they are appropriate.
The use of outcome measures during psychological therapy involves the regular review of feedback from measures of symptoms, functioning, or other common aspects of treatment, such as therapeutic alliance, reported by patients and/or therapists (Carlier et al., 2012; Knaup, Koesters, Schoefer, Becker, & Puschner, 2009). Outcome measures have been promoted in healthcare over the last three decades and it has become integral to modern, evidence-based, person-centred care founded on transparency and accountability (Department of Health, 2001; Gilbody, House, & Sheldon, 2002b; Newnham & Page, 2010; National Health Service (NHS) England, 2015; SAMHSA’s National Registry of Evidence-based Programs and Practices, 2015). More recently, the policy has recommended the implementation of outcome measures in child and adolescent mental health services (CAMHS) across England in order to facilitate clinical practice and service evaluation as part of the Children and Young People’s Improving Access to Psychological Therapies (CYP IAPT) service transformation programme (Department of Health, 2011).
Evidence suggests that there is a disparity between policy recommendations and the use of outcome measures in clinical practice. Previous evidence has found the use of outcome measures in CAMHS to be about 15–30% (Batty et al., 2013; Gilbody, House, & Sheldon, 2002a; Johnston & Gowers, 2005; Patterson, Matthey, & Baker, 2006). Nonetheless, due to initiatives such as CYP IAPT and the Child Outcomes Research Consortium (CORC) there has been an increase in the routine use of outcome measures (Hall et al., 2013). A range of barriers to outcome measures has been reported as explaining this disparity between policy and practice (Hall et al., 2014).
One of the most prevalent barriers to using outcome measures is the information technology systems for collecting measures and producing feedback (Bickman et al., 2016; Gleacher et al., 2015). On one hand, patient acceptance of outcome measures is generally high but on the other hand, clinicians question the clinical utility of outcome measures (Dowrick et al., 2009; Hall et al., 2014; Stasiak et al., 2012). In particular, a disparity has been reported in that clinicians may hold a generally positive attitude towards outcome measures, but still feel that they do not help to improve patient care (Martin, Fishman, Baxter, & Ford, 2011; Valenstein et al., 2004). For instance, the vast majority of clinicians reported not using data from measures in treatment planning or monitoring even when they had access to them (Garland, Kruse, & Aarons, 2003).
This reluctance may stem from clinicians placing more importance on their clinical judgment rather than on outcome measures (Crocker & Rissel, 1998; Dowrick et al., 2009; Hall et al., 2014). It has also been argued that measures can be depersonalising and unrepresentative by attempting to objectively measure work that is often subjective in nature (Batty et al., 2013; Moran, Kelesidi, Guglani, Davidson, & Ford, 2011; Norman, Dean, Hansford, & Ford, 2014; Wolpert, Curtis-Tyler, & Edbrooke-Childs, 2016). Concerns have been raised that data from outcome measures are not used for the best interests of service users but rather those of managers in terms of service evaluation and efficiency savings (Johnston & Gowers, 2005; Norman et al., 2014).
However, outcome monitoring has been shown to produce a deeper clinical understanding by increasing the availability of meaningful evidence and facilitating better communication between patients, parents and clinicians (Emanuel, Catty, Anscombe, Cantle, & Muller, 2014). The feedback process of outcome monitoring has similarly been shown to make a difference to treatment effectiveness compared to patients receiving no feedback, on at least one treatment outcome (Gondek, Edbrooke-Childs, Fink, Deighton, & Wolpert, 2016). Training child mental health clinicians to use outcome measures has been shown to be associated with higher levels of positive attitudes and self-efficacy related to administering measures and using feedback from measures, which may suggest that training could help to overcome the above reluctance to using measures (Edbrooke-Childs, Wolpert, & Deighton, 2016).
Aim of the present research
The above evidence suggests that there is a disparity between the policy recommendations for the use of outcome measures in CAMHS and the prevalence of use in clinical practice. Therefore, the aim of the present research was to explore clinician attitudes to outcome measures and, in particular, the facilitators and barriers to implementing outcome measures. An up-to-date exploration of clinician attitudes is especially needed in the context of recent policies on the implementation of outcome measures in CAMHS (NHS England, 2015). Findings of the present research will add to the literature by identifying lessons from clinicians’ experience to inform recommendations to support the use of outcome measures in CAMHS.
Method
Semi-structured interviews were conducted with nine CAMHS clinicians from a Mental Health Trust in South London. Clinicians had seven separate job roles, meaning that they had completed different types of training and used a range of therapeutic modalities. Correspondingly, they had been trained in outcome measures at different stages in their career. Three clinicians reported using outcome measures regularly in every session as a result of CYP IAPT training; four clinicians reported using measures at assessment, review and discharge; and two clinicians reported not using outcome measures regularly. The two clinicians not using outcome measures regularly worked in specialist services, with higher intensity cases.
The interview schedule was designed to explore clinicians’ attitudes towards outcome measures. Interviews were conducted by a co-author (VG) working within the same Trust. All interviews were audio-recorded and transcribed verbatim and lasted between 17 and 41 minutes. The Trust Research and Development department classified the project as service evaluation and gave approval for the conduct of the interviews, which was confirmed by our local University Research Ethics Committee. All participants gave consent before taking part.
The overall analysis took a critical realist approach, aimed at interpreting the reality of the data within the contexts of the participants and the researcher to discover underlying meanings and constructs (Willig, 2012). An inductive thematic analysis was used with the objective of allowing emerging patterns in the data to inform the themes within the parameters of the research question. This approach consisted of four initial phases (Braun & Clarke, 2006). Phase 1: familiarisation with data through transcribing and rereading. Phase 2: NVivo (Bazeley & Jackson, 2013) was used to code interesting aspects of the data in a systematic way. Phase 3: similar codes were then grouped into themes based on the researcher’s judgment with two overarching categories of interest identified (service level and session level; see Results for details). Phase 4: a section of the data was then checked by the interviewer to ensure consistency in coding.
Results
Four superordinate themes emerged from the analysis – standardisation, training, practical experience and resources – each of which is discussed in detail below. In addition, two levels of implementation of outcome measures emerged from the analysis: (1) the service level, regarding the implementation of outcome measures across a service to inform service improvement and (2) the session level, regarding the implementation of outcome measures within individual clinical sessions. Each of these levels is considered in relation to each of the four themes below.
Standardisation
Standardising (or mandating) implementation was described as facilitating use at the service level but as a barrier at the session level. Two-thirds of the clinicians explained how the use of outcome measures in sessions with patients was compulsory within their service (i.e. because of CYP IAPT). On one hand, this was reported as facilitating the use of outcome measures. Two clinicians identified that the established credibility of CYP IAPT due to it ‘being rolled out nationally’ (Clinician 6) and because its ‘systems are better evolved’ (Clinician 8) encouraged them to use measures more consistently. Clinicians described several ways in which ‘actually [using] measures in a more consistent way’ (Clinician 2) encouraged an ‘evidence-based’ approach to practice (Clinician 1) and service improvement through ‘the collaborative thinking about using this, sort of meeting the targets in terms of data reporting’ (Clinician 7) and sharing ‘what was going on in other services’ (Clinician 6). On the other hand, approximately half of the clinicians also expressed some concerns about having a mandatory set of measures across the service, which was described as a ‘one-size-fits-all approach’ (Clinician 4).
Despite some clinicians reporting standardisation to be a facilitator at the service level, clinicians unanimously reported it as being a barrier at the session level. One clinician voiced this by stating that ‘we have to do them regardless of whether we feel they are helpful or not’ (Clinician 5). Clinicians raised several ways in which standardising the use of outcome measures was potentially having a negative impact on clinician–therapist relationships; for example, by taking time away from session discussions, not considering the differences in client needs, and that it ‘risks somewhat alienating clients’ (Clinician 3). One clinician went as far as to say that the mandatory use of measures can feel completely pointless, ‘like a tick-box exercise, I am not really interested in what they show’ (Clinician 7).
In particular, if service users were resistant to using measures in a session, clinicians reported that they felt the need to encourage the continued use of measures, which was reported as being detrimental to the therapeutic alliance by some, with others stating that service users ‘don’t really bat an eyelid, they just do it’ (Clinician 2). A particular cause of service user resistance reported by clinicians was the language of outcome measures. A third of clinicians raised concerns that patients have misinterpreted the questionnaires, due to the language not being accessible, which caused them to question their accuracy. Some clinicians raised concerns over service users finding the language distressing or difficult to understand, potentially leading to inaccurate responses (also see the theme ‘Practice’). Explaining the reasons for using outcome measures with service users was described as particularly important to ensure use in a ‘meaningful way so that families don’t get fed up’ (Clinician 7).
Relatedly, some clinicians noted that there could be inaccurate data collected by outcome measures. For example: ‘sometimes young people [who] struggle with endings … they may use the routine outcome measures as a way of reporting symptoms returning’ (Clinician 6) and there can be ‘huge gaps in session-by-session measures’ (Clinician 6) due to session non-attendance.
Training
Training was described as facilitating use at both service and session levels but ongoing support was recommended to sustain use. Two thirds of the clinicians expressed several ways in which they had been supported by their service or team, or ways that they could be supported, to use outcome measures. In contrast to the previous theme, recommendations were to increase the structure of use of outcome measures by, for example, creating a ‘strong supervision structure … to look at the graphs a lot and to monitor the ratings’ (Clinician 6), assigning an outcome measures champion to add ‘enthusiasm and meaning’ (Clinician 8) or training using observations or ‘doing seminars on the different outcome measures and how to use them’ (Clinician 1). Training was mentioned by the majority of clinicians, with a third identifying it as a means of increasing knowledge and confidence.
Still, one clinician noted that although training may create a ‘flurry’ of activity, it may not necessarily be sustained and ‘sort of dies all over again’ (Clinician 9). Nevertheless, those who had received training reported it as giving them an advantage over their colleagues as it allowed them to learn about the benefits of outcome measures and how to integrate them into their daily practice. One clinician explained that ‘it’s a natural part of my clinical practice now’ and they did not feel that continued use was much of a culture change (Clinician 1).
Practical experience
Practical experience of using measures was described as a facilitator of use at both service and session levels. When the use of outcome measures was seen as appropriate for a session, practical experience was described as one of the main facilitators to using measures. At the service level, there was a tension in the reported use of outcome measures for informing commissioning decisions if the ‘same outcomes happen after ten sessions as they do with six sessions … in terms of keeping waiting lists down and pressures on services … that’s going to be quite helpful’ (Clinician 1) against reports about ‘anxieties about how the data might be used’ (Clinician 2).
At the session level, the majority of clinicians felt that the measures were useful to establish a basis for treatment or to inform direction, but that the ultimate autonomy and flexibility of use should lie with the clinicians themselves. A third of clinicians explained how some patients liked the structured format of the measures, giving a framework to the discussion and monitoring of progress, making presenting problems ‘an easier thing to talk about’ (Clinician 4). Most clinicians described how outcome measures had helped them to reflect on therapeutic work and improvements in service users’ outcomes. One clinician emphasised this by explaining how measures had encouraged collaborative practice during sessions and allowed patients ‘to be active in their own treatments’ (Clinician 2). Presenting and discussing change during the course of therapy was recognised as a benefit for engaging families in care and ‘helps people feel really taken seriously’ (Clinician 8) or ‘drive[s] them to use those sessions or value those sessions more’ (Clinician 3).
In contrast, presenting and discussing a lack of change or even deterioration was described as a barrier to implementation. Other clinicians felt that more broadly, the outcome measures needed to represent something meaningful and useful for service users and their families (also see the ‘Standardisation’ section).
Resources
Resources such as data systems and administrative support across a service were described as having the potential to facilitate use, but in practice tending to be a barrier. A third of the clinicians identified that information technology systems could help to streamline the processes of using outcome measures for easier implementation, in terms of entering data and providing feedback. However, introducing new systems was identified as a barrier. One clinician explained: ‘I think we lost a lot of people because we kept changing systems’ (Clinician 2). Moreover, another clinician added that systems can be a barrier to implementation if they are ‘clunky’ and do not allow for linking of data, causing ‘a disconnect between the clinical work and the data of measures’ (Clinician 7).
Administrative support to ensure the organisation of measures and paperwork was also mentioned as making ‘a big difference’ by several clinicians. Despite this, some clinicians raised concerns of central administration control over allocation and distribution of outcome measures being a barrier, reflecting the views raised in the theme ‘Standardisation’. One clinician voiced this by saying that ‘autonomy is taken away from us in the sense that they are routinely sent out by the service before we’ve seen them’ (Clinician 1). Another voiced frustration that ‘collecting an outcome on some kind of questionnaire and sticking it in a pile to be uploaded by an administrator in about six months’ time, it is really rather pointless’ (Clinician 7).
At the session level, not having time to complete measures was described as a barrier. For example, ‘some of the things that [patients] really were hoping to speak about in their session with their therapist, there wasn’t time for those things to be discussed because they had to spend a lot of time doing outcome measures’ (Clinician 9). Another clinician explained that not having the capacity to score up measures and feed them back led to the conclusion that ‘that wasn’t a useful session’ (Clinician 2).
Summary of findings
Two levels of implementation emerged: the service level and session level. Training in and practical experience of using outcome measures were reported as facilitating implementation, with ongoing support needed to sustain use. Resources, in terms of information systems, administrative processes and time within sessions, were more often described as being prohibitive, rather than supportive, of use. A tension emerged in clinicians’ descriptions of the standardised use of outcome measures. On one hand, this was described as a barrier to implementation at the session level, with clinicians and service users struggling to use them when they were not seen as appropriate. On the other hand, it was also described as facilitating the use at the service level, with recommendations to support use in the future including further systematisation (e.g. through training and supervision). Similarly, clinicians described the structured content of measures as resulting in them, at times, being misinterpreted by service users, or causing distress and disengagement. In contrast, the structured content of measures was also described as being useful to service users to help frame discussions of presenting problems and treatment planning.
Discussion
The aim of the present research was to explore clinician attitudes to outcome measures and, in particular, the facilitators and barriers to implementing outcome measures. It highlights lessons from clinicians’ experience to support the use of outcome measures in CAMHS.
The two levels of implementation that emerged from the analysis are in line with previous evidence showing that training clinicians to use outcome measures was associated with increases in levels of positive attitudes and self-efficacy related to administering outcome measures and using feedback from measures (Edbrooke-Childs et al., 2016). Clinicians discussed the value of gaining practical experience of using outcome measures to facilitate use in sessions, with examples including promoting collaborative practice between service users, families and clinicians; having the ability to review progress; and informing decisions about diagnosis and treatment.
In line with previous research (Valenstein et al., 2004), although clinicians reported that bespoke resources could, in principle, be a facilitator to using outcome measures, in practice they were actually prohibitive. Clunky information systems, burdensome administrative processes or ones which bypassed the clinician in the process of sending and receiving outcome measures, and insufficient time within sessions to complete measures were the main barriers reported.
A tension emerged in clinicians’ descriptions of the standardised use of outcome measures. On one hand, this was described as a barrier to implementation at the session level, with clinicians and service users struggling to use them when they were not seen as appropriate, in line with previous research (Martin et al., 2011). On the other hand, it was also described as facilitating the use at the service level, with recommendations to support use in the future including further systematisation (e.g. through training and supervision). Similarly, clinicians described the structured content of measures resulting in them, at times, being misinterpreted by service users or causing distress and disengagement. In contrast, the structured content of measures was also described as being useful to service users to help frame discussions of presenting problems and treatment planning, also in line with previous research (Hall et al., 2014; Unsworth, Cowie, & Green, 2012).
Limitations should be considered when interpreting the findings of the present research. The sample was recruited from one Mental Health Trust and the sample size was small. Although this is consistent with qualitative research, saturation was achieved, and clinicians in our sample had a range of roles and experiences of using outcome measures. Future research should replicate the present findings in larger, more heterogeneous samples. To triangulate interview data, future research should collect other forms of data, such as observations of practice, to provide more objective data on actual use of outcome measures. Finally, the interviewer worked in the same Trust as the participants. On one hand, this may have influenced clinicians’ responses and they may have felt inhibited to express negative attitudes towards outcome measures, particularly as the Trust was involved in a service transformation programme – a central part of which is the implementation of outcome measures. On reflection, we think it is unlikely that this occurred as clinicians expressed a range of both positive and negative attitudes and experiences of using outcome measures. Moreover, we think that the interviewer’s role may have actually made clinicians feel more able to discuss negative attitudes, particularly as the rest of the research team are based in an institution perceived to be supporters of outcome measurement. The interviewer’s role provided helpful contextual knowledge when analysing and writing up the findings.
Findings from the present research suggest that clinicians view training in outcome measures, including the opportunity to gain practical experience of using them, as a crucial facilitator to their use at both a service level and session level. Ongoing support is needed, which could be through supervision or, as previous evidence suggests (Hall et al., 2014), with a learning collaboration. Either way, help overcoming local contextual barriers, such as resources, information systems and administrative processes, is vital. Existing training available for clinicians to use outcome measures (Edbrooke-Childs et al., 2016) teaches that outcome measures should only be used when it is clinically appropriate to do so, that is, where a patient has the capacity to formulate a rational answer, for example not when very depressed or psychotic (Emanuel et al., 2014). Training should also be provided at the service level to ensure this message is spread across a service. This may be one means of helping services and clinicians to strike the balance of a standardised or mandatory approach, reported as a facilitator at the service level but a barrier at the session level. In doing so, clinicians may be supported to use outcome measures with all services users for whom they are appropriate.
Footnotes
Acknowledgements
The authors would like to thank members of the Policy Research Unit in the Health of Children, Young People and Families: Terence Stephenson, Catherine Law, Amanda Edwards, Ruth Gilbert, Steve Morris, Helen Roberts, Cathy Street and Russell Viner.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article:The Policy Research Unit in the Health of Children, Young People and Families is funded by the Department of Health Policy Research Programme. This is an independent report commissioned and funded by the Department of Health. The views expressed are not necessarily those of the Department.