Abstract
Objectives
To examine the challenges to using systematic review evidence to develop guidance for decommissioning ineffective health services, and the problems experienced by clinicians and commissioners when they attempt to implement the evidence from this guidance.
Methods
Interviews with 23 clinicians and 15 commissioners from nine commissioning organizations (Primary Care Trusts) in the south of England.
Results
Participants identified generic and intervention-specific barriers to using systematic review evidence to develop and implement decommissioning. Generic barriers included: contradictions within the health care system arising from policy; managing a high volume of evidence; difficulty in applying the evidence to the local context; and patient or parent expectations. Intervention-specific factors included: the influence of industry; an absence of systems for monitoring local implementation of guidance; and the availability of different codes for the same procedure which made monitoring some practices unreliable.
Conclusions
The micro practices of commissioners are shaped by the wider system of health policy, the knowledge producing and delivery agencies associated with health care, and power dynamics within the health care system. If decommissioning is to be guided by evidence, then adequate resources to support the process are necessary. This includes long-term engagement of clinicians, providing alternatives to the decommissioned activity and tackling perverse incentives. An important precursor to decommissioning is obtaining data on the nature and extent of current clinical practice and using these data to monitor variation in the implementation of guidance.
Introduction
The return on investment in research, through the translation of research findings to health care, continues to be a concern.1–3 Efforts to maximize the translation of evidence include audit and feedback, clinical practice guidelines and patient decision aids 4 and have largely focused on clinicians as adopters and users of research evidence.5–8 Recently, attention has shifted to policy makers and commissioners as intermediaries in the evidence translation process. 9 For example, in England, the Audit Commission has drawn attention to the need for research evidence to guide the decommissioning of ineffective health care interventions. Decommissioning is defined as stopping the provision of a service or a significant part of a service in order to bring about an improvement to existing service provision. 10 The objectivity and evidence base of commissioning has been criticized, and the ability to systematically incorporate decommissioning has recently been questioned.11,12
Evidence from systematic reviews is essential to knowledge translation as it provides a transparent and robust method for identifying interventions that are either ineffective or unproven. We examined the challenges to using evidence from Cochrane systematic reviews to develop guidance for decommissioning (when a decision is made that a product or service is no longer required), and the problems experienced by clinicians when they attempt to implement the evidence from this guidance into clinical decisions.
Methods
Recruitment and sampling
We interviewed health care commissioners from Primary Care Trusts (PCTs) who produce commissioning guidelines and clinicians who apply these guidelines in the South Central region of England. Initially, participants were identified through contact with the two priority setting forums serving nine PCTs: Milton Keynes, Oxford, East and West Berkshire and Buckinghamshire (MOBBB) and Southampton, Hampshire, Isle of Wight and Portsmouth (SHIP). We then used the ‘snowballing’ technique to identify commissioners and clinicians who had knowledge of the interventions we had selected as potential candidates for decommissioning.
Interviews
Participants were interviewed in their professional capacity using a purposive sample of ‘difficult’ and ‘easy’ decisions around diverting funding from five health care interventions that are ineffective or where there is insufficient evidence supporting their use. Participants from areas of relatively high and low use areas for these interventions were invited to be interviewed. Usage data for the interventions were obtained from Solutions of Public Health, a non-for-profit organization which provides specialist health intelligence services. The interviews followed a semi-structured guide which included factors that may facilitate or limit the application of evidence.
We selected the following five interventions as examples of interventions used in primary or secondary care (Box 1):
Arthroscopic debridement for knee osteoarthritis.
17
Grommets (ventilation tubes) for hearing loss associated with otitis media with effusion in children.
15
Long-acting insulin analogues compared with NPH insulin (human isophane insulin) for type 2 diabetes mellitus
14
and short-acting insulin analogues compared with regular human insulin in patients with diabetes mellitus.
13
Minocycline for acne vulgaris: efficacy and safety.
18
Tonsillectomy or adeno-tonsillectomy versus non-surgical treatment for chronic/recurrent acute tonsillitis.
16
Conclusion from the Cochrane Systematic Reviews used in the interviews.
Interviews were conducted face to face or by telephone between January and May 2011. The participants agreed for their interviews to be recorded. Interviews continued until no new themes were mentioned.
Analysis
The interview data were triangulated by RA, SD and SS. Each person read the transcripts and identified key themes; we then met to identify and discuss the main, recurrent and most important themes. We categorized these as macro (national policy factors), meso (factors at the level of the health care organization) and micro level factors (individual factors) that might influence participants’ use of evidence. This provided an explicit framework to capture the interplay between national policy, organizational context and clinicians and commissioners use of evidence. This builds on previous research which explored the utilization of research by health care managers. 19 We used Oxman and Flottorp's barriers to the adoption of evidence-based practice as a conceptual framework to guide this thematic analysis. 20 In addition, we adopted a data-driven approach to ensure we identified themes related to decommissioning. 21 This process was facilitated by coding all the interview transcripts using NVivo software. 22
Results
Twenty-three clinicians (13 hospital consultants, six general practitioners (GPs) and four specialist nurses) and 15 commissioners (two GP commissioners, three pharmacists, six public health consultants and four health care managers) were interviewed. Just over half (55%) were from high use areas. Twenty-five interviews were face to face and 13 by telephone and lasted for an average of 40 min. There was some variation in responses between commissioners and clinicians but little variation in responses between high and low use areas for each of the five interventions.
Macro level factors
A number of policy-related variables were cited, by commissioners and clinicians, as introducing contradictions within the health system. A consequence was that the implementation of some policies objectives, for example a reduction in waiting lists or a shift of care from secondary to primary care, prevented decommissioning. Contradictions observed included the different interests multiple players brought to the process of implementing policy, the complexity of running different policy agendas in tandem and performance-based payment incentives. Prescribing insulin analogues illustrates this as their introduction coincided with the shift of care for people with diabetes from secondary to primary care.
‘Insulin analogues, particularly increasingly so now, are initiated in general practice, so there’s a much wider group of people who we need to influence, both general practitioners and nurses, there’s even the wider primary care health community as well who might interact with the patient, pharmacists, district nurses, all of that. So there’s a much bigger group of people who we need to get to, to get on board with this issue’. [Commissioner]
‘… and there’s the escalation in GPs trying to reach their QOF targets and, and the analogues particularly being put out as an easy way of starting somebody on to, you know [reach those targets even though] they’re not sticking to the NICE guidance … it’s supported by possibly inappropriate QOF targets, you know, there’s been a lot of focus on driving HBA1c down’. [Commissioner]
‘And of course the issue is HBA1C creep which is “I’m not hitting my QOF targets anymore” which is suddenly the QOF target has made it important for me to do this’. [Clinician]
‘Payment by results has a lot to answer for because you don’t actually have much control as a commissioner anymore because whatever the provider chooses to do, they do and you are contractually bound to pay for it unless you have specifically said that you don’t’. [Commissioner]
‘So in a culture where, you know, you’re paid for every operation you do, you might do more operations, in a culture where there are financial limitations then you, we’re going to be doing less operations. So evidence-based medicine I think is, is great, I don’t think we’ve got it though’. [Clinician]
‘………… there is a complete lack of a system in this organisation that picks up and highlights clinical information like this that is useful. So I would say that there is no particular information feed that routinely says right, what’s come out of NICE, what’s come out of Cochrane, what’s comparing our, you know, we do it as a one-off exercise looking at the Croydon list and all of these and benchmarking ourselves. But, you know, there’s nothing regular to say we’re doing an awful lot of arthroscopic debridement’. [Commissioner]
Meso level factors
The findings from our interviews highlighted the complex environment that commissioners work in and, for many, decommissioning is a novel process which is evolving. The lack of an explicit framework can result in strong resistance and skepticism from clinicians, can lead to ambiguity within the health system and can create tension between clinicians and commissioners. Some clinicians expressed a lack of faith in the process and concern that cost considerations were placed ahead of effectiveness. Decommissioning was described as a social process and one that lacked structure to monitor the process. Commissioners valued working with the same clinician for the duration of a decommissioning process, viewing this as essential in reducing the level of ambiguity in decision-making. ‘Decommissioning is more difficult to do because it upsets people and it goes against … If you’ve always done it and then suddenly someone’s saying “Actually we shouldn’t be doing this”, it’s a harder one to swallow. And commissioning something’s easy because everybody’s keen, everybody wants to do it, we think we’re doing a good thing, we’re putting our money into something that we think is going to have good outcomes’. [Commissioner] ‘So having a framework [for decommissioning] is key, and we are actually trying to decide what are the parameters for a disinvestment and we haven’t yet … because it’s actually very difficult to look at the combination, the whole matrix of factors that we need to consider’. [Commissioner]
‘It’s about weighing up the amount of effort you need to put in to change behaviour and what are the financial benefits to it’. [Commissioner]
‘So whenever you are decommissioning I think you’ve got to look at the balance of decommissioning something but also to say “Well that practice is going to lose that much income, is that going to destabilise it?”’ [Commissioner]
‘Well this is actually an intervention that’s quite difficult to monitor through our standard mechanisms...but, the trouble with this particular policy is that the intervention, arthroscopy, is used for other things. And therefore actually monitoring it, the activity, is actually really quite difficult. So what we have had is conversations with the clinicians who assure us that they’re not doing it, but that is really unsatisfactory from our point of view’. [Commissioner]
Micro level factors
The main challenges experienced by individuals to decommissioning were dealing with clinical uncertainty, prevailing opinion, skills and training and patient expectation. Clinicians reported experiencing these to a greater extent than commissioners. Factors raised by clinicians included the challenge of identifying a priori precisely which patients benefit. Inevitably, if there is uncertainty about whether or not a health care intervention delivers benefit to a patient, some patients will receive an intervention unnecessarily.20,24
‘I don’t think there’s any evidence to suggest it’s (tonsillectomy) not an extremely good operation in certain people, I think that’s the difficulty’. [Clinician]
‘I admit it’s very difficult to know which patients are going to benefit from these treatments, but when the only alternative is something along the lines of an osteotomy or an arthroplasty, it’s very difficult to decline someone arthroscopic surgery even if we’ve only got say a fifty percent chance of making them better’. [Clinician]
‘… we always think back to our historical patterns of practice because they’re the ones we know and love and they’re tried and tested and they’ve been around for ages. And so if we’re scratching for the exact details, the default position is more “What did we do last time?”’ [Clinician]
‘So I think for a surgeon we base [practice] on surgical confidence, surgical experience and the teaching that we’ve had, which is cultural and the way we were taught’. [Clinician]
‘But still in some of our practices, I think our data actually are skewed by a very small number of practices, so we have a very small number who, who for various cultural reasons do not like to say “No” to their patients’. [Commissioner]
Discussion
Main findings
There are difficulties faced by health care commissioners and clinicians when using evidence from systematic reviews to develop and implement guidance for decommissioning. Participants identified generic and intervention-specific barriers to decommissioning, which have relevance to health systems beyond the NHS in terms of delivering cost-effective health care. Generic barriers included unintended consequences of policy, systems for managing information, translating evidence to the local context (including sub groups of patients) and patient or parent expectations. Intervention-specific factors included the influence from industry in the form of training specialist primary care nurses in the care of people with diabetes 25 and the availability of different codes which made monitoring some practices inaccurate. An important precursor of decommissioning is capturing the nature and extent of current clinical practice; without these data, it is not possible to monitor use. 26
The concern that NHS commissioning has failed to live up to expectations is receiving increasing attention. The House of Commons Health Select Committee 12 cited a number of missed opportunities to improve the delivery of cost-effective care. Examples include the absence of levers to ensure that hospitals use NICE guidelines, a lack of capacity and skills and perverse incentives arising from national policies which incentivize hospitals to increase activity. These factors correspond with our findings and require attention if interventions of limited clinical value are to be successfully decommissioned.
Decommissioning is a complex challenge for health systems and one that cannot be tackled in a programmatic way. It depends on a set of social processes which include: sensing and interpreting new evidence and integrating it with existing evidence (including tacit evidence); reinforcement by professional networks and communities of practice; relating the new evidence to the needs of the local context; and discussing and debating the evidence with local stakeholders before taking joint decisions about its enactment and changing practice.
Features of contextual receptivity for decommissioning include structural features (the degree of system complexity and volume of clinical work) and aspects of process (the historical development of services and the foundation of good prior relationships) together with some features of action (credible opinion leaders and project management skills amongst commissioners; and support from senior management). A more sophisticated notion of decommissioning is needed that acknowledges local contexts which can be seen both as external to decommissioning and internal.
Limitations of this research
First, the pattern of responses did not differ between the high and low users of the five interventions. One explanation is that by relying on aggregate data about usage, we may have interviewed clinicians with high usage rates who happened to be located in a low use PCT (or vice versa). A second limitation is that the study was limited to one region of England. Third, we examined the use of only one type of evidence, from Cochrane systematic reviews, and recognize that other factors have a major role to play in commissioning decision-making such as other sources of evidence, 27 equity, ethics, affordability and the quality of provision. 10
Conclusion
It is well established that the simple availability of evidence in the form of guidance is not sufficient to prompt its adoption.28,29 In this paper, we have presented a range of factors that we found to mediate the decommissioning process. It would be a mistake to see macro, meso and micro factors as somehow separate. The micro practices of commissioners are shaped and nuanced by the workings of the wider system of health policy, the knowledge producing and delivery agencies associated with health care and the existing power dynamics within the health care system. Context cannot be seen as a set of static and independent variables or an ordered series of hierarchical layers but as a syndrome of forces which interact in complex ways and lead to unintended outcomes. As in other studies of health care,30 we noted the potential for tension between the professionally distinct groups of clinicians and commissioners which hindered the uptake of evidence. Tension was most evident regarding the credibility and legitimacy of the (de-) commissioning functions, reflecting anxiety that finance rather than evidence drives the process. The House of Commons Health Committee 12 also noted that relations between these groups could be adversarial. Whether or not the quest for credibility and legitimacy will be satisfied by the replacement of PCTs with Clinical Commissioning Groups from April 2013 remains to be seen. If decommissioning is to be guided by evidence then adequate resources to support the process are necessary, including long-term engagement of clinicians, monitoring variation in implementation, providing alternatives and tackling perverse incentives.
Footnotes
Note
The views in this article are those of the researchers and not necessarily those of the Department of Health.
Funding
This project was funded by an NIHR Cochrane NHS Engagement grant.
