Abstract
Objectives
To examine whether the Health System Reforms delivered the promise of being a coherent and mutually supporting reform programme; to identify the underlying programme theory of the reform programme; to reflect on whether lessons have been learned.
Methods
Documentary analysis mapping the implicit and explicit programme theories about how the reforms intended to achieve its goals and outcomes. Semi-structured interviews with policy-makers to further understand the programme theory.
Results
The Health System Reforms assumed a ‘one size fits all’ approach to policy implementation with little recognition that some contexts can be more receptive than others. There was evidence of some policy evolution and rebalancing between the reform streams as policy-makers became aware of some perverse incentives and unforeseen consequences. Later elements aimed to restore balance to the system.
Conclusions
The Health System Reforms do not appear to comprise a coherent and mutually supportive set of levers and incentives. They appear unbalanced with the centre of gravity favouring suppliers over commissioners. However, recent reform changes have sought to redress this imbalance to some extent, suggesting that lessons have been learned and policies have been adapted over time.
Introduction
International health care reforms vary over time and space. Toth 1 claims that there have been distinct and sometimes opposing waves of reform over the last 20 years, characterized from competition to cooperation and back. Countries have different starting points and reform journeys.1,2 Between 1997 and 2010 the UK Labour government produced an ambitious programme of reform for the English National Health Service (NHS). It composed many different elements,3,4 with policy reforms being layered to form a complex temporal path.
In this paper, we focus on the Health System Reforms as set out in Health reform in England Update and Next Steps 5 , which can be seen as part of Toth's third wave reforms that focus on patient rights through choice and competition. The Health System Reforms were part of an evolving system that wove existing policies together and sought to incorporate future policies into the NHS.2,5 In short, it was a reform programme rather than a single event that set out the direction and momentum of reform but also established strong processes for identifying any unintended consequences, feeding back lessons learned and, in turn, for adapting the reforms over time.
The main aim of this paper is to examine whether the Health System Reforms in the English NHS delivered the promised ‘coherent and mutually supporting set of reforms’. This concern reflects a recent interest in ‘realist’ evaluation of health policy that is based on different ‘schools’ such as ‘realistic’ evaluation7,8 and ‘theories of change’.9,10 We aim to identify the underlying ‘programme theory’ and context-mechanism-outcome (CMO) configurations of the Health System Reforms.
The various realist approaches start from the viewpoint that while experimental logic can claim the programme a ‘success’, we actually learn nothing about why it works. In paying attention to know how and why a programme works, evaluators of programme theory concern themselves with questions about the internal structure of programmes. They argue that social programmes are based upon explicit or implicit theories about how and why the programme will work. The task of evaluation is to surface those theories, identify the key assumptions and test their validity. 11
In a report of 75 pages Coleman et al. 8 use programme theory to examine practice-based commissioning (PBC). However, PBC constitutes one policy within one of the four streams of the Health System Reforms. Consequently, due to space constraints, we identify rather than test programme theory. 12 Similar to Pawson and Tilley 11 who present ‘a history of evaluation in 28 1/2 pages’, we draw on documentary analysis and interviews with key policy-makers to present a programme theory of the Health System Reforms in nine pages.
The Health System Reforms
Health reform in England: update and next steps 5 describes how the reform programme was intended to work together. This system reform programme (Figure 1) represents four related streams (demand, supply, transactional and system management reforms) which are described as ‘a coherent and mutually supporting set of reforms, which together provide systems and incentives to drive improvements in health and health services, increase responsiveness to patients and help to achieve reductions in health inequalities'’.
The Health System Reforms aimed to achieve ‘the right balance of incentives, patient choice, plurality and transparency in the system’. In doing so it intended to make the ‘patient's voice and choice heard' and give clinicians and managers the ‘tools’ to make good use of resources and to respond flexibly to their patients and population. 5
We aim to critically examine this balance in two main ways. First, what was the ‘fit’ of the streams? Were the reforms a ‘post hoc rationalization’? 6 Were they unbalanced in the sense of one stream dominating the others? Second, what was the ‘fit’ over time? It is possible to map the four reform streams presented in Figure 1 onto earlier and later documents. We will also examine whether there were unintended consequences, lessons learned and policy adaption over time.
Framework for health reform in England
Programme theory and CMO configurations
Pawson and Tilley 11 set out some basic guidance about programme theory and CMO configurations. Theories must be framed in terms of propositions about how mechanisms (M) are fired in contexts (C) to produce outcomes (O):what might work for whom in what circumstances. In other words, the relationship between causal mechanisms and their effects is contingent rather than fixed. They present an example of realism in the car park with eight mechanisms and six contexts. They stress that there is nothing about CCTV in car parks which intrinsically inhibits car crime. CCTV must work by instigating a chain of reasoning and reaction. Put another way, we need a theory, which relates to human choices and capabilities, of why CCTV may be effective. They later argue that the policy-maker's account has a rather specific significance as a source of testable theory, which takes the form of CMO configurations.
Methods
Our aim is to surface and refine the programme theory of Health System Reforms through a documentary analysis and semi-structured interviews with policy-makers. 12 Documentary analysis examined the policy assumptions contained in the Health System Reforms programme. This carried out a micro level textual analysis that aimed to surface the implicit and explicit programme theories about how the reforms intended to achieve its goals and outcomes. Detailed analysis of the Health reform in England document 5 examined the way in which the reforms were pulled together. Essentially, this looked to trace the reform journey to highlight the causal assumptions underpinning the Health System Reforms. In particular, the analysis aimed to tease the connections or linkages between the reforms and discuss how the reform streams fit together within the overall programme theory. This analysis culminated into causal maps as presented in Figures 2 and 3.
Mapping the programme theory of Health System Reforms
The patient benefits associated with Health System Reforms
In support of the documentary analysis in understanding programme theory we carried out interviews with policy-makers involved in the formulation of the Health System Reforms programme. These interviews with eight policy-makers aimed to further understand the programme theory in policy documents. These interviewees were selected and recruited as individuals having a direct involvement in the policy-making process during the period from the NHS Plan 13 onwards. They held different positions and were in post for different periods of time but had all contributed to the Health System Reforms programme. The interviews took place in between May and September 2009. Within each interview we asked for their particular perspective on the formulation and interaction of the reform streams. Interviews asked for their reflections on the process, the intended and unintended outcomes of the policy and the contextual drivers that led to the arrival of the Health System Reforms diagram. Data analysis of the interviews looked to gather and present evidence on whether policy-maker perspectives supported the Health System Reforms programme theory.
Results
This section presents the programme theories and CMO configurations underpinning each of the four reform streams, and then moves to examine the interactions between the streams. The mechanisms are the policies within the four streams. There is little explicit discussion of context, which suggests that it is assumed that mechanisms will work for all in different situations. In other words, it seems to be assumed that mechanisms are universalistic rather than contingent. The Health reform in England document points out that most work to date on the reform programme has addressed the acute and hospital care sector, but the introduction of new incentives will enable health care professionals and NHS managers to better respond to the needs and preferences of their patients. It continues that in early 2006, a White Paper (following the Your Health, Your Care, Your Say consultation) will set out both a strategic vision for health and health care services in the community. These new incentives and flexibilities will be used to support ‘better quality and more responsive and patient-focused services in primary and community settings’. Similarly, the Health System Reforms programme will also act as a key driver to deliver the vision for the future of community services. There is no explicit discussion about whether the reforms are expected to operate in different settings such as urban and rural areas, where levels of choice and competition can vary significantly. 14
It is difficult to pin down the aims or stated outcomes of the reforms. The Health System Reforms diagram (Figure 1) lists the rather vague outcomes of ‘better care; better patient experience; and better value for money’. Later, it is claimed that the programme will ‘drive improvements in health and health services, increase responsiveness to patients and help to achieve reductions in health inequalities'’. Moreover, the intended outcomes are ‘better-quality patient services and improved value for taxpayers’ money. In Appendix A, aims are laid out for the individual streams: to create more knowledgeable, assertive and influential users of services (demand); to create more flexible, responsive and innovative service providers (supply); to ensure that the impact of patients' choice is understood, and that good provider response is rewarded (transactional); and to ensure safety and to safeguard core standards in all services and to provide a transparent, rules-based framework for key management and decisionmaking functions in a more dynamic system (system management and regulation).
Demand side reform
Demand side reform is composed of choice, voice and commissioning elements It is claimed that achieving the outcomes of better care, better patient experience and better value for money would be achieved through levers and incentives that ‘create more knowledgeable, assertive and influential users of services’ (p. 3). In other words, more information will contribute to more empowered patients who will drive service improvement through both choice and voice. The creation of a commissioning framework that included PBC will provide the necessary ‘tools’ to innovate, and influence local service development . The cumulative effect of commissioning reform will increase the focus on moving to care into primary and community settings. PBC, in particular, would provide a lever in shifting health care spending towards preventative services delivering better health outcomes. Despite repeated emphases on innovation and responsiveness, the precise causal links are never spelled out. While it is clear that providers must be responsive to customers in a competitive market, it is unclear whether the ‘customers’ are largely PCT or individual ‘patient-choosers’, nor is it clear on what basis customers make their choices. Moreover, the claim that PBC would be a major driver for preventive services and community settings appears rather optimistic.
There was recognition among our policy interviewees that the choice policy had developed over time from a means to tackle waiting times to something that was more powerful as a driver for competition. Some informants stressed that not all consumers had to choose in order for the policy to work. In short, contestability or the perceived threat of competition may be more important than the level of actual competition.
We never needed much by the way of patient choice to create these kind of incentive effects, we didn't need every patient going through some massive rational cost benefit analysis. We needed the threat of choice (P1).
You didn't need that many patients to actually switch as long as it was a credible enough threat that would produce some kind of behavioural effect on the part of the hospitals and surgeons (P4).
Concerns about applying an ‘urban’ solution to ‘rural’ areas were not given great weight.
There are concerns about rural areas, but only about 10% of Britain is rural, so actually there was always this great danger of letting the rural tail wagging the urban dog (P1).
There is a view that we live on a small island with densely populated areas and therefore choice and competition could become a reality for 80% of the population. The fact that you couldn't make it work in Cumbria or rural East Anglia or Cornwall was not an argument for failing to apply those principles for the majority of the population (P5).
In relation to commissioning reform, it was broadly agreed that commissioning had developed more slowly and was weaker than the supply side. There were some concerns over the capacity of PCTs as commissioners, partly due to reasons of structure, personnel and potential destabilization. In general there was a sense of disappointment and by 2003 a realization that changes were needed, with the introduction of PBC in 2005. On the one hand, PBC was seen as a form of continuation of GP fundholding and as a method of demand management. However, PBC was also perceived as a weak link in contributing to the reform programme.
I found a weak spot in the set-up was commissioning… I think the people that introduced them hoped that PCTs were going to have the best of health authorities and the best of GP fundholding, you know, they were in a sense GP-led health authorities. Actually I think they had the worst of both. They weren't big enough to really deal properly with the big trusts, the big hospital trusts, and they weren't small enough to play the market in the way the GP fundholders had, and they didn't really mobilize the GPs again in the way the GP fundholders had, they became another bureaucracy and so on and so on. So I was very keen to try and reintroduce GP fundholding, and indeed did, sort of, with PBC (P1).
Supply side reform
The introduction of supply side reform aimed to create more ‘flexible, responsive and innovative service providers’ in order to achieve the outcomes of better care, better patient experience and better value for money. Of particular note, the freedom and flexibilities of foundation trust (FT) status gives frontline healthcare professionals and local managers the incentive to improve services and innovate in response to the needs of their patients and local populations. The diversification of providers with the introduction of private and third sector providers would generate more capacity, innovation and new ways of working. Within this, the development of a wider range of primary and community services will provide ‘new and innovative models of care’. Alongside ‘diverse providers’, workforce reform aimed to introduce more flexible working practices leading to changes in practice and culture (p.10). The assumptions here appear to be private and third sector providers are more innovative, and can force their NHS competitors to innovate. Moreover, it appears to be assumed that greater freedoms will automatically lead to greater innovation.
The need for greater plurality of providers was seen an important component of the reforms. One reason was to increase capacity to reduce public and private waiting lists but also through FTs to give successful NHS providers greater autonomy. However, there were some different views on FTs from our interviewees:
The freeing up of FT from the dead hand of the state was a perfectly rational thing to do. The supply side reforms needed more diversity within the NHS (P8).
I got very hooked on constructing an organizational form for public services which could not be bought by the private sector and which would not distribute shareholder profit (P2).
We needed autonomous public providers, but the worry I had about FT is that they would be accountable in too many different directions (P1).
The introduction of independent sector treatment centres (ISTCs) through a first wave of centrally commissioned contracts was seen as a means of increasing capacity and challenging the NHS. Despite this, the initial policy ambitions were scaled back before the policy had got off the ground.
I would like to have seen more use of the independent sector [problems over the ISTC programme] (P1).
Transactional reform
The goal of the transactional reform stream is essentially to link the supply and demand reforms; to ‘ensure that the impact of patient choice is understood and good provider response is rewarded’. Combined with patient choice and better information for patients, payment by results (PbR) will bring greater financial transparency to the system. It will benefit ‘respected and productive services’, and provide incentives to improve for those ‘providers struggling on quality or cost control’. PbR also provides incentives for the development of alternative primary and community services where these are more clinically effective and cost effective than hospitalization. In addition, more and better clinical and management information will enable providers to understand and respond to patients' needs. Practices and PCTs will use such information to drive commissioning decisions to align services with population needs. Better information will also help practices, PCTs and regulators to track quality and performance (p. 11).
PbR replaced block contracts with payment for activity. It was seen as a key driver of competition for creating incentives to reduce waiting lists:
Money following the patient as the vehicle for driving competition (P3).
However, it was recognized that tariffs needed to be refined to ensure that they did not simply lead to greater secondary activity:
The decision to have fixed prices was in part recognition of weak commissioners and was expected to drive technical efficiency within the acute sector. However the original tariff was a beta release 0.1. Patricia Hewitt in a speech to the NHS Confederation in about June 2005 said that the tariff needed to be reformed. She talked about creating a tariff based on best practice and a tariff that incentivised care outside of hospitals and so even at that point we were clear the tariff was not fit for the long term purpose (P8).
System management reform
The elements of system management reform (SMR) are given as governance; standards; licensing providers; competition policy; performance regime; and setting prices. SMR aims to create a framework of levers and incentives to ensure safety, standards and transparency. Here, the Health reform in England document suggests performance management and regulatory functions would achieve this goal through the creation of ‘standards and monitoring compliance’ (p.11). Processes for ensuring quality, licensing providers and price setting created a framework that promoted competition but protected essential services. The goal of system management would also be achieved with a wider view of regulation. Connections are made here with transactional reform as the new regulatory system combined with NHS tariff to create incentives for improving services, health care outcomes and increasing productivity.
Among those we spoke to there was little recognition that the system of targets and performance management continued to co-exist alongside the market reforms or about the role of SHAs in wider system management. The main focus was on the creation of independent regulators to assure patients of quality and safety and to have greater transparency about the quality of care.
Patients should not be choosing between good and bad services, but should be safe and appropriate …. In the same ways that when you buy house you should be able to take it as a given that it is not going to fall on you (P8).
Interactions between the streams
The Health reform in England document stresses the importance of interactions between the streams at many points. For example, this gradual introduction of incentives and flexibilities within a context of ‘system rules’ will build the momentum needed to deliver more benefits for patients and taxpayers. Better and more responsive services will be driven by the combined effect of more information on quality, patients exercising choice on the basis of such information and advice, and money following the patient. The combination of PBC and better commissioning by practices and PCTs will make it easier to invest in local services. A clear rules-based system will provide confidence for patients, and will encourage innovation as providers understand the rewards and risks (p. 12). Figure 2 presents the results of analysis of Chapter 3 of the Health System Reforms document that maps the causal assumptions underpinning the Health System Reforms approach.
However, programme theory is often far from clear. As noted above, the broad high level outcomes are listed, but it is difficult to determine the precise causal linkages. Moreover, analysis of the ‘Benefits for Patients’ appears to give a rather different picture (Figure 3). If we link the goals with mechanisms, we find that mechanisms appear to be heavily skewed towards the demand side.
Above all, incentives appear to be the key mechanism of the reforms, but ‘choice’ seems to be more important than ‘voice’. A very crude content analysis of the document gives the following counts: choice (52); information (45); incentives (26); and voice (8). All of the reforms are designed, in different ways, to introduce more incentives for hospitals and commissioners to fulfil patients' expectations and, crucially, more incentives for those who are not meeting patients' expectations to improve (p. 13).
The ‘programme theory’ of policy-makers suggested that the Health System Reforms programme was an attempt to make sense of the various elements of the reform agenda and how they fitted together. Crucially this was in part to satisfy a new ministerial team as there ‘wasn't a good understanding of how things were meant to fit together' (P7). Clarity and coherence for the new Secretary of State (Patricia Hewitt) was needed.
With the new Secretary of State, the department was struggling to create a narrative to explain what the health reforms were set out to do (P8).
There was a ‘new Secretary of State and Minister of State … asking how does all this work and why and nobody was really able to explain it'’ (P3).
One informant was able to articulate the link between the specific reform mechanisms and the outcomes though this was seen to be problematic:
Choice was the primary driver for a better patient experience, regulation as a primary driver for better quality; tariff as a primary driver for better value for money. So they operated an interlinked set but some of them were more important in certain aspects (P8).
There was a sense that the means had sometimes been elevated above the ends (P3). While ends were drawn from what would have been the department's key outcomes from time immemorial (P7), it was generally considered that the main target was the 18 week wait.
Reducing waiting times was THE focus, that was the overriding policy objective (P5).
Some interviewees expressed the view that the ‘fit’ between the streams was not fully clear, and that they were unbalanced to some extent.
I still think that the system architecture is correct. I just think that we're using Windows 3 when everybody else has moved onto Vista (P8).
Some predictable but unwanted outcomes were also identified, including the expenditure growth associated with the strength of providers and the incentives from PbR.
Powerful providers who were getting their income from fee for service and so had a very strong incentive to try and hoover up as much activity as possible. Unless you had a strong instrument for managing demand you got a classic recipe for, in a few years time, massive cost inflation (P1).
Proper checks and balances were not as strong as they need to be. For example, PbR gives incentives on providers to get more activity through (P7).
It was recognized that PbR acted as a barrier to moving care outside the hospital. the tariff was a blockage to moving money outside of hospital (P8).
Moreover, some views were expressed about the validity of policies beyond elective conditions.
[I]t was designed really around elective surgery I think; designed really for the hospital services surgery… . I don't think that we have fully completed the jigsaw of what the right blend of policy mechanism has been for different pathways. We … need to think about how you get the right mix of mechanisms for different pathways in different places (P3).
When asked about the interactions between the levers, informants recognized that there were tensions and that some areas of reform had been stronger and been given greater attention.
There is a real risk in uneven or very differently paced developments. My perception was that supply side reforms had been more advanced and were picked up more quickly that the demand side, which would have been reversed in an ideal world (P7).
The system was originally unbalanced when it was designed but it did not matter as the first generation of reforms were created in the context of long waiting lists. As it became successful it made itself out of date. To use the interviewer's weights example: ‘the ballast was right but as you were unloading the cargo you needed to change the ballast’ (P8).
We have to ask it is the package of reforms that are fit for purpose rather than individual items. For example [name] has spoken many times that the failures of FT may be the failures of commissioning rather than the failures of FT perse(P5).
On balance looking across the reforms the informants we spoke to felt that the supply side reforms had been pursued at the expense of the demand side reforms and that in terms of order, more attention should have been paid to commissioning in the early phases of reform.
[I]f I knew where they were going to end up I wouldn't have started by doing FT. So quite a lot of phase 2 reforms were designed to catch up on the commissioning side (P8).
One of the well rehearsed criticisms is that one should have sorted out commissioning before doing any of the supply side stuff. In the abstract one can see the attractions of that argument. In practice there are a series of political problematics… and the first was around waiting times (P4).
Discussion
A number of points emerge from the analysis of documents and interviews. First, programme theory was not fully clear. While the programme appeared to fit together at the macro level, this was less obvious for micro level linkages. In particular, there was a tendency to claim that individual streams led to global outcomes rather than considering how different streams might lead to conflicting outcomes. For example, unreformed PbR might lead to greater secondary activity and undermine ‘care closer to home’. Some dogs did not bark. For example, there was little discussion of voice or workforce reform, and little consideration of how the reforms might lead to the reduction of health inequalities. While the comments of our interviewees were more nuanced and filled in some of the gaps in the document, this did not make programme theory fully clear. In short, there is little explanation of ‘fit’ beyond the diagram. It remains unclear why there are four streams rather than, say, three or five. There are few clues as to how the mechanisms fit with each other or with the outcomes.
Second, there was generally little recognition of context in the document, although more recognition among our interviewees. However, in there was a broad assumption of universalistic rather than contingent mechanisms. For example, there was little recognition of the differences between acute and community care settings. In other words, it was assumed that ‘one size fits all’ with little recognition that some contexts would be more receptive than others. 15
Third, there was some policy evolution, and rebalancing between the streams. Some later policy levers were added or changed as problems emerged. For example, it became clear that PbR was more accurately viewed as ‘payment by activity’. This led to discussions of ‘best practice’ and ‘unbundled’ tariffs that linked payment partly to quality, patient experience and patient satisfaction rather than simply to quantity.16,17 Policy-makers were also aware of other perverse incentives and unforeseen consequences. They attempted to pull the appropriate policy levers with some later elements aiming to restore balance to system, for example, PBC to balance PbR, and World Class Commissioning to challenge the dominance of FTs.
Conclusion
The main conclusion of this paper is that it is difficult to develop ‘programme theory’ for the Health System Reforms beyond the fairly general level of the diagram. It is particularly difficult to detect a CMO configuration at the micro level. We have found the linkages between M and O unclear with little explicit recognition of C. The reforms do not appear to comprise a coherent and mutually supportive set of arrangements, and appear ‘unbalanced’ in that the ‘centre of gravity’ favours suppliers over commissioners. However, recent reform changes have sought to redress this imbalance to some extent, suggesting that lessons have been learned and policies have been adapted over time. It remains unclear whether the 2010 Conservative/Liberal Democrat coalition government plan for the NHS 17 represents evolution or revolution in health care reform. Like the Health System Reforms, it is also stated that the reform proposals ‘are interconnected and mutually reinforcing’, and it claims that ‘levers’ and ‘incentives’ will lead to certain outcomes e.g. a new system of economic regulation will promote financial discipline, efficiency, transparency and fairness in the way that resources are used (para 7.28). Moreover, in the language of programme theory, we have seen some changes in context, mechanism and (intended) outcomes. For example, strengthening the local authority role as integrator of commissioning across the NHS, public health and social care will deliver more integrated care, improve efficiency and offer better user experience. The development of economic regulation will strengthen the drive for provider efficiency through greater price transparency, increased competition, and a clear and independent provider failure regime.
The implications of our findings suggest that drawing attention to such programme theory configurations is important in three ways. First, simply linking all of the reform streams to global outcomes, such as improved health, is insufficient. Greater and more specific CMOs for individual policies and streams are required. Second, more attention needs to be paid to the interactions between the various reform policies as some tensions were clearly evident. Third, greater consideration of context is required. What works for urban areas and elective conditions may not work for rural areas and long-term conditions. Reformers need to understand that healthcare systems are like complex and fragile ecosystems: changes to one area may trigger changes in other areas. By paying attention to the programme theory of health care reform, we hope to shed light of these implicit and explicit assumptions. In doing so, we provide a map from which to navigate through the complexity of reform policy, illuminating casual links and intended and unintended consequences.
Footnotes
Acknowledgements
This paper draws on independent research commissioned and funded by the Policy Research Programme in the Department of Health. The views expressed are not necessarily those of the Department.
Conflicts of interests: None declared.