Abstract
Introduction
Improvement of health service performance is a central concern in all countries – what is the evidence on how costs can be reduced and quality can be enhanced? A major challenge in the context of healthcare is how best to define and measure quality, the subject of a substantial body of health services research. This theme begins with research that sought to incorporate quality measures in estimates of productivity and to investigate variation in performance to highlight the potential for improvement. An important advance in our ability to measure quality, from the perspective of recipients of health care, was the development and roll-out of Patient Reported Outcome Measures, the subject of our second case study. Having better measures of quality in turn allows research to identify effective mechanisms for achieving performance improvement in specific contexts and circumstances, the topic of the final two case studies. Research demonstrates the potential for competition in the hospital sector to drive quality improvement, but only when operating in the context of centrally determined fixed prices; otherwise, there is a risk that quality will deteriorate. Evidence from a natural experiment shows that there is a role for performance targets in producing performance gains in healthcare systems, but only where the targeted outcome can be measured directly and accurately; if it is not, the system may be open to manipulation, potentially reducing patient benefits.
Measuring health service productivity
Summary
Research on the measurement of overall National Health Service (NHS) productivity at the national, hospital and individual consultant level, has had an impact on policy and practice. Methods have been developed for measuring NHS productivity, which changed the way that the Office of National Statistics values the NHS in national accounts. The methods, which take into account improvements in the quality of care, have been incorporated into submissions to the Comprehensive Spending Reviews that determine the NHS budget and are internationally influential. Research on productivity at hospital level has influenced the tariffs set for reimbursement of specialist hospital care. Research on the productivity of hospital consultants influenced the reviews of doctors' pay and rewards by the Doctors' and Dentists' Pay Review Body and formed the basis of benchmarking tools distributed for use in the NHS.
Research and impact
Productivity is a key indicator of efficiency and competitiveness. Measures of public service output and productivity are important elements of accountability to the public for how the £106 billion annual NHS budget is spent.
At a national level, novel and improved methods to measure the productivity of the NHS in England were developed and have been continuously refined and improved. Productivity estimates were derived from secondary analysis of routinely collected NHS data. They represented an advance on standard practice in three important ways. 1 First, the index of outputs was comprehensive, capturing activities in 5,381 healthcare categories for all NHS patients treated by either NHS or non-NHS providers, incorporating information about every patient treated across acute and community settings. Second, the quality of output was assessed by including indicators such as health outcomes, patient satisfaction, waiting times and readmissions. Third, better measures of NHS input, particularly labour and capital, were compiled. Estimates of NHS input, output and productivity growth are produced annually for the Department of Health (DoH), 2 incorporating updated methods and new data each year. They include a range of sensitivity analyses to ensure robust estimates. Methods to analyse regional productivity have also been developed, with estimates showing significant variation across the country, which indicate potential savings of £3.2bn. 3
By developing improved methods of measuring NHS productivity and providing methodologically robust evidence on annual NHS productivity growth, research has influenced the processes by which the size of the NHS budget is determined. The 2002 Spending Review committed the Government to a new Public Service Agreement target of 2% improvements in productivity per year. The DoH turned to researchers to find a way to incorporate quality of care improvements into the measurement of the output of the NHS, which had not previously been attempted. The methods developed were adopted as national policy and have been used continuously. These methods are incorporated into the Office for National Statistics (ONS) estimates of NHS productivity, which feed into the annual UK National Accounts. 4 Every year since, the ONS has used the methods for quality adjustment as a key element in their triangulation of evidence about health service productivity, which has added an average of 0.5 percentage points to estimates of annual output growth. 5 Estimates of input, output and productivity growth are produced regularly for the DoH for use in their calculations of NHS productivity and a tool has been produced for DoH analysts to use in order to explore in detail the impact of different assumptions on future productivity estimates. The estimates provided through the research are important for negotiations with ministers and with agencies that have the power to influence the size of the NHS budget. The research has been used directly to provide numerical answers and context for Health Select Committees, Public Accounts Committees and a Public Expenditure Inquiry.6–8 Researchers have provided workshops on methods of productivity measurement to policy makers, statisticians and academics nationally and internationally, and have been influential in Italy, Sweden and Japan.
At the hospital level, research on the costs of specialist hospital care has applied advanced econometric methods to hospital data for over 26 million patients. This isolated the extra costs associated with the provision of specialist care, accounting for a range of other factors that may also influence hospital costs. Results showed that higher costs were legitimately associated with the provision of specialised care only for a small number of conditions and groups, including cancer, cystic fibrosis and children's care. 9 The research was commissioned as part of ‘a fundamental review of the current methodology’ used to calculate specialised service top-ups for the Payment by Results tariff and resulted in changes in policy in 2011/2012, including the introduction of new top-ups for neurosciences and spinal surgery and revision of the level of existing top-ups for children's services and orthopaedics. 10
Research has also focused at measuring productivity at the level of individual hospital consultants. 11 Linking NHS data with information about consultants from the Medical Workforce Census for the first time, datasets of inpatient activity were derived for all consultants in 10 specialties in England. Using multi-level modelling, factors were identified that predict consultant productivity. Consultants with a ‘maximum part-time contract’ (permitting substantial private practice) were found to treat more NHS patients on average than their full-time NHS colleagues, as did those with clinical excellence awards (bonus payments). 12 This finding was reinforced in a later study, which included exploration of associations between clinical excellence awards and consultant productivity. 13 Finally, using interrupted time series, it was shown that reform of the consultant contract in 2003 failed to improve consultant productivity, and indeed, in some specialties, such as trauma and orthopaedic surgery, the effect was negative. 14
Staff costs are the largest single component of NHS expenditure and research into productivity at consultant level has had significant policy impacts. First, the method of describing and exploring variation in consultant productivity was adopted as a benchmarking tool by the DoH's Workforce Directorate and the NHS Institute of Innovation and Improvement as part of their ‘Delivering Quality and Value’ programme, which in 2008 distributed comparative data on consultant clinical activity to all hospital Trusts in England. 15 Second, this research influenced the review of clinical excellence awards by the Doctors’ and Dentists’ Review Body (DDRB). In evidence to the Review Body, researchers recommended ‘introducing an extended career structure for doctors, with earned increments and a senior consultant grade’ 16 and this recommendation was adopted by the DDRB in their report, 17 which recommended a ‘principal consultant grade’ as part of contract negotiations with the British Medical Association.
Acknowledgements
Institutions listed as contributors to research: University of York.
References
Research Impact Case Studies
Introducing patient reported outcome measures into the National Health Service
Summary
Research to develop and test patient reported outcome measures (PROMs), which measure health outcomes from the patient perspective, has demonstrated the feasibility of routinely collecting such measures before and after elective operations. In 2009, the Labour government mandated that PROMs should be collected on all National Health Service (NHS) patients in England undergoing one of four surgical operations, a policy endorsed by the Coalition government following the 2010 election. This remains the only nationwide programme of its kind worldwide, providing essential data for comparing providers' performance, patient choice and other quality improvement approaches.
Research and impact
Patients' views are essential to achieving high quality care. Their perspective complements that of clinicians, providing unique insights into their perceptions of health status and health-related quality of life. It is therefore important to find ways of involving patients in reporting on their own health outcomes.
The goal of this programme of research since 1996 has been to create PROMs – measures of patients' health and health-related quality of life collected before and after surgery, involving patient interviews and the development and testing of questionnaires. The research fell into three phases: the development and psychometric testing of PROMs (1996–2004); methodological research to ensure accurate analysis and interpretation of these measures (1993–2012) and applied research into the feasibility of the routine use of PROMs to assess the quality of care of providers (since 2005).
In terms of developing and psychometrically testing patient questionnaires, major contributions were made to the development of new PROMs for a wide range of surgical procedures: stress incontinence (1996); benign prostatic hypertrophy (1998); menorrhagia (1998) 1 ; venous disorders (2003); plastic surgery on hands/arms and on head/neck procedures (2004) and coronary revascularisation (2004). 2 Work in the second phase was undertaken on a range of methodological aspects of the use of PROMs, including the influence of patients' preoperative expectations; the impact of late response and non-response to follow-up; 3 the use of minimally important differences 4 and many others.
The focus in the third phase was on the routine implementation of PROMs in the NHS. A systematic review of instruments for the routine assessment of outcomes following five common elective operations (hip and knee replacement, varicose vein surgery, hernia repair and cataract surgery) led directly to a study to develop pre- and postoperative questionnaires for four procedures and to test the feasibility of using them routinely in the NHS. This study confirmed that it was possible to recruit patients, follow them up and make risk-adjusted comparisons of providers, all at reasonable cost. To confirm these findings, a much larger study with about 35,000 patients was undertaken. Studies have included equity of use and outcomes, the impact of choice of metric 5 and clinicians' and patients' views of how best to feed back comparisons of health care providers' performance to maximise the likelihood that the data will stimulate improvements in the quality of care. 6
This research into PROMs has led directly to their introduction across the NHS in England for four elective surgical procedures. It is the first time that such measures have been introduced on a nationwide scale with the aim of comparing the performance of hospitals.
The route to achieving this impact started with the decision, based on the research report on PROMs, by the NHS Management Board that it would be mandatory from 2009 for all NHS patients undergoing the designated operations to be invited to complete pre- and postoperative PROMs questionnaires. 7 The Chair of the DH PROMs Stakeholder Reference Group recognised that the research was pivotal in giving the DH the confidence to start the National PROMs Programme and Lord Darzi, then Minister of Health, addressing the Health Select Committee, stated: ‘I think the best investment we have made was actually in this report’. 8 A government White Paper on the NHS provided a strong endorsement: ‘Just as important [as clinical measures] is the effectiveness of care from the patient's own perspective which will be measured through patient-reported outcome measures.’ 9
Despite the change of government in 2010, there has been seamless political support for the use of PROMs demonstrated by Ministers' positive views on PROMs 10 and by the inclusion of PROMs in the NHS Outcomes Framework. 11 The value of PROMs for more sophisticated estimates of NHS productivity has been recognised by the Office of National Statistics 12 and the National Audit Office. 13
Since April 2010, PROMs data have been published online (http://www.ic.nhs.uk/proms) for the use of clinicians, managers, commissioners and the public. The huge scale of the programme is indicated by the 515,000 patients who participated over the first 3 years (over 70% of those eligible). In 2012, the DH's Branch Head of Strategy, Finance and NHS Directorate, David Nuttall, commended the research after the introduction of PROMs as having ‘informed decisions on how data should best be analysed, presented and used’. Another indication of the impact of this research is that coronary revascularisation was included from 2014 using a PROM (CROQ) developed through research.
The research has raised awareness and understanding of PROMs among clinicians, NHS managers and patient organisations through many national and international conferences and researchers have advised the US National Institute of Medicine and the American Medical Association.
Acknowledgements
Institutions listed as contributors to research: London School of Hygiene and Tropical Medicine.
References
Research Impact Case Studies
Changing policy on competition in the UK health-care market to benefit patients and taxpayers
Summary
National and international policy on the use of competition in health care since 2006 has been strongly influenced by research evidence on how competition affects patient care. The research has underpinned a series of political decisions to use competition and choice in the National Health Service (NHS) and informed the design of regulatory structures to support these policies. Specific decisions influenced by the research relate to the structure of prices, policy on mergers and policies to promote greater use of choice and competition to benefit patients and taxpayers.
Research and impact
Research investigated the impact of market structure on the quality of hospital care by examining the NHS internal market reforms, which were introduced in 1991 and abolished in 1997.1,2 In these reforms, hospitals competed on both price and volume. Results showed that such competition led to a decline in quality (indicated by higher death rates in competitive markets) and suggested that these reforms cost 1030 lives at a cost of around £355.35 million. Subsequent research examined the Labour government's ‘pro-choice’ reforms of 2006, which re-established competition between hospitals, but under a system of regulated prices. Research suggested that the reforms saved approximately 4791 life years without increasing resources used, 3 implying that price regulation allowed competition to occur on quality. Further research examined whether the reforms increased the elasticity of demand with respect to quality, a prerequisite for pro-competitive reforms to have a positive impact on the quality produced by hospital suppliers. Results suggested that the elasticity of demand with respect to quality increased post-reform, particularly for sicker patients. 4 Investigation of the relationship between management, quality of care and competition suggested that one of the ways in which competition might bring about improvements in care was through better management. 5
Another strand of research exploited the large-scale closure of hospitals in England between 1997 and 2003 to examine whether mergers in hospitals had similar negative outcomes to those in the private sector. The research analysed a range of outcomes relevant to both patients and taxpayers, including death rates following emergency admissions, waiting times, expenditure, deficits and staffing levels. The research design compared merged hospitals with appropriate controls. It was found that mergers' promised gains were not realised and instead resulted in longer waiting times and larger deficits, with no evidence of improvement in clinical outcomes. 6
Government policies relating to decisions on the structure of the healthcare market, the degree of choice and competition allowed and the regulations governing the market, directly affect patients through the quality of care delivered, and taxpayers, through the costs associated with implementing the policies. The research has had a major impact on decisions made throughout the process of developing a structure of market regulation. It formed part of the underpinning for the overall (highly controversial) policy on the use of choice and competition contained in the Health and Social Care Bill of 2012. 7 The research has also been used to decide who should be allowed to compete with NHS providers: 3 it was used as evidence in the review of the operation of Any Willing Provider for the Provision of Routine Elective Care, with the Department of Health accepting the key recommendations. 7 Research has also been used to demonstrate that there is an incentive for providers to improve quality. 4 Impact was achieved through the provision of extensive presentations and individual briefings to policymakers and politicians, including the Secretary of State for Health and the Number 10 policy advisor on health, as well as presentations to the Department of Health, the Prime Minister's Strategy Unit, the Treasury and the Cabinet Office.
The body of research relating to the role of prices in the NHS played a role in the Coalition government's decisions to retreat from initial plans to allow hospitals to negotiate prices freely; instead choosing to set prices centrally and allow competition only on quality. The research was used as evidence in internal presentations, alongside similar papers, to set out the empirical support for the theory that non-price competition can lead to better quality outcomes for patients. 7 Reports by the health care regulator, Monitor, regularly acknowledge the research 8 and media reports credit the research as leading to the policy reversal. 9
Decisions on hospital mergers have drawn on the research3,6 and it has been used as evidence in eight advisory reports of the Co-operation and Competition Panel (CCP) to the Secretary of State for Health and Monitor on whether proposed hospital mergers were consistent with quality. 7 The research has also been used in evidence in internal presentations to provide a general context on the tendency of hospital mergers to fail to achieve efficiencies or to improve quality for patients.7,10
Acknowledgements
Institutions listed as contributors to research: University of Bristol in collaboration with Imperial College, London School of Economics and Political Science, Carnegie Mellon University and Stanford University.
References
Research Impact Case Studies
Using targets and incentives to improve the quality of public services
Summary
From 2000, the four devolved governments of the United Kingdom allocated unprecedented increases in health spending, each set similar goals for improved performance, and yet performance was transformed only in the National Health Service (NHS) in England. By utilising this natural experiment, research explained why this happened and changed the understanding by key policy actors about the use of targets to achieve performance goals. The evidence base it created also influenced the governance of health services, notably a general shift to the use of targets for all the UK NHSs.
Research and impact
In the late 1990s, across the four UK healthcare systems, there was a crisis of poor quality indicated by long hospital waiting times. From 2000, the government sought to transform the quality of the NHS in England through two policies. The first policy in England was sustained and generous increases in NHS funding each year. This policy was followed by the other governments of the devolved countries. The second policy was to abandon the old model of governance, which rewarded failure to deliver targets by NHS organisations with extra resource. In its place a new model was proposed, which set ambitious targets with sanctions for failure; this policy was implemented in England only. The sanctions that applied from 2000 to 2005 in the regime of annual ‘star ratings’ were ‘naming and shaming’ and ‘targets and terror’ (i.e. Chief Executives were at risk of being dismissed). This second policy was not followed by the non-English governments, which continued with the old model of governance. These differences in governance between the systems in England and the devolved countries formed the basis of the ‘natural experiment’.
The ‘natural experiment’ was evaluated in a number of research studies: at a national level, across the four NHSs for hospitals and ambulance services;1–3 in comparisons of England and Wales for ambulance services, 4 and hospital waiting times, 5 and the effects of the presence or absence of ‘naming and shaming’. 6 Three principal findings emerged. First, the regime of ‘star ratings’ transformed reported performance of the NHS in England. Second, outcomes in England were not matched by the performance of health services in the devolved countries. Third, there was gaming of the target system in the English NHS.
These findings suggested two strategic lessons for policy design. Policies that link high-powered incentives to targets can work for public services where targets accurately measure key dimensions of performance and gaming is of secondary importance (e.g. hospital waiting times, ambulance response times and school league tables). However, where the targets can only be proxies (and hence not an accurate measure) for important dimensions of performance, the dysfunctional effects of gaming may make their use problematic (e.g. in the police services).
Governments seek to improve public services and like to claim that their policies are evidence-based. But, the nature of policy making means that there is typically little good experimental evidence, hence the importance of this research across the four healthcare systems of the United Kingdom, which were so similar except for different models of governance.
The research had impacts through public engagement, influence over the terms of the debate about targets, and downstream policy changes, especially in devolved governments. It featured in the media and policy debates, including a House of Lords/Parliamentary Seminar on Benchmarking Public Services for Excellent Performance. Internationally, it was cited by WHO's report on assessment of the performance of health systems 7 and the research papers appear on the World Bank website following presentation at seminars on performance measurement in developing countries.
The research has influenced the terms of debate, being covered extensively in the media and by Scottish Parliament and being cited as evidence of the effectiveness of governance by targets in commentary on the shift to a quasi-market model for the English NHS by the then Secretary of State for Health. 8 The Nuffield Trust Report on public reporting of performance, written for the Secretary of State for Health in England, extensively cited the research into ‘star ratings’. 9 It has been cited by a member of the police force 10 as helping to identify scope for gaming in response to targets and how the inflexible nature of targets in policing led to such dysfunctional consequences that they should not be used in attempts to improve performance of the police.
The research is also strongly associated with policy changes in the health services of the four devolved nations through its contribution to an evidence base about targets, which the devolved governments could not ignore. An influential report published by the Kings Fund 8 provides evidence of the impact of this research, stating ‘In terms of day-to-day management of the NHS… one of the most obvious examples of lessons learnt is over the use of targets’. In particular, it was noted that the research on the ‘targets and terror’ approach provided very strong evidence on the effect of targets on waiting times and, as a result, the target-driven approach implemented in England was eventually adopted by Wales and Scotland. The coalition government in England, having first given the impression that it was abandoning waiting time targets, also then reaffirmed their use. 8
Improved governance leads to improved performance of public services in terms of the objectives that matter to the public and hence to governments, such as reducing long hospital waiting times and good performance in GCSEs at age 16. The ‘natural experiment’ following devolution shows which types of governance are and are not effective in achieving highly salient measures of performance of public services. It also provided a model for other research enquiries into health services and school systems, and into emerging policies for policing.
Acknowledgements
Institutions listed as contributors to research: London School of Economics and Political Science.
References
This Measuring and improving performance themed article is based on the following case studies:
Measuring health service productivity. http://results.ref.ac.uk/DownloadFile/ImpactCaseStudy/pdf?caseStudyId=43411 Introducing patient reported outcome measures (PROMs) into the NHS. http://results.ref.ac.uk/DownloadFile/ImpactCaseStudy/pdf?caseStudyId=41461 Changing policy on competition in the UK health-care market to benefit patients and taxpayers. http://results.ref.ac.uk/DownloadFile/ImpactCaseStudy/pdf?caseStudyId=40285 Using targets and incentives to improve the quality of public services. http://results.ref.ac.uk/DownloadFile/ImpactCaseStudy/pdf?caseStudyId=35163