Health services research related to performance indicators and benchmarking in Europe

Abstract

Objective:

Measuring quality of care through performance indicators and subsequently using these to compare, learn, and improve (benchmarking) has become a central component of health care policy. This paper aims to identify the main themes of health services research in this area and focuses on opportunities for improving the evidence underpinning performance indicators.

Methods:

A literature survey was carried out to identify research activities and main research themes in Europe in the years 2000-09. Identified literature was categorized into sub-topics and for each topic the main methodological issues were identified and discussed. Experts validated the findings and explored the potential for related further European research.

Results:

The distribution of research on performance and benchmarking across EU member states varies in time, scope and settings with a large amount of studies focusing on hospitals. Eight specific fields of research were identified (research on concepts and performance frameworks; performance indicators and benchmarking using mortality data; performance indicators and benchmarking related to cancer care; performance indicators and benchmarking on care delivered in hospitals; patient safety indicators; performance indicators in primary care; patient experience; research on the practice of benchmarking and performance improvement). Expert discussions confirmed that research on performance indicators and benchmarking should focus on the development of indicators, as well as their use. The research should involve the potential users and incorporate scientific approaches from biomedicine and epidemiology as well as the social sciences. Further progress is hampered by data availability. Issues which need to be addressed include the use of unique patient identifiers (UPIs) to facilitate linkages between separate databases; standardized measurement of the experiences of patients and others; and deepening collaboration between Eurostat, the World Health Organization (WHO), and the Organization for Economic Co-operation and Development (OECD) to facilitate the availability of internationally comparable performance information.

Conclusions:

This study suggests a number of themes for future research. These include testing and improving: the validity and reliability of performance indicators, especially related to avoidable mortality and other outcome indicators; the effectiveness and efficiency of embedding performance indicators in the various governance, monitoring and management models, and their effect on health systems, services and professionals; and the effectiveness and efficiency of linking performance indicators to other national and international strategies and policies such as accreditation and certification, practice guidelines, audits, quality systems, patient safety strategies, national standards on volume and/or quality, public reporting, pay-for-performance and patient/consumer involvement. The field would benefit from strengthening the clearinghouse function for research findings, training of researchers and appropriate scientific publication media. Results should be systematically shared with policy-makers and managers, and networking stimulated between the growing number of regional and national institutes involved in quality measurement and reporting.

Introduction

Benchmarking and performance indicators have become key challenges in health care policy in European countries now that accountability and transparency are central notions in current ideas about health system governance and stewardship.^1–3 Whether looking at health care from an economic perspective, a public health perspective, a medical perspective, or a legal perspective, the performance of health care services and systems needs to be made explicit and used as the basis for improvement strategies. This assumes that the performance of health care services (meso level) and health care systems (macro level) is measured with valid, reliable and relevant performance indicators (performance measurement),⁴ and that this information is interpreted correctly to assess the relative position of health services or health care systems (bench-marking),^5–7 resulting in appropriate decision-making to improve the outcome of health services and health systems. Measuring and managing performance is thus not only essential for policies that aim to improve the quality of health care (defined in this paper as effectiveness, safety, and patient-centredness), but is also a precondition for policies focusing on increasing efficiency and value creation.

The notion of stewardship and governance through performance indicators has been endorsed by ministers of health in the World Health Organization (WHO) Euroregion in the Tallinn Declaration in 2009, and more recently by ministers of health of member countries of the Organization for Economic Co-operation and Development (OECD).⁸ Both the development and use of indicators should be evidence-based, and thus call for health services research (HSR).

Health Services Research into European Policy and Practice (HSREPP) is an initiative aimed at identifying, evaluating and improving the contribution of health services research (HSR) to the health policy process in Europe which includes the aforementioned challenges of HSR related to benchmarking and performance indi-cators.⁹ This article aims to identify the main themes of HSR in Europe related performance indicators and benchmarking, and focuses on the opportunities for improving the HSR evidence to support policy developments in the field of performance indicators and benchmarking rather than giving a complete overview of all research in this field. Based on this information, priorities are identified and recommendations provided for setting the future European research agenda in this area.

Performance is restricted to quality, defined in terms of effectiveness, safety, and patient-centeredness. Performance domains such as costs, access, and equity are not addressed here. Focus is put on the underlying need for data and the quality of the related national information infrastructure.

Methods

Database search

We conducted an iterative search to identify the main themes and opportunities for improving the evidence related to performance indicators and benchmarking. First, we searched for literature using key authors in the field. These authors were identified by expert opinion as well as by the impact of their studies on European health care policy. More specific search terms were derived from the scientific publications of these authors, and were subsequently entered in a snowball search strategy using the PubMed search database at www.pubmed.org. Doing so, identified further search terms and relevant literature.

To be as thorough as possible within the chosen approach, we checked to see if a systematic search would add any significant literature that had not yet been identified. Therefore, search terms derived from the literature that had already been identified were used in the thesaurus of Medical Subject Headings (MeSH) Database in PubMed. Unfortunately, the relatively low number of articles that were identified suggests that the search terms we used such as ‘quality indicator’ were too specific for the field of HSR, which is relatively new and still developing. However, broadening the MeSH terms decreased specificity to such an extent that it was not feasible to analyse the data with our detailed approach. A detailed description of the search is provided in an appendix on benchmarking and performance indicators in HSREPP's final report.⁹

The time span of the search was from 1 January 2000 to 1 January 2010. References on research from outside Europe (identified by a manual scan) were removed. In a later step, articles were further analysed according to the main topic they addressed.

Project search

We conducted a search in the CORDIS (Community Research and Development Information Service) project database, the information service for both current and past EU Framework Programmes (http://cordis.europa.eu/search/index.cfm?fuseaction=proj.advSearch). The search ranged from January 2000 to January 2010. The search terms ‘benchmarking’, ‘quality indicator’ and ‘research’ were used in different combinations.

The database of the EAHC (Executive Agency for Health and Consumers, http://ec.europa.eu/eahc/projects/database.html) was searched over the same period. The combination of the search terms ‘quality indicator’ and ‘benchmarking’ was used. We used the internet search engine Google with the terms ‘quality indicator’ and ‘benchmarking’ and consulted websites of leading European organizations in the field in searching for projects.

All projects identified were then classified according to the topic areas of the sample of articles as described in the HSREPP final report.⁹

Expert meeting/conference discussion

At the working conference, Health Services Research in Europe, at the Hague in April 2010, participants discussed in a carousel format the categorization of findings into topics, and future directions of HSR related to performance indicators and benchmarking. In particular, the validity of performance indicators, their use and misuse and the requirements needed for national information infrastructures came up as points for discussion.

Results

Database and project search

A total of 1508 articles were identified and used to draw conclusions on overall research activities and to identify main research themes in the last 10 years in EU countries. Full details can be found in the appendix of the final report under the corresponding topic heading.⁹

Figure 1 presents the cumulative number of references between 2000 and 2009. The number of publications increased in this period by an average of 18% annually. The year 2010 was not taken into account in the calculations, because the literature search was conducted in January 2010.

Figure 1

Cumulative number of references, 2000–09

Identified articles were categorized into eight main topics. These are presented in Figure 2 as percentages of references per topic. The highest proportion of research activities was identified in the field of performance indicators and benchmarking on care delivered in hospitals (23%), followed by research in the field of primary care (16%). The lowest proportion of research activities was identified in the cluster of research on concepts and performance frameworks (3%).

Figure 2

Percentages of references per identified topic

Sub-topics were identified during a subsequent step of analysing the articles in certain categories (performance indicators and benchmarking related to mortality and cancer, performance indicators on care delivered in hospitals and patient safety indicators). Once again, the highest number of sub-topics was identified in the field of performance indicators and benchmarking on care delivered in hospitals (see Figure 2).

The following sections give more detailed descriptions of the eight identified fields of HSR related to performance indicators and benchmarking and provide some general methodological reflections identified in the articles. An overview of each field and its subtopics is presented in Figure 3.

Figure 3

Identified topics and sub-topics in health services research related to performance indicators and benchmarking

Research on concepts and performance frameworks

Given their dependency on context and goals, the work of performance measurement has become embedded in a growing body of research on concepts and frameworks used for performance measurement and benchmarking. One line of research links specific quality aspects to specific sets of indicators (e.g. research questions such as ‘what is a balanced set of indicators?’). Another line of research looks at how sets of indicators can be grouped into broader conceptual frameworks to maintain an overview of all aspects of performance, and be able to derive strategic meaning from developments in performance in various domains and aspects at the same time. Frameworks from industry (such as the Balanced Scorecard) are used as examples. These types of frameworks can be found for assessing the performance of entire health care systems and the quality of health care (e.g. by OECD or WHO) as well as for individual services such as hospitals (e.g. PATH [Performance Assessment Tool for Quality Improvement in Hospitals] designed by WHO¹⁰).

A specific area of research is the link between quality and costs. As two domains of health services and health system performance, they are usually considered separately. Although not the main focus of our study, we are aware that there are international standards for measuring costs, most notably those embodied in national accounting schemes and international efforts such as the system of health accounts (SHA) work of the OECD, the EU and WHO. Linking quality and costs as part of overall performance management seems less common in Europe, the only example found in the literature stems from Ontario, Canada.

Although quality and cost methods from industry have been applied,^11,12 in the reported studies it is observed that the external re-imbursement methods in health do not, as in industry, reward better quality (i.e. in health care there does not seem to be a business case for quality). Policy-makers have already recognized this, and performance indicators measuring quality are increasingly being used in ‘pay-for-performance’ schemes and selective contracting. Evaluation research on these policy interventions – although quite prevalent in North America – is still limited in Europe.

Performance indicators and benchmarking using mortality data

Historically, death statistics have been the basis for assessing the performance of health care systems. In many international reports, life expectancy and perinatal deaths are still used to compare the relative performance of health care systems (e.g. WHO and the World Bank^13,14), and several adjustments have been made to refine the measures (e.g. the use of ‘disability-adjusted life years’). With respect to performance indicators and benchmarking, two research lines are interesting to note. The first relates to ‘avoidable mortality’. After initial research by Holland et al. in the 1980s,^15,16,17 avoidable mortality has recently become a topic of interest again. Following research by Nolte^18,19 and McKee,^20,21,22 an EU-funded project (AMIEHS – Avoidable Mortality in the European Union – Towards better Indicators for the Effectiveness of Health Systems²³) is, at present, reviewing the causes of avoidable mortality and refining the measurement methods and international comparability of data. Several European countries are using this method for comparing performance within their own countries and the OECD is looking into reporting on avoidable mortality rates as part of its international comparative work.

A second line of mortality statistics-based research is the growing popularity of ‘hospital standardized mortality rates’. Initially developed by Jarman,^24,25,26 this method for comparing the performance of hospitals based on adjusted hospital mortality rates has been taken up by a number of countries. Methodological debates are focusing on validity and the international comparability of these data (or the lack thereof), given the differences in hospital systems and recording. Even so, there is substantial policy interest.

Opportunities to use mortality statistics as performance indicators and for benchmarking seem to be hampered by differences in coding practices for causes of death, and also by the varying ability in different countries to link mortality statistics from hospital administrative databases with overall death statistics. Unique patient identifiers (UPIs) and linking databases are the keys to making further progress in this area.

Performance indicators and benchmarking related to cancer care

Apart from being the major cause of death in EU countries, cancer is also the area in which relatively complete statistics are available in some countries on the performance of cancer care for various types of cancer (such as breast, cervical and colon cancer). Recent work by the CONCORD (cancer survival in five continents) group has reported on cancer worldwide,²⁷ and the EU^28,29 and the OECD are also active in improving the measures for international comparability on the performance of cancer care. For this work to make progress, the quality of (national) cancer registries is of key importance. As long as some countries do not have cancer registries that cover their entire population, valid performance data are difficult to produce. Furthermore, for cancer data to be useful for benchmarking, it is essential to have not only mortality data, but also cancer staging data. As with mortality data, linkage between cancer registries and administrative databases such as those in hospitals are essential to conducting meaningful research on the relationship between the quality of cancer care, and the use of services and resources. Although coding has been internationally standardized in cancer compared with other disease areas, further standardization is warranted to increase the potential for valid benchmarking.

Performance indicators and benchmarking on care delivered in hospitals

A great deal of indicator development work is being undertaken in the field of hospital indicators. Indicators are being developed and tested in countries such as Spain, Italy, Portugal, France, Germany, the UK, the Netherlands, Belgium and Denmark. Popular categories for indicators are 30-day case fatality rates (e.g. in acute myocardial infarction and stroke), readmission rates, complication rates in surgery, hospital infection rates, bedsores, volumes of specific treatments, waiting times, systematic measurement of patient experiences, and systematic measurement of experiences of hospital staff. Most HSR in this area focuses on the development and testing of indicators. Sources for indicators are mainly administrative databases and medical records.

The following are generic problems identified by this work and by international comparative work by WHO² and the OECD³⁰ using administrative databases: •

Quality of coding practices for administrative databases (ICD 9–10; International Statistical Classification of Diseases and Related Health Problems);

•

Lack of nationally and internationally standardized procedure codes;

•

Lack of coding for secondary diagnoses;

•

Lack of coding for whether a certain condition was present at admission; and

•

Lack of opportunities for linking the administrative databases of individual hospitals with other databases (e.g. by using a UPI).

Apart from the present limitations of administrative databases for HSR on performance indicators and benchmarking, deriving the appropriate data from medical records also poses problems. Although the techniques for audits based on medical and nursing records have been improving, methodological flaws are still reported. The approach taken in the USA by McGlynn et al.³¹ in their study on the quality of care for adults contains important lessons for Europe. Furthermore, in most countries not enough progress has been made in implementing electronic health records (EHRs) for them to be used as a primary source of data for calculating performance indicators. Most of the problems surrounding the development of EHRs to derive population-based statistics are political rather than technical. Privacy legislation and an insufficient focus on standardizing data requirements from a public information perspective have hindered HSR in this area.

Patient safety indicators

Patient safety has become a focal point of health policies over the past ten years. Following the publication of the report, ‘To Err is Human'³² in the USA, the EU-initiated several activities to coordinate policy development and research in this area. In 2007, a meeting was held in Porto, Portugal, to provide an overview of the ongoing research efforts.³³ At present, the EU-funded EUNetPaS (European Network for Patient Safety) project is trying to coordinate national efforts.³⁴ Also, WHO has launched programmes on a global scale that include inventories of ongoing research. Many European countries have executed studies to assess the magnitude of adverse events in their respective countries, mostly based on detailed audits of medical records. Several countries have also set up national patient safety agencies, which are also primarily involved in running adverse event reporting programmes. Furthermore, a growing body of knowledge has been created on studying safety culture, the implementation of safety systems and implementation programmes for specific safety projects on topics such as handovers, medication errors and reducing hospital infections. In the area of patient safety indicators (PSIs), the example of the PSI reporting system of AHRQ (Agency for Healthcare Research and Quality) in the USA³⁵ has been broadened to 17 other counties (including many European ones) as part of the OECD's Health Care Quality Indicators Project.³⁶

The type of HSR applied in all these efforts is not fundamentally different to the types of questions and methods discussed earlier in this article, and focuses on measuring risks, adverse events and errors and their contexts, as well as implementation research that evaluates the effectiveness of interventions. With respect to data collection, the problems identified are similar to those in collecting data for quality indicators of hospital care: •

Many studies depend on the quality of medical records;

•

EHRs are often an insufficient source of the necessary data;

•

Administrative systems often do not have sufficient secondary diagnoses coded to calculate PSIs;

•

Administrative databases often do not record whether relevant conditions (e.g. infections, bedsores) were present at admission; and

•

Linkage to other databases within the hospital (e.g. laboratory, pharmacy) or outside the hospital (e.g. primary care databases) is often not possible or not allowed.

If these data availability problems are not solved, the opportunities for, and impact of, HSR in the area of patient safety will remain limited with respect to performance measurement and benchmarking.

Performance indicators in primary care

Alongside hospital care, the design and functioning of primary care is a focal point for health policy. A primary care system that functions well is seen to help contain costs and improve the quality of care. HSR on the organization of primary care is discussed in another paper in this supplement. As part of the assessment of HSR related to performance indicators and benchmarking, it is apparent that data on the quality of primary care are still relatively scarce and heterogeneous. This is partly due to the fact that the information infrastructure in primary care is often still patchy and less developed than the administrative databases and availability of health records (including electronic health records) in hospital care. Although there are notable examples of countries that have further developed part of their primary health care information infrastructure for governance purposes (England, Denmark, the Netherlands), an assessment by the OECD in 2007 showed that national databases were not sufficiently developed or sufficiently comparable to merit cross-national collection and comparison of performance indicators on primary care.³⁷ The OECD is, at present, assessing the quality of primary care by looking at the rates of avoidable hospital admissions derived from the hospital administrative databases. Despite the limitations of the data systems, a great deal of HSR is being undertaken in primary care, often focusing on specific diseases (especially chronic diseases) and related care arrangements (e.g. disease management or implementation of guidelines). Benchmarking studies in primary care seem to be rare.

Patient experience

The systematic measurement of patient experience has become a fundamental element in assessing the performance of health care services and systems. HSR in this area consists of methods for determining the domains and topics that are considered to be important for performance (methods such as focus groups, interviews and concept mapping), and all types of questionnaire development and testing. Instrumental research on valid methods for assessing and reporting on patient experience is at the core of HSR in this area. Following the example of CAHPS (Consumer Assessment of Healthcare Providers and Systems)³⁸ in the USA and Picker³⁹ in the UK, a growing number of European countries are standardizing and institutionalizing the systematic measurement of patient experience. Apart from the validation of patient questionnaires, work has been reported on population-based surveys for measuring people's experiences with, and opinions of, health care (e.g. Eurobarometer, Commonwealth Fund surveys, WHO, CQ [Consumer Quality] Index). Apart from the ongoing validation of sets of questions, work is taking place on using vignettes to capture opinions.

Research on the practice of benchmarking and performance improvement

Although HSR focusing specifically on the technique and methods of industrial benchmarking in health care is still rare, there is a substantial body of evidence on implementation and innovation in health care coming from biomedicine but even more from the social sciences. Literature suggests that implementation of strategies for using performance indicators should not be considered as a separate research topic, but rather be seen as one strategy alongside others for improving the quality of health care. In a group of 489 hospitals, the EU funded MARQuIS Project (Methods of Assessing Response to Quality Improvement Strategies) demonstrated the synergy between strategies for using indicators, measuring patient experience, performing audits, and implementing clinical guidelines, patient safety systems and Total Quality Management (TQM).⁴⁰

Expert meeting/conference discussion

During a workshop at the working conference, Health Services Research in Europe, in April 2010, conference participants discussed three predefined topics in the field of benchmarking and quality indicators. Their discussions are summarized below:

Research on the validity of indicators

Conference participants regarded the underlying data for calculating indicators as an essential component of constructing performance indicators. The way data is collected as well as why it is being collected needs to be assessed in terms of validity. Another topic that was discussed was the low or even non-existent number of secondary diagnoses in most clinical registries and administrative databases. The great variation in available secondary diagnoses found nationally would seem to hamper international comparison. Because there seems to be a great deal of uncertainty about the amount of variation in the interpretation of definitions of indicators and related coding practices, this must be studied in more detail.

Considering the validity of indicators, a great many cross-sectional designs are currently being used. The participants agreed that more longitudinal data based on sound databases are needed to study the validity of indicators. The delegates agreed that indicators can be used for very different purposes (e.g. accountability, selective contracting, choice, quality improvement) and that the ultimate use of the performance indicators should be taken into account when studying their validity.

Requirements for national information infrastructures

Conference participants agreed that information sharing is hampered by a great Europe-wide difference in information infrastructures. For example, only a few countries allow patient level data to be linked using UPIs, which is of great importance to a comprehensive picture of quality of care. To allow data linkage, there clearly needs to be uniformity in registration software. At the moment, there is considerable variation in software systems and software providers. Participants also concluded that the evaluation of the quality of electronic databases for deriving population statistics is an area that has been under-researched. On a regional level, the ownership of data (e.g. in the case of Germany) and the inter-regional differences of coding can be a problem. More EU involvement in supporting the national and international comparability of information on health care through data infrastructures was considered to be useful (e.g. coherent definitions, similar software systems, and coding systems), as well as on policies to balance privacy and data-protection concerns with availability of data for calculating population-based performance indicators.

Use and misuse of performance indicators

Conference participants reported misuse or non-use of performance indicators (e.g. league tables, gaming), which led to the observation that systematic research on use and misuse is both rare and in its infancy in Europe. There were doubts about the extent to which patients used performance indicators. To be able to identify misuse of indicators, their meanings as well as their embedding in governance and managerial structures and processes must be known. To enhance the use of indicators, they should be integrated into an established system, and thus linkages with other quality strategies such as practice guidelines, accreditation, audits and quality systems should be sought.

Discussion

HSR can help both in the development of performance indicators and their eventual use through benchmarking. Our literature study and expert consultations identified a large number of existing research initiatives within the EU. The discussions during the conference on HSR in the Hague confirmed the following about performance indicators and benchmarking: •

Research should focus on the development of indicators (validity, reliability, relevance), as well as on their use (effective embedding in policy and management);

•

As a consequence of the above, HSR on these topics should always involve the potential users; and

•

Scientific approaches are needed from biomedicine and epidemiology (validity) as well as the social sciences (implementation and use).

The further progress of HSR on performance indicators and benchmarking is hampered by data availability. Experts agreed that the following issues need to be addressed: •

Use of UPIs to facilitate linkages between separate databases;

•

Further standardization of coding;

•

Use of present-at-admission codes in administrative databases;

•

Recording of secondary diagnoses in administrative databases;

•

Facilitating secondary data use from EHRs;

•

Facilitating standardized measurement of experiences by patients and other people; and

•

Continued collaboration between Eurostat, WHO and OECD to improve the availability of internationally comparable performance information.

HSR research on benchmarking and performance indicators at European level would benefit from a better exchange of research findings. It was observed that publications on HSR in general and on performance indicators in particular tend to be written in national journals and consequently in other languages than English. Efforts should be made to make this information accessible for other countries. This might ask for other media than the classical scientific publication media; also European wide training of researchers on health care indicators might help to enhance a true European Health Services Research community.

Results of HSR research on benchmarking and performance indicators should be systematically shared with policy-makers and managers of health services and systems to assure a good fit with local contexts. Networking should be stimulated at European level between the research groups involved in this kind of work and the growing number of regional and national institutes involved in quality measurement and reporting.

HSR that provides the evidence base for performance indicators and benchmarking is a field that has been expanding rapidly in EU countries over the past ten years. It is an area, par excellence, that should benefit from EU-wide initiatives, as this would increase comparability between member states and thus the potential of benchmarking between countries. Apart from policy initiatives to address the quality and comparability of national information infrastructures in health care through the kind of initiatives that are mentioned in this paper, it also constitutes a potentially fruitful area for HSR in the forthcoming Eighth Framework Programme.

Present research seems to be skewed towards measuring the performance of hospitals, with fewer studies on underlying theories and concepts, or empirical studies on the use of indicators for quality improvement. Based on the literature study and the discussions during the HSR in Europe conference in the Hague in April 2010, the following three areas are recommended for future research: (1)

Testing the validity and reliability of performance indicators (including more outcome-oriented indicators) related to avoidable mortality, cancer care, primary care, hospital care, mental health care, prevention and health promotion, palliative care, and long-term and social care;

(2)

Assessing the effectiveness and efficiency of embedding performance indicators in governance, monitoring and management systems, and their effect on health systems, health services and professionals;

(3)

Assessing the effectiveness and efficiency of linking performance indicators to other national and international strategies and policies, such as accreditation and certification, practice guidelines, audits, quality systems, patient safety strategies, national standards on volume and/or quality, public reporting, pay-for-performance and patient/consumer involvement.

Footnotes

Acknowledgements

The research summarized in this supplement was funded as part of the European Commission's Seventh Framework Programme (FP7/2007-2013) under grant agreement 223248. We are grateful for the ideas provided by participants to a workshop session dedicated to performance indicators and benchmarking during the working conference ‘HSR in Europe’, and to the experts who facilitated and reported on these discussions: Rosa Suñol (Avedis Donabedian Foundation, Spain), Sandra García Armesto (Health Sciences Institute in Aragon, Spain), Diana Delnoij (CKZ, the Netherlands), Thomas Plochg (NPFH, the Netherlands) and Helen Anema (AMC-UvA, the Netherlands).

Conflict of interest: None declared.

References

Agere

Promoting good governance: principles, practices and perspectives. London: Commonwealth Secretariat, 2000

World Health Organization. The world health report 2000. Health systems: improving performance. Geneva: WHO, 2000

Arah

, Westert

, Hurst

, Klazinga

NS.

A conceptual framework for the OECD Health Care Quality Indicators Project.

Int J Qual Health Care 2006; 18 (Suppl. 1): 5–13

Kelley

, Hurst

Health Care Quality Indicators Project Conceptual Framework Paper. OECD Health Working Papers, 2006

Zairi

, Leonard

Practical benchmarking: The complete guide. London: Chapmann & Hall, 1994

Camp

Benchmarking: The Search for Industry Best Practices that Lead to Superior Performance. New York: QR, 1989

Benson

HR.

An introduction to benchmarking in healthcare.

Radiol Manage 1994; 16: 35–39.

OECD. Health System Priorities in the Aftermath of the Crisis. Paris: Ministerial Meeting, 2010

Klazinga

, Fischer

, Ten Asbroek

Health services research related to performance indicators and benchmarking. In: HSR-Europe. Health Services Research into European Policy and Practice. Final report of the HSREPP project. Utrecht: NIVEL, 2011. See: http://www.healthservicesresearch.eu

10.

Groene

, Klazinga

, Kazandjian

The World Health Organization Performance Assessment Tool for Quality Improvement in Hospitals (PATH): an analysis of the pilot implementation in 37 hospitals.

Int J Qual Health Care 2008; 20: 155–61.

11.

Custers

, Klazinga

, Brown

AD.

Increasing performance of health care services within economic constraints: working towards improved incentive structures.

Z Arztl Fortbild Qualitatssich 2007; 101: 381–8.

12.

Custers

, Hurley

, Klazinga

, Brown

AD.

Selecting effective incentive structures in health care: A decision framework to support health care purchasers in finding the right incentives to drive performance.

BMC Health Serv Res 2008; 8: 66

13.

World Health Organization. The world health report 1995. See: http://www.who.int/whr/en/index.html (last accessed 23 August 2010)

14.

World Bank. Data Catalog. See: http://data.worldbank.org/data-catalog (last accessed 2 August 2010)

15.

Holland

, Paul

, Lakhani

Avoidable deaths.

Lancet 1988; 2: 169

16.

Holland

WW.

The “avoidable death” guide to Europe.

Health Policy 1986; 6: 115–7.

17.

Charlton

, Holland

, Lakhani

, Paul

EA.

Variations in avoidable mortality and variations in health care.

Lancet 1987; 1: 858

18.

Nolte

, Shkolnikov

, McKee

Changing mortality patterns in East and West Germany and Poland. II: short-term trends during transition and in the 1990s.

J Epidemiol Community Health 2000; 54: 899–906.

19.

Nolte

, McKee

Measuring the health of nations: analysis of mortality amenable to health care.

BMJ 2003;327:1129. Erratum In: BMJ 2004; 28: 328: 494

20.

McKee

, Hunter

Mortality league tables: do they inform or mislead?

Qual Health Care 1995; 4: 5–12.

21.

McKee

, Coles

, James

‘Failure to rescue’ as a measure of quality of hospital care: the limitations of secondary diagnosis coding in English hospital data.

J Public Health Med 1999; 21: 453–8.

22.

McKee

, Shkolnikov

Understanding the toll of premature death among men in eastern Europe.

BMJ 2001;323:1051–5. Review. Erratum In: BMJ 2001; 323: 1423

23.

AMIEHS: Nolte

, McKee

Does health care save lives? Avoidable mortality revisited. London: The Nuffield Trust, 2004. See: http://www.nuffieldtrust.org.uk/ecomm/files/21404avoidablemortality2.pdf (last accessed 27 July 2011)

24.

Jarman

, Gault

, Alves

Explaining Differences in English Hospital Death Rates Using Routinely Collected Data.

BMJ 1999; 318: 1515–1520.

25.

Jarman

, Aylin

, Bottle

Acute stroke units and early CT scans are linked to lower in-hospital mortality rates.

BMJ 2004; 328: 369

26.

Jarman

, Bottle

, Aylin

, Browne

Monitoring changes in hospital standardised mortality ratios.

BMJ 2005; 330: 329

27.

Coleman

, Quaresma

, Berrino

CONCORD Working Group. Cancer survival in five continents: a worldwide population-based study (CONCORD).

Lancet Oncol 2008; 9: 730–56.

28.

Richards

EUROCARE-4 studies bring new data on cancer survival.

Lancet Oncol 2007; 8: 752–3.

29.

Verdecchia

, Francisci

, Brenner

EUROCARE-4 Working Group. Recent cancer survival in Europe: a 2000-02 period analysis of EUROCARE-4 data.

Lancet Oncol 2007; 8: 784–96.

30.

Armesto Garcia

, Lapetra Gil

, Wei

, Kelley

and the members of the HCQI Expert Group. Health care quality indicators project 2006. Data collection update report. DELSA/HEA/WD/HWP(2007)4. See: http://www.oecd.org/dataoecd/57/22/39447928.pdf (last accessed 2 August 2010)

31.

Schuster

, McGlynn

, Brook

RH.

How good is the quality of health care in the United States? 1998.

Milbank Q 2005; 83: 843–95.

32.

Stelfox

, Palmisani

, Scurlock

The “To Err is Human” report and the patient safety literature.

Qual Saf Health Care 2006; 15: 174–8.

33.

European Commission. Patient Safety Research: shaping the European agenda. Porto: conference report, 2007. See: http://ec.europa.eu/health/archive/ph_systems/docs/ev_20071122_rd01_en.pdf (last accessed 16 November 2010)

34.

EUNetPaS (European Union Network for Patient Safety). See: http://www.eunetpas.eu/ (last accessed 16 November 2010)

35.

AHRQ Quality Indicators - Guide to Patient Safety Indicators. Version 2.1, Revision 2 (22 October 2004). AHRQ Pub.03-R203. Rockville, MD: Agency for Healthcare Research and Quality, 2003

36.

Marshall

, Klazinga

, Leatherman

OECD Health Care Quality Indicator Project. The expert panel on primary care prevention and health promotion.

Int J Qual Health Care 2006; 18 (Suppl. 1): 5–13

37.

OECD Health Policy Studies Improving Value in Health Care: measuring quality. Paris: OECD, 2010: 40–42

38.

Goldstein

, Farquhar

, Crofton

Measuring hospital care from the patients’ perspective: an overview of the CAHPS Hospital Survey development process.

Health Serv Res 2005; 40: 1977–95.

39.

Jenkinson

, Coulter

, Bruster

The Picker Patient Experience Questionnaire: development and validation using data from in-patient surveys in five countries.

Int J Qual Health Care 2002; 14: 353–8.

40.

Suñol

, Garel

, Jacqerye

Cross-border care and healthcare quality improvement in Europe: the MARQuIS research project.

Qual Saf Health Care 2009; 18 (Suppl. 1): 3–7