Abstract
Recognizing that robust information on health systems in other countries can provide valuable lessons for the English National Health Service, the Department of Health commissioned an academic team to provide an ‘On-call Facility for International Healthcare Comparisons’ in 2005. This paper describes the work of this novel approach to informing policy and reviews the experience of the first two years. It illustrates the well-documented challenges of comparative analysis of health systems. One important issue is understanding the health system context so as to interpret phenomena and draw appropriate policy conclusions. Other challenges include the potential tension between academic interest and rigour, and the need for timely analysis to inform the Department of Health's rapidly changing policy agenda. The diversity and nature of topics covered, as well as the rapid turn-around time have meant that the Facility has had to balance rigour and timeliness carefully to ensure the value and relevance of reports. A strong research base linked with an international network of country experts promotes the provision of high quality analyses at relatively low costs. However, such an arrangement can only be sustained if it provides scope for additional primary research. A formal evaluation of the influence on health care policy-making in England is not yet available. Such knowledge will be of crucial importance for the development of similar resources elsewhere.
Introduction
There has been a long-standing interest in cross-national comparisons of health systems among health policy analysts. They offer a means to capture the range of approaches countries have adopted to address similar challenges and so allow the experience of each country to provide ‘an experimental laboratory for others’.1,2 They offer opportunities for mutual learning and consideration of policies, cross-fertilization, or even policy transfer, where appropriate.3,4 ’ The Department of Health in England has recognized the potential for robust information on health systems in other countries to provide valuable insights relevant to policy development in the English National Health Service (NHS) and, in 2005, it took the novel step of commissioning an academic team to provide an On-call Facility for International Healthcare Comparisons’. The Facility was to ‘undertake a programme of work on international health care comparisons, provide intelligence on new developments in international health policy and operate on an “on-call” basis to ensure rapid response as specific requirements arise’. 5 We describe the work of the Facility which represents an innovative approach to informing policymaking and review the experience of its first two years.
Comparative analysis of health systems
Cross-national comparisons of health care systems can be traced back to at least the 1930s, rooted in an interest in their historical evolution, as exemplified by the work of Sigerist, 6 much of which had the goal of informing developments in national health policy.7,8 From the 1960s onwards, cross-national comparisons received increasing attention, with influential examples including works by Abel-Smith,9,10 Roemer,10-12 Anderson 13 and Mechanic. 14 More recently, the notion of learning from experience elsewhere has been supported by the work of the Organisation for Economic Co-operation and Development (OECD) on international benchmarking of health systems, through a series of international studies published from the mid-1980s, designed to provide an empirical basis for a comparative understanding of the differences and similarities between OECD countries’ health systems.4,15 In addition, structured descriptions of health systems in Europe have been carried out by the European Observatory on Health Systems and Policies.16-18
Cross-national comparisons are now widely undertaken by academics and policy-makers alike. However, they can be problematic. Unless they are based on a detailed understanding of the nature of health systems and the settings in which they are embedded, they may be superficial, leading to misinterpretation of phenomena and misleading conclusions. 19 Many comparisons are based on routinely collected data that reflect what can be counted rather than what is important, 20 potentially producing ‘a lot of measurement but not much understanding’. 21 And as Klein has noted, cross-national curiosity is not always a neutral intellectual exercise. There is always a risk that the ‘experience of other countries may be used in domestic policy debate to inspire either emulation or repudiation’. 19 In the UK, recurring calls to replace tax funding with social health insurance are an illustration of a contextually ill-informed debate.22-24
The Department of Health's initiative in setting up the On-call Facility on International Healthcare Comparisons (the Facility) was based on an expectation that learning from other countries would assist health policy-making by giving proposed developments an international perspective and adding to the range of policy ideas and options emerging from domestic policy processes. The Facility was also set up to contribute to health policy-making in a new way. Its remit to provide a rapid response was motivated by the desire to make its work as relevant to, and useable by, policy-makers as possible. By establishing the Facility, the Department of Health (DH) took an innovative approach which, to our knowledge, no other government has done. This report on the experience of the Facility aims to inform the development of any similar initiative elsewhere.
We begin by briefly describing the Facility's structure and the nature of its analyses, process, timeframe, outputs and dissemination. We then review its experience to date, discussing lessons learnt and options for taking the project forward.
An ‘On-call Facility’ for International Healthcare Comparisons
The structure
The Facility was set up in October 2005 to work for an initial period of three and a half years. It brings together experts from Australia, Canada, Denmark, Finland, France, Germany, Italy, Norway, The Netherlands, New Zealand, Spain, Sweden and Switzerland (one in each country), together forming the International Healthcare Comparisons (IHC) Network. It is coordinated and managed by a research team based at the London School of Hygiene and Tropical Medicine (LSHTM), University of London. The Department of Health had identified the countries of interest in its commissioning brief. Country experts were selected by the LSHTM team on the basis of their expertise in health policy in their own countries and their capacity to access information on a diverse range of topics through their involvement with a wide network of contacts in their countries. The majority of experts are based in academic institutions, representing health services research, health economics, health care management and related disciplines, with many also occupying roles as government advisors or active members of national/regional policy bodies. Experts are paid a small annual fee for their participation in the network rather than for each request for their expertise.
The LSHTM team operates closely with the Department of Health through a designated research liaison officer and an Advisory Group comprising members of the Department. It has also established its own Steering Committee of senior academics, policymakers and research directors drawn from the European Observatory to advise periodically on general policy issues as well as the design and quality of individual on-call reports. The specific arrangements for coordination and project management are set out in Figure 1.
Arrangements for project management and coordination
Type and format of policy analysis
The over-arching approach to the work is comparative analysis based on reports from individual country experts, backed up by a synthesis of published and grey literature. Thus the primary data sources are the country experts in 13 countries. Their input is based not only on their own expertise but also consultations with in-country policy informants and other researchers. Additional data sources include governmental and non-governmental policy documents (green papers, white papers, consultation documents, proposals, committee reports, parliamentary hearings, legislative bills, policy analyses, etc.). The Facility also draws on a range of secondary data sources such as the European Observatory's ‘Health System Reviews’; and the International Network for Health Policy and Reform's Health Policy Monitor’ reports. 25
The process
The nature, scope and timing of the work of the Facility is determined in collaboration with the Department of Health to ensure targeted, relevant and concise policy analyses. The Advisory Group comprises members from selected parts of the DH, largely representing NHS policy and strategy, including staff from the Strategy Unit. It has a key function in the work of the Facility by inviting bids for comparative policy reports from selected teams within the Department (customers). Bids are coordinated by the research liaison officer within the Department, who plays a crucial role in ensuring that the communication and interactions between the LSHTM team and the DH run smoothly. Using a template, the DH customers make a case for the specific request, outlining the policy context, the added value of international evidence, alternative routes to obtain the evidence and the level of urgency. Requests are discussed in regular, usually quarterly, meetings between the Advisory Group and the LSHTM team, and ranked in terms of relevance, urgency and research feasibility. Usually, one or two requests are selected for the ensuing period of two to six months. Bids that are not selected may be kept in play and discussed for prioritization at a later date. This structured process is complemented by ad hoc, more rapid turn around (between one day and two weeks) requests that are commissioned when need arises, although their number is limited to fewer than five per year.
The precise nature, scope and timeline of the individual request is then discussed further between key Departmental contacts, representing the customers, and the LSHTM team. The team develops a draft work plan and, where appropriate and necessary, a questionnaire to obtain detailed information from the country experts and/or others. It reviews the literature on the topic, using standard methods, collates the responses from country experts, and then analyses and synthesises the experts’ information in the light of published and grey literature. This process is highly iterative, with intermediate findings discussed in meetings between the LSHTM team and the DH customers. Also, interim analyses are shared with the country experts who have provided input to review and confirm individual country accounts. The overall process is depicted in Figure 2.
Producing comparative policy analysis: the process
Time frame
The turn-around time of rapid response requests was originally set at three to four weeks on the assumption that the work of the Facility would comprise a mix of rapid and longer-term work, such as an annual overview of policy developments in each country. The time frame for rapid response requests was subsequently adjusted to accommodate better the realities of working with the country experts and responding to the demands of DH customers, who have generally focused on shorter-term rather than longer-term policy analyses, and more specific issues rather than general overviews of trends in different countries. A rapid response project now aims to take about six to 12 weeks from the agreed request to production of the policy report, with medium-term work addressed within two to six months. The main reason for lengthening the response time is the multiple competing commitments of country experts, which challenges the collection and provision of detailed and accurate information within a very short time frame. Country experts cannot always drop their other commitments to give priority to a Facility request. The rapidly changing policy agenda of the Department of Health has meant that, so far, requests cannot be agreed more than a few months ahead.
Outputs and dissemination
The requests for comparative analyses from the Department of Health have covered a very wide range of topics. Work over the first two years addressed a variety of health policy issues of high priority, most of which are closely related to the health reform agenda in England, 26 including, among others:
access to health care outside hospital;
health care capacity planning;
reimbursing highly-specialized hospital services.
The formal product is normally a policy report of 8000 to over 30,000 words, all including short summaries. The rapid turn-around of requests has thus far precluded the establishment of a formal external peer-review process. However, individual country case studies that normally form an integral part of reports are reviewed by the relevant country expert to ensure that the account produced by the LSHTM team is correct. Also, some reports have undergone ad hoc review by experts external to the Facility (for example, if the content is regarded as technically challenging).
The longer-term intention of the LSHTM team is to convert in-depth policy reports into policy briefs published through the European Observatory, following review by its senior staff (two have been published to date) and, where possible, articles for publication in professional and peer-reviewed journals.27-29 Policy reports and related publications are available through the project's website (http://www.lshtm.ac.uk/ihc) with the Department's agreement under the standard research contract terms.
Experience to date and lessons learnt
A formal external evaluation of the influence of the Facility on health policy-making in England is not yet available. However, one year into the project, the LSHTM team undertook its own evaluation of the work, eliciting views from country experts using a structured questionnaire, and from DH customers and members of the Advisory Group through discussion in a formal meeting. The internal evaluation aimed to identify potential concerns about the work of the Facility, both in terms of processes and outputs, to optimize collaboration with the country experts and to enhance its value and relevance for health policy in England.
Department of Health
Feedback from customers and members of the DH Advisory Group has confirmed the timeliness and usefulness of the Facility's outputs for their work. It is not only the final product, i.e. the policy report, which has been rated as useful. The process of identifying, discussing and clarifying the policy issue, its translation into a researchable question in the light of existing evidence and the potential to learn from international experience have all been described as being of high value in themselves. As a result of this process, some selected requests were not pursued further either because comparative analysis was not deemed appropriate for a particular subject or because the LSHTM team was able to identify sufficient relevant published material.
Another indication of the usefulness of the Facility for policy-makers is the number of on-call requests generated by the Department and the number of requests jointly agreed with the Department. Both have slightly increased over time, although, at present, the Facility is only available to those sections of the Department involved, directly or indirectly, with the NHS health reform agenda. As yet there is no mechanism for tracking the use of the Facility's policy reports within the Department. One clear indicator will be whether the Department of Health wishes to continue the contract beyond April 2009.
Country experts
Country experts were invited to complete a brief structured questionnaire in October 2006 that aimed to assess their perspectives on the timing of information requests, communication between the LSHTM team and experts, interest in and relevance of health policy topics to their own systems, usefulness of project outputs, and the main perceived achievements of, and challenges facing the Facility. The response rate was 69% (9/13; of the four non-responders, one had joined the project only in the month preceding the evaluation and one is also a member of the LSHTM research team).
Country experts valued the opportunity for networking, found the work interesting and the reports useful for their own countries. Most experts expressed satisfaction with the way that their information had been used and noted that the requests were of value for their own work. Most experts had used the policy reports on one or more occasions: for teaching, research, in discussions with policy-makers or for other publications and policy briefs.
However, experts also expressed some concern related to the timing and the context or relevance of information requests. They pointed out that deadlines set for responding to an information request needed to be more flexible to allow them to respond in an informed way. This is mainly because the Facility is additional to experts’ other work and information requests often require fact-finding and interviews with other informants. In addition, there was concern as to whether health policy topics of high priority in England were equally topical and of interest in other health system contexts. The experts also noted that more attention should be paid to the specific system context when analysing and presenting information on health policy or health services across countries (an issue that the LSHTM team tries to bring out in all its reporting). At the same time, the Facility was viewed as providing opportunities to compile international data which could lead to researchers and policy-makers developing new questions for primary collaborative research.
Learning from others?
The work of the Facility illustrates many of the challenges international health care comparisons face and these have been addressed in some detail else-where.20, 30-33 Here we focus on different dimensions of the process of providing intelligence on developments in international health policy.
Importance of context
One important issue is understanding the health system context so as to enable fair interpretation of observed phenomena and the drawing of appropriate policy conclusions. For example, Klein argues that if cross-national learning is to be successful, comparisons not only require an analysis of the relevant political institutions, but also of ‘civic traditions, tax systems, and administrative resources’, and that this should take priority over the analysis of specific health policy issues. 19
This requirement can pose challenges to the provision of rapid policy analysis, which often limits the depth of analyses of the system context. However, successive policy analyses undertaken by the Facility have increased the LSHTM team's and some of its customers’ levels of insight into different health system contexts. It is noteworthy that each new set of customers at the Department of Health has had to be briefed, in varying degrees, about the culture and institutional history of individual health systems included in the analysis so as to enable better understanding of the content of the policy (the customers’ usual focus) in light of the context, processes and actors giving rise to it. 34
Timeliness and rigour
A related challenge is how to reconcile academic interest and rigour with the Department's rapidly changing policy agenda, which determines the type of requests. There is a potential tension between the interests pursued by policy-makers (in the Department of Health) and academics (the LSHTM team), and the diversity of sources and types of knowledge they draw upon to inform their work. As Klein notes, although they ‘both operate in the field of health policy […] their concerns and questions will be rather different’. 19 Indeed, the diversity and nature of topics covered so far, as well as the rapid response time have meant that the Facility has had to balance rigour and timeliness carefully to ensure the value and relevance of reports for health policy in England.
Addressing scarcity of evidence
Challenges also arise from the relative novelty of some of the policy issues to be addressed that have only been looked at in a few countries, such as new approaches to caring for people with chronic conditions, and which are not well documented. Without undertaking primary research, it is often only possible to describe policies with limited scope for assessing their consequences. For some policy topics, particularly those that are at an early stage in only a small number of countries or are changing rapidly, such as the use of risk-weighted capitation resource allocation formulae, it has been more appropriate to convene a meeting of experts from countries where relevant policies have been tested. Organized by the LSHTM team, face-to-face discussions with representatives from the Department rather than written reports based on little sound, published evidence, are more likely to provide insight. On one occasion, the LSHTM team facilitated a rapid study tour for DH representatives to selected countries for a given topic.
Self-examination
Klein has argued that policy learning ‘is as much a process of self-examination - of reflecting on the characteristics of one's own country and health care system - as of looking at the experience of others’. 19 Yet, perhaps surprisingly, particularly in retrospect, the IHC network does not include an expert for England. This is based on the implicit assumption that the LSHTM team and DH customers possess sufficient in-depth knowledge about relevant aspects of the English health system. Yet, as experience has shown, this is not necessarily the case. The commissioning brief for the Facility did not envisage that reports would normally include England as a case study. Sometimes, the LSHTM team has requested an English case study from DH customers to provide a benchmark without which accounts from other countries would be difficult to interpret, in particular in terms of deriving any policy implications for England. The discipline of providing an account of how a policy issue is currently addressed in England has proved valuable for the Department itself and is now part of the Facility's routine way of working.
The same but different?
Increasing interest among policy-makers in cross-national comparisons of health systems and the potential to learn from experience elsewhere has stimulated the establishment of a range of international initiatives to support this process, including the European Observatory on Health Systems and Policies and the International Network for Health Policy and Reform's Health Policy Monitor’.16,25 Similar to the Facility, these resources aim to inform policy-making through the analysis of available evidence 16 and/or the compilation of regular reviews and updates on health policy trends and developments in a range of industrialized countries. 25 All three resources work on the basis of networks of health policy experts in relevant countries as the principal source of information. They also collaborate closely, drawing on many of the same experts and some of the Facility's reports are published through the European Observatory. However, the Facility is unique in that its output is determined in close collaboration with and tailored to the specific needs of a single customer, the Department of Health. In contrast, the outputs of the European Observatory and the Health Policy Monitor are targeted at a much broader audience. Lavis and colleagues distinguish four principal models for linking research to (policy) action: push efforts by research producers; (research) user pull efforts; exchange efforts; and integrated efforts. 35 They describe the European Observatory as a resource that facilitates research user pull through the provision of evidence to inform policy in the form of written work and information about the best available research. Using this classification, the Facility may be described as a direct user-pull effort in which the research user (the Department of Health) has developed structures and processes to promote the use of research in decisionmaking. However, the Facility also shows elements of what has been described as exchange efforts in which the producers of research (here the LSHTM team) develop a partnership with the group using the research (the DH) to establish ‘a shared understanding about the questions to ask, how to answer them […] and how to weight the research and other types of information that each group brings to the table’. 35 This is facilitated by the fact that the Facility works closely with a single customer (albeit a large central government department).
The future
One of the strengths of the Facility is its strong base in health systems research which ensures the provision of high quality analyses at relatively low cost compared to the cost of management consultants. However, from an academic perspective, the sustainability of the current arrangement can only be ensured if it provides for the development of a more nuanced approach with scope for some primary research on important but poorly documented and analysed health policy topics. This will also help provide knowledge for the Department of Health that cannot be provided by the expert network alone.
At present, the IHC network operates on the basis of one paid expert per country, principally for budgetary reasons. As indicated above, the country experts contributing to the Facility were selected to combine an academic background with active involvement in policy-making. They possess wide expertise in health policy in their countries and can thus provide a well-informed overview of the health system in which they operate. However, the nature of some requests has required detailed insight into very specific issues or technical expertise which has meant that additional experts had to be approached. Allowing time for more primary research is likely to support this process, as might restructuring of the current arrangement of one expert per country in favour of a team of experts in each country able to cover different aspects of the relevant health system. The cost implications of a revised structure would, however, have to be carefully balanced against the likely gain in terms of expertise and insight.
The work undertaken to date has been largely topic-based rather than country-based, and the applicability of selected health policy topics to individual countries has changed with each topic, as has the DH customer requesting the comparative analysis. It thus remains a challenge to develop an in-depth understanding of the formal and, particularly, the informal institutional structures and relationships within each of the individual health systems included in the IHC network. However, detailed knowledge of the particularities of a given health system is essential for appropriate cross-national understanding and potential learning. As a result, it will be important in future to maintain continuity of experience in comparative health systems analysis in the LSHTM team. Likewise there are considerable advantages if there is some continuity of personnel in the involvement of the Department of Health, particularly in the liaison officer role and in membership of the Advisory Group.
Finally, there is a need to identify more systematically whether and, if so, how, the work of the Facility makes a difference to national policy-making in England. This is not only of academic value but important for the development of any such resource elsewhere. The Facility is due to be formally reviewed in 2008, at which point these judgements can be made.
Footnotes
Acknowledgements
This work is funded by the Department of Health, grant no. 0510002. EN is funded by a Career Scientist Award from the National Institute for Health Research. The Department of Health does not accept any responsibility for the views expressed.