Evidence-based policy as reflexive practice. What can we learn from evidence-based medicine?

The example of evidence-based medicine

In health care, the example of evidence-based medicine (EBM) is often invoked in this situation. The EBM movement, as generally understood, has succeeded in bringing about a radical change in the way medicine in taught and practised. Just as with the notion of evidence-based policy (EBP), this has brought both praise and criticism. Critics argue that EBM has lead to standardized care in which the individual patient has suffered under the violence of the evidence produced in randomized controlled trials, and in which the autonomy of individual doctors has been denied. ¹⁰ Advocates of EBM, on the other hand, have argued that EBM has led to reducing bad quality of care and that standardization has strengthened rather than weakened the profession. Both groups, however, tend to plead their cases by reference to a rather formalistic version of EBM that does little justice to the ways in which EBM is practised. ¹¹

A similar emphasis on the formalistic, external characteristics of EBM can be found in arguments to translate EBM to the world of policy and management.^12,13 These authors thereby sidestep the situated and emergent character of EBM as described especially by scholars from Science & Technology Studies. Despite all claims, EBM is most certainly not a universalistic operating machine but displays much local variation. ¹⁴ Moreover, translating the formalistic character of EBM to policy contexts has proven to be of little practical use. Even Chalmers, in the just cited paper, has to admit that despite all attempts to do just this, EBP has not come about. For example, the Dutch Centre for Healthy Living’s list of evidence-based interventions has not led to interventions that can be used, let alone be successful in practice. ¹⁵ The complexity and dynamics of policy contexts are usually not taken into account in these translations and political realities are ignored.

In this essay, I want to explore another, more pragmatic, picture of EBM so as to contribute to the discussion on the (im)possibilities of using EBM as a model for EBP. I take an insider’s perspective – without trying to suggest that an outsider’s critique of EBM or EBP would not be interesting or valuable – in which I am especially interested in the question of how EBM, and in its wake EBP, can be successful. I will do this by analysing three episodes in the practice of EBM. Those three episodes can also be seen as three consecutive phases in the practice of EBM: the production of evidence through medical research (especially the clinical trial); the gathering of evidence in the production of clinical guidelines; and the use of evidence in health care practice through the implementation of guidelines. The analysis of those episodes is based on research I have been involved in.

The clinical trial as a reflexive space between research and practice

The randomized clinical trial (RCT) is probably the greatest contribution of the EBM movement and referred to by Timmermans and Berg as the ‘gold standard’. ¹⁴ The RCT was developed during the 20th century in reaction to the disadvantages of a medicine based on clinical practice by translating methodological prescriptions from the sciences to health care, and developing an ethics of medical research (e.g. the practice of informed consent).^16,17

Like EBM in general, the RCT also has its critics. Again, the standardization central to RCT is the main target. To perform a clinical trial, one has to standardize the intervention and the context in which it is delivered, so that similar individuals are treated in the same way. This necessity produces frictions. Standardizing the group being researched leads to the exclusion of all kinds of groups – children, older people and people with multiple diseases. Interventions are kept simple, that is ‘evaluable’, because complex interventions are harder to research in the RCT framework. The dynamics and complexity of practical contexts are usually ignored. RCTs thus lead to knowledge production about ‘ideal’ patients in ‘ideal’ circumstances that usually have little value for practice. As a result of these criticisms, other forms of experimental research have been developed, like the ‘pragmatic trial’, but these usually have the same problems. ¹⁸

The criticism of the RCT on the one hand is justified but on the other it also misses out on an important point: the practice of medical research is different from what is prescribed in the methods textbooks, or described in the methods sections of scientific papers. Trials in practice are much messier than in the methods literature, if only because of problems with the inclusion of patients, the impossibilities of completely standardizing treatments or the difficulty of managing different contexts.

An example is the ‘PreCare’ trial, a randomized trial in youth care, in which an intervention that was originally developed and tested in the USA on ‘problematic teen mothers’ was translated to the Dutch context. ¹⁹ The intervention entailed youth workers paying a total of 18 visits to mothers in the research group on the basis of a standardized protocol. In practice, this proved to be near to impossible, due to no show of the mothers, or due to their home situation, e.g. the presence of aggressive partners, etc. Also, the youth workers were dissatisfied with the trial because they did not want to withhold an intervention they thought to be effective from the control arm of the trial.

To deal with these and similar problems, within the context of the trial, ‘practice groups’ were organized, bringing together the researchers, the youth workers and their managers. These groups mediated between the research and practice, and devised situated, practical solutions for the frictions found. Not all conflicts could be remedied through this procedure; the youth workers remained dissatisfied with not being able to give the intervention to the control group, although they better understood the necessity of this for the research – but many practical problems were resolved. In many cases, this meant that the trial had to compromise on the standardized protocol, and the researchers had to find solutions to legitimate this in the context of the trial methodology.

As this example shows, trials are not standardized evaluation machines for the uniform application of interventions in practice, but need flexibility and compromise for their execution. The PreCare trial operated as a reflexive space in which situated, pragmatic solutions for frictions between research and health care practice could be formulated and applied. It is this reflexive space that makes the trial – and by extension, EBM – possible and workable.

Guidelines and dealing with uncertainty

Clinical guidelines emerged as a response to reports from clinical epidemiologists in the USA in the 1980s that there were huge variations in medical practice and health care outcomes between medical doctors and health care organizations. Guidelines at first were mainly intended to support medical practitioners in making decisions in diagnosis and treatment. Over time, guidelines have developed to be increasingly based on evidence from RCTs and meta-analyses. A guideline industry has emerged that is itself governed by numerous guidelines. In the Netherlands, the method of evidence-based guideline development (‘evidence based richtlijn ontwikkeling’ or EBRO) is popular, offering many methodological handles, for example, on the weighting of studies from the literature in which systematic, experimentally gathered evidence is preferred over other types of studies. The newest kid on the block, Grading of Assessment, Development and Evaluation (GRADE) allows for more leeway and, among others, considers the relevance of the evidence found in the literature in weighting studies to be used in guidelines. ²⁰

Guidelines have met similar criticism as the RCT, in that they promote the standardization of care and thus overlook the complexity of individual patients. Moreover, guidelines tend to become ever more comprehensive and as a result give less guidance to practice. In most countries, guidelines have been taken up to control medical practice by third parties (governments, payers) and have thus neglected their original purpose of supporting practical decision-making. Critics have also pointed out that patients have too little influence on the development of guidelines and that in guideline development the voice of health care practitioners is bypassed. ²¹

Here again, critics (and advocates) have largely ignored the fact that the practice of developing guidelines differs from the methodological guides such as EBRO and GRADE. Sociologist of science Tiago Moreira has shown, for example, that in the development of guidelines by the English National Institute for Health and Care Excellence (NICE), at least four types of evaluation are used, of which the methodological is only one. ²² Additionally, guidelines show large international differences of which differing practice in dealing with uncertainty or conflicting evidence is one possible explanation. ²³

To gain insight in the practice of guideline development, we researched the ways in which guideline developers try to deal with uncertainty. While guidelines were originally a means to give medical practitioners support in dealing with uncertainties, guideline developers themselves are also confronted with all kinds of uncertainties, for example, because of a lack of evidence. From interviews with guideline developers in different sectors of health care (e.g. medical specialist care, public health, general practice), it appeared that guideline developers had developed all kinds of mechanisms to deal with such uncertainties. ²⁴ The methodological guidelines play a role in this, but intertwined with a practice-based strategy in which the knowledge and experience of medical practitioners is used for guideline development. Guideline development for infectious diseases proved to be the most advanced in this respect, probably driven by a lack of research results related to the spread of new infectious diseases combined with a need to act immediately. Guideline developers in this area had developed a network with practitioners from public health services across the Netherlands in order to be able to respond to new developments and monitor the effects of the proposed guidelines.

With the development of clinical guidelines, we thus see a much more reflexive practice than would have been expected on the basis of the guidelines for guideline development, and which allows the gathering and use of all kinds of knowledge, alongside that produced in experimental research. Following sociologists May and Finch, we could speak of a practice of reflexive monitoring, ²⁵ allowing for and stimulating the mediation between evidence and practice, and stimulating productive practices of guideline development; that is: stimulating the development of guidelines that both reflect the state of the art in research and can be used in practice.

Using guidelines in health care practice

Within the EBM movement there have long been concerns about the use of clinical guidelines in practice. While guidelines were meant to reduce practice variation and quickly disseminate the results of experimental research into health care practice, the results of this endeavour seem to be disappointing. Practice variation has in fact hardly been reduced over the last decades, as is shown by the unending stream of publications on the topic, ²⁶ and the take-up of guidelines in medical practice has also not been met with much enthusiasm. Research on the implementation of guidelines consistently shows that guidelines are used in about 50% of cases. ²⁷

This has given impetus to a whole range of flanking policies and practices that should stimulate the uptake of guidelines. As a result, guidelines play an ever-increasing role in regulation and procurement of health care. Also, many large-scale quality programs have been set up to ‘roll out’ guidelines – in the Netherlands, these include programs such as ‘Better Faster’ and ‘Care for Better’.^28,29 Also, the science and practice of implementation have been further developed so that ‘implementation science’ in health care is now defined as an academic field in its own right, with its own journals and academic associations. ³⁰

Research on the use of guidelines in practice, however, often uses a rather problematic perspective on use, often just seen as the ‘correct’ application of a guideline, in which ‘correct’ is more often than not defined by the researchers. Questionnaires, analysis of medical records or observations are then used to ‘measure’ the percentage of patients or medical actions in which the guideline is applied. Again, ethnographic research on medical practice paints a different picture, in which the use of guidelines and protocols receives new meaning. Care, as shown in this type of research, is not just the application of rules from guidelines, but a much more messy process in which different options are constantly weighed and tested given the often complex situation in which patients (and professionals) find themselves. ³¹

In a study on the use of a guideline in heart failure by cardiologists, we started off studying patient records to see in which and how many cases the guideline was followed. ³² Consistent with the literature, this appeared to be in about 50% of cases. In a next step, we interviewed the cardiologists – with the patient’s records to hand – to get a feel for their experiences in using the guideline and find out why they did (not) use the guideline. From these interviews, it not only appeared that the cardiologists were well aware of the existence of the guideline and applied it in their practices, but also that often they had good reasons not to apply the guideline with specific patients. Often this entailed the sensitivity of a patient to specific drugs or the fact that patients suffered from multiple conditions which made the direct application of the guideline to those patients impossible. In the end, it remained unclear why the guideline had not been applied in only 10% of cases – a completely different picture than often found in the literature on guideline use.

Clinicians thus appear to take guidelines much more into account than is often assumed. Guidelines do give direction to their actions. This, however, does not mean that they follow guidelines uncritically. It is precisely in the confrontation between the guideline and the patient (and possible other factors) that decisions on treatments emerge. Again, this leads to the conclusion that the use of evidence is a reflexive practice in which guidelines together with their contexts of application lead to specific diagnostic or therapeutic interventions. Guidelines are an essential part of this, but, instead of dictating specific actions, they offer a kind of ‘reflexive guidance’. As in RCTs and the development of guidelines, the use of guidelines in health care practice can be described as a practice of mediation between the guideline and the specificities of the context of application.

Implications of the practice of evidence-based medicine for evidence-based policy

I started this essay with the question: what we can learn from evidence-based medicine for EBP? Rather than basing this on formalistic prescriptions of EBM often found in the literature, I took a practice-based approach, focussing on the ways in which EBM is enacted in practice. By analysing three episodes in the practice of EBM, I have tried to show that the success of EBM is based on completely different mechanisms from those often assumed. Three such mechanisms seem crucial: creating reflexive spaces for the mediation of research and practice; the reflexive weighting of evidence in the light of real world contexts; and the use of evidence as a form of reflexive guidance. What do these insights mean for the practice of EBP? And, what are the prospects that these insights – themselves a product of research – can be translated to policy contexts?

Before addressing the first question, let me start with a cautionary note. When translating evidence from one field to another, one always risks not taking into account qualitative differences between the settings. Some authors have, for example, argued that EBP is by nature much more complex than EBM as it has to deal with more heterogeneous contexts and builds on more practical epistemologies.^9,33 While I agree that context matters and that EBP should take into account the varying complexity of the problems and contexts it tries to tackle, ⁸ I hope to have showed in this essay that the practices of EBM are complex in themselves and that the differences between EBM and EBP in this regard are not essential. Learning across domains moreover needs a more practice-based approach since this gives a deeper understanding of why a specific practice is successful (or not). The formalistic translations of EBM to EBP arrangements that I cited earlier are unproductive precisely for the reason that they lack such a practice-based approach. Too large a separation between research and policy, for example, is not desirable as it prevents the necessary reflexive mediation between the two – similar to the connection between science and practice in EBM.

The analysis of EBM presented above also means that practices of reflexive monitoring should be taken much more seriously in EBP, and should lead to mutual sense-making between researchers and policy actors, leading to modification of policies. This then also entails that policy arrangements should indeed leave room for reflexive monitoring and should create platforms for reflexivity for policy actors and researchers. Additionally, one may conclude that research cannot be productively organized outside the policy process. This is because, on the one hand, research needs to take into account the dynamics and complexities of policies and use the insights of policy practitioners to develop research designs, and, on the other hand, research results cannot act as the final arbiters of policies but should be seen as just one ingredient – albeit an important one – in the policy-making process.

What then are the prospects for arrangements that create reflexive spaces between researchers and policymakers and allow for reflexive monitoring and the gradual modification of policies? I can best speak here for the Dutch context as this is the one I know best. Within the Netherlands, the prospects are both good and bad. Bad, because as a result of cuts in health care and in health services research, there is an increasing focus on ‘quick wins’ and on the implementation and ‘roll out’ of research results, without much attention to changing contexts. The space for incremental policy implementation with forms of reflexive guidance is therefore limited. Research policy is, moreover, increasingly focused on ‘valorization’ of research; while this means that societal relevance of research is emphasized, the values used are almost exclusively economic. The quality assessment systems of universities, still focussed as much as possible on the production of papers in English language journals (that are hardly read by Dutch policymakers and practitioners), offer little encouragement for action-research and other more engaged approaches to research that are necessary to develop productive relations with policy practices.

Luckily there are also positive developments. Recent years have seen increasing discussion of the position of universities, and research policies, amongst others fuelled by the Science in Transition movement (see http://www.scienceintransition.nl/english, accessed 8 July 2016). The Ministry of Education has moved towards developing a national research agenda that seems to leave room for more practice-informed input (http://www.wetenschapsagenda.nl/?lang=en, assessed 8 July 2016). Interestingly, the failure of many policy initiatives also provides room for reflection. The generally felt crisis in the health care transparency programme is leading to the emergence of new types of accounting for care – see, e.g., the work of the association for long-term care organizations on ‘narrative accountability’. ³⁴ In reimbursement policies, the discussion of ‘appropriate care’ may open up space to discuss the relations between research and care in different ways than has been usual until recently. Many policy fields are showing signs of developing of new ways to organize the relation between research and policy. New organizational arrangements for this relation have been developed, such as ‘Academic collaborative centres’ that allow for new relations between policymakers, practitioners and researchers. ³⁵ The ACCs are partnerships between research organizations, policy agencies and practitioners, stimulating interaction across organizational and epistemological boundaries, and furthering both policy-relevant research and evidence-informed policies. Beginning in public health, these collaborative centres have now spread to all kinds of policy fields, such as youth care, supervision and reimbursement policy. Academic collaboratives, if well embedded, offer potential reflexive spaces for the mediation of policy, practice and research. Their success – or failure – is, however, dependent on the extent to which actors from these different worlds do indeed use these spaces to experiment.