Commentary on ‘The art and science of non-evaluation evaluation’

In writing this commentary, the first thing I must say is ‘mea culpa’. I have used realist methods, and eliciting programme theories is often a starting point for my studies. You could argue, therefore, that I am one of the targets of this essay, and you might imagine that my comments would mount a defence of this approach. But, of course, I agree with all that Jones has to say. ¹ Single methods, rigidly applied, are limiting. I abhor ‘checklists’ policing methodological purity, and am passionate about the right of qualitative researchers to ‘hang about’ and see what is happening. Social science theory is important, and being alert to and mindful of agendas, politics and power is essential if policy research is to be meaningful. So what else is there to say? Where I think Jones possibly errs is in setting these things up in opposition to one another. If all we did was elicit programme theories and assess how far they had been implemented, then she would be absolutely right in criticizing us. If a realist approach precluded an exploratory and ‘bottom up’ approach, then it should rightly be shunned. However, in order to make her argument I think the author has simplified a complex situation in which there are multiple stakeholders.

There are two over-arching issues missing from this account. The first is research funding. Who pays for the time it takes for researchers to ‘hang about’, and for the large qualitative data sets to be analysed? In England we are fortunate in that we have a wide range of funders to whom we can apply to do research on how the National Health Service (NHS) works. We have an independent Economic and Social Research Council (ESRC) which will fund social science studies whose primary aim is to contribute to social theory. Research funded by the National Institute for Health Research Health Services and Delivery Research programme is a little more instrumental in intent, intended to support research that can ‘make a difference’ to the NHS within a reasonable time period. The Policy Research Programme (PRP) is more ambiguous. Historically ‘owned’ by what is now the Department for Health and Social Care, PRP has funded both research infrastructure – Policy Research Units – and commissioned studies exploring the operation and impact of policy. Commissioned by an independent panel, studies under this workstream have usually had to be signed off by policy makers. This is usually accomplished by civil servants, but for large funding calls ministers may be involved. This policy audience is obviously looking for different things in a study than an independent Research Council, for whom academic contribution may be paramount, with the political impact of potential findings a legitimate concern for a policy audience. So designing a research study must start with considering what the funder wants. What looks credible to ESRC – built upon social science theory, and aiming to contribute in this space – looks obfuscatory to civil servants, whilst an evaluation built upon a programme theory approach might excite an audience of policy makers but look pedestrian to a social scientist sitting on an ESRC funding panel.

The second important issue is the need for research findings to speak to a relevant audience on the ground. In my experience, one of the values of the programme theory approach is that it has face validity with policy makers and with local managers. Asking ‘what were you trying to achieve, and why do you think this set of policies will achieve it?’ is a great way in with managers and policy makers, and, even when the finding is, as Jones so neatly illustrates, that a particular policy has no logic or doesn’t really exist, that is, in itself, an important and interesting finding.

The trick, as far as I can see, is to set up your research so that you can do all these things at once. This isn’t an easy thing to achieve, but it can be done. There are great examples of policy-related research which provide a relatively straightforward and descriptive account for the funders and for other relevant stakeholders, highlighting factors affecting implementation, for example, or demonstrating the conflicts in the relevant programme theories, whilst at the same time generating high-quality social science papers. For example, in a PRP-funded study evaluating a number of pilot new genetic services in the NHS, Martin et al., ² published some relatively straightforward research reports exploring the ‘barriers’ and other organizational issues faced by the pilot sites. At the same time, they worked at a deeper theoretical level, contributing a range of insights relevant far beyond the narrow study focus.^3–5 With perhaps not quite so much success, we carried out a study of the development of a commissioning initiative in the NHS in England, using a programme theory approach to provide insights for policy makers which spoke to their concerns about how to make the initiative work in practice, ⁶ whilst at the same time contributing more theoretically informed papers to social science journals.^7,8

However, Jones does give us pause for thought here. Her argument, I think, is that messy, social science theory-informed research can and should also speak to a policy or NHS audience. This is a challenge, which I think those of us who do policy-related research need to take seriously. Segmenting our audiences, as I have described, is, perhaps, the easy way out. Social science theory is an attempt to explain the world, by identifying recurring patterns in social situations. It allows us to generalize from our local and particular studies. Explaining these theories in a way that makes sense to our funders and participants is something we surely should be able to do, and I am sure our research would be all the better for it.

So in summary I don’t think Jones and I are far apart. Her eloquent essay is a useful reminder that messy, ambiguous, theoretically informed and exploratory social science research has an important role to play in illuminating important policy issues and programmes, and I would wholeheartedly agree that a search for methodological purity will yield diminishing returns. On the other hand I think that policy makers should be at liberty to fund evaluations that speak to their immediate concerns, and an approach which foregrounds programme theories certainly has a role here. The challenge that Jones offers us is to be more ambitious in our commitment to our theory and methods. The core skill of a policy researcher has always been the ability to deliver what funders want and need, whilst at the same time carrying out the sort of messy and exploratory research which yields useful social scientific insights. To this, Jones challenges us to add a third skill: that of explaining social science theories to a wider variety of audiences in ways which make clear their value.