‘The harmony of social theory in evaluation’ – commentary on ‘The art and science of non-evaluation evaluation’

The art and science of non-evaluation evaluation ¹ provides useful insights into the experiences of conducting evaluations of dynamic health care interventions. The author’s thesis is that social science theory is invaluable in understanding ‘what is going on’ when evaluating these interventions and learning implications for policy and practice. Jones chooses to make these points through a critique of realist evaluation (generally referred to in the paper as programme theory evaluation) but we argue that the way realist methodologies are described here rather undermines what the paper tries to achieve. In this commentary we, as realist methodologists, would like to address these misapprehensions. Not least because, as we will show, a better understanding of realist methodologies supports Jones’ central thesis. Like her, we agree that social theory should be part of any explanation of health systems.

Jones reports her growing unease with health services research that ‘thingifys’ methodology. We take this to mean reifying methodology: implementing techniques, strategies or instruments mechanically with little, if any concern for the epistemological justification for their use. Jones cites the recently published RAMESES reporting standards for realist evaluation ² as exemplifying her concern with thingifying methods. Yet, Wong et al. are careful not to fall into this trap. They note that in a realist methodology, programmes are characterised as ‘theory incarnate’. In a realist evaluation, methods are chosen to test and refine these theories. For the moment, it is sufficient to describe these theories as empirically testable about what works, for whom, in what circumstances and why. ³ According to Wong et al., ² realist evaluations should ‘describe and justify the data collection method: which ones were used, why and how they fed into developing supporting, refuting or refining programme theory’. Realist evaluators are expressly encouraged not to apply cookbook methods unthinkingly, but to adopt the most appropriate methods, quantitative, qualitative or mixed methods, to test theory. ‘Thingifying’ methods is rejected in preference for theory informed pluralism.

Characterizing theory in relation to the objects of our enquiry is a key concern in a realist methodology. ⁴ We do not agree that theory in a realist methodology is expressed through and is indeed reducible to, logic models, suggesting linearity, determinism and predictability. On the contrary, and like Jones, realist evaluators accept that health (and any social) programmes are dynamic, multiple, conflicted and contested. This does not mean that logic models never have a place in realist evaluations. But their limitations are recognized. A logic model can be a useful representation of the multiple sources of information needed to understand how a programme works, in particular circumstances, for specified individuals or groups. It can also help evaluators to articulate tentative hypotheses about how a programme is intended to produce change. Logic models, with their neatly drawn arrows, boxes and succinct text, may also be useful as communication tools with a programme’s participants and stakeholders. They can, when used as part of an ongoing discussion with stakeholders, provoke disagreement, discussion and revision, so enabling an articulation of the contested nature of social programmes. When things get complicated, multiple logic models might prove useful to do all these things. As Ebenso et al. ⁵ note, ‘logic models depict linear and simplified relationships between inputs, activities and outputs, or between outputs and outcomes of programmes’. Like Jones, realist methodologists recognize that logic models restrict research outputs to description, rather than analysis.

Another limitation of logic models is that they thingify, to use Jones’ language, the context in which programmes and policies happen. Context is, invariably, described in spatial terms: the programme or policy is happening here. Jones observes that ‘[a] key principle of realist evaluation, in the context of policy research is that the effects of a policy are mediated by context’. But realist methodologies do not treat context as a ‘thing’ in the way Jones proposes. Pawson and Tilley ³ characterize the context as the ‘spatial and institutional locations’ (emphasis added) of social situations. Policies are constitutive of the norms, values and interrelationships found in these locations. More recently, Pawson ⁶ developed the notion of context in realist research further, proposing the 4I’s formulation in which the context may refer to any characteristic of (emphasis in the original):

the individuals who partake in the programme;

Policies, their formulation, interpretation and expression are woven into this relational understanding of context. Contexts are most definitely not limited to location. It is not a description of ‘a thing’ that moderates or mediates programmes and policies. ⁷ In a realist methodology, context is not ‘treated differently to policy’, as Jones suggests. It is recognized to be part of the very social fabric of policy.

Understanding the intimate relationship between context and its expression is captured nicely in Jones’ description of policy as ‘an ethos’. This is the kind of language a realist methodologist might use; a disposition that shapes actions in a health care programme in particular circumstances.

Jones’ thesis is that the social science theory is invaluable in researching health care systems. These theories capture ‘the strategies adopted by actors for asserting identity, maintaining autonomy, extending jurisdiction and promoting particular visions of health care’. These kinds of explanations are necessary because, as Jones observes, 'intentions behind changes are transformed by the process of change'. This too is a contention that realist methodologists agree with. The practical challenge faced by both Jones and realist methodologists is how to include these accounts of agency in their explanations about what is going on in health systems and why. It is, we suggest, the source of the dissatisfaction that Jones so eloquently articulates in The art and science of non-evaluation.

Here, a realist methodology that expressly rejects thingyfied description can help. Earlier in this commentary, we noted that realism is ‘theory incarnate’. We discussed theory in one way, as middle-range bundles of hypotheses to be tested empirically. But realist methodologists think about theory in another way too, as Jones does. Theories express relationships between agency and structures that shape what we can see in the health care programmes we evaluate. Realists contend that we can unpick these relationships empirically. ⁸ Without social theory, we are blind to these powers, these generative mechanism and dispositions. This is the important message in The art and science of non-evaluation evaluation. As we have shown, read more carefully and there is much that Jones and realist methodologists can agree upon. Together these insights, particularly about the importance of theory and its relation to methodology and method, can play their part in better evaluating health care programmes and policies.