Building Realist Program Theory for Large Complex and Messy Interventions

What Is Already Known?

Methods for developing initial theories in realist research are varied and underreported. Additionally, existing abstract theories are often used to substantiate rather than inform program theory development.

What This Paper Adds?

This paper provides an account of program theory development in a realist evaluation of positive comprehensive youth sexual health services. It offers a rationale for early development of a framework of abstract theories to improve coherence, quality, and transparency in realist research and a set of criteria for selecting abstract theories.

Developing Theory for the Delivery of Positive Youth Sexual Health Services

English and international policy contains an ambition for a positive approach to youth sexual health services, one which prioritizes and promotes young people’s sexual well-being (Faculty of Sexual and Reproductive Healthcare [FSRH], 2015; Great Britain, Department of Health, 2013; World Health Organization [WHO], 2010). However, the dominant model of delivery represents a risk-based, rather than positive approach, focused on treating or preventing sexual ill-health and teenage pregnancy (FSRH, 2015). This is despite support from a wide range of scholars (Patton et al., 2016; Wellings & Johnson, 2013) and advocates for young people (Brook, 2016; Family Planning Association [FPA], 2011) for a positive approach. The WHO (2010) recognizes a need for theory and evidence to support the development of positive, comprehensive youth sexual health services (hereafter positive services). The aim of this research project was therefore to gather evidence and ideas about what works (has worked, could work) to deliver positive services, and for whom, under what circumstances, and why.

Realist Methodology for Investigating Youth Sexual Health Service Design and Delivery

Complex interventions are characterized by multiple parts that interact with each other and the political, historical, social, and geographic contexts in which they are situated to produce outcomes (Clark, 2013b). Youth sexual health services can be described as complex interventions because they cover a range of different issues, for example, prevention and management of sexually transmitted infections, preventing unwanted conceptions and psychosexual concerns, are delivered in a range of settings by a variety of clinically and nonclinically trained staff for all young people with their different needs and experiences. Such complexity needs to be reflected in any research evaluating these interventions. Research studies of sexual health interventions must be designed to consider local contexts; experimental designs alone are not sufficient to understand why certain ideas work, or do not, in particular contexts (Michielsen et al., 2016; Santelli & Schalet, 2009). Several scholars have argued that research approaches, rooted in a realist philosophy of science, may support the accrual of knowledge concerning how complex interventions, such as sexual health services, work (Clark, 2013a; Pawson & Tilley, 1997; Westhorp, 2012). Realist approaches are particularly focused on uncovering causal processes rather than simply outcomes and may be most effective when dealing with issues of complexity, that is, where many causal factors interact. This was the case for the project reported here concerning positive sexual health services, hence the choice of a realist approach.

Two Principles of Realism

A goal of realist research is to explain causal processes. Causation, according to realist philosophy can be attributed to underlying mechanisms which, triggered under particular contextual conditions, lead to the outcomes we are interested in (Bhaskar, 2008). Mechanisms are often hidden, for example, at the level of human reasoning or social interactions and therefore cannot be directly observed (Sayer, 2000). It follows that we need to use other methods to uncover these mechanisms, the contexts in which they are triggered and the outcomes that ensue; Pawson (2013) summarizes this as the context mechanism outcome (CMO) framework. Given that the mechanisms are not directly observable, the search for them is led by the theories about them; in other words, we look for the operation of CMOs in places that the theories about them guide us to look. For example, sexual health services may be placed in a discrete location because it is assumed that possible embarrassment and shame, associated with sexual health issues, might prevent people attending, if the services were highly visible. We cannot see the user’s feelings of shame, affecting their decision-making, nor the cultural conditions contributing to these feelings, but our theories about them would direct us to consider these mechanisms in our data collection.

A second principle is that realist research embraces the idea that complexity is inherent in social systems (Westhorp, 2012). Social interventions are always played out in “open” settings where various contextual features at different social strata, such as individual demographics, interpersonal relationships, and political and economic structures, interact affecting the outcome (Clark, 2013a). This is not necessarily a linear relationship, whereby A leads to B, but more like a web of causal processes which, in combination, generate the outcomes (Sayer, 2000). Realist scholars call this web of causal processes leading to an outcome generative causation (Bhaskar, 2008). One of the aims of realist research is to make explicit the ways in which the various contexts interact and affect the outcomes of an intervention via the triggering or inhibiting of key mechanisms (Pawson & Tilley, 1997). Both qualitative and quantitative methods are legitimate tools for extracting, developing, or testing theories that articulate these ideas (Pawson & Tilley, 1997).

Realist research should therefore, and as stated above, be theory-led and use tools that support the analysis of the complexity inherent in the system.

Program Theories are a Central Aspect of Realist Research

Pawson and Tilley (1997), in setting out a realist approach to evaluation, argue that the “evaluand” (i.e., the thing evaluated) in such studies should not be the program, intervention, or policy itself, such as would be the case in other evaluative methods, for instance, a randomized-controlled trial, but the causal program theory underpinning it. Broadly speaking, this program theory relates to why and how the program brought about the changes observed. There are some differences in the way in which “program theory” has been conceptualized. This is in part due to the fact that such theories can either represent a highly specific causal explanation or a more abstract explanation. Pawson (2010, 2013), for example, uses program theory somewhat interchangeably with middle-range theory, which is at a higher level of abstraction and can be generalized across different contexts. Other scholars make a distinction between program theory and middle-range or grand theories, by which they mean abstract theories that are not attached to a specific context (Davidoff et al., 2015).

For the purpose of this article, we will refer to program theories in the narrower sense concerning how a specific intervention is theorized to lead to a goal (Davidoff et al., 2015; Funnell & Rogers, 2011). However, these program theories are not free-floating; there are relationships between them and the more abstract theories in the middle-range or grand theories (Walker & Avant, 2005), see Figure 1, which gives a visual representation of these relationships. For example, the more abstract theories can be harnessed to guide the development of program theories by highlighting key concepts and relations that might be influential (Westhorp, 2012). In turn, testing program theories, in different contexts, has the potential to refine more abstract theories. Thus, effective program theories may well be rooted in one or more abstract theories (Westhorp, 2012).

OPEN IN VIEWER

Figure 1.

Relationships between grand, middle-range and program theory.

Accordingly, certain aspects of program theories, which are rooted in more abstract theories, will not be unique to individual settings or interventions but may be commonly applied across a wide range of policy areas (Pawson & Tilley, 1997). Examples are given such as “naming and shaming” theories that operate across criminal justice, health care, and education settings among others. The task of the research practitioner is to identify whether, when, how, and why the abstract theory applies in a particular context. This leads to the central question in their seminal work: “what works, for whom, in what circumstances and why?” (Pawson & Tilley, 1997). The outputs from such a study would ideally be well-articulated program theory to support the development of the intervention in context (Davidoff et al., 2015) as well as new or refined abstract theory, most likely in the middle range, which can be generalized to other settings.

Adapting Program Theory Building for Large, Complex, and Messy Interventions

Realist methodology has been applied in a wide range of research studies. Some of these concern interventions which are well defined with distinct boundaries and outcomes against which the project could be evaluated, such as crime reduction programs (Pawson & Tilley, 1997). However, other practitioners have attempted realist evaluations of interventions, including policy reform and system transformation, which are highly complex, large scale, and/or messy (Greenhalgh et al., 2009). The intervention, which is the focus of this article, falls into the latter category because it is looking at system transformation and organizational culture change within publicly funded health services, rather than the discrete addition of a new intervention. In addition, the “program” itself is not a well-defined intervention—more an idea or set of ideas which have been tried, but not in a systematic or uniform way.

Arguably, realist methods need to be adapted to address different research questions (Davis, 2005; Pedersen & Rieper, 2008). In particular, the method for explicating and developing program theory in large, “messy” interventions may pose specific challenges because it is unlikely to be explicitly stated and may be highly convoluted and multistranded. This particular task is now considered in detail below.

Strategies for Building IRPTs

Guidance on conducting realist work suggests that the starting point in realist evaluation and realist synthesis is to develop an initial rough theory or set of program theories, henceforth referred to as IRPTs (Wong, 2015; Wong et al., 2016; Wong, Westhorp, et al., 2013). These IRPTs become the object of the inquiry and the structure and framework for examining and synthesizing diverse evidence (Pawson, Greenhalgh, Harvey, & Walshe, 2005; Rycroft-Malone et al., 2012). As projects progress, the IRPTs are frequently revisited, revised, and refined according to new information as it becomes available until ultimately they can be presented as a refined program theory, albeit fallible and partial (Pawson, 2013).

The RAMESES guidance suggests that IRPTs may be elicited from a number of sources (Wong et al., 2016; Wong, Westhorp, et al., 2013). An exploratory review of the literature suggests that, where initial theory building is reported, different approaches are indeed employed. Some use the program theory that is explicit within the program development documentation. For example, Tolson, McIntosh, Loftus, and Cormie (2007), in their evaluation of delivering Managed Clinical Networks, cite program theory used by the Scottish Executive Health Department. This may be possible when interventions are well defined with clear boundaries but less so with messy interventions.

Where there is no explicit program theory, written in policy or service documents, researchers are required to build it (Pawson, 2013). There are various processes for building IRPTs that can be used singly or in conjunction with one another (Lipsey & Pollard, 1989). Four of the possible strategies, used in realist program theory building, are outlined below:

Building program theory using the latter two strategies, that is, by using data drawn from stakeholders or literature alone, can give rise to problems which we look at below.

Issues Associated with IRPT Development from Data Alone

There are at least three potential issues with data-driven approaches to building IRPTs. First, one may simply rediscover what is already well established in the theoretical literature and not add substantively to our understanding of the concept, for example, that trust between stakeholders leads to better outcomes. Second, it is reported that data-driven approaches generate an overabundance of candidate theories, which can be overwhelming (Pawson, 2013). A third, related problem is the developing theory may be unstructured, that is, not clearly relatable to levels of social strata (e.g., individual, interpersonal, institutional, and infrastructural; Pawson, 2006), and as a result lack coherence as they will not fully acknowledge the role of mechanisms at these different levels, nor explain the patterns that they form. Arguably, these problems are exacerbated in interventions that are large, complex, and less well defined because there are considerably more aspects of the theory which could be explicated.

In the face of this abundance, RAMESES guidance stresses the importance of prioritizing or focusing the research (Wong et al., 2013). Pawson (2013) suggests a number of strategies, including the use of conceptual platforms, cycles of hypothesis selection and shedding, focusing on policy discord, or developing lines of inquiry.

Additionally, in explicating the role of different mechanisms at different levels of social structure, Westhorp (2012, 2013) uses the metaphor of climbing up and down ladders. The ladder rungs refer to different levels of social strata. These may have corresponding layers of theory: micro (relating to individual), meso (relating to interpersonal), and macro (relating to institutional, infrastructural, and cultural).

It has been argued that a sound understanding of a broader, more abstract conceptual framework, incorporating theories that relate to different layers of social structure, may help to overcome each of the three highlighted issues (Westhorp, 2012). It may also direct and frame a more detailed analysis of causal explanation (Westhorp, 2012). The rationale for this is further developed below.

The Case for a Conceptual Framework of Abstract Theories to Inform IRPT Development

It is the central thesis of this article that an initial conceptual framework of abstract theory could be a valuable asset for formative assessments of large, complex, and messy interventions. This framework, if informed by theory at different levels of social strata, may provide a number of benefits. First, it can highlight common features and relations that are likely to play a role in the program theory, and second, it can provide a structure within which to situate more detailed analysis. This marries with Salter and Kothari’s (2014) suggestion that a conceptual framework may, in general, facilitate the identification of important relationships between concepts. The method for developing one such framework and its contribution to IRPT building is outlined below.

Phase 1: Concept Defining

Any program theory is made up of concepts, which define the fundamental characteristics of the program in question (Walker & Avant, 2005). Realist methodology calls for explication of concepts and strives for clarity where they are contested. An essential first step of theory building is therefore to articulate the concepts and shared or contested understanding of the program under review such as “what the program is?”; “who is the supposed target?”; and “what is the supposed outcome?” (Pedersen & Rieper, 2008).

A process of concept mining (Rycroft-Malone et al., 2012) and refining was adopted in this project, not least because the concept of “positive approaches to youth sexual health” has no set definition. This process had a number of stages as detailed below.

Concepts, constructs, and definitions of positive approaches to youth sexual health services were identified through a systematic search of four electronic databases using the search terms “sexual health” in combination with “sex positive,” “young people,” “service,” and other synonyms (for details, see Shearn, Piercy, Allmark, & Hirst, 2016). More specific inclusion and exclusion criteria were then applied to identify articles that related to universal youth sexual health provision, developed countries, written in English. Of 1,162 articles, three services meeting the inclusion criteria were reported in the literature. Reference, citation, and gray literature searches resulted in 25 sources concerning the development and evaluation of these services. Data referring to the overarching aim, outcomes of interest, characteristics, and principles were extracted. These were then synthesized, and principles and characteristics distinguishing positive approaches from other models of care were identified. In brief, these principles were first, an acknowledgement that young people had a sexual identity and rights associated with this; second, a desire to support young people to achieve sexual well-being and recognition that this is influenced by individual, interpersonal, and societal factors; and third, a commitment to place young people’s needs, as opposed to political, professional, or societal needs, at the center of decision-making. We then investigated the extent to which these principles and characteristics were present in current policy, to refine our definition of positive youth sexual health services. This definition was then verified via a group of multiagency practitioners and researchers in sexual health.

The output of this stage was a provisional definition of positive youth sexual health services which described its principles, characteristics, and the organizational outcomes associated with such an approach and initial data regarding how such outcomes might accrue. The process of concept mining, however, alerted us to the fact that many possible interpretations of positive services exist, for example, as a marker of quality overlaying a clinically orientated service or as a reorientation of services to help young people achieve sexual well-being. These contested concepts were therefore incorporated into our initial theories as possible contexts leading to unintended outcomes.

Phase 2: Proposition Development

Selecting existing theory on the basis of explanatory power

Initially, we asked ourselves the question “what is this intervention an example of?” This gave the more abstract, general answer: “the adoption of a new, potentially controversial, model of service delivery.” A purposive search for middle-range theories to support an understanding of what might work to deliver this type of change was undertaken. An initial short list of 15 theories was established by drawing on the work of scholars in the field of sexual health, other realist scholars looking at similar service transformation and our own expertise in psychological and sociological fields.

The short-listed theories were then appraised according to four criteria:

The level within the social system, that is, the extent to which they offered guidance for explaining phenomena at or between micro-, meso-, or macrolevels.

Their potential fit with the aims of the current research project, that is, the extent to which they offered guidance, in this case, for explaining the likely phenomena observed when looking at the transformation of youth sexual health services.

Their simplicity—how readily they inspired theory generation.

Their compatibility with realist notions of causation, that is, the extent to which they offered guidance for articulating underlying causal processes building on Westhorp’s (2012) notion of complexity consistency theory. For example, these theories would address some of the following: the constituent elements of the system, interactions within and between levels of a system, and the properties that may result in one level of the system as a consequence of the interactions at other levels (Westhorp, 2012).

Three theories were selected from the short list which best fit the criteria. Each operated at a different level of social structure. These were the morphogenetic approach (MA; Archer, 1995), normalization process theory (NPT; May & Finch, 2009), and the capability, opportunity, motivation model of behavior change (COM-B; Michie, van Stralen, & West, 2011).

At a macrolevel, Archer’s (1995) MA provides a realist perspective of how structure, culture, and agents interact. Her approach describes the ways in which agents are conditioned by structure and culture to behave or react in certain ways, and hence, our choices are constrained. How agents choose to act then reproduces or transforms the social structures/culture. This theory contributed to our thinking around the overall process of change but also the role of agents within the system and the effect of time and sequencing of events.

At the mesolevel, NPT describes how organizations change to adopt new practices. May and Finch (2009) propose that normalization “work,” by which they mean “what people do,” concerns four broad constructs: (i) coherence—work that defines and organizes objects of a material practice, (ii) cognitive participation—work relating to actors within the system engaging with the change, (iii) collective action—work relating to all parts of the system working toward the same goal, and (iv) reflective monitoring—work relating to assessing patterns of work and outcomes. This theory has clear applicability to the current project aims, which is looking at the adoption of a new model of practice.

At the microlevel, Michie, van Stralen, and West (2011) assessed a wide range of behavior change theories and distilled them to three key factors they suggest are necessary for individual change: COM-B.

The three theories, Archer’s (1995) MA, May and Finch’s (2009) NPT, and Michie et al.’s (2011) COM-B, were assimilated to form an overarching conceptual framework. This framework was used to guide and inspire our program theory development and subsequent data collection and to frame the analysis. Figure 2 below depicts the simplified framework, in which theories are positioned in relation to macro-, meso-, and microlayers of the social system.

OPEN IN VIEWER

Figure 2.

Conceptual framework of theories.

Using the conceptual framework to build theory propositions

The next step was to use the conceptual framework to inspire the development of program theory propositions, for example, to connect concepts that might explain local buy-in to positive approaches. A series of explanations for the underlying causal processes leading toward local buy-in were postulated using the conceptual framework and information from the concept defining stage. Explanations were derived by iteratively hypothesizing about how the step might be achieved based on a theoretical understanding of (i) how it has been achieved in other circumstances (retroduction) and (ii) developing hypotheses on the basis of data which are not explained by current theories (abduction; Oh, 2014). This exercise gave rise to a number of theory propositions. One example is given below. Summarized, in prose, this is that when commissioners, managers, and practitioners, who intend to embed a positive approach in their work, share an understanding of positive principles and characteristics with other local decision makers and pursue the same goals, they will be motivated to work together as this will enhance their chances of success.

Table 1 below illustrates which aspects of the proposition are supported by the conceptual framework and data (gleaned to that point).

OPEN IN VIEWER

Table 1.

Proposition Development and Sources.

Proposition	Data	MA	NPT	COM-B
When commissioners, managers, and practitioners, who intend to embed a positive approach in their work	*	*	*	*
share an understanding of positive principles and characteristics with other local decision makers	*	*	*
and pursue the same goals,	*	*	*
they will be motivated to work together as this will enhance their chances of success		*	*	*

Note. COM-B = capability, opportunity, motivation model of behavior change; MA = morphogenetic approach; NPT = normalization process theory.

Other propositions supporting change relating to “conviction,” “integration with other contextual features,” “consistent policy,” “evidence-based practice,” “devolved decision-making,” “young people’s voice at the center of decision-making,” and those hindering change relating to “tension between practice requirements” and “professional silos,” were also identified and explained using this approach.

Phase 3: Connecting Propositions to Form Theories

The final stage of developing the IRPT was to draw the connections between the theory propositions. As Pawson (2006) notes, it is the combination of attributes, the fact that they happen together in a process over time, which provides the trigger for system transformation.

Figure 3 below illustrates the relationships between some of the theory propositions. Some were viewed as contextual features, representing social phenomena at macro-, meso-, and microlevels. These propositions were thus arranged to illustrate how they might condition local, individual, and group responses. Other propositions were viewed as possible outcomes or causal processes that might be triggered as a result of such responses within a long implementation chain which serves to transform services. Positioning these theories within a web of causation allowed for a rich picture to develop but also highlighted gaps in the overall theory. Hypotheses were formed about the gaps using the abductive and retroductive inferences, in much the same way as the original propositions.

OPEN IN VIEWER

Figure 3.

Connecting propositions to develop initial rough program theories.

The theories are presented as relatively linear and sequential below, for the sake of clarity. It is, however, a representation of a more complicated picture where aspects of the theory compound or conflict with one another and where feedback loops reinforce or reduce their influence. For example, young people’s voices, demanding a positive approach, may be directly heard within the organization through engagement strategies and indirectly heard through increasing practitioners’ conviction in a positive approach.

The resulting IRPT (or collection of theories) can be summarized as follows. In circumstances where there are multiple and competing influences on optimal service design, but some degree of autonomy in local settings, individuals with a clear understanding of positive services (differentiated from other models of care), conviction in their efficacy for reducing youth sexual ill health, and a sufficient degree of influence within the organization may be able to instigate a positive approach by positioning it as the most effective means for reaching mandated service requirements. A model of positive services may be sustained if local agencies share principles and values and work toward common aims and if suitable evidence is collected to support it.

Each of the propositions held within this could be further unlocked and interrogated by asking “when?” “why?” “how?” “who?” and “in what circumstances?” These are the questions that can be posed directly in future data collection initiatives.