Improving Process Evaluations of Health Behavior Interventions

Process Evaluation and Context

Health behavior interventions with multiple components are complex but are also set within complex settings (Shiell, Hawe, & Gold, 2008); understanding this context is critical for the generalizability of study findings. This is especially important when interventions are tailored to suit local conditions or in pragmatic trials, where the roll out of the intervention is locally or programmatically distinct since applications to other contexts may be uncertain. However, context is an underdeveloped aspect of process evaluations of health behavior interventions (Wells, Williams, Treweek, Coyle, & Taylor, 2012). A key issue is the tendency for evaluations to be acontextual, addressing the somewhat closed world of the intervention, particularly where process evaluations are nested within trials. Process is conceptualized as how the trial is implemented and received, and as such attention is paid to internal elements such as retention and fidelity. Context has been addressed to a limited degree in some health behavior process evaluations by including context as one of these elements for analysis (Moore, 2010). Additionally, context is a key consideration for the few health studies that have adopted Pawson and Tilley’s “Realist Evaluation” approach (Pawson & Tilley, 1997) which emphasizes three key elements: context, mechanism, and outcome, where the mechanism operates within a particular context to produce an outcome in a particular configuration (e.g., Byng et al., 2008).

Several perspectives within health behavior research have engaged with context more extensively, rather than simply identifying it as an element of process evaluation. For example, approaches to context may examine the organizational environment in which an intervention is implemented, which may have a significant effect on an intervention (Wells et al., 2012). Theoretical models relevant to organizational factors affecting implementation, such as Diffusion of Innovations and Normalisation Process Theory (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004; Murray et al., 2010), have been applied to health behavior interventions to a limited extent and have been used as frameworks to understand how new ideas or initiatives are taken up by organizations. Socio-ecological perspectives take a broader approach and can be used to conceptualize context in terms of different levels, from the individual through to the family, the community, and the societal levels. They thus draw attention to context in its different scales and, further, may also attempt to explain how different levels are related to each other, for example how local organizations (the community level) and regulatory environments (the societal level) might complement health education for behavior change (the individual level) (Palareti & Berti, 2009; Stokols, Allen, & Bellingham, 1996). However, in practice interventions generally focus on one of these levels rather than linking a context, such as community organizations, with an individual health behavior (Richard, Gauvin, & Raine, 2011) and also tend to focus on the individual rather than family, community, or societal interventions in any case (Golden & Earp, 2012). A number of studies have addressed a wider context by adopting environmental approaches, but these have tended to be limited to physical environments such as access to green spaces (e.g., Charreire et al., 2012).

An additional consideration for context is that it changes and evolves over time (Pawson & Tilley, 2004; Wells et al., 2012) and thus requires a process evaluation that can accommodate dynamics of change. Hawe et al. provide an example of engaging with this issue in a more sophisticated way by theorizing an intervention as a “critical event in the history of a system” (Hawe, Shiell, & Riley, 2009, p.267), rather than a package of activities delivered. One then considers the settings, social networks of those involved and change over time, which form a system into which the intervention is introduced. The outcomes produced are conceptualized as emerging from the reactions produced in the “intervention field” rather than simply as a result of causal processes within an intervention (Virtanen & Uusikylä, 2004).

The section above has summarized various attempts in the health behavior literature to engage with context as a multi-level and dynamic factor in process evaluations. However, the wider social, cultural, and political environment is reflected in this literature to a lesser degree, although this is also a significant context with implications for the processes and outcomes of interventions and is one that social science perspectives can elucidate. Context can extend to economic, historical, and global factors, which have a distinct relevance to poverty and health inequalities through their influence on income, housing and other significant drivers of health (Edwards & Di Ruggiero, 2011; Marmot et al., 2010). In particular, a key challenge for health behavior interventions and evaluations is engagement with social, cultural and political contexts of groups that tend to have poor health, in contrast to the “rational behavior” of the health-seeking individual who often populates health behavior interventions at the conceptual level. Health behavior perspectives have tended to employ health psychology theories that have a bias toward cognitive processes of the individual and that incorporate the social world in a fairly limited way (Burke et al., 2009). This emphasis on “rational action” by autonomous individuals has remained despite the increasing incorporation of other perspectives noted above (Krumeich, Weijts, Reddy, & Meijer-Weitz, 2001). This perspective discounts the notion that “rational behavior” always occurs within a social context and that what is rational in one sociocultural setting may not be in another. Social contexts of low-income groups, elucidated by social science studies, can be starkly different from those of the professional policy maker or researcher and may indicate one reason why existing interventions are ineffective in addressing health inequalities. For example, shorter expected life spans in low-income groups may negate appreciation of long-term smoking risk and therefore interventions based on investing in a healthy older age will have little relevance (Lawlor, Frankel, Shaw, Ebrahim, & Smith, 2003). There are a number of texts that explore lifeworlds affected by deprivation and low incomes (e.g. Abrams, 2002; Ehrenreich, 2001) and others that specifically explore the relationship between sociocultural contexts and health behavior (Krumeich et al., 2001; Parry, Mathers, Laburn-Peart, Orford, & Dalton, 2012); these could be incorporated into process evaluation research to inform elements such as intervention recruitment or acceptability.

Bigger range sociopolitical perspectives from the social science literature may also be helpful, for example, Burke et al. (2009) comment on Bourdieu’s concept of “habitus” and Gidden’s concept of “structuration” and their relevance for understanding the dynamic relationship between social contexts and health behaviors. One theoretical point, more closely aligned to Bourdieu than Giddens, is that social environments such as class hierarchies shape people’s behavior, their sense of what is morally right and their ideas about what is logically coherent. This occurs without the individual necessarily being conscious that they have internalized the historical and social forces that shape their social world. Whitehead (2007) addresses the relationship between theory and interventions by providing a typology of interventions based on different theoretical ideas about health inequality, referring to approaches such as strengthening communities, and improving workplaces. This is not to overstate the scope for small or large scale social theories to explain social problems or how interventions might work: social theories of ‘big’ social problems always have potential for improvement (Chen & Rossi, 1980) and persistent social problems which resist intervention and change have been characterized as “wicked issues” within the social policy community (e.g., Page, 2000). Furthermore, the challenge is to incorporate these types of social science perspectives, which in illuminating context add to conceptual complexity, into approaches that can also accommodate the dynamic processes of interventions discussed by Hawe et al. (2009).

Process Evaluation and Process

Currently published process evaluations in health behavior research tend to address various elements of intervention implementation such as recruitment, fidelity, retention, and acceptability. For example, the Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework categorizes aspects of an intervention at different levels for: reach, efficacy, adoption, implementation, and maintenance (Glasgow, Vogt, & Boles, 1999). This approach to conducting a process evaluation is relatively atomistic, although useful in that it defines certain domains of interest for research activity. What these categories lack as a whole, however, is a sense of a process. It is somewhat implied in the chronological order of RE-AIMs elements, beginning with intervention “reach” and ending with “maintenance” of the intervention but is not explicit. The concept of a “process” indicates a series of linked events and change over time, and the sequential order of elements of the intervention is a core aspect of what the process is. For example, in smoking cessation group interventions, it is not just whether people attend that is important or how acceptable they find the intervention but what steps they progress through to become a non-smoker that are the substance of the intervention and account for its outcomes. This includes what events take place, such as peer support or quitting techniques supplied by the group leader as well as how participants respond to planned or unplanned elements of the intervention. The order of these may matter as well, for example peer support might be important for keeping the participant attending the group sessions long enough to learn sufficient quitting techniques from the leader. It is striking when one compares the list of elements that tend to be employed in health behavior process evaluations to approaches used in the social sciences literature, as the latter very often provide a model, represented by a graphic, which is structured around a conceptual timeline. This is represented by text boxes or similar, connected by arrows, where the arrows show the direction of time. Models start from the beginning of an intervention through to what is theorized to occur during the intervention to what the expected outcomes are (Margoluis, Stem, Salafsky, & Brown, 2009; Zantal-Wiener & Horwood, 2010). Many contain similar elements such as “input,” “output,” “outcome,” and “context” and are usually arranged in a fairly linear way to demonstrate a process or a causal chain (Nesman et al., 2007). Some distinguish between short or intermediate outcomes and longer term outcomes (McLaughlin & Jordan, 1999; Sidani & Sechrest, 1999). Some address different levels such as individual, interpersonal, collective (James & Meezan, 2002) or micro, meso, macro (Virtanen & Uusikylä, 2004). In other respects, these graphical models vary in the amount of detail provided, the exact elements they contain, and the overall style of graphic. Possibly the model most familiar to health behavior researchers is the realist evaluation one (Pawson & Tilley, 2004) that prescribes the three elements “mechanism, context, and outcome” and the configuration between them. Two other types of process models in use are “logic models” and “logical frameworks,” which are used for program design as well as evaluation. Logic models have been promoted by the Kellogg Foundation (Kellogg Foundation, 2004) and the term is used in public health literature to an extent (e.g., Petticrew et al., 2009). Logical frameworks have been used in international development for both monitoring and evaluation. They outline, in a relatively concise and prescriptive grid, the anticipated pathway that an intervention is predicted to take, taking assumptions including features of the external environment into account (e.g., Gasper, 2000). Logic models and logical frameworks, similarly to the other graphics described above, include within them a narrative about a process of change. These various ways of modeling process in the social science literature encapsulate a better grasp of the nature of process since they represent a chain of events that are linked to each other and that lead to intervention outcomes. The “atomistic elements” in health behavior process evaluations mentioned above could be arranged in a linked chain of events—they are not redundant—but do not explain a process since they only describe certain components, often in isolation. Social science-based models of process, by contrast, make explicit the idea of an intervention as introducing change over time and the possibility of explaining the various elements of a process of change through the causal links represented.

Social science process evaluations, as well as addressing process more explicitly, are also more likely to link processes with outcomes and therefore to address causal explanations to a greater degree. Where process evaluation has been adopted in health behavior studies it tends to be a rather separate enterprise, where process evaluation is nested within the main study that largely examines the outcomes of an intervention using quantitative data. Social science models such as those described above outline a chain of events that link input, process and outcomes and in doing so produce a causal account of an intervention. For example an account of social participation in a community project which included accounts of processes such as confidence building among participants would also be linked to an explanation of outcomes such as increased employment levels, with an explicit pathway explaining job-seeking behavior as being supported by greater confidence, which in turn was promoted by the community project: there is a narrative of events that take place, involving the participant and the intervention, in which the outcome is the last event. This representation of a causal chain provides a basis for explaining intervention outcomes in more depth and detail. Models can represent causality in different ways and do not have to be complex to be useful (Rogers, 2000; Virtanen & Uusikylä, 2004). Process in health behavior research reporting, however, tends to be divorced from analysis of outcomes. Demonstrating robust and convincing evidence for causal relationships in research can be difficult and therefore identifying causes through the modeling of processes will not necessarily be straightforward. However, process evaluation models of change processes at least represent a tentative explanation that can be explored further and can facilitate the structuring of future research activity around exploring causal pathways. Models that represent interventions are limited by the fact that they are, by their nature, simplistic and generally linear representations of complex, messy, and reiterative processes (Mackenzie & Blamey, 2005). Interactions between elements of an intervention and feedback loops occur (Davies, 2005) as well as interactions with external factors that make accurate description and modeling of causal chains more difficult. This indicates the need to develop more sophisticated process models which provide a basis for exploring and understanding relatively complex dynamics of change.

Process Evaluation and Theory

All academic disciplines engage with theory; the MRC guidance for complex interventions recommends commencing evaluation with theoretical considerations (Craig et al., 2008a). Theory has a practical use for process evaluation as it explains how an intervention is expected to have an effect (Weiss, 1997a). A single intervention may be explained by several different theories: for example, a weight loss group might be theorized to help people lose weight through weight loss education, social support, competition between members, regular monitoring of progress in weekly weigh-ins or through the role modeling of a group leader who has lost weight and encourages others to do the same. These could be regarded as competing theories or the process evaluator might want to consider two or more as complementary theories. “Programmes are theories incarnate” (Pawson & Tilley, 2004, p.3), although theories may be lurking rather than explicit, superficial rather than extensive, articulated fully in advance or addressed as an afterthought. Theory matters for improving interventions, for building and refining scientific knowledge, for removing inactive or harmful elements, and for providing a rationale for one type of an intervention over another. Theory also has implications for validity and reliability, as this will inform whether the intervention is likely to work elsewhere or for different populations (Michie, Fixsen, Grimshaw, & Eccles, 2009). Theory is needed at the level of social problems, models of programs and interventions, and the interaction of the program with its environment (Chen & Rossi, 1980). Theory in the health behavior literature, apart from a limited discussion of health psychology theory, tends to be more implicit than explicit and is often not discussed in depth when results are reported.

Several relevant social science models of process evaluation address theory more extensively. “Theory-based evaluation” conceptualizes and organizes evaluation based on the theory of how an intervention has an effect. Weiss (1997b) distinguishes between program theory which is defined as what is done to implement an intervention, and implementation theory, which attempts to explain the effect of the intervention on its recipients. A classic sociological text that analyzes the former is Lipsky’s “Street-Level Bureaucracy” which emphasizes the importance of the discretion of frontline public service employees (the street-level bureaucrats) for how public services are delivered (Lipsky, 1980). Tilley further distinguishes between what a program ought to be doing—“supposed to do theory” (STD)—and unexpected processes which he terms “otherwise/also does theory” (OAD) (Tilley, 2004). Theory may be adopted in process evaluations using different models of application of theory (Lipsey & Pollard, 1989) and theory-based approaches can be applied to suit the particular process evaluation (Blamey & Mackenzie, 2007). For example, Dickinson (2006) recommends combining realist evaluation and theory of change approaches within a critical realist framework, and Weitzman, Silver, & Dillman (2002) discuss using theory-based evaluation within a quasi-experimental study. Logic models and logical frameworks do not always explicitly mention theory in conceptualizations of process; however, logic models are often used as a conceptual tool to represent a program theory or theory of change. The Kellogg Foundation Logic Model Development Guide (Kellogg Foundation, 2004) includes a section on theory-based approaches, and several studies have explicitly integrated the two approaches by using logic models to articulate the intervention theory (Fear, 2007; Jordan, 2010; Judge & Bauld, 2001; Nesman et al., 2007; Nilsen, 2007).

Two major subsets of theory-based evaluation are “realist evaluation” and “theory of change.” Realist evaluation was developed by Pawson and Tillley (1997) who challenge the explanatory power of RCTs on the grounds that one needs to explore and understand generative mechanisms at work rather than simply noting which outcomes are correlated with which interventions. Their model, as described earlier, identifies three key elements—context, mechanism, and outcome—and their configuration. The other main subset is “theory of change,” promoted by the Aspen Institute (Anderson, 2005; Connell, Kubisch, Schorr, & Weiss, 1995). Theory of change is distinctive in that it adopts a participatory approach where the theory is “surfaced” and discussed or negotiated between the evaluator and project implementers at the beginning of an intervention; this theory then guides and focuses the evaluation (Hughes & Traynor, 2000; Mackenzie & Blamey, 2005). An intervention to provide additional sports sessions for inactive school children, for example, would be discussed to identify the theory of how the intervention is expected to bring about change; this might be to draw inactive children into sports though the appeal of the camaraderie of teams, in which case the process evaluator would examine social aspects of the sessions. Alternatively, the anticipated mechanism might be to provide extra practice and skills coaching to increase children’s confidence, in which case the process evaluation would concentrate on teaching and skills development. Although theory ought to be developed and articulated prior to an intervention in theory of change approaches, in practice if this has not happened an evaluator may have to uncover it in discussion with implementers (Moore, 2010). A theory may also be constructed by the evaluator rather than implementers (Chen & Rossi, 1980). Mason and Barnes (2007) advocate building theory as an intervention progresses; this could be useful where theory is unclear or where there are many unknown factors at the beginning. Depending on the evaluation approach, multiple theories may emerge from different stakeholders, including the evaluator who may be aware of competing theoretical possibilities (Chen & Rossi, 1980); this could be useful in comparing the utility of different theoretical explanations. There are, then, different ways to apply theory to evaluation activity; the benefit of constructing process evaluations around theory is that the evaluation activity is directed, coordinated, and mapped more closely onto the underlying rationale of the intervention. A process evaluation is thus constructed to answer pertinent research questions rather than comprising a standardized tick-box exercise covering elements such as recruitment and fidelity.

As well as using theory in different ways in process evaluations, the scale of theory is also a consideration. Lipsey distinguishes between “small” program theories which articulate how interventions operate and “big” sociological theory that is usually applied at more abstract or generalized levels (Leviton & Lipsey, 2007); this distinction might be helpful for analysis at different levels of context. Lipsey (1997) further argues that findings from multiple individual intervention evaluations ought to be reviewed together to build more generalized theory about interventions. Realist evaluation also aims to develop “middle range theory” that operates between big theory and particulars of individual interventions, and which can operate across commonalities in intervention approaches to draw lessons from more than one (Pawson & Tilley, 2004). One might, for example, investigate what types of interventions seem to work best for hard-to-reach groups and, after finding that interventions close to where people live tend to have higher recruitment and retention rates for this population, theorize that a key element in their participation is proximity to home, and perhaps develop further theory about the geographies of social participation for these groups. In health behavior interventions, however, this is difficult because there is a tendency for elements and mechanisms of interventions, and their connection to theory, to be under-specified in reporting of results (Michie & Abraham, 2004). The examples from the social science literature presented here demonstrate how models of theory can be adopted in process evaluations and how theory, which if specified, can be used explicitly and flexibly in different types of process evaluations to investigate different aspects and levels of interventions.

Implications for Methodology

Developing process evaluation practice in health behavior research and addressing broader social factors will require engagement with new areas and methodologies (Smith & Petticrew, 2010). Several examples of process evaluations of health behavior interventions cited in this article reflect a positive trajectory of engagement with the social world and with social science evaluation approaches to a degree. Nevertheless, a challenge for the improvements suggested in this paper is the associated methodology required to develop engagement with context, process, and theory further, particularly qualitative methodologies which have much to offer in this area. However, there have traditionally been reservations about methods across the paradigm divide, including in health behavior research. There is a tendency still for biomedical scientists to regard only experimental methods as robust (Albert, Hodges, Regehr, & Lingard, 2008) and therefore to reject the qualitative methods often necessary in process evaluation. Conversely, there have also been reservations about the value of randomized controlled trials (Mackenzie, O’Donnell, Halliday, Sridharan, & Platt, 2010). An alternative, more pragmatic, perspective is acknowledgment of how mixed methods add to explanation and rigor in health research. For example, a qualitative investigation of implementation processes may improve the internal validity of an intervention study. However, qualitative studies are not used in all trials or integrated well with quantitative data (Glenton, Lewin, & Scheel, 2011; Lewin, Glenton, & Oxman, 2009), and if they are carried out are not always reported in process evaluation findings (Moore, 2010).