Abstract
This article argues for the importance of theory and theorizing for an evaluation in the form of a process theory of change. A process theory of change centers its theoretical attention on key episodes that explain how things worked, in which the causal linkages are unpacked. The key lies in answering why actors do what they do (and thus whether these actions can be traced back to the intervention). This theorization has three steps: (1) definition of intervention and potential contribution; (2) theorization of potential contribution pathways; and (3) unpacking the process. This procedure is illustrated with a hypothetical example.
Keywords
Introduction
Theory-based evaluation methods (TBEMs) hold the promise of enabling analysis of how and why interventions produce real-world contributions. This is achieved through the theorization of the causal linkage(s) between interventions and contributions that are evidenced empirically in a case study. TBEMs include realist evaluation (RE), contribution analysis (CA), and process-tracing methods (Lemire et al., 2020).
While RE and CA are very useful for assessing the activities and outputs/outcomes of interventions, the conceptual language they provide for theorizing what links them together does not enable the evaluator to unpack the sequence of actions and interactions between program actors and relevant stakeholders that produced a contribution (Rothgang and Lageman, 2021; Schmitt and Beach, 2015). Yet if the inner workings of an intervention are not unpacked theoretically, it is difficult for evaluators to claim that they have empirical evidence of whether and how an intervention produced a contribution.
Process-tracing methods offer a theoretical language that enables evaluators to develop more granular theorizations of actual processes of change through unpacking the interactions between project activities and those who are affected by them using what can be called process theories of change (pToCs) (Cartwright, 2021: 13110). A pToC in effect answers the “how does it work question” for an intervention. However, in the literature, there is little practical guidance for evaluators for how good pToCs can be theorized. The contribution of this article is therefore to provide step-by-step guidance for developing more granular pToCs that can be used in practice in different real-world evaluation settings. The article focuses on the theoretical side of process tracing in terms of what is being traced and how it differs from the types of theories used in RE and CA approaches. The article does not discuss the methods for evidencing a pToC empirically (for more on this, see, e.g., Beach and Pedersen, 2019; Befani, 2021).
More granular theorization in the form of a pToC 1 has two main analytical benefits in evaluations. First, by empirically tracing a pToC linking an intervention and contribution together, more credible evidence-based contribution claims can be made (Schmitt, 2020: 12). A pToC enables evaluators to engage in a more robust dialogue between the process theory and empirics to figure out whether and how an intervention actually produced a contribution. Empirical material is only evidence of a causal process when a pToC is used that helps us assess what empirical material is actually evidence of. Second, an empirically validated pToC sheds more light on how an intervention produced a contribution (i.e., how it actually worked), thereby providing actionable knowledge that can both help improve implementation in the case in hand (if on-going), and inspire program design and implementation in other cases if contextual differences are taking into account (Schmitt, 2020: 12; Sridharan and Nakaima, 2012: 385).
A good pToC should avoid the extremes of either drowning in detail or simplifying so much that the workings of the intervention remain in a black box. An excessively detailed pToC is not tractable as an evaluation method because it would require evidencing (almost) every action and interaction by every actor during each moment of the implementation period of the program instead of focusing the analysis on the key episodes seen from a causal perspective. Even if it could be evidenced, a very detailed pToC would not provide a clear overview of how a contribution was produced. On the contrary, an oversimplified pToC does not provide the theoretical scaffolding required for making credible evidence-based contribution claims, as it black boxes the key linkages that we want to evidence empirically.
A good pToC is therefore a theoretical model that captures the key episodes of interaction and causal linkage between different actors and activities that together produce a contribution. Key episodes are defined as the actions and interactions between program actors and stakeholders that together overcome challenges and barriers that otherwise stand in the way of the achievement of the intended results of the program. For example, one challenge in an intervention aimed at using data to influence policy would be getting policymakers to notice the data in the first place. Therefore, a pToC of the data-policy influence process would unpack the interactions between program actors and policymakers that could (in theory) have made them notice the data. Without overcoming this barrier, the process would have stalled, and no amount of policy dialogue based on the data would have succeeded if policymakers were not aware of the data and its implications.
The article proceeds in three steps. In the next section, we discuss the theoretical questions related to processes that are left unanswered in both RE and CA, as they typically do not unpack sequences of action and interaction that explain how they were able to produce the contribution. Then, we introduce the conceptual language drawn from process-tracing methods, in which actors, actions, and linkages are used to develop more granular pToCs. We then develop a step-by-step guide for theorizing pToCs.
Existing TBEMs
TBEMs are used to evaluate empirically the causal arrow(s) linking interventions and results (i.e., contributions). We contend that while RE and CA approaches have many strengths when used to understand different aspects of a program’s logic and the results produced, in practice, they tend to model processes of change as static one-offs because of the conceptual language they adopt. Sequences of interactions between program actors and other stakeholders are not unpacked theoretically because they conceptualize what happens after program activities are performed either as causal link assumptions or as micro-level mechanisms. Process tracing and the theoretical extensions to the approach that we develop in this article provide a friendly amendment to both approaches by developing a conceptual language that allows the evaluator to put forward testable hypotheses about the actions and interactions during the implementation of an intervention that unpacks how the contribution might have been produced.
RE seeks to answer the question of “(w)hat works for whom, in what circumstances and in what respects, and how?” (Pawson and Tilley, 2004: 2). In RE, the theoretical focus is on explaining why micro-level actors (aka individuals) react as they do in response to interventions in a particular context (Pawson and Tilley, 1997: 75–77). Activities and the resources, opportunities, and constraints from the context trigger actors to react based on “mechanisms” that are the underlying preferences, reasoning, and motivations that trigger decisions and choices that produce a given behavioral outcome (Aston, 2020). Realist theories of change (ToCs) focus on conceptualizing configurations of context (C), mechanisms (M), and the outcomes they are hypothesized to produce (O).
Figure 1 depicts an example of a ToC from an actual RE of a program that sought to explain “how, when and/or why the MomConnect program messages trigger the expected results” (Kabongo et al., 2020). The ToC depicts four different theorized mechanisms that are hypothesized to link the intervention activities and outcomes/impact through a combination of empowerment of mothers, encouragement, increased motivation, and awareness of services. The ToC does not unpack the interactions between program actors and mothers nor the sequencing of how the mechanisms would play out, instead treating it as a one-off set of activities and responses.
A realist evaluation theory of change for MomConnect program.
CA uses ToCs as models that describe the links between inputs and activities of a project and the outputs, outcomes, and impacts, focusing on making clear the assumptions underlying postulated causal linkages in the form of a results chain or COM-B ToC (e.g., Mayne, 2017a, 2019); 2 The core of CA is to claim an intervention made a contribution by searching for empirical evidence that can confirm the ToC of an intervention, that is, that the impact pathways, the causal links, and related causal narratives, as well as the assumptions behind the causal links, are present, and thus causality can be inferred from this evidence (Mayne, 2019: 173).
Using an example from Mayne (2015) of an intervention involving training mothers in nutrition, a CA ToC is depicted in Figure 2. If the proper contextual conditions are present, the desired outcome is assumed to occur. The linkages between project activities and outputs/outcomes and results are treated as causal linkage assumptions. The ToC does not unpack the process whereby mothers acquire knowledge and what the teacher is doing to impart the lessons, nor what they were doing when they adopt new feeding practices at home.
A nutrition intervention theory of change.
Despite their many merits, for evaluations that intend to understand theoretically how an intervention worked and evidence it empirically, both approaches tend to produce static ToC models because the temporal dimension of the interactions is not unpacked theoretically (Aston, 2020; de Souza, 2022: 73; Wauters and Beach, 2018). Instead, they tend to depict activities followed by outcomes/results/impacts without unpacking the sequence of interactions going on in the arrow drawn between the two.
Realists treat the arrows in an additive sense that when coupled with context and mechanisms produces outcomes. CA views the arrows as causal link assumptions that are “events and conditions that have to occur for the causal link to work” (Mayne, 2015: 125, 126), resulting in a type of if-then theorization of the process. As a result, ToC in both approaches tend to be one-sided theories that focus on the delivery of planned activities and the outcomes/results/impacts that follow, thereby downplaying the relational character of change processes.
However, real-world project interventions are typically not activities performed at a particular moment in time but instead are a series of interactions with other actors (targeted groups and other stakeholders) over a period of time during which a program is implemented. For example, there might be a series of interactions between the teacher and mothers that are focused on building a trust relationship before mothers are asked to try to implement lessons at home. Without first building trust, advice by the teacher might be ignored, resulting in no behavioral change. In addition, mothers implementing lessons at home in ways that actually result in better child nutrition might not be one-off actions. They might have been produced through a process in which the teacher might have provided a series of lessons supporting mothers as they experiment with what works for them at home. Assuming that mothers are reached, the material is appropriate, that they are motivated, and so on, the ToC does tell us what factors must be present for the intervention to work. However, it does not shed much light on what actors (teachers, mothers, and other stakeholders) are actually doing in the implementation process.
If an evaluation intends to understand how the program actually produced the results (i.e., “how it works”), this requires unpacking the arrows into episodes of interaction between actors. The lack of theorization of interactions between actors also implies that what is going on in-between program activities and results/outcomes are typically not evidenced empirically (see Rothgang and Lageman, 2021: 533; Schmitt and Beach, 2015: 431). This would not be problematic for impact evaluation methods such as randomized controlled trials (RCTs) because the controlled comparison of results/outcomes in cases where activities were present and those where they were absent enables causal inferences to be made. However, not unpacking the interactions within the causal arrows is more problematic in theory-based evaluation case-study methods. Here it can be tempting to claim that a sequential order of activities, output, and outcome implies that there is a causal relationship between them. However, this would be a flawed inference as we would not have actual empirical evidence of linkages as they were not traced empirically. Using a metaphor, a ToC-based evaluation that does not unpack interactions is similar to a series of still-shot pictures over time (t0, t1, . . ., tn) in which we should not infer that the picture at t1 is caused by what is depicted at t0; only that there is temporal correlation between the pictures.
To be able to understand change processes requires unpacking in more granular detail who did what during the process and why as a sequence of interactions. A pToC can be used to unpack processes of change theoretically into episodes of interaction, thereby also enabling them to be evidenced empirically in an evaluation. By making this an explicit requirement, evaluators using process-tracing methods switch from treating linkages as assumptions or unobservable mechanisms to trying to theorize them in ways that enable empirical assessment. 3
In the next section, we build on the existing political science/policy studies literature on process tracing for the conceptual language for unpacking causal processes (e.g., Beach and Pedersen, 2019). 4 Given that the existing process-tracing literature lacks guidance for what elements of a causal process should be focused on, we develop the language of key episodes. Furthermore, there are conceptual elements from both RE and CA theorization that can improve existing process-tracing guidance for theorization. This includes how realists treat “mechanisms” underlying behavior, and the emphasis on context in both approaches.
A practical step-by-step guide to theorizing a pToC
The three steps of theorizing a pToC are as follows:
Defining the intervention and potential contribution;
Identifying potential contribution pathways and challenges/barriers in the process; and
Unpacking the pathway(s) into key episodes in a more granular pToC.
While they are depicted sequentially, in practice, earlier steps are often returned to when the evaluation finds that the original pToC did not function as originally theorized. For instance, in unpacking and tracing empirically a pToC, we might become aware that the originally defined intervention was too vague in relation to the more limited activities that triggered a process of change. Or the empirical analysis might find that the expected contribution did not occur in a case, which should then lead us to return to Step 1 to redefine the contribution (if there was one at all).
We use a running hypothetical example to illustrate the different steps in the procedure, inspired by evaluations of several existing vaccination programs. In the example, the program’s desired outcome is to increase immunization rates among children under 5 years old against vaccine-preventable diseases in several different countries in Africa. Program activities include increasing vaccine access and availability by supplying more doses, setting up camps in hard-to-reach areas, and different awareness-raising activities. Parallel to this running example, we illustrate the benefits of this level of granularity by drawing from existing evaluation literature using process tracing.
Before we walk through the steps in theorizing, it is first important to discuss the elements that constitute a “good” pToC.
What constitutes a “good” pToC?
A pToC is a granular, step-by-step theoretical depiction that attempts to unpack the inner workings of a program by detailing who did what and why for each key episode in the process linking an intervention with a contribution. Contributions are defined as the outcomes or results produced due to the intervention.
In a pToC, the conceptual language of actors and actions is used to conceptualize the interactions that constitute a process of change (Schmitt and Beach, 2015; Wauters and Beach, 2018). In simple terms, this means theorizing who is doing what. However, only theorizing what actors are doing is not enough to understand why the actions of one actor led other actors to do things. To do so requires that we supplement the existing language of process tracing with what Cartwright and Hardie (2012) term causal principles, defined as reasons why a given action might plausibly lead another actor to do something (p. 22). When theorizing how individuals respond to actions, causal principles can be similar to what realists term “mechanisms,” in which the motivations and perceptions of individual actors are critical to understanding why they responded in a particular way. However, realists tend to focus on micro-level (aka individuals) motivations and reasoning, meaning they lack language to capture interactions between meso-level actors (e.g., groups and organizations) (Aston, 2020). Depending on the process being evaluated, there can be other types of causal principles than motivations and perceptions, especially when theorizing processes involving meso-level actors interacting with each other. Returning to the better nutrition example, there might be group-level social-psychological causal principles such as skepticism based on in/out-group dynamics that are triggered when they meet the teacher. Different types of social science theories will be relevant for theorizing causal principles, ranging from cognitive or social-psychological to theories drawn from policy studies and public administration, depending on what is being evaluated. Note that, for each part of a pToC, different actors, actions, and principles underlying them will be theorized.
Theorizing both actions and the causal principles linking them together is not merely for theoretical understanding. By making actors, actions, and linkages clear theoretically, they can also be empirically evidenced (Raimondo, 2020: 49). If we want to be able to make causal inferences, we need to make the causal principles that provide linkages explicit in a way that can be evidenced empirically, since “well-developed and testable ToCs are a prerequisite for effective process-tracing” (Aston and Wadeson, 2023: 9). Even though we might not be able to evidence a causal principle providing the linkage directly, making it explicit can help us search for indirect (aka circumstantial) evidence that provides at least some confirmation that things worked for the reasons theorized. Furthermore, a good pToC can help us rule out a contribution if it quickly becomes evident that there is no evidence, direct or indirect, of a linkage having operated.
As realists and contribution analysts have argued, context is important for the operation of a process of change (Blamey and Mackenzie, 2007; Cartwright and Hardie, 2012: 45; Mayne, 2012; Woolcock, 2022). The context for a pToC is similar to what CA depicts as causal link assumptions as “events and conditions that have to occur for the causal link to work” (Mayne, 2015: 125, 126) or what Cartwright and Hardie (2012) term “support factors,” that is, contextual conditions that have to be present for a linkage to work.
A pToC can be visualized in abstract terms as in Figure 3, where each part is composed of an actor (who), action (what), and causal principle (why) formulated using the clause “because. . .,” along with the contextual conditions that are expected to enable the pToC to work as theorized.
An abstract process theory of change.
A good pToC is not a blow-by-blow account of every event between an intervention and outcome, as this level of granularity would be unrealistic to evaluate empirically. In this sense, a good pToC is always an analytical simplification that focuses on the key episodes of interaction of the process, enabling us to understand how an intervention produced a contribution. Key episodes are defined as a discrete set of interactions that describe how a particular challenge or barrier in a process was overcome. 5 If there is only one challenge or barrier blocking the achievement of an intended result, overcoming it is synonymous with the intended result. For example, a common challenge faced by nongovernmental organizations (NGOs) advocating for policy change is that policymakers might not recognize that a given problem even exists. Here a set of interactions between an NGO and policymakers might lead them to recognize the problem, which would then be the contribution produced by the advocacy activities of the NGO. More typically, there will be multiple challenges/barriers in a process. If there is more than one challenge, overcoming it will result in an intermediate contribution.
Returning to the better child nutrition example, there might have been two challenges that had to be overcome to produce better child nutrition: (1) lack of trust of the teacher, meaning that mothers would not follow the recommendations of the teacher, and (2) difficulties in implementing lessons properly in practice. A pToC of the program would therefore focus on unpacking the interactions between the teacher and mothers that first built trust (an intermediate contribution of the process) and then detail the interactions in which the teacher supported the implementation of lessons.
When unpacking key episodes, they must be detailed enough that it can be wrong empirically. Furthermore, the degree of granularity—that is, the number of key episodes included—depends on the purposes of an evaluation and the complexity of the intervention. If actionable knowledge about how an intervention produced a contribution is sought, a more detailed pToC with more episodes that are unpacked in more steps would be required, whereas a very simplified pToC focusing only on one key episode can be enough to evaluate whether there is any evidence linking an intervention and contribution, or if we want to compare how interventions worked across different cases. When dealing with very complex interventions, it might be necessary to theorize many episodes and even multiple parallel pathways that together produced the results/outcomes.
A pToC can shift between different levels of actor (i.e., micro-level interactions between individuals and meso-level interactions between collective actors). For instance, in an evaluation on the work of the International Institute for Environment and Development (IIED) to ensure that the Least Developing Countries’ (LDC’s) position regarding climate change was reflected in the Paris Agreement, the actors were both collective (the LDC’s) and individuals such as the chair of the group (Monzani, 2018: 13). A pToC in this evaluation would theorize how the IIED was able to interact repeatedly with both the LDC as a group and the chair individually (and how an interaction with one might have impacted the other’s actions) in a sequence of key episodes leading to the contribution of having the LDC’s position in the final agreement. Other evaluations might focus on pToCs working at two different levels and theorize them separately if those actors do not interact in a way relevant to the contribution. For example, a study on gender-responsive budgeting theorizes a pathway with collective actors, citizen groups, and another pathway with technocrats, each with their own specific logic, but where they together had an impact on the provision of maternal health care (Bamanyaki and Holvoet, 2016).
Taken together, a “good” pToC is defined by three elements:
Actors (who), actions (what), and causal principles (why) that capture the interactions that result in challenges/barriers to change being overcome;
The context in which the pToC is expected to operate is made explicit; and
The pToC is as simple as possible, focused on one or more key episodes.
We now turn to the three steps in theorizing a good pToC for an evaluation.
Defining interventions and potential contribution
Theorizing a pToC first requires that the intervention and contribution are clearly defined. When working with a pToC, we suggest using the term contribution to refer to the outcomes/results of the intervention in light of the challenges and barriers that existed. Without challenges/barriers (e.g., if policymakers were already aware of the problems in an issue), the intervention would not be required to produce the contribution of overcoming the challenges/barriers because the intended contribution already existed. A pToC is a theorization of how the challenges and barriers were overcome.
While defining intervention/contribution can seem very common sense, it can be difficult to develop a good pToC linking vaguely defined interventions and/or unrealistic contributions. It is therefore vital that the intervention must be something that could plausibly have produced a given contribution, and the contribution should be something that could plausibly have resulted from the program being implemented.
Contributions can be identified either by starting with program documents or through desk research and talks with program staff to identify a plausible contribution. The actual contribution of an intervention might not be what was detailed in program documents. Many program plans include relatively unrealistic desired outcomes because they have been developed using approaches such as logframes which do not involve unpacking the causal process linking activities with the outcome. In a logframe, a real-world problem is identified together with its causes. One or more causes are then chosen as the target of an intervention. The desired outcome (solution) of the intervention is then defined as the inverse of the causes. This can result in unrealistic contributions being included in program documents because they are decoupled from a good theory for what challenges/barriers existed, and how the intervention could plausibly have overcome these barriers to produce the contribution. Furthermore, the desired contribution might not have been what was actually achieved in the selected case.
Relatively simple interventions might only trigger one pathway with a distinct contribution, whereas more complex programs working over an extended period of time can trigger multiple pathways that are not necessarily mutually exclusive, and that can either lead to the same contribution (equifinality) or result in different contributions (multifinality). Given that process tracing is an analytical demanding evaluation method, when engaging in evaluations of complex interventions, it will typically not be possible to trace in-depth all the interventions and contributions they produced. Therefore, evaluators and commissioning agents will have to choose one or a small number of particularly interesting potential contributions from specific aspects of a program for process tracing, with the rest of the program evaluated using other methods. Process tracing is especially relevant when trying to evidence the contribution of more intangible interventions like expert advice or capacity-building, and/or when actionable lessons are sought based on either studying how success was produced or how to avoid failure.
The choice of intervention(s) and contribution can be influenced by the purposes of the evaluation. If the evaluation is interested in drawing best-practice-type lessons, the focus will be on identifying one or more cases where the desired outcome actually manifested itself. In this type of “success” case, it cannot be assumed that the intervention defined by the program was what actually produced the desired outcome (i.e., contribution). 6 Here the evaluator should try to reason backwards from the contribution to identify what intervention might in theory actually have produced it. In other instances, a “representative” case might be selected to understand how a typical contribution across a series of cases sheds light on how the program actually works across many cases (Beach and Pedersen, 2019: 96–102; Raimondo, 2023: 10–12). A “failure” (aka deviant) case might be traced in which the intervention pToC broke down to learn about what went wrong and why (Beach and Pedersen, 2019: 96–102).
In defining the intervention to be traced, the first question is whether it is a single intervention at one point in time or a set of interactions between actors over an extended period of time? Were there multiple interventions taking place in parallel? Desk research of the program documents can help map the specific activities. Evaluators should also check with program staff and other relevant stakeholders to understand both the scope of the project/program and whether it was implemented as designed. In many situations, program documents are vague as to the specifics, and implementing partners and organizations might have adapted the activities to better suit the context and the local needs.
When determining what contribution might have been produced, it is also important to explore whether there might be other, competing explanations outside of the scope of project activities that could also account for the contribution. When a contribution could have been produced either by project activities or outside factors, they are rivals that should be tested against each other to see which better accounts for the evidence found. In other situations, project activities and other factors might act in complementary ways, especially when dealing with complex, major changes. Here the evaluation should try to identify the aspects that were produced by the intervention, thereby narrowing the definition of the contribution. In the case of the IIED evaluation discussed above, other agencies also supported the LDC Group, but the evaluation also found evidence that the assistance provided by IIED made a distinct contribution on its own (Monzani, 2018: 18).
A useful first step can be to engage in initial fieldwork to map the different actors in the program and try to follow a “typical” beneficiary through the program. Whom did they interact with? In our example, we might have followed a mother going to a vaccination center, explored whom she interacted with, and probed which project actors interact with her and what is going on. Through this initial fieldwork, we might have found that the intended result (increased immunization rates) was (still) not present. Because mothers were very reluctant to have their children vaccinated, program activities had to be re-directed toward overcoming this hesitancy—meaning that the contribution to be assessed would be re-defined.
Theorizing potential contribution pathways to overcome challenges and barriers
Before a more granular pToC can be developed, it can be useful to identify simplified causal explanations for how specified challenges/barriers can have been overcome. This can take the form of simplified “contribution pathways” that capture the core causal dynamic. A contribution pathway can be thought of as a “one-liner” that summarizes what actors were doing to overcome a given barrier/challenge. For any given barrier/challenge, there can in theory be multiple pathways that might link them. Returning to the child nutrition example, the skepticism of mothers might be overcome through trust-building or through exemplification of the efficacy of lessons.
Ideas about potential challenges/barriers and contribution pathways can be drawn from existing theoretical literature in the social sciences on the topic or from repositories of evaluative evidence in the gray literature. If there is not good existing material, challenges/barriers and contribution pathways can also be developed using a theoretical brainstorm focused on speculating about what might hypothetically be blocking or hindering an intended result from being produced and what might plausibly be used to overcome it. For example, in an evaluation of how indicators might shape policy, a critical challenge might be in overcoming the opposition of certain actors to change. This could potentially be overcome using either building public support for change, elite shaming, or by leveraging transnational pressures (te Lintelo et al., 2020: 1316). After assessing which pathway is most plausible in a given context, one or more pathways can be used as the starting point for theorizing more detailed pToCs.
Contribution pathways should be selected based on whether they can plausibly overcome a given challenge/barrier in the context of the intervention. At this stage of the evaluation, it can be helpful to cast our net widely, trying to identify challenges and multiple plausible pathways in the literature. These can be summarized in a “contribution pathway tree” table (see Table 1), after which our evaluation would choose to focus on the most plausible pathway, at least as a starting point.
A contribution pathway tree of links between an intervention on vaccination awareness and reducing vaccine hesitancy.
In our running example, we identified reducing vaccine hesitancy as the contribution that might have resulted from program activities. Given that the program could not provide material incentives, the intervention instead had to use other tools to overcome reduced hesitancy. Based on a literature review on vaccine hesitancy and how to overcome it (Cooper et al., 2018; Hussain et al., 2022; Peretti-Watel et al., 2015, 2019; Simas and Larson, 2021), there are several distinct pathways whereby information coupled with other types of activities might produce a contribution. Based on this review and what we otherwise know of the context of the intervention, we identified three plausible contribution pathways: (1) community engagement, (2) tailored communication, or (3) interpersonal dialogue using local healers or other trusted local authorities as intermediaries (Enria et al., 2021; Hussain et al., 2022; Simas and Larson, 2021), as depicted in Table 1. Each of these contribution pathways would involve different sets of interactions, with, for instance, community engagement working through a series of longer-term interactions with different groups, whereas tailored communication would focus on the production and delivery of messages by program staff that might resonate locally.
The contribution pathway tree should also include the contextual conditions that could plausibly impact whether and how the contribution pathway works. For example, initial conversations with project staff might indicate that community engagement might work very differently in urban and rural environments. Or a review of the literature on community engagement might lead us to conclude that membership stability of the “community” is a key contextual condition for this pathway to work, with membership of poorer urban communities in the region tending to fluctuate more over time, making it less plausible it would work in an urban setting. Finally, discussion with stakeholders and additional desk reviews might tell us that in the given context, traditional healers are trusted in large sectors of the rural population, meaning that they might have acted as intermediaries (Enria et al., 2021; Madamombe, 2006).
Beyond using a contribution pathway tree to help identify plausible theories from the literature, the tree can also be used after an evaluation as a tool for categorizing what the evaluation findings tell us about a broader class of interventions/contributions. Findings from an evaluation of a community engagement pathway in an urban environment would not tell us much about similar interventions in rural contexts, nor would they tell us about how tailored communication works (or does not work). However, we could compare an evaluation of how community engagement in an urban environment worked with a case in a similar context, helping to develop cumulative, context-sensitive knowledge about how interventions work. Given the promise of enabling more cumulative knowledge about how things work from evaluations, we suggest further research into how contribution pathway trees can be used.
Unpacking a pToC
Once the key challenges/barriers have been identified and one or more plausible contribution pathways have been selected, each of the key episodes needs to be unpacked into a more granular, step-by-step process of change that explains how we move, causally, from the original situation to the expected intermediate or final contribution due to the intervention. That is, we need to unpack the interactions that led to the challenge or barrier being overcome.
When there are multiple plausible contribution pathways in a key episode, the evaluation can either select only one or proceed with multiple pathways in parallel. In a gender-responsive budgeting study, the authors assessed two paths that led from the intervention to the enhancement of maternal health service delivery, one operating through district-level technocrats and female councilors and another working through mobilized citizen groups (Bamanyaki and Holvoet, 2016), illustrating that more than one pathway can link one intervention with one contribution simultaneously.
Unpacking a contribution pathway in a key episode into a pToC can involve a combination of: (1) cataloging who did what and when through initial fieldwork; (2) seeking inspiration in empirical examples from existing evaluation studies; and (3) logical brainstorming. In practice, the formulation of a pToC is often a back-and-forth between empirics and theory. Beginning with hunches about a possible pToC—based on either logical brainstorms or case knowledge—the evaluator would then engage in a preliminary plausibility-probe round of initial fieldwork. At this stage, the pToC will be incomplete, and there might be many unknown actions/causal principles for parts of episodes. Putting these hunches and question marks into the preliminary pToC flags for the evaluator everything that is not known, and that therefore should be the focus of empirical probing asking “how did it work” in fieldwork. Initial hunches are often wrong, which should result in an updating process in which the theorized pToC is modified to match what was found. If possible, it is recommended to conduct explorative fieldwork to assess the initial pToC and, based on these findings, revise and assess it more systematically in a second round of fieldwork.
First, through our initial fieldwork, we can identify what program actors and those they interact with were doing and in what sequence, along with some intermediate results for each plausible contribution pathway. A second tool that can be used is reviewing existing empirical literature on similar programs of the intervention/contribution (Raimondo, 2020: 49). In some instances, an existing evaluation might include enough empirical details that it is possible to develop good ideas about what might also have happened in the case in hand. For both procedures, the result is a descriptive “what happened” story that helps identify the most central actors and the actions in a key episode that should be included in the pToC. However, it is not yet a pToC. Describing what actors are doing does not explain why the actions of one actor are causally linked to the actions of another actor in response. However, they act as important bricks and mortar for unpacking key episodes in a pToC.
Based on this, the evaluator can then start brainstorming about potential sequences of actors/actions and potential causal principles linking action with reactions by other actors. To help identify potential causal principles, it can be helpful to engage in a far-reaching review of social science literature relevant to the particular pathway. For instance, the descriptive “what happened” from both initial fieldwork and similar evaluations might have identified healthcare providers meeting with traditional healers several times before launching a public health campaign or a local health center, or that women in rural areas in Africa went to gynecological exams accompanied by a traditional healer who had previously met several times with the hospital staff. The form of the interaction between health care providers and traditional healers suggests that there might be some form of trust-building going on, suggesting that we look toward social-psychological literature on the subject for inspiration for what is actually going on in a causal sense in the trust-building interactions.
Returning to our running example, Figure 4 provides an illustration of a simplified descriptive cataloging of what happened for each of the three identified contribution pathways that might have been able to overcome vaccine hesitancy.
A descriptive “what happened” for three vaccine hesitancy pathways.
Another useful tool for theorizing a pToC is logical brainstorming, using existing knowledge (initial fieldwork, other evaluations, and theories from the broader literature) as inspiration. This can be done by either forward gaming from the intervention to a contribution or gaming backward from a contribution to the process that might have produced it, or a combination of both. One particularly useful tool is to use what Cartwright and Hardie (2012) term a “pre-mortem,” where we imagine that the intervention did not work and we game through what types of barriers or challenges might it have faced that resulted in it not working. This can help us identify key episodes that we have to include in our pToC (p. 94).
When theorizing a pToC, it is important not to include everything that occurred. Indeed, as using process tracing tends to be resource-intensive, a pToC should only focus on one or a small number of key episodes, even for the most complex interventions. This raises the question of when can we stop adding episodes and details? As discussed earlier, a good pToC has enough detail that it sheds light on how interventions work, and it can be wrong empirically. However, excessive detail means that the pToC would not be able to be evidenced empirically because it is way too complex. Evaluators will need to consider how feasible it is to evaluate a pToC with the resources they have available (Aston and Wadeson, 2023). Focusing the evaluation on the most important challenges/barriers and the interesting parts of the intervention/contribution pathway used to overcome them is the way to avoid drowning in excessive details. A pToC will never be a perfect representation of what actually happened; it will always be a simplification of reality. However, a pToC should be detailed enough that we gain a better understanding of how an intervention works in the real world.
Focusing on challenges and barriers when theorizing a process can also help when producing a more abstract pToC because they enable us to think about what functions actors and their actions play in relation to overcoming the barrier. This enables us to develop abstract pToC that in theory might work in functionally equivalent ways in other cases. As regards actors, instead of using formal nouns to describe the actual actors in a case, we should describe them in more abstract terms based on their role. For instance, instead of using formal nouns like “governor Cruz,” we could use more abstract terms such as political leader in a sub-unit of a country, which would mean that it is a role that might be played by different actors in other cases, but with equivalent causal functions in overcoming a given challenge in the pToC. Similarly, actions can be defined very specifically (actor provides three in-person capacity-building workshops) or more broadly (any form of action that provides capacity-building functions to another actor) in relation to what it is going to overcome the challenge/barrier.
In practical terms, unpacking a pToC can be done using post-it notes for each key episode (actor, action, and principle that explains why a previous action triggered an action). Notes can then be removed or added as needed, with the pToC being re-arranged through further refinements.
Returning to our running example, after initial fieldwork, we might have found that the interpersonal dialogue contribution pathway is of most interest because it is not only the most surprising in relation to what we would otherwise expect but also the most promising in terms of results. After discussion with the evaluation commissioner, a decision could be taken to focus on understanding how this particular contribution pathway worked.
The next step would be to translate the descriptive “what happened” into a pToC that unpacks how challenges/barriers are overcome. Here there is only one key episode, focused on vaccine hesitancy. We might know from discussions with local staff and existing literature, so unpacking the next episode in which local healers who had been brought on board were able to convince mothers is not very interesting because mothers have such a high degree of trust in local healers. The most interesting part of the causal story is how local healers were brought on board. Furthermore, it might be the area where the evaluation findings make the largest contribution to our knowledge because little research had been conducted on how an outside group could build a trust relationship with local healers and then enlist them as allies in a subsequent healthcare intervention. Furthermore, our logical brainstorming suggested that, without this trust-building episode, the process would not have worked, suggesting that we should focus on unpacking this episode further in our pToC.
It can be important to have attempted to unpack to some degree what you believe are key episodes before conducting a second round of more systematic fieldwork. Without an explicit pToC, you are hoping that the “facts” will speak for themselves, which they seldom do. Attempting to unpack an episode before returning to the field sheds light on both what we think we know but that needs to be critically assessed empirically, and everything we do not know that needs to be explored empirically. A good preliminary pToC is specific enough in detailing who, what, and why for key episodes that it can be wrong empirically. In our running example, before the second round of systematic fieldwork, we would want to disaggregate the key episode of “alliance-building” into a set of plausible interactions between local staff and local healers and how they plausibly could have led to the local healers becoming an ally to the vaccination program.
In Figure 5, this key episode is unpacked. In the pToC for this episode, actors are engaging in actions and the causal principles linking actions together in a particular context are made explicit.
A detailed process theory of change for the interpersonal dialogue pathway.
The pToC details the interactions that lead local healers to change their attitudes toward vaccination and the causal principles driving the linkages. It is hypothesized to work because local healers maintain their legitimate place in the community and enhance it vis-à-vis the local project team, as they have signaled that they need them for their own health interventions. The “alliance-building” happens through repeated interactions, where local healers gradually start trusting the local project team because they perceive the team looks at them as a peer, and thus they feel they can conduct a dialogue on equal terms, which makes them more receptive to listening to the benefits of vaccination. They become convinced over time during a series of meetings that allow trust to be built. We can see the difference in this depiction from what we had in the previous figure, where this was contained in the single “local healers acquire knowledge and feel seen and accepted as legitimate” line. Although the step was necessary for the contribution pathway to work, it does not provide us with a step-by-step theory that we can then evidence, as we would not know how the local healers would acquire the knowledge nor why they felt seen and accepted.
Finally, although this key episode in the pToC has been unpacked, actors and actions are still defined in relatively abstract terms. Although no cases are going to be identical, they might play out in functionally equivalent ways in different cases. Local healers might be traditional healers in one case but midwives or other types of more traditional local health authorities in other cases. A local project team in one context might require nurses to gain trust because of their knowledge of also dealing with the psychology of patients, whereas in others, it might be a doctor who might be more likely to gain the trust of local healers.
Discussion
The aim of this article was to provide a step-by-step framework for theorizing pToCs that evaluators can use when engaging in process tracing in policy evaluation. First, we showed why ToC developed using RE and CA methods tends to result in static models that do not unpack the interactions between actors that produce contributions. We then provided suggestions for how to theorize a more granular pToC that captures the links between actors and actions in ways that can be evidenced empirically. The procedure has three steps: (1) defining the intervention, challenges and barriers, and potential contribution; (2) identifying potential contribution pathways to overcome challenges and barriers; and (3) unpacking the process into a more granular pToC focused on key episodes. The steps can be repeated as more information is uncovered during the evaluation.
A good pToC should enable us to evaluate empirically whether an intervention produced a contribution, and if so, how it did so for one or more key episodes. Since it explicitly theorizes actors (who), actions (what), and causal principles (why) that link the parts of the process together, a pToC allows the evaluator to understand how change actually happened (if at all). Due to the level of empirical scrutiny that understanding linkages requires, a pToC for evaluation purposes should focus on the key episodes of how the intervention made a particular contribution.
Although there are evaluations that have used process-tracing methods, the method is still a relatively new variant of TBEMs (Aston and Wadeson, 2023; Lemire et al., 2020). While there are several good examples of process tracing that unpack ToC to some degree (e.g., Bamanyaki and Holvoet, 2016; Schmitt and Beach, 2015; World Bank, 2023), the causal principles binding actions together have typically not been made explicit enough to enable empirical evidencing of linkages. Yet to understand how contributions are actually produced, evaluators should strive to unpack both actions and the causal principles that are theorized to trigger action by other actors.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Independent Research Fund, Denmark (grant number 0133-00115B).
