Twinning “Practices of Change” With “Theory of Change”

ToC and Advocacy Evaluation

In international development, many planning and evaluation tools are used to gain control over program effectiveness. ToC evolved from the linear logic model, the log frame. The log frame only lists inputs and outputs, without looking at the different elements in relation to each other. In its monitoring agenda, it focuses on intended outcomes, which is problematic for processes where not everything can be planned. Additionally, the log frame does not allow for insight into the processes leading to outcomes or space to incorporate influences from the changing context (Prinsen & Nijhof, 2015). ToC was meant to resolve these problems.

ToC was developed from program theory in the tradition of theory-based evaluation and is concerned with how and why an intervention works (Weiss, 1997, 2000). Program theory studies the interrelation between the mechanisms of change, the program, and the outcomes it intends to achieve. A growing number of studies claim ToC is the best approach for dealing with complex social and political change processes because it emphasizes the links between objectives, strategies, outcomes, and assumptions (Beer & Reed, 2009; Louie & Guthrie, 2007; Stein & Valters, 2012; Vogel, 2012). There is no universally agreed understanding of the exact nature of ToC, although it is often referred to as a road map, theory for action, or blueprint for strategic planning and learning (Reisman et al., 2007). Despite ToC being celebrated and widely used, studies have shown that tensions exist between its ambitions and its implementation, demonstrating that the approach is not yet living up to these ambitions (Eyben et al., 2015; Mowles, 2013; Valters, 2014).

For this reason, ToC has been criticized as both a planning and monitoring and evaluation tool. Mowles (2013, pp. 47–49) found that ToC is predominantly used as an outcome-based approach that evokes cause–effect thinking (i.e., if we do this, then this will be the result). Consequently, this emphasizes the intended outcomes foreseen by policy-relevant theories and plans (Coryn, Noakes, Westine, & Schröter, 2011; Valters, 2014). Moreover, ToC as a monitoring and evaluation approach has triggered highly critical voices questioning whether it is the best approach for complex interventions or rather a new donor-driven results tool.

ToC follows a cause–effect and if–then logic that tends to focus on how the program expects to achieve its intended outcomes (Vogel, 2012). Therefore, it does not pay attention to other outcomes, such as those that are not anticipated or intended. ¹ Van Es and Guijt (2015) illustrated how ToC as an approach, both ex ante and ex post, is limited by the political pressure focusing on intended results, which does not enable a reflective and learning environment. Valters (2014) argued along the same lines, adding that, in the ToC approach, the institutional environments (i.e., bureaucracies, funding agreements, and program planning) and limited resources (i.e., time and funding) mean that accountability trumps learning. Eyben (2015, p. 21) conceived of ToC as results artifact, helping to achieve the politically desired outcomes that must be accounted for. This reality contradicts the ambition of ToC to be a reflective and critical approach. Thus, ToC as an evaluation approach becomes a framework for testing predefined hypotheses as was also confirmed by Bamberger, Tarsilla, and Hesse-Biber (2016).

Barnes, Matka, and Sullivan (2003) stressed that the evaluation of nonlinear processes involving multiple stakeholders and relations demands multiple theories of change. They argued that the contested interpretations of stakeholders, who all have different interests and demands, need to be addressed. Building on this, we consider advocacy a complex intervention because it pursues multilevel change, works in multistakeholder settings, crosses borders, and concerns diverse levels of policy arenas. Rogers’s (2008, p. 39) work helps with the understanding of the outcomes of advocacy interventions as emergent rather than intended or planned. She considers outcomes to emerge through interactions during the process of implementation. Emergent outcomes, including unintended outcomes, can be mistaken for ineffective management, although these outcomes actually point to evolving, strategic and flexible approaches as well as complex processes. If evaluators focus merely on intended outcomes, these emergent outcomes are easily overlooked.

At the same time, multiple advocacy researchers have claimed that ToC is essential and strategically important for understanding the changes advocacy achieves (Jones, 2011; Klugman, 2011; Stachowiak, 2013). In our case study, the program did not articulate a ToC specifically but was assessed on its outcomes using the ToC method. We reconstructed a ToC for that purpose. We noticed serious tensions between the cause–effect logic in the reconstructed ToC approach and the observed practices of doing advocacy (i.e., adapting to changing circumstances, communications, and interactions), including recursive strategizing resulting in emergent outcomes. These tensions included the focus on intended outcomes, with other outcomes remaining concealed. There was no explanation in the ToC for how these unintended outcomes came about. Seeking to do more justice to the dynamic advocacy processes, we therefore suggest twinning theoretical models and theorized relations, as seen in the ToC, with practices of change, which provide space for recursiveness and emergence.

Creating Space for Practices of Change

The idea of ToC adheres to the notion that planned models equal the real world. When plans are made in anticipation of change, there is an implicit understanding that change and the implementation of these plans are driven by policies. This can be overcome by emphasizing and articulating practices in addition to theorizing how change evolves. Strategy as practice offers an analytical lens that can make a difference in the evaluation of complex advocacy interventions.

Breaking down the existing body of knowledge around ToC, we note that “strategy” is an important element. However, ToC is not suited to the explicit integration of strategy as an evolving practice central to change processes but rather theorizes that if we use this strategy, then this will be the outcome. This emphasizes the intended outcomes and places the theory above changing practice. From public health intervention studies, process evaluation is another approach that pursues a shift from outcome focus to processes (Hulscher, Laurant, & Grol, 2003). However, this approach also underscores the notion that planned theoretical models equal the real world while it focuses on intended processes. It does not provide space for understanding practices as strategies and interactions. Inspired by management studies, we turn this around by focusing on practices in addition to program theorizing. By looking at the processes as they happen in practice, we see strategy as something actors do while adapting to changing circumstances.

SAP is a strand of management studies that was developed from the acknowledgment that organizational processes should be understood from practice. This makes both the theorizing and the practices of strategy meaningful. SAP explains how strategy is dynamic, as it is enabled and constrained by actors, human interactions, and organizational and societal practices (Vaara & Whittington, 2012). Jarzabkowski (2005, p. 7) defined strategy as a situated, socially accomplished flow of activity that has consequential outcomes for the direction and/or survival of an organization. She showed that interactions are always pursued while keeping a future mission in mind. This implies that actors act strategically, with a mission (i.e., agenda, interest) that guides their work. At the same time, they must also deal with the daily dilemmas of reconciling a constantly changing world with the need for stability because this is essential for an organization to function effectively and efficiently (Jarzabkowski, 2004, p. 530). Jarzabkowski and other SAP scholars (e.g., van Wessel, van Buuren, & van Woerkum, 2011) have addressed actions and interactions as dynamic and complex, acknowledging that organizations are not necessarily a coherent whole, but rather fragmented, pluralistic, and contested. SAP therefore provides a shift from performance-based approaches (in the sense of a results agenda) to actual understanding of how strategy evolves. Strategy, then, is a type of activity that is dynamic and shaped by practitioners and practices (i.e., routines, tools, methods, discourses, meetings, communication, and behavior).

Using a SAP lens enables the understanding of practices as they evolve, rather than framing them in line with a predefined theoretical model. This provides space for interaction between the theory and the practices. In other fields that bring together theory and practice, it is considered important to focus on strategies to understand how these are used to address challenges and needs, shape interactions, and influence trust building (see Coburn & Penuel, 2016). Feldman and Orlikowski (2011, p. 1249) have argued that there is a need for the development of a practice theory to enable a focus on empirical and microlevel processes constructed through relations. Relatedly, Langley (2007) has shown that bridging microlevel relations and macro environments requires process thinking. Process thinking considers “how and why things—people, organizations, strategies, environments—change, act and evolve over time” (Langley, 2007, p. 2). SAP provides an analytical lens to gain better insight into such processes by zooming in on the practices.

As our case study will show, addressing “how” and “why” questions in advocacy evaluation requires an understanding of strategies as dynamic practices. Focusing on these practices provides insight on the recursiveness of advocacy. Recursiveness means that each decision, direction, or step results from the previous ones and generates new starting conditions for the rest of the process through interaction, learning, and decision-making (Crozier, 2007). This recursiveness demands the continuous reconsidering, redefining, and restrategizing of advocacy strategies through feedback loops of interactions between stakeholders and organizational/environmental contexts. Through these feedback loops, new starting conditions are developed, based on learning and integrating new ways of thinking, decision-making, and strategizing. Crozier (2007, pp. 10–13) has illustrated this point with political communications, which are interpreted through and influenced by actions and interactions. Communication and knowledge are transformed through feedback loops between the sender and the receiver, depending on how the information is received and interpreted. What happens in these feedback loops is thus largely unforeseen (Crozier, 2007). This also implies that outcomes inherent to a recursive process should be considered emergent. Giving space to strategies as practices enables insight into these loops and the relations between strategizing, interactions, and outcomes.

To date, recursiveness has not been given prominence in advocacy or in studies discussing ToC as planning, monitoring, and evaluation approach. A limited number studies have mentioned recursiveness in relation to international development interventions. These studies have considered recursiveness in the context of nonlinearity, multiple pathways of change, multidimensionality and emergence. Patton (1997, p. 232) discussed recursiveness as an element of multidirectional and multilateral relationships. For advocacy, we argue that these relationships, as practices, are key to the strategies. We therefore consider recursiveness fundamental. Looking at advocacy interventions, Rogers (2008) argued that strategies need to be revisited continuously, as practices to pursue change are recursively generated through feedback loops that create new starting conditions with each output.

Whereas ToC focuses on theorizing change from policies, SAP emphasizes that change is initiated from practices and then should find its way back into policy. Rather than seeing processes as linear or causally linked, SAP provides space for understanding practices as recursive and their outcomes as emergent. In the next sections, we present our case study findings, which demonstrate the importance of focusing on practices to grasp the meaning of outcomes. We show how advocacy outcomes came about through recursive strategies as practices, rather than as planned change processes, as was an initial assumption in our ToC methodology. This justifies using SAP as an analytical lens for understanding the practices of change and illustrates that advocacy outcomes are emergent, not necessarily either intended or unintended.

Method and Case Study

This study was part of a broader research project based on a transnational advocacy evaluation (2012–2015) commissioned by the Netherlands Ministry of Foreign Affairs in cooperation with the Foundation for Joint Evaluations. The evaluation was administered by the Netherlands Organisation for Scientific Research. We were involved as evaluators and researchers. The evaluation assessed eight transnational advocacy programs that were funded by the Netherlands Ministry of Foreign Affairs and implemented by a variety of Dutch nongovernmental organizations (NGOs) in cooperation with partners worldwide. For this article, we drew from one of the eight advocacy program. This program was the advocacy component of a broader child rights program. It was executed by an alliance of Dutch NGOs but implemented solely by an international NGO (INGO) in Ethiopia that focused on Pan-African child rights. The Dutch alliance referred to this INGO as a “strategic” partner.

Our research was a multisited ethnographic study of this Pan-African child rights advocacy program, which we studied through time and space (see Marcus, 1995). We gathered data using diverse methods (see Arensman et al., 2015, for full details). We conducted 147 semistructured interviews and held formal and informal meetings with staff and management, program managers, and the board of the Ethiopia-based INGO. We interviewed the Dutch alliance partners and staff members of targeted organizations, including African policy makers and other African partners. We also analyzed more than 200 documents, including documents internal to the organization and external public and policy documents. We observed five strategic advocacy events in 2012, 2013, and 2014, and we gathered feedback during formal and informal meetings with staff members, individual program managers, and with the organizations’ management. We conducted participant observation and outcome tracing in the Netherlands, Ethiopia, Kenya, Mozambique, Uganda, Namibia, South Africa, and the United States (New York), but time and space limitations did not allow a survey of the full Pan-African scope of the advocacy program. We were unable to visit Francophone or Maghreb countries or Central or Western African countries. Therefore, the data cannot be generalized or considered representative of the Pan-African scope of the program.

We invested in trust building while remaining independent outsiders to the program. This approach gave us the opportunity to work closely with the advocacy program’s staff, enabling us to improve our understanding by gaining access to privileged information. The longitudinal nature of the evaluation made it possible to monitor how information and stakeholders developed, thus acquiring progressive insight. For confidentiality reasons, the names of interviewees are not given in this article. The next section describes the case in more detail.

Case Study Findings

Theory and Practices of Change

In the evaluation methodology, ToC was at center stage. When we arrived in Ethiopia in 2012, we were thus harnessed with concepts including “pathways of change,” “outcomes,” and “outputs” and with tools such as contribution analysis. However, we decided to approach the program with an open mind, enabling reflection on our preliminary findings from our document analysis.

Our preliminary study of documents and interviews with the Dutch alliance in the Netherlands had not surfaced the ToC ACPF was pursuing. The Dutch alliance partner had stressed that ACPF was mainly doing research. What advocacy or outreach was being undertaken in daily practices was not explicitly known. We therefore first tried to reconstruct the ToC and intended outcomes ourselves (a paper exercise), but interviews and documentation provided by ACPF communicated another reality as to how change was pursued and achieved in daily practices. Noting this, we questioned staff members about how they had bridged the multiple realities. Jointly, we tried again to retrieve the ToC in order to evaluate how it had worked out in practice. We understood that the notion of ToC had been introduced to ACPF by the Dutch partner. This was elaborated in a joint ToC workshop that was organized in 2012 with the goal of inspiring improved effectiveness. The staff within ACPF had accepted ToC as a donor-given tool, but they had not explicitly used it in their way of working. The ToC as defined in the workshop was not really an issue of discussion with the Dutch funding agent.

Although ACPF did not have a clearly spelled-out ToC, they did think about and reflect on what they pursued and how they went about this. One of the program managers emphasized this point:

We do not have an overall theoretically written ToC; we have a ToC in practice, because we have internal discussions and you are always asked by the team to account for what you are doing and for the choices made, for example, on the justification of the project or the choices made. […] At our organisational level, we have a strategy document titled From Era of Rhetoric to an Era of Accountability. That gives us an overarching direction. (Program manager, 2012)

The staff members thus implicitly emphasized the practice side of their work over the theory, explaining how they continually and recursively worked and strategized in practice. This meant that their advocacy interventions were experienced as evolving in practice rather than in a structured on a theoretical basis or institutionalized in the organizational way of working.

We also found that ACPF was, in theory, predominantly targeting its actions toward the AU, African governments, and NGOs. Figure 1 demonstrates the theory through which they aspired to pursue change. Their reports, strategic documents, and interviews framed this as direct and explicit advocacy toward these three targets. They often stressed how they were influencing governments:

We constantly organise launches, which we see as advocacy events due to our identification of the country-specific issues. In country briefs, we report where a country is lagging. We have different countries to deal with, and we have local partners who take it up. Local and international bodies take it up. (Program manager, 2012)

OPEN IN VIEWER

Figure 1.

African Child Policy Forum (ACPF) theory of change (Arensman et al., 2015).

This interview extract illustrates how staff members spoke about their output and facilitative role as self-evidently influencing change. They had not distinguished between the immediate output of their work and the effect it had on their targets (outcome), a point that will be further discussed in the following section. Content-driven advocacy as a strategy was often discussed as a self-explanatory cycle that naturally trickles down to results and influence: “We are about advocacy, because whatever we produce has to have an effect, a result. We advocate for those results” (Program manager, 2012).

In ACPF’s annual monitoring reports, they spoke mainly of program outputs, which they often presented as outcomes. We started to look beyond the output and, together with the staff, reviewed the narratives on what happened in practice to reach their targets and achieve advocacy results (outcomes). In these narratives, we found not only additional outcomes but also strategies other than those that had been outlined in theory. Their advocacy was taking shape in a recursive way in much more strategic and emergent forms.

In addition, it appeared that governments were not a direct target for ACPF. Rather, they were targeted through the AU. Further, the country-specific activities were limited. Thus, we concluded that, in reality, ACPF followed another pathway toward change, targeting, and cooperating with different stakeholders (Figure 2). Comparing Figure 2 with Figure 1 demonstrates the difference between the theory (Figure 1) and the practice of change (Figure 2) pursued by the organization. This practice was not reflected in any theoretical model. We found that the organization’s staff members were not fully aware of the influence they wielded. Consequently, many of the outcomes they achieved were not seen, reported or followed up. In other cases, more outcomes could have been achieved if they had realized what they had set into motion with their output. In one of the countries, for example, a workshop was conducted to introduce one of the child rights reports. This workshop was highly appreciated by the participants. However, the report (recommending strengthening child rights) was never translated into the national language, and it was scarcely distributed; thus, its outreach remained limited. The next section discusses how this advocacy work evolved and developed in an emergent way.

OPEN IN VIEWER

Figure 2.

African Child Policy Forum (ACPF) practice of change (Arensman et al., 2015).

Practices of Change

ToC, in its current form, is used with a focus on control and predictability. This focus is at odds with the need and ambition to provide space for understanding how processes develop and evolve in reality. Our case study demonstrates the need to appreciate practices in order to understand effectiveness. Our findings prove that advocacy is recursive and emergent rather than linear and causal. In their strategic decision-making, advocates interact, react, and adapt. When organizations focus on what they intend to achieve (ToC), often in relation to donors’ agendas and accountability demands, they overlook what is actually achieved in practice. We suggest refining the use of existing ToCs to understand change as initiated from practices in which human interactions are central: practices of change. This is not another model; rather, it exposes the theoretical model to the real world of practices.

Theory-based approaches to evaluation such as ToC are valuable because they construct (ex ante) oversight and insight into the envisaged processes of change pursued by a program. Critically questioning assumptions and beliefs encourages program staff to reflect on their roles, missions, agendas, and strategies. Questioning helps to pave the way for examining the human interactions and relations. However, although it is valuable, ToC does not clarify human interactions in everyday practices. We found that the theoretical ideas about change differed from what we identified in practice, and we learned that ToC as an evaluation approach has two interrelated shortcomings. First, although it describes how and why an intervention works in theory, it overlooks practice. This results in the second shortcoming: ToC thus fails to pay attention to outcomes that were not intended. What actually happens in practice (emergent and recursive) can therefore easily be overlooked (see also Jabeen, 2016). To help us evaluate the programs, we felt theory was one thing, but we also needed to understand practices.

The complex and unpredictable nature of advocacy demands that evaluators investigate both theory and practices. This requires a framework that provides room for both. According to Jarzabkowski (2005, p. 172), “a framework is only valuable in relation to the conditions under which it applies.” As our case study shows, change may be strategized in theory, but it actually happens in the interactions between actors, influenced by organizational environments, policy arenas, and social and cultural contexts. This multilevel playing field is shaped by theories, practices, and the interactions between the two. To understand advocacy outcomes as they are, rather than seeking to assess outcomes against a predefined theoretical framework, evaluators must examine how strategy develops in practice.

What does this mean for evaluation practice? Strategy in ToC is one theorized element (if we do this, then this happens), but it becomes central when looking at the practices—not as if–then logic, but rather as how-did-it-evolve logic. SAP foregrounds human interactions and is a way to look at advocacy processes and understand how advocacy strategy develops through daily practices. It is important to explore how such strategy is shaped over time, across multiple levels, stakeholders, borders, and perspectives. Understanding strategy as something actors do rather than something organizations have gives room to practices while also making it possible to relate practice with theory (ToC). This means that it takes human interactions and processes—rather than theoretical, predefined outcomes—as a starting point for understanding effectiveness. For evaluation, looking at the practices of change through the lens of SAP and recursiveness provides a new perspective for understanding the outcomes of complex processes, such as advocacy. This includes looking at outcomes as emergent, beyond intent.

By twinning practices of change and ToC in the way evaluators and organizations work, we believe that change, even if it happens over a long time period—which is often the case in advocacy—can be captured in terms of how it actually evolves. Twinning practices of change and ToC thus provides evaluators and program staff with the space to follow and trace advocates’ progress over time (including small steps, practices changed, strategies adapted, and achievements), reflect on the practices together, and establish plausible connections between these practices and changes (even if these changes occur over a long time period). This approach thus encourages advocates to more openly record their steps as they evolve, develop, and flow in their day to day practices.

In advocacy evaluation, we argue that understanding practices of change demands more than questioning and theorizing processes as a sequence of events that will lead to a desired outcome (Vogel, 2012) or testing and proving the theorized causal hypothesis (Mowles, 2013). This clearly has implications for the role and position of evaluators, who need to look beyond theory to observe, investigate, and explore practices. Doing so will require a shift from cause–effect thinking, which emphasizes outcome planning and reporting, to putting advocates at the center. Advocates act strategically while they make practical judgments, maneuvering through changing circumstances and acting in interaction with contexts (both organizational and environmental) and theory (plans and objectives). These practices need to be understood and reflected upon in terms of strategies, recursiveness, and human interactions. Only then can an evaluator assess the broader scope of a program’s achievements.

Besides establishing common ground, evaluators should take an open, qualitative approach to collecting data. Evaluators should listen well and without prejudice, and they should critically question and investigate the strategic practices. Narratives around strategies as dynamic practices become key in establishing how change processes are shaped and developed; what human interactions are meaningful; and how this relates to diverse roles, perspectives, theories, and achievements. This means that close cooperation with program staff is necessary to create space and build the trust required for transparency (i.e., open discussions, learning, and joint reflection). This demands fostering moral courage to tell an honest story, even addressing those things that failed. To optimize reflection and learning, it is advisable to start evaluation of practices during the implementation.

For evaluation purposes, we consider practices of change a necessary complement to “ToC.” Additionally, practices can be used to mirror theory. This may provide plausible explanations for how outcomes (practice) relate to policy (theory) and its program planning. The tensions between theory and practice are a well-documented field of concern in development practices (see Eyben et al., 2015; Wallace, 2006).

Conclusions

The above discussion demonstrates that ToC is potentially suitable as an approach for planning dynamic advocacy interventions, but we argue that, for evaluation purposes, ToC should be twinned with practices of change. Whereas ToC focuses on theorizing change from policies, the idea of practices of change emphasizes that change is initiated from strategies as practices shaped by human interactions. This approach provides a practical solution to the problem posed by Teles and Schmitt (2011) who argued that current evaluation approaches are limited in terms of evaluating advocacy and suggested evaluating the advocates instead of the outcomes. The kind of twinning we suggest makes the case that theory and practice cannot be separated in evaluation.

Our approach provides the space necessary to elucidate processes that are unpredictable in nature (not predicted in any ToC), encouraging evaluators to look beyond the pursued outcomes. We argue for emphasizing practices as a mirror to theories, challenging the traditions of prediction and control. Understanding the correlation between policy (theory) and outcomes (practice) clearly requires evaluators to explore and analyze connections between theories and practices. Theorizing how change works provides insight into the beliefs, assumptions, and ideas about change, whereas exploring how change is pursued and achieved in practice provides insight into interactions, strategies, and decision-making.

Therefore, we argue that the theory-based approach (ToC) should be twinned with the practice-based approach. The combination of both approaches provides useful insights into the complex processes between policy and implementation that can bridge the gap between the world of theory and the world of practices. This combined approach also provides space to take distance from and reflect on theory while doing justice to practices. New theories can be constructed on the basis of understanding practices. This should then find its way back into practice.

Our case study specifically focused on advocacy evaluation, but our findings may be relevant beyond advocacy to the broader field of working with and evaluating complex interventions that continuously evolve over time. We realize that evaluation and recording (documenting) of Practices of Change in advocacy requires further study and methodology development, acknowledging advocacy as strategic in interactions and as form of practical judgment. There is a wide body of evaluation methodologies available to draw upon also in other sectors, such as the education field. Two of the authors are also involved in developing and designing a methodology specifically for advocacy evaluation that takes a practice approach, looking at advocacy as strategic in interactions and as a form of practical judgment. We encourage more case studies to be done to accompany (advocacy) evaluations enhancing understanding of outcomes. But also to contribute to elaborating a practice theory of evaluating complex interventions such as advocacy. Enhanced understanding and analyses of practices should also contribute to policy improvement.