Dealing with complexity through actor-focused planning,monitoring and evaluation (PME)

Abstract

This article reports on the results of a collaborative action research project (2010–12) in which 10 development organizations (nine Dutch and one Belgian), together with their southern partners, explored different actor-focused Planning, Monitoring and Evaluation (PME) approaches with the aim of dealing more effectively with complex processes of social change. A major challenge that organizations were trying to address during this action research pertained to the demonstration of observable results in complex contexts where such results are not always easy to measure or to quantify and where causal links between cause and effect cannot always be predicted. Drawing from recent literature, the article presents an analytic framework to assess the effectiveness of a PME approach in dealing with complex social change. This framework is then used to explore how actor-focused PME approaches can help international development programmes to manage complex processes of social change by stimulating processes of real-time results-based learning.

Keywords

action research actor focused complexity monitoring results-based learning

Introduction

Over the last decade, there has been a growing international call for results-based management, whereby development actors are asked to be accountable for and demonstrate achievement of ‘measurable’ results (Paris Declaration – OECD, 2005; Accra Agenda for Action – OECD, 2008; High-Level Forum on Aid Effectiveness in Busan – OECD, 2011). However, after more than two decades of implementing a results agenda, critical information and learning needs of donors and policymakers, as well as those of southern stakeholders and the change agents responsible for the implementation of aid programmes, are still not always met (Vähämäki et al., 2011). Recent Planning, Monitoring and Evaluation (PME) literature also points out that actors in the field are faced by the limited relevance of dominant results-based PME approaches that follow a logic of linearity, predictability and control when dealing with complex processes of social change (Eyben, 2006; Hummelbrunner, 2010; Jones, 2011; Mowles, 2010; Ramalingam and Jones, 2008; Stern et al., 2012).

Results-based management can also be approached from a more complexity-oriented worldview. A complexity perspective accepts that the relation between cause and effect in complex change processes is unpredictable and comes with a high level of uncertainty (Ling, 2012) and emergent outcomes (Rogers, 2008). While the use of linear logic models such as the logframe remains widespread within the international development sector (Davies, 2005; OECD/DAC, 2009), a rich variety of more complexity-oriented PME approaches has been developed and implemented over the years across a wide variety of development programmes and contexts. For the field of impact evaluation, various approaches have recently been documented and discussed by Stern et al. (2012); for monitoring and evaluation, see the work of the Oversees Development Institute (Jones, 2011) and Ling (2012). Some authors have connected complexity-oriented evaluation practice with the field of real-time evaluation, which supports reflexive learning and informed adaptation (Ling, 2012). This argument has been forwarded in the context of humanitarian work, where real-time evaluation is high on the agenda, but can also be extended to other complex social development programmes. Such a perspective acknowledges that in complex environments the dividing lines between monitoring activities and evaluation work tend to blur, especially in those cases where regular monitoring cycles end up providing important spaces for much needed evaluative thinking to review programme strategies in the light of quickly changing operating environments. This may also require a stronger involvement of programme stakeholders in data collection and reflection and learning from the monitoring information as such involvement was shown to be crucial for building evaluative knowledge (Cook, 2006; Marra, 2004) and negotiated understanding about how a programme contributes to long-run changes (Callaghan, 2008).

An important gap in the body of knowledge of the evaluation community, and the broader PME field as a whole, in this area is the lack of evidence on what kind of PME approaches and PME practice actually support organizations to deal with complexity. There are few systematic longitudinal studies which document the experiences of development organizations with PME as a tool to navigate complexity (Jones, 2011; Van Ongevalle et al., 2012). We could also not identify a specific theoretical framework which allows researchers and practitioners to unpack the relationship between complexity and PME. And while the evaluation field has developed fast and has been theorized extensively over the last few decades, for monitoring there is ‘no such diversity or depth or theoretical articulation, not such an elaborate body of literature’ (Guijt, 2008: 28). This lack of knowledge is especially problematic in view of the returning observation that due to climate change, globalization, and hyper-connectivity it is very likely that social development will be confronted with more shocks and complexity (World Economic Forum, 2012).

This article seeks to contribute towards addressing this knowledge gap. We first discuss the implications of complexity for PME practice and present an analytic framework for assessing to what extent a PME approach is complexity-oriented. We then apply this framework as we report on the results of a collaborative action research project (2010–12) during which 10 organizations (nine Dutch and one Belgian), together with their southern partners, explored different actor-focused PME approaches with the aim of dealing more effectively with complex processes of social change. These approaches included outcome mapping (OM), most significant change (MSC), client satisfaction tools (CSI), Sensemaker and ‘personal goal’ exercises. In line with principles of real-time evaluation, most cases are focusing on the planning and monitoring part of their PME practice.

Implications of complexity for PME

What distinguishes complex change from other types of change? Over the years, a growing body of literature has identified characteristics of environments and processes which make social development become non-linear and unpredictable (Crawford, 2004; Crawford and Pollack, 2004; Forss et al., 2011; Rogers, 2008). In this article, we argue that working in complex environments has important implications for PME approaches and practices, which we cluster around the following four main areas: dealing with multi-stakeholder situations; learning from unexpected and intangible results; adaptive capacity; and accountability. These also constitute the theoretical framework that guides this action research.

Dealing with multiple actors, perspectives and relationships

The non-linear and unpredictable nature of complex change is often compounded by the influence of many different actors attempting to deal with social issues. This is unavoidable: as no single actor or organization has the capacity to solve complex problems (Jones, 2011). In addition, these actors often have different understandings of and perspectives on the same issue. What is a problem for one may not be a problem for someone else; or the problem might be understood in different ways. Also, different forms of interactions can exist between the actors involved in a social-change process. Collaboration, negotiation, dialogue, influencing, lobbying and conflict are just a few examples of such interactions. Guijt (2010) refers to ‘messy partnerships’ to describe this interplay of different actors. A messy partnership is a convergence of different actors for concerted action driven by a common overarching vision and a perception of value added by collaboration. The various differences that can exist between the actors – such as governance structure, culture, mandate, capacities, priorities and commitment to collective efforts – are the basis of this messiness (Guijt, 2008). Programmes that support complex change are often built around multiple actors that may hold different understandings of the programme’s objectives, how to achieve these, and what the roles and responsibilities are of each of these actors (Jones, 2011). Resources, such as knowledge, financing and networking are dispersed among various actors, that in addition are interdependent (Klijn and Koppenjan, 2000; Van Kersbergen and Van Waarden, 2004). Jones (2011) formulates this as ‘the capacities to tackle complex problems are often distributed across a range of (interdependent) players’. There is not one single actor who has control over a programme’s progress towards a particular objective, and action may require different forms of interaction with a variety of actors (Booth, 2012). Hence, PME approaches and practices will only be relevant to the extent that they provide information about the ‘messy’ day-to-day social interactions between different programme stakeholders and facilitate and support these interactions.

Learning from unexpected and intangible results

Development organizations face a challenging paradox. On the one hand, they set themselves ambitious poverty-reduction objectives which unavoidably come with all the risks, uncertainty and conflicts associated with such high-level objectives (Eyben, 2006); on the other hand, development organizations are often pressured by a results-based agenda to claim that their solutions are certain and can be achieved without risk of failure. Such a paradox can lead to a fear of failure, which in turn can deprive organizations of the ability to learn and understand through experimentation (Snowden and Boone, 2007). However, when dealing with processes of complex change, such learning is essential because in unpredictable and non-linear contexts, it is not useful to predict outcomes and then try to control the implementation of predetermined plans to achieve these outcomes (Marra, 2011: 328).

Opening up to unexpected or intangible aspects of results, context or relationships in situations involving complex human interaction can help to break this paradox (Guijt, 2008; Mowles, 2010). This is because it allows people to move away from what Mowles (2010) calls a dualistic way of thinking characterized by a discourse of ‘were objectives achieved or not achieved?’ and ‘were projects on target or not?’ (Mowles, 2010: 767). Within such dualistic lines of thinking, unpredictability and non-linearity are seen as problematic, and may need to be avoided or contained as much as possible. From a complexity science perspective, it is more relevant to implement a programme as ‘a process of creating the enabling conditions for triggering emergent change mechanisms’ (Marra, 2011: 328). Stern et al. (2012) refer to Morell (2010) and Rogers (2008) who argue for the need to receive speedy feedback about a programme’s effects in the face of uncertainty in order to learn quickly if a programme is moving in the right direction. This presents a strong argument for ‘real-time’ monitoring and evaluation designs (Stern et al., 2012) and a developmental evaluation approach (Patton, 2011) where PME supports learning, not only about the progress towards the objectives that were originally set out, but also about unforeseen and intangible effects.

Learning to adapt

Supporting complex change is a two-way process. That means that any organization that is supporting social-change processes will itself also change (Earl et al., 2001). This is in line with insights of complexity science which suggest that programmes ‘are involved within a mutually adaptive relationship with their environment’ (Marra, 2011: 327). Being able to change and adapt to the changing context is crucial for organizations or programmes to remain effective and relevant. The following five elements of the core capability ‘to adapt and self-renew’ from the five core capability framework (5-C model) helps to explain what is meant by adaptive capacity: 1) to improve individual and organizational learning; 2) to foster internal dialogue; 3) to reposition and reconfigure the organization; 4) to incorporate new ideas; and 5) to map out a path for growth (Baser and Morgan, 2008). While many organizations look to PME to strengthen their adaptive capacity, reality teaches that our PME practice often doesn’t live up to that task (Crawford, 2004; Guijt, 2008). Even organizations that are able to attain the results they set out to achieve risk neglecting the need for continuously adapting to the changing environment in the process of being busy (Baser and Morgan, 2008). In such cases it is difficult to change deeply engrained behavioural patterns in organizations, such as a lack of time for reflection and learning (Britton, 2005; Fisher, 2010; Roper and Petit, 2003; Smit, 2007). Complex environments, therefore, put high demands on PME to support the adaptive capacity of development programmes.

Dealing with different accountability needs

A more dynamic understanding of accountability goes beyond upward accountability towards the donor, and entails a wider set of stakeholders (Whitty, 2008). Forms of downward accountability to beneficiaries (i.e. rights-based approaches) and public accountability towards the wider public are becoming more common. Eyben (2006) refers to Lindblom (1990) to explain that processes of accountability can be strengthened through investing in relationships and fostering mutual responsibility derived from shared learning through trial and error. This is especially relevant in complex contexts as such focus can encourage different actors to work with each other through mutual communication of their particular knowledge of the system or the problem at hand. Eyben (2006: 56) also warns of possible unequal power relations between the different actors in development programmes; in such cases, partners and donors ‘must see mutual responsibility as an aspiration to aim for, rather than something that is easily achievable’.

A wider understanding of accountability as described above helps us to recognize the existence of different actors’ different and not always compatible information needs in complex social-change processes (James, 2009). Besides donors’ upward accountability needs, implementing partners or NGOs might want the PME system to provide information that helps them to learn about what works and what does not work, in order to inform future planning and implementation. Furthermore, beneficiaries might have an interest in PME information to make and keep the programme or project accountable to their needs. Some compromise will be needed between these various information needs when developing a PME system (James, 2009). The information needed and its projected use will determine which approaches and tools are most suitable within a PME system for planning, data collection, data analysis and honest reporting (Simister and Smith, 2010).

The resulting theoretical framework

The four implications of complexity for PME that emerged from the literature discussed above, give us the building blocks for a theoretical framework (see Figure 1) that can be used to assess to what extent a PME approach is helping a programme deal with complex change. Each dimension of the theoretical framework corresponds with a specific implication of complexity for PME practice. This framework was tested during the collective action research described in the next sections.

Figure 1.

Analytic framework to assess if your PME approach is complexity oriented.

Using collaborative action research to explore and strengthen PME practice

A collaborative-action research design was chosen as a research method that was applied by 10 development NGOs (nine Dutch and one Belgian) and their southern partners to explore if and how a variety of PME approaches helped them deal with processes of complex change. Action research is research by particular people on their own work, to help them improve what they do, including how they work with and for others (Kemmis and McTaggart, 1992). The action research process is characterized by a spiral process of planning – acting – observing – and reflecting. This spiral process allowed the organizations who participated in the action research to try out specific PME approaches, and adjust how they were implemented according to lessons learned along the way. Collaborative Action Research differs from usual day-to-day practice by being more systematic and collaborative in collecting evidence on which to base rigorous group reflection (Kemmis and McTaggart, 1992). The collaborative aspect of the action research was characterized by 10 collective learning moments over a period of three years, where participating organizations shared the lessons learnt from their respective research processes.

The action research started with each participating organization clarifying their respective research questions based on the PME challenges they faced. This was done via an interactive process involving participants from the various cases (two collective reflection meetings), coaching sessions with research coordinators (two work meetings per case) and through internal reflection sessions with case representatives and their colleagues within their respective organizations.

PME challenges related to learning, showing intangible results, strengthening relationships, satisfying accountability requirements and stimulating ownership for PME activities form a golden thread across the various research cases. Each case then developed its own research plan in which they specified the type of data they were going to collect about their PME pilot, how they were going to reflect on these data, and who was going to be involved in data collection and reflection. The organizational research plans were developed in collaboration with the action research coordinators in such a way that they would help the organizations to answer not only their own organizational research questions but also the collective questions related to the implications of complexity for PME and which constitute the theoretical framework shown in Figure 1 above.

The action-research approach used in this study is not a value-free research process in which the researchers behave as expert independent observers. Instead, in line with the definition of action research by Reason and Bradbury (2001), the research has brought together action and reflection, theory and practice, in collaboration with research participants in order to explore practical solutions towards improved PME practice. Instead of being worried about objectivity, distance and controls, as in conventional research, as action researchers we were worried about ‘relevance, social change, and validity tested in action by the research participants’ (Brydon-Miller et al., 2003: 25). The quality and validity of the knowledge or evidence that emerged from the individual action research cases is therefore strongly determined by the extent to which this knowledge was useful and relevant for the stakeholders within the cases to better understand or improve their PME practice. It allows for an informed recognition of worthwhileness that is found through a synthesis of different understandings, involving a range of practitioners within a particular context (Cook, 2006). The action research provides a framework that helps practitioners to develop this knowledge into something they can then recognize and act upon (Cook, 2006). This also means that the findings from each case, while they may help to gain deeper understanding about what worked for what reason in which context, do not allow for wide generalizations or conclusions beyond the particular case contexts. Nevertheless, for the purpose of the cross case analysis, we used evidence from the individual cases as case-specific examples to illustrate the potential advantages and limitations of particular PME approaches as they are used in a particular way within a specific context. By exploring some commonalities according to the mechanism, context and outcomes of the PME experiments within the 10 action research cases, we are able to gain some deeper insights in the general potential of the explored PME approaches for helping organizations to deal with complex change.

Major limitations of the action research included the institutional insecurity of several research participants due to considerable cuts in funding from the Dutch government at the time of the research. Also, the organizations needed considerable time to develop and introduce their PME approaches, and only managed to implement one or two monitoring cycles during the research.

Focusing PME on the programme actors

The PME approaches used in this action research were selected by the participating organizations in the hope that they would be helping them to deal with specific challenges, which their existing PME systems – based on the logframe – could not address. Once the selection was made, the research team compared the communalities and differences of the selected approaches. A common characteristic of the PME approaches that made them particularly attractive for the action research cases turned out to be their focus on the actors whom a programme is trying to influence directly or indirectly (see Figure 2). Rather than focusing on the ‘hoped for changes in state’ (e.g. changes in income levels, agricultural production or health, for example) they direct attention on what people do (e.g. behaviour, practices, relationships) in order to contribute to the hoped for changes in state and/or people’s perceptions about the progress towards the hoped for changes of state.

Figure 2.

Visualization of actor focus of the PME approaches piloted in the action research.

Table 1 below summarizes the specific actor focus of each PME approach as well as the organizations where the approaches were piloted.

Table 1.

Outline of the actor focus of the PME approaches piloted in the action research.

PME approaches and organizations where they were piloted	Actor focus
Outcome Mapping (OM) (piloted by: Light for the World, STRO, MCNV, Cordaid)	• Initial planning and on-going monitoring is focused on changes in behaviour and relationships of intermediate local actors whose actions are believed to contribute towards changes at the level of final beneficiaries. (See also: www.outcomemapping.ca)
Most Significant Change (MSC) (piloted by Oxfam Novib, ETC Compas, Cordaid)	• MSC is a form of participatory monitoring and evaluation that involves many stakeholders in deciding the sorts of change to be recorded and in analysing the data. • Representatives from intermediate actors and final beneficiaries are asked about any significant changes that they experienced as a result of the programme. (See also: http://mande.co.uk/special-issues/most-significant-change-msc/)
Client Satisfaction Instruments (piloted by ICCO and Woord & Daad)	• CSI are downward accountability tools to monitor satisfaction among users of a particular service (See also: www.clientsatisfactioninstruments.org) • Clients are given an opportunity to provide direct feedback about their satisfaction with the services received. Clients can be the final programme beneficiaries (e.g. hospital patients) or could be the staff within local partner organizations.
Sensemaker (VECO)	• Target groups provide feedback about a certain issue of interest for the programme (e.g. inclusiveness in agricultural value chains) through a short story (micro narrative) and analyse their story according to predetermined criteria. (http://sensemaker-suite.com/index.htm)
Personal goal exercises (War Child Holland)	• Through this tool, youth participating in War Child Holland’s life skills workshops were directly involved in setting their own change objective and monitoring their progress in achieving this objective as well as in providing feedback on each module and the entire intervention. (http://www.warchildlearning.org/trainingtools/evaluation)

Action research results

In this section we use the theoretical framework presented above to describe the main results from the action research. Based on the four dimensions of this framework and using illustrations from the individual cases, we describe to what extent the actor-focused PME approaches have helped the cases to: 1) deal with multiple actors, perspectives and relationships; 2) promote learning; 3) strengthen adaptive capacity; and 4) satisfy accountability needs.

Dealing with multiple actors, perspectives and relationships

The potential of the piloted PME approaches to deal with the reality of multiple actors was explored through the ‘relationship’ dimension of the analytic framework. Across the 10 cases we observed that actor-focused PME approaches helped programmes to clarify and negotiate different programme stakeholders’ expectations, roles and relationships. This was shown to have important implications for programme design and implementation.

First, the use of Client Satisfaction Instruments (ICCO, Woord & Daad) and personal goal exercises (War Child Holland) allowed quick feedback from programme beneficiaries or programme actors who are influenced directly by the programme. This was shown in the three cases to encourage programme staff to reflect on their programme activities in function of the expectations and feedback of their target groups, something that was not done before using these PME approaches.

Second, in the four cases that used Outcome Mapping (Light for the World, Cordaid, MCNV and STRO), the ‘spheres of influence’ framework helped programme stakeholders to develop a better and more shared understanding of who is influencing who within the respective programmes and who is expected to do what in order to contribute to the programme’s objectives. As a result, programme actors were not abstracted but took a central role within the theories of change. These took the form of different spheres of influence containing different stakeholders depending on the particular influence and role they have within the programme. An example of an actor-focused theory from the Light for the World case is given in Figure 3 below.

Figure 3.

Simplified actor focused theory of change of the Light for the World inclusive education programme.

In the Light for the World’s inclusive education programme in Cambodia, the concept of the spheres of influence from outcome mapping provided a language among programme stakeholders to discuss the objectives of the programme and the roles and expectations of these actors according to their positions in the different spheres of influence of the programme. This helped to reorient the focus of the regular monitoring cycles on changes in behaviour and practice of the actors situated in the programme’s sphere of direct influence. Previously, monitoring and evaluation was mostly focusing on changes at the level of the children with a disability in the sphere of indirect influence. By reorienting its focus on the actors in the sphere of direct influence, Light for the World changed its strategy from service delivery to capacity development of local stakeholders. It also opened up a new results level where programme effects could be monitored on a more regular or ‘real time’ basis.

Third, the enhanced interaction between programme implementers and direct or indirect target groups, stimulated by the actor-focused PME approaches, was shown to bring the voices of the target groups to the attention and the consideration of the programme. This was observed across all the 10 cases and was shown to empower programme stakeholders and to strengthen relationships. For example, the monitoring approach in the Cordaid’s violence against women programme in Colombia, OM and MSC, helped members of various geographically dispersed women’s networks to gain more access to monitoring information on a more regular basis which in turn helped them to increase their understanding of the common agenda and the roles of the different network members. This was shown to empower them to take up leading roles and change established power positions with their networks.

Also, in the cases of Oxfam Novib and ETC Compas, it was shown that the use of MSC stories helped to strengthen relationships between programme staff and beneficiaries, who felt empowered through the MSC process:

MSC allows us to hear the voice of community members and not just community leaders that we hear from usually. The beneficiary moves from object to subject, and hence it empowers the individual. (Reflections from MSC workshop Oxfam Novib, Addis Ababa, 2011)

Finally, the Sensemaker approach was used in VECO’s agricultural value chain programme to learn about the inclusion of smallholder farmers within modern markets. Inclusiveness is in essence a relational aspect of value chain development. It refers to the extent that ownership, voice, risk and reward are being shared and distributed among the different chain actors (farmer organizations, local NGOs and private companies). The issue of inclusiveness was integrated into the Sensemaker signification framework that was used to signify the stories by the story tellers. In this way, Sensemaker helped to reveal patterns related to inclusiveness. Using this information during collective reflection sessions with key chain actors was shown to strengthen relationships and partnerships within VECO’s value chain programme.

The fact that the actor-focused PME approaches gave the organizations a practical framework to work on the relationships among programme stakeholders was seen as a major added value across the 10 cases. The changes in those relationships that emerged during the action research were also identified as important and unexpected results that were often neglected or missed before. They were seen as essential to learn on a regular or ‘real time’ basis if the programme was moving in the right direction. However, it was also evidenced across the 10 cases, that the increased social interaction that comes with an actor-focused approach requires the necessary leadership and resources to sustain such interaction over time. Setting up an actor-focused PME approach takes time and as illustrated by a majority of the cases, constituted a considerable shift from a more traditional PME practice based on measuring and reporting against predetermined indicators. Furthermore, the 10 cases also illustrate that building strong relations requires considerable facilitation skill and doesn’t happen automatically merely by involving people in the PME process. The availability of such skill was a considerable challenge in a majority of the cases.

Learning about programme progress

All the 10 cases used the logical framework approach (LFA) before the start of the action research. This approach resulted mainly in programme activity reports and rather quantitative information about predetermined SMART (i.e. Specific-Measurable-Appropriate-Relevant-Timed) indicators. The monitoring information was shown to provide only limited insight into intangible and unexpected programme effects. Reorienting a PME’s focus to programme actors was shown in the 10 cases to contribute to useful additional monitoring information about changes at actor level that could be obtained on a regular basis.

Learning about intangible results

Outcome mapping was found particularly useful to gain more clarity about the changes in behaviour, practice or relationships a programme was hoping to see at the level of the actors in a programme’s sphere of direct influence (see Figure 3). In the four cases that used outcome mapping it was observed that such clarity stimulated regular monitoring at these actor levels. The monitoring information that emerged was found useful to inform critical reflection about the effectiveness of programme activities and to inform or adjust programme plans. This is illustrated by the quote below from the Cordaid’s violence against women programme:

We were very busy with doing, doing, doing; knocking on the city mayor’s door, for example. The monitoring (using elements of OM and MSC) helped us to focus our actions and make them more effective. We reflected on the processes: was it worthwhile what we were doing or not? Then we identified the changes and adjusted the activities . . . There is more discussion about what political lobby is. We now understand that there are different levels of lobby. For some of us it was enough to have the city mayor participating in the council. But is that really the change we need? It made us reflect on the type of changes we want to achieve. (Representative of women network supported by Cordaid’s violence against women programme)

Also MSC was shown to help identify intangible effects such as gender-related deep cultural change in Oxfam Novib’s gender programme and changes in various forms of wellbeing (e.g. spiritual, leadership and women’s empowerment) in a sustainable agriculture programme of ETC Compas:

While the logical framework indicators would provide information about the number of terraces cultivated and the amount of crops produced, MSC stories allowed farmers to explain some deeper reasons why they are happy with the results of the intervention. Mr Suduappu, for example, was happy he was able to feed his family without having to kill other creatures. This was in line with his Buddhist worldview and was made possible thanks to the organic farming practices promoted by the programme. This link with the farmers’ Buddhist worldview is now used by ETC Compas partners to promote organic farming practices more successfully. (ETC Compas case report)

Furthermore, the use of personal goal-setting exercises allowed War Child Holland to draw deeper lessons about changes in psychosocial wellbeing of youth participating in its life skills workshops in a participatory and child friendly way. Previously, the implementation of validated questionnaires had resulted in limited findings on effectiveness which contradicted with the various positive observations of such changes by programme facilitators, teachers and parents.

Finally, the use of Sensemaker in the VECO case, as we have seen in the previous section, allowed the programme to make the intangible concept of ‘inclusiveness’ of agricultural value-chain actors more explicit and discussable. By making the state of affairs concerning this intangible concept more explicit, it became an issue that could be discussed objectively among programme stakeholders and could inform programme planning. Before the use of Sensemaker, programme actors knew implicitly that there were challenges related to inclusiveness, but they were not able to discuss it as they didn’t have tangible data or evidence.

Gaining insight into the unexpected

An important difference between the actor-focused PME approaches that have been explored in this action research and the more linear planning approaches, such as the logical framework approach, is their non-predictive character concerning the information they seek to track during programme implementation. The actor-focused approaches didn’t make use of SMART indicators with predetermined targets and timing. This open perspective towards the possible results of a programme contributed towards surfacing information about unexpected and unintended programme effects and about the local context. Across all cases, we found instances of unexpected effects that were captured due to their respective actor-focused PME approach.

The MSC story collection process, for example, in the Oxfam Novib and ETC Compas cases was left relatively open without predetermining domains of change nor specifying in advance who should be the story tellers. Instead, a non-prescriptive story collection framework was used that implied that stories should relate to changes experienced by people that were affected in one way or the other by the programme. This resulted in unexpected results as illustrated by the following example from the Oxfam Novib case:

After a gender training, I (female teacher) decided to form a girls’ football team and discussed my idea with the headmaster of the school. The headmaster was motivated by me and agreed. In May 2011, I formed a girls’ football team in cooperation with the headmaster. The members of the football team became skilled through regular exercise . . . Now the football team has become familiar in the Union. Other girls’ schools are now trying to follow this example and have taken initiatives to form such football teams. (Oxfam Novib, Bangladesh 2012)

Asking for direct feedback from target groups was also shown to highlight useful and unexpected contextual information. In the ICCO case, one partner hospital in Malawi was surprised by the patient feedback the client-satisfaction instruments obtained. Patients complained about the high price that they had to pay. Hospital management assumed that the medical costs were covered by the National Service Agreement in the district where the hospital is located, but it emerged that the patients who complained lived in a neighbouring district but the hospital of that particular district was much further away. This insight opened up internal discussions about the need for the programme to start lobbying the government to address this issue.

Learning doesn’t come easy

An important lesson across the 10 cases is that having the necessary monitoring information will not automatically lead to learning. Making sure that the monitoring information is used to draw meaning about the programme was an up hill task in all the cases. The main challenges faced in stimulating learning are summarized below:

Availability of careful and customized facilitation to promote critical reflection and identification of action points.

Availability of time and money for people to be able to meet and reflect regularly.

Reflection and learning activities not being integrated as a regular event in programme cycle management.

Too much focus on data collection instead of using the data for deeper analysis and collective sense making.

Organizations getting stuck in planning and taking a long time before moving into regular monitoring cycles. This can result in programme actors not immediately seeing the benefits of the monitoring process and lacking motivation to continue participating actively.

Strengthening adaptive capacity

The adaptive capacity domain of the analytic framework helped us to explore if the piloted PME approaches contributed to visible changes in organizational practice that have the potential to be sustained over time. In all 10 cases we observed that the experimentation with the actor-focused PME approaches had stimulated the creation of space and time for programme stakeholders to reflect and learn about the monitoring information. The fact that collective reflection and learning about the monitoring data is an essential aspect of the actor-focused PME approaches helped organizations to do the effort to bring stakeholders together for such collective reflection and learning sessions and to overcome the challenge of colleagues or partners being too busy or not seeing it as a priority. The importance of such reflection and learning spaces is illustrated by the Cordaid case (see quote below), where representatives from women groups meet twice a year to reflect on their achievements in the programme using a process of structured dialogue:

Further than the M&E findings, the process of being engaged in the M&E process and actively participating in the spaces designed for data collection, analysis, reflection and decision making, helped the women of the pilot regions to reinforce their sense of belonging to the program. They managed to reflect and identify the added value of networking for being more effective in their policy work (individually and collectively), as well as for being recognized as legitimate stakeholders on the women’s rights field. (Cordaid case report)

The added value of the piloted PME approaches is also evidenced by the fact that eight of the 10 cases had integrated elements of these approaches into other programmes or in the organization’s overall PME practice. Examples include Oxfam Novib integrating MSC in their country programmes in Nigeria and Zimbabwe, or the STRO case which saw partner organizations using outcome mapping to help them involve local producers and businesses in defining their roles and responsibilities in a new programme funded by the Ford Foundation.

The extent to which the observed changes in PME practice are appropriated by a whole organization or by a small number of individuals varies. For example, two organizations decided to explore their PME methods only after all head office staff members were involved in clarifying the specific challenge that they would address through this PME approach and the modalities for its implementation. The PME pilot was thus carried by the whole organization. In the other eight cases, however, one or a few individuals in the head office of the donor NGO introduced the idea of a particular new PME approach because they considered it useful to address specific PME challenges, mainly at the level of the partner organizations. An important implication of such a scenario is that the PME pilot is dependent on the motivation and availability of the individuals who introduced the new approach and also on the partners who are supposed to be the main implementers. This was shown to result in the following challenges:

First, the enthusiasm that came with the introduction of a new PME approach faded after some time in most cases. This posed a risk for sustained implementation, especially if it was dependent on the interest of some individuals and not part of an organization-wide policy or follow-up system.

Second, the lessons learned from the PME pilot risked being ignored by management because of a lack of critical mass or because it was considered to be an interesting but one-off side event.

Third, the responsibility of mentoring and follow-up in the field was often left in the hands of local consultants together with local partners. This came with the risk that the head office of the supporting NGO would be excluded from the learning process.

Accountability

The increased interaction and dialogue between programme stakeholders, triggered by the actor-focused PME approaches, was shown to help organizations to satisfy different accountability needs. Horizontal accountability was seen to be strengthened in those cases where an actor-focused PME approach went hand-in-hand with increased dialogue among different programme actors situated in the same circle of influence or control. Evidence of such increased dialogue was particularly observed in the four cases that used outcome mapping. This was illustrated by the Cordaid case, in which regular monitoring meetings strengthened horizontal accountability among women’s networks at local, regional and national levels. These meetings were found to contribute to increased transparency among the women groups involved in the programme as it allowed them access to monitoring information, to share information about the use of funds and to participate in agenda setting and making decisions about programme activities.

Downward accountability on the other hand was seen to be promoted in those cases where the actor-focused PME approach provided a practical methodology that helped programme actors to give critical feedback about the programme. This was most evidenced in the two cases that used client satisfaction tools. In the ICCO case, for example, one hospital in Malawi started offering integrated diagnostic services in response to the complaints that emerged through the client satisfaction tools from patients being denied treatment when there was no specialized doctor for their specific health complaint. In addition, downward accountability processes were also strengthened when the PME approach helped a programme to organize collective reflection sessions during which donor NGOs and partners could reflect on the partner’s progress and the effectiveness of the support provided by the donor NGO. This was observed in three cases that used outcome mapping where programme plans were adjusted during monitoring meetings involving representatives from both the donor NGO and the partners.

Finally, the different actor-focused PME approaches, across all the 10 cases, were also shown to contribute to upward accountability by providing information about changes at the level of the actors whom the programme is influencing directly or indirectly. These changes represent an important results level that was often ignored before the actor-focused approach was introduced and allowed the cases to report more widely about the results of their respective programmes. This was illustrated, for example, by the ETC compas case where the use of MSC provided qualitative information about perceived changes in wellbeing in addition to the quantitative scores obtained for the wellbeing indicators (e.g. number of ceremonies held in sacred forests or numbers of pesticide poisoning). This qualitative information mainly helped the local partner NGOs to ‘develop a better understanding of the complex reality of community wellbeing from the local perspective’ and to learn about the deeper effects of their sustainable agriculture programme. Most significant change stories were then included in the monitoring report to the donor.

At the same time, in the War Child Holland case, head office staff questioned the reliability of the monitoring information that emerged from the personal goal exercises which involves a process of self-assessment by the participants to the life skills workshops of War Child Holland. There was still the concern that such monitoring information could be biased by possible cherry picking of positive stories. Additional external in-depth research was therefore considered necessary to complement the actor-centred monitoring approach in order to satisfy upward accountability needs.

Conclusions

This article described how we used insights from complexity theory to identify four practical implications for PME of development programmes that are dealing with complex change. These implications relate to the ability of PME approaches to help programmes:

to clarify the multiple perspectives, roles and responsibilities of the various actors and to strengthen relationships;

to stimulate collaborative learning about the results of the programme, including intangible and unexpected results;

to strengthen adaptive capacity of the programme stakeholders; and

to satisfy the accountability needs of the various stakeholders involved.

We used these four implications to construct a theoretical framework that was used to guide reflection and analysis during a collaborative action research project where 10 organizations experimented with various actor-focused PME approaches in order to help them deal with processes of complex change.

Our first conclusion reflects on the usefulness of the theoretical framework for analysing to what extent a PME approach is complexity-oriented. Stimulating case participants to engage with the theoretical framework was not easy in the initial stages of the action research as it was seen as something that was forced upon them by the facilitating research team. However, over time, it did help participants to develop a deeper understanding of the meaning of complex change and this understanding started ‘to “disturb” their satisfaction with what they do in terms of PME and helped them to “move beyond the familiar to learn something new from their work”’ (Cook, 2006: 429). This was evidenced by the observation that case participants gradually became more critical and reflective (rather than descriptive) about their PME practice as they started utilizing the theoretical framework to reflect on their PME approach. Also there was a considerable improvement in the quality of analysis in the final case reports (September 2012) which included a structured reflection around the four domains of the framework as compared to the mid-term case reports (May 2011) where the framework had not yet been widely used. We therefore consider this action research as a first successful test of the framework as it helped to stimulate critical reflection about the extent to which actor-focused PME approaches are suitable for dealing with complex change. Of course, future longitudinal case research will be needed to determine if the current four dimensions of the theoretical framework are sufficient or need further refinement.

Our second conclusion refers to the results of the collaborative action research. These affirm that an actor-focused PME approach, if done well and followed through with the necessary leadership, can provide development organizations working towards complex change with the means not only to demonstrate this complex change (i.e. show their results) but also to learn how this change happened and how they contributed to it. This can help organizations to adjust their strategies according to lessons learned, making them more effective, adaptive and more accountable. This conclusion is based on evidence from the 10 cases that implemented various actor-focused approaches and is summarized in Figure 4. Actor-focused PME practice is therefore not just an interesting complement to more mainstream linear planning logic; we consider it an essential component of learning-centred programme management, particularly in contexts of complex change.

Figure 4.

Advantages of actor-focused PME approaches in dealing with processes of complex change.

We are fully aware that implementing an actor-focused PME approach is not a silver bullet that will solve all problems related to dealing with complex change and the pressures related to the results agenda. Figure 5 summarizes the main challenges faced when implementing an actor-focused PME approach in relation to dealing with relationships, learning, accountability and adaptive capacity.

Figure 5.

Challenges related to implementing actor-focused PME approaches.

Several implications for PME practice can be taken home from the action research. First, actor-focused PME approaches can help to broaden a programme’s results ‘radar’. In other words, tracking changes in behaviour, relationships or practices of target groups in different spheres of influence can bring to light important information about a programme’s effects which might otherwise remain hidden. Second, regular monitoring of programme results resulting in lessons that inform programme adjustments is key for dealing with complex change processes. This may require shifting perceptions of the meaning and value of regular monitoring practice. Monitoring is still often perceived as the little brother (or sister) within monitoring and evaluation. It is often limited to a regular follow-up of programme activities and activity-based progress reports. Evaluation, on the other hand, is much more associated with deeper learning about programme results. At the same time, such evaluations are often carried out by external consultants, and the learning therefore externalized. We argue that effective monitoring by programme staff needs to go beyond activity monitoring but should systematically and regularly track programme effects and motivate learning processes that can inform programme adjustment whenever the programme is getting off track. Strong leadership that motivates and mandates regular learning-centred monitoring of programme effects will be key.

For donors we present the following recommendations based on the results from the action research. First, there is need to adopt a wider notion of what results can entail. We observed that very useful programme results can be harvested in terms of changed behaviours, relations or perceptions among social actors directly or indirectly influenced by a programme. While such changes may not provide objective measurements of changes in state, which may be the specific objective of a programme (e.g. increased production or income, improved health), they are crucial to making these changes in state sustainable. Second, donors could ask funded programmes to demonstrate that they have developed and implement PME systems that are learning-centred and ask for specific accounts of how lessons learned were used for programme improvement or for planning. Third, donors could request for funding proposals that are clear and explicit about the various actors in a programme’s sphere of control (i.e. who is responsible for inputs, activities, outputs), spheres of direct influence (direct target groups) and spheres of indirect influence (indirect target groups or/and final beneficiaries). Finally, explicit appreciation could be shown for programmes that are able to demonstrate a deepened understanding of their theory of change over time, even if this means that the original theory of change had to change. This would help safeguard and promote flexibility in programme planning.

Footnotes

Acknowledgements

The authors would like to thank all the organizations who participated in this action research for their systematic reflection on their PME practice and for producing rich narratives of their experiences in the form of detailed case reports. A special word of thanks goes to Anneke Maarse, Cristien Temmink and Eugenia Boutylkova from PSO, who helped to coordinate this research.

Funding

This work was supported by the Dutch organization: PSO Capacity Building in Developing Countries (Thematic Learning Programme on PME of complex processes of Social Change).

Jan Van Ongevalle is research manager at HIVA-KU Leuven where he leads the development cooperation research group (). His research interests include complexity-oriented ‘planning, monitoring and evaluation’, participation and collaborative learning within programmes of international development.

Huib Huyse is head of the research group on sustainable development at HIVA-KU Leuven (). He coordinates research on emerging trends in global development, and studies various aspects of the work of development actors.

Peter Van Petegem, PhD, is a full professor at the Institute for Education and Information Sciences of the University of Antwerp, Belgium. He leads the research group EduBROn (www.edubron.be) and the Center of Excellence in Higher Education (). He is currently editor-in-chief of Studies in Educational Evaluation.

References

Baser

Morgan

(2008) Capacity, change and performance. Discussion Paper no. 59B, April. Maastricht, The Netherlands: European Centre for Development Policy Management (ECDPM). URL (consulted 20 September 2013): http://siteresources.worldbank.org/INTCDRC/Resources/CapacityChangePerformanceReport.pdf

Booth

(2012) Development as a collective action problem: addressing the real challenges of African governance. Report for the Africa Power and Politics Programme, October. London: Overseas Development Institute. URL (consulted 20 September 2013): http://www.institutions-africa.org/filestream/20121024-appp-synthesis-report-development-as-a-collective-action-problem

Britton

(2005) Organisational learning in NGOs: creating the motive, means, and opportunity. Praxis Paper no 3, March. Oxford: INTRAC.

Brydon-Miller

Greenwood

Maguire

(2003) Why action research? Action Research 1(1): 9–28.

Callaghan

(2008) Evaluation and negotiated order: developing the application of complexity theory. Evaluation 14(4): 399–441.

Cook

(2006) Collaborative action research within developmental evaluation: learning to see or the road to myopia. Evaluation 12(4): 418–36.

Crawford

Pollack

(2004) Hard and soft projects: a framework for analysis. International Journal of Project Management 22(8): 645–53.

Crawford

(2004) Aiding aid: a monitoring & evaluation framework to enhance international aid effectiveness. PhD thesis, Institute for Sustainable Futures, University of Technology, Sydney, Australia.

Davies

(2005) Scale, complexity and the representation of theories of change part II. Evaluation 11(2): 133–49.

10.

Earl

Carden

Smutylo

(2001) Outcome Mapping: Building Learning and Reflection into Development Programs. Ottawa: International Development Research Centre (IDRC).

11.

Eyben

(2006) Relationships for Aid. London: Earthscan.

12.

Fisher

(2010) Between pragmatism and idealism: implementing a systemic approach to capacity development. IDS Bulletin 41(3): 108–17.

13.

Forss

Marra

Schwartz

(2011) Evaluating the Complex. Attribution, Contribution and Beyond. Comparative Policy Evaluation, vol. 18. London: Transaction Publishers.

14.

Guijt

(2008) Seeking surprise: rethinking monitoring for collective learning in rural resource management. PhD thesis, Wageningen University and Research Centre, The Netherlands.

15.

Guijt

(2010) Rethinking monitoring in a complex messy partnership in Brazil. Development in Practice 20(8): 1027–44.

16.

Hummelbrunner

(2010) Beyond logframe: critique, variations and alternatives. In: Fujita

Nobuko

(ed.) Beyond Logframe; Using Systems Concepts in Evaluation. Tokyo: the Foundation for Advanced Studies on International Development, 1–33.

17.

James

(2009) Just do it: dealing with the dilemmas in monitoring and evaluating capacity building. Praxis Note no 49, November. Oxford: INTRAC.

18.

Jones

(2011) Taking responsibility for complexity: how implementation can achieve results in the face of complex problems. Working Paper no 330, June. London: ODI. URL (consulted 17 September 2013): http://www.odi.org.uk/resources/download/5275.pdf

19.

Kemmis

McTaggart

(1992) The Action Research Planner, 3rd edn. Victoria: Deakin University Press.

20.

Klijn

Koppenjan

JFM

(2000) Public management and policy networks. foundations of a network approach to governance. Public Management 2(2): 135–58.

21.

Lindblom

(1990) Inquiry and Change: The Troubled Attempt to Understand and Shape Society. New Haven, CT: Yale University Press.

22.

Ling

(2012) Evaluating complex and unfolding interventions in real time. Evaluation 18(1): 79–91.

23.

Marra

(2004) The contribution of evaluation to socialization and externalization of tacit knowledge: the case of the World Bank. Evaluation 10(3): 263–83.

24.

Marra

(2011) Some insights from complexity science for the evaluation of complex policies. In: Forss

Marra

Schwartz

(eds) Evaluating the Complex. Attribution, Contribution and Beyond. Comparative Policy Evaluation, Vol. 18. London: Transaction Publishers, 315–34.

25.

Morell

(2010) Evaluation in the Face of Uncertainty: Anticipating Surprise and Responding to the Inevitable. New York: The Guilford Press.

26.

Mowles

(2010) Successful or not? Evidence, emergence, and development management. Development in Practice 20(7): 757–70.

27.

OECD (2008) Accra Agenda for Action. Paris: OECD Publishing. URL (consulted 17 September 2013): https://dx-doi-org.web.bisu.edu.cn/10.1787/9789264098107-en

28.

OECD/DAC (2009) Managing aid, practices of DAC member countries. Paris: OECD Publishing. URL (consulted 17 September 2013): http://www.oecd.org/dataoecd/58/29/42903202.pdf

29.

OECD (2011) Proceedings of the High Level Forum on Aid Effectiveness. Report, OECD, Busan, Republic of Korea. URL (consulted 17 September 2013): http://www.oecd.org/dac/effectiveness/fourthhighlevelforumonaideffectiveness.htm

30.

OECD (2005) The Paris Declaration on Aid Effectiveness: five principles for smart aid. Report, OECD, Paris. URL (consulted 17 September 2013): http://effectivecooperation.org/files/resources/Paris%20Declaration%20in%20Brief%20ENGLISH.pdf

31.

Patton

(2011) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: The Guildford Press.

32.

Ramalingam

Jones

(2008) Exploring the science of complexity: ideas and implications for development and humanitarian efforts. Working Paper no 285, October. London: ODI. URL (consulted 17 September 2013): http://www.odi.org.uk/resources/download/583.pdf

33.

Reason

Bradbury

(2001) Handbook of Action Research: Participative Inquiry and Practice. London: SAGE.

34.

Roper

Petit

(2003) Development and the learning organisation: an introduction. Development in Practice 12(3–4): 258–71. URL (consulted 17 September 2013): https://dx-doi-org.web.bisu.edu.cn/10.1080/0961450220149654

35.

Rogers

(2008) Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation 14(1): 29–48.

36.

Simister

Smith

(2010) Monitoring and evaluating capacity building: is it really that difficult? Praxis Paper no 23, January. Oxford: INTRAC.

37.

Smit

(2007) We are too much in to do mode. INTRAC. Action research into supporting international NGOs to learn. Praxis Paper no 16, February. Oxford: INTRAC. URL (consulted 17 September 2013): http://www.ethiopia-ed.net/images/1517707581.pdf

38.

Snowden

Boone

(2007) A leader’s framework for decision-making. Harvard Business Review 85(11): 68–76.

39.

Stern

Stame

Mayne

Forss

Davies

Befani

(2012) Broadening the range of designs and methods for impact evaluation. Working Paper no 38 for The Department for International Development, April. London, UK. URL (consulted 17 September 2013): http://mande.co.uk/blog/wp-content/uploads/2012/06/2012-Stern-Working-Paper-38-final-April-20121.pdf

40.

Vähämäki

Schmidt

Molander

(2011) Review of results based management in development cooperation. Report for the Riksbankens Jubileumsfond. 30 November. Sweden. URL (consulted 17 September 2013): http://www.rj.se/f23f-87f4cf866f-dfe04be2d7125a47

41.

Van Kersbergen

Van Waarden

(2004) Governance as a bridge between disciplines: cross-disciplinary inspiration regarding shifts in governance and problems of governability, accountability and legitimacy. European Journal of Political Research 43(2): 143–71.

42.

Van Ongevalle

Maarse

Temmink

Boutylkova

Huyse

(2012) Dealing with complexity through actor-focused planning, monitoring & evaluation. Report, PSO and HIVA, Leuven, Belgium: November. URL (consulted 17 September 2013): https://partos.nl/webfm_send/34462

43.

Whitty

(2008) Accountability principles for research organisations. Report, One World Trust, London.

44.

World Economic Forum (2012) Global risks 2012: seventh edition. Report. Risk Response Network, World Economic Forum, Geneva, Switzerland. URL (consulted 17 September 2013): http://www3.weforum.org/docs/WEF_GlobalRisks_Report_2012.pdf