Methodological Considerations in Evaluating Long-Term Systems Change

The Objective of the Paper

To increase the evidence base on the relationship between multisectoral development interventions and long-term population level change, the UK Department for International Development (DFID) together with the National Planning Commission of Nepal commissioned an independent study in 2010 to look back after 40 years of development intervention delivered through Overseas Development Assistance and then by DFID. This unique and innovative study aimed to assess the impact of multiple development interventions on the long-term process of economic and social change, using a particular region (the Koshi hills) of Nepal as a case study.

This focus was identified based on the concentration in time and space of development interventions in the region and the perceived availability of data at district level from project documents and other sources. ¹ The study sought to determine whether the Koshi hills region experienced long-term social and economic changes over the study period and, if so, the degree to which these changes could be attributed to specific development interventions.

This article describes and analyses the approach and methodology adopted in the study. Overall, it aims to contribute to the reappraisal of retrospective evaluation studies and more specifically to the assessment of development interventions over the long term. It aims to support future best practice in the design and implementation of such studies both in Nepal and globally. We will highlight the challenges and opportunities associated with retrospectively viewing the impact of development interventions through a systems and long-term lens based on available data.

We found that the biggest challenge in this approach is the need to define and conceptualize what constitutes a “change” over the long term. This required both an assessment of intended results and change processes identified within individual program and project frameworks and of the broader context examined from an interdisciplinary perspective; history, sociology and anthropology, geography and development, politics, economics, and so on. This required a broadening of scope to consider the full range of potential drivers of systems change; not necessarily captured by conventional program management and evaluation tools such as logical frameworks, implementation plans, and program-specific Theory of Change (TOC) documents that normally characterize development programming and evaluation today (Coryn, Noakes, Westine, & Schroter, 2011; Frechtling, 2007).

The article first considers briefly the role of evaluation in development—with respect to policy formulation and planning as well as to specific development interventions—and issues around data availability within the field of long-term retrospective evaluation. It then describes the approach and methodology adopted in the Koshi hills study itself. The article concludes with an assessment of the challenges and general lessons learned from this experience that might be applicable to other researchers or commissioners of long-term evaluation studies through a systems lens in other contexts.

Evaluation in Development

The focus on measuring and demonstrating results of both the Paris Declaration and the Accra Agenda for Action has put evaluation at the forefront of development agendas for donor and recipient countries. This has resulted in increasing efforts dedicated to assessing the effectiveness or otherwise of their investments and interventions and identifying the impact or effects of programs and projects through which they have intervened (Organization of Economic Cooperation in Development [OECD], 2008; Rao & Woolcock, 2003).

A commonly used approach to evaluation in this context is retrospective evaluation, which attempts to assess impact after a program has been implemented by establishing program treatment and comparison groups ex post. Although considered generally less likely to produce credible results of program attribution as compared to prospective evaluations, these evaluations and “impact studies” are now, generally, an integral part of the standard project cycle, largely due to cost and feasibility concerns (Gertler, Martinez, Premand, Rawlings, & Vermeersch, 2011; Hageboeck & Frumkin, 2009).

The trend has also increasingly moved from evaluation being an internal program function to a more “independent” and “objective” function, often contracted to external consultants and organizations (Bamberger, 2009; Bamberger, Vijayendra, & Woolcock, 2010). Most evaluations however remain strongly “program and project focused” in their scope, with the normative assumptions that only the intervention will have an effect (Miyoshi, 2013). This focus draws evaluations into concentrating often exclusively on the linear linkages between specific interventions and those changes identified as program and project “outcomes” and “effects,” and being less concerned, if at all, with other causal linkages and changes less clearly attributable to the program or project.

This may lead studies to falsely attribute a given change to a particular program, project, or other “intervention” and obscure wider economic and social dynamics as contributing factors, or indeed alternative projects, operating within the context of larger, complex systems which may have contributed equally to the observed impact. Widening the lens is therefore key to more effective identification of program attribution and contribution to impact.

The field of prospective impact evaluations using counterfactual analysis, including experimental designs such as randomized control trials to demonstrate causality, is an accepted approach to attempt to overcome some of these issues. These methods underlie the standard evaluation model for many health programs, but programs intended to influence multiple variables and impact upon socioeconomic or political outcomes which are less tangible and harder to measure have been less subject to these designs, though this is changing (Banerjee, 2007; Banerjee & Duflo, 2009; Duflo & Kremer, 2005; Morra & Rist, 2009; Rossi, Freeman, & Lipsey, 1999).

Furthermore, the analysis and assessment of change are rarely examined from the perspective of people who are at the center of change and thus lack some explanatory power. The field of participatory evaluation attempts to address this, but these evaluations are considered less rigorous than experimental designs in their ability to estimate program attribution (Chambers, 1994; Cousins & Earl, 1992; Stern et al., 2012).

Finally, though practitioners may recognize and understand that while some changes may take place within a short period of time, “development” as a multifaceted phenomenon is a long-term and systemic process, evaluations rarely examine effects of a program beyond their immediate program focus, largely due to the short-term project and funding cycles (Hageboeck & Frumkin, 2009).

Evaluation of Long-Term Change

Along with a growing trend in program-specific evaluation, in recent years, donors have also begun to show more interest in assessing the overall effects of investments on poverty alleviation and improvements on the quality of life beyond a project cycle and within the context of more broadly defined systems than those bounded by sectoral definitions. This has caused a shift to look not only for evidence that their interventions resulted in the changes envisioned but also for more broadly derived evidence regarding the determinants of development and change over the long term. For instance, DFID, in recent years, has commissioned a number retrospective evaluations to look at the overall impact of a DFID country program on poverty across a program portfolio (Thornton et al., 2010).

A number of methodologies have also been employed to study change in the long term (Magnussen & Bergmann, 1990; Menard, 1991). These include historical studies, in which a range of sources of evidence is drawn upon to provide a description and analysis, and sometimes even an explanation, of changes in a given location or region or country—“a narrative” (Blossfield & Rohwer, 1995). Another is “the anthropological study,” in which fieldwork over a relatively long period (at least many months and sometimes years) provides the major basis for research into the past, through “oral history” (drawing on local people’s “indigenous” knowledge and memories) and other materials (sometimes documents; Davies & Dale, 1994).

In addition, restudies are often undertaken by government or other agencies for the purposes of national and regional or sectoral planning (i.e., National Living Standards Surveys, Household Income and Expenditure Surveys, Demographic and Health Surveys, Agricultural Surveys, etc.). These studies are based on representative sample surveys which claim to collect data on the basis of carefully defined samples so that statistical analysis is possible and correlations may be made with relative confidence regarding trends and tendencies within a prescribed population (Coleman, 1981; Gershuny, 1998; Ruspini, 1999).

A major limitation of many of these evaluations in producing strong and credible findings of program impact as compared to prospective evaluations are their reliance on the existing sources of information which are available to the evaluator. The limitations in the availability of robust and comparable data across programs and over time pose challenges to estimating a valid counterfactual over the long term and ultimately affect the strength of evaluation findings in terms of estimating attribution and contribution.

It is within this new endeavor that this particular evaluation of long-term impact in Nepal was framed. This article refers to the challenges of retrospective evaluation in a “data-poor” setting. By this we mean that though this particular region of Nepal was chosen for its perceived data availability, the availability of robust and comparable data over the 40 year study period posed a major challenge to the ambition of the evaluation. Data of various kinds were widely scattered, between the offices and archives of diverse institutions, both governmental and nongovernmental, and in numerous libraries, bookshops, and so on, making collation and ordering a significant challenge. Moreover, it was diversely focused making comparative analysis at the same level—change in the Koshi hills as the system of analysis—difficult. Although some restudies and population-based data were available, the majority of the data collected were program reports reporting on project-specific results over a short-time frame. The next section describes how the study design and methodology attempted to overcome these challenges to respond to the overarching research questions.

Study Setting and Design

A study of this complexity and scope required careful consideration of a combination of quantitative and qualitative approaches. The right mix was expected to explore the notion of change from both an aggregated and disaggregated trend analysis level and the objective and subjective perspective of people living in poverty. Evidence gathered using both qualitative and quantitative methods was intended to enrich the possibilities to interpret findings through the eyes of people living in poverty, to challenge externally imposed assumptions supported through available data, and to build retrospective theories of change to explain how and why (and under what conditions) change occurred.

For the purpose of this study, “the Koshi hills region of Nepal” was understood to comprise four districts of the Koshi zone in the Eastern Development Region of Nepal: Bhojpur, Dhankuta, Sankhuwasabha, and Terhathum districts (Figure 1). The study team used these “districts” as well as subdistricts or “village development committees (VDCs)” as the local units of analysis and comparison across study sites.

OPEN IN VIEWER

Figure 1.

Location of the Koshi hills and surrounding districts, Nepal.

Given the complexity of the study, a broad range of evaluation methodologies were combined within an appropriate theoretical framework to guide study implementation. The team used a multidisciplinary and chronological framework to approach the collection, synthesis, and analysis of data. This was used to address challenges to the accurate measurement and evaluation of development impact presented by complex pathways to change, confounding effects and inputs, unintended as well as intended consequences, and the intangible nature of many of the impacts that were hoped to be achieved over the years in the study area.

It was recognized from the outset that not only would it be difficult to define and measure “long-term impact,” but developing an understanding of what contributed to these changes within a larger system also represented a fundamental challenge. The study thus used a mixed-methods approach using reviews and analysis of quantitative, qualitative, spatial, and historical data to develop complex causal pathways of change and to identify the nodes of social and economic drivers of changes over decades and generations of people living in the study area. Combining the different methods was proposed to make it possible to a certain extent to overcome limitations and to enhance the strength of analysis (Bamberger et al., 2010; Stern et al., 2012).

Individual Research Components

A number of individual research components were conducted to produce the data required to assist in pulling together the complex narratives of social and economic changes over the 40-year study period.

Analytical Approach

The findings of the individual research components were ultimately compiled and analyzed as a comprehensive data set of quantitative and qualitative data in order to integrate findings and draw overall conclusions. The analytical approach developed was similar in structure to a ToC framework, using analytical techniques including causal process mapping, key node analysis, and strength of evidence assessments to identify key drivers in a complex system. The retrospective construction of causal chains irrespective of a development “sector lens” led to the identification of unanticipated cross-sector linkages and relationships which may not have been evident through the framework of a particular program or sector ToC. This approach attempted to recognize the complexity of interrelations within a system and diverse pathways to change. Three key stages of analysis were conducted.

Key node analysis

Once causal process maps linking collected evidence along the causal chain to long-term impact had been developed, key nodes, or drivers, of change were identified on the basis of those development interventions and other contextual factors that recurred most regularly and with the strongest plausible causal links with a specific observed change. The aim was to critically investigate interrelationship and the reinforcing nature of interactions. For example, with regard to the identified long-term change of growth in alternative livelihoods outside of agriculture within research sites, evidence linking this change to improvements in road network was identified numerous times in the secondary data review, the RCA study, and numerous other research findings. This provided a strong basis for a plausible causal relationship between road transport development and the development of alternative livelihoods.

Figure 2 presents a causal map which frames a complex narrative of change. This causal map indicates the series of complex interrelationships between development intervention and other factors which drove long-term changes over the study period. The causal processes were nonlinear and often cyclically reinforcing and thus, to the extent possible, key processes were highlighted within a nonlinear and complex system. Thus, for example, migration was both an end point in poverty alleviation by way of economic opportunity and increased agency, but also a key driver in the process, as increased money from remittances was spent on further migration opportunities. This is depicted in the figure, where permanent migration is encircled as a key node with forward and backward causal linkages. This causal map provided the basis for the study’s findings in regard to the impact of development interventions (and other factors) on long-term change in the Koshi hills.

OPEN IN VIEWER

Figure 2.

Example of a causal process map and key nodal analysis leading to plausible narratives of change in Koshi hills: changes in economic opportunities available within Koshi hills. Note. Ag = Agricultural.

The Transformative Effect of Cardamom Production

Differentiated from small-scale diversification of agricultural production as a resilience mechanism that has a long history in the Koshi hills, changes detected in commercial agriculture during the study period included the regular growth of agricultural surplus for sale, the adoption of new technology and inputs, and specialization in a saleable product. In relation to this, a significant trend was the growth in production of high value crops.

The story of cardamom as a high value crop in Koshi hills is an illustrative example of the complexity and nonlinear nature of change in the region. Over the study period, the cropping area in the Koshi hills dramatically increased from 7 ha in 1971 to 3,930 ha in 2009. The study identified that cardamom production expanded through a number of drivers of change, including expansion of the transport network, changes to the price in cardamom due to the collapse of production in other regions following the spread of disease, and changes in access to technical information to help small holder farmers increase agricultural production and competitiveness.

Analysis of land holdings dedicated to cardamom production indicated that Ilam and the Koshi hills both began the production of cardamom before the roads in the late 19th century. However, the rapid acceleration of production only began after the completion of the roads and coincided with price increases as a result of higher market demand when production collapsed in Bhutan and Sikkim. This strongly suggests that road construction had an amplifying effect on cardamom production. GIS data demonstrating changes to cardamom production along the main trail to the Tibetan border during the 1990s suggest that it was changes in the transport network as a whole, and not only roads per se, that stimulated production in new areas and on a significant scale.

Analysis suggested that development interventions in the form of agricultural support programs were also a strong factor in the development of cardamom and other high value crop production, as small holder farmers benefited from better technology and information about production parameters and market structures. Studies suggest that farmers became more confident about markets and prices of high-value crops in general with the support of projects such as the Centre for Environment and Agriculture Policy Research Extension and Development and the Koshi Seed and Vegetable Project and provision of technical support and information from projects such as the Pakhribas Agriculture Centre.

This increased utilization of technology and information as well as access to and confidence in markets due to an expanded road network, coinciding with cardamom prices increases, was the right combination to make cardamom an important cash crop in the Koshi hills, and this lifted many households out of poverty.

Implications for Ability to Disaggregate Findings

Challenges were also experienced in obtaining appropriate socially and economically disaggregated data. Disparities in poverty and social outcomes in Nepal (and in the Koshi hills) cut across gender, caste, ethnicity, religion, and geographic regions. Yet, the history of the availability of socially disaggregated data is fairly recent. The census of 1991 was the first to identify in detail the 100 or so different caste and ethnic groups in the country. Since then, however, the classification of the population in terms of a range of criteria has become a major preoccupation, with different dimensions of poverty, social discrimination and disadvantage, gender, caste, ethnicity, age, region, and so on all considered when devising “socially inclusive” policies, programs, and projects. While this is the case, the lack of data disaggregated by these factors, specifically gender-disaggregated data, for much of the period reduced the ability of the team to conduct much needed disaggregated analysis and to disentangle the differential impacts of development interventions on these different social groups.

Implications for the Ability to Assess Cross-Sector Systems Level Change Versus Programme-Level Change

Although the intention was to take a perspective on long-term change outside the typical program and project development lens, there was a tendency for both data collection and analysis to be “development-led,” as much of the data was drawn from program and project reports. The Koshi hills, as defined by the study, was not a single geographical or administrative unit; but rather, a development region defined largely by an early foreign aid development intervention (the Koshi Hills Area Development Project). This influenced the type of data collected and available during the time frame. For example, the study had a larger focus on changes in the 1980s, 1990s, and 2000s in part because it was during those decades that development interventions in the Koshi hills were greatest and in part because there was relatively limited quantitative data prior to 1971. Building a coherent narrative of change beyond this “development lens” was therefore problematic. This inevitably led to a less sharp focus on drivers of change beyond “development interventions,” although it was clear that from the mid-1990s onwards, foreign labor migration and the flow of remittances into the region (as elsewhere in Nepal) were having a major effect. Moreover, an additional issue was presented by the lack of continuity and comparability of data generated through diverse project Monitoring and evaluation and reporting systems. The interoperability and short time frames of project data sets pose a consistent challenge to long-term change trend analysis.

Implications for Comparative Counterfactual Analysis

The lack of representative baseline information meant that only the most significant drivers were identified through causal analysis. Without adequate baseline data that covered all the different sectors, the disparate existing data had to be pieced together strategically to develop a coherent narrative of the context in the beginning of the study period. The causal process mapping proved an effective tool in retrospectively constructing baselines and timelines of change, leading to plausible conclusions of contributory impact. This process however was deliberately selective to address only the most significant drivers of change, where the strongest evidence was available. Thus many factors where strong evidence was not apparent in the available secondary data were omitted from analysis, however important they were considered to be. For example, there was little systematic evidence to enable the reliable identification of differential impacts or the nuances of change between different social groups (i.e., defined by class, ethnicity, caste, or gender). This was a major weakness of the methodology.

As such, the study was able to produce a body of interesting findings, but, apart from the economic analysis, which indicated specific correlations between investments and actual changes, the conclusions drawn suggested plausible narratives and complex scenarios of probable cause and effect within larger systems rather than conclusive findings of linear causal (cause-effect) relationships. It is important, however, to recognize that even where data are good, the dynamics of causation may be too complex to allow for simplistic and reductionist models of cause and effect, and that the emphasis on plausible narratives and complex scenarios is not simply a consequence of poor data.

Study Design and Implementation

Retrospective evaluation of long-term impact requires a strongly led multidisciplinary team able to combine their distinctive disciplinary approaches within a single research framework and a clear strategy for iterative data collection and analysis. We derived the following lessons learned in this regard that could be relevant for future researchers.

Looking at Systems Change

Two important lessons from this study include the importance of looking at systems change and the necessity of delineating and understanding overall patterns and dynamics of change before attributing causation. The causal mapping process used as a main analytical approach highlighted the linkages and interrelatedness of change over time, as well as the synergistic effects of interventions. Reviewing the causal maps reveals the nonlinear nature of change and the fact that it is difficult and in some ways not useful to disaggregate change a priori to the program or project level but indicates that an ongoing process of developing scenarios and causal narratives at varying levels of detail is a desirable part of the overall research process.

By combining conventional program-specific TOC approaches as a starting point for each sector in conducting final analysis, it was possible to identify the overlap between different programmatic- or sector-specific TOCs and how outcomes originally conceived within sectors interacted across sectors over time to provide long-term impact in peoples’ lives—extending the delineation of analysis to the complex system level. For instance, changes in attitudes toward the role of women and girls in society cannot be limited to a single program or development intervention, as these changes cut across education, health, and livelihoods.

When considered together, this emphasized the complex nature of change and the risks associated with the narrow view that any one program or even sector can directly “cause” long-term change at a population level. Program or sector-specific theories of change for any one community, locality, or sector will never capture the wider context, or even consider the assumptions and externalities inherent outside that self-induced boundary. Unless an integrated approach is adapted to development planning incorporating all actors and stakeholders (government, donors, private sector, and civil society), there will inevitably be missed opportunities for the identification of synergies and complementary cross-sector dynamics as well as instances where interventions “pull in different directions” or where a promising start is not followed through due to changing priorities or theories.

As such the implementation of prospective studies similar in method to the current assignment but designed to provide a strategic framework to future program planning may be considered. This would provide a strong evidence base for contextual understanding and allow more effective planning of interventions on a longer time horizon and with a cross-sectoral appreciation.

Building on this, the scope and focus of theory-based evaluations, both in short-term and long-term, project-specific evaluations, should be considered within the context of systems change. The recent use of theory-based evaluation in development programs, investigating and testing causal links in a project’s or program’s impact theory, attempts to address this by explicitly stating assumptions in proposing relationships between program interventions and outputs and development outcomes and impacts (Coryn et al., 2011; Donaldson, 2007).

The Koshi hills study shows that long-term socioeconomic change is routed in systems, complex and multisectoral (or rather ‘non-sectoral’), and attempting to deal with this complexity in a project-focused evaluation by controlling for these additional factors in analysis (as is standard practice in economic analysis) on a short, program-defined time horizon, does not provide the analytic sophistication necessary to answer research questions confidently. This raises questions about the value for money or utility of the usual (heavily economic) project-level impact evaluations of interventions of known effectiveness at scale.

These questions are a source of much debate among impact evaluators and development stakeholders. Results from this study can contribute to this debate, demonstrating the complexity of socioeconomic change over a long horizon and the inability of any one programme to demonstrate attributable and sustained changes in a population over time in isolation and in the absence of the myriad of other changes at play at any given time.

Even if attribution is demonstrated in the short term by one intervention through rigorous evaluation methods, our study argues that it is methodologically difficult within a project focus to fully understand how this attributable change will interact positively or negatively with the other developmental and external factors to which a population is routinely exposed over time. Particularly when attempting to demonstrate attribution at the higher impact levels of change, it is well understood that program-level attribution declines and contribution increases.

This issue of estimating attribution in the long term might lead one to conclude that impact evaluation at scale could be reserved to look at the long-term socioeconomic change across development sectors within a community, attempting to understand the mix of development interventions that contributed to this change rather than focusing on impact evaluations of particular programs.

The pace of change is also accelerating and thus outcome level data must be more accessible on a 4–5-year horizon looking across sectors. Program intervention thus might be reserved for specific studies into the effectiveness of delivery and if programs achieve the desired outputs. This has direct implications for how governments and donors plan and fund evaluation activities.

Data-Related Implications for This Type of Research in the Future

Governments and donors interested in measuring long-term change in the future must take data storage and accessibility into consideration. It is axiomatic that effective policy making and planning rely heavily on reliable data. Yet many countries, including Nepal as the focus of this study, still lack a central archive bringing together routine large-scale surveys (undertaken by central statistics agencies and/or other government agencies) and other surveys and studies relating to development (undertaken by government, nongovernment, international nongovernmental organizations as well as local organizations, university, and research institute agencies and organizations).

An effective archiving and storage system for data which makes data available and accessible at the most disaggregated level to any interested parties is thus critical for the ability of other researchers in the future to look in detail as well as in broad terms at long-term change. Equally the establishment of data standards and indicators and variables of change commonly agreed upon and universally applied in all related data collection, whether program focused or long-term indicators, would have a significant positive impact on the ability of researchers and policy makers in the future to understand change and implement policy from a string evidence base. This equally applies to other settings and should be seriously considered by governments and donors.