Development Trajectories and Complex Systems–Informed Theories of Change

Abstract

This article presents an approach to use complex adaptive systems thinking to construct a theory of change to plan and evaluate transformational interventions. The article draws on the concepts of domains, scales, agents, and emergence to build a model of the system targeted by the intervention, to identify the chains of causality driving the system, and to identify the most influential enabling conditions affecting the system's development trajectory. Using the case study of the Indonesia SMART-Fish project, the article illustrates how to use such a theory of change to understand how a program interacts with the phenomena and how to assess the extent to which a program contributes to a development trajectory consistent with the intended long term transformational objectives. The article also illustrates how to use network analysis tools and simple data visualization techniques in ways that engage stakeholders in evaluation design, data collection and analysis.

Keywords

theory-based evaluation environmental system evaluation policy change

The Sustainable Development Goals (SDGs) include a set of ambitious objectives that require fundamental changes in the way societies operate. Such changes must address problems that have multiple causes and at various levels (Brondizio et al., 2009; Selomane et al., 2019). In this context, development practitioners face the challenge of developing sufficiently comprehensive interventions that lead to the desired long objectives but are also manageable (Folke et al., 2016). Additionally, the full effects of transformative interventions can only be expected years after an intervention ends. Thus, development practitioners also face the challenge of knowing the extent to which interventions are contributing to a path conducive to the stated policy objectives. Theories of change (TOC) are tools often used to address these challenges (Koleros & Mayne, 2019).

Useful TOCs must be readable and not unduly complicated while at the same time sufficiently comprehensive and detailed to capture the phenomena in question. For a TOC to be useful, it must also consist of a set of “if…then…” propositions that help explain how interventions work. Particularly crucial in TOCs is the clarity on how the different elements of the TOC are interacting to generate the desired outcomes (Davies, 2018).¹ Systems thinking seeks to understand how the interaction between the components produce a whole (or system). The whole is different from the parts and exhibits behaviors or shapes that cannot be explained by any of the parts or their sum (Arnold & Wade, 2015). The emphasis on understanding the interaction of the elements of a system is especially useful in the development of TOCs that strive to model the chains of causality of projects or programs seeking transformational changes. Good TOCs also make assumptions explicit (Bamberger et al., 2015). Given the uncertainties inherited in complex processes and the longtime change frameworks, it is also essential to periodically revise TOCs based on information generated during implementation.

The literature of systems thinking is broad and includes many different perspectives (Laszlo & Krippner, 1998). The social–ecological systems (SESs) proponents have developed a set of concepts to understand the interlinked dynamics of social and natural subsystems (Fischer et al., 2015; Jerneck et al., 2011). SESs are composed of social and ecological systems that contain units that interact in ways that are interdependent (Anderies et al., 2004, p. 18). Given the emphasis on the link between social and ecological processes, SES approach is particularly useful in the analysis of cases such as fisheries, forestry, or coastal management, where there are strong direct interactions between social processes and the natural environment. SES thinkers also integrate into their analysis justice and sustainability for humans and the earth (Preiser et al., 2018). This article proposes a methodology based on SES concepts to construct TOCs that can help steer complex development processes. These TOCs are also useful to assess the extent to which interventions are likely to contribute to long-term goals such as the United Nations SDGs (Selomane et al., 2019). While the TOC presented here was used as an evaluation tool, TOCs are more effective when done during the preparation of interventions as they are a useful tool for ex ante design to monitor and adapt programs in the light of information generated during implementation. This article consists of three parts. The first part explains a few key concepts to construct models of SES. The second part describes how to use those concepts to build TOC to plan, monitor, and evaluate transformational interventions. The third part provides examples of how evaluators used an SES-based TOC to assess a project contribution consistent with long-term policy objectives.

Key Concepts to Construct Models of SESs

Concepts that are particularly useful in the construction of TOCs (see Text Box 1) are the SES and its boundaries, domains, scales, agents, adaptive behavior, and the emergence and development trajectory.

Text Box 1.

Elements of a Complex System.

Domains

Temporal scales

Spatial scales

Agents

Adaptive behavior

Emergence

Development trajectory

The SES and Its Boundaries

The boundaries of a system are established through the identification of the entities that have more connections among each other than the connections they have with other entities. Ostrom (2009) describes the SESs as a set of multiple subsystems that are nested and linked. In a complex SES, subsystems can be relatively separable. Still, these different subsystems interact to produce outcomes at the SES level, which in turn, feedback to affect the different parts of the system (Ostrom, 2009). SES are complex adaptive systems because they can reorganize their parts and learn when facing internal or external drivers (Anderies et al., 2004; Dooley, 1996).

Domains, Scales, and Boundaries

Domains and scales are crucial elements of SES that help to delineate the phenomena pertinent to the system. Further domains are areas of knowledge (fields of cognition) or activity characterized by a set of concepts, terminology, and behaviors (Couture & Valcartier, 2007). Domains are representations of subsystems of phenomena that interact with other subsystems and affect outcomes at the SES level. Domains that are typically considered relevant to development interventions include social–cultural, ecological, economic, governance and science, knowledge & technology (Zazueta, 2017). Each domain comprises nested subdomains such as the economy encompassing the market and production or governance containing the legal and the administrative system. The identification of the specific domains that are pertinent to a set of long-term objectives requires research and consultations on the root causes of barriers to achieving those objectives. The relevant domains and subdomains provide an organizational framework of the conditions that need to be changed or reinforced in order to steer the system in the direction of the policy objectives. Domanial conditions take place across diverse scales.

Scales are conceptual tools that can help to categorize aspects of the phenomena related to time and space. Spatial and temporal scales consist of sets of regions or levels expressed in quantitative forms. Thus, the spatial scales include regions or levels related to the micro–meso–macro continuum. The temporal scales include regions or levels associated with the short–medium–long duration continuum (Gibson et al., 2000). Levels are thus units of analysis located in different areas within a scale. For example, within the ecological domain, spatial scales consist of natural systems that are geographically nested: a coral reef that is within a bay and a bay that is within a sea. The levels within an ecological spatial scale are organized based on natural systems criteria. Within the governance domain, the levels along the spatial scale of the administrative subsystem are defined based on formal political or jurisdictional criteria; an example of such a scale includes the levels of the municipality, province, and national government. The temporal scales can consist of different time frames or durations; for example, an intervention can have different levels of accomplishment in building the capacities of administrative systems in the short, medium, and long term. D. Cash et al. (2006), using the criteria developed by Gibson et al. (2000), expanded the types of scales including frequencies as one of the temporal scales and expanded the spatial scales to include administrative, geographical, ecological, and sociocultural.

In complex systems, the boundaries between different types of scales might coincide or not. Ecological scales often cut across several administrative scales. One example is the California Current Ecosystem in the Pacific Ocean, which spans across nearly 3,000 km and includes multiple jurisdictions in Canada, the United States, and Mexico. Sociocultural scales can take place within or across jurisdictional scales, such as the case of the Papago Native Americans, who reside in the Sonora desert between Mexico and the United States. Mismatches between the different types of scales significantly affect outcomes at the system level. For example, mismatches between administrative and ecological scales can substantially influence the management effectiveness of resources in problems related to pollution, migratory fisheries, and aquifer management in which coordination and consistency of approaches of the resource management are needed (W. D. Cash et al., 2006; G. Cumming et al., 2006; Wilbanks, 2007). The links between scales make it possible that changes that originate at one scale can trigger developments at other scales and across the system (Selomane et al., 2019).

System Components, Agents, and Adaptive Behavior

Interactions across the system take place through the system components, which can be human or nonhuman. Human components (typically referred to as agents) include individuals, communities, organizations, institutions, or any organized expression of human agency. Nonhuman components include water, air, forests, minerals, and animals. Nonhuman components also command agency in as far as they interact with and affect the behavior of other system components and display changes in response to the actions of other agents or system components. Moreover, in some cultural settings, nonhuman agents are also perceived and treated by human agents as having objectives and power. Human agents may have different belief systems with diverse values and objectives, as well as access to various resources, information, or power (Ives et al., 2020). SES models assume that agents interact with diverse degrees of autonomy from one another and that actions of an agent can trigger a chain reaction of the adaptive behavior of other components.

The adaptive behavior of components is a significant factor contributing to the dynamism and complexity of the system. While agents operate from different places, scales, or domains, they are linked (either directly or through other agents). The system works through the actions and reactions (adaptive behavior) of the agents. Even relatively minor agents under the right conditions can generate reverberating behaviors across the system (Brugha & Varvasovszky, 2000; Clement, 2013). The agents’ adaptive behavior can take many forms, such as imitation, cooperation, conflict, and coalitions, which feedback to influence other agents’ reactions (Allen & Garmestani, 2015; Holland, 2006; Levin, 2003).

Emergence and System Development Trajectory

The combination of the interactions of the agents results in the emergent properties of the system; these are system properties for which no individual agent is responsible. These properties make systems unique and can take many diverse forms. The chain reaction of the adaptive behavior of agents across scales and domains is a significant factor that contributes to the system complexity (Holland, 1995, 2006). Due to this complexity, the same intervention or action can have very different results in different circumstances. Given the multiple interactions of the agents involved, the changes triggered can be unpredictable and nonlinear; this is to say that the ultimate outcomes in a process are not necessarily proportional to inputs.

Three central aspects of SES dynamics are resilience, adaptability, and transformability. Resilience is the capacity of an SES to change and adapt while remaining within critical thresholds. Adaptability is the SES capacity to adjust responses to external and internal processes while staying within the current development trajectory. Transformability takes place when the reorganization of parts of the SES crosses the threshold, resulting in a transition to a different system development trajectory (Folke et al., 2010). In SES, system transitions take place when ecological, economic, or social conditions make the existing system untenable (Cumming et al., 2006; Feeny et al., 1990; Walker et al., 2003). The notion of system development trajectory assumes that systems are dynamic. While complex systems are unpredictable, past events influence how and in what direction systems change (in systems terminology, this is known as path dependency). The mechanisms involved are positive feedback loops that cross domains and scales that can steer the system toward a given direction. One example of a development trajectory is how the policies, technology, and information influence the interests and behavior of humans to adopt patterns of consumption that have led to and reinforce a trend of CO₂ emissions, which is predicted to transform the earth’s climate and life as we know it. In the case of fisheries, it is frequently the case that market incentives to increase capture, improvements in the efficiency of capture technology, weak institutions, and energy subsidies are contributing changes that are highly likely to lead to the collapse of many fisheries around the world. The concept of system development trajectory is particularly important for development interventions because the purpose of development interventions is precisely to steer SESs to conditions that can sustain human life with dignity as well as ecosystems that can provide services for all life on earth. While having enough information to know that our systems are in a trajectory that is not sustainable, due to the complexity of the system, we cannot predict what specific form the system will adopt or the timing changes will take place. Given the unpredictability of these systems, SES thinkers propose that development interventions mimic other agents in the system and adopt adaptive management as the approach to steer the development trajectory of the system (Hartman & De Roo, 2013). Adaptive management entails clear goals, identification of alternative management objectives, the development of a set of the hypothesis of causation and procedures for data collection, and ongoing evaluation (Allen & Garmestani, 2015).

An SES-Based TOC of SMART-Fish Program

Indonesia is the second largest fish producer in the world and a global hot spot for conservations with one of the highest levels of marine biodiversity. It is also one of the most fish-dependent countries in the world for food security (Dhina, 2017). In 2012, around 6.4 million people were engaged in inland and marine fishing and fish farming; most of them were small fishermen and women (Food and Agriculture Organization of the United Nations, n.d.). With its marine resources spread to almost 18,000 islands, 6,000 of which are populated, a coastal area of 54,716 km—the second longest in the world, and exclusive economic zone of 6,159,032 km², Indonesia faces many challenges in managing marine resources. The country confronts urgent pressures generated by overfishing, climate change, coastal development, and pollution. In this context, weak institutions and poor enforcement are leading to the rapid degradation of marine resources (California Environmental Associates, 2018). In 2009, the Government of Indonesia (GoI) requested assistance from the United Nations Industrial Development Organization (UNIDO) to make national fisheries more competitive, equitable, and environmentally sustainable. During the following year, UNIDO carried out a series of studies and stakeholder consultations and concluded that the Indonesian seafood sector was high volume but low value that contributed to the degradation of the country’s marine resources. Most products were exported in bulk as raw products or sold in the national market with little added value. This prevented the realization of economic potentials across the value chain and particularly among fishermen and women (UNIDO, 2012). In 2010, UNIDO presented a program to make the fisheries sector more competitive, equitable, and sustainable. The program SMART-Fish for Indonesia—as it is known its short name—operated US$3.7 million from 2013 to 2019, in collaboration with the Indonesian Ministry of Marine Affairs and Fisheries (MMAF).²

UNIDO (2019) commissioned an evaluation close to the end of the program in 2019. The purpose of the evaluation was to assess the extent to which the program had contributed to fisheries that are more competitive, equitable, and environmentally sustainable. It was not realistic to expect that a relatively small program implemented for 5 years could resolve the challenges faced by a country with one of the largest and most complex fishing sectors in the world. The evaluation, therefore, focused on assessing the extent to which the program contributed to a development trajectory that makes efficient use of resources and generates benefits across the value chain in environmentally sustainable ways. The evaluation required a good understanding of the fisheries systems that the program targeted. The evaluation also developed a set of propositions that the program interacted with. The program included six significant components for each value chain (Figure 1).

Figure 1.

Transforming fisheries theory of change.

TOCs are best developed during program planning and in close interaction with the stakeholders. In the case of SMART-Fish, no TOC was developed during project preparation. The project preparation team carried out a detailed diagnosis of the fisheries sector in the country and identified problems in the sector and several root causes. As there was no explicit theory of change (TOC) for the program at design, during the evaluation inception phase, the evaluation and project teams jointly developed a draft TOC drawing from the project document and other analyses carried out during the preparation phase. The daft TOC was subsequently presented and discussed with stakeholders in a workshop in Indonesia with the participation of key stakeholders from government, fishers associations, and academia.

The elaboration of the TOC was as follows.

Identifying the System and the Long-Term Objective

The long-term objective of the program is to support the transition to fisheries that are more efficient and environmentally sustainable and that have increased value across the market chain, especially among the small fishermen and women. Each of the three value chains selected for the program (pangasius, seaweed, and pole and line capture tuna [P&L tuna]) is composed by closely linked subsystems (production and marketing) in which specific human and nonhuman components interact across multiple levels (local, provincial, national, and international). Pangasius, seaweed, and P&L tuna were selected because they are essential sources of revenue for small fishers across the country. Like other fisheries in the country, these three fisheries were also inefficient and characterized by practices that are destructive to the environment and are not likely to be sustainable over time. The project sought to introduce new practices that would make better use of natural resources and increase value generation, particularly among small fishers and farmers upstream in the value chain, in a way that was environmentally sound or at least reduced the environmental footprint. The three value chains are different in as far as they pertain to different species and stakeholders. They also have a distinct ecological, technological, organizational, market, and regulatory challenges—fish farmers grow pangasius mostly in inland ponds, seaweed farmers cultivate seaweed by the seashore, and P&L tuna fishers capture fish mainly in the sea.

Identifying the Relevant Domains and Enabling Conditions for Behavioral Change

The objective at this stage was to develop a set of propositions of the conditions that are likely to enable the behavior changes that, in the long run, would lead to the achievement of the program objectives. In the SMART-Fish evaluation, this stage was carried out jointly by the evaluation team and the project management team through brainstorming the critical enabling conditions and by clustering them under a small number of domains. During this stage, the team took into account lessons from previous evaluations of transformational interventions, which broadly pointed to five key domains: technological, policy and regulatory, institutional, financial, and sociocultural (Zazueta, 2017). The TOC was subsequently presented and discussed in a one-and-half-day workshop in Indonesia with the participation of key stakeholders from government, fishers’ associations, and academia.

The steps followed were:

Step 1 consisted of the brainstorming on enabling conditions. As indicated, this was the result of the brainstorming of the evaluation and the project technical team. The brainstorming responded to the question, “What are the conditions that will increase the generation of value across the value chain (specifically among fishers) in ways that are environmentally sustainable?” The team identified 32 of such enabling conditions.

Step 2 consisted of the clustering of the 32 enabling conditions and grouping each cluster into a domain. The 32 enabling conditions were clustered in six groups and labeled as domains: governance, production, markets and trade, financing, science and technology, and quality and standards (see Appendix for a list of the 32 enabling conditions clustered under the six domains).

Step 3 consisted of the cross-referencing of enabling conditions and development of a table that indicated the instances in which each of the 32 enabling conditions had an enabling function to the rest of the conditions. This step was carried out mostly by drawing in the expert knowledge within the evaluation and project management team. Step 3 identified 236 cases in which the 32 conditions have an enabling function to other conditions. Figure 2 presents a network diagram that includes the 236 interactions among the 32 conditions. This step provided a model of the interactions of enabling conditions across the system.

Figure 2.

Map of the Enabling Functions Among the 32 Enabling Conditions Affecting the System.

Step 4 consisted of running several tests to identify the conditions that had the most influence across the system. This analysis assumed that the influence of an enabling condition across the system was correlated with the reach and influence across the system. Using the program NodeXL, the team ran five network analysis tests to rank the influence of each of the 32 enabling conditions across the system. Three tests were particularly indicative of such influence: One is out-degree, which measures the number of times that a condition directly provides an enabling function to other conditions in the system. Direct connections and shortest paths are essential indicators of influence across the network. The higher the out-degree, the more significant number of conditions influenced across the system. Closeness centrality is a measure of the average shortest distance from each condition to the other conditions. A condition with lower closeness centrality score is therefore more influential in the system as it has the shortest distance to all other conditions. Eigenvector centrality measures the degrees of the conditions with which any given condition is connected to. This metric is similar to “degree,” but eigenvector centrality extends itself to calculate connections to the second degree of distance. It indicates how connected each of the conditions is across the system. Think of it as a way of determining how influential someone’s friends are. Therefore, a high eigenvector score means that a condition is linked to many other conditions with high eigenvector score. The purpose of evaluation was to understand the contribution of the program to changes in the most influential conditions. Out-degree is the most directly related test as it just measures direct influence. The out-degree test identified the following five enabling conditions as the most influential across the system. The five enabling conditions that have the largest influence across the system are as follows. Figure 3 presents the result of the test.

23. Awareness and common understanding of the challenges, opportunities, and trends in the fisheries sector

13. Sector policies conducive to sustainable fisheries development (including legal and regulatory frameworks)³

14. Inter-sectoral policy coherence and coordination

24. Robust science, technology, and innovation capacity that generates knowledge in the sector

15. Legal and regulatory framework conducive to sustainable development

Figure 3.

Influence of enabling conditions across the system.

This stage of analysis helped to begin setting boundaries of the evaluand and also helped to identify the relevant contextual aspects. The network diagram helps visualize the complexity of the system the program targeted. The network analysis software allowed to isolate the links for each of the 32 conditions. This ability to disaggregate the network to its components made it possible to trace the chains of influence across the system and allowed other stakeholders to confirm or dispute any such links. Network analysis helped the team identify a set of conditions that are likely to have the most influence across the system. The five high-ranking enabling conditions provided a manageable number of conditions that the evaluation could further explore to assess the extent of actual contributions of the program.

Identifying Spatial Scales

The evaluation team identified three sets of scales as the most relevant to understand how the three value chains work. One is market scales, including local, national, and global markets. Another is the governance or administrative scale, referring to hamlet, village, province, country, and global scale. In Indonesia, there are frequent mismatches in the objectives and capacities of agents operating in governance and administration at the scales of the town, province, and central government. Stakeholder objectives at the local scale or policy objectives at the provincial scales need not fully correspond to national policy objectives. The third scale is the ecological scale, which referred to the natural systems that differ according to the specific pertaining resources traded within each value chain.

Identifying Key Stakeholders

The evaluation team used the domains, enabling conditions, and scales previously defined as a framework to help identify the agents or key stakeholders in the system. It did this by responding to two questions: (1) Who are the agents that had a role, interest, or ability to achieve or obstruct any of the enabling conditions in the different scales? and (2) Who are the agents that could be affected (positively or negatively) by achieving such conditions? The team identified the following stakeholders or human agents: fishermen and women; fish feed traders; intermediaries/processors; national and international buyers; extension workers; government officials at the local, provincial, and national levels; and representatives of value chain associations. Based on this framework, the evaluation team selected key informant interviews across agents from different domains, scales, and geographies. Nonhuman agents included the resources targeted by the value chains, which include tuna fish, pangasius, and seaweeds. Other examples of relevant, not human agents, in this case, include pangasius diseases and the weather.

Criteria to Address Mismatches in Temporal Scales

Given the challenges involved, system transformations are likely to take much more time than the typical 5–7 years of projects or programs. Like many projects, the SMART-Fish sought to test, adapt, demonstrate, and promote the adoption of technologies and organizational approaches and helped develop the awareness that would contribute to changes in behavior among stakeholders consistent with the project goals. These projects were pilots that included trial at scales that were sufficiently small to allow for quick adjustments and learning. The TOC required attention to the conditions and mechanisms to ensure positive feedbacks that would bridge the gap between project duration and the time necessary for the broader adoption of the new behaviors that would redirect the system to the desired long-term objectives. One condition refers to the extent to which stakeholders perceive benefits and demonstrate a commitment to the new behaviors and approaches. A second condition refers to the point to which capacities are in place to continue supporting the trajectory shift to the policy objectives. A third condition refers to the extent to which efforts or actions of stakeholders seeking to bring about the different enabling conditions are sufficiently linked across scales and in mutually supportive ways.

A critical factor in achieving long-term changes beyond project duration is to put in place a transformation mechanism during project implementation. This mechanism that ensures the durability of the new trajectory consists of three elements: (1) mainstreaming, when information, specific lessons, or results of projects are incorporated into broader mandates and initiatives (e.g., governments, private sectors, or development agencies) such as laws, policies, regulations, and programs; (2) replication in which the supported initiatives are copied or adapted at other geopolitical or ecological scales, often in another geographical area or region; and (3) scaling up in which the supported initiatives are implemented at a larger geographic scale (Global Environment Facility Independent Evaluation Office/United Nations Development Programme Independent Evaluation Office, 2016; Zazueta, 2017).

Use of the Complex Systems–Based TOC to Assess the SMART-Fish Contribution to a Trajectory Consistent With the GoI Policy Objectives

The following are three examples of how SESs-based TOC and the use of tools developed by sophisticated system thinkers helped to evaluate the extent to which the project contributed to a trajectory consistent with the long-term program goals.

Contributions to Enabling Conditions

The SES-based TOC helped the evaluation team identify the ways that the project interacted with each of the value chains to help bring about the enabling conditions to redirect the system development trajectory toward the long-term goals of the project. After having confirmed the SES-based TOC model in a stakeholder workshop, the evaluation team carried out three focus groups with stakeholders, one for each of the value chains targeted by the program. Each group rated the state of the 32 enabling conditions before the project started and at the project completion. Subsequently, the groups rated the extent to which SMART-Fish contributed to the changes in these conditions.⁴

The stakeholders reported some differences among the three value chains. P&L tuna stakeholders reported the most improvement in the domains of trade and market, production, and science and technology, with some improvement in the domains of quality standards and governance and a modest improvement in the financial domain. This marked improvement across most domains of the P&L tuna value chain is a result of the support to this value by multiple donors over nearly 10 years. Stakeholders in the pangasius value chain also reported most conditions improving in the domains of trade and market, finances, and production, with significant contributions by SMART-Fish in the domains of trade and production, and less so in aspects related to governance. Seaweed stakeholders reported improvements in most domains, also with significant contributions by SMART-Fish.

When aggregating the responses on the three value chains, changes in the enabling conditions to the long-term objectives were most pronounced in the domains of trade and markets, governance, and production (Figure 4). These were three domains that the project targeted most actively and in which stakeholders reported SMART-Fish making its most substantial contribution. Pertaining trade and markets, stakeholders reported progress on nearly all the five enabling conditions identified in the TOC model, with the most progress taking place on the development of demand and effective marketing strategies. In the production domain, the most substantial contributions were to the adoption of technologies and best practices and the availability of inputs at competitive prices and quality. Stakeholders acknowledged the contribution of SMART-Fish in the domain of science, technology, and innovation but also reported that the progress made in this domain was relatively small compared to the needs in the area. Stakeholders considered that the project contributed to the changes in the domains of finance, and quality and standards were very modest.

Figure 4.

SMART-Fish and changes in domains.

The evaluation team also used the focus group data to assess the extent to which the project had contributed to the five key enabling conditions with the most influence on the system previously identified through the network analysis test: Condition 13—sector policies conducive to sustainable fisheries development; Condition 23—awareness and shared understanding of the challenges, opportunities, and trends in the fisheries sector; and Condition 24—robust science, technology, and innovation capacity that generates knowledge in the sector (Figure 5). Stakeholders reported the highest project contributions for Conditions 13 and 24. Stakeholders reported lower project contributions to Condition 14—inter-sectoral policy coherence and coordination, and Condition 15—legal and regulatory frameworks supportive of sustainable fisheries. It is interesting to notice that SMART-Fish contributions closely match the conditions in which most changes had taken place, indicating that it is very likely that the project was a factor in the progress made in all the five conditions.

Figure 5.

SMART-Fish contribution to five catalytic conditions.

Progress Toward the Long-Term Objectives of the Program

The evaluation team also used the SES-based TOC to assess the contribution to the long-term objectives of the program (in the form of more efficient use of resources that are produced in environmentally sound ways and which also generate benefits across the value chain, particularly among the fishers). The best aquaculture practices promoted by the program were related to the nutritional needs and environmental conditions for pangasius, algae, and P&L tuna. Initially, the program focused on the introduction of innovations that were simple and made economic sense to the farmers and fishers. For example, in the case of pangasius, the focus was on the reduction of the use of antibiotics by introducing probiotic practices that reduced the ammonia content in the ponds. Pangasius support also included the development of feed formulas that used ingredients locally available to the farmers. In the case of algae, the project focused on seed selection and placement of wracks to allow for water circulation, cultivation periods, and postharvest treatments. In the case of P&L tuna, the project supported the development of new live bait formulas that substituted the more expensive marine captured live bait for cultivated fish. These practices helped improve efficiency and economic benefits to farmers and fishers while at the same time reduce waste and environmental degradation. Despite the accomplishments achieved, when considering the interaction among the broader scales and the behavior of stakeholders not engaged in the project, the development trajectory of the value chains appears uncertain. P&L tuna is mainly produced for the international market, and it is well-integrated from the local level to the international levels. The foreign buyers, while not directly participating in the program, were engaged in similar initiatives supported by several philanthropic foundations and thus had already moved in the direction of purchasing sustainably harvested fish. The program worked with the fisher’s association to find ways to reduce operating costs for P&L tuna, such as by developing new formulas for live bait. Nevertheless, P&L fisheries face intense competition from illegal, unreported, and unregulated (IUU) fishing and declining fish stocks. Despite the improvements achieved by the project and other initiatives, unless IUU fishing is addressed, a downward trend is likely to continue in the P&L tuna fishery with declining stocks and benefits for the human populations.

In the case of seaweed, Indonesia is the second global producer of seaweed, mostly for the international market. New market trends for Indonesian seaweed, in particular as a result of the growing demand in China, have influenced the sector in significant ways. While in the past there was a robust domestic sector that processed seaweed for different products, over the last few years, the market had been increasingly cornered by one international buyer that supplies markets in China which do not differentiate between high or low quality of the product, which acts as a disincentive for improvement of quality. An underlying cause of this problem is the accuracy of data and uncoordinated policy. The seaweed industry believes unrealistically high official estimates of national production capacity led the government of Indonesia to adopt a welcoming policy to attract foreign investment in the processing sector, whereas in reality, the national processing sector was already struggling to secure supply of raw materials for processing.

The program was able to demonstrate ways to improve productivity and quality, temporarily raise the income of seaweed farmers, and contain harmful environmental impacts through better aquaculture practices. But the program did not address a critical level in the value chain: the international market driving the demand for seaweed. These new international seaweed buyers are currently displacing domestic carrageenan seaweed processors. By controlling the international market outlets, this new buyer is steering the market toward a more extractive, environmentally harmful trajectory. The new buyer is interested in bulk and gives no attention to quality, which contributes to a long-run trend toward lower value generation along the upstream sections of the value chain. In the short term, the program was successful in introducing more efficient and environmentally sound production practices. The program also improved the seaweed quality, which resulted in higher income for the participating small seaweed farmers. Nevertheless, trends are taking place at broader temporal and spatial scales that are steering the system trajectory away from the program’s long-term objectives.

Pangasius in Indonesia is mostly produced for the fresh local markets and the national frozen fillet market. The pangasius market chains tend to be more regional and fragmented, and fresh market fish is often of low quality. The program introduced the best aquaculture practices that improve the fish quality and, at the same time, promoted the use of probiotics to enhance the quality of water and reduce the environmental footprint. The program, in coordination with the MMAF, developed a branding campaign and the adoption of standards that resulted in a significant expansion of the frozen fillet markets and a demand for a product at a higher price. The program also helped the GoI open new international markets in the Middle East. But the uptake of the latest technology differed greatly depending on social and market conditions affecting the agents. In Tulungagung where the value chains were oriented to national markets of frozen fish fillets, processors, traders, and farmers saw an opportunity for higher prices and market expansion. Farmers, endeavored to meet the higher quality standards, rapidly adopted the methods and modes of stakeholder interaction proposed by the project. The best aquaculture practices introduced by the program led to better sanitation and water quality in the ponds and healthy fish. Traders and industry, interested in meeting the national standards for frozen fish fillet, were also willing to pay higher prices to pay a premium to farmers for quality. In regions such as Mauro Jambi where the fresh pangasius regional markets are not integrated into the frozen fillet national market, the project had a lower success rate in improving the prices for farmers or in expanding market opportunities. Fresh local fresh pangasius markets have much lower quality expectations than the national freezer markets. Local fresh markets required high volume but did not reward quality. As opposed to Tulungagung, where farmers carefully monitored water quality and the fish health, in Mauro Jambi, farmers adopted extensive farming practices. Low water quality and inadequate sanitation often render the conditions of the ponds beyond the thresholds that fish could tolerate, which resulted in high mortality rates. Farmers reacted by changing their practices very selectively and mostly to reduce fish mortality. Still, they had no incentive to apply the full technological packages. Moreover, Mauro Jambi farmers were heavily indebted to feed suppliers and fresh fish traders who had high stakes in selling more feed to farmers and supplying low-cost fish to compete in the local markets. Without massive financial support—a component that could not be directly targeted by the project—the farmers were unable to break away from the established relations. They remained captive to strategies that favored fish feed suppliers and fish traders. Two key factors that affected the degree to which the project generated the intended outcomes or benefits were (1) the extent of the match or mismatch of the interests and objectives of the agents and (2) the extent to which human production strategies were compatible and conducive to the natural conditions required by seaweed, tuna, and pangasius.

Addressing the Temporal and Spatial Scales: Mechanisms to Continue to Support a Development Trajectory Consistent With the Program Long-Term Objectives

The program was designed to develop and test practices that, if widely adopted, could change the development trajectory of the system. By design, it reached only a small fraction of the millions of fishers and farmers in Indonesia. The desired transformation requires adoption by many more agents at a much broader scale, which can only happen over time. The project supported the capacity development of three stakeholders’ associations, one in each of - the value chains, aiming at improving communication, exchange, and collaboration among stakeholders in the chain. Drawing on the configuration of domains and scales identified earlier, the evaluation team held a workshop with stakeholders to assess the extent to which the stakeholder associations would continue to function as a mechanism supporting the trajectory of the long-term project objectives after the project ended. A key consideration was the extent to which the associations included the key agents operating at different domains and scales. Stakeholders in the seaweed and pangasius associations concluded that their associations were essential to ensure the exchange of information, development of new partnerships, and engagement in the policy dialogue with the Ministry of Fisheries and Marine Affairs. They also drew a network diagram which indicated that members are better linked and interconnected than the P&L tuna association. Nonetheless, the self-assessment concluded that all three associations excluded some stakeholders who were key to overcome challenges. Conspicuous among the missing stakeholders were institutions from the financial sector and those government departments responsible for the construction and maintenance of infrastructure. These two factors present significant constraints to the further development of efficient value chains. As indicated earlier, the program did not reach a key stakeholder who is having a substantial impact on the seaweed value chain and is driving the system in a development trajectory that is not consistent with the long-term objectives of the program. The program also helped to build capacities in local universities and with MMAF to expand the extensionists to continue to support farmers and replicate the use of the best aquaculture practices introduced by the program. Universities also developed some research capacity to support the improvement of technology and to inform regulations. Of the three value chains, the Government of Indonesia favored the strengthening of the pangasius farm extension system as part of its food security policies. Extensionists in seaweed were few to start with, and the number remained small as it was not a high priority for the government to provide subsidized technical assistance to an export commodity.

Conclusion

This article has presented a method to construct TOC based on the SES and complex system thinking. SES scholars have developed a set of concepts to help understand and communicate how human and natural subsystem interactions give rise to a development trajectory. The concepts of domains, scales, agents, and emergence provide valuable tools to construct models to identify the chains of causality among the enabling conditions conducive to long-term objectives. This article illustrates how an SES-based TOC is useful to understand how a project or program interacts with the phenomena and helps identify the information needed to assess the extent to which a given intervention contributes to the intended long-term objectives. The article has also illustrated how to use network analysis to develop models of the interactions and chains of causality among the enabling conditions affecting the system. A key lesson in the design and evaluation of transformational interventions is that it is not enough to support specific enabling conditions. It is also critical to ensure that conditions and capabilities are integrated and linked in a mutually supportive way. It is also crucial to ensure that the initiative reaches all relevant scales of the system and that mismatches of stakeholder interests, between the social and natural subsystems, and policies are addressed.

While the experience presented in this article pertains to a TOC developed during an evaluation, TOCs are most useful when they are developed in the early planning stages of an intervention. Modeling the interaction of the components of the system (e.g., domains, scales, and agents) helps understand the phenomena as a dynamic system. Thus, evaluators and project managers can use the TOC during implementation as a tool to develop a better understanding of the phenomena. Subsequently, it is also the job of the evaluators and project managers to adjust the TOC and the evaluation or the development intervention.

Another important observation concerning the TOC example presented in this article is that it did not have the participation of the key stakeholders from the start. This is explained by the fact that the evaluation team was introducing in the organization the complexity-based TOC approach as a multipurpose tool for planning, monitoring, and evaluation of programs. The availability of the stakeholders was also a factor, as the engagement from the start was likely to be time-consuming. Still, it is crucial to engage key agents (the stakeholders) from the beginning to capture and reflect in the TOC the multiple interests and objectives involved accurately. Yet, when such engagement is not possible, the evaluation and management teams can hold a stakeholder workshop to discuss and modify the TOC. Once the key stakeholders have developed a good understanding of the TOC, the management team can use it as a tool to engage the stakeholders in planning, monitoring, and evaluation. Learning from this evaluation, the management team has started the planning of a follow-up project with the formulation of a TOC. This is taking place through a series of stakeholder workshops. The follow-up project also includes a technical working group (TWG) formed by representatives of the diverse stakeholders involved in the project who will be periodically engaged in the review of the project. The evaluation and management teams are also collaborating with the TWG in the development of a monitoring system based in the TOC, which the program will use to trace the accomplishments, confirm chains of causality, and adapt the program to address changing circumstances or new information.

Appendix

Domains and 32 Conditions Enabling the Policy Trajectory Change

Production

Infrastructure and utilities to support production processes are in place.

Suitable technologies and best practices adopted by relevant actors across the sector

Inputs of the necessary quality, competitive price, and quantities available to producers

Investment in sustainable fisheries takes place.

Fisheries perceived as attractive investments and suitable business models for sustainable fisheries exist.

Qualified labor is available.

Compliance on quality, environmental, health, and labor standards across the production process

Trade (market)

Effective demand for sustainable fisheries products

Effective market development strategies are implemented by the sector.

The market recognizes that the sector complies with quality, standards, and costs and delivers requirements of target markets.

Functioning infrastructure (physical and virtual) facilitates the trade of fisheries products.

LINK TO Production: Market intelligence is available to producers.

Governance

Sector policies conducive to sustainable fisheries development

Inter-sectoral policy coherence and coordination

Legal and regulatory framework supportive of sustainable fisheries

Trade agreements favorable to sustainable fisheries

Capable institutions and clear division of roles and responsibilities

Effective control and surveillance of fisheries

Finance

Public and private financial resources available for investment in the sustainable development of fisheries

Positive investment development horizon perceived for fisheries

Incentive structures to encourage investment in fisheries in place

Financial business models suitable for fisheries

Science and technology

Awareness and common understanding of the challenges, opportunities, and trends in the fisheries sector

Robust science, technology and innovation capacity that generates knowledge in the sector

The capacity of the fishery sector to respond and adapt to megatrends

Incentives structure that encourages adaptation

Capacity to adapt to emerging standards

Quality and standards system

Globally harmonized national standard

Recognized and affordable conformity assessment services are available.

Services available to promote and support the compliance on quality, environmental, health, and labor standards across the production process

Competent quality infrastructure institutions (to support conformity assessment services)

Quality policy in place

Footnotes

Declaration of Conflicting Interests

The author(s) declared potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The views expressed herein are those of the author(s) and do not necessarily reflect the views of the United Nations Industrial Development Organization.

Funding

The author(s) received no funding for the research, authorship, and/or publication of this article.

ORCID iD

Aaron Eduardo Zazueta

Thuy Thu Le

Notes

References

Allen

C. R.

Garmestani

A. S.

(2015). Adaptive management of social-ecological systems: The path forward. In Garmestani

A. S.

(Ed.), Adaptive management of social-ecological systems (pp. 255–262). Springer Netherlands. https://doi.org/10.1007/978-94-017-9682-8_14

Anderies

Janssen

Ostrom

(2004). A framework to analyze the robustness of social-ecological systems from an institutional perspective. Ecology and Society, 9(1). https://doi.org/10.5751/ES-00610-090118

Arnold

R. D.

Wade

J. P.

(2015). A definition of systems thinking: A systems approach. Procedia Computer Science, 44(2015), 669–678.

Bamberger

Vaessen

Raimondo

(2015). Dealing with complexity in development evaluation: A practical approach. Sage.

Brondizio

E. S.

Ostrom

Young

O. R.

(2009). Connectivity and the governance of multilevel social-ecological systems: The role of social capita—Google search. Annual Review of Environment and Resources, 34, 254–278. https://doi.org/1543-5938/09/1121-0253

Brugha

Varvasovszky

(2000). Stakeholder analysis: A review. Health Policy and Planning, 15(3), 239–246.

California Environmental Associates. (2018). Trend of marine resources and fisheries management in Indonesia. A 2018 review. https://www.ceaconsulting.com/wp-content/uploads/Indonesia-Report-Executive-Summ-6.26-1.pdf

Cash

Adger

W. N.

Berkes

Garden

Lebel

Olsson

Pritchard

Young

(2006). Scale and cross-scale dynamics: Governance and information in a multilevel world. Ecology and Society, 11(2). https://doi.org/10.5751/ES-01759-110208

Cash

W. D.

Adger

Berkes

Garden

Lebel

Olsson

Pritchard

Young

(2006). Scale and cross-scale dynamics: Governance and information in a multilevel world. Ecology and Society, 11. http://www.ecologyandsociety.org/vol11/iss2/art8/main.html

10.

Clement

(2013). For critical social-ecological system studies: Integrating power and discourses to move beyond the right institutional fit. Environmental Conservation, 40(1), 1–4.

11.

Couture

Valcartier

(2007). Complexity and chaos—State-of-the-art; overview of theoretical concepts. Minister of National Defence, Canada.

12.

Cumming

G. S.

Cumming

D. H. M.

Redman

C. L.

(2006). Scale mismatches in social-ecological systems: Causes, consequences, and solutions. Ecology and Society, 11(1). https://www.jstor.org/stable/26267802

13.

Davies

(2018). Representing theories of change: Technical challenges with evaluation consequences. Journal of Development Effectiveness, 10(4), 438–461. https://doi.org/10.1080/19439342.2018.1526202

14.

Dhina

(2017, May 15). ‘Eating fish campaign’ in Kupang: Improving Indonesian women’s health and nutrition. MAMPU. http://www.mampu.or.id/en/news/eating-fish-campaign-kupang-improvingindonesian-women%E2%80%99s-health-and-nutrition

15.

Dooley

K. J.

(1996). A nominal definition of complex adaptive systems. Chaos Network, 8(1). http://s3.amazonaws.com/academia.edu.documents/44741737/A_nominal_definition_of_complex_adaptive20160414-11368-137c9kw.pdf?AWSAccessKeyId=AKIAIWOWYYGZ2Y53UL3A&Expires=1485095247&Signature=7uiIQmUXha1FwTkjUm%2FJGIyp3G0%3D&response-content-disposition=inline%3B%20filename%3DA_nominal_definition_of_complex_adaptive.pdf

16.

Feeny

Berkes

McCay

B. J.

Acheson

J. M.

(1990). The tragedy of the commons: Twenty-two years later. Human Ecology, 18(1), 1–19. https://doi.org/10.1007/BF00889070

17.

Fischer

Gardner

T. A.

Bennett

E. M.

Balvanera

Biggs

Carpenter

Daw

Folke

Hill

Hughes

T. P.

(2015). Advancing sustainability through mainstreaming a social–ecological systems perspective. Current Opinion in Environmental Sustainability, 14, 144–149.

18.

Folke

Biggs

Norström

A. V.

Reyers

Rockström

(2016). Social-ecological resilience and biosphere-based sustainability science. Ecology and Society, 21(3), 41.

19.

Folke

Carpenter

Walker

Scheffer

Chapin

Rockström

(2010). Resilience thinking: Integrating resilience, adaptability and transformability. Ecology and Society, 15(4). https://doi.org/10.5751/ES-03610-150420

20.

Food and Agriculture Organization of the United Nations. (n.d.). Fisheries and aquaculture. Country profile. Indonesia. Retrieved February 4, 2020, from http://www.fao.org/fishery/facp/IDN/en

21.

Gibson

C. C.

Ostrom

Ahn

T. K.

(2000). The concept of scale and the human dimensions of global change: A survey. Ecological Economics, 32(2), 217–239.

22.

Global Environment Facility Independent Evaluation Office/United Nations Development Programme Independent Evaluation Office. (2016). Impact evaluation of GEF support to protected areas and protected area systems. http://www.gefieo.org/sites/default/files/ieo/evaluations/files/BioImpactSupportPAs-2016.pdf

23.

Hartman

De Roo

(2013). Towards managing nonlinear regional development trajectories. Environment and Planning C: Government and Policy, 31(3), 556–570.

24.

Holland

(1995). Hidden order: How adaptation builds complexity. Basic Books. https://books.google.com/books?hl=en&lr=&id=ukc3sOTMfgUC&oi=fnd&pg=PR17&dq=Hidden+Order:+How+Adaptation+Builds+Complexity&ots=u5rkTQpc-X&sig=bgMXfHdFC-QWQFcarU4LjGAZSf8#v=onepage&q=Hidden%20Order%3A%20How%20Adaptation%20Builds%20Complexity&f=false

25.

Holland

(2006). Studying complex adaptive systems. Journal of Systems Science and Complexity, 19, 1–8.

26.

Ives

C. D.

Freeth

Fischer

(2020). Inside-out sustainability: The neglect of inner worlds. Ambio, 49(1), 208–217.

27.

Jerneck

Olsson

Ness

Anderberg

Baier

Clark

Hickler

Hornborg

Kronsell

Lövbrand

(2011). Structuring sustainability science. Sustainability Science, 6(1), 69–82.

28.

Koleros

Mayne

(2019). Using actor-based theories of change to conduct robust contribution analysis in complex settings. Canadian Journal of Program Evaluation, 33(3), 292–315.

29.

Laszlo

Krippner

(1998). Systems theories: Their origins, foundations, and development. Advances in Psychology, 126, 47–76.

30.

Levin

(2003). Complex adaptive systems: Exploring the known, the unknown and the unknowable. Bulletin of the American Mathematical Society, 40(1), 3–19.

31.

Ostrom

(2009). A general framework for analyzing sustainability of social-ecological systems. Science, 325. http://science.sciencemag.org/content/325/5939/419

32.

Preiser

Biggs

De Vos

Folke

(2018). Social-ecological systems as complex adaptive systems: Organizing principles for advancing research methods and approaches. Ecology and Society, 23(4). https://doi.org/10.5751/ES-10558-230446

33.

Selomane

Reyers

Biggs

Hamann

(2019). Harnessing insights from social-ecological systems research for monitoring sustainable development. Sustainability, 11(4), 1190. https://doi.org/10.3390/su11041190

34.

United Nations Industrial Development Organization. (2012). Project document for the increasing trade capacities of selected value chains with in the fisheries sector in Indonesia.

35.

United Nations Industrial Development Organization. (2019). Independent terminal evaluation. SMART-Fish program: Increasing trade capacities of selected value chains within the fisheries sector in Indonesia”. https://www.unido.org/sites/default/files/files/2020-01/120110_Indonesia_SMART%20Fish_Terminal%20evaluation.pdf

36.

Walker

B. H.

Holling

C. S.

Carpenter

S. R.

Kinzig

A. P.

(2003). Resilience, adaptability and transformability in social-ecological systems. Ecology and Society, 9(2, Article 5). https://www.researchgate.net/publication/42764046_Resilience_Adaptability_and_Transformability_in_Social-Ecological_Systems

37.

Wilbanks

T. J.

(2007). Scale and sustainability. Climate Policy, 7(4), 278–287.

38.

Zazueta

A. E.

(2017). Principles for the development of integrated climate change and chemicals and waste (A STAP information document). The Global Environment Facility Scientific and Technical advisory Panel. http://stapgef.org/sites/default/files/publications/Final_Principles%20for%20%20Development%20of%20Integrated%20CCM%20and%20ChW%20Projects_October%202017.pdf