Abstract
In 2020, for the first time in history, the whole world has shared a crisis of severe proportions. The COVID-19 pandemic has been disruptive and traumatic. Its multiple interconnected shocks—physical, economic, financial, psychological, and political—have shattered societies’ stability and comforts. It has brought out the best and the worst in nations and their leaders. It has made some move apart while drawing together others. It has made us all reassess our assumptions about life and living, about priorities and what matters, and about our roles in families, communities, and the world.
Now, the aftermath brings a unique opportunity—a chance for all humanity to reimagine and reinvent our societies and their relationship with the natural ecosystems on which everything depends, redefine our personal and professional commitments and priorities, and revisit our worldviews and how we conduct ourselves. This is also the chance to change how the field of evaluation is viewed and developed.
In our view, seizing this opportunity is essential. Calls for the transformation of systems on which humanity depends have been growing rapidly and with increasing urgency (Club of Rome, 2020; Diaz et al., 2019; Sachs et al., 2019). The urgent need for transformational change is at the core of the 2030 Agenda for Sustainable Development, the global compact to which all nations signed up in 2015 (United Nations [UN], 2015). Yet in his last report to the UN General Assembly, in July 2020, Philip Alston, the UN special rapporteur for extreme poverty and human rights, argues powerfully that there is no chance that the world will achieve the sustainable development goals by 2030 (Alston, 2020), noting the “scandalous lack of ambition” in the World Bank’s definition of the international poverty line (IPL) of US$1.90 a day—a “staggeringly low standard of living, well below any reasonable conception of a life with dignity.” He berates the “celebratory accounts” of global efforts to alleviate poverty, given that the vast proportion of the billion people lifted out of poverty over the last three decades is due to the extraordinary achievements of one country, China, where the number of people below the IPL dropped from more than 750 million in 1990 to 10 million in 2015 (World Bank, 2020).
Furthermore, the hope that humanity can avert emerging disasters even more serious than the COVID-19 pandemic is rapidly fading. Risks are rapidly multiplying (World Economic Forum, 2020), the result of humanity’s actions that have heralded in the Anthropocene—an era defined by the overshoot of planetary boundaries (Rockström, 2009) as a result of runaway capitalism, rapid population growth, overconsumption, and the notion of growth at all costs. Severe biodiversity loss, staggering inequalities, and rapidly advancing climate change have followed (Ripple et al., 2020). With the Global North responsible for 92% and the Global South for only 8% of climate breakdown (Hickel, 2020), low-income countries are bearing the brunt of what some refer to as “climate apartheid” or “atmospheric colonization” (Alston, 2020). Under these circumstances, the field of evaluation cannot be allowed to evolve slowly and incrementally. There is no greater challenge to evaluation professionals than showing the value of this field of work at this critical time. It cannot be business as usual, with only slight adjustments for the more challenging nature of fieldwork and increased reliance on technology and remote work as a result of the constraints posed by the pandemic. Evaluation specialists today carry the responsibility to revisit, redesign, and reconfigure, with a sense of urgency, evaluation theories, and practices to be valuable and essential instruments in support of the large-scale, transformative changes our societies and ecosystems need. And our responsibility does not end there. We have to become adept at demonstrating and advocating for our value in ways that are credible and visible among those with influence and, especially, among those intended to benefit from our contributions. Unless we can show that evaluation can powerfully contribute to the new world humanity has to create, it is better that our craft slowly fades into irrelevance rather than being a persisting obstruction to the transformations needed.
Evaluation can be deployed in two capacities related to large systems change and transformation. First, to help understand the results and implications, from the local to global scale, of ongoing transformations (such as the COVID-19 pandemic, climate change, and the effects of the Fourth Industrial Revolution; see Schwab, 2018) and how to deal with them. And second, to inform and provide ongoing developmental feedback for engineered systems transformations, such as efforts to escape poverty traps, to move to an ecological civilization, to reduce inequalities, and to eliminate racism and the undervaluing of the other.
In this process, evaluations can be transformative. Furthermore, if it is to help address the complex challenges the world faces today, the field of evaluation itself will have to transform or, at the very least, accelerate its evolution. This has been argued by some of the most prominent voices in evaluation, such as Schwandt (2019), in a powerful keynote speech on “postmodern evaluation” at the 2018 European Evaluation Society Conference in Thessaloniki, Greece; Patton (2019), in Blue Marble Evaluation; and Picciotto (2020), in a recent paper on a complexity framework for transformative evaluation. It is also present in the pioneering work of the growing number of indigenous evaluators around the world (Cram et al., 2018), as well as in the growing engagement by evaluators who share the values underpinning South–South Cooperation among countries in what is known as the “G77 + China Group” (UN Office for South-South Cooperation, 2020). The G77 is a UN geopolitical economic designation, which refers to the group of least developed countries. It originally included 77 such countries but has since expanded to include approximately 130 low-income countries in what is loosely referred to as the Global South (although not formally part of the group, it includes China in all its deliberations). South–South Cooperation evaluation complements the much more established and dominant evaluation of North–South Cooperation of the international aid and development programs offered by donors.
Prominent tensions and shifts in awareness and action worldwide indicate the types of efforts to which evaluation must contribute on a much larger scale. “Interdependencies,” “systems,” and “complexity” are words suddenly on everyone’s lips. Powerful calls for more innovative finance and responsible investment are growing. Social entrepreneurs concerned about the destruction of ecosystems and the overshoot of planetary boundaries are forming massive networks of individuals and organizations that can act as change agents in transformation systems. Prominent economists are calling for fundamental changes to economics, a discipline and field of work that often determines the fate of nations. Leaders of societies caught in poverty traps have been learning from remarkable success stories in countries that did not follow conventional models of development, while protests in economically rich societies are sparked by exploding frustration with ineffective and/or corrupt political systems as well as by excessive inequalities brought about by unfettered capitalism taken to the extreme.
Unfortunately, in the evaluation field, conventional results-based management, linear theories of change, and an obsession with measuring and attributing predetermined impacts have given a false sense of control when we should have been dealing with a world defined by uncertainties and interdependencies. We believe, in spite of established resistance, that to be effective, every designer and implementer of policies, projects, and programs and every evaluation commissioner and evaluator should be conversant with at least the basic principles of systems thinking and complexity science.
Yet there are also many encouraging initiatives in the field of evaluation, which offer the potential for more impactful contributions. The underutilized power in the global evaluation architecture can be unlocked through synergistic action by key initiatives such as EvalPartners, the International Organization for Cooperation in Evaluation, the International Development Evaluation Association, the Eval4Action campaign, the emerging Global Evaluation (Capacity) Initiative, and the International Evaluation Academy. BetterEvaluation, a not-for-profit organization that operates globally with the mission to work collaboratively to create, share, and support the use of evaluation knowledge, is an example of a powerful on-line resource that can help create common understanding around the world and support learning that will enable evaluation knowledge to stay current. Evaluators are discussing the importance of new technologies such as artificial intelligence, blockchain, and remote sensing. Several evaluation conferences have had transformation as a theme, and many articles and books have been written about the incorporation of systems and complexity concepts in evaluation practice. Systems-informed approaches such as realist evaluation, Developmental Evaluation, Principles-Focused Evaluation, Dynamic Evaluation, and Blue Marble Evaluation—especially if coupled with a focus on social justice as espoused in Donna Mertens’s work on Transformative Evaluation—are important guideposts. Systems-informed methods are being developed and efforts are advancing to build bridges across sectors with similar interests—in the philanthropic government and the private sectors and among social impact entrepreneurs and scientific researchers.
Before the COVID-19 pandemic, one could observe the growing trends in the field of evaluation reflecting the recognition of a need for transformation and bold action but still operate in a business as usual context. That is no longer viable. We face a new reality, and our field must catch up with the rapidly changing new normal.
What we have sketched in this introduction represents our rationale for the focus in this section on the intersection of evaluation and transformation and responds to this new-normal zeitgeist. As in this inaugural American Journal of Evaluation section on International Developments in Evaluation, we intend future offerings in this section to present the voices of authors (familiar and new) representing perspectives from countries and cultures around the world, from inside and outside the field of evaluation. In this issue, we hear perspectives from authors from Latin America, Asia, Africa, Europe, and North America. In subsequent issues, we will seek to present voices from a wide range in the Global South and Global North and from indigenous societies around the world.
This inaugural section highlights some of the fundamental aspects we have to consider when advancing our field in future: the criteria that determine what and how we evaluate when dealing with transformations, the implications of complex systems approaches and society–nature linkages when developing and using theories of change, how we conceptualize transformational change and the supportive role of evaluation, and the need to understand the role of evaluation in fostering living systems leadership. The articles are not connected in terms of their conceptualization and proposals for practice. Instead, they display some of the diversity of experiences and perspectives that have been shaping evaluators’ engagement with this emerging and important field of work.
In the first article, “Evaluation Criteria for Evaluating Transformation: Implications for the Coronavirus Pandemic and the Global Climate Emergency,” Patton challenges the global evaluation community to move well beyond the ubiquitous business as usual Organization for Economic Cooperation in Develoment's Development Assistance Committee (OECD DAC) criteria. The debates that accompanied the recent revision of these criteria, and the results of the revision, disappointed those wishing for a more progressive stance that encourages the use of evaluation criteria more in tune with the demands of our new normal. As an illustration of the type of thinking that is needed, Patton proposes six criteria to guide evaluations when dealing with major systems transformations, with many implications for practice. Among others, transformation-specific criteria have to reflect and highlight the scale, scope, urgency, and challenges of the problems to be addressed. Evaluators will be expected to assess trajectories toward transformation rather than determine whether transformations have occurred. Application of the criteria will change how we think about how change happens; truly transforming systems requires integrating several theories of change to build critical mass transformational tipping points. Evaluators will have to engage in practice with complex systems concepts, with full-cost accounting that calculates the more comprehensive costs and benefits of changes that have taken place, adopt a dynamic rather than static understanding of sustainability, and reflect systems-level diversity, equity, and inclusion as design features from the start.
The second article, “Advancing Evaluation and Learning on Transformational Change” by Joseph Dickman, Anna Williams, and Regan Smurthwaite, summarizes the first phase of the ongoing work of the Transformational Change Learning Partnership (TCLP) established by the Climate Investment Funds (CIF). It has been a pioneering evidence-based learning and collaboration on transformational change in the area of climate action, using the experiences of CIF with its more than US$8 billion investment that since 2008 has spanned over 300 projects across 72 countries. Apart from their novel approach to establishing a platform for systematic collective learning about systems-level transformational change, the TCLP developed a set of framings for evaluating transformational change in the context of climate change. They consist of dimensions—that is, key qualities that have to be displayed for change to be considered transformational or on a potential path toward transformation, stages of advancement toward transformation, and signals of transformational change that highlight progress along the dimensions.
The third article, “Development Trajectories and Complex Systems-Informed Theories of Change” by Aaron Zazueta, Nima Bahramalia, and Thuy Thu Le, focuses on theories of change using a social-ecological systems (SES) perspective—that is, social and ecological systems that emphasize the relationship of humans with nature, display resilience and complexity, and are linked through feedback mechanisms. They tackle some of the challenges that development practitioners face when trying to develop sufficiently comprehensive, yet manageable, interventions that can lead to desired long-term systemic changes that honor the relationship between people and nature. Policy makers and practitioners face considerable challenges when working toward the sustainable development goals. Among others, they need to have some inkling of the extent to which such interventions are contributing to a development trajectory in line with broader policy objectives. Based on an evaluation of a program in the fisheries sector in Indonesia, the authors introduce a methodology to construct environmentally sensitive theories of change for complex development processes based on the concept of SES. Commissioned by the UN Industrial Development Organization, the evaluation focused on assessing the extent to which the program contributed to a development trajectory that made efficient use of resources and generated benefits across the value chain in environmentally sustainable ways. The article highlights the importance of designing and evaluating interventions that may contribute to transformation with due consideration of the enabling conditions, all relevant scales of a system, and the mutually supportive integration and linking of conditions and capabilities. It also considers potential mismatches of stakeholder interests and mismatches between the social and natural subsystems and policies.
In the final article, “Valuing Those Who Lead for Transformation in Living Systems” by John Atkinson, Florence Lasbennes, and David Nabarro, we receive a call to action proposed by authoritative voices from outside the global evaluation community. We hope to include such reflections also in future sections to help stimulate interest and debate on issues where evaluation still needs to demonstrate its value. David Nabarro is a highly influential expert on global health and at present a special envoy to the COVID-19 pandemic response of the World Health Organization; with his colleagues, they represent the Skills, Systems, and Synergies for Sustainable Development (4SD) initiative. Their article argues for the critical importance of living systems leadership, where leaders have the mindset and behaviors that allow them to be in tune with the health of the systems they lead and that foster understanding of the skills necessary to design and adapt their approaches for the benefit of societies and ecosystems. They appeal to the evaluation community to do more to understand and evaluate the performance of this type of leadership and to use the function of, and findings from, evaluations to inspire this type of leadership. We trust this call to action will further inspire many already perceiving the need for moving in this direction and create awareness of the need to do so among others. We also hope that it will help catalyze leadership interactions within the global evaluation community and across diverse, yet like-minded, disciplines. We believe that in practicing elements of living systems leadership ourselves, we can also help nurture essential transformations in the field of evaluation.
We have developed this inaugural section to provide insight into international developments around some of the most important and timely issues in the evaluation field globally today. Multiple cascading crises have put us in a race against time. As Dickman et al. aptly noted: “If ever there was a moment to seize new frontiers and possibilities, armed with our best tools and the knowledge that we are in unprecedented and uncharted territory, it is now.” We trust that these articles will inspire you, dear reader, to help accelerate progress in the evolution of the field of evaluation and multiply the value that evaluation can offer the world. “We now live and work in a business as unusual world, a post-normal world, a global emergency world, a time-is-running-out world,” Patton reminds us. Let us all work with a sense of audacity and urgency to make the field of evaluation the best it can be in service of this critical moment in time.