Abstract
In today’s talk, I will focus on the developing context of evaluation over the next couple of decades, and I will approach this from my own areas of expertise. This context is challenging because the real question is not only how we can evaluate better, but also how we can contribute to the transition towards sustainability in times of urgency.
From a global sustainability perspective, we are facing not only unprecedented challenges, but also improved knowledge. Many of us have seen the latest International Panel on Climate Change (IPCC) report which is quite alarming. Then there is the Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services (IPBES) report which is equally alarming and talks about the sixth great extinction. Perhaps lesser known is the International Resource Panel, of which I am a member, which looks at the fundamentally unsustainable resource use in our systems of production and consumption, and the impacts that has on the planet. And I would here also mention the World Health Organization with its work linking environment and climate change to health issues.
Taken together, these add up to a context of urgency. We find ourselves in a pivotal decade in which windows of opportunity to deal with these matters are closing. We are already seeing ‘irreversibilities’ (irreversible changes); we are seeing tipping points in natural and social systems, and all of these issues are interconnected. So that is the context.
In terms of policy and political responses at the global level, I think it is fair to say that the Sustainable Development Goals (SGDs) are probably the most integrated and accepted framework to think about a future that can deal with these issues. And it is also fair to say that we are now actually 30 years on from the SDGs – we already had SDGs in 1992 at the Rio Conference, just that back then they were called Agenda 21.
The question is this: why are we less sustainable now than we were in 1992, even when we had very similar goals to those we see now in the current SDGs introduced in 2015? There are reasons for this failure, and I think the evaluation community can contribute to understanding these reasons better. So the question then becomes: what do we need to change? We are not currently on a trajectory towards sustainability, so what do we need to do differently – urgently and fundamentally – in the next 30 years from what we have done in the last 30 years?
At a European level, the response has been the European Green Deal (EGD), and one could ask if this is a paradigm shift in politics and policy. Part of the answer to that fundamental question can come from work done by the evaluation community.
The goals of the EGD (European Commission, 2019) are well known: climate neutrality, strong biodiversity and nature restoration, the breakthrough of the circular economy as reality rather than concept, zero pollution, a vision on the food system with the Farm-to-Fork Strategy, the just transition, and the link to sustainable investment and the industrial strategy. In fact, if you look at the section on the EGD on the European Commission website, one of the first things it says is that that our most pressing challenge and greatest responsibility is to keep the planet healthy. Europe must lead the transition to a healthy planet. And ‘transition’ here has a particular meaning: it is about systemic changes, which I will come back to later. It also talks about existential threats, and I can tell you that in EU official language, a document like the EGD – which is now the key document driving the European institutions – does not use terms like ‘existential threat’ lightly.
Taking a step back, I look at the EGD as the political, economic and investment priority for Europe. It is much more than an environment or climate plan. It is the boundary condition for political, economic and investment priorities. There is a strong systemic transitions logic embedded in it. There is a strong link with traditional sectoral policies. Much of it talks about the interconnected nature of the issues and solutions. It works with longer time horizons than traditional policies would do. There is a clear link with the social dimension which we have not seen at this level before. There is a link with innovation and digitalization, and with the governance agenda.
So the EGD is ambitious, innovative, interconnected and systemic. I think that we, as an evaluation community, need to respond by reflecting on our own level of ambition, our own innovative character, how interconnected we are and how we deal with systemic challenges. And we do that in a context that is risky not only from a governance and scientific perspective, but also from a business perspective. The World Economic Forum’s annual risk report shows how more and more green issues are appearing. We are also dealing with the changing nature of evaluation, raising questions such as:
What are the goal of evaluators?
What is their place in the policy process?
When does evaluation take place?
What should forward-looking evaluation look like?
All of these questions are becoming important. The traditional approach concentrates on the goals of evaluation through a variety of lenses:
Did the policy work?
What are the (un)intended side-effects on the economy or society?
Should there be a revision of legislation or changes to a policy instrument or target?
What is the quality of the participatory nature of the policy process?
What about the political legitimacy of policies?
I would claim that there is also a new approach – evaluation having a role in catalysing systemic change, (see Figure 1). Helping to show how we can connect the boundary conditions of the planet to our goals around human health, wellbeing and social cohesion. That means that we have to look at systems – food, energy, mobility, the built environment – through the lens of drivers of change, systemic interlinkages, policy gaps. But also through connections to society-wide action, and not only government interventions.
New approach: a role in catalysing systemic change.
Evaluation practitioners should themselves be aware of system dynamics at multiple levels. As demonstrated by the work of Geels and Schot (2007) and Loorbach and colleagues (Loorbach et al., 2017) system transitions are complex, face society with options and risks and demands for constant learning.
Risk-taking is part of fundamental systemic change. What can evaluation contribute to a context in which risk-taking should be stimulated? What sort of corrective action is needed along the way, during transition processes that will take several decades?
Transformations will be of a deep character – how much experience do we have of evaluating such deep transformations? There are no silver bullets to solve complex challenges. As evaluators we are often asked to look at simple solutions or rather limited policies. But what are the ‘lock-ins’ that prevent deep sustainability transitions? And what are the barriers to fundamental change?
Let me use mobility as an example. Historically, one of these supposed ‘silver bullets’ for the transport system has been legislation that would focus on creating better engines and fuel efficiency. Lots of evaluation has happened in this domain, so is it a silver bullet? No. You cannot decarbonize with a combustion engine. What about another silver bullet – biofuels? A lot of evaluation effort has gone into analysing policy instruments around biofuels. If you look at things from a systemic perspective, it becomes obvious that biofuels present many trade-offs. First, there are questions around scaling up, but at a fundamental level, we should ask ourselves whether taking a percentage of crops that are grown on virgin agricultural land and placing them into a combustion engine, which is a 19th-century technology with an efficiency of 25 per cent on a good day, is the right solution. And at the end of it all we would still be stuck with the same old system. Then came electric cars, and there has been plenty of evaluation around what that would mean in terms of the system, infrastructure, the link with sustainable energy, and so on. But even then we would still be stuck in traffic. Evaluating transport and mobility for the future will require a completely different view.
Another proposed silver bullet has been policy integration. The Common Agricultural Policy (CAP) has integrated environment and climate issues for a long time. Recent evaluations, including by the European Court of Auditors, have been clear. It has been largely unsuccessful and even counterproductive.
Unsustainable agriculture remains the main threat to biodiversity and natural capital in Europe. We see the continued pollution of soil water, air and food, and the over-exploitation of natural resources. Greening of the CAP has been ineffective.
We need to ask ourselves: are we focusing on individual projects and programmes of the CAP, or on overall delivery of objectives? How do we evaluate lock-ins in a food system? It’s not only the CAP, but also consumption habits and production methods. And ultimately: how do we disentangle lock-ins from pathways that might lead to sustainability?
Now let’s talk about ‘medium term-ism’ which threatens to be a new risk. The use of gas-fired electricity plants in Europe can be evaluated in a variety of ways: cost, potential for reducing GHGs in the next decade or two, and so on. But if you look at the longer term, and you want an energy system that is zero carbon, then there are certain risks in investing heavily in this medium term solution. Are our evaluations capturing the risks and side effects of medium termism? Systemic change is disruptive as illustrated in Figure 2
Systemic change is disruptive: the ‘x-curve’.
And what about the core of the system – the financial part. How does an evaluation community deal with a new taxonomy on sustainable finance? This will lead to fundamental questions about pathways of investment. Are we evaluating investments at certain moments based on a narrow understanding? Are they adding to potentially stranded assets; locked-in investments that will never get us to a low carbon society? Or are we evaluating them for their fundamental contribution to breakthrough investments?
Social vulnerability and just transitions are also a fundamental part of the EGD. However, they are under-defined concepts. We have very little good data on them and evaluating them will be critical for any societal transition. So here we see a new challenge: for evaluators to work with scientists working on social justice to build a credible knowledge base for the future.
We often focus evaluation on things we need to start doing: on whether experimentation is accelerated, institutionalized, and contributing to sustainability. But I think it is equally important to evaluate where we are on phasing things out – whether policies contribute to phasing out things like fossil fuels, environmentally harmful subsidies, unsustainable taxes, unsustainable spatial planning. . . Because phasing out unsustainable practices is at least as important as promoting breakthrough practices.
What about evaluation in the context of long-term change? Things like climate adaptation measures that will help deal with sea level or temperature rise towards the end of the century. Do we have the methods for that type of evaluation and can we do it in a credible way? What sort of networks of scientists, scenario builders and modellers will we need to collaborate with evaluators?
Time is indeed a critical element that has been underspecified and under defined. It is vital that we pick it up as an evaluation community. Too often evaluation reporting is mis-timed; and implementation not examined.
If you look at much of the literature on what is needed to live up to the SDGs or the EGD, there is a lot of focus on scaling up of solutions. I think it is equally important that we speed up solutions. Speeding up is not a consequence of scaling up. Speeding up is an autonomous factor. We often don’t consider it in this way and that has dangers. If you underestimate time, you could underestimate potential ‘irreversibilities’, leading to lower quality outcomes. You are rewarding laggards (countries or actors) and you may be focusing on the wrong time dimension in sustainability questions. So I’m pleading very strongly to take the time dimension more seriously in evaluation work.
So what is the place of evaluation in the policy process overall? It has to become an integral part of systemic change and transition pathways. These will require iterative evaluations integrated into transition pathways, part of a learning toolbox with strong links to monitoring and reporting. There also needs to be a mandate for evaluators to be engaged in this way.
Another critical element is future orientation – to move beyond the ‘rear-view mirror’. I challenge you to think about the link between foresight and evaluation. How are the two linked in a methodologically sound and responsible way? How can we link evaluation to the challenge of resilience, not only in the environment but also in society?
In conclusion, I think the context for evaluation is changing fundamentally. This requires a new reflection on the role of evaluation, not only from within the evaluation community itself, but also in dialogue with those making use of evaluation, and with broader societal actors that might benefit from the specific capacities of evaluators. It means investing in innovation in the evaluation community and ensuring that evaluation is embedded and integral to the transition process.
I hope that these perspectives will be helpful to your discussions, and I hope to exchange views with you further in the future.
Thank you very much.
Footnotes
Funding
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.