Evaluating the Impact of a Futures-Oriented Organization

Abstract

Foresight and futures work aim at imagining, rethinking, and setting conditions for systemic changes in the society. The societal change as a starting point of evaluation poses challenges to the traditional evaluation approaches. To analyze whether something has an impact on systemic changes, we need methods that consider the dynamics of the operating environment, the multi-actor perspective, and the long-time span of societal changes. In this article, we explore the design and methodological questions related to evaluating the impact of actors, which have set goals for societal impact of their futures work. Our case is based on the wide-scale evaluations of the Finnish Innovation Fund Sitra. In this article, we describe the process and thinking behind the evaluation of the futures work at Sitra and illustrate it with a case study of an evaluation of one of Sitra’s impact goals. Based on our experience from this and other impact evaluations, we provide good practices and recommendations on how to evaluate futures work and foresight in a way that helps steer an organization’s actions.

Keywords

foresight futures work impact evaluation foresight evaluation capacity futures-oriented organization

Introduction

Evaluation of foresight and futures work is not a new thing. In the previous studies, evaluation of foresight and futures work has mainly focused on specific projects or processes, how well they work, and what are their outputs and outcomes (Amanatidou and Guy 2008; Georghiou and Keenan 2006; Haegeman et al. 2017; Miles 2012). The value of evaluation for the futures work is to increase knowledge and understanding of what works and what does not work, and above all, how foresight and futures work can be developed to better meet the needs. To redeem this promise, we argue that evaluations need to consider more strongly the impact of foresight and futures work in society. Furthermore, there is a need to analyze the significance, value, or merit of an action taking account the wider societal context. More and more foresight and futures work occur in different networks of social actions, actors, and organizations. Thus, as typical for complicated or complex actions in a highly connected world, it is not clear where the boundaries of the actions or actors are and what should be evaluated when evaluating the impact. The crucial purpose of evaluation of a future-oriented work and practices is to support learning and development while enhancing understanding of how and why things succeed or not. Therefore, evaluation differs from the other form of assessment, auditing, which is more focused on compliance and determining whether or not some result, activity, or statement meets predetermined criteria (Mayne 2006, 12–13). Therefore, the potential strength of evaluation is the ability to deal with complex issues where uncertainty is the norm, and definitive and simple answers, or predefined measures, are not possible (Mayne 2006, 32).

There are many definitions to foresight and futures work. In this article, we define futures work as all futures-oriented activities, including continuous foresight, futures-oriented projects, and development. By foresight we mean systemic analysis of drivers, alternative futures, and their implications for present action. Foresight and futures work consist of preparing for the future by analyzing trends and drivers, planning for the future by creating visions, discussing preferable futures and making strategies, and broadening the range of possible futures by focusing on emergence, surprises, and discontinuities (Miller 2018). Futures work is participatory by nature, aiming to create systemic change, which entails challenging mental models, creating strategy, and coordinating action.

Futures work is done by many actors and in multiple different ways (Dufva and Ahlqvist 2015b). It can be carried out in a specific foresight project, ranging from a small intraorganizational workshop to a large national foresight project. It can be integrated into organizations as a team or department, and organizations can be a part of a network of different actors doing futures work. There are also different consultants and think tanks dedicated to futures work.

In this article, we describe how the evaluation of futures work is done at the Finnish Innovation Fund Sitra. Sitra is a think-and-do-tank operating under the Finnish Parliament. Sitra’s mission is to build a successful Finland of tomorrow, and its work is guided by the vision of the next era of well-being, a fair and sustainable future. Sitra’s role as a future-oriented fund involves creating preconditions for reform, spurring different actors toward making a change and providing opportunities for co-operation. Sitra has a wide range of methods and tools for its future-oriented work and is continuously testing and developing new ones.

Sitra has an internal foresight team, which coordinates the Finnish National Foresight Network together with the Prime Minister’s Office, and it funds and facilitates many foresight projects. It thus covers many ways of doing foresight. However, Sitra also differs from most of the organizations that are doing futures work. It is funded by returns on endowment capital and capital investments, which means it is financially independent. It is located directly under parliament, not government, which means that while it is not driven by current policy needs, it is connected to policy making. It is also not just a think tank that publishes reports, but rather a “think-and-do” tank creating systemic change by funding experiments, coordinating joint projects, networking and increasing the long-term policy capability in Finland.

Sitra’s futures work focuses in three theme areas, which are the capacity for societal renewal, the transition to a carbon-neutral circular economy, and the creation of new working life and a sustainable economy. In addition, Sitra organizes societal training and produces long-term foresight data and studies in anticipation of the future as well as supports the Finnish society in interpreting and making use of these data. Sitra’s foresight products include, for example, analyses on megatrends (Hautamäki et al. 2017; Kiiski-Kataja 2017; Neuvonen 2017), societal visions (Kiiski-Kataja et al. 2018), and weak signals (Dufva 2019). Sitra runs also foresight projects that are more action-oriented. For example, the purpose of the Timeout project (Sitra 2017) is to strengthen people’s participation in society, mutual trust, the understanding of future developments, and their connection to decision-making by utilizing dialogue.

To support and develop the futures work, there is a need constantly to evaluate the activities and to ask whether the organization is doing the right things right, at the right time, and in an effective manner? In this article, we describe how Sitra has developed its internal evaluation capacity to answer these evaluative questions, including a rationale, an evaluation framework and practices as well as resources required for commissioning and utilizing evaluation. First, we present how and why the impact evaluation framework was developed. After that, we focus on the case study of evaluating one of the Sitra’s impact goals. Finally, the article presents key notions and recommendations for other foresight organizations that would like to develop their evaluation expertise.

Development of the Impact Evaluation Framework

Already for several years, Sitra has monitored and assessed its projects by using the input–output–outcome–impact results chain (IOOI) (e.g., Organisation for Economic Co-Operation and Development [OECD] 2013) in the goal-setting and setting measures for the goals. The IOOI framework helps to illustrate a theory of change of a project, that is, what kind of resources, outputs and activities and concrete changes are required to bring about the societal impact, the ultimate goal of the action. The traditional use of the logic models bases usually on the linear thinking, but we see our actions and the operational environment rather complicated or complex because Sitra addresses innovations aimed at systemic change. They are often generated in networks of different actions, actors, and environmental factors and as a combined effect of them, which makes them complex. Although IOOI framework is always a simplification, it is a valuable tool for impact management and a good baseline for constructing theory of change and for communicating both intended and achieved impact paths.

In 2016, there was a need to shift the focus from individual projects’ results to an analysis of the bigger picture of impact and what has the organization’s contribution been on systemic changes. We created an impact evaluation framework to complete the previous model of internal monitoring with the external impact evaluations. The framework was a written document that explicitly defined the rationale of the impact evaluation. It was accepted by the management and the board who were also the important intended users of the evaluation results.

Impact is an ambiguous concept, and there are several ways to define and use it; some of them focus on very precise understandings of impact, while others cast a much broader net. How impact is defined and used has a significant effect on the design, management, and evaluation (Hearn and Buffardi 2016). As defined in the evaluation framework, Sitra’s approach relies on a broad definition of the concept of impact, in which impact is seen as positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended (also OECD-DAC Glossary 2001).

The principles defining an approach to Sitra’s impact evaluation were formulated in the impact evaluation framework. Based on the relevant evaluation studies (e.g., Befani et al. 2015; Hyytinen et al. 2014; Nieminen and Hyytinen 2015; Preskill et al. 2014; Stern et al. 2012; W.K. Kellogg Foundation 2004), we identified the needs and also the elements for an approach that is suitable for impact evaluation of an organization operating in a complex environment. The main principles are (1) taking into account the long-time span of societal changes, (2) applying a holistic approach to systemic changes, (3) focusing on contribution of an organization (instead of attribution when it is not possible), (4) supporting learning and development during the evaluation process, and (5) using methodologies that are appropriate for the evaluation purpose and context. The last one emphasizes that the appropriateness of the methods strongly depends on the phenomena and the contexts that are being evaluated. The evaluation framework acknowledges that each method involves its own strengths and weaknesses, and there is no method that would be universally best (e.g., Chen 1990, 23–28).

The principles guide the most important part of evaluation design which is formulating the tasks and questions that need to be answered by an evaluation. The evaluation framework stated the purpose of the Sitra’s impact evaluation, that is, (1) to produce high-quality, unbiased information on impact of Sitra’s activities; (2) to support strategic steering and development; and (3) to increase knowledge and understanding of the progress of the impact goals in the society. The impact evaluations combine the three classical purposes of an evaluation: accountability, learning and developing, and knowledge production (Chelimsky 1997). Each of the evaluation has to answer three key questions (see Figure 1): (1) What progress has been made toward the goals and what or who, besides Sitra, has contributed? What actions are needed in future? (2) How has Sitra contributed to achieving the goals, and what are the paths, mechanisms, partnerships and other means that have created the impact? and (3) How well the strategic choices have succeeded? The essential question is “Are we doing the right things in a right way at the right time?” With our impact evaluation approach, we emphasize the applicability of the evaluation in developing our own activities and in learning in futures work. A good evaluation is a remarkable investment, and that is why it should not only look backward. An ambitious task was to create an evaluation framework that increases knowledge and understanding of the factors that breed success for sustainable well-being of tomorrow.

Figure 1.

The tasks and questions for evaluating the futures work of Sitra.

At the same time as we renew the evaluation framework, we also launched and implemented the shared goals for societal impact. This means that instead of having separate, loosely interlinked but poorly communicated and evaluated impact goals for each project, Sitra’s operations were now guided by eight shared goals for impact, all of which are interlinked and aim at achieving the sustainable well-being.

The impact goals are the following:

Addressing well-being in a holistic way.

Adjusting to planetary boundaries.

Empowering individuals and communities.

Moving to a regenerative and collaborative economy.

Building competencies for a complex world.

Developing inclusive and adaptive governance.

There is a widespread awareness of potential future developments in Finland.

Sitra is a nationally and internationally recognized, responsible, and independent think-and-do-tank.

By nature, Sitra’s impact goals define a shared direction and vision of what kind of a society we want to build. Thus, in Sitra’s operations, impact refers to long-term systemic change. They are closer to principles steering the action in a complex world (Patton 2017) than accurate predefined measures or indicators. Each of the eight impact goals were evaluated separately for the period 2017–2019 by external evaluators that were chosen through a public procurement process.

To illustrate the approach and methods used in more detail, we will next focus on the evaluation of the impact goal concerning “the Widespread Awareness of Potential Future Developments in Finland.” This is the primary impact goal of our foresight activities and the futures work of the organization as a whole. In this article, we describe what kind of approach and methods we have used and illustrate with some examples the knowledge and results the methods have produced.

Case: Evaluating the Impact Goal “There is a Widespread Awareness of Potential Future Developments in Finland”

It is essential that the evaluation set-ups and methods take into account the nature of our work for the future, the operating environment and the dynamics and time span of systemic changes in a society. We have considered this when identifying the evaluation questions and developing the evaluation set-ups for the impact goal “There’s a Widespread Awareness of Potential Future Developments in Finland.” We will next describe some aspects of the evaluation in each of the three key evaluation questions in more detail. The time period of the evaluation starts from the year 2011 and ends till 2018.

For creating an appropriate evaluation design, it is necessary to understand the nature of actions that will be evaluated. Evaluation is not a “one size fits all” activity. Therefore, we conducted first a research review on foresight evaluation practices. The aim of the research review was to better understand the evaluation frameworks of the foresight and futures work as well as to identify existing evaluation practices and methodologies (see Parkkonen and Vataja 2019). The review clarified the nature of foresight and futures work, but as a result, we also noticed that there was no ready-made tool, like a measure or a questionnaire, that could be used for our evaluation purposes. Sitra not only produces futures knowledge but also aims at building futures and foresight capabilities in the society at many levels. Hence, the existing foresight evaluation approaches and practices that base on evaluating the success of a specific foresight process or project were not suitable for evaluating Sitra’s contribution.

The first evaluation question considered whether there has been progress toward the impact goal, and where should the developments and changes be seen in the society. To answer the question, we conducted media analysis, interviews, and workshops with stakeholders and experts. For example, to get a good understanding of the general awareness of futures and future developments, we commissioned a survey resulting in a Futures Barometer (Dufva et al. 2019). The purpose of the Futures Barometer was to study the public’s awareness and attitude toward the future, knowledge concerning the different megatrends, and the capability to make an impact on the future. The survey was targeted at fifteen- to eighty-four-year-old Finns, and the data were gathered via the Kantar TNS Internet panel from over two thousand one hundred people. The survey was conducted on a sample representing the Finnish society in a sociodemocratic manner and it considered the respondents’ political views. According to the results of the survey, 88 percent of the respondents are interested about the future, 83 percent believe that they can make an impact on the future but only 63 percent know how to make the impact. The megatrends as defined by Sitra are well known and Finns tend to see more threats in them than opportunities.

The second evaluation question addressed how Sitra has contributed to achieving the impact goal. We approached this question by gathering evidence on how Sitra’s futures knowledge and foresight products such as the megatrend and weak signal analyses, megatrend cards, and visions have been used, and what has been Sitra’s role in the public discussion concerning different futures topics. For example, we commissioned a media and document analysis on futures discourse in the public. The goal of the analysis was to conduct an environmental scanning to better understand what futures topics are been discussed in the media and the society and by whom, and what is Sitra’s role and contribution to the discussion. The analysis was based on Sitra’s media analyses and media archives in 2012–2016 and on a big data consisting of over sixty-nine thousand articles in digital and editorial media as well as over two hundred twenty thousand social media messages in 2017–2018.

The media analysis shows that Sitra’s engagement and relative share in the public futures discourse has increased during review period. It also provided more detailed information on what topics are included and how they have changed over time. The most discussed topics in the Finnish media focused on, for example, the future of artificial intelligence, urbanization, circular economy, health care, digitalization, and climate change. Sitra has been an active debater in discussions that relate to topics such as the future of democracy, data economy, electronic health care, circular economy, and climate change. All these topics are covered in Sitra’s megatrend work and analysis. Sitra has also contributed to the discussion of weak signals—the growth within the overall share of media discussion increased from 0 to 35 percent between the years 2017 and 2018, and the engagement rate increased even 80 percent. According to the results, Sitra is most often present in articles that provide an optimistic view of the futures, especially related to the topics such as the sustainable well-being, circular economy, and life-long learning. The analysis concludes that Sitra has influence on the public futures discourse in Finland and Sitra’s impact on the discourse is seen most clearly in areas where Sitra has launched new concepts in Finnish, for example, the circular economy and sustainable well-being.

In addition, we commissioned a bibliometric analysis on the use of Sitra’s futures and foresight products and publications. The bibliometric analysis explored in what context and by whom Sitra’s publications and products have been cited. The information was gathered from different open databases including the scientific publications, the annual reports of the listed Finnish companies and nonprofit organizations, government and parliamentary reports, and studies and other expert publications within the time period reaching from 2011 to 2018. Based on the bibliometric analysis, Sitra appeared in a total of one hundred twenty-five publications. Sitra’s publications and products were used most often in the scientific publications, in the annual reports of the listed Finnish companies, and in the reports of the Committee for the Future. The first two uses came as a little surprise to us. Moreover, we conducted a survey to the megatrend card users and asked how they have used the product. The most common uses were for education, strategy building, and ideation. Together, these studies indicate how widely, by whom, and for what purposes Sitra’s foresight publications and products are being utilized, which then helps in evaluating Sitra’s contribution to the progress in the society. The results show that Sitra’s futures and foresight material have reached different stakeholders and audiences who have used the material for different purposes. This is relevant from the point of view of the theory of change. To argue that Sitra’s work has created impact on the widespread awareness of potential future developments, it is necessary to know whether the futures products have been used.

The analysis recognized other relevant actors beside Sitra in the field of futures work and it is relevant to assess Sitra’s role in relation to other actors and their influence. However, the challenge is that there are no benchmarks and standards to refer while evaluating Sitra’s worth or merit.

The last question of evaluation examined the success of Sitra’s strategic choices, reflecting against the progress (the first evaluation question) and the contribution of Sitra’s futures work (the second evaluation question): “Are we doing the right thing at the right time?” We commissioned an external and unbiased evaluation that made a synthesis of the above-mentioned evidence, complemented the material with the interviews, and evaluated Sitra’s role, contribution, and the success of the strategic choices based on all the material. The external evaluators interviewed a wide range of stakeholders and experts from different parts of the society, representing the views of the academia, the public and private sector, labor-market organizations, the civil society, the culture sector, and the political decision-makers. Altogether, twenty-two people were interviewed. The external evaluation is still ongoing, and the results will be available at Sitra’s website later in the fall 2019. All evaluations are published on our website to support transparency. The results will give valuable knowledge on how well the evaluated impact goal is reached at different levels and within different stakeholders in the society and how well Sitra has succeeded in meeting the impact goal. The results of the external evaluation will be utilized in the organizational steering and development.

We have summarized the key evaluation questions and methods in the Table 1. The table illustrates a need for using various data sources for tracing and harvesting impacts and contribution of the futures work.

Table 1.

The Evaluation Questions, Evaluation Focus, and Methods.

Evaluating the impact goal: there is a widespread awareness of potential future developments in Finland
Evaluation questions	Evaluation focus	Data gathering methods
What has been the progress toward the impact goal? Where the developments and changes should be seen in the society?	• What kind of discussion is there on the different futures? • What futures topics have been discussed in the public and in the media during the time period? • What kind of futures knowledge and future-oriented studies have been published and available? • How is the futures knowledge been used in the decision-making and the policies, for example, in the governments’ programs, strategies, and decisions? • What type of foresight practices and activities do different organizations have, for example, in public or private sector? • Are the citizens and the decision-makers aware of the potential future developments?	• Media and document analysis • Interviews and workshops with stakeholder and experts • Surveys, as futures barometer for citizens (Dufva et al. 2019)
How has Sitra contributed to achieving the impact goal? How could it be verified?	• What actors work with and talk about the different futures? • How has Sitra succeeded in broadening the knowledge and debate on different futures? • How has Sitra contributed to the public discourse on futures topics? • How and by whom have Sitra’s futures work and foresight products (e.g., megatrend and weak signal analysis, megatrend cards, and societal visions) been used? • How do the stakeholders see Sitra’s role and contribution in building the awareness of the potential future developments in Finland?	• Media, document, and big data analysis of the public discourse on future-related topics • Bibliometric analysis • Stakeholder and expert interviews • Surveys, as the feedback from the product users
How well the strategic choices have succeeded?	• Have the choices regarding the focus and targets of the foresight projects been right and timely? • Have the projects been executed with the right partnerships and stakeholders? • Has Sitra identified and contributed to the relevant futures discussions? • Are we doing right things at the right time?	• An external evaluation that makes a synthesis of the progress toward the impact goal and of analysis of Sitra’s contribution to the changes and based on them evaluates what has been Sitra’s role and contribution in advancing the impact goal.

Discussion

Based on our experiences from the impact evaluations at Sitra, we discuss four key notions related to context, futures knowledge, organizational learning, and practical considerations that could be useful for others too. First, it is the importance of understanding different contexts in which futures work is carried out and futures knowledge is used. To make sense of the impact of foresight and futures work, it is important to pay attention to the operational environment. What kind of societal impact are you trying to achieve with your futures work, and where and when should the changes be seen? Who are the main target groups? The context of the work should guide the evaluation methods too: how much emphasis is put into media and document analysis? Would outcome harvesting be useful in exploring both intended and unintended results? Could interviews or questionnaires help to track changes in the awareness and capacity of the people and decision-makers? Evaluation is not just measuring results and monitoring indicators. To be able to evaluate societal impact, there might be a need to make sense of the outcomes of futures work and to understand how these outcomes have been generated in the interaction of different actors.

The second notion concerns the importance of identifying and making explicit how futures knowledge is understood. This is also strongly contextual question. For example, what does it mean that people are aware of potential future developments? Is it enough that there are reports published on key trends, or should futures knowledge be understood more as an ongoing conversation about future prospects and actions that need to be taken today? In our experience, the latter is more fruitful, although also more demanding. For evaluation, this understanding of futures knowledge more as a network of concepts than a block of information (Dufva 2015) means paying attention to how widely the outputs of foresight are used and how they are linked to decision-making on all levels.

Using futures knowledge is connected to supporting organizational learning. The outcomes of the evaluation and the evaluation process itself should improve the capability to do foresight and futures work better. This requires ensuring the quality of evaluation, starting by asking right questions. We highlight that the suitability of approaches and methods used for impact evaluation depends on the nature of the object and context of evaluation as well as on what information is needed. In organizational context, it also means supporting evaluative thinking and capacity (Schwandt 2018). It is also crucial to build bridges between evaluation, strategy, and operational work both in an organization and with the stakeholders—in essence to increase futures literacy (Miller 2018), futures consciousness (Ahvenharju et al. 2018) or futures preparedness (Rohrbeck and Kum 2018).

There are also practical things to consider when evaluating foresight, especially regarding the relation between the external evaluator and the organization. Our third notion is that there is a need to familiarize external evaluators properly with your organization’s vision, mission, strategy and activities, as well as the needs and expectations for evaluation. The evaluation framework presented in this article includes basic ideas of utilization-focused evaluation that highlight the role of intended users of evaluation from the beginning and the importance to involve them in the evaluation process that can also serve learning (Patton 2008).

As the fourth notion, we considered it essential to outline and increase explicitly our own understanding, inside the organization, of how future-oriented work promotes our impact goals. As a good practice, we created so-called impact claims that explained Sitra’s own hypothesis of the last step of the results chain—from outcomes to impact. These claims have been a useful tool to communicate our theory of change better to the external evaluators, and they have laid foundations for theory-based evaluation that external evaluators have produced for us. According to our experiences, good evaluation of foresight and futures work is similar to good foresight process: it is participatory and interactive, considers multiple viewpoints, and is seen as a part of a larger whole.

Conclusion

In the article, we have presented the impact evaluation framework of future-oriented organization Sitra and described its rationale and development. Our case illustrated how the primary impact goal of our foresight activities and futures work, “There are the Widespread Awareness of Potential Future Developments in Finland,” is evaluated to have systematic and reliable knowledge of the progress in a society, Sitra’s contribution, and whether or not we are doing the right things at the right time.

When foresight and futures work happen in networks of many actions and actors, there are no clear boundaries for evaluating the impact of the work. The evaluation framework presented in the article aims to take account the complicated, or even complex nature of the action with multiple components. It takes a systemic perspective in which the action is analyzed taking account its context and how the context has developed over time. It also considers the actions of an organization in relation to what other actors are doing and thus views foresight as a system (cf. Dufva and Ahlqvist 2015a).

Our case illustrated the methods that have been utilized in the evaluation. However, we also recognize a need for new innovative methods that help to understand even better why and how the impact is generated and serve continuous learning during the whole evaluation process. The key notions stress the need for internal capacity building in an organization, including adequate resources, to be able to design, commission, and manage evaluations. Learning is also needed from an outside evaluator for understanding the nature and special needs of the futures work. Evaluation should thus be seen as a process of capacity building, organizational learning, and co-learning, so it can spur impactful foresight and futures work.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Pinja Parkkonen

Author Biographies

Katri Vataja DSc is a senior lead on impact evaluation and strategy processes at the Finnish Innovation Fund Sitra. In her current work she manages evaluations, ensuring that high-quality information is obtained through evaluations, and she has a versatile experience of organizational development, strategy work and evaluation. Her dissertation in social and health management examined self-assessment as a method for generating and managing change in the organizations.

Mikko Dufva DSc is a leading specialist on foresight at the Finnish Innovation Fund Sitra. In his work he examines future trends, weak signals and images of futures. His dissertation was about knowledge creation in foresight from a systems perspective and he has broad methodological expertise ranging from systems thinking, decision analysis and optimization to interactive planning, scenario analysis and participatory methods.

Pinja Parkkonen MSocSci is a specialist on impact evaluation at the Finnish Innovation Fund Sitra. In her work she coordinates evaluation projects, facilitates impact assessments and produces analysis on future developments. She is passioned about integrating futures and evaluative thinking into organizational development.

References

Ahvenharju

Sanna

Minkkinen

Matti

Lalot

Fanny

. 2018. “The Five Dimensions of Futures Consciousness.” Futures 104:1–13.

Amanatidou

Effie

Guy

Ken

. 2008. “Interpreting Foresight Process Impacts: Steps towards the Development of a Framework Conceptualizing the Dynamics of ‘Foresight Systems.’” Technological Forecasting & Social Change 75:539–57.

Befani

Barbara

Ramalingam

Ben

Stern

Elliot

. 2015. “Introduction: Towards Systemic Approaches to Evaluation and Impact.” IDS Bulletin 46 (1): 1–6.

Chelimsky

Eleonor

. 1997. “Thoughts for a New Evaluation Society.” Evaluation 3 (1): 97–109.

Chen

Huey-Tsyh

. 1990. Theory-Driven Evaluations. Newbury Park: Sage.

Dufva

Mikko

. 2015. Knowledge Creation in Foresight: A Practice- and Systems-oriented View. Espoo: Aalto University.

Dufva

Mikko

. 2019. “Heikot Signaalit Tulevaisuuksien Avartajina” [Weak Signals as Expanders of Futures]. Sitra’s Studies 142. Vantaa: Erweko. https://www.sitra.fi/en/topics/weak-signals/.

Dufva

Mikko

Ahlqvist

Toni

. 2015a. “Elements in the Construction of Future-Orientation: A Systems View of Foresight.” Futures 73:112–25.

Dufva

Mikko

Ahlqvist

Toni

. 2015b. “Miten edistää hallituksen ja eduskunnan välistä tulevaisuusdialogia?” [How to enhance the futures dialogue between the government and parliament?]. Publications of the Government’s Analysis, Assessment and Research Activities. http://julkaisut.valtioneuvosto.fi/handle/10024/79546.

10.

Dufva

Mikko

Laine

Paula

Lähdemäki-Pekkinen

Jenna

Parkkonen

Pinja

Vataja

Katri

. 2019. “Tulevaisuusbarometri 2019. Millaisena suomalaiset näkevät tulevaisuuden?” [The Futures Barometer 2019. How Do the Finns See the Future?). Sitra’s Studies 147. Helsinki: Erweko.

11.

Georghiou

Luke

Keenan

Michael

. 2006. “Evaluation of National Foresight Activities: Assessing Rational, Process and Impact.” Technological Forecasting & Social Change 73:761–77.

12.

Haegeman

Karel

Spiesberger

Manfred

Könnölä

Totti

. 2017. “Evaluating Foresight in Transnational Research Programming.” Technological Forecasting & Social Change 115:313–26.

13.

Hautamäki

Antti

Leppänen

Juha

Mokka

Roope

Neuvonen

Aleksi

. 2017. From Pause to Play: Work and Income in the Next Era. Helsinki: Sitra.

14.

Hearn

Simon

Buffardi

Anne L.

2016. “What Is Impact? Overseas Development Institute.” https://www.odi.org/publications/10326-what-impact.

15.

Hyytinen

Kirsi

Ruutu

Sampsa

Nieminen

Mika

Gallouj

Faïz

Toivonen

Marja

. 2014. “A System Dynamic and Multi-Criteria Evaluation of Innovations in Environmental Services.” Economics and Policy of Energy and the Environment 2014:29–52.

16.

Kiiski-Kataja

Elina

. 2017. From the Trials of Democracy towards Future Participation. Helsinki: Sitra.

17.

Kiiski-Kataja

Elina

Laine

Paula

Jousilahti

Julia

Neuvonen

Aleksi

. 2018. The Next Era of Well-Being: Ideals, Vision and Solutions. Edited by Lahti

Vesa-Matti

. Helsinki: Sitra.

18.

Mayne

John

. 2006. “Audit and Evaluation in Public Management: Challenges, Reforms, and Different Roles.” The Canadian Journal of Program Evaluation 21 (1): 11–45.

19.

Miles

Ian

. 2012. “Dynamic Foresight Evaluation.” Foresight 14 (1): 56–68.

20.

Miller

Riel

. 2018. Transforming the Future: Anticipation in the 21st Century. London: Routledge.

21.

Neuvonen

Aleksi

. 2017. Rewiring Progress. Helsinki: Sitra.

22.

Nieminen

Mika

Hyytinen

Kirsi

. 2015. “Future-Oriented Impact Assessment: Supporting Strategic Decision-Making in Complex Socio-Technical Environments.” Evaluation 21 (4): 448–61.

23.

Organisation for Economic Co-Operation and Development. 2013. “Development Results: An Overview of Results Measurement and Management. Briefing Note.” https://www.oecd.org/dac/peer-reviews/Development-Results-Note.pdf.

24.

OECD-DAC Glossary. 2001. “Glossary of Key Terms in Evaluation and Results Based Management.” https://www.oecd.org/dac/evaluation/2754804.pdf.

25.

Parkkonen

Pinja

Vataja

Katri

. 2019. “Näkökulmia ja lähestymistapoja tulevaisuustyön ja ennakoinnin arviointiin” [Perspectives and Approaches to Evaluating Futures Work and Foresight]. Futura 60–73.

26.

Patton

Michael Q.

2017. Principles-Focused Evaluation. New York, NY: Guilford Press.

27.

Patton

Michael Quinn

. 2008. Utilization-Focused Evaluation. 4th ed. Thousand Oaks: Sage.

28.

Preskill

Hallie

Gopal

Srik

Mack

Katelyn

Cook

Joelle

. 2014. “Evaluating Complexity: Propositions for Improving Practice.” FSG. http://www.pointk.org/resources/files/Evaluating_Complexity.pdf.

29.

Rohrbeck

René

Kum

Menes Etigue

. 2018. “Corporate Foresight and Its Impact on Firm Performance: A Longitudinal Analysis.” Technological Forecasting & Social Change 129:105–16.

30.

Schwandt

Thomas A.

2018. “Evaluative Thinking as a Collaborative Social Practice: The Case of Boundary Judgment Making.” New Directions for Evaluation 158:125–37.

31.

Sitra. 2017. Timeout Project. https://www.sitra.fi/en/projects/timeout/.

32.

Stern

Elliot

Stame

Nicoletta

Mayne

John

Forss

Kim

Davies

Rick

Befani

Barbara

. 2012. “Broadening the Range of Designs and Methods for Impact Evaluation. Report of a Study Commissioned by the Department for International Development.” Working Paper 38, Department for International Development. https://www.oecd.org/derec/50399683.pdf.

33.

W.K. Kellogg Foundation. 2004. Evaluation Handbook. Battle Creek: W.K. Kellogg Foundation.