Aspects of the institutionalization of evaluation in Finland: Basic,agency,process and change

Abstract

This article draws on neo-institutional theoretical ideas to empirically examine the institutionalization of evaluation in the national government of Finland. The results indicate ambiguity in the basic institutionalization of Finnish evaluation, and imprecision in the agency of the actors that carry out or commission evaluations or utilize the evaluation results. Some Finnish institutional practices of evaluation enhance formal rationality such as efficiency and effectiveness, some support legitimation, and others do both in combination. The strength of coupling of evaluation to decision-making varies greatly. For future research, the article suggests studies on the institutionalization of evaluation in other countries. For evaluation practice, the results highlight the position of evaluation along the rationality-legitimation axis, and the variable linkages of evaluation to decision-making.

Keywords

agency coupling effectiveness efficiency Finland institutional change institutionalization legitimation rationality

Evaluation is a practical art but one that utilizes a wide variety of scientific approaches and that often contributes substantially to the development of these approaches. Given that there is such a variety of approaches to evaluation, this article considers only one of these. The article takes its conceptual framework from global neo-institutional research (Lowndes and Roberts, 2013; Peters, 2011). On the theoretical grounds discussed below, the article poses four research questions and empirically seeks answers to these questions through a country case study on the institutionalization of evaluation in Finland. The four questions are:

(1) What are the processes and outcomes that characterize the basic institutionalization of evaluation in Finland?

(2) From where do evaluation actors derive their capacity to act – their ‘agency’?

(3) Observing the global proliferation of evaluation, which processes of institutionalization contribute to its shaping and persistence?

(4) Observing that evaluation may not only evolve gradually, how do active change agents exert influence its shaping?

Academic research on evaluation and evaluation practice can progress without reflecting on the key notions of evaluation, simply taking them for granted. Guba and Lincoln (1989), for their part, advise us to abandon efforts to define evaluation on the basis that context-free definitions are vacuous. However, academic research commonly requires either universal or contextual definitions of evaluation, and evaluation clients also make analogous requests (see, for instance, Stufflebeam and Shinkfield, 2007). This article observes these common requirements.

The following section elaborates the conceptual approach, discusses the motivation for the four research questions, and links these questions to each other. The next section explicates the research methodology. Each of the subsequent four sections seeks to answer one of the research questions posed. The last section discusses the contributions and implications of the study.

The research approach

Within neo-institutional research (Lowndes and Roberts, 2013; Peters, 2011), this article represents what is commonly called sociological neo-institutionalism. This is a variant launched by John W. Meyer and Brian Rowan (1977) during the incubation years of neo-institutionalism, and later substantially elaborated by Meyer and his colleagues (Kruecken and Drori, 2009). This article formulates its four research questions by way of four complementary perspectives drawn from Meyerian neo-institutionalism. The four perspectives, respectively, focus on:

the examination of the roots and processes of basic institutionalization;

‘agency’ as the capacity of the institutional actors to act;

broader processes and outcomes of the ‘going concern’ of institutionalization;

intense agent-driven institutional change.

The basic institutionalization of evaluation

The neo-institutionalism that this article draws upon sees basic institutionalization to be formed by habitually taken-for-granted patterns of interaction between the relevant actors, which also catalyzes the social knowledge that these actors share (Meyer and Rowan, 1977: 341). During the early years of neo-institutionalism, Zucker (1977: 726) noted that this ‘social knowledge, once institutionalized, exists as a fact, as part of objective reality’. Following this neo-institutionalist perspective, ‘facts’ include institutional vocabularies, terminologies, conceptual systems, classifications, categorizations, boundaries within and between institutions, and the identities of institutions and institutional actors.

Institutional routines and habitual patterns pose particular methodological challenges, and suggest caution in using institutional insiders as informants. Schneiberg and Clemens (2006: 214) suggest that we examine ‘breaches’, ‘deviant events, or conflicts that reveal … undiscussed boundaries of taken-for-granted understanding’. The intricacies of institutional taken-for-granted routines led this investigation to dive below the surface of evaluation. This is why it examines ambiguity, hybrid forms, equivocal language, ‘leaky’ classifications, categorizations and boundaries, and the incomplete identities of evaluators, evaluation practices, and institutions of evaluation. Emphasizing these characteristics is a consequence of methodological choice, not a criticism of evaluation.

The capacity to act in evaluation: Evaluation agency

This article moves on from basic institutionalization of evaluation to examine the ‘evaluation agency’ of actors in the contingent circumstances in which these actors may either succeed or fail. Meyer and Jepperson (2000: 117) analytically distinguish three types of agency. ‘Agency for itself’ (or for ‘themselves’) vests in individuals and collectivities such as professions, specialist organizations and national states acting on their own behalf. ‘Agency for others’ is carried out, for instance, by professions, organizations and states on behalf of others whether they are individuals, families, groups, organizations or other states. ‘Agency for standards and principles’, as its name suggests, acts on behalf of a principle, for example, human rights, transparency, good governance, social responsibility, and science-driven rationalization (Drori and Meyer, 2006; Drori et al., 2009).

This article looks for:

‘agency for itself’ in the self-evaluation of actors and in evaluations conducted by evaluators on their own initiative;

‘agency for others’ in evaluations commissioned by other actors and carried out by evaluation professionals, organizations and institutions consistent with their mandate;

‘agency for standards and principles’ in the approaches, practices and principles of evaluation itself.

Diffusion, rationality, legitimation and institutional coupling in evaluation

The neo-institutional framework adopted directs attention to the diffusion of models and scripts for new or revised institutional elements, the modification of these models and scripts in use, and the sedimentation of the new elements atop those that have been institutionalized earlier (Strang and Macy, 2001). The article considers evaluation in these terms.

Accounts of institutional actors trying to control their contingent circumstances by means of institutional elements that enhance formal rationality such as efficiency, performance or effectiveness convey only a partial truth (Meyer and Rowan, 1977). This is because the actors may also introduce and maintain institutional elements that enhance institutional legitimation.

Neo-institutionalism agrees with certain other research orientations that institutional elements may be introduced to enhance rationality, often fulfill their purpose, and support institutional legitimation in this way. However, since Meyer and Rowan (1977), neo-institutionalism also considers the possibility that legitimation is enhanced by institutional elements that are introduced with the promise of rationality independent of what is actually achieved. It is useful to note that the decoupling (or loose coupling) between institutional elements that actually enhance rationality and those that enhance legitimation may usefully protect core institutional elements from institutional outsiders. In actual practice, research on evaluation has recognized the complex relationships between rationality and legitimation years before the emergence of neo-institutional research (Eaton, 1962; Weiss 1970; but see also Dirsmith et al., 2000).

Intense agent-driven institutional change in evaluation

Lowndes and Roberts (2013: 111–43) distinguish two dimensions of institutional change: its tempo; and the strength of the agency of the actors that initiate change. Combining these two dimensions, they distinguish four types of change. Three of these types have been covered in the previous sections, but the fourth type is relevant here: intense agent-driven institutional change (Lowndes and Roberts, 2013: 122–5). For examining the institutionalization of evaluation, an early neo-institutional view is relevant:

[W]hen emergent culture is the focus, then the problem of establishing facticity becomes the central problem. It is here that the moral character of social facts becomes the central concern. (Zucker, 1977: 726)

In the neo-institutional understanding to which this article subscribes, actors that successfully carry agency for change induce the re-evaluation of what count as the most acceptable social facts with regard to the institutionalization of evaluation. This re-evaluation substitutes a new preferred institutionalization for prior institutionalization that now becomes obsolete (see, for instance, Meyer and Hammerschmid, 2006).

Methodology

This article follows common criteria of qualitative research: the cogent formulation of theoretically informed research questions, the clear explication of the data collection and analysis, the ‘saturation’ of the available data in the analysis, and the credibility and trustworthiness of the results (Silverman, 2011: 27–56). Empirically, the article examines the institutionalization of evaluation within Finland’s national government of twelve ministries and numerous agencies and offices (Salminen et al., 2012).

Had Finland a Westminster-style government with a more unitary political executive, it would be sufficient for the article to draw upon such sources as White Papers elaborating government policies and programs. However, Finland’s legalist hierarchy of norms, derived from a written constitution, justifies reliance on the contents of the national legislation database Finlex. The analysis first uses this database to examine the basic institutionalization of evaluation, looking for instances of the Finnish word arviointi for ‘evaluation’, and searching for ambiguity in the use of this word.

The necessary material to examine institutional agency for evaluation in Finland includes both official documents and research studies. Analogous material and Finlex support the examination of diffusion, rationality versus legitimation and the coupling or decoupling of institutional aspects that concern evaluation, such as the link between evaluation and decision-making. The study of intense agent-driven institutional change in evaluation requires, in principle, the observance of methodological conventions of historical research (Jordanova, 2006), although this article type allows only for examples of historical change.

Besides documents, the research material includes five expert interviews to check facts and tease out interesting ‘breaches’ in the institutionalization of evaluation. Interview questions were not shown to the interviewees but only used as prompts in each interview situation (Gubrium et al., 2012; see Table SD1 in Supplementary Data on http://evi.sagepub.com/supplemental). All interviewees had experience of evaluation-related research and evaluation practice, each held or was studying for a doctorate, and the survey included both male and female respondents. The average age of the interviewees was fifty-five. To encourage the interviewees to speak freely, the interviews were conducted under assurances of anonymity.

Basic institutionalization of evaluation in Finland

Table 1 provides an overview of the evaluation vocabulary in Finnish legislation. In the Finnish legislative database Finlex, as in the Finnish language in general, the word arviointi may signify not only ‘evaluation’ but also, for instance, ‘assessment’, ‘inspection’, ‘valuation’, ‘ascertainment’, or mere ‘checking’ of legal compliance. Nominally, arviointi is present in quite a number of acts and statutes (see Table 1), but polysemy (words and expressions carrying different meanings) means that most instances have little to do with evaluation as understood within the global evaluation community. The common Finnish expression tuloksellisuuden arviointi, despite its literal translation as ‘performance evaluation’, refers to performance measurement ascribing quantitative values to performance indicators. Moreover, the majority of the current 315 acts or statutes that use the expression vaikuttavuuden arviointi (‘effectiveness evaluation’; see Table 1) are simply upgrades of technical ‘inspection’, or environmental or other ‘impact assessment’. Some varieties of assessment are closer than second cousins to evaluation, but in order to use different terms we must understand their different meanings and concepts before using them analytically.

Table 1.

Evaluation vocabulary in Finnish legislation, 2014.

Words and expressions in Finnish		Closest equivalents in English		Number of acts or statutes with the word or expression
Word or words	Expression	For words	For expressions	Current acts or statutes	Original acts or statutes
arvioin*	N.A.	evaluation	N.A.	399	1,980
arvioin* AND vaikut*	vaikuttavuuden arviointi, vaikuttavuusarviointi	evaluation AND effectiveness	effectiveness evaluation, impact evaluation	315	976
arvioin* AND yhteiskunnall* AND vaikut*	yhteiskunnallisen vaikuttavuuden arviointi	evaluation AND societal AND effectiveness	evaluation of societal effectiveness	42	88

Source: Finlex database, 11 December 2014. Asterisks (*) indicate truncation to catch words in all their linguistic forms. AND comprises a mere connector.

The core of the current Finnish legislative vocabulary of effectiveness evaluation is summed up in the notion yhteiskunnallisen vaikuttavuuden arviointi. Its official Finnish translation into English, the ‘evaluation of societal effectiveness’, is no less ambiguous than the Finnish original. No Finnish legal text specifies the meaning of this notion, but from the characteristic usage of the notion we can infer that it comprises a catch-all notion referring to the accomplishment of any politically preferred or other substantial and widespread effects in society. The notion was found in 42 current acts or statutes (Table 1) but some of the 42 do not deal with ‘societal effectiveness’ in the proper sense and several other instances of utilization regulate various types of impact assessment without undisputed elements of evaluation. Eliminations left three domains to examine more closely. One includes the universities, removed from the government budget and the civil service in 2010, but with a statutory obligation to pursue ‘societal effectiveness’. The second is the standing orders of the State Council, comprised of the ministers and the twelve ministries together, also prescribing the evaluation of societal effectiveness, as do the standing orders for each ministry. The third domain, established in legislation in the early 2000s (BA, 2013; BS, 2013), instituted the evaluation of societal effectiveness in the annual budget preparation, the medium-range planning in each ministry for its sector of administration, the annual reporting of the agencies and offices, the annual reporting of the ministries on their sectors, and the annual government report to Parliament on the closure of government accounts.

During this research process colleagues continued asking why not use other sources than Finlex to investigate the uses of the Finnish word for ‘evaluation’, arviointi. Besides Finland’s entrenched legalist traditions, the results of an earlier study (Ahonen and Virtanen, 2005) justified the choices of data source. By the middle of the first decade of the 2000s, arviointi had come to be very broadly interpreted in Finland. Many earlier types of study, research or monitoring were renamed arviointi but without introducing changes in substance. Had this article exploited other documents than legal texts, this would have excessively complicated the analysis. However, future research should utilize such tools as big data methods to examine the hundreds of Finland’s annual evaluation reports.

Agency for evaluation in Finland

Agency of the evaluation actors for themselves

The ambiguities in the basic institutionalization of evaluation in Finland need not discourage us from examining agency for evaluation. There is no overarching evaluation institution in the country. The State Audit Office has the independence, but lacks the statutory mandate to re-badge its performance auditing – often undertaken in parallel with evaluation – as evaluation proper (Table 2, item 1.2). Despite efforts, there has been no effective coordination of evaluation within Finland’s national government (Table 2, item 2.0.2; PMO, 2011; MF, 2012).

Table 2.

Aspects of the institutionalization of evaluation in the national government of Finland.

	I Characterization	II Comments	III Estimated emphasis on rationality or legitimation	IV Estimated coupling to decision-making	V Examples of origins of institutional elements
1 Parliament	Commonly obliges government to present ex post evaluations of parliamentary legislation	N.A.	Emphasis on rationality limited by the short term covered by the evaluations	Formally tight, but varies because of Finland’s multi-party rule	Implements global institutional models and scripts; e.g., MF, 2013b
- 1.1 Audit Committee	May commission studies with evaluation characteristics	N.A.	Primarily legitimation	The Audit Committee is none of the committees with foremost legal preparation roles	Created in 2007 analogously as in 2003 in Sweden
- 1.2 State Audit Office (SAO)	Carries out performance auditing and compliance auditing; since 2001 no longer affiliated with government but Parliament	Many performance audits analogous with evaluations, but SAO lacks the mandate to use the term	Mandate accentuates rationality	Independent, excellent public image, but coupling to decision-making rather loose despite re-audit activities	Networking with its global peer institutions (e.g., INTOSAI, 2007)
- 1.3. Social Insurance Institution	May carry out studies with evaluation characteristics	N.A.	Rationality rather than legitimation	Loose except for evaluations of functions under its own executive responsibility	Cooperates with its global peer institutions
2 Prime Minister’s Office	N.A.	N.A.	By and large, see points 2.0.1–2.0.5	N.A.	N.A.
	2.0.1 Evaluates the implementation of the government political program	N.A.	Rationality orientation limited and legitimation orientation enhanced by the four-year electoral cycle		Influenced by global institutional models and scripts; e.g., PMO, 2013b
	2.0.2 Together with MF (see 7 below) loosely coordinates evaluation activities in government	N.A.	Accents on legitimation rather than rationality	Coupling to decision-making generally rather loose	PMO, 2011; MF, 2012
	2.0.3 Outsourced ex ante assessments of European Commission legal and other initiatives, since 2015	Both 2.0.3 and 2.0.4 tap a new funding medium of the PMO introduced in 2014, growing to 12.5 million euros p.a. by 2016	Accents on legitimation rather than rationality	Intended coupling to decision-making tight rather than loose	Uusikylä et al., 2015
	2.0.4 Other possible outsourced assessments and evaluations		N.A.	N.A.	PMO, 2013a
	2.0.5 Evaluation components in studies funded by Council of Strategic Research according to principles defined by government	Another new funding medium; will grow to 63 million euros p.a. by 2016; the first projects start in 2015	N.A.	N.A.	PMO, 2013a
3 Ministry for Foreign Affairs	Evaluation of Finland’s development co-operation, ever since the 1970s	N.A.	Legitimation (intangibles), rationality (program and project management, funding)	Looser (intangibles, legitimation) or tighter (tangibles, rationality)	Harmonized with global and EU guidelines; MFA, 2013
4 Ministry of Justice	Maintains guidelines for the ex ante evaluation of the impacts of all proposed new legislation	The guidelines only indicate the impact types to consider	Legitimation (intangibles), in principle also rationality (tangibles)	Made looser by the absence of the compulsory operationalization of the impacts	Influenced by global models and scripts of legal impact evaluation; see MJ, 2008
- 4.1. Institute for Legal Policy Research	Carries out studies with evaluative characteristics	Formally a unit of the University of Helsinki until 2015, but retains previous functions	Legitimation (intangibles), rationality (processes of legal preparation and implementation)	Made ever looser by new status within a university faculty and strengthening emphasis upon issues of criminology	Networks with its global peer institutions
5 Ministry of the Interior	N.A.	N.A.	N.A.	N.A.	N.A.
6 Ministry of Defense	N.A.	N.A.	N.A.	N.A.	N.A.
7 Ministry of Finance	N.A.	N.A.	By and large, see items 7.0.1–7.0.3 and 7.1		N.A.
	7.0.1 Regulates reporting on the results of evaluations carried out by the ministries, agencies and offices	N.A.	Rationality and legitimation	Made looser by information overload and the high cognitive demands of effectiveness evaluation	Influenced by global models and scripts of evaluation and extended accounting; BA, 2013; BS, 2013
	7.0.2 Coordinates the evaluation of the basic public services	N.A.	Legitimation (intangibles), performance (tangibles)	Loose, as the service providers mostly comprise self-governing municipal organizations	MF, 2013a
	7.0.3 Evaluation of public administration reform policies	N.A.	Both legitimation and performance	Loose rather than tight	Active in the 1990s (e.g., Pollitt et al., 1997), later abated
- 7.1 Government Institute for Economic Research	Characterizes some of its work as policy evaluation	E.g., economic evaluations of social programs	Entrenched provider of studies with evaluation characteristics	Made looser because of staff instead of line organization status, but lately turned visibly tighter	Ilmakunnas et al., 2008
8 Ministry of Education and Culture	N.A.	N.A.	By and large, see points 8.1 and 8.2		N.A.
- 8.1 Center of Education Evaluation	In 2014, assumed responsibilities for evaluation types 8.1.1.-8.1.3 from their previous provider organizations			MEC, 2013a	N.A.
- 8.1.1 Evaluation of compulsory education	Studies of the educational achievement of students finishing their compulsory education	Part of the OECD-PISA studies	Rationality, but spillovers to legitimation from Finland’s excellent results	Coupling made looser by prohibition to publish the results by schools	The PISA studies are an OECD project; see MEC, 2013b
- 8.1.2 Evaluation of vocational and related education	Evaluation of vocational education and adult education	N.A.	Varies	Loose rather than tight	N.A.
- 8.1.3 Higher education evaluation	Audits of quality assurance systems of higher education, evaluations by the Center of Education Evaluation or the universities themselves with Center assistance	Legislation obliges the polytechnics and the universities to be subject to the evaluations	Legitimation rather than rationality	Generally loose	Influenced by global institutional models and scripts; ENQA, 2013
- 8.2 The Academy of Finland	Carries out evaluations of research policies and research fields	Comprises scientific boards and an administrative office	Rationality rather than legitimation	Coupling varies, made looser by staff as opposed to line status of the AF boards and the AF office	Finland’s science policies have been strongly influenced by global and EU models and scripts
9 Ministry of Agriculture and Forestry	See the fourth column	Has important research institutes within its sector of administration	The institutes are entrenched providers of studies with evaluation characteristics	The staff as opposed to line status of the institutions does not prevent their reasonably tight coupling with decision-making	N.A.
10 Ministry of Transport and Communications	N.A.	N.A.	Rationality rather than legitimation	Varies	N.A.
11 Ministry of Employment and the Economy	N.A.	N.A.	Too diffuse to indicate in detail; however, let us mention that the ministry carries out regulatory impact assessment (RIA) using the common standard cost model to assess regulatory burdens		RIA derives from global institutional models and scripts
- 11.1 Technological Research Institute	May carry out evaluation studies on commission	N.A.	Entrenched provider of studies with evaluation characteristics	Looser because of staff instead of line organization status	Following international examples, will turn into a government-owned company in 2015
- 11.2 Funding Agency for Technology and Innovation	Has the programs and projects that its funds systematically evaluated	N.A.	There is too much variance in these aspects to allow for estimation		Evaluation much influenced by EU institutional models and scripts
12 Ministry of Social Affairs and Health	Among its other duties, maintains within its organization the Finnish Office for Health Care Technology Assessment (FINOHTA)	Promotes impact assessment related to health, welfare, gender, social and human aspects	More rationality orientation in FINOHTA, more legitimation orientation in the other assessment activities	Tighter where rationality receives emphasis, looser where legitimation is important	Health technology assessment in Finland derives from global models and scripts (INAHTA, 2013)
- 12.1 National Institute of Health and Welfare	N.A.	Carries out evaluations of health care and social welfare, including in its FinSOC unit for social work evaluation	Entrenched provider of studies with evaluation characteristics	Loose rather than tight; generally tighter in health than in social welfare	N.A.
- 12.2 Finnish Institute of Occupational Health	Carries out academic and applied evaluation studies	N.A.	Entrenched provider of studies with evaluation characteristics	Made looser by the corporatist influence of the employees and employers in occupational health	Connected with its global counterpart organizations
13 Ministry of the Environment	N.A.	A foremost actor in statutory environmental impact assessment (EIA)	Rationality (in principle), legitimation (where evaluation is more of a formality)	Looser where vested interests are strong	EIA is a global innovation (Pope et al., 2013) diffused in Finland
- 13.1 Finnish Environmental Institute	Carries out academic and applied evaluation studies	N.A.	Entrenched provider of studies with evaluation characteristics	Looser because of staff instead of line status	N.A.

Explanation: Subordinate bodies of functions are indicated by a dash below the name of their main body.

Table 2 implies that typically evaluators within Finland’s national government are employees of agencies and offices subordinated to the ministries rather than officials in the ministries themselves. Many other evaluators work for global and domestic consulting companies and other organizations that win evaluation contracts from government organizations.

Evaluation agency on behalf of others

The constitutional obligation of Finland’s government to respond to parliamentary requests for comprehensive ex post reporting on the results of legislative reforms establishes an important type of evaluation agency in the country (Table 2, item 1). Further types of evaluation agency are written into legislation on individual policy fields; in result contracts between ministries and their subordinate agencies and offices; and business contracts between outsourced providers of evaluation services and government organizations.

Interviewees reported that no profession proper has evolved in Finland that could be said to have agency for evaluation. Since 1992, those who have sought to acquire Finland’s Certified Degree of Auditors of Public Administration and the Public Finances (JHTT) have had to pass evaluation assignments as part of their examination. However, most of those who have acquired this degree (JHTTs) take on few evaluation roles. Since its establishment in 1999, the Finnish Evaluation Society (FES), with approximately 200 members, has made valuable contributions. However, it has not forged a unitary body of evaluators, and many evaluators have never joined the society.

At the time of this study, only a few companies – micro or small – indicated exclusive specialization in evaluation in Finland, although many larger companies offered evaluation expertise alongside other services. This reflects a small evaluation market, although the Finnish market has not been too small to attract some foreign companies in taking over Finnish-owned evaluation operations.

Evaluation agency representing general principles and standards

Neither the members of Finland’s evaluation community nor the government have produced their own evaluation standards and principles. One of the interviewees argued that market competition provides better quality assurance than formal standards and principles. Furthermore, a competitive evaluation market can be identified between the approximately 100 organizations commissioning evaluations in Finland’s national government; and the providers of evaluation services.

Finland’s evaluators frequently apply international standards and principles (e.g. AEA, 2013), and some government organizations have explicitly harmonized their evaluation guidelines with these standards and principles. One early example is the evaluation of Finland’s development cooperation (Table 2, item 3) which has for several decades subscribed to international – and since the 1990s also to EU – evaluation standards and principles (MFA, 2013).

A set of softer and more academic standards is also applied by Finland’s scholars and practitioners of evaluation, derived from all three types of evaluation roots distinguished by Alkin (2012): methods, valuing and utilization. Evaluation deriving from the ‘methods’ roots evolved from early quasi-experiments on speed limits (Salusjärvi and Kontiala, 1975) in line with the evaluation classic of Campbell and Ross (1968). Evaluative studies of school achievement also sprang up early on (Renko, 1971). Certain non-experimental and experimental economic analyses have been re-named evaluations (Ilmakunnas et al., 2008), and evidence-based evaluation within health care applies entrenched conventions of randomized control trials (Hakama et al., 2012). Some of Finland’s education scholars have stressed Alkin’s ‘valuing’ roots of evaluation (Keskitalo et al., 2012), and Alkin’s ‘utilization’ roots (Patton, 2008) have also featured (Berg and Hukkinen, 2011). Harder-to-classify evaluation types – found in Finland no less than elsewhere – include ‘fourth-generation evaluation’ (García-Rosell and Mäkinen, 2013; Guba and Lincoln, 1989), evaluation that implements critical scientific realism (Holma and Kontinen, 2010; Pawson and Tilley, 1997), and studies in evaluation ethics (Laitinen, 2008; Schwandt, 2002).

Numerous doctoral dissertations have been written in Finland on evaluation since the earliest study (Sintonen, 1981), but evaluation is not among the country’s academic disciplines awarding master’s and doctoral degrees, nor is it the main theme in any of the thematic master’s programs. Interviewees vented their frustration over common academic reluctance to assign full intellectual value to research that involved evaluation. However, the vigorous global publication activity by Finland’s evaluation scholars questions the claims of deficiency.

Diffusion, performance, legitimation and coupling

Having considered both the basic institutionalization of evaluation and agency for evaluation in Finland, let us have a look at the ‘going concern’ of Finnish evaluation in its practical institutionalization. As the conceptual framework implies, the examination has to provide indications of the diffusion, modification and sedimentation of global institutional elements of evaluation in the Finnish case (Table 2, last column). The analysis also considers the enhancement of rationality and the enhancement of legitimation by means of evaluation, and at the tight or loose coupling between evaluation and decision-making. Alongside evaluation proper, the analysis includes practices close enough to evaluation to merit inclusion.

High politics, the elevated authority of institutions and the political importance of issues crowd out rationality and accentuate legitimation in evaluation (Table 2, items 1 and 1.1). In one case, the relative shortness of the electoral cycle – four years at the maximum – appears to crowd out rationality (item 2.0.1), in another the proximity to high politics exerts an analogous impact (items 2.0.2 and 2.0.3), and in a third case, limited operationalization restricts rationality (item 4). Within higher education policy, legitimation is emphasized (item 8.1.3). A spillover from the excellent results in evaluating the efficiency and effectiveness of compulsory education in Finland is that they also provide this country’s system of compulsory education with general legitimation (item 8.1.1). Finland’s foremost impact assessment practice builds on institutionalized statutory procedures enacted in the name of rationality enhancement, but does not necessarily ensure actual rationality, rather guaranteeing formal compliance (item 13). However, rationality and legitimation commonly coexist as aspects of evaluation.

The coupling of evaluation with decision-making is generally loose rather than tight in Finland’s national government. However, tight and loose coupling commonly co-exist (items 1, 1.3, 3, 7.0.1, 9, 10, 12 and 13). The coupling is looser in high politics (items 1, 2.0.1 and 2.0.2), if operationalization is limited (items 4 and 7.0.2), or if constitutional or other statutory separation between legislative and executive powers exerts an influence (items 1.2, 1.3, and 7.0.2).

Insofar as organizations or functions have a staff status (with preparatory or other assisting tasks) as opposed to a direct line status (with direct responsibilities under the command of organizational leadership), this weakens the coupling of their evaluation results with the decision-making of the government ministries (items 4.1, 7.1, 8.1.3, 8.2, 9, 11.1, 12, 12.1, 12.2 and 13.1). In one case, the publication of output other than summary evaluation results is prohibited; this is to prevent tight coupling to decision-making on individual schools (item 8.1.1). Vested interests of the corporatist variety may also influence the coupling of evaluation to decision-making (12.2). There have nonetheless been more recent government efforts to tighten the coupling of evaluation to government decision-making (items 2.0.3. and 2.0.4), including the introduction of a new assessment practice (item 2.0.3).

Intense agent-driven institutional change

Intense agent-driven institutionalization within Finnish evaluation has happened in some sectors and circumstances. Since the 1960s development co-operation led to the first institutionalization of evaluation proper in the country in order to legitimate allocations of the government resources (Table 2, item 3). Quasi-experimental studies on speed limits and studies on teacher efficiency (Renko, 1971; Salusjärvi and Kontiala, 1975) referred to earlier also fall into the ‘intense’ category.

In 1987–91, evaluation was largely overshadowed in Finland by New Public Management emphases on operational performance, and in 1991–94, by the exigencies of preventing the general collapse of the economy (on this, see the chapter on Finland in Pollitt, 2013). Finland’s accession to the EU in 1995 catalyzed the ex ante evaluation of the structural funds programs and projects (Table 2, item 11). Later, ex ante evaluation expanded into legal policy-making and legal preparation (items 4 and 4.1; Tala, 2010). Finland harmonized the evaluation of its development co-operation and its research and technological development policies with their EU counterparts by 1995 (items 3, 8.2 and 11.2; MFA, 2013). Evaluations required by the ‘open method of coordination’ of EU policies, evolved since 1997, have been also implemented in Finland (Ahonen and Virtanen, 2008). Since 2015, Finland implements systematic national ex ante assessments of the Finnish impact of European Commission legal and policy initiatives (Uusikylä et al., 2015; Appendix, item 2.0.3.; see also items 2.0.4–2.0.5).

An important threshold year for evaluation in Finland was 1995, marking the multiplication in types and targets of evaluation in ways that suggest partial substitution of what can be called ‘post-NPM’ for previous NPM procedures (see Pollitt and Bouckaert, 2011). Evaluation expanded, for instance, to cover national public administration reforms (Pollitt et al., 1997), public services, individual government organizations, and policies, programs and projects. However, as indicated previously, the word arviointi (‘evaluation’) became a buzzword connected with many pursuits, many of which had little to do with evaluation as understood within the global evaluation community.

In the future, focused studies should examine further cases of intense agent-driven change in Finnish evaluation. This concerns, for instance, the 2003–4 reform that introduced statutory procedures covered by the ambiguous notion of ‘societal effectiveness’ (BA, 2014; BS, 2014; MF, 2003). This evaluation reform evolved not only as an extension of government accounting, but also as a novel dimension of budgetary accountability, and a new content element in the annual government accounting. This reform substituted an effectiveness orientation for the traditional compliance orientation, and further supplemented the efficiency orientation of NPM with post-NPM characteristics.

Conclusions and implications

This article has sought answers to four research questions derived from a variant of neo-institutional theory (best summarized in Kruecken and Drori, 2009) in its examination of the institutionalization of evaluation in the national government of Finland. The article has contributed with a country case study on the institutionalization of evaluation grounded on a systematic theoretical framework. The empirical results indicate ambiguity in the basic institutionalization of evaluation in Finland and in the agency of the actors carrying out, commissioning or using evaluations. However, evaluation principles, standards, approaches and methods give coherence. With tighter or looser coupling to decision-making, some of the Finnish institutional practices of evaluation enhance formal rationality such as efficiency and effectiveness, some support legitimation, and the other practices contribute to both ends. There have also been instances of intense agency-driven institutional change in Finnish evaluation.

A referee usefully asked if evaluation in Finland’s national government is institutionalized at all. One interviewee proposed that evaluation is institutionalized up to strong routinization, although this routinization is characteristically confined only to ‘particular knowledge practices’ (erityiset tietokäytännöt). In general, the neo-institutionalism that this article applies fully agrees with views that global pressures towards institutional isomorphism– that also affects evaluation – may not lead to homogeneity because of the wide variety of contexts in which these forms find their applications (see Beckert, 2010; Pollitt, 2013).

At its beginning, this article promised to open up one and only one perspective from among the others to examine the institutionalization of evaluation. However, this limitation by no means rules out examination of the institutionalization of evaluation in other countries than Finland in analogous theoretical and methodological terms as in this article.

The implications of this article for practice are stronger and more general than its immediate implications for future research. The analysis of the basic instutionalization of evaluation suggests that the evaluation practitioner should try to live with ambiguity with respect to notions of evaluation and the agency of the evaluator, the characteristics of evaluation commissioners and the multiple possible uses of the evaluation results. A clear-headed evaluator should find good use for the rationality versus legitimation division in evaluation and the various possible degrees of coupling between evaluation and decision-making. Last but not least, would not many a researcher on evaluation and many an evaluation practitioner prefer to turn into an agent of intense change for the very best of evaluation?

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Pertti Ahonen (DSocSc, MScEcon) is Professor of Political Science at the University of Helsinki, and has been an evaluation scholar and practitioner for 35 years.

References

AEA (American Evaluation Association) (2013) Program evaluation standards. URL: http://www.eval.org/p/cm/ld/fid=103.

Ahonen

Virtanen

(2005) Unpublished book manuscript with an empirical study of evaluation in Finland, in Finnish. Tampere: University of Tampere.

Ahonen

Virtanen

(2008) Understanding evaluation in the social sector: Two reflexive metaevaluation case studies inspired by Pierre Bourdieu. International Journal of Public Administration 31(10–11): 1146–66.

Alkin

(ed.) (2012) Evaluation Roots: A Wider Perspective of Theorists’ Views and Influences, 2nd edn. Thousand Oaks, CA: SAGE.

BA (Budget Act) (2014) Originally 1988, with later amendments. In Finnish. URL: http://www.finlex.fi/fi/laki/ajantasa/1988/19880423.

Beckert

(2010) Institutional isomorphism revisited: Convergence and divergence in institutional change. Sociological Theory 28(2): 150–66.

Berg

Hukkinen

(2011) Beyond effectiveness: The uses of Finland’s national programme to promote sustainable consumption and production. Journal of Cleaner Production 19(16): 1788–97.

BS (Budget Statute) (2014) 1992, with later amendments. In Finnish. URL: http://www.finlex.fi/fi/laki/ajantasa/1992/19921243.

Campbell

Ross

(1968) The Connecticut crackdown on speeding: Time-series data in quasi-experimental analysis. Law & Society Review 3(1): 33–54.

10.

Dirsmith

Fogarty

Gupta

(2000) Institutional pressures and symbolic displays in a GAO context. Organization Studies 21(3): 515–37.

11.

Drori

Meyer

(2006) Scientization: Making the world safe for organizing. In: Djelic

M-L

Sahlin-Andersson

(eds) Transnational Governance. Cambridge: Cambridge University Press, 33–52.

12.

Drori

Meyer

Hwang

(2009) Global organization: Rationalization and actorhood as dominant scripts. In: Meyer

Sahlin

Ventresca

. (eds) Ideology and Institutions. Bingley: Emerald, 17–43.

13.

Eaton

(1962) Symbolic and substantive evaluative research. Administrative Science Quarterly 6(4): 421–42.

14.

ENQA (European Association for Quality Assurance in Higher Education) (2013) Homepage. URL: http://www.enqa.eu.

15.

Finlex (2012) Database of Finland’s legislation. URL: http://www.finlex.fi.

16.

García-Rosell

J-C

Mäkinen

(2013) An integrative framework for sustainability evaluation in tourism: Applying the framework to tourism product development in Finnish Lapland. Journal of Sustainable Tourism 21(3): 396–416.

17.

Goertz

Mahoney

(2012) A Tale of Two Cultures: Qualitative and Quantitative Research in the Social Sciences. Princeton, NJ: Princeton University Press.

18.

Guba

Lincoln

(1989) Fourth Generation Evaluation. Newbury Park, CA: SAGE.

19.

Gubrium

Holstein

Marvasti

. (ed.) (2012) The SAGE Handbook of Interview Research: The Complexity of the Craft. London: SAGE.

20.

Hakama

Malila

Dillner

(2012) Randomised health services studies. International Journal of Cancer 131(12): 2898–902.

21.

Holma

Kontinen

(2011) Realistic evaluation as an avenue to learning for development NGOs. Evaluation 17(2): 181–92.

22.

Ilmakunnas

Jukka

Uusitalo

(eds) (2008) Vaikuttavaa tutkimusta: miten arviointitutkimus palvelee päätöksenteon tarpeita? A study on evaluative economic research for decision-making, in Finnish. Helsinki: Government Institute for Economic Research.

23.

INAHTA (International Network of Agencies for Health Technology Assessment) (2013) Homepage. URL: http://www.inahta.org.

24.

INTOSAI (International Organization of Supreme Audit Institutions) (2007) INTOSAI Working Group on Program Evaluation, Program Evaluation for SAIs: A Primer. London: National Audit Office.

25.

Jordanova

(2006) History in Practice. London: Bloomsbury Academic.

26.

Keskitalo

Määttä

Uusiautti

(2012) Sámi education in Finland. Early Child Development and Care 182(3–4): 329–43.

27.

Kruecken

Drori

(ed.) (2009) World Society: The Writings of John W. Meyer. Oxford: Oxford University Press.

28.

Laitinen

(2008) Autenttinen arviointietiikka. A study in the ethics of evaluation, in Finnish. Rovaniemi: University of Lapland.

29.

Lowndes

Roberts

(2013) Why Institutions Matter: The New Institutionalism in Political Science. Houndmills: Palgrave Macmillan.

30.

MEC (Ministry of Education and Culture) (2013a) Koulutuksen arviointikeskuksen perustaminen. On founding the Center of Education Evaluation, in Finnish. URL: http://www.minedu.fi/OPM/Koulutus/koulutuspolitiikka/vireilla_koulutus/.

31.

MEC (Ministry of Education and Culture) (2013b) OECD Pisa Survey. URL: http://www.minedu.fi/OPM/Koulutus/artikkelit/pisa-tutkimus/?lang=en.

32.

Meyer

Jepperson

(2000) The ‘actors’ of modern society: The cultural construction of social agency. Sociological Theory 18(1): 100–20.

33.

Meyer

Rowan

(1977) Institutional organizations: Formal structure as myth and ceremony. American Journal of Sociology 83(2): 340–63.

34.

Meyer

Hammerschmid

(2006) Changing institutional logics and executive identities: A managerial challenge to public administration in Austria. American Behavioral Scientist 49(7): 1000–14.

35.

MF (Ministry of Finance) (2003) Parempaan tilivelvollisuuteen: valtion tilinpäätösuudistuksen periaatteet. Report on the reform of government accounting closure, in Finnish. Helsinki: MF.

36.

MF (Ministry of Finance) (2012) Uudistamme yhdessä valtionhallintoa. Brochure on the government program on effectiveness and performance, in Finnish. Helsinki: MF.

37.

MF (Ministry of Finance) (2013a) Peruspalvelujen arviointi. The evaluation of the basic services, in Finnish. URL: http://www.vm.fi/vm/fi/13_hallinnon_kehittaminen/03_palvelujen_laatu/.

38.

MF (Ministry of Finance) (2013b) Valtioneuvoston selonteko eduskunnalle aluehallintouudistuksen toimeenpanosta ja toteutumisesta. Government report to Parliament on the effects of the reform of regional administration, in Finnish. Helsinki: MF.

39.

MFA (Ministry for Foreign Affairs) (2013) Evaluations. URL: http://formin.finland.fi/public/default.aspx?nodeid=15454&contentlan=2&culture=en-US.

40.

MJ (Ministry of Justice) (2008) Impact Assessment in Legislative Drafting: Guidelines. Helsinki: MJ.

41.

OECD (2010) Finland: Working Together to Sustain Success. Paris: OECD.

42.

Patton

(2008) Utilization-Focused Evaluation, 4th edn. London: SAGE.

43.

Pawson

Tilley

(1997) Realistic Evaluation. London: SAGE.

44.

Peters

(2011) Institutional Theory in Political Science: The ‘New Institutionalism’, 3rd revised edn. London: Continuum.

45.

PMO (Prime Minister’s Office) (2011) Poliittisen toiminnan tietopohjan parantaminen. On improving the information basis of political decisions, in Finnish. Helsinki: PMO.

46.

PMO (Prime Minister’s Office) (2013a) PMO budget proposal for 2014, in Finnish. URL: http://vnk.fi/ajankohtaista/tiedotteet/tiedote/fi.jsp?oid=391631.

47.

PMO (Prime Minister’s Office) (2013b) Hallitusohjelman tavoitteiden ja toimien toteuma. Mid-term evaluation of the government political program, in Finnish. Helsinki: PMO.

48.

Pollitt

(ed.) (2013) Context in Public Policy and Management. Cheltenham: Edward Elgar.

49.

Pollitt

Bouckaert

(2011) Public Management Reform: A Comparative Analysis – New Public Management, Governance, and the neo-Weberian State, 3rd edn. Oxford: Oxford University Press.

50.

Pollitt

Hanney

Packwood

. (1997) Trajectories and Options: An International Perspective on the Implementation of Finnish Public Management Reforms. Helsinki: Ministry of Finance.

51.

Pope

Bond

Morrison-Saunders

. (2013) Advancing the theory and practice of impact assessment: Setting the research agenda. Environmental Impact Assessment Review 41(1): 1–9.

52.

Renko

(1971) Opettajan tehokkuus oppilaiden koulusaavutusten ja persoonallisuuden kehittämisessä. A study on teacher efficiency, in Finnish. Jyväskylä: University of Jyväskylä.

53.

Salminen

Viinamäki

O-P

Jokisuu

(2012) Finland. In: Verhoest

Van Thiel

Bouckaert

Lægreid

(eds) Government Agencies: Practices and Lessons from 30 Countries. Houndmills: Palgrave Macmillan, 223–33.

54.

Salusjärvi

Kontiala

(1975) Tiekohtaisten nopeusrajoitusten kokeilu 1973: vaikutukset liikenneonnettomuuksiin. A quasi-experimental study on speed limits, in Finnish. Otaniemi: VTT.

55.

Schneiberg

Clemens

(2006) The typical tools for the job: Research strategies in institutional analysis. Sociological Theory 24(3): 195–227.

56.

Schwandt

(2002) Evaluation Practice Reconsidered. New York: Peter Lang.

57.

Silverman

(2011) Interpreting Qualitative Data, 4th edn. Los Angeles, CA: SAGE.

58.

Sintonen

(1981) An Approach to Economic Evaluation of Actions for Health. Helsinki: Ministry of Social Affairs and Health.

59.

Strang

Macy

(2001) In search of excellence: Fads, success stories, and adaptive emulation. American Journal of Sociology 107(1)L 147–82.

60.

Stufflebeam

Shinkfield

(2007) Evaluation Theory, Models, and Applications. San Francisco, CA: Jossey-Bass.

61.

Tala

(2010) Better regulation through programs and quality standards – are new perspectives needed? Legisprudence 4(2): 193–212.

62.

Uusikylä

Ahonen

Takanen

(2015) EU-aloitteiden vaikutusten ennakkoarvioinnin tehostaminen. A study on improving the ex ante assessment of European Commission legal and policy initiatives in Finland, in Finnish. Helsinki: Prime Minister’s Office.

63.

Weiss

(1970) The politicization of evaluation research. Journal of Social Issues 26(4): 57–68.

64.

Zucker

(1977) The role of institutionalization in cultural persistence. American Sociological Review 42(5): 726–43.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.20 MB