Abstract
This article
Introduction
The final decade of the last century initiated an ‘evidence turn’ in public policy-making across OECD countries that changed the interrelation between politics and evidence. As a result of “an upsurge of interest in evidence-based policy and practice” [1] ever since, evidence-informed policy-making2 (EIPM) seems to have become the new normative normal of good public governance that is supported by international organisations such as the United Nations, the OECD, and the EU [4, 5, 6]. The beginnings of evidence-based policy originated after World War II in the area of medicine and extended from there in the 1980s to health-related social policies [7, see also 8]. Linked to this development was the “claim that rigorous evaluation practices can significantly improve attainment of cost-effective outcomes” [8]. The same idea was reflected in the neo-liberal doctrine of the era and the parallel rise of new public management (NPM) as “one of the most striking international trends in public administration” [9] to rationalise public service delivery through assessment and monitoring. NPM promoted priorities that are also reflected in the rationale of EIPM, such as the improvement of public administrations through new managerialism, accountability for performance, performance measurement and auditing, strategic planning and management, policy analysis and evaluation as well as rationalisation and streamlining of administrative structures [10].
EIPM became increasingly relevant for policy-making during the late 1990s and early 2000s, when the “‘evidence-based policy’ movement …sought to promote rigorous analysis of policy and program options, with the intention of providing useful inputs for policy makers in their ongoing consideration of policy development and program improvement” [8]. This new uptake of EIPM in the 1990s was prominently promoted by newly elected social democratic governments across Europe, especially for the development of alternative social and healthcare policy paradigms and the establishment of expertise-led approaches to rationalise policy-making. In this period, the United Kingdom emerged as a cradle of EIPM with a “New Labour enthusiasm for evidence-based policy” [11] which “at the policy level” gave the “evidence-based approach …a significant boost by the advent of a Labour government with a pragmatic, antiideological stance” [1] of its ‘what matters is what works’ approach [11, 12].3 In the context of these developments, policy programmes as well as their delivery and outcomes measurement became subject to structural and procedural innovation. These changes adhered to the logics of EIPM that impact not only on policy content, but also on policy-making, scientific expertise and forms of evidence provided and used to design policies. For 21
Evidence-informed policy-making in the 21
century
EIPM is broadly defined as policy-making informed by evidence that influences all stages of the so-called ‘policy process’, which – even if in a simplified linear perspective – includes agenda setting, policy formulation, legitimation/decision-making, implementation, evaluation, as well as maintenance, succession or termination [2, 3, 17, 18]. In its most consequent form, EIPM results in decisions being based on evidence. This connection of policy-making with the use of evidence impacts procedurally, structurally, ideationally and strategically on politics of which it is both a means and an end as it “represents both an important set of professional practices and aspirations; and also a political rhetoric seeking to legitimate forms of decision-making” [13].
In its assumptions, the concept of EIPM reflects a differential impact of the use of evidence on actor groups involved in policy-making in different stages of the policy process. It acknowledges that, “[w]hile access to accurate information is very important in all agencies, the specific administrative practices and procedural rules governing information selection and use in each type of organization are crucial for the way evidence is identified and utilized. Thus, patterns of evidence use and information management vary across policy domains (e.g., social policy, economic development, environmental regulation) and across organizational types associated with different public sector functions (e.g., service delivery, regulatory oversight, and policy development)” [8]. Moreover, “the use of evidence for policy has been described as ‘qualitatively different’ [19] than its use in technical decision-making arenas (such as clinical medicine)” [20]. In this sense, public evidence-informed policy-making is acknowledged to be different from “technical decision making. Rather, policymaking typically involves trade-offs between multiple competing social values, with only a very small proportion of policy decisions simply concerned with technical evidence of the effects of interventions” [20]. When analysing EIPM, therefore a rational choice and a behavioural perspective inform the debate. The former takes a ‘positivist empiricists’ stance, while the latter offers a ‘critical interpretivists’ view on the concept. They reflect the normative dimension of either a strong belief in, or doubts about EIPM and its capacity to inform policy-making through knowledge-sharing and policy-learning [20, 21].
Systemically, the concept of EIPM emphasises the input and throughput side of the policy process and indicates that evidence has been injected into policy-making, which is that policy-making has been informed by science and evidence. With this focus, the assumptions about the effects of EIPM are more flexible [22] and cautious about the real impact of evidence on policy-making than those of the concept of ‘evidence-based policy-making’ (EBPM) [20, 23]. The latter embraces a throughput- or output-oriented perspective and assumes that the evidence injected into the policy process indeed influences policy-making not only procedurally, but also in terms of actual content. This assumption however has been criticised “as ‘naïve rationality’ – incorrectly assuming that policymaking is merely an exercise in ‘decision science’, when the policy process is, instead, a ‘struggle over ideas and values’ [24]” [20]. With this nuancing [25], EIPM – more than EBPM – pays tribute to the fact “that policy making is an inherently political process … [26, 27, 28], involving ideology, vested interests, institutional norms and path dependencies … [29]” [11, see also 8].
Science and knowledge as resources
Within EIPM, science and different forms of knowledge (including data) become important political resources.4 Science-based advice, knowledge management and communication of evidence therefore become central features and aims of the policy process itself. The selection of evidence within the different stages of the policy process is influenced by the capacity of actors involved in policy-making to assess, evaluate and understand research and scientific evidence just as much as by the capacity of scientists to explain and translate evidence for policy-making to produce evidence that can inform the policy process. Essential skills actors involved in policy-making need to acquire for EIPM are hence to properly obtain, interrogate, assess, use and evaluate evidence and the adequate engagement with stakeholders [5]. Apart from such basic evidence literacy skills, evidence selection also depends on ‘rational’ (goal-oriented selection of evidence) and ‘irrational’ (stricto sensu belief- and/or emotion-based prioritisation of information and trust in evidence and experts) behaviour of actors involved in policy-making [18]. In this understanding, the use of evidence as political resources in policy-making is also influenced by academic, political, individual and organisational incentives to disseminate and use certain forms of evidence and, if the incentives are right, can inspire increased interaction between political practice (political actors) and science (including academics and experts).
Institutionalisation of science advice
Such increased interrelations between experts and actors involved in policy-making inspires the institutionalisation of science advice in the policy process to an extent that EIPM has strong effects on the governance of the science-policy-interface (SPI) and on governance in general. This institutionalisation of the SPI reflects the assumption that for “addressing complex policy and program areas …collaborative approaches to knowledge sharing and adaptive management in light of experience will be necessary” [8].
A plethora of different systems of science-based advice has been institutionalised across all levels of governance with the independence of expertise being one of the key trust catalysts in these institutional contexts. Prominent examples for such institutionalisation are positions of national chief science advisors (like in Ireland or Estonia); scientific advisory councils (as in Bulgaria, the Czech Republic, Denmark, France, Malta, or The Netherlands); ad hoc or permanent expert commissions (like in Germany); strategy units attached to the offices of the heads of state and government (like in Austria and Finland); networks of science advisors in ministries; governments’ strategic research councils; parliamentary research services; national academies of sciences; informal networks of public knowledge organisations; the European Commission’s Group of Chief Science Advisors;5 the European Science Advisors Forum;6 the International Network for Government Science Advice;7 and the United Nations Convention to Combat Desertification’s Knowledge Hub’s Science Policy Interface.
Less institutionalised and more flexible arrangements result from the fact that, also due to the demand for independent scientific advice, “relationships between research, knowledge, policy and practice are always likely to remain loose, shifting and contingent” [31]. Such more loosely coupled evidence arenas, like Communities of Practice, are alternatives to the formal institutionalisation of science-politics relations. Actors engaged in Communities of Practice pursue a common professional practice and share a common knowledge base that interlinks them across institutional or organisational boundaries without formal institutionalisation. Common challenges, problems and preferences for solutions frame their common knowledge pool. Such Communities of Practice usually “arise spontaneously or come together through seeding and nurturing …[;] to legitimize the community as a place for sharing and creating knowledge, recognized experts need to be involved” [32]. These recognised experts can be core groups of thought leaders, experts that document practices through collection and organisation of knowledge, or those that connect the Communities of Practice to other communities. Following Wenger [33, 34], Communities of Practice constitute social learning systems that transcend organisational borders. They are “informally bound by what they do together …and by what they have learned through their mutual engagement in these activities” [32, see also 35]. They support taking “collective responsibility for managing the knowledge they need”; they “create a direct link between learning and performance”; they enable practitioners to “address the tacit and dynamic aspects of knowledge creation and sharing” and “are not limited by formal structures: they create connections among people across organizational and geographic boundaries” ([34] for all citations in this sentence). With these characteristics they form more integrative evidence arenas for the generation, adaptation, validation, selection and sharing of knowledge in specific contexts that aim at connecting practitioners, exploring and exploiting knowledge, nurturing creative thinking, facilitating practitioner feedback on policies and programme design, and replicating good practices as in the case of the UN’s Solution Exchange programme in India.8 The EU’s Community of Practice for better self- and co-regulation of digital markets9 or the European Commission’s Joint Research Centres’ 2020 workshop series on ‘Strengthening and connecting eco-systems of science for policy in Europe’10 and its associated inter-disciplinary, inter-sectorial and multi-national expert group is another example that reflects features of Communities of Practice at EU level.
In this way, Communities of Practice are essential for the development and diffusion of knowledge and evidence in organisations or areas. Within EIPM, they can constitute a good nucleus for knowledge- and capacity-building and for the creation of joint understandings for the quality and limitations of evidence. They do not formalise knowledge, but rather curate it as a ‘living’ resource that can be contextualised where necessary. “Communities of practice preserve the tacit aspects of knowledge that formal systems cannot capture” [32], potentially making evidence more actionable. As problem-focussed collaboration is one of their essential features, they are platforms to adapt knowledge and to innovate and as such they have proven to be useful tools in policy-making in the EU and UN examples provided above.
Feeding-in evidence
In terms of the translation of information and knowledge, “the gap between the needs of policy makers and the ways researchers present evidence” [36] is a central challenge for the injection of evidence into policy-making which, in turn, enhances “the non-systematic or almost accidental feature of the processes leading to its inclusion” [37]. Patterns of inclusion of evidence in policy-making are consequently not uniform, but respond to different needs, such as ad hoc or permanent science advice, general advisory functions, or the requirements of policy evaluation and impact assessment. Following from that, the translation and “processing of …information and expert knowledge is problematic and highly variable across organizations” [8]. This challenge of EIPM was already at the heart of the ‘two communities’ theory of research utilisation of the 1970s. The theory held that actors engaged in EIPM (both in the production and the use of evidence) do not form a uniform evidence group, but are divided into two different subgroups – researchers/scientists (academia/supply) and politicians/public officials (policy/demand). These subgroups “were poorly connected and operated under different rules, spoke different languages, and were motivated by different rewards systems” [38]. This early analysis is, however, increasingly contested by empirical research, which shows that the different communities – also due to lose coupling such as in Communities of Practice – are neither completely separated, nor fully integrated.
Adding to the intervening factors for feeding-in of evidence into policy-making is what Bannister and O’Sullivan describe as the “evidence-policy nexus”. It claims that “the relationship between evidence and policy is fundamentally conditioned by the policy issue at hand, the point in the policy cycle (from initiation to evaluation) at which the relationship is judged, as well as broader considerations such as resource (money, time) availability” [11]. Based on the complexity of policy ecosystems, the “politics of decision-making inherently involves a mixing of science, value preferences, and practical judgements about the feasibility and legitimacy of policy choices” [13]. As a consequence, biases and the structural-procedural logics of EIPM impact on the selection of evidence; on the transparency, accountability and legitimacy of evidence production, selection and use; on discursive politics and deliberative practices [18, 39]. In the most dysfunctional case, dangers of ‘policy-based evidence making’ [40] can emerge as a sign of flawed interrelations within the “evidence-policy nexus” [11]. Contextualisation is therefore an essential requirement and a major challenge of EIPM to identify the adequate “nature of policy, of evidence, and of the relationship between them” [11]. EIPM thus needs to embed “the use of evidence and the knowledge conversion process in a multi-actor policy trajectory” [41] that relates political actors and interests; knowledge reception, perception, production and content; evidence quality; governance structures; policy paradigms and instruments to increase quality, efficiency and transparency of reason and rationale behind decisions. Supportive evidence-based policy design frameworks, such as New Zealand’s ‘Policy Methods Toolbox’,11 have been created to support this conversion process and the use of evidence in wider policy ecosystems.
Data as evidence in policy-making
History informs us that the connection between knowing and governing links statistics and politics ever since the 17
Data as governance tool
While a lot of academic reflection focusses on the micro level of data development and statistical methods, macro level analysis of the overall position and relevance of data in EIPM are essential to understand the challenges ahead for data providers and actors involved in policy-making related to the production and use of data (and metadata) as evidence. In this macro level understanding, “numbers are a technology of government” [17] and statistical data are key governance tools. ‘Numbers’ represent data, information, narratives and knowledge and are one of the most essential sources of evidence for policy-making [45].
As such, data12 provide a “vector of acting by knowing” [45] in EIPM. Data manifest approximations to social reality, they stand in dialectical relation with social action and are essential means of policy-making. In the form of indicators, indices, composite indicators, and scoreboards, they are key contemporary instruments to quantify, qualify and compare complex processes, structures, and situations. In this way, they support the performance assessment of nations, are essential means of collective political action [45] and “one of the ways of building coalitions in governance” [17].
The connection between statistics and politics materialises the essential feature of ‘governing by knowing’ in 21
Within this framing of social reality, conflicting impacts are at play. The inherent logic of EIPM is assessing and steering based on evidence; yet, if politics regulate what is measured and measurable, aspects that are difficult to measure (such mental well-being or the multi-dimensionality of intelligence) or complex phenomena (like corruption and inter-generational equity) risk facing less evidence-informed deliberation in policy-making due to missing or less standardised data. The downsides for an encompassing policy-making and for progress in general seem obvious.
Data as robust form of evidence
Generally, evidence for policy-making can be abstract, practical, objective, or subjective. In qualitative terms, it can be statistical, narrative, or anecdotal. Data evidence in policy-making can derive from statistical and survey data, administrative data, big data, (composite) indicators, indexes, aggregations such as national and environmental accounts, modelling, cohort or case studies, ex-ante or ex-post policy evaluation, impact assessments, scientific and research results, systematic reviews, or information based on randomised controlled trials [11, 26, 46].
Going beyond the purely metrological purpose of quantification and measurement, as one of the most robust types of evidence, statistical data and especially official statistics have become one of the most important forms of factual evidence in EIPM. The impact of data as evidence in EIPM is manifold. It covers directions that Radermacher frames as “Data to policy/politics” (which entails “statistics …driving policy development, priority setting, foresight” and the “Governance and institutionalisation of sense-making”) and “Facts to policy/politics” (meaning the “Preparation/use of information content of facts for policy advice”). In a cyclical way, the impact of data as evidence also covers the feed-back from policy-making to data generation, what Radermacher calls “Policy/politics to data” (including the development of “Framework conditions and governance for the generation of data”), “Data to facts” (in particular the “Generation of data …, evaluation of data sources” and generally “Research-based evaluations of microdata”), and “Policy/politics to facts” (especially the “Policy-relevant design of quantifiable variables”, and “Questions of knowledge creation governance”) ([30] for all quotations in this paragraph).
Used to provide the factual basis for evidence-informed policy development (“Data to policy/politics”), statistical data serve multiple purposes. They help measure and compare performance; they inform the monitoring of progress; they enable comparability; they support evaluation and assessment of policies; they increase insight into complex concepts; and they constitute independent sources of information that open government decision-making to scrutiny and make them subject to accountability. Statistical data also take over different functions. They support strategic planning in multilevel political structures; define common goals for progress and development; enhance multi-dimensional performance assessment; increase transparency of decision-making processes and policy instruments; inspire innovation; and serve various other instrumental, conceptual, tactical, symbolic and political purposes.
Effects of data
As a consequence of using data to support policy-making (“Data to policy/politics”), measuring itself turned into an essential “way of doing politics” [47], justifying and rationalising political decisions and making progress visible and comparable. Data have, hence, become an essential part of governing with and through knowledge to bring “objective consistency” [48] of social phenomena into politics. As such, they are important means of identifying demand for political action and of holding decision-makers accountable for their policy choices. In this way, the use of data as evidence increases the transparency of policy choices and becomes an important scrutiny tool in EIPM. As such it can be regarded to functionally amend the ‘four powers’ of political systems (executive, legislative, judiciary, and media), contributing a ‘factual fifth power’ quality to 21
In this “Data to policy/politics” understanding, data have fundamental effects on knowledge and governance within the policy process. The use of data as evidence (“Data to facts”) alters the systemic logics of policy-making and opens the policy process to wider participation and network governance around evidence communities. Manifesting ‘post-metrological trends’, data, especially in the form of (composite) indicators, provide new forms of relevant and accepted evidence for policy-making. As such, they turn into advocacy and policy tools that perform new qualitative functions and become tools of scrutiny. This metamorphosis of data into evidence (“Data to facts”) into policy tools (“Data to policy/politics”) fundamentally affects the genuine work of actors involved in policy-making and data providers alike (“Policy/politics to data” and “Policy/politics to facts”).
Challenges for data as evidence
As discussed above, the use of data (“Data to policy/politics”) as evidence (“Data to facts” and “Facts to policy/politics”) in EIPM affects knowledge (“Policy/politics to facts”) and governance (in general and in the sense of Radermacher’s “Policy/politics to data”). Consequently, quantification can become inherently political. When it does so, it can define multiple relationships of power given that data are instruments that impact on governance by influencing the policy process as alternative modes of governance and political steering. They also configure political relations and create political priorities by influencing assessment and judgements. In this way, they impact in a particularly strong way when they “exercise regulatory functions …: by transforming local or particular models of governance and normative frameworks into apparently global technical standards and benchmarks (good governance, the rule of law, sustainability), indicators, rankings, and ratings can become technologies of policy transfer and diffusion” [45]. As a result of the systemic comparison and competition provoked by such rating, ranking and benchmarking exercises, “several states have actively sought to improve their ranking by adopting targeted reforms that are calculated to raise their overall score [49], while public discussion of a negative ranking could also result in strategic policy responses in order to be seen as “doing something” about the judgement” [45]. In the global governance debate about measuring as part of Sustainable Development Goal 16, this strong (“Data to policy/politics”) impact of indicator processes and in particular the “adoption of a Western-centred idea of ‘good governance’ that ignores how such views may be ‘transplanted’ in countries that remain target of development aid and assistance has been widely criticised” [50].
While the power of data is highly contextual, they shape policy-making through soft modes of governance, such as bench-marking processes; peer review and monitoring exercises; expert exchange; performance-based management, self-evaluation, audit cultures; and forecasting or horizon scanning. They influence formal and informal regulatory practices through normative frames and paradigms, and technical standards. Their “shared ontologies, rationalities, models and technical standards of governing often develop momentum as an independent force of collective ordering” [51]. Moreover, by ‘activism through numbers’, they function as naming-blaming-shaming tools in order to promote policy change [42].
From the “Data to policy/politics” point of view in EIPM also arise new challenges for the impact and role of data in politics as the use of data as evidence impacts on systemic preconditions and alters established ‘truisms’ about the structure and (alleged) linearity of the policy process [18]. This perspective acknowledges that, in terms of injection of evidence, EIPM results in the greater permeability of the different stages of the policy process given that feeding-in evidence becomes an overarching concern of policy-making at all stages of the policy process. Instead of few access points at the beginning of the policy cycle, policy-making is to be understood as a continuous framing of policy options and narratives for which evidence is the basis. Such options and narratives are further developed throughout EIPM and evidence is therefore used at every stage of the policy process [18] rendering it potentially even more complex. Consequently, in order to reduce analytical complexity, public policy-making should be deconstructed into its component parts – actors, institutions (rules and norms), networks, belief systems (core beliefs, ideas, paradigms), policy conditions and events [52] – rather than perceived as a linear cycle of consecutive stages. Knowledge (“Facts to policy/politics”) and data (“Data to facts” and “Data to policy/politics”) production and use hence need to accompany policy-making throughout the entire policy process and a solid trust relationship between knowledge producers and users forms a precondition for the uptake of evidence.
Politicisation of data and data use
The use of statistical data in EIPM poses many challenges to non-statisticians. As an instrument of public policy-making, data underlie the general principles of legitimacy, transparency, and accountability (“Policy/politics to data”). As part of the public policy process, their use is yet also affected by the logics of interest-formation, preference-building and political negotiation, because given “the multiple interests, perspectives, and problem frames mobilized by policy actors, the linkages between evidence and policy are deeply mediated by diverse evolving contexts, interpretations, negotiations, and organizational practices” [8].
Consequently, data, including their production and use (“Policy/politics to data”, “Data to facts”, “Policy/politics to facts”), can become instrumentalised and politicised in its different dimensions (“Data to policy/politics”, “Data to facts” and “Facts to policy/ politics”) within the policy process and within politics in general. A case in point is the 2019 (seemingly incorrect) official national statistics on Tanzanian GDP growth depicting the country as far from risking economic contraction, that led to institutional disputes with the International Monetary Fund (IMF) over a related report on the unreliability of official national data, the publication of which was stopped by Tanzania’s President Magufuli [53, 54, 55, 56]. The politicisation of statistics in the Tanzanian case leaned towards political ‘weaponisation’ of statistical data since the 2018 amendments [57] to the 2015 Statistics Act [58] that criminalised the collection and dissemination of statistical information “which is intended to invalidate, distort or discredit official statistics” [59, see also 56] (see also new Section 24A(2) of [57]) with a fine of ten million shillings, at least three years in jail or both (see amendment of Section 37, subsection (4), [57]). Already the 2015 Statistics Act had criminalised the publication or communication of official statistics that may ‘distort facts’ with the punishments reconfirmed by the 2018 amendments (see Section 37(5), [58]). Based on the 2015 Statistics Act, Zitto Kabew, an opposition party member of the Tanzanian Parliament, had been arrested “for violating the law for remarks he made about Tanzania’s economic growth” [60]. The World Bank called the 2018 amendments of the Statistics Act “deeply concerning” [61] for their impacts on the free flow of information and on civil society organisations’ potential to scrutinise official national statistics and government policies based on such data. The World Bank warned “that the amendments, if implemented, could have serious impacts on the generation and use of official and non-official statistics, which are a vital foundation for the country’s development” [61]. Furthermore, the World Bank assessed the amendments to be “out of line with international standards such as the UN Fundamental Principles of Official Statistics and the African Charter on Statistics” [61] and requested “Tanzanian authorities to ‘protect openness and transparency’ in the use of official data” [59]. Finally, the World Bank maintained that it was “critical for Tanzania, like any country, to utilize statistics laws to ensure that official statistics are of high quality and are trusted, and also protect openness and transparency in their use, to further public dialogue for the benefit of the citizens” [59]. The IMF urged the Tanzanian government “to ensure an open and independent statistics agency” [56]. After fierce national and international reaction [60], the 2018 amendment of the 2015 Statistics Act was partially revoked in 2019. The 2019 law “removes a threat of prison for civil society groups that publish independent statistical information” ([60]; new Section 26, [62]) and offers the right to statistical data collection and dissemination to everyone, even if it still entrusts the National Bureau of Statistics with “the right to challenge the misuse or interpretation of statistical information disseminated by any other person if such statistical information contains fundamental errors of does not abide with the principles specified under Section 26(3)” ([62], see Section 27), the latter refer to professional considerations, scientific principles and professional ethics on methods and procedures.
Other prominent cases of such politicisation of data use in policy-making are the repeated mis-interpretation and mis-representation of statistical information and factual evidence on the COVID-19 pandemic by some heads of states and government, or the case of Andreas Georgiou, the head of ELSTAT, the Greek national statistical office, which was assessed by the European Statistical System Committee (ESSC) as a “serious interference with the professional independence of the statistical institute in one specific country, namely Greece, revealed by the end of 2009” [63]. Georgiou and two ELSTAT senior officials were “accused of wrongdoing [to have] massaged the figures” [64] and for not having sought approval of revised public finance data by the ELSTAT board prior to publication in 2009. These accusations ignored the fact that the “European Statistics Code of Practice unequivocally states that the Head of the National Statistical Institute has ‘sole responsibility for deciding on statistical methods, standards and procedures, and on the content and timing of statistical releases”’ [64]. The “National Accounts figures published by ELSTAT in 2010 and afterwards were approved and published by Eurostat following standard procedures of scrutiny and verification” [65] of their compliance with EU legislation [63] and Eurostat explicitly refuted “allegations that the deficit of 2009 was over-estimated or that any pressure was put on ELSTAT to falsify data” [66]. Nevertheless, Georgiou was prosecuted and convicted to a “two-year suspended sentence for ‘breach of duty’ during his stint at the head of the statistics agency, Elstat – a travesty that goes beyond the unfair treatment of one statistician to questions about the progress of Greece’s economic recovery and the sustainability of the eurozone” [67]. The European Statistical Governance Advisory Board (ESGAB), in 2019, repeated its concerns “that the proceedings have the potential to adversely affect the public perception of the credibility and objectivity of Greek official statistics” [65]. In 2020, the persecution of Georgiou is still ongoing [68]. Such cases of deep politicisation of statistical data, of their use and of data producers underline the essential role of data in 21
Transparency and diversity of data
Polarisation, partisanship, instrumentalisation and politicisation of data evidence also challenge the way data is collected (“Policy/politics to data” and “Politics/politics to facts”) and used (“Data to policy/ politics” and “Data to facts”) given that both the collection and use of data become subject to public discourse, scrutiny and contestation that target the narratives and power structures that emerge from and result in their production and use. The use of data consequently inspires and requires epistemic community engagement of independent data providers and of informed data users. It also pushes for participatory structures and co-creation (for instance in the case of surveys, citizen science approaches or the use of indigenous knowledge) in (data) Communities of Practice that perceive evidence as an opportunity structure for the (normative) framing of evidence arenas and policy content. These developments have immediate impacts on political institutions and their capacity to select, evaluate and process data, which, in turn, is influenced by challenges to develop common understandings of complex social phenomena and to oversimplify complexity in politics. In this way, statistical data become part of the knowledge and evidence production (“Data to facts”) in policy-making that supports discourse and deliberation in evidence-informed processes (“Data to policy/politics”). As such, institutional checks and balances (“Policy/politics to data”) are required to guarantee neutrality of data and open access to data to increase transparency about who defines what at which stage of the evidence-informed policy process. Such institutional checks and balances demand responsiveness of data generation and the policy context that statistical data is to inform which can result in a more direct interlinkage of central knowledge and data producers, brokers, and users.
From such transparency demand also stem further instrumental challenges of data as policy tools that are related the links of political interests to processes of measurement. Questions in this context centre around a certain rhetoric-reality gap in measurement (e.g. can complex phenomena such as human well-being really be measured?) and a means-ends dilemma (e.g. is fiscal sustainability measured to demonstrate fiscal space for social policy intervention or is it measured to prove fiscal sustainability in international monitoring processes?). Difficulties of common understandings of complex phenomena; the life-cycle of policy paradigms (such as potentially human well-being) and the resulting data ecosystems that derive from the statistical approximation to such complex realities are to be scrutinised thoroughly.
Increased complexity is also becoming more of a factor against which both EIPM and the contestation of data evidence develop. As both policy problems (such as climate change, fracking, cyber security) and required policy solutions are complex and multidimensional, data evidence informing policy development also requires a broader, if not holistic, and multidimensional approach to cover the different facets of the policy challenges at hand. This increase in complexity oftentimes goes together with an increase in contestation of expertise and experts leading to the paradoxical situation that “expert advice is being sought with growing urgency across a proliferating array of policy and public questions. At the same time, and often on the same issues, the legitimacy of evidence and expertise has rarely been so fiercely contested” [69]. As a result, parallel to the demand for data, the contestation of factual evidence and expertise in policy-making increases to an alarming extent for the scientific and policy-making communities. An increased focus on (re-)building trust in politics and science (communication) is therefore required.
Resulting from the proliferation of evidence and knowledge sources for EIPM, also the diversity of data increases. Official statistical data are increasingly complemented by more disaggregate, micro-level, local experience-based measures to support the development of more targeted policy interventions. At the expense of comparability, the latter are more targeted to trigger policy action and steer reforms on the ground. Increasing the post-metrological trend in EIPM, such new forms of evidence and knowledge have entered in competition with statistical data and include evidence not based on statistical definitions. Examples for such data are experimental studies; observational qualitative and quantitative studies to identify causal mechanisms; functional models; normative deduction from principle norms; citizens and practice-informed knowledge or indigenous knowledge and experience [70, 71, 72, 73, 74, 75]. A divide over methodological flaws accompanies this multiplication of data evidence sources. As their non-standardised quality impacts on the credibility of data evidence of this type, levels of uncertainty around their validity potentially also influence the perceived authority of data evidence in general and might, in the worst case, lead to a rejection of data evidence in EIPM.
Objectivity and contestation of data
Another challenge for data as evidence in EIPM is their alleged objectivity. As outlined in the previous subchapter, although “numbers are associated with neutrality, science and expertise, concealing their political character” [76], statistics can define normalcy (and hence deviancy) and codify behaviour [50]. Measuring is therefore not just counting, but defining and creating ‘objective’ norms [45]. The decision on what to measure is thus framing reality. Through active codification and categorisation, (social) phenomena can become reality. In the same vein, deciding not to measure social phenomena creates reality by exclusion and prioritisation. In this perspective, statistics can hence become inherently political.
The transformation of EIPM “from paradox to principles” [69] of policy-making and the “paradox of scientific authority” [77] are accompanied by the above described rise in politicisation of data (and evidence in general) that increases the contestation of expertise in politics [78, 79]. An erosion of trust in data and other forms of factual evidence is what seems to follow in many political systems [80]. Mis- and disinformation, “deceptive misinterpretations” [81], post-truth and post-fact phenomena heavily impact on EIPM across the world and partially result in the failure of data evidence to impact in policy-making. This recent ‘emotional turn’ in the understanding of political decision-making [82] requires stronger attention to perception-based and emotional motivations in data use in policy-making. Biases13 and bounded rationality might be strongly influencing the uptake of data as evidence in EIPM, which in turn might favour alternative data sources or different forms of knowledge to inform policy-making. Such contestation of data as evidence can be supported by (dysfunctional) political communication and mis-information strategies that can currently be witnessed in some countries [69, 83, 84, 85]. A thorough evaluation of data use in EIPM, such as for example the ones performed by the Multilateral Organisation Performance Assessment Network (MOPAN),14 can help to promote and underline the benefits of data use in EIPM as compared to “ideological or faith-based policy-making” [13].
Conclusion
The use of data as evidence in EIPM materialises essential features of ‘governing by knowing’ and statistical data have become central evidence sources and instruments of collective political action in 21
From these developments, some conclusions on the production and use of data in EIPM can be drawn. First, statistics are part of knowledge production process within societies (“Facts to data”, “Data to facts” and “Policy/politics to data”) and as such inspire open discourse and deliberation in evidence-informed processes. Data, their production, methods, and use as well as uncertainties surrounding them therefore need to be open to productive discussion and public scrutiny to avoid politicisation and political weaponisation of statistics. Both within the political process and the process of data production, the underlying normative character of statistical data and the implications of their use for the creation of realities need to be made transparent. Narratives and power structures in which data root and that emerge from statistical data need to be openly acknowledged and discussed to better understand the consequences of measuring or not certain phenomena for their perceived (in-)existence.
Second, the institutional, collective, and personal capacities to select, evaluate and process data are of fundamental relevance to EIPM. Evidence and data literacy are essential to create common understandings of measured reality and the value of the data at hand.
Third, participation and co-creation can offer responses to contestation of data as evidence. The data production process can serve as an opportunity for inclusion, joint framing, and interlinkage of epistemic and evidence Communities of Practice. Co-creation at the design and data collection stage can increase trust in the transparency and validity of the data, potentially strengthening its position in the overall evidence ecosystem in EIPM. Official statistical data providers acquire a particularly important role in such participatory approaches to guarantee soundness of methodology and data quality; to secure the credibility of its use as data evidence; and to ensure the independence of both public and private actors in data co-creation.
Finally, there is an essential need for transparency of data and for guaranteeing neutrality of access to it. What follows is the requirement of professional, institutional, and political independence of official data production given that official statistical data can be assessed as a particularly robust form of evidence. It must hence be made explicit who defines what at which stage in official data generation. The independence of official statistical offices and a responsive, transparent feedback cycle between data creation and the policy process can not only enhance the trust in data as evidence, it can also improve their contribution to 21
Footnotes
Evidence-informed policy-making is understood to be relevant for the entire policy process (problem definition, agenda setting, consideration of policy options, legitimisation/decision-making, implementation, evaluation, maintenance/succession/termination) [2, see also 3], impacting to different extents on the practice of public officials and politicians.
Solesbury links this “upsurge of interest in evidence-based policy and practice [to] the utilitarian turn in research funding policy towards economic and social priorities and the provision of research results that are both useful and useable” [1, see also 13]. At this meta-level, the functional intersection and perceived demand-supply relation of EIPM research and politics can be acknowledged as a driver of activities in both academia and political practice [
].
See
See
See
See
See
See
See
For the purpose of reducing linguistic complexity, the terms ‘data’, ‘statistics’ and ‘statistical data’ are used as proxies for the universe of metrological instruments that count as statistical evidence in policy-making; they include measurement tools such as statistics, indicators, indices, composite indicators, scoreboards, etc.
Such as framing effects, representativeness heuristic, availability heuristic/processing fluency, prospect theory, cognitive dissonance, need for coherence, status quo biases/sunk costs fallacy/optimism biases, or groupthink.
See