Guidelines on indicator methodology: A mission impossible?

Abstract

Indicators are part of daily life, modern politics and public debate in times of crises and their solutions. The question arises to what extent indicators embody a special form of statistical information, whether a separate indicator-chapter in statistical methodology is needed and what the content of this chapter should be. This paper considers different perspectives of indicators produced by official statistics agencies. Starting from the definition of indicators, it then discusses statistical quality, impact and interaction between producers and users of indicators. To introduce and illustrate the points made in the paper a choice of cases is used in order to derive the essential elements of a methodological framing. The overall objective is to enhance the added value of official statistics indicators as they are communicated and expected to lead to trustworthy evidence for policy making.

Keywords

Official statistics indicators information quality methodology

1. Background, purpose and scope of the paper

‘Indicator’ comes from the late Latin “indicator” which means “who or what indicates”. Indicators represent economic, social or environmental phenomena quantified with a view to an intended use, for example in a decision-making process, and within the framework of an interpretation model. “In this perspective, an indicator is not simple crude statistical information but represents a measure organically connected to a conceptual model aimed at describing different aspects of reality” [1]. This makes it clear that the main characteristic of indicators is their purpose and the context of interpretation (i.e. the ‘indicandum’). It is therefore not only the ‘how’ of their construction [2], but also the ‘for what’ that marks them [3]. For example, a demographic statistics variable is “divorce rate”. In order to consider it as an “indicator of country’s well-being”, one has to define it in terms of the conceptual framework of wellbeing.

To what degree does such an abstract definition help us to approach the topic of ‘indicators’? Only to a limited extent. What is genuine behind using the term ‘indicator’ depends on the statistical field of application in which one does so. Very different histories, quality ethics and cultures coexist in official statistics, so it is difficult to deduce a general theory about indicators from a conceptual definition.

Nevertheless, it is considered necessary to dedicate a specific chapter in statistical theory to the field of indicators. Unlike other chapters in the statistical textbook, this chapter however cannot be exhausted by describing the statistical methods that are relevant to the field of indicators. Rather, the goal must be to do justice to the very special role and task that the indicators have in the value chain for the production of knowledge.

Indicators embody a relay, a communicative bridge between the producers of statistics on the one hand and the users of statistical information on the other. This means that these two sides (and the interaction between them) exert influence as well as aspects that need to be taken into account in an indicator theory. Indicators are, in a sense, the incarnation of what has been labeled governing-by-the numbers [4].

The paper will approach this two-sided question by first presenting a few concrete, exemplary practical examples from which generalised conclusions will be drawn.

2. History of indicators in (official) statistics

Before we will approach a definition and determination of indicators, a few cases may be used as examples and their history may serve as illustration.

2.1 Case A. Social Indicators

Social Indicators’ history dates back to the early 1960s. Under this name, a development took shape that was concerned with the side effects and impacts of research projects (such as NASA’s space programme) in particular and with the social side effects or feedbacks of economic policies in general [5, 6].

“Frustrated by the lack of sufficient data to detect such effects and the absence of a systematic conceptual framework and methodology for analysis, some in the project attempted to develop a system of social indicators – statistics, statistical series, and other forms of evidence – with which to detect and anticipate social change and to evaluate specific programs and determine their impact. The results of this part of the project were published in a volume [7] called Social Indicators.” [8]

The aim of the social indicators was and is to be able to draw a more complete picture of the progress of a society, of social welfare, etc. than the National Accounts and especially GDP are able to do. The discussion about the limits of GDP is therefore as old as GDP itself [9, 10]. Many approaches to improve and above all to complete the coverage of GDP have been successful, but many have also failed. As a consequence, a turnaround in the methodological approach is being made; instead of a single aggregate, a dashboard of indicators is proposed.

“GDP should be dethroned. In its place, each nation should select a “dashboard” – a limited set of metrics that would help steer it toward the future its citizens desired. In addition to GDP itself, as a measure for market activity (and no more) the dashboard would include metrics for health, sustainability and any other values that the people of a nation aspired to, as well as for inequality, insecurity and other harms that they sought to diminish.” [11]

Note the nuances in the objective of these two quotes, which are separated by some 70 years of development. The ambition in the initial phase (statistics, time series and other forms of evidence) has been joined (especially in the past two decades) by forms of indicators that claim to condense the desired quantitative statements as much as possible. Dashboards, composites indices and rankings derived from them are now part of the standard repertoire of indicator presentation.

To use a multivariate approach with indicators of different dimensions instead of a univariate economic accounting approach, is a major step from an economic point of view. What this does not reflect, however, are the associated issues of condensing and aggregating a very large number of statistical variables into a manageable indicator dashboard. It is all too easy to be satisfied with a large set of indicators, with hundreds of indicators instead of a concise dashboard, as the example of the 231 UN Sustainable Development Indicators more than clearly shows. This does not really take us any closer to a solution with relevance for policy advice. Such extensive collections of indicators still cannot compete with the highly condensed (and in some cases much more up-to-date) economic aggregates. The urgent question arises as to which path and which methods lead out of this dilemma. How can we succeed in reducing and condensing the enormous variety of facets and variables of reality so that they are suitable as a tailor-made information package for upcoming decisions, but without crossing the line to value judgements and (improper) monetary valuations?

2.2 Case B. Governing-by-the-Numbers

Indicators, indicator sets, composite indices etc. are used in different stages and for different purposes within the life cycle of politics, as the twenty-year-old chart already illustrates (Fig. 1).

This interaction has also left its mark on the indicator; new possibilities for political governance by means of indicators have emerged. For example, the long-term UN strategy until 2030 was not only defined with Sustainable Development Goals, but also (whenever possible) with quantitative targets, the achievement of which is monitored with the help of Sustainable Development Indicators [14]. One could call this form of decision-making ‘augmented’ on the basis of evidence [15].

The European Union has even gone one step further in linking indicators, thresholds and political consequences in the Stability and Growth Pact, which refers specifically to Government Finance Statistics Indicators.1 With the aim of keeping the leeway for political negotiations regarding public debt and deficits as small as possible, fixed threshold were set, the exceeding of which was linked to sanctions. To a certain extent, decisions should not only be augmented in this way, but automated. With this intended depoliticisation of the annual monitoring of the solidity of the public finances of the EU member states (in the euro area) and the accompanying guarantee of stability, however, a very great responsibility has been transferred to statistics, which must provide reliable and comparable indicators to an extraordinarily high degree; indicators with authority, so to speak. Such high claims were undermined in 2009 at the latest, when it turned out that the corresponding Greek results did not correspond to the facts, but had been manipulated [16].

Figure 1.

Social indicators used for policy making. ${}^{2}$

Close relationships between indicators and decision making have considerable power and political impact. Unfortunately, however, they are not free of side effects and risks, as with any effective medicine. The famous ‘Goodhart’s Law’ applies: “When a measure becomes a target, it ceases to be a good measure.”3 Increasing scientization of politics on the one hand and politicization of science on the other pose considerable dangers for both sides “and open debate is needed about what scientific evidence – which is often expressed and mediated by means of numbers – can realistically do in and for politics.” [17].

High policy relevance of statistical indicators must not mean that they become victims of political interference. For this reason, it is necessary to put in place appropriate safeguards (if necessary also through appropriate legal provisions) as part of statistical governance. The more ‘authority’ indicators are given, the stricter the mechanisms must be to ensure compliance with valid statistical standards. Of course, it depends very much on the respective situation which degree of security and verification is considered appropriate, whether certification is sufficient or whether a complete audit with the possibility of inspection and control is considered necessary. In any case, it can be stated that governing-by-the-numbers must be balanced by adequate statistical governance or more briefly: Data4Policy in turn requires Policy4Data [4].

2.3 Case C. COVID-19

In contrast to the development of social indicators over many years and decades, the provision of adequate quantitative information regarding the Corona pandemic had to be extremely fast in 2020. In the following, reference is made to two papers in which, shortly after the start of the pandemic in the spring and summer of 2020, US-American [18] and Australian [19] experts summarise their recommendations regarding the main statistical issues. For the eminently important political decisions, evidence was (and is) needed, which, however, could not be provided or at least not to the level of quality, form and timeliness of well-rehearsed and routine data producers and corresponding processes. First of all, it is about quantitative answers to “the most basic questions:

How many people are infected by the virus?

How many people who are infected are not yet exhibiting symptoms?

How many people who are infected do not exhibit symptoms (asymptomatic)?

How many who are infected and exhibit symptoms but have not been tested?

How many people have recovered from the virus (only known for those who have been tested and were positive)?” [19]

Comparable to the development of a vaccine, it was therefore necessary to look for new ways and solutions in the shortest possible time. An initial and very rapid consultation conducted by the Societal Experts Network of the US Science Foundation resulted in initial priorities for the types of data needed and corresponding quality criteria:

“The seven data types are: the number of confirmed cases, hospitalizations, emergency department visits, reported confirmed COVID-19 deaths, excess deaths, fraction of viral tests that are positive, and representative prevalence surveys (including both viral and antibody tests). The five criteria are: representativeness; bias; uncertainty, and measurement and sampling error; time; and space. The importance of any of these five criteria depends on the nature of the decision being made, and each data type has different strengths and weaknesses.” [18]

These expert’s reports – apart from their relevance for policy in a global crisis situation and the speed with which they were submitted – are interesting for our study of indicators. The Australian paper, which was prepared by statisticians, goes into detail about the information needs and the monitoring systems in the different phases of the pandemic and deals very concretely with the processes of data collection. The term ‘indicator’ is not actively used here; the authors are concerned with information, data, figures and there use for policy purposes. Although the US-American paper refers to indicators in that the seven types of data are to be used as such to assess the course of the pandemic, there is no reflection on what the use of indicators means in a broader policy (and communication with citizens) environment.

One may therefore wonder why the concept of indicator (sets), which has been widely developed and used elsewhere, has not been applied in this subject matter area. Is it owing to the fact that there is no such thing as an indicator methodology as an overarching and neutral element of statistical theory? Do different communities in statistics, such as social statisticians, macroeconomic statisticians or sustainable development statisticians, perhaps also use different forms and definitions of what they each call indicators? What is the difference, if any, between data, statistics and indicators? Has the buzzword ’data’ perhaps taken over and is draping itself over all differences as a veil of fuzziness and lack of distinction?

Apparently, it was particularly urgent for the authors of the two papers used as references to address the data set and the production of decision-relevant statistical evidence, whereas the interaction with users (in the sense of the wider public) is thematically absent. If a linguistic distinction is made between a technosphere and a sociosphere, the contributions focus on the technosphere. Yet the Corona pandemic must have made it very clear to us how important it is that the information generated in the world of experts (and following their scientific codes) is translated into the world of politics and into the language of ordinary citizens. The assumption that this is a kind of by-product of the actual statistical work falls considerably short and fails to recognise the possibilities and also the risks of communication and interrelation between statistical producers and users, between the indicator and the indicandum [20]. In order for this communication to take place without serious misunderstandings, a special effort is needed and the creation of a novel statistical product, namely the indicators. In the development of statistics related to the COVID pandemic that has taken place since then and is still ongoing, the great importance of smart, clear and user-friendly tools and methods of communication can be observed. An integration of such proven communication with its methodological details would be desirable to be explicitly included in recommendations on pandemic statistics.

2.4 Case D. Data Report Germany [21]

“The Data Report is a social report published by the Federal Agency for Civic Education (bpb) together with the Federal Statistical Office (destatis), the Social Science Research Center Berlin (WZB) and the Socio-Economic Panel (SOEP) of the German Institute for Economic Research (DIW).”4 The fact that this report is only available in German does not play a significant role in our case study. Rather, it is about the fact that a cooperation between different partners that has existed since 1983 has manifested itself in it and has proven to be successful. The very neutral title ’Data Report’ is somewhat surprising, as it deals with a publication of social indicators in the broadest sense. The report combines data from official statistics with those from social research and creates a comprehensive picture of the living conditions and attitudes of people in Germany. The individual chapters highlight social issues: population trends, for example, or current changes in education and on the labour market. In 2018, the data report focusses on the living conditions for families and especially for children and young people.

In the history of this publication, it is worth mentioning that it was not until 2008 that there was a continuous content structure according to themes, within which the various sources were presented in a linked manner. Prior to this edition, it was not considered possible or opportune to link the results of official statistics and those of social research. A separate presentation was chosen, which of course made reading more difficult.

One important conclusion can be drawn at this point: Indicator sets and reports that are intended to cover a broader spectrum of social, ecological and economic issues (and their combination) often require a combination of different sources and cannot be reduced to official statistics alone. This naturally raises a number of serious questions, especially about the quality of information and its comparability. Indicator partnerships presuppose that it is possible to agree on a common denominator in which the different participants can find themselves with their possibilities, limits, standards as well as their ambitions.

Another aspect of interest for us is the cooperation of a public institution for political education. “The data report is thus not only a veritable social report on the state of the republic, but also an important instrument of political education. It provides users with material they need to form their own well-founded judgement.”5 This claim is an essential part of the function and, for this reason, of the entire conception of the report.

2.5 Lessons learnt

As far as the term ‘indicator’ is concerned, there is no clear definition applied in practice and above all no demarcation from statistical results in general. One can however observe that statistical facts are called ‘indicators’, if they concern more abstract objects and concepts, describing reality, such as poverty, quality of life or the like. Implicitly, therefore, one can assume a meaning and intended use, different from the very detailed tables of basic statistics. However, there is no theoretical concept in this respect that would fit the very heterogeneous applications; perhaps it was so far not considered necessary.

3. Towards an indicator concept – A first approximation

These very few and selected cases have taught us that, on the one hand, there is no simple, clear-cut concept for indicators, but that, on the other hand, there are elements that can be utilised to delineate a conceptual framing. These elements are the following:

Empirical: Data sources and producers may be inside but also outside of official statistics; in these cases it is important to establish reliable partnerships and agree on a robust compromise regarding the quality profile of the indicators [22].

Theoretical: The quantification of phenomena of the social and economic sphere, as well as their interaction with the environment, presuppose that a description and delimitation of these phenomena with a theory (i.e. qualitatively). Such theoretical descriptions belong to the respective discipline (or several of them in appropriate conjunction).

Aggregation: Compression of information and filtering out a signal from the noise that is irrelevant to the question is basically the core and characteristic of all statistical work. With indicators, however, one goes very far in this direction, so that statistical-methodological requirements arise as well as questions of information quality and normative loading (in the case of weightings). For the main macro-economic indicators (such as GDP), the answer lies in the use of one unit of measurement: money; for others (such as the consumer price index), weighting schemes are used that are themselves derived from statistical surveys. It always becomes difficult when indicators are generated from different variables with different units and scales.

Political: “Choices as to defining and carrying out concrete ‘measurement’ of such indicators is, however, not only a technical question but concerns the whole political context in which they take place. The procedures for debating and systematically investigating this subject have yet to be invented.” [23] According to the maturity and stage of a policy lifecycle (problem framing, policy framing, policy implementation, monitoring and evaluation), the requirements for the indicators will be very different in terms of their quality profile.

Goals: The explicit formulation of policy goals and especially their quantification in the form of targets or thresholds are popular nowadays and are understood as belonging to the repertoire of modern politics and to the principles of good governance in the public sector. All the more is it imperative to know the risks and side effects of such evidence-based policies and to take precautions accordingly.

Communication: Indicators are first and foremost tools of communication. Communication between those who produce such evidence and those who use it for information or decision-making requires a common platform in the form of a language that both sides understand. But this communication becomes even more challenging when the general public, the citizen, the entrepreneur, the teacher, the pupil, the nurse and the doctor are also expected to understand this language. Then it is also required that these people have the necessary language skills to a certain extent in order to be able to read and evaluate statistics and to be able to recognise and assess their quality. This proves that successful and undisturbed communication of indicators requires efforts that go beyond modern, clear and simple tools of communication. Equally important are endeavours to improve statistics education and to create a statistics culture.

Governance: Indicators should be policy relevant; they must not be politically driven! In this contradictory objective, it is necessary to find the right mix of instruments from the toolbox of “politics for facts”; not an easy task. If, for example, one chooses a too demanding defensive control (as in the case of the indicators with high political authority mentioned in case B.), one runs the risk of making the statistics inflexible and hostile to innovation, which in turn harms relevance. If, on the other hand, flexibility (and less rigid governance) is chosen, there could be risks for the comparability and accuracy of the indicators. It therefore crucially depends on the individual case and the political situation for which dosage one should decide.

One might assume that these elements are of such great importance that they would have already been studied and answered with great intensity, both in scientific theory and in statistical practice. This is not the case, however. On the contrary, one can get the impression that these considerations, which cannot be confined exclusively within the statistical discipline, but also extend to questions of sociology, epistemology, etc., are rather a marginal phenomenon in statistical methodology. Exceptions are the works of Theodore Porter [24, 25] and especially of Alain Desrosières [26] or, more recently, the work primarily in sociology circles, e.g. [27].

For our elaboration we are making use of the following differentiation introduced by Alain Desrosières: “between several aspects of statistics, 1) that of quantification properly speaking, i.e. the making of numbers,62) that of the uses of numbers as variables, and finally, 3) the prospective inscription of variables in more complex constructions, models.” [23] Desrosières stresses that such a distinction is necessary for a number of reasons and is helpful for a more detailed examination. These include the fact that different cognitive schemes are applied and that in a statistical value chain based on the division of labour, different types of experts are at work who regard the inputs from the respective upstream production stage as given and thus as ‘real’.

In this tripartite structure of Desrosières, indicators belong to the third level; they are refined statistical products, i.e. variables based on more complex constructions or models. Indicators are a specific product type of statistics at the interface between the statistical production processes and the processes of interpretation and use of statistics. This makes it necessary to look at this interface in both directions linking the questions of design, production and communication of indicators on the one hand with those that make up their use on the other.

Our overarching goal is the production of indicators fit for purpose and on generating information quality. On the one hand we consider aspects and possibilities of statistical methodology used to condense (multifactorial) indicators, and on the other hand we consider the interaction between producers and users of indicators, which are of a more sociological nature. In this paper we will focus on the latter aspect.

4. Indicators and the reduction of complexity

4.1 The data value chain

In principle, a decomposition of the data value chain is possible. It provides a wide range of professional areas and statistical skills.7“In research on information literacy, often a step-by-step model such as the DIKW model (Data, Information, Knowledge, Wisdom) is chosen. It shows schematically how raw data are processed in the human brain into information, knowledge and wisdom through an increasing degree of organisation. After cleansing and linking individual data elements to form meaningful information, we search for patterns, apply analysis principles and structure the information, for example, by classi?cation or categorization. Before this, there is the measurement as a process of mapping real world objects into data.” [28] Accordingly, those who in the broadest sense apply statistical methods developed by academic statistics, has diversified enormously; users of statistical methods can be found at all stages of the data value chain8 (Fig. 2).

Figure 2.

The data value chain.

The construction of indicators can involve the process of synthesising indicators through aggregative – compensative and noncompensatory approaches [29]. These methods apply a synthesis of units (cases, subjects, etc.) with reference to one or more macro-indicator aiming at aggregating the individuals’ value observed at a microlevel.

Operating a synthesis in a complex structure such as a system of indicators may represent a very difficult task, urging solutions which are able to apply different instruments in combination, including graphical devices. While numerical approaches aim at a reduction of many values in just one (or, at least, very few), the graphical perspective concerns the deduction of many values in a visual display, such as dashboards [30].9

In official statistics, indicators have been used traditionally in a large variety of types and applications. The most popular (and politically important) indicators are those, which belong to macro-economic statistics, such as GDP (growth), Current Account Balance, Public Deficit (over GDP ratio), Consumer Price Index, (Labour) Productivity etc. These types of indicators belong to the category of ‘models’ in the terminology of Desrosières, mentioned above. Typically, these macro-economic indicators do not emerge directly from statistical surveys, but rather from a further processing and refinement in National Accounting (within the framing of the SNA10 or the SEEA11) or complex models, such as the one for the Consumer Price Index. Furthermore, these accounting procedures solve the aggregation problem by adopting common denominators to which all statistical elements (transactions, assets, energy flows, etc.) are converted. Traditionally (in the SNA) this is done in monetary units, following a valuation by the market. In the case of environmental phenomena, all of these are converted into energy units or mass units where possible, before producing higher aggregates. This gives such indicators coherence and consistency that generates a high aggregation and summary of information.

In areas not covered by the macro-economic accounts, such as social statistics, transport, agriculture, education, etc., indicators and sets of indicators have been typically distilled more directly from the results of surveys or official statistics based on administrative data sources.

Macro-economic indicators often link to scientific theory derived from economics. Non macro-economic indicators partly cannot rely on a comparable homogeneous and comprehensive theoretical framework, which raises issues with their interpretations and possible statistical aggregation.

In general, the aim of an indicator is to allow decision makers (and decision processes) to get to the heart of an issue, with supporting evidence. This assumes an association between an indicator and the issue it is designed to represent. No doubt, this description should apply to every statistical variable. Any variable, any statistical object is an operationalisation of a qualitative concept (a system) for the purpose of measurability, i.e. for the generation of data. In the case of indicators, we are considering a specific context that gives them their characteristic function and form. We have described earlier in the paper which elements belong to this context. However, the proximity to and relevance for decisions and very often the complexity of the aggregates at stake (e.g. poverty, welfare, etc.) should be particularly emphasised.

The crucial question regarding the quality of indicators is whether they are capable of providing the answer to the question posed, with the required accuracy, timeliness, consistency, etc. In other words, it is about designing, producing and communicating indicators so that they are ’fit for purpose’ and provide adequate information quality. But what are the criteria for assessing this quality? With which methodological framework and which approach can we approach this extremely complex question?

One can approach this issue from two sides, a statistical-scientific and a statistical-practical angle:

A structured scientific framework for assessing information quality is provided in Kenett and Shmueli [34, 35, 36]. In that concept, information quality is defined as the utility in applying a method of analysis to a data set, conditioned on the analysis goal. Eight dimensions are defined, which determine information quality. These are: data resolution, data structure, data integration, temporal relevance, chronology of data and goal, generalisability, operationalisation and communication. Conditional to the given goals, all these dimensions need to be properly addressed, in order to achieve adequate information quality.

In the practice of official statistics, a quality management system was developed in response to experiences, crises and their countermeasures. It sees statistical information as products whose quality is ensured by comprehensive management. Quality assurance by the European Union, consisting of a Code of Practice, Peer Reviews and a Quality Framework, is a mature and proven example of this.12

For both frameworks mentioned, however, the question arises as to whether they are also suitable to do justice to the very specific characteristics, tasks and risks of indicators, or whether they need further development and appropriate supplements.

4.2 Indicator uses and purposes

Indicators are now used in large numbers. For example, the United Nations Sustainable Development Goals (SDG) initiative aims at reaching 17 goals that are defined in a list of 169 SDG Targets. Progress towards these Targets is tracked by 231 unique indicators.

The fact that indicators are everywhere and used by different communities for their purposes, results in a great diversity in the terminology used. In addition, the integration of new data sources (big data) and new disciplines (e.g. data science) has added even more confusion. Differences between ‘data’, ‘statistics’, ‘facts’, ‘information’, ‘indicators’ or ‘metrics’ are largely blurred. All these terms are treated as synonyms and all come together under the umbrella term and buzzword ‘data’.

The situation in statistics, however, does not necessarily look much better. While indicators have found their place as a valuable tool for the dissemination and communication of statistics,13 not much attention is devoted yet to them in terms of harmonised methodology, applicable equally as a standard to all statistical domains.

A couple of important agreements concerning indicator methodology do already exist, however. Just to mention a few, the statistical domain of social indicators can look back on a tradition of methodological discussion, conceptual harmonisation and practical implementation since the 1980s;14 composite indicators have been developed and methodologically harmonised;15 Eurostat has published a series of guidelines, which could serve as a first step into the direction of a standardised methodology.16

Nevertheless, it can be it can be argued with all caution that there is discrepancy between the large dynamics in the quantity and variety of indicators and the comparatively low degree of methodological harmonisation, which implies significant risks:

Inflation and proliferation of indicators might lead to misperception in such a way that it does not require the professional treatment of statisticians, but that anyone without proper training of techniques and methods can handle indicators.

Anecdotal, spontaneous design of indicators might be perceived as appropriate, if it is all too easy to ask for the provision of indicators in the preparation of decisions or when political negotiations are difficult. In such situations, a statistician should first be consulted before any further requests are manifested. The costs (production costs, response burden) and benefits of additional indicators should be considered as part of the impact assessments for regulations and political projects.

Fragmentation of methods and terminology and unnecessary variation create confusion on both sides, producers and users of indicators.

Unrealistic expectations from the political and management level result in frustration, misinterpretation and conflicts. Adaptations in practice ‘solve’ those problems at the expense of quality.

Political influence might dominate the design, production and communication of indicators, so that statistical principles and the quality of indicators are threatened.

4.2.1 Indicators vis-à-vis statistics

The essential lesson from all the above criticisms is that indicators have a very specific role to play in the statistical information portfolio and that, consequently, a specific methodology has to be applied. The methodology used here needs, however, to be tailored for this particular type of information and it’s utilisation. Ensuring that indicators are not politically driven, albeit policy-relevant, is ultimately the crucial issue on which the decision between trust and mistrust depends. In this respect, questions of communication and governance must be anchored from the outset in the methodology.

Of course, quantitative indicators are statistics. Not all statistics are however indicators. Indicators are a special type of statistical information in two respects:

First, it is a particularly high level of compression, focus and synthesis with regard to the relevant statistical message (they are ‘models’ positioned at the third level in Alain Desrosières’ typology, mentioned earlier).

Second, it is the close connection to a scope, a purpose, and the associated user community.

In contrast to detailed basic statistics, indicators have a different quality profile with a particular emphasis on the criterion of relevance, while other properties and strengths of basic statistics (e.g. level of detail, accuracy) are less critical. They condense and communicate the informational content contained in statistics in such a way that it can be understood and used by the respective target group. Indicators are pointers to particular feature. In contrast to multi-purpose basic statistics, indicators are designed (or at least should be designed) in such a way that they serve one (partly very specific) purpose.

4.2.2 Indicators and the policy life cycle

It is now crucial to consider the two peculiarities and dimensions of indicators as interrelated and interdependent. A special focus, aggregation and consolidation of various pieces of information into one indicator is only possible if it is known for what purpose it is planned to use that indicator. Even an already highly condensed aggregate of national accounts, such as GDP, requires further specifications in order to be customised as an indicator to suit a specific application: shall inflation be filtered out, shall an index for growth be calculated, is it a net amount without capital depreciation, or is it a seasonally adjusted quarterly GDP that is expected? Once again, the context and the question ultimately determine which indicator is derived from the GDP with the help of further refinements and additional methodological filters.

Depending on the phase in the life cycle of a policy area targeted by the indicator, different characteristics are expected and needed. In a phase of awareness raising for a new phenomenon, it might, for example, be sufficient to work with indicators of lower granularity or a wider confidence interval while for the monitoring of target achievement, very high precision and resolution is a necessary feature of provided indicators.

Indicators can reveal, suggest, distort and conceal.17 Which of these characteristics is actually built in the design of an indicator or an indicator set will determine their impact on the debate in the political sphere, the ‘bazaar’.18

4.2.3 Co-Design, Co-Construction, Literacy

It follows that it is neither meaningful nor possible to develop indicators external to and isolated from the system that is going to use this information, in a separate area of statistics or solely through collaboration of statistics with science (or possibly through an experienced statistical expert, who combines a multitude of competence fields). Rather, a co-construction is required, where different stakeholders contribute and participate. From a systemic point of view, the observing and observed systems cannot be isolated from each other.

Taking note of this fact leads consequently to different questions, expectations and approaches concerning also an envisaged indicator methodology. As in the example of Sustainable Development, it is not a question of carrying out a ‘measurement of sustainability’ as a mere academic and analytical undertaking, which is then fed into the political discussion. Rather, the point is to develop and constantly improve in close connection between the political and statistical worlds the measurement and use of these measurements.

The particular challenge is, here, to ensure the quality of indicators when they are produced under these circumstances. It must therefore be the goal to ensure both this quality and convince the users of this quality in communication. Firstly, it is important that the indicators provided by the statistics are indeed of the best quality and that this quality is also credibly certified. With such a seal of quality in the sense of a confirmation that high quality standards have been met, a big step has been taken towards credibility and trust. However, this alone will not achieve the goal if the users of the indicators cannot tell the difference between reliable quality and ’fake news’. An initiative for digital literacy, for improving statistical skills and for creating a culture that is open to statistical evidence is therefore an imperative part of the official statistics programme.

4.2.4 Multi-disciplinary cooperation, iteration and adaptation

Methodological development and research in the field of indicators is spread over different disciplines, each of them focusing on their respective field of expertise. Furthermore, specific methodological approaches and concepts have been developed side-by-side in single statistical communities for various sets of indicators. While economics statistics, given that their observation units are often expressed in monetary terms resulting from market transactions, prefer an accounting approach to obtain highly aggregated economic indicators, such as inflation, growth, or productivity, this is difficult to do similarly in social and environmental statistics, where many essential variables are not monetised on actual markets.19 This is one of the reasons why statistical methods of synthesis and aggregation come into play.20 Mutual fertilisation through methodical cooperation across the disciplines should therefore be supported – not least because of new opportunities arising from new data sources, the interest in successful methodologies in neighbouring areas (e.g. geographical sciences and statistics) or new disciplines (e.g. data science) as a source of innovation and efficiency.

In parallel with the statistically oriented fields, a growing group of researchers has been dealing with the sociological aspects of co-construction of indicators (general or specific ones) in recent years, pointing out the particular challenges of indicators in terms of communication or concerning required governance provision. Which function, for instance, should indicators have in the political process? Should they be used to facilitate an exchange of views (‘opening-up’) or should they be used to shorten or close a discussion (‘closing-down’)? What is the relationship between targets, goals and indicators? In which sequence should they be defined and by whom? How much evidence is available concerning the (correct) effectiveness of indicators? Is this evidence systematically fed into the learning cycles of statistics? To what extent and at what stage of the process is a consultation or even stakeholder participation (i.e. civil society) taking place? Such questions would require significantly intensified research work, and this in collaboration between statisticians, communication experts, and sociologists or other social scientists.

4.3 Indicators with authority

Indicators with a high and more or less direct political impact, such as the price index, the number of unemployed or the GDP, did not arise overnight. Their strength stems from the fact that they have evolved over years and decades, in a constant interplay between new user demands, conceptual advancements based on scientific work and new data sources, and statistical methods. Therefore, it cannot be expected that, for complex issues (such as Sustainable Development), an indicator or system of indicators can be born with a forced act, even if the political will so demands. It is more reasonable to assume that such a system evolves and learns, that all participants (science, statistics, society) contribute to this evolution, and that this long-term and multifaceted process requires order, governance and management.

Indicators with a high political impact, such as the indices of indebtedness of the public sectors in Europe, which are closely linked to the Excessive Deficit Procedure (EDP, introduced with the common currency by the Maastricht Treaty)21 belong to an extraordinary category of indicators that deserves a special degree of attention. As history has shown, the fact that these indicators are used for surveillance purposes and that they could result in serious consequences contains the risk of manipulation and interference by politics. The special degree of authority that has been assigned to these indicators therefore requires special governance that must go beyond the usual dose of institutional competencies of statistics.

Indicators with a high potential to influence the markets, such as the inflation rate or the quarterly GDP, deserve special attention with regard to equal access and in particular pre-release access.22 Like many statistical offices, Eurostat has discussed this subject with representatives of the media and has recorded the results in a ‘Protocol on impartial access to Eurostat data for users’.23

Rankings and (composite) indicators with a weighting not based on observations or available market values might be difficult to be used as part official statistics. Even if the chosen aggregation methods meet the scientific standard, the extent of the assumptions, estimates or other model parameters they contain may have too great an influence on the result. Nonetheless, it is very useful, through close cooperation between applied research and official statistics, to offer users a range of highly condensed information that combines the best available statistics with the best practices for their compression.

Political goals are manifest value judgments of a society. Without being in charge to assess their validity, statisticians can make use of them as a reference for statistical indicators. This opens up new possibilities for a simpler, more condensed and clearer representation of multidimensional indicator sets both numerically and, above all, graphically.

Nonetheless, there are also critical aspects of a procedure that is geared to target values. Sakiko Fukuda-Parr has addressed the intentional and unintended effects of a target-based policy under the UN Millennium Development Goals (MDGs) [48], and similarly questions the Sustainable Development Strategy (SDG) [49].

An interplay between the statistical process of the indicator design and the political process of defining objectives can succeed if this is done gradually. In an idealised sequence, the following steps should follow one after the other: First, goals should be distilled out and politically agreed, which subdivide a broad topic and qualitatively formulate the priority questions. In the second step, statisticians are required to provide a set of quantitative answers in the form of indicators for qualitative questions. If it has been agreed that these indicators are suitable, then the third step on the part of the politicians is to fix quantitative targets to be set up at an agreed time in the future. In the fourth step, a monitoring can then begin, in which the indicators and their development are assessed against the targets in order to be able to determine whether the target is being approached as scheduled.

If one does not stick to this stepwise sequence but makes, for example, the third before the second step (setting of targets before the indicators are designed and agreed upon), one may generate targets for which there are no indicators, targets may remain partly qualitative (they are only subdivisions of the priority goal) or the number of indicators may be inflated unnecessarily, resulting in redundancies and inconsistencies.

5. Lessons learned for indicators

The preceding observations should have clarified that it is necessary and useful to work on a comprehensive indicator methodology for official statistics. The momentum generated by the Sustainable Development Indicators (SDI) certainly helps to come considerably closer to this ambition. There are important lessons to be learned for the future of indicators in and by official statistics.

Overall, it seems important to reduce the fragmentation of the indicator landscape by consistently working to bring together and bridge developments in different communities. For official statistics, it should be possible to come to a (at least in core) standardised terminology and methodology covering the entire production process from data to the composite indicator. Similarly, in such a comprehensive methodology manual, it is necessary to do justice to the specific nature of the indicators empirical by including the components of communication, participation, market research, and governance.

Especially in the area of Sustainable Development (SD), it should be possible to use the indicator system for the entire planning and production process (mainstreaming SDI) be it through user-friendly links to basic statistics and accounts or through adaptation of communication and media work by prioritising indicators in long-term planning.

Partnerships with science and neighbouring institutes can help to join forces and increase the effectiveness of the evidence provided.

In terms of governance, it is important to grant statistics the professional degree of freedom it needs to provide quality information. Determining the details of an indicator system, as with the enormously large number of targets in the case of the SDGs, reduces these degrees of freedom sensitively and can ultimately lead to unsatisfactory results.

Specific advice for improving indicators is summarised in the following non-exhaustive and non-exclusive list:

Promote epistemological knowledge and sensitivity amongst scientists, practitioners and users; promote media and information literacy to counter disinformation

Improve evidence about the impact of indicators through research and impact assessments

Promote a focus on quality and be critical with unrealistic expectations

Make use of state-of-the art aggregation methods, minimising normative (value-based weighting) ingredients; be transparent, when normative components (judgements, weights, values) are mixed with observation-based indicators

Connect indicators with other components of the information system of official statistics (accounts, basic statistics); make use of the complementarity of statistical tools (in particular accounts)

Promote indicator-related research and innovation; promote indicator-related methodological competencies with appropriate training

Establish platforms and channels that facilitate communication amongst scientists, indicator producers and users of indicators; find regular opportunities for indicator-related exchange of views in (international) conferences

Ensure a functioning dialogue between goal setting and indicator design processes; assign power to indicators for the redefinition of targets and goals

Promote an evolutionary perspective, where continuous improvement of the indicator design is part of the strategy

Develop tools for empowering users and journalists to tackle disinformation and foster a positive engagement with fast-evolving information technologies

Safeguard the strengths and independence of institutions and researchers working in the area of indicator development, production and knowledge management

Enhance transparency about the quality of indicators and compliance with ethical principles and good governance guidelines

In official statistics, important methodological agreements and standards are laid down in manuals, guidelines or even standards of international level, depending on the respective level of maturity. Measured against this, there is still relatively little in the way of guidance for indicators in which the various aspects elaborated in this article are prepared and summarised in such a generalised manner that they can serve as orientation aids and compasses for the designers, compilers and communicators of indicators in their daily work. As a summary recommendation at the end of this paper, it is therefore only logical to call for such guides and manuals. An already existing fundament for this are OECD/JRC’s “Handbook on Constructing Composite Indicators” [39] and Eurostat’s three-part series of indicator manuals [40, 41, 42]. An international statistical guideline based on these sources, updated, enhanced and complemented, supported by standard-setting bodies, would certainly be a major step towards improving the quality of indicators in general.

Footnotes

https://ec.europa.eu/eurostat/statistics-explained/index.php/Government_finance_statistics#Introduction.

https://www.oxfordreference.com/view/10.1093/oi/authority. 20110803095859655.

https://www.destatis.de/DE/Service/Statistik-Campus/Datenrep ort/datenreport-artikel.html.

https://www.bpb.de/nachschlagen/datenreport-2018/.

Today one would probably use the term ‘data’ here.

See for example the “The Digital Competence Framework 2.0” https://ec.europa.eu/jrc/en/digcomp/digital-competence-framework or the “European framework for ICT Professionals in all sectors” https://www.ecompetences.eu/.

For more details of the Data-to-Policy framework, see W.J. Radermacher “Governing-by-the-numbers” [].

Using graphical approach to synthesis could represent a step forward with respect to the analytical approaches, especially in its potential capacity to represent a system of indicators (or part of it) in particular ways able to support users in making statements and asking questions about the indicators and allowing a critical understanding the reality they try to represent. This represents a big challenge especially if we think that new sources of data (big data) would support the construction of indicators []. Visual complexity may represent an alternative/complementary approach to synthesis. It is located exactly at the crossroad of different competences, i.e. image, word, number and art. This field of studies refers directly to the concept of complexity and recalls the main characteristics of a system.

System of National Accounts [].

System of Environmental Economic Accounting [].

For an overview see Radermacher “Official Statistics 4.0” [].

See for example the large selection of ‘EU Policy Indicators’ provided by Eurostat (http://ec.europa.eu/eurostat/).

See the Handbook on Social Indicators [].

See Handbook on Constructing Composite Indicators – Methodology and User Guide [].

See the Guidelines for indicators, parts 1 to 3 [40, 41, ].

See [43, ].

In this sense indicators are of high relevance for human rights and (global) governance [].

Even if this might change, following the analysis of Cook [].

For an overview over the statistical methods and approaches see Filomena Maggino’s Complexity in Society: From Indicators to their Synthesis [].

For more details see http://ec.europa.eu/eurostat/web/govern ment-finance-statistics/excessive-deficit-procedure.

See Eurostat’s policies here (http://ec.europa.eu/eurostat/about/ policies/dissemination).

See http://ec.europa.eu/eurostat/documents/4187653/5798057/ IMPARTIAL_ACCESS_2014_JAN-EN.PDF.

References

Maggino

. Indicators Definition. 2018. In: Wiley StatsRef: Statistics Reference Online [Internet]. John Wiley and Sons. Available from: doi: 10.1002/9781118445112.stat08095.

Maggino

. Indicators Construction. 2018. In: Wiley StatsRef: Statistics Reference Online [Internet]. John Wiley and Sons. Available from: doi: 10.1002/9781118445112.stat08194.

Maggino

. Systems of Indicators. 2018. In: Wiley StatsRef: Statistics Reference Online [Internet]. John Wiley and Sons. Available from: doi: 10.1002/9781118445112.stat08096.

Radermacher

. Governing-by-the-numbers – Reflections on the future of official statistics in a digital and globalised society. Statistical Journal of the IAOS. 2019; 35: 519-37.

Noll

H-H

. Social Indicators and Social Reporting – The international Experience. In: Development CCoS, editor. Symposium on Measuring Well-being and Social Indicators. Ottawa: Canadian Council on Social Development, 1996, pp. 1-24.

Noll

H-H

. Social Indicators and Quality of Life Research: Background, Achievements and Current Trends. In: Genov

, editor. Advances in Sociological Knowledge Over Half a Century. Wiesbaden: VS Verlag für Sozialwissenschaften, 2004, pp. 151-81.

Bauer

. Social indicators. Cambridge, MA: M.I.T. Press, 1966.

Social Indicators. Encyclopedia of Sociology, 2020.

Coyle

. GDP: A Brief but Affectionate History. Princeton: Princeton University Press, 2014.

10.

Fioramonti

. Gross Domestic Problem: The Politics Behind the World’s Most Powerful Number: Zed Books, 2013.

11.

Stiglitz

. GDP Is the Wrong Tool for Measuring What Matters – It’s time to replace gross domestic product with real metrics of well-being and sustainability. Scientific American. 2020; 2020 (August 1).

12.

Land

Ferriss

. Conceptual Models for the Development and Use of Social Indicators. In: Glatzer

Habich

Mayer

, editors. Sozialer Wandel und gesellschaftliche Dauerbeobachtung. Wiesbaden: VS Verlag für Sozialwissenschaften, 2002, pp. 337-52.

13.

IRESEARCHNET [Internet]2020. [cited 2020 28.12.2020]. Available from: https://www.iresearchnet.com/research-paper-examples/sociology-research-paper/social-indicators/.

14.

United Nations. Sustainable Development Goal indicators website New York: United Nations; 2018 [Available from: https://unstats.un.org/sdgs/.

15.

Umbach

. Of Numbers, Narratives and Challenges: Data as Evidence in 21st Century Policy-making. Statistical Journal of the IAOS. 2020; 36: 1043-55.

16.

European Commission. REPORT ON GREEK GOVERNMENT DEFICIT AND DEBT STATISTICS. Brussels: European Commission, 2010.

17.

Prutsch

. Science, numbers and politics in a “post-truth” world. Statistical Journal of the IAOS. 2020; 36: 1035-41.

18.

National Academies of Sciences, Engineering, and Medicine. Evaluating Data Types: A Guide for Decision Makers using Data to Understand the Extent and Spread of COVID-19. Washington, DC: The National Academies Press; 2020. Available from: https://www.nap.edu/download/25826.

19.

Fisher

Trewin

. A proposal to enhance Australia’s capability to manage epidemics – The critical importance of expert statistical input. [Recommendation to the Australian Government]. In press 2020.

20.

Davies

. How statistics lost their power – and why we should fear what comes next. The Guardian, 2017.

21.

Bundeszentrale für politische Bildung, Statistisches Bundes- amt, Wissenschaftszentrum Berlin für Sozialforschung, Sozio- oekonomischen Panel des Deutschen Instituts für Wirtschaft-sforschung. Datenreport. Bonn: Bundeszentrale für politische Bildung; 2018. Available from: https://www.destatis.de/DE/Service/Statistik-Campus/Datenreport/Downloads/datenreport-2018.pdf?__blob=publicationFile.

22.

MacFeely

Nastav

. “You say you want a [data] revolution? Aproposal to use unofficial statistics for theSDG Global Indicator Framework. Statistical Journal of the IAOS. 2019; 35.

23.

Desrosières

. A Politics of Knowledge-tools – The Case of Statistics. In: Sangolt L, editor. Between Enlightenment and Disaster. Brussels: P.I.E. Peter Lang, 2010.

24.

Porter

. Trust in numbers : the pursuit of objectivity in science and public life. Princeton, N.J.; Chichester: Princeton University Press; 1995.

25.

Porter

. The flight of the indicator. In: Rottenburg

Merry

Park

Mugler

, editors. The World of Indicators: The Making of Governmental Knowledge through Quantification (Cambridge Studies in Law and Society). Cambridge: Cambridge University Press, 2015, pp. 34-55.

26.

Desrosières

. The Politics of Large Numbers – A History of Statistical Reasoning, Cambridge Massachusetts: Harvard University Press, 1998.

27.

Rottenburg

Merry

Park

Mugler

. The World of Indicators: The Making of Governmental Knowledge through Quantification. Cambridge: Cambridge: Cambridge University Press, 2015.

28.

Schüller

. Ein Framework für Data Literacy. AStA Wirtschafts- und Sozialstatistisches Archiv. 2019; 13(3): 297-317.

29.

Ivaldi

Ciacci

. Indicators Definition: Non Compensatory Methods. Wiley StatsRef: Statistics Reference Online: John Wiley & Sons, Ltd., 2021.

30.

Maggino

. Synthesis of Indicators. 2018. In: Wiley StatsRef: Statistics Reference Online [Internet]. John Wiley and Sons. Available from: doi: 10.1002/9781118445112.stat08195.

31.

Lima

. Visual Complexity: Mapping Patterns of Information. New York: Princeton Architectural Press, 2011, 240.

32.

European Commission, International Monetary Fund, Organisation for Economic Co-operation and Development, United Nations, World Bank. System of National Accounts 2008. New York, 2009, 635.

33.

United Nations. System of Environmental-Economic Accounting 2012 – Central Framework. UNSD, editor. New York: United Nations European Union, FAO, IMF, OECD, The World Bank, 2014.

34.

Kenett

Shmueli

. On Information Quality (with discussion). Journal of the Royal Statistical Society. 2014: 3-38.

35.

Kenett

Shmueli

. From Quality to Information Quality in Official Statistics. Journal of Official Statistics. 2016; 32: 1-22.

36.

Kenett

Shmueli

. Information Quality: The Potential of Data and Analytics to Generate Knowledge: John Wiley & Sons, 2016.

37.

Radermacher

. Official Statistics 4.0 – Verified Facts for People in the 21st Century. Heidelberg: Springer Nature Switzerland AG; imprint Springer, 2020.

38.

United Nations. Handbook on Social Indicators. New York: United Nations, 1989.

39.

OECD, European Commission JRC. Handbook on Constructing Composite Indicators – Methodology and User Guide. Paris: OECD, 2008.

40.

Eurostat. PART 1 – INDICATOR TYPOLOGIES AND TERMINOLOGIES. Luxembourg: Eurostat, 2014.

41.

Eurostat. PART 2 – COMMUNICATING THROUGH INDICATORS. Luxembourg: Eurostat, 2017.

42.

Eurostat. PART 3 – RELEVANCE OF INDICATORS FOR POLICY MAKING. Luxembourg: Eurostat, 2017.

43.

Lehtonen

. Indicators: tools for informing, monitoring or controlling? In: Jordan

Turnpenny

, editors. The Tools of Policy Formulation – Actors, Capacities, Venues and Effects. Cheltenham: Edward Elgar Publishing; 2015, pp. 76-99.

44.

Ravetz

Hild

Thunus

Bollati

. Sustainability indicators – Quality and quantity. In: König

, editor. Sustainability Science. New York: Routledge, 2018, pp. 271-95.

45.

Merry

. Measuring the World – Indicators, Human Rights, and Global Governance. Current Anthropology. 2011; Volume 52, Supplement 3(April 2011).

46.

Cook

. The Pricing of Progress – Economic Indicators and the Capitalization of American Life. Harvard: Harvard University Press, 2017.

47.

Maggino

. Complexity in Society: From Indicators Construction to their Synthesis: Springer International Publishing, 2017.

48.

Fukuda-Parr

. Global Goals as a Policy Tool: Intended and Unitended Consequences. In: Fukuda-Parr

Ely Yamin

, editors. The MDGs, Capabilities and Human Rights. New York: Routledge, 2015, pp. 14-27.

49.

Fukuda-Parr

. Is goal setting a good way to define global development agendas? European Commission Workshop The Impacts And Methodology Of Indicators And Scoreboards; 22–23 MARCH, 2018; Ispra, Italy: Joint Research Center, 2018.