Guidelines on indicator methodology: A mission impossible?

2.1 Case A. Social Indicators

Social Indicators’ history dates back to the early 1960s. Under this name, a development took shape that was concerned with the side effects and impacts of research projects (such as NASA’s space programme) in particular and with the social side effects or feedbacks of economic policies in general [5, 6].

“Frustrated by the lack of sufficient data to detect such effects and the absence of a systematic conceptual framework and methodology for analysis, some in the project attempted to develop a system of social indicators – statistics, statistical series, and other forms of evidence – with which to detect and anticipate social change and to evaluate specific programs and determine their impact. The results of this part of the project were published in a volume [7] called Social Indicators.” [8]

The aim of the social indicators was and is to be able to draw a more complete picture of the progress of a society, of social welfare, etc. than the National Accounts and especially GDP are able to do. The discussion about the limits of GDP is therefore as old as GDP itself [9, 10]. Many approaches to improve and above all to complete the coverage of GDP have been successful, but many have also failed. As a consequence, a turnaround in the methodological approach is being made; instead of a single aggregate, a dashboard of indicators is proposed.

“GDP should be dethroned. In its place, each nation should select a “dashboard” – a limited set of metrics that would help steer it toward the future its citizens desired. In addition to GDP itself, as a measure for market activity (and no more) the dashboard would include metrics for health, sustainability and any other values that the people of a nation aspired to, as well as for inequality, insecurity and other harms that they sought to diminish.” [11]

Note the nuances in the objective of these two quotes, which are separated by some 70 years of development. The ambition in the initial phase (statistics, time series and other forms of evidence) has been joined (especially in the past two decades) by forms of indicators that claim to condense the desired quantitative statements as much as possible. Dashboards, composites indices and rankings derived from them are now part of the standard repertoire of indicator presentation.

To use a multivariate approach with indicators of different dimensions instead of a univariate economic accounting approach, is a major step from an economic point of view. What this does not reflect, however, are the associated issues of condensing and aggregating a very large number of statistical variables into a manageable indicator dashboard. It is all too easy to be satisfied with a large set of indicators, with hundreds of indicators instead of a concise dashboard, as the example of the 231 UN Sustainable Development Indicators more than clearly shows. This does not really take us any closer to a solution with relevance for policy advice. Such extensive collections of indicators still cannot compete with the highly condensed (and in some cases much more up-to-date) economic aggregates. The urgent question arises as to which path and which methods lead out of this dilemma. How can we succeed in reducing and condensing the enormous variety of facets and variables of reality so that they are suitable as a tailor-made information package for upcoming decisions, but without crossing the line to value judgements and (improper) monetary valuations?

2.2 Case B. Governing-by-the-Numbers

Indicators, indicator sets, composite indices etc. are used in different stages and for different purposes within the life cycle of politics, as the twenty-year-old chart already illustrates (Fig. 1).

This interaction has also left its mark on the indicator; new possibilities for political governance by means of indicators have emerged. For example, the long-term UN strategy until 2030 was not only defined with Sustainable Development Goals, but also (whenever possible) with quantitative targets, the achievement of which is monitored with the help of Sustainable Development Indicators [14]. One could call this form of decision-making ‘augmented’ on the basis of evidence [15].

The European Union has even gone one step further in linking indicators, thresholds and political consequences in the Stability and Growth Pact, which refers specifically to Government Finance Statistics Indicators.1 With the aim of keeping the leeway for political negotiations regarding public debt and deficits as small as possible, fixed threshold were set, the exceeding of which was linked to sanctions. To a certain extent, decisions should not only be augmented in this way, but automated. With this intended depoliticisation of the annual monitoring of the solidity of the public finances of the EU member states (in the euro area) and the accompanying guarantee of stability, however, a very great responsibility has been transferred to statistics, which must provide reliable and comparable indicators to an extraordinarily high degree; indicators with authority, so to speak. Such high claims were undermined in 2009 at the latest, when it turned out that the corresponding Greek results did not correspond to the facts, but had been manipulated [16].

OPEN IN VIEWER

Close relationships between indicators and decision making have considerable power and political impact. Unfortunately, however, they are not free of side effects and risks, as with any effective medicine. The famous ‘Goodhart’s Law’ applies: “When a measure becomes a target, it ceases to be a good measure.”3 Increasing scientization of politics on the one hand and politicization of science on the other pose considerable dangers for both sides “and open debate is needed about what scientific evidence – which is often expressed and mediated by means of numbers – can realistically do in and for politics.” [17].

High policy relevance of statistical indicators must not mean that they become victims of political interference. For this reason, it is necessary to put in place appropriate safeguards (if necessary also through appropriate legal provisions) as part of statistical governance. The more ‘authority’ indicators are given, the stricter the mechanisms must be to ensure compliance with valid statistical standards. Of course, it depends very much on the respective situation which degree of security and verification is considered appropriate, whether certification is sufficient or whether a complete audit with the possibility of inspection and control is considered necessary. In any case, it can be stated that governing-by-the-numbers must be balanced by adequate statistical governance or more briefly: Data4Policy in turn requires Policy4Data [4].

In contrast to the development of social indicators over many years and decades, the provision of adequate quantitative information regarding the Corona pandemic had to be extremely fast in 2020. In the following, reference is made to two papers in which, shortly after the start of the pandemic in the spring and summer of 2020, US-American [18] and Australian [19] experts summarise their recommendations regarding the main statistical issues. For the eminently important political decisions, evidence was (and is) needed, which, however, could not be provided or at least not to the level of quality, form and timeliness of well-rehearsed and routine data producers and corresponding processes. First of all, it is about quantitative answers to “the most basic questions:

How many people are infected by the virus?

Comparable to the development of a vaccine, it was therefore necessary to look for new ways and solutions in the shortest possible time. An initial and very rapid consultation conducted by the Societal Experts Network of the US Science Foundation resulted in initial priorities for the types of data needed and corresponding quality criteria:

“The seven data types are: the number of confirmed cases, hospitalizations, emergency department visits, reported confirmed COVID-19 deaths, excess deaths, fraction of viral tests that are positive, and representative prevalence surveys (including both viral and antibody tests). The five criteria are: representativeness; bias; uncertainty, and measurement and sampling error; time; and space. The importance of any of these five criteria depends on the nature of the decision being made, and each data type has different strengths and weaknesses.” [18]

These expert’s reports – apart from their relevance for policy in a global crisis situation and the speed with which they were submitted – are interesting for our study of indicators. The Australian paper, which was prepared by statisticians, goes into detail about the information needs and the monitoring systems in the different phases of the pandemic and deals very concretely with the processes of data collection. The term ‘indicator’ is not actively used here; the authors are concerned with information, data, figures and there use for policy purposes. Although the US-American paper refers to indicators in that the seven types of data are to be used as such to assess the course of the pandemic, there is no reflection on what the use of indicators means in a broader policy (and communication with citizens) environment.

One may therefore wonder why the concept of indicator (sets), which has been widely developed and used elsewhere, has not been applied in this subject matter area. Is it owing to the fact that there is no such thing as an indicator methodology as an overarching and neutral element of statistical theory? Do different communities in statistics, such as social statisticians, macroeconomic statisticians or sustainable development statisticians, perhaps also use different forms and definitions of what they each call indicators? What is the difference, if any, between data, statistics and indicators? Has the buzzword ’data’ perhaps taken over and is draping itself over all differences as a veil of fuzziness and lack of distinction?

Apparently, it was particularly urgent for the authors of the two papers used as references to address the data set and the production of decision-relevant statistical evidence, whereas the interaction with users (in the sense of the wider public) is thematically absent. If a linguistic distinction is made between a technosphere and a sociosphere, the contributions focus on the technosphere. Yet the Corona pandemic must have made it very clear to us how important it is that the information generated in the world of experts (and following their scientific codes) is translated into the world of politics and into the language of ordinary citizens. The assumption that this is a kind of by-product of the actual statistical work falls considerably short and fails to recognise the possibilities and also the risks of communication and interrelation between statistical producers and users, between the indicator and the indicandum [20]. In order for this communication to take place without serious misunderstandings, a special effort is needed and the creation of a novel statistical product, namely the indicators. In the development of statistics related to the COVID pandemic that has taken place since then and is still ongoing, the great importance of smart, clear and user-friendly tools and methods of communication can be observed. An integration of such proven communication with its methodological details would be desirable to be explicitly included in recommendations on pandemic statistics.

2.4 Case D. Data Report Germany [21]

“The Data Report is a social report published by the Federal Agency for Civic Education (bpb) together with the Federal Statistical Office (destatis), the Social Science Research Center Berlin (WZB) and the Socio-Economic Panel (SOEP) of the German Institute for Economic Research (DIW).”4 The fact that this report is only available in German does not play a significant role in our case study. Rather, it is about the fact that a cooperation between different partners that has existed since 1983 has manifested itself in it and has proven to be successful. The very neutral title ’Data Report’ is somewhat surprising, as it deals with a publication of social indicators in the broadest sense. The report combines data from official statistics with those from social research and creates a comprehensive picture of the living conditions and attitudes of people in Germany. The individual chapters highlight social issues: population trends, for example, or current changes in education and on the labour market. In 2018, the data report focusses on the living conditions for families and especially for children and young people.

In the history of this publication, it is worth mentioning that it was not until 2008 that there was a continuous content structure according to themes, within which the various sources were presented in a linked manner. Prior to this edition, it was not considered possible or opportune to link the results of official statistics and those of social research. A separate presentation was chosen, which of course made reading more difficult.

One important conclusion can be drawn at this point: Indicator sets and reports that are intended to cover a broader spectrum of social, ecological and economic issues (and their combination) often require a combination of different sources and cannot be reduced to official statistics alone. This naturally raises a number of serious questions, especially about the quality of information and its comparability. Indicator partnerships presuppose that it is possible to agree on a common denominator in which the different participants can find themselves with their possibilities, limits, standards as well as their ambitions.

Another aspect of interest for us is the cooperation of a public institution for political education. “The data report is thus not only a veritable social report on the state of the republic, but also an important instrument of political education. It provides users with material they need to form their own well-founded judgement.”5 This claim is an essential part of the function and, for this reason, of the entire conception of the report.

2.5 Lessons learnt

As far as the term ‘indicator’ is concerned, there is no clear definition applied in practice and above all no demarcation from statistical results in general. One can however observe that statistical facts are called ‘indicators’, if they concern more abstract objects and concepts, describing reality, such as poverty, quality of life or the like. Implicitly, therefore, one can assume a meaning and intended use, different from the very detailed tables of basic statistics. However, there is no theoretical concept in this respect that would fit the very heterogeneous applications; perhaps it was so far not considered necessary.

4.1 The data value chain

In principle, a decomposition of the data value chain is possible. It provides a wide range of professional areas and statistical skills.7“In research on information literacy, often a step-by-step model such as the DIKW model (Data, Information, Knowledge, Wisdom) is chosen. It shows schematically how raw data are processed in the human brain into information, knowledge and wisdom through an increasing degree of organisation. After cleansing and linking individual data elements to form meaningful information, we search for patterns, apply analysis principles and structure the information, for example, by classi?cation or categorization. Before this, there is the measurement as a process of mapping real world objects into data.” [28] Accordingly, those who in the broadest sense apply statistical methods developed by academic statistics, has diversified enormously; users of statistical methods can be found at all stages of the data value chain8 (Fig. 2).

OPEN IN VIEWER

The construction of indicators can involve the process of synthesising indicators through aggregative – compensative and noncompensatory approaches [29]. These methods apply a synthesis of units (cases, subjects, etc.) with reference to one or more macro-indicator aiming at aggregating the individuals’ value observed at a microlevel.

Operating a synthesis in a complex structure such as a system of indicators may represent a very difficult task, urging solutions which are able to apply different instruments in combination, including graphical devices. While numerical approaches aim at a reduction of many values in just one (or, at least, very few), the graphical perspective concerns the deduction of many values in a visual display, such as dashboards [30].9

In official statistics, indicators have been used traditionally in a large variety of types and applications. The most popular (and politically important) indicators are those, which belong to macro-economic statistics, such as GDP (growth), Current Account Balance, Public Deficit (over GDP ratio), Consumer Price Index, (Labour) Productivity etc. These types of indicators belong to the category of ‘models’ in the terminology of Desrosières, mentioned above. Typically, these macro-economic indicators do not emerge directly from statistical surveys, but rather from a further processing and refinement in National Accounting (within the framing of the SNA10 or the SEEA11) or complex models, such as the one for the Consumer Price Index. Furthermore, these accounting procedures solve the aggregation problem by adopting common denominators to which all statistical elements (transactions, assets, energy flows, etc.) are converted. Traditionally (in the SNA) this is done in monetary units, following a valuation by the market. In the case of environmental phenomena, all of these are converted into energy units or mass units where possible, before producing higher aggregates. This gives such indicators coherence and consistency that generates a high aggregation and summary of information.

In areas not covered by the macro-economic accounts, such as social statistics, transport, agriculture, education, etc., indicators and sets of indicators have been typically distilled more directly from the results of surveys or official statistics based on administrative data sources.

Macro-economic indicators often link to scientific theory derived from economics. Non macro-economic indicators partly cannot rely on a comparable homogeneous and comprehensive theoretical framework, which raises issues with their interpretations and possible statistical aggregation.

In general, the aim of an indicator is to allow decision makers (and decision processes) to get to the heart of an issue, with supporting evidence. This assumes an association between an indicator and the issue it is designed to represent. No doubt, this description should apply to every statistical variable. Any variable, any statistical object is an operationalisation of a qualitative concept (a system) for the purpose of measurability, i.e. for the generation of data. In the case of indicators, we are considering a specific context that gives them their characteristic function and form. We have described earlier in the paper which elements belong to this context. However, the proximity to and relevance for decisions and very often the complexity of the aggregates at stake (e.g. poverty, welfare, etc.) should be particularly emphasised.

The crucial question regarding the quality of indicators is whether they are capable of providing the answer to the question posed, with the required accuracy, timeliness, consistency, etc. In other words, it is about designing, producing and communicating indicators so that they are ’fit for purpose’ and provide adequate information quality. But what are the criteria for assessing this quality? With which methodological framework and which approach can we approach this extremely complex question?

One can approach this issue from two sides, a statistical-scientific and a statistical-practical angle:

A structured scientific framework for assessing information quality is provided in Kenett and Shmueli [34, 35, 36]. In that concept, information quality is defined as the utility in applying a method of analysis to a data set, conditioned on the analysis goal. Eight dimensions are defined, which determine information quality. These are: data resolution, data structure, data integration, temporal relevance, chronology of data and goal, generalisability, operationalisation and communication. Conditional to the given goals, all these dimensions need to be properly addressed, in order to achieve adequate information quality.

For both frameworks mentioned, however, the question arises as to whether they are also suitable to do justice to the very specific characteristics, tasks and risks of indicators, or whether they need further development and appropriate supplements.

Indicators are now used in large numbers. For example, the United Nations Sustainable Development Goals (SDG) initiative aims at reaching 17 goals that are defined in a list of 169 SDG Targets. Progress towards these Targets is tracked by 231 unique indicators.

The fact that indicators are everywhere and used by different communities for their purposes, results in a great diversity in the terminology used. In addition, the integration of new data sources (big data) and new disciplines (e.g. data science) has added even more confusion. Differences between ‘data’, ‘statistics’, ‘facts’, ‘information’, ‘indicators’ or ‘metrics’ are largely blurred. All these terms are treated as synonyms and all come together under the umbrella term and buzzword ‘data’.

The situation in statistics, however, does not necessarily look much better. While indicators have found their place as a valuable tool for the dissemination and communication of statistics,13 not much attention is devoted yet to them in terms of harmonised methodology, applicable equally as a standard to all statistical domains.

A couple of important agreements concerning indicator methodology do already exist, however. Just to mention a few, the statistical domain of social indicators can look back on a tradition of methodological discussion, conceptual harmonisation and practical implementation since the 1980s;14 composite indicators have been developed and methodologically harmonised;15 Eurostat has published a series of guidelines, which could serve as a first step into the direction of a standardised methodology.16

Nevertheless, it can be it can be argued with all caution that there is discrepancy between the large dynamics in the quantity and variety of indicators and the comparatively low degree of methodological harmonisation, which implies significant risks:

Inflation and proliferation of indicators might lead to misperception in such a way that it does not require the professional treatment of statisticians, but that anyone without proper training of techniques and methods can handle indicators.

4.3 Indicators with authority

Indicators with a high and more or less direct political impact, such as the price index, the number of unemployed or the GDP, did not arise overnight. Their strength stems from the fact that they have evolved over years and decades, in a constant interplay between new user demands, conceptual advancements based on scientific work and new data sources, and statistical methods. Therefore, it cannot be expected that, for complex issues (such as Sustainable Development), an indicator or system of indicators can be born with a forced act, even if the political will so demands. It is more reasonable to assume that such a system evolves and learns, that all participants (science, statistics, society) contribute to this evolution, and that this long-term and multifaceted process requires order, governance and management.

Indicators with a high political impact, such as the indices of indebtedness of the public sectors in Europe, which are closely linked to the Excessive Deficit Procedure (EDP, introduced with the common currency by the Maastricht Treaty)21 belong to an extraordinary category of indicators that deserves a special degree of attention. As history has shown, the fact that these indicators are used for surveillance purposes and that they could result in serious consequences contains the risk of manipulation and interference by politics. The special degree of authority that has been assigned to these indicators therefore requires special governance that must go beyond the usual dose of institutional competencies of statistics.

Indicators with a high potential to influence the markets, such as the inflation rate or the quarterly GDP, deserve special attention with regard to equal access and in particular pre-release access.22 Like many statistical offices, Eurostat has discussed this subject with representatives of the media and has recorded the results in a ‘Protocol on impartial access to Eurostat data for users’.23

Rankings and (composite) indicators with a weighting not based on observations or available market values might be difficult to be used as part official statistics. Even if the chosen aggregation methods meet the scientific standard, the extent of the assumptions, estimates or other model parameters they contain may have too great an influence on the result. Nonetheless, it is very useful, through close cooperation between applied research and official statistics, to offer users a range of highly condensed information that combines the best available statistics with the best practices for their compression.

Political goals are manifest value judgments of a society. Without being in charge to assess their validity, statisticians can make use of them as a reference for statistical indicators. This opens up new possibilities for a simpler, more condensed and clearer representation of multidimensional indicator sets both numerically and, above all, graphically.

Nonetheless, there are also critical aspects of a procedure that is geared to target values. Sakiko Fukuda-Parr has addressed the intentional and unintended effects of a target-based policy under the UN Millennium Development Goals (MDGs) [48], and similarly questions the Sustainable Development Strategy (SDG) [49].

An interplay between the statistical process of the indicator design and the political process of defining objectives can succeed if this is done gradually. In an idealised sequence, the following steps should follow one after the other: First, goals should be distilled out and politically agreed, which subdivide a broad topic and qualitatively formulate the priority questions. In the second step, statisticians are required to provide a set of quantitative answers in the form of indicators for qualitative questions. If it has been agreed that these indicators are suitable, then the third step on the part of the politicians is to fix quantitative targets to be set up at an agreed time in the future. In the fourth step, a monitoring can then begin, in which the indicators and their development are assessed against the targets in order to be able to determine whether the target is being approached as scheduled.

If one does not stick to this stepwise sequence but makes, for example, the third before the second step (setting of targets before the indicators are designed and agreed upon), one may generate targets for which there are no indicators, targets may remain partly qualitative (they are only subdivisions of the priority goal) or the number of indicators may be inflated unnecessarily, resulting in redundancies and inconsistencies.