Techniques for analyzing and presenting official statistics indicators

1. Background, purpose and scope of the paper

This paper is about how official statistics indicators are constructed and produced, with an emphasis on their statistical properties. Indicators are a specific product of statistics at the interface between the statistical production processes and the processes of interpretation and use of statistics. This makes it necessary to look at this interface in both directions linking the questions of design, production and communication of indicators with their use. Moreover, it is necessary to clarify whether these two sides, the producer and the consumer, are independent of each other and sequential, or whether they influence each other.

Our overarching goal is the production of indicators fit for purpose and on generating information quality. On the one hand we consider aspects and possibilities of statistical methodology used to condense (multifactorial) indicators, and on the other hand we consider the interaction between producers and users of indicators, which are of a more sociological nature. The next section is about indicators and the reduction of complexity to simplify the analysis. Section 2 is focused on statistical methodology of official statistics indicators. Section 3 is on the level of aggregation of indicators, Section 4 on assigning a differentiated level of importance to components used to compute indicators. Section 5 is on the integration of indicators to get an overall perspective, Section 6 on multivariate analysis and Section 7 on graphical presentation of indicators. The paper concludes with a discussion section.

2. Statistical methodology of statistical indicator

A common, but problematic approach used in official statistics reports, is to view a list of indicators as independent, i.e. not accounting for mutual effects of indicators. This poses several challenges in analyzing, communicating and operationalizing such indicators:

Analyzing refers to how a multivariate data set is analyzed by statistical and analytic methods. Statistical analysis is available in a multitude of platforms that can handle numbers, text, images, recordings, social media networks etc. In particular, administrative data and questionnaire-based data can be combined to provide deeper insights and generate information of high quality. We discuss this in Section 5.

Communicating is about the visualization tools to present such indicators. Indicators, like all outputs of statistical analysis need to be presented, in an appropriate way, to the relevant persons, at the proper time. Reporting platforms can include traditional printed reports but also interactive dashboard that can be queried for deep dive investigations and linked graphics.

Operationalization is the consideration of such a set of indictors by decision makers and policy formulators in support of their decisions. In order to achieve results, Deming [1] posed 3 questions to managers: 1) What do you want to accomplish, 2) By what methods will you accomplish it and 3) How will you know when you accomplished it? We need indicators to answer (1) and (3).

To help achieve concrete results, the reduction of complexity in statistics means using methods to separate the important from the unimportant. Clear and concise metrics are needed to convey useful information. In economic statistics, this objective is achieved by referring to indicators based on monetary currency. This approach condenses statistics on national production into a single aggregated figure: the GDP. Economists also evaluate other phenomena in monetary terms. From this perspective, The conventional economic approach can be seen as the opposite of an approach using various indicators. In the former, the challenge of aggregating multivariate variables is circumvented by monetarisation at the micro level. This approach requires monetary valuation of market transactions and needs to be justified when producing official statistics. In markets that do not yet exist such as environment related public goods, monetary valuations need to be simulated. This, in turn, is possible and justifiable only to a limited extent within the ethical and qualitatively defined standards of official statistics (42). With this background, an OECD-based expert group led by Joseph Stiglitz called for a dashboard of indicators, rather than a single aggregated figure, to better measure societal progress: “No single metric will ever provide a good measure of the health of a country, even when the focus is limited to the functioning of the economic system. Policies need to be guided by a dashboard of indicators informing about people’s material conditions and the quality of their lives, inequalities thereof, and sustainability. This dashboard should include indicators that allow us to assess people’s conditions over the economic cycle. Arguably, policy responses to the Great Recession might have been different had such a dashboard been used.” [2]

We address these challenges by considering the multivariate structure of official statistics indicators. Developing such capabilities will provide an improved policy support platform that indicates the overall effect of local changes.

Statistical issues affecting common practice in computing and displaying indicators range from the identification of latent variables, accounting for local variability, statistical aggregation and clustering of indicators and providing a multivariate model accounting for associations between indicators. We treat these aspects in the next section. Issues raised by the common approach of weighting individual indicators to derive an overall score have been raised in [3].

3. Aggregation of indicators

Identifying the level of synthesis in aggregating indicators is not a trivial issue. The decision depends on the meaning that the final aggregation should have. We demonstrated this with an example.

Subjective wellbeing is observed by collecting individual level data on both positive and negative effects representing wellbeing. The wellbeing indicator represents by the effect balance, defined as the difference between positive effects and negative effects. Even if the difference can be obtained at a macro level, the synthesis should be performed at an individual micro level.

Selecting the indicators to be included in a wellbeing aggregation represents a fundamental step since it operationally determines the concept defining the phenomena that the aggregation is supposed to measure. This step aims at

testing (reflective model) or investigating (formative model) the level of complexity of the concept measured by the indicators in terms of dimensionality.

Dimensional analysis is applying correlation between indicators. According to the reflective model for aggregating indicators, the selected indicators are considered as caused by the same latent variable. They can be considered multiple measures that to contribute to the measured variability. Consequently, highly correlated indicators prove that the model of measurement is reflective. The statistical property allowing the aggregation of reflective indicators is internal consistency. An approach to reflective models is Factor Analysis (FA), which can be applied in order to test the hypothesized dimensional structure underlying the selected indicators. In particular, it allows indicators that fit better the latent dimensional structure to be synthesized. FA is based on the assumption that the total variance of each indicator represents a linear combination of different components (additive assumption): the common variance (due to the dimensional structure), the specific variance (due to the specificity variance of each indicator), and the error. By estimating the amount of common variance (communality), the reflective approach is tested. Often, Principal Component Analysis (PCA) is applied to test dimensional structures. This practice is however criticized as the main goal of PCA is not to test a (dimensional) model but to decompose the correlations among basic indicators by calculating linear combinations of variables. PCA describes the variation of a data set using a smaller number of dimensions than the number of the original indicators.

The latent construct in FA can turn out to be multidimensional with the variance of each indicator explained by one or more dimensions. Aggregating indicators by referring to single dimensions of the latent variable is acceptable, and the aggregated score can be easily interpreted. Creating a single score out of several dimensions (i.e., aggregating indicators referring to all dimensions of the latent variable) may be also consistent and meaningful.

In the case of formative models for aggregating indicators, the correlation informs us about the level of overlap in indicators measuring the same unobservable variable.

From the technical point of view, several instruments are available:

Correlation Analysis which is useful in selecting indicators that are not redundant and avoidinf multicollinearity (double counting) in composite indicator construction.

In some cases, the approaches can be combined (e.g., tandem analysis or factorial k-means analysis) [4].

Consequently, a high level of correlation between two indicators presents a redundancy and suggests the selection of only one of them. This decision should give preference to the indicator allowing trend analysis and wide comparisons and proving to be available for a large number of cases are preferable.

The nature of a latent variable defined in the formative context is multidimensional. Consequently, aggregation of formative indicators raises several issue. In general, when concepts are truly multidimensional, collapsing indicators to just one synthetic indicator is very questionable. The nuances and ambiguities of the data can be forced into a conceptual model where all the features affecting one-dimension indicators are considered as noise to be removed. Moreover, synthetic scores can be biased towards a small subset of basic indicators, failing to give a faithful representation of the data.

4. Defining the importance of each indicator

In case of formative models, stating the importance of indicators requires a criterion able to reproduce as accurately as possible the contribution of each indicator to the overall synthesis. With this perspective, the definition of the weighting system constitute an improvement and refinement of the model of measurement.

While equal weighting does not necessarily imply unitary weighting, the adoption of the differential weighting procedure does not necessarily correspond to the identification of different weights but rather to the selection of the most appropriate approach in order to identify the weights [4]. Assigning differential weights can be problematic, especially when the decision is not supported by theoretical reflections that provide a meaning to each indicator or consideration of its impact on the synthesis In this sense, apart from the applied approach, the defined weights represent judgment values. The methodology to be adopted should be carefully evaluated and made formally explicit. From the technical point of view, the weighting procedure consists of defining and assigning a weight to each indicator. The weights are used in the successive computation of the individual aggregate score when each weight is multiplied by the corresponding individual value of the indicator. In order to define the weighting system, some decisions need to be adopted:

The proportional size of weights (equal or differential weighting)

The approach is considered objective when the weights are determined through an analytic process, that is through

statistical methods (correlation, Principal Component Analysis, Data Envelopment Analysis, Unobserved Components Models). The adoption of statistical methods in weighting socio-economic components has to be considered carefully since, by removing any control over the weighting procedure from the analysts, it may give a false appearance of mathematical objectivity that is actually difficult to achieve in social measurement.

The approach is considered subjective when shows the possibility to involve more individuals (experts or citizens) in the process of defining weighting systems for social indicators. These approaches are defined in the perspective of giving more legitimacy to social indicators by taking into account citizens’ importance (values) and not – as usually done in the past – statistical importance.

Weights can be kept constant or can be changed according to particular considerations concerning each application. In both cases, the researcher needs to rationalize the choice. The former approach can be adopted when the aim is to analyse the evolution of an examined phenomenon. The latter is useful when the aim is to define particular priorities.

In any case, we have to consider that a set of weights is able to express in a perfect way the actual and relative contribution of each indicator do not exist.

5. Integrating indicators

The selection of the proper aggregation technique should consider: (i) if it admits or does not admit compensability among indicators; (ii) requires or does not require comparability (with reference to nature of data) among indicators; (iii) requires or does not require homogeneity in indicators’ level of measurement

An aggregating technique is compensatory when it allows low values in some indicators to be compensated by high values in other indicators. A compensatory technique can be useful in some contexts especially when the purpose of applying indicators is to stimulate behaviours aimed at improving the overall performance by investing in those ambits showing lower values. The selection of the aggregating technique is an important choice also to avoid inconsistencies between the weights previously chosen – in terms of theoretical meaning and importance – and the way these weights are actually used. In other words, in order to continue interpreting the weights as “importance coefficients”, a non-compensatory aggregating procedure has to be preferred, such as a non-compensatory multi-criteria approach, like Multi-Criteria Analysis (MCA).

The literature offers several aggregation tech-niques [4]. The linear aggregation approach (additive techniques) is the most widely used. By contrast, multiplicative techniques (following the geometrical approach) and the technique based upon multi-criteria analysis (following the non-compensatory approach) allow the difficulties caused by compensation among the indicators to be overcome.

6. Multivariate analysis of indicators

A common approach to present indicators is to use a wheel display or radar plots. Another approach is to use line plots (see for example [5]). These methods present a frozen perspective which only indirectly reflects the links between indicators. To overcome this, reports of indicators typically include upward or downward arrows reflecting sort term trends over time.

League tables are sometimes prepared on the basis of indicators. Such leagues are notorious for misrepresenting the complexities of reality and have been widely criticized in scientific work and in the press [3, 6]. They are strong in communication but poor in information.

An important advance has been the methodology of sensitivity analysis proposed by Saltelli and colleagues [7, 8]. This work helps decision makers evaluate the robustness of alternative decisions, for example in movement restrictions during a pandemic, through their impact on indicators measurements.

Bayesian networks [9, 10], provide a graphical dynamic perspective that can be used for diagnostic and predictive analysis. This extended capability supports decision makers in assessing the causes of given situations and predicting the impact of policy decisions.

We present in this section, with examples, four approaches to the statistical analysis of indicators: that can be used in data aggregation, in comparability of indicators, and in representing homogeneity of indicators. These tools are:

Control charts analysis

These approaches aim at leveraging the multivariate structure of indicators and account for variability in time to identify significant effects. They give us the means to account for variability in the value of indicators, identify units with similar behaviour, reduce the dimensionality of a large number of indicators and provide graphical synthesis of indicators. As mentioned, we focus on examples, providing only a general theoretical introduction to these methods. For more details see [10, 11].

As constantly stated in this paper, representing a complex and multidimensional world through indicators produces a complex and multidimensional system [14].

While the numerical approach concerns the “reduction” of many values in just one (or, at least, very few), the graphical perspective concerns the deduction of many values in a visual display which typically uses a bi-dimensional representation. The main issue is, how can we represent the world of measures on a mere flatland? [15].

Operating a synthesis in a complex structure may represent a very difficult task, urging solutions which are able to apply different instruments in combination, including graphical devices.

Using a graphical approach to synthesis could represent a step forward with respect to the analytical approaches, especially in its potential capacity to represent a system of indicators (or part of it) in particular ways able to support users in making statements and asking questions about the indicators and allowing a critical understanding the reality they try to represent. This represents a big challenge especially if we think that new sources of data (big data) would support the construction of indicators [16]. Visual complexity represents an alternative/complementary approach to synthesis. It is positioned at the crossroad of different competences, i.e. image, word, number and art. Visual complexity represents the intersection of two technical and cultural fields, networks and visualization [16]. While the first has some tradition in social sciences, the explosion of online social networks and the increasing availability of web information linked to each other have raised the need of having the possibility to visually represent them.

However, similarly to what we saw for the analytical approaches, the graphical approach to synthesising indicators is not extraneous to risks and criticisms. In order to allow patterns, relations, dimensions to be communicated in a more efficient and clear way, many decisions should be taken in managing graphical instruments. For example, the relationship among indicators could be represented in terms of network visualization which may emphasize different aspects (density, organic growth, instability, dynamism) and/or different structure (symmetry, top-down, stable dimensions). One of the risks is that the synthetic representation of indicators could be created in order to obtain just more aesthetically interesting description. By adopting Tufte’s words, “what is to be sought in designs for the display of information is the clear portrayal of complexity. Not the complication of the simple; rather the task of the designer is to give visual access to the subtle and the difficult – that is, the revelation of the complex.” [17].

8. Discussion and outlook

The paper reviews non-economic indicators used in analysing and presenting official statistics. Some of the discussion parallels the framework on information quality proposed in [18, 19]. Eventually, indicators provide an evidence based foundation for stakeholders interested in action operationalisation initiatives. Action operationalisation is one of the eight information quality dimensions. In other words, indicators take you from a construct, reflecting abstract conditions, to an action aiming at affecting such conditions. Indicators are designed to provide a wide coverage of perspectives. The multivariate approach to indicators presented here provide policy makers with an integrated perspective on social and economic systems such as transportation, healthcare, education and agriculture. Furthermore. we show that it is of critical importance for the production and use of high-quality indicators that both the technical-statistical and the communicative-application-oriented aspects of indicators are considered.

In direct comparison with the statistical areas in which surveys and accounts are used and produced, internationally established methodical standards are lacking for the younger statistical area of indicators. In this respect, it is obvious to recommend as a resumé the development of such standards and guidelines by the international statistical community. This does not refer to guidelines that relate to a specific set of indicators (such as the SDG indicators). Rather, there is a lack of useful handbooks (or rather ‘headbooks’) in which the various aspects elaborated in this article are prepared and summarised in such a generalised manner that they can serve as orientation aids and compasses for the designers, compilers and communicators of indicators in their daily work. Fortunately, there are already some such guidelines, such as OECD/JRC’s “Handbook on Constructing Composite Indicators” [20] or Eurostat’s three-part series of indicator manuals [21, 22, 23]. An international statistical standard based on these sources, updated, enhanced and complemented, and supported by standard-setting bodies, would certainly be a major step towards improving the quality of indicators in general.