SDG progress assessment;comparing apples with what?

Abstract

This paper looks at a selected number of metrics developed by different international organizations for measuring progress towards the Sustainable Development Goals (SDGs) and aims to shed light on differences and highlight where harmonization is most necessary. It shows that inconsistency in results is more likely to be driven by different interpretations of concepts not methodologies, emphasizing that this has to be a priority in order for any harmonization to be successful. The paper provides a set of principles for orchestrating SDG progress assessment efforts across international organizations.

Keywords

Sustainable Development Goals progress assessment SDG indicators normalization

1. Introduction

With just under a decade left to achieve the Sustainable Development Goals (SDGs), countries around the world endeavour to understand if their current pace of progress is fast enough to achieve their 2030 targets and where acceleration or course correction is needed. Many international development organizations have supported such efforts by developing frameworks to measure SDG progress of countries and regions [2, 3, 4, 6, 7]. This has created a useful dialogue on whether this diversity in the proposed approaches is a sign of inconsistency and to what extent it may confuse, rather than assist, in monitoring implementation of the SDGs. The reviews of technical aspects of alternative metrics [1, 5] show that despite all being considered progress measures, they differ in underlying concepts as well as measurement techniques.

This paper explores various terminologies associated with the concept of “progress” and examines differences in metrics applied by five organizations. The paper argues that regardless of normalization and indexing techniques employed, the metrics are identical or at least provide identical ranking when underlying concepts are same.

Progress measures may vary in three different dimensions: evidence base (indicator, data and targets), underlying concepts (understanding of progress), and techniques (normalization, target setting, projections etc). This paper places more emphasis on underlying concepts and shows that when the same concept is being measured, metrics are mathematically interchangeable (if not equivalent) and results are unlikely to lead to conflicting comparisons. Similarly, when two different aspects of progress are well-defined, properly measured and clearly communicated, results are more immune to misinterpretation.

To understand and measure one question about progress assessment in different ways without clear communication of the underlying concepts is a recipe for confusion, to ourselves as professional statisticians as well as to users. Therefore, the harmonization of evidence base and statistical techniques are necessary but insufficient without a common understanding of concepts. Careful choice of terms and understanding, as well as clear communication of underlying concepts is fundamental as its absence will leave users in the dark regardless of sound and sophisticated statistical methods.

2. Understanding underlying concepts

What is “progress”? The term “progress” appears on the cover page of almost every data-driven report on the SDGs. We seem to have taken our common understanding of the term for granted. At least three different interpretations of progress may be derived from existing SDG reports: trend, achievement, and relative progress.

Trend defines progress as taking at least one step further in desirable direction. It measures the size of change for an indicator over time.

This interpretation of progress has been applied on SDG indicators without explicit targets [3, 4]. In this approach an annual growth rate (in our examples, compound annual growth rate (CAGR)) is used as a measure of trend and evaluation of current growth against desirable direction provides a subjective assessment for progress.

$\displaystyle\textit{CAGR}=\left({\frac{x_{t}}{x_{t_{0}}}}\right)^{\frac{1}{t-% t_{0}}}-1$

Where, $x_{t}$ and $x_{t_{0}}$ are the latest value (at year $t$ ) and value of indicator at base year ( $t_{0}$ ) respectively.

Achievement defines progress as distance to a milestone. In the context of the SDGs, it measures the difference between the latest value of an indicator and a pre-specified target value.

Given huge data gaps for the SDG indicators, this may be the most popular interpretation of progress as it does not require time series data. At least three reports take this approach either only for indicators with explicit target [4] or for all indicators with at least one data point [6, 7].1 For an indicator with decreasing trend as desirable direction, two different mathematical expressions have been used:

$\displaystyle\textit{FAO}_{A}\&\textit{SDSN}_{A}=\frac{x_{t}-x^{w}}{x^{\ast}-x% ^{w}}$ $\displaystyle\textit{OECD}=\frac{x_{t}-x^{\ast}}{\textit{STD}({x_{t}})}$

Where, $x^{w}$ and $x^{\ast}$ are worst value of indicator across countries and target value for the indicator respectively. STD is standard deviation of the latest indicator value across countries. It is shown in the Annex that these two mathematical expressions are interchangeable and provide exactly same ranking of achievements.

Relative progress defines progress as current trend benchmarked against a desirable trend. It measures the current rate or size of change in indicator in relation with total change needed to achieve a milestone.

At least three metrics have been proposed by different organizations. For an indicator with decreasing trend as desirable direction, ESCAP uses two different metrics to measure current status of progress and anticipated progress at the target year.

current status of progress:

$\displaystyle\textit{ESCAP}_{RP1}=\frac{x_{t_{0}}-x_{t}}{x_{t_{0}}-x^{\ast}}$

anticipated progress:

$\displaystyle\textit{ESCAP}_{RP2}=\frac{x_{t_{0}}-x_{p}}{x_{t_{0}}-x^{\ast}}$

where $x_{p}$ indicates predicted value of indicator at target year.

And three organizations [3, 4, 7] measure relative progress as ratio of actual growth rate and desirable growth rate for achieving specific target.

$\displaystyle\textit{FAO}_{RP}\&\textit{Eurostat}_{RP}=\frac{\textit{CAGR}}{% \textit{CAGR}^{\ast}}$ $\displaystyle\textit{SDSN}_{RP}=\frac{\textit{LAGR}}{\textit{LAGR}^{\ast}}$

where $\textit{CAGR}^{\ast}$ is the compound annual growth rate required to achieve the target ( $x^{\ast}$ ) and LAGR and $\textit{LAGR}^{\ast}$ are actual and required linear annual growth rates respectively (see Annex).

Table 1
Information requirements for measuring each of the progress concepts

Policy question	Progress concept measured	Information requirements
		Baseline	Target	Time series
How fast and where are we heading?	Trend	$\surd$	$\times$	$\surd$
How far are we from the targets?	Achievement	$\times$	$\surd$	$\times$
How much have we progressed in achieving the targets?	Relative progress	$\surd$	$\surd$	$\surd$

It can be shown that under two interpretations (achievement and relative progress) where more than one metric has been proposed, metrics are mostly interchangeable (i.e. each metric can be either transformed to a linear function of another or provide same ranking as other metrics) (Annex). The actual choice, therefore, is not among metrics but among various interpretations of progress – are we using the word “progress” to mean trend, achievement or relative progress?. Shifting from one interpretation to another disrupts comparability (no matter which metrics are used) and mixing results from various interpretations opens the door to confusion.

3. Unpacking the question

First things first: Much of the discussion around harmonization of SDG progress measures is focused on the choice of statistical methods. For instance, whether or not we should set targets, and whether it is appropriate to use time series in the face of data gap. I argue, these important issues can only be resolved after we decide what interpretation of “progress” has the most policy relevance. The three interpretations serve different purposes, answer different questions, and have different information requirements (Table 1).

Trend analysis (concept one) can only point to the direction and pace of change and requires no target value. Nevertheless, it provides limited information for policy prioritization as slow pace of progress should not raise any red flag for indicators with high level of achievement. To be policy relevant, trend analysis must be presented together with data on achievement (concept two), which in turn demands for setting a target value. Additionally, the measure of achievement presents the challenge of normalization. Normalizing this measure with other countries’ achievements [4, 7, 6] may provide a good basis for ranking but has less relevance to national policy prioritization.

Alternatively, one may decide to apply a relative progress approach (concept three) which combines both trend and achievement information and benchmarks current pace of progress against the desirable pace. In other words, it is a measure of achievement normalized by target and baseline values of the same country, rather than other countries’ achievements. This provides required information for identifying indicators and targets that need prioritization for acceleration of progress. But it does not come without a cost. It requires more data points and masks information on current level of achievement. The latter could be misleading in cases where level of achievement is high (close to target) prior to a baseline year.

It is clear from Table 1 that our decision on information requirements (baseline, target, and time series) is not purely a statistical choice. It determines what policy questions we can answer and defines the limits for our analysis. It is therefore natural that the choice of information should be driven by policy questions and not the other way round.

Behind the math: The consistency and comparability of results should raise concerns only when we mix different interpretations without clearly distinguishing them. As shown earlier, we often have multiple options of mathematical expression under each interpretation which are interchangeable and provide identical ranking (see Annex). But when different interpretations are combined (regardless of metrics), we are comparing apples and oranges and provide a recipe for confusion and inconsistency. Table 2 illustrates this by an example on national poverty rate in three countries: Thailand, Georgia, and Kyrgyzstan for two underlying concepts: achievement and relative progress.

Table 2
Progress assessment for population below national poverty line (%) in 2018 ${}^{2}$

Country	National poverty rate (%)		Achievement		Relative progress
	2015 ( $x_{t_{0}}$ )	2018 ( $x_{t}$ )	$\textit{SDSN}_{A}\&\textit{FAO}_{A}$	OECD	$\textit{ESCAP}_{RP1}$	$\textit{ESCAP}_{RP2}$	$\textit{SDSN}_{RP}$	$\textit{FAO}_{RP}\&\textit{Eurostat}_{RP}$
Thailand	7	10	8	8	$-$ 6	$-$ 10	$-$ 10	$-$ 10
Georgia	22	21	6	6	1	2	1	0.2
Kyrgyzstan	32	22	6	6	3	9	4	1.4

Note: Target ( $x^{\ast}$ ) $=$ 2%, STD $=$ 12.2, worst value ( $x^{w}$ ) $=$ 54.5%.

No matter which metric is used to measure relative progress, all results provide exactly the same ranking. But findings from achievement and relative progress send opposite signals about performance of three countries. The best performer in terms of achievement (Thailand) has the worst relative progress and is in fact regressing.

A similar scenario could be true for three indicators in one country. It is obvious that relying only on measures of achievement does not help preventing negative trends emerged in the first row and does not recognize the significant progress made in the third row in Table 2.

In practice, it is possible that progress assessment aims to address multiple policy questions (interpretations). In this case, results from different interpretations could still be presented side-by-side with clear distinction. But mixing results from two interpretations in a single visualization must be avoided, unless differences are clearly highlighted and explained to the user. This is especially the case when we treat indicators with and without explicit targets or indicators with time series and those with only one data point differently and combine the results.

Dilemma of setting target values: Perhaps target setting is the most controversial issue around SDG progress assessment. On one hand, it makes progress assessment more meaningful and policy relevant, on the other hand explicit target values do not exist for more than 70 percent of the SDG targets and few countries have set national targets. Two questions are often asked: what role statisticians play in setting national targets?, and can we set targets for global and regional progress assessment when most of the indicators need country specific targets?

Two factors are important to keep in mind in answering these questions: First, in an ideal setting, targets are set by policy makers based on evidence provided by official statistics. In other words, statisticians play an important role in target setting by providing several “objective alternatives”, out of which policy makers can select one feasible and aspirational target considering existing resources, national priorities and political realities. In the absence of the latter, and for the purpose of progress assessment, one of the objective alternatives could still be selected based on historical data and is preferable to any subjective assessment of progress. Statistical community is instrumental for increasing awareness about role of target values in policy monitoring and can inform stakeholder consultations on national targets. Second, when set at regional/global level, a target value used for progress assessment should not be interpreted as a commitment for every country to fulfil. Regional/global targets serve as benchmarks to evaluate “average” progress and make sense even for indicators which require individual country target values as national commitments. In statistical term, a regional/global target is used to assess how distribution of an indicator is shifted over time and should not be used to assess any individual country progress, unless nationally adopted.

Three alternative approaches are proposed for dealing with indicators without explicit target values, or existing international standards. First is setting a target at average value of the indicator among top performers [6, 7]. Second is to categorise current rates of change based on expert/agency evaluations [3, 4]. The third is finding feasible yet stretching target rates based on historical national data [2]. In practice, the last two approaches are similar in setting target rates (as opposed to absolute target values), one with subjective and incremental thresholds and the other with one objective cut-off as target rate.

4. Conclusion

Apart from underlying data and indicators, two sources of inconsistency in SDGs progress assessment measures are interpretations of the underlying concept of progress and metrics used for measuring the chosen progress concept. This paper has introduced three different interpretations for “progress” (trend, achievement, relative progress) to describe metrics currently used by five different international organizations. It is shown that under each interpretation, different metrics can be used interchangeably (i.e. they provide almost identical ranking of progress across indicators or countries). The paper argues the first step in developing a measurement framework for SDG progress assessment must be deciding on policy questions which we aim to answer. Policy questions should define our interpretation of progress and guide us in selecting relevant metrics and information requirements (baseline, target, and time series).

The following are four generic principles that apply to every attempt for assessing progress towards the SDGs:

Harmonize the evidence base: Regardless of the approach taken for measuring progress, comparability and consistency of results depend on the evidence base (indicators and data). Countries have endorsed a common set of indicators developed by an Inter-Agency and Expert Group on SDGs indicators (IAEG-SDGs)3 and the UN Development System regularly collects and disseminates harmonized national data on the SDGs.4 These provide a basis for consistent and comparable national and international reporting on SDG progress. Countries are encouraged to integrate official SDG indicators into their national follow up and review processes and when additional indicators are needed to complement the global SDG indicators, international standards and definitions must be followed.

Ask the right questions(s): For any progress assessment to be relevant and to serve national policy prioritization, it must answer specific policy question(s). The way we understand and interpret “progress” dictate restrictions on the use of the results and information requirements. When understood as trend, progress assessment provides very limited information for prioritization. However, progress as relative change or achievement of pre-defined targets enable policy makers to identify areas that require acceleration of progress or that are at risk of set-back.

Set targets when needed: Availability of target values and time series data should not dictate the type of progress we measure. It must be the other way round; if the purpose of the analysis is to understand achievement and/or relative progress, target values are a necessary part of measurement and cannot be avoided. Ideally, targets must be set by policy makers at national level, in consultation with all relevant stakeholders and based on official statistics. Statistics must provide alternative milestones among which policy makers select an ambitious target given political realities which is feasible within available resources. But in the absence of such efforts at national level, past trends and good practices can provide regional and international organizations with useful benchmarks as “objective, feasible, and challenging milestones” which can be used as regional/global target values for analytical purposes only, without imposing any political commitment on countries.

Compare apples with apples: It is likely that more

than one interpretation of progress is desirable to provide a full range of information to policy makers. For instance, measures of achievement combined with relative progress or trend data is necessary to identify areas of high achievement with slow progress. Moreover, data availability varies considerably across SDG indicators and a degree of flexibility in measurement is needed to fully utilize available data. It is therefore crucial to remember that metrics from different interpretations of progress do not provide comparable results and should not be mixed in a single graph or table without being clearly distinguished and communicated to the user. For instance, when indicators with and without explicit targets in an SDG framework or indicators with time series data and those with only one data point are treated differently and measured by metrics of different interpretations (points (b) and (c) explain why this is not a preferable approach), the results are not comparable and should be clearly differentiated.

Footnotes

Sachs et al. 2021 method is labelled as Sustainable Development Solution Network (SDSN).

https://unstats.un.org/sdgs/indicators/indicators-list/.

https://unstats.un.org/sdgs/indicators/database/.

Annex: Technical note

This annex provides mathematical details comparing metrics under two different interpretations of progress. It is assumed that decreasing trend is desirable for indicator $x$ and target value is equal to the minimum value of the indicator ( $x^{\ast}=x^{\min}$ and $x^{w}=x^{\max}$ ).

Achievement: Two proposed metrics measuring achievement are linear transformations of one another and therefore provide identical assessment.

$\displaystyle\textit{FAO}_{A}\&\textit{SDSN}_{A}=\frac{x_{t}-x^{w}}{x^{\ast}-x% ^{w}}=\frac{x_{t}-x^{\max}}{-\textit{range}}=1-\frac{x_{t}-x^{\ast}}{A\times% \textit{STD}({x_{t}})}=1-\frac{1}{A}\textit{OECD}$

Note: For any given location-scale distribution, standard deviation is proportional to range ( $\textit{range}=A\times\textit{STD}$ ). Also, $x^{\max}=\textit{range}+x^{\ast}$ .

Relative progress:

$\textit{SDSN}_{RP}$ is proportional to $\textit{ESCAP}_{RP1}$ .

$\displaystyle\textit{SDSN}_{RP}=\frac{\textit{LAGR}}{\textit{LAGR}^{\ast}}=% \frac{x_{t}-x_{t_{0}}}{x^{\ast}-x_{t_{0}}}\times f(t)=\textit{ESCAP}_{RP1}% \times f(t)$

Where $\textit{LAGR}=\frac{x_{t}-x_{t_{0}}}{t-t_{0}}$ , $\textit{LAGR}^{\ast}=\frac{x^{\ast}-x_{t_{0}}}{t^{\ast}-t_{0}}$ and $f(t)=\frac{t^{\ast}-t_{0}}{t-t_{0}}$

$\textit{ESCAP}_{RP2}$ , $\textit{FAO}_{RP}$ and $\textit{Eurostat}_{RP}$ provide identical ranking, when predicted values are estimated based on geometric growth rates.

$\displaystyle\textit{ESCAP}_{RP2}=\frac{x_{t_{0}}-x_{p}}{x_{t_{0}}-x^{\ast}}=% \frac{x_{t_{0}}-({\textit{CAGR}+1})^{({t^{\ast}-t_{0}})}\times x_{t_{0}}}{x_{t% _{0}}-({\textit{CAGR}^{\ast}+1})^{({t^{\ast}-t_{0}})}\times x_{t_{0}}}=\frac{1% -({\textit{CAGR}+1})^{({t^{\ast}-t_{0}})}}{1-({\textit{CAGR}^{\ast}+1})^{({t^{% \ast}-t_{0}})}}$ $\displaystyle\equiv\textit{FAO}_{RP}\&\textit{Eurostat}_{RP}$

Where $\textit{CAGR}=\left({\frac{x_{t}}{x_{t_{0}}}}\right)^{\frac{1}{t-t_{0}}}-1$ and $\textit{CAGR}^{\ast}=\left({\frac{x^{\ast}}{x_{t_{0}}}}\right)^{\frac{1}{t^{% \ast}-t_{0}}}-1$

Note: Due to using linear growth rates, $\textit{SDSN}_{RP}$ may provide slightly different ranking compared to $\textit{FAO}_{RP}\&\textit{Eurostat}_{RP}$ in rare/extreme cases. The results are therefore comparable most of the time.

References

Bidarbakht-Nia

. Measuring Sustainable Development Goals (SDGs): An inclusive approach. Global Policy. 11(Issue 1) (Feb 2020), 56–67.

ESCAP. Asia and the Pacific SDG Progress Report 2021. United Nations Publications, Bangkok; 2021. https://data.unescap.org/publications/sdg-progress-report-2021.

Eurostat. Sustainable development in the European Union, Overview of progress towards the SDGs in an EU context. Publications Office of the European Union, Luxembourg; 2020 Edition. https://ec.europa.eu/eurostat/documents/4031688/11010788/KS-01-20-192-EN-N.pdf/ae63aff0-a6f3-1d47-da83-c6886b9daaab.

FAO. Tracking progress on food and agriculture-related SDG indicators 2020: A report on the indicators under FAO custodianship. Rome; 2020. http://www.fao.org/sdg-progress-report/en/.

Gennari and D’Orazio. A statistical approach for assessing progress towards the SDG targets. Statistical Journal of the IAOS. 36 (2020), 1129–1142.

OECD. Measuring distance to the SDG targets 2019: An assessment of where OECD countries stand. OECD Publishing, Paris; 2019. https://www.oecd.org/sdd/measuring-distance-to-the-sdgs-targets.htm.

Sachs

Kroll

Lafortune

Fuller

Woelm

. Sustainable Development Report 2021. Cambridge: Cambridge University Press; 2021. https://www.sdgindex.org/.