The Difficult Task of Evaluating MDG-4

Introduction

The development process involves substantial health, economic, and social changes, which affect directly the lives of billions of people around the world. The Millennium Development Goals (MDG) set up by the United Nations around year 2000 were targets associated with a selection of these changes and were based on relevant indicators measuring progress in development. ¹ Some of these targets were economic (poverty, hunger) or social (primary education, gender equality, women empowerment), and several were related to health (mortality, morbidity). Among the latter category, the fourth goal (MDG-4) targets the mortality of young children. As stated in the original document, MDG-4 proposed to reduce by two thirds, over a 25-year period (1990-2015), the under-5 mortality rate (U5MR), expressed as the number of deaths of children under the age of 5 years per 1000 live births. The U5MR is one of the leading indicators of the health transition and focuses on the age group in which mortality is the highest in high mortality populations, outside of very old ages. This indicator varies from some 500 per 1000 and above in very high mortality situations to 5 per 1000 and below in low mortality situations. This very wide range of variations, a scale from 1 to 100, makes it highly sensitive to measure progress in the health sector.

Mortality decline is primarily the result of modern public health (hygiene, preventive medicine, curative medicine, nutrition) and is highly responsive to health services and programs. So levels and trends in mortality are choice indicators for monitoring progress in the health sector. However, mortality levels and trends also depend on external circumstances, such as emerging infectious diseases (eg, HIV/AIDS, resistant malaria), health and environmental hazards (eg, industrial pollution, pesticides), behavior (eg, smoking, alcoholism), natural disasters (eg, drought, tsunami), occupational hazards (eg, mines, industry), man-made disasters (eg, wars, civil unrest, famine), as well as social circumstances (eg, poverty, discrimination). So mortality levels and trends may reflect not only progresses in public health but also the local circumstances, whether favorable or unfavorable.

The aim of this article is to review evidence of changes in under-5 mortality in sub-Saharan Africa and their implications for evaluating MDG-4. Some 35 countries with available DHS surveys were selected. Several essays of reconstructing mortality levels and trends were considered: those made by international organizations and those made by the author. The main period investigated was from 1985 to 2010, and comparison with the previous 25 years (1960-1985) is also provided. Several papers have already addressed this issue of evaluating MDG-4, but without discussing in detail data quality and local circumstances.^2-16

Data and Methods

Most sub-Saharan African countries have no reliable registration of births and deaths, as developed countries, and therefore have no time series that could be used for investigating changes in under-5 mortality. There are a few exceptions with comprehensive vital registration, in particular islands (eg, Mauritius, Reunion, Cape Verde). In continental Africa, only South Africa now has high level of coverage of registration of births and deaths, but this is a recent situation (since the mid-2000s), and registration is still incomplete for deaths of infants and children. So for sub-Saharan Africa one has to rely on other sources for investigating mortality trends.

Two major sources of data have been used in the past: population censuses and sample demographic surveys, which have their own specific features and weaknesses. Censuses provide a full coverage of mortality for the country as a whole, but data tend to be of lower quality, simply because population censuses are huge operations conducted in a short period of time during which data quality is hard to maintain. In contrast, demographic surveys provide mortality estimates for only a sample of households. Data quality tends to be higher because more time is devoted to data collection and quality control, but mortality estimates are hampered by statistical variations due to the sample size (usually a few thousands of households) and due to the sampling scheme (usually stratified cluster sampling). Several large programs of censuses and surveys have provided mortality data in African countries: the decennial census program (since the 1960s), the INSEE survey program (1960s and 1970s), the World Fertility Survey program (1974-1983), the Measure-DHS program (in place since 1985), the MICS survey program (since the mid-1990s), and a few others. These surveys vary in data collection methods, in data quality, and in geographical and time coverage. Interpretation of levels and trends from all these different sources is therefore delicate.

Methods of mortality data collection vary between sources. Censuses often rely on deaths in the 12 months preceding the census, which may provide reliable estimates when great care is devoted to the precise timing of the death and to the age at death, but might be seriously distorted otherwise. Censuses and some surveys (such as MICS surveys) often rely on indirect methods, based on children ever-born and children surviving to women in their reproductive ages. ¹⁷ This method has serious shortcomings, in particular because it cannot provide a point estimate, but only a weighted average of many point estimates. Furthermore, in order to convert the proportion of children who died into standard under-5 death rates, several hypotheses are required, in particular regular mortality change, no fertility change, and use of reliable model life tables. Violation of theses hypotheses, and possible underdeclaration of children who died in early infancy, often lead to estimates that are far from true values. The most reliable method of data collection is the recording of full births histories, with precise dates of birth and dates of death, as done in DHS surveys. These data provide robust estimates of levels and trends in under-5 mortality in the years preceding the survey. In certain cases, they may be somewhat biased, because they are based on births to women aged 15 to 49 at time of survey, so births that occurred during the same time period to older women are missed. This tends to overestimate mortality levels in earlier periods, because children born to younger women have higher mortality, and therefore overestimate declining trends. But the most critical issue of these mortality estimates derived from DHS surveys is the sample size and the sampling scheme.

The DHS surveys are the most useful source for the purpose of monitoring levels and trends, and are presented in detail elsewhere. ¹⁸ An average DHS survey in sub-Saharan Africa is based on a sample of 7750 households, 24 400 births, 4200 deaths, of which some 1120 are deaths of under-5 children in the past 5-years, the basis of under-5 mortality point estimates. As a result, an average level of under-5 mortality of 128 per 1000 is given with a standard error of 5.85 per 1000 (of which about 63% is due to sample size and the rest to design effect), that is, a precision of about ±10% (range 5% to 35%). These sampling variations imply that 2 surveys conducted 5 years apart do not permit to assess a 10% mortality change, which is the order of magnitude of expected mortality decline over 5 years. However, beyond point estimates, the same DHS surveys allow to measure mortality trends with much higher precision. Details on the method to investigate trends are given elsewhere.^19,20 In brief, trends are estimated directly from each yearly data point of U5MR and fitted with a regression line. This procedure provides narrower confidence intervals and more robust trends.

Examples of discrepancies between under-5 mortality estimates are given below. Point estimates are notoriously imprecise, because of data collection, method of analysis, and sample size. Mortality trends are usually better documented, but here again with variations depending on the source. Some institutions use a mix of sources and methods (census and surveys, direct and indirect methods), which is often confusing because the differences between sources may be larger than real differences in mortality levels. This analysis focuses on a reconstruction of under-5 mortality trends based solely on DHS survey data and direct calculations.^19,20 The main advantage of this approach is to ensure consistency in trends. Furthermore, despite similar methods of data collection and data analysis, one sometimes finds inconsistencies between surveys in the same country, usually because of different sampling schemes, and sometimes because of varying data quality. As will be seen in the Results section, the reconstruction using a robust method may lead to conclusions different from that obtained with strategies mixing data sources and calculation methods, with major implications for evaluating MDG-4.

Results

This section presents variations in levels and trends in under-5 mortality and consequences for evaluating MDG-4 from 4 sources: reconstruction of trends from DHS data available at FERDI, ²¹ estimates made by the United Nations Population Division (UNPD), ²² estimates made by UNICEF and the World Health Organization (WHO) under the Inter-agency Group for Mortality Estimation (IGME), ²³ and estimates made by the Institute for Health Metrics and Evaluation (IHME). ²⁴ All these data have open access on the Internet. This study is restricted to 35 countries with available DHS data, for the 1960 to 2010 period, and divided into two 25-year periods: 1960 to 1985 and 1985 to 2010. The second period is used for testing the capacity of available data to evaluate mortality changes. The MDG-4 targeted originally the 1990 to 2015 period, but data for this period will be available only around year 2020, and the point of this article is simply to show the difficulties in pursuing such an evaluation. The first 2 data sets are complete for the whole 1960 to 2010 period, but the IGME set is incomplete for many countries before 1980, and the IHME set starts only in 1970.

Overall Trends

Overall, for the 35 countries considered together, mortality decline has been impressive over the 1960 to 2010 period (Figure 1). Estimates vary somewhat between the first 3 sources, but the picture is the same: a steady decline at average speed (about −1.7% per year) from 1960 to 1985, a slower decline from 1985 to 1997 (about −1.2% per year), and a faster decline from 1997 to 2010 (about −3.7% a year). Overall, under-5 mortality was reduced by 65% in 50 years, from 275 per 1000 in 1960 to 96 per 1000 in 2010. Only the IHME estimates differed, with a lower slope, due to lower levels in 1970 (210 per 1000) and higher levels in 2010 (139 per 1000), the main discrepancies being in the last 10 years. The overall mortality decline in Africa can be compared with that in Sweden a century before (1860-1910), where the decline was similar in levels and trends, from 240 per 1000 in 1860 to 117 per 1000 in 1910, a 51% reduction in 50 years. ²⁵ In Africa, mortality decline in the last 25 years (1985-2010), although impressive, failed to meet the ambitious MDG-4 (47% mortality reduction instead of 67% targeted).

OPEN IN VIEWER

Figure 1.

Trends in under-5 mortality in 35 sub-Saharan African countries, 1960-2010 (by source), and comparison with expectations from MDG-4, and with Sweden (1860-1910).

Variations by Country

If the overall pattern emerged clearly, variations between sources were large and were due to estimation methods and data interpretation (Table 1). Not only slopes differed in absolute value between sources, but even the sign of the slope (positive or negative) was different in a number of cases. In the 2 Congos, the slope was positive (mortality increase) in the IHME estimates but negative in the other sources; in Lesotho, in Swaziland, and in Zimbabwe, 2 sources found a mortality increase, but the other 2 found a mortality decline; in Nigeria the IHME estimates indicated a mortality increase, contrary to the other 3 sources. In all these cases, the DHS yearly data indicated no change in mortality trends in the recent period: steady mortality increase in Lesotho, Swaziland, and Zimbabwe and steady mortality decline in the 2 Congos and in Nigeria, whereas indirect estimates from various sources were confusing. In the 3 Southern African countries, mortality increase was obviously due to HIV/AIDS, before prevention and care could inverse the increasing mortality trends, as already happening in South Africa and Namibia after 2005. In Nigeria, the 3 recent DHS surveys (2008, 2010, 2013) all indicated a steady mortality decline. In Congo-Brazza, the last 2012 DHS survey also indicated a steady mortality decline, as was the case in Congo-Kinshasa with the 2013 DHS survey.

OPEN IN VIEWER

Table 1.

Estimates of Change in Under-5 Mortality From 1985 to 2010 According to Sources.

Country	Reconstruction From DHS Data (FERDI)	United Nations Population Division (UNPD)	UNICEF/WHO (IGME)	Health Metrics (IHME)
Angola	39.5%	37.7%	19.9%	8.9%
Benin	66.7%	43.2%	52.0%	39.7%
Botswana	37.5%	42.2%	7.6%	47.2%
Burkina Faso	45.5%	33.0%	47.7%	30.1%
Burundi	41.3%	23.5%	47.2%	2.5%
Cameroon	20.0%	20.2%	30.3%	22.0%
Chad	29.5%	23.0%	29.8%	20.1%
Comoros	57.4%	33.2%	41.9%	63.7%
Congo-Kinshasa (RDC)	35.4%	9.1%	29.4%	−6.6%
Congo-Brazza (RC)	33.3%	12.9%	38.9%	−29.9%
Cote d’Ivoire	31.6%	27.1%	28.9%	33.3%
Ethiopia	60.8%	64.2%	66.1%	43.2%
Gabon	31.2%	34.4%	36.5%	32.6%
Ghana	50.2%	43.5%	46.4%	38.1%
Guinea	54.3%	47.6%	57.3%	55.2%
Kenya	39.3%	20.8%	17.4%	13.0%
Lesotho	−53.8%	25.4%	−12.4%	35.8%
Liberia	58.7%	61.2%	65.1%	24.8%
Madagascar	67.8%	64.8%	65.6%	49.6%
Malawi	59.3%	48.7%	67.1%	50.7%
Mali	67.4%	37.1%	51.8%	34.7%
Mozambique	60.5%	48.3%	59.3%	25.8%
Namibia	38.5%	51.4%	35.6%	48.3%
Niger	62.4%	57.2%	62.8%	39.1%
Nigeria	36.1%	37.8%	38.0%	−5.6%
Rwanda	65.0%	51.9%	59.8%	15.4%
Senegal	65.6%	53.9%	61.7%	46.8%
Sierra Leone	47.0%	22.6%	37.2%	19.5%
South Africa	10.0%	17.5%	27.3%	5.7%
Swaziland	−36.5%	12.3%	−7.1%	44.0%
Tanzania	63.6%	52.0%	65.3%	31.6%
Togo	43.8%	31.2%	41.5%	37.0%
Uganda	56.2%	48.8%	58.4%	25.5%
Zambia	41.5%	35.2%	41.9%	1.8%
Zimbabwe	−19.0%	30.3%	−20.3%	25.8%

Abbreviations: IGME, Inter-agency Group for Mortality Estimation; WHO, World Health Organization; IHME, Institute for Health Metrics and Evaluation.

Beyond the direction of the trends, serious discrepancies could also be noted in the magnitude of the slope between sources, with consequences for evaluating MDG-4. This happened for instance in South Africa, because of the complex changes in mortality, with major increase (due to increasing HIV/AIDS prevalence) followed by major decrease in mortality (due to prevention and treatment of HIV): if 3 sources (but IHME) agreed on the pattern, they differed somewhat in the timing and the magnitude of the changes, with consequences for the average change over the 1985 to 2010 period. A similar problem was found in Namibia, also for similar reasons. In Cameroon, the first 2 sources agreed on the speed of the mortality decline, whereas the third source indicated a stronger mortality reduction (30% vs 20%). Other differences were of smaller magnitude.

As a consequence, the number of countries meeting goals such as 30%, 40%, or 50% mortality reduction differed markedly by source. For instance, out of the 35 countries investigated, 15 countries were meeting the 50% mortality reduction from 1985 to 2010 according to the first source, 8 countries with the second (UNPD), 13 countries with the third (IGME), and only 3 countries with the last source (IHME). These differences reveal the difficulty in evaluating MDG-4 from imperfect data (Table 1).

Erratic Mortality Changes

A goal such as MDG-4 assumes implicitly a steady mortality decline and targets its speed as an indicator of regular progress in public health services and programs. But steady mortality decline was not a feature of most African countries. Among the 35 countries investigated, only 9 had a steady mortality decline. Examples of such countries are shown on Figure 2. Even in these smooth cases, there were at times variations in the speed of mortality decline, reflecting a variety of difficulties in each case, which complicate the trend analysis between 1985 and 2010.

OPEN IN VIEWER

Figure 2.

Selected countries with steady under-5 mortality decline, sub-Saharan Africa (from DHS data).

The other countries endured periods of mortality increase because of political crises, economic crises, or social crises (19 cases) or because emerging or resurging diseases (11 cases), namely, HIV/AIDS and malaria. Erratic trends complicate the interpretation of mortality changes over a fixed period of time, such as 1985 to 2010. Figure 3 displays selected countries with large fluctuations in mortality trends. In Rwanda, the civil war period (1991-1999) was associated with major disruption in already erratic trends.^26,27 In Zambia, the copper crisis period (1975-1992) and HIV/AIDS induced major changes in trends. ²⁸ In Lesotho and Swaziland, HIV/AIDS induced drastic increases in mortality. In Congo-Brazza, as in Congo-Kinshasa, major political crises induced large fluctuations in under-5 mortality. Many other countries (not shown) also underwent serious disruption in mortality trends for similar reasons.^19,20 These chaotic situations hamper any simple trend analysis and make the evaluation of MDG-4 difficult without taking the local context into account.

OPEN IN VIEWER

Figure 3.

Selected countries with erratic under-5 mortality decline, sub-Saharan Africa (from DHS data).

Furthermore, trend analysis between 2 points such as 1985 and 2010 depend not only on the regularity of the decline but also on the situation in 1985 (normal or outstanding) and the consistency of data points. For instance, the country with the fastest mortality decline from 1985 to 2010 was Madagascar. This was due in part to the fact that the 1985 baseline year had abnormally high mortality due to a 12-year economic and political crisis peaking in 1985, which was followed by a quick recovery. Had one taken 1973 as the baseline year, results would have been different. Some of best performers were countries with high mortality levels at baseline, and steady mortality decline, such as Benin and Mali. However, in Benin, mortality decline seemed exaggerated by the fact that the last 2012 DHS survey showed lower mortality levels than previous surveys. The same phenomenon was visible in Mali with the 2013 DHS survey.

On the other side of the spectrum, some countries endured major mortality increase because of HIV/AIDS. This was the case in all countries located in Southern Africa, in several East African countries, and in some West African countries. Had HIV not been prevalent in these countries, mortality decline would probably have been steady over the 1985 to 2010 period, because public health services and programs did in fact improve dramatically. So mortality trends follow first the prevalence of HIV (say from 1985 to 2000), and second the prevention of mother to child transmission (since the early 2000s) and more important antiretroviral therapies (since the mid-2000s). Last, resistant malaria caused a mortality increase in the 1990s in selected countries (as in Senegal), and the new programs of prevention (bednets) and care (new antimalarial drugs) caused major mortality declines in many countries with high malaria mortality. Therefore, these erratic trends are in part due to external causes (emerging diseases or medical innovation) and not to the simple functioning of the health system.

Discussion

Overall and despite serious difficulties, African countries underwent a major mortality decline from 1960 to 2010, and especially in the last 10 years. At this baseline level of mortality, the decline in Africa compares favorably with than in Sweden a century before. This is impressive in itself, and can be considered as a very positive achievement, especially given the level of income and of social organization of these countries, and the strong perturbations provoked by the HIV/AIDS epidemics and by malaria. Note that impressive mortality declines were achieved in countries with low levels of income and education, as in some of the Sahelian countries.

Formally, MDG-4 was not met, except by a few countries, and even these few cases could be discussed at length. MDG-4 appears therefore as overambitious, especially when compared with past experience in Africa (1960-1985), and with past experience in Europe. Why African countries should undergo a faster mortality decline than European countries at the same level of mortality could be questioned. Furthermore, had the HIV/AIDS epidemic not occurred, mortality decline would have been faster, showing that these countries made in fact major efforts in public health.

This analysis raised the issue of data quality. Some of the estimates appeared far from those shown in DHS data, usually considered as the most consistent and reliable. In a few cases, trends were in fact in the opposite direction, which is surprising. More work is needed to better interpret alternative data sets, in particular estimates derived from indirect methods, which can be confusing when hypotheses on which the methods are based are too strongly violated.

Trends are better estimated while using consistent sources, such as DHS surveys. But even in this case, one could underestimate or overestimate levels and trends because of the large fluctuations due to sample size and sampling scheme. Ultimately, one wishes to have reliable birth and death registration in order to obtain more robust mortality estimates, and a better grip on achievements in MDGs.

Last, a formal analysis of achievements in public health outcomes must take the local context into account. Only a detailed analysis of the local situation will allow one to understand whether the country did well or not, and if not, why it did not. One should remember that mortality decline was not always smooth in Europe, Russia, or Asia in the 19th and early 20th century at similar levels of mortality. Increases in mortality could be due to wars (world wars or civil wars), diseases (tuberculosis, Spanish flu), severe economic or political crises, increasing inequalities and poverty, and other factors. Africa is no exception to these earlier experiences.