Abstract
Statistics are the currency of debate and the basis for sound decisions. If they are misused the currency of debate is devalued and the basis for decision making is undermined. Without confidence in statistics, decision makers are flying blind when they make their choices and citizens are in the dark in seeking to hold those decision makers to account. Misuse of statistics undermines trust and by doing so it undermines democracy. This paper explores the safeguards available to protect against misuse of statistics. It describes the nature of the threats and how they are changing before assessing the responses. It concludes that there is unfinished business to be taken forward.
Introduction
The Fundamental Principles of Official Statistics [1] are a pillar of the Global Statistical System. They have been endorsed by the United Nations General Assembly and are of universal applicability to all nations and at the regional and global levels. They set the standard for statistics that serve the public good, helping decision makers make well founded decisions and enabling others to hold those decision makers accountable for their promises.
During the current pandemic, people across the world have been talking about statistics. People are looking for data they can trust about how to make choices in their daily lives in order to keep safe, how to interpret the actions their governments are taking and how to judge future prospects for the economy and society. The value of the Fundamental Principles and the statistical organisations that are guided by them has been brought into sharp focus: trustworthy official statistics are a vital public service that plays a unique role in helping guide our actions.
A particular feature of the current climate is the danger of false statistics that can mislead and result in poorer decisions. Such false statistics may be the result of an inadequate design, implementation or communication or a deliberate attempt to deceive in order to serve a vested interest. Either way false statistics result in poorer decisions and therefore lost lives, weaker economies, less just societies and a future for our children that is not as sustainable as it could have been.
The 4
The paper recognises that this is not a new issue but argues that the case for action grows ever stronger. It builds on debates within the statistical community over many years including at the World Statistics Congress in Marrakech (2017) [2] and the IAOS Special Meeting: NSO’s Professional Independence: Threat and Responses in Paris (2018) [3]. More recently, presentations made at an International Statistical Institute/ International Association for Official Statistics webinar [4] held on 6 October 2020 describe the current situation and case for change.
What do we mean by misuse of statistics
Misuse of statistics is about misrepresenting reality. Each of us sees the world in a different way based on our experience, our beliefs, our current situation and how we are feeling. How we act will also differ depending on our priorities and the range of choices available to us at that moment. However, if someone is cloaking a vested interest in an apparent statement of fact that does not stand up to scrutiny or is trying to hoodwink us into believing in a fantasy view of the world, then we are at risk of making decisions that will not turn out well for us.
Statistics has always been the source of jokes on “how to lie with statistics”, but with so many new tools available the scope for conscious misleading has grown substantially. As Vincent Hendricks and Mads Vestergaard [5] have compellingly argued, “in the digital age, misinformation has become a global challenge joining the family of anthropogenic climate change, increasing economic inequality, water supply shortage, global health problems, and a range of other urgent problems …without sound information as bedrock for formation of political opinion, decision-making, and action, individual agency and political sovereignty of the people are crippled”.
Misuse of statistics is an abuse of power.
Misuse of statistics can take several forms. One form that is particularly prevalent is manipulation or presentation of statistics to support a claim that does not stand up against an impartial and professional assessment of what the statistics really show. It is wrong to assert that something is getting better on the basis of carefully chosen numbers selected to show just that, when other more relevant selections would have shown the reverse. It is wrong to claim that a number is a fair representation of reality when it has been derived in a way that is unrepresentative of the issue at hand. It is wrong to shout loudly that something has a great meaning when the evidence is just not good enough to make that claim.
Another form of misuse of statistics is to hide them from view. If a government has claimed itself to be open and transparent and yet produces statistics and analysis that it keeps secret, it is misusing those statistics. One element of this form of misuse is the situation where a government insists on having sight of statistics in advance of wider publication in order to give itself an advantage in presenting the findings in a good light. Another manifestation of this type of misuse is where a public authority prevents the collection of statistics that would inform the population about matters of public concern. More insidious still, as Andreas Georgiou [6] has pointed out, is self-censorship, when the statistical agency itself chooses not to address an important statistical question for fear of the consequences.
A linked form of misuse of statistics is to attack or undermine them once they are published. All statistics should be open to scrutiny to evaluate whether the findings can be replicated and confirm how potential biases have been addressed and uncertainty assessed and communicated. However, unless this scrutiny can be scientifically justified, it can represent a dangerous from of misuse of statistics. Attacks on statistical evidence about vaccines are a topical example of this. Attack on statistics can get personal when it is not just the statistics but the statistical institution publishing the statistics or members of its staff who are unfairly accused or even prosecuted.
The case of Andreas Georgiou is an example that has been highlighted by the International Science Council on 10 December 2020, Human Rights Day, in the following terms:
Dr Andreas Georgiou is a Greek economist and statistician, and former head of Greece’s national statistical office, the Hellenic Statistical Authority (ELSTAT). Dr Georgiou has been involved in a series of legal proceedings for nine years, related to his tenure as president of Greece’s national statistical office from 2010 to 2015. Dr Georgiou has been investigated, tried, and acquitted on three separate occasions on identical charges of conspiring to artificially inflate Greece’s deficit. In addition, he has been subject to criminal investigations for seeking to protect statistical confidentiality of the information of households and enterprises in accordance with statistical principles. The statistical processes and ethics for production and dissemination of the official statistics published by Georgiou are considered by the international statistical community to have been fully consistent with European Statistical Standards and international principles and ethics. Eurostat repeatedly verified the accuracy and reliability these figures and the methodologies used [7].
Why does this matter?
Statistics are the currency of debate and the basis for sound decisions. If they are misused the currency of debate is devalued and the basis for decision making is undermined. Without confidence in statistics, decision makers are flying blind when they make their choices and citizens are in the dark in seeking to hold those decision makers to account.
Misuse of statistics undermines trust and by doing so it undermines democracy.
What safeguards are in place?
The Fundamental Principles of Official Statistics were initially developed by the Conference of European Statisticians in the aftermath of the collapse of the Berlin Wall in 1989. They drew on the International Statistical Institute’s Declaration on Professional Ethics [8] (first adopted in 1985) and were designed to make a statement that in the new Europe there could be no place for misuse of statistics. The role of trustworthy statistics as essential to a well-functioning democracy was enshrined. The principles developed in Europe were rapidly seen as of universal application and were adopted by the United Nations Statistical Commission in 1994. Both the Declaration on Professional Ethics and the Fundamental Principles have since been periodically reviewed to assure their validity. The Fundamental Principles were endorsed by the United Nations General Assembly in 2014.
The safeguards set out in the Fundamental Principles and Declaration on Professional Ethics are also echoed in statistical laws and codes of practice for statistics. At the international level they can have a strong legal force on member countries of institutions such as the International Monetary Fund [9], Organisation for Economic Cooperation and Development [10] and European Union [11].
In the United Kingdom, there is a dedicated Office for Statistics Regulation [12]. The most recent OSR business plan sets out its role to “intervene where we identify significant or persistent issues with how statistics are being used or misrepresented”. It has a track record of drawing public attention to the misuse of statistics, including by the most senior figures in government.
What are the threats?
There are many mechanisms used to threaten the integrity and validity of statistics and to foster their misuse. These were well described in William Seltzer’s 1994 paper: Politics and Statistics: Independence, Dependence or Interaction [13]? Eleven specific threats are identified and analysed. These are:
Mission of the statistical service (is the aim of the statistical service tied to the interests of one political player or is it broader?) Financial resources and controls (can resources for statistical work be cut if the data produced is seen to be inconvenient?) Staff (are professional standards upheld or undermined?) Statistical fields or series targeted for expansion or suppression (who decides on the statistical programme?) Definitions, concepts and methodologies (is the basis for the statistics determined on scientific principles?) Terms and nomenclature (are things described in ways that aid understanding and acceptance?) Altering specific numbers The extent and timing of the release of data (is it controlled by the statistical agency or by others?) Threats to data confidentiality (how well are they dealt with?) Use of the statistical agency for political analysis or other political work (can the statistical agency avoid crossing “the unmarked border into policy advocacy”?) Active campaign to discredit statistical service outputs, methods or staff (how to avoid a bad outcome for a messenger bringing an uncomfortable message?)
One or more of these threats is present in many countries. In the case of Argentina it has been argued [14] that between 2007 and 2015 all eleven were present. Victor Beker explores the case of Argentina in detail as well as other examples of abuse of statistics in Latin America. He draws two important lessons from these experiences. First, given its role as “de facto, involuntary, auditor of the national government’s performance”, “the first priority is to ensure the autonomy of the National Statistics Office”. Second, that the institution must “strive to achieve greater credibility and respect among the population, making its procedures as transparent as possible”. It must “interact with the life forces of society – entrepreneurs’ chambers, trade unions, political parties – as well as users in the public and private sector” [15].
How are the threats changing?
The existing framework of safeguards dates from the 1980s and 1990s. Whilst it has been reviewed and updated since then it is operating in an environment that has changed radically. Indeed, it is a notable coincidence that the catalyst for the Fundamental Principles of Official Statistics was the fall of the Berlin Wall in 1989, the same year as the World Wide Web was invented by Tim Berners-Lee.
The World Wide Web has heralded the rise of the internet and the data revolution. When existing ethical codes for statistics were conceived data was a scarce resource with few actors involved in its generation, analysis and dissemination. Today the situation is transformed.
Data is generated continuously through sensors and capture of online transactions in real time. The amount and variety of data is at levels that would have been unimaginable 30 years ago and the internet puts vast quantities of that data in the hands of everyone. Data is also much more dynamic, capturing social and other interactions.
As Martine Durand (former Chief Statistician at the OECD) has pointed out, sources of misinformation and misuse have grown markedly. Much of the data available to us is unfiltered, with no mechanism to help us judge whether it is wrong, inaccurate or incomplete. The scope for manipulation and falsification is increased. New channels dominate creating echo chambers and a climate that can reinforce beliefs over facts. All these challenges are in turn compounded by a lack of statistical literacy and errors of interpretation [16].
Alongside this datafication of society, the political landscape has become much more complex and fluid. The period between 1945 and 1989 had generated parallel statistical and political structures both to mediate relations between countries and to govern within them. Since 1989, the combination of radical changes in the worlds of politics and data has challenged existing statistical frameworks as well as political ones.
How should we react?
Throughout history, the messenger who brings bad news is at risk. Statistics is a messenger bringing news that can be very uncomfortable for people in power who have strong vested interests in particular outcomes. The purveyors of statistics therefore need very powerful protection and general recognition that the role of statistical messenger should be safeguarded whatever the temptations of expediency.
Even the self-interest of the person made uncomfortable by the statistics is best served if impartial statistics are available and accepted, otherwise no claims made will have any value and the powerful person ultimately has the most to lose from that. However, in the short term, the messenger is at risk and needs others to rally round and give support. Building this understanding and network of support across the political spectrum, within the media and internationally, is a critical task for politicians and statisticians.
When statistics, statistical offices and individual statisticians are attacked the attack should be called out. It is especially potent when someone other than members of the statistical community stand up in this way and most potent of all if it is allies of the source of the attack who explain why such attacks are likely to rebound.
At the same time, the statistical office, and those involved in all stages of the production of statistics, should encourage scrutiny and challenge as a mechanism for learning. Statistics are always uncertain and statistical knowledge is always provisional and can be improved. Openness to challenge is itself a protection against attack whilst defensiveness arouses suspicion and frustration from critical friends with reasonable concerns.
Where statistics are misused or suppressed this should be called out. A most positive development in many countries in recent times has been support for Fact Checkers. Such bodies have no axe to grind in the debate and can earn high levels of trust. They can also be adept at navigating a tricky boundary between robust and partisan debate and misleading use of statistics.
Fact checkers and other commentators are helped enormously if the statistical offices communicate their findings well and make background data and methodologies openly available. Such actions also mitigate the risk of misuse. The job of the statistician is not completed when the results are known, only when they are effectively communicated to the intended audience in a way that that audience can readily understand.
Emphasis on communication inside the statistical office also helps the statisticians themselves to assess their results and avoid making claims that do not stand up. The communication of uncertainty is especially important as is avoiding overstepping from description of what has been found to advocacy about what should be done about it.
What more needs to be done?
When we think about statistical leadership and capacity building in the current context it is essential to consider these outward facing and political aspects as well as technical and professional considerations. In the new data ecosystem, the system of official statistics plays a critical role in building partnerships and the social licence for the use of data to serve the public good. Inevitably this draws the statistical office into questions related to the harms of misuse.
Accordingly, there is more to be done by most statistical offices to make a contribution to national data strategies and more to be done by national governments to invite statistical offices to bring their invaluable knowledge and experience to the table. The special role of national statistical offices in data stewardship is being increasingly recognised but so far remains patchy. Similarly, the idea of statistical regulation at the national scale is still quite new and the effectiveness of codes of practice is uneven. In most countries there remains a need to strengthen and update statistical laws to recognise the new data landscape and prepare for whatever future further changes we can expect. At the same time there is a need for leadership in statistical offices with the ability to be confident and capable in the face of these new challenges.
In a globally interconnected system, parallel changes are needed in the architecture of the international agencies involved in the standards and governance of official statistics. There has been serious discussion of this issue before. The 2013 Oxford Martin Commission for Future Generations [17] called for the establishment of WorldStat, a body that would “improve the availability and reliability of global statistics” and specifically “call attention to the misuse of statistics” [18]. Improvements to statistical governance have also been discussed by the United Nations Statistical Commission but whilst there has been lots of goodwill, responses have been limited.
Conclusion
Misuse of statistics is an abuse of power. The threats are wide-ranging and have been changing fast in recent years as the data revolution has gathered pace. A good range of safeguards are in place and remain relevant but need further bolstering to be robust to the face of the changes taking place.
Pali Lehohla has called for us to “wake up and smell the coffee” [19]. There is unfinished business. A strengthened data ecosystem that does not tolerate misuse of statistics is needed at the national and global levels if the public good that could be generated is not to be seriously undermined. We should cheer on those who produce statistics that serve the public good and call shame on those who suppress, distort, manipulate and misuse numbers to mislead, cover up and divert attention from what is really going on.
There is an opportunity to give fresh impetus to efforts to implement a system of governance for statistics serving the public good that will serve the citizens of all nations. Now is the time to grasp that opportunity.