Abstract
Algorithmic decision-making is neither a recent phenomenon nor one necessarily associated with artificial intelligence (AI), though advances in AI are increasingly resulting in what were heretofore human decisions being taken over by, or becoming dependent on, algorithms and technologies like machine learning. Such developments promise many potential benefits, but are not without certain risks. These risks are not always well understood. It is not just a question of machines making mistakes; it is the embedding of values, biases and prejudices in software which can discriminate against both individuals and groups in society. Such biases are often hard either to detect or prove, particularly where there are problems with transparency and accountability and where such systems are outsourced to the private sector. Consequently, being able to detect and categorise these risks is essential in order to develop a systematic and calibrated response. This paper proposes a simple taxonomy of decision-making algorithms in the public sector and uses this to build a risk management framework with a number of components including an accountability structure and regulatory governance. This framework is designed to assist scholars and practitioners interested in ensuring structured accountability and legal regulation of AI in the public sphere.
Introduction
This paper is concerned with a specific challenge facing contemporary society namely managing the risks that arise from the use of computer systems for supporting or making decisions in the public sector. This is not a new risk, but it is one that is being amplified by the increasing use of artificial intelligence (AI) in such decision-making. However, while much of the focus in this paper is on AI, we use the term “algorithm” rather than AI. This requires a brief explanation. To do this it is first useful to set out what is understood by AI and then briefly discuss the nature of algorithms.
Artificial intelligence is a broad church encompassing, in the view of some, everything from spell checkers to the self-aware computers of science fiction such as HAL in 2001 – A Space Odyssey or 3CPO in Star Wars to name but two. The EU High Level Expert Group on Artificial Intelligence (2019) has sought to capture that breadth, proposing the following definition:
“Artificial intelligence (AI) systems are software (and possibly also hardware) systems designed by humans that, given a complex goal, act in the physical or digital dimension by perceiving their environment through data acquisition, interpreting the collected structured or unstructured data, reasoning on the knowledge, or processing the information, derived from this data and deciding the best action(s) to take to achieve the given goal. AI systems can either use symbolic rules or learn a numeric model, and they can also adapt their behaviour by analysing how the environment is affected by their previous actions. As a scientific discipline, AI includes several approaches and techniques, such as machine learning (of which deep learning and reinforcement learning are specific examples), machine reasoning (which includes planning, scheduling, knowledge representation and reasoning, search, and optimization), and robotics (which includes control, perception, sensors and actuators, as well as the integration of all other techniques into cyber-physical systems).” (EUHLEG 2019, p. 5)
A notable feature of AI is that its impact has been steadily spreading and deepening over a long period of time. Some technologies such as, say, social networking or the World Wide Web, emerged and became an important part of our social fabric in less than a decade. Others, such as the Internet (invented in 1969) and AI have been burrowing their way into people’s lives for 50 years or more. Many developments in so-called weak AI such as expert systems or voice recognition, both of which were developed and improved over a long period, were once crude and unreliable, but are now sophisticated and widely adopted. More recent developments include facial recognition and machine learning.
Not all algorithms qualify as forms of AI. An algorithm is nothing more than series of steps designed to achieve a particular outcome. It is neither necessarily mathematical nor computer based. Amongst their many applications, algorithms are used to solve problems, to optimise, to carry out daily tasks and to make predictions. However, almost all non-trivial algorithms involve choices/decision-making at some level. From deciding when to lower the flaps on an aircraft to determining whose tax returns should be audited, computer systems across the world make vast numbers of decisions every minute. Almost all of these decisions are uncontroversial and/or do not have life changing implications for individuals. But some do. Of particular interest are complex and difficult to understand algorithms that make or recommend decisions that have material implications or even life-changing consequences for individuals, groups and/or societies. The danger of faulty algorithms was vividly and tragically illustrated by the crashes of two Boeing 737 800 Max aircraft in 2018 and 2019. Much less tragic errors, but errors with other forms of undesirable consequences for individual citizens, happen every day as a result of problems embedded in software. What is possibly even more worrying is that some of these problems may not be bugs, but features, i.e. policies and/or prejudices consciously or unconsciously built into the code and/or problems caused by biases in the underlying data that they use (Ntoutsi et al., 2020).
Consequently, while, as AI based algorithmic decision-making becomes more and more widespread, the focus of concern going forward has to be on this technology, it is necessary to consider all decision-making algorithms. Algorithms exist on a risk spectrum; they range from the insignificant to the momentous. We need to learn how to recognise, for any given algorithmic decision-making process, not just its benefits, but its potential risks and how to calibrate and manage such risks. We need to learn how to control algorithms before they come to control us any more than they already do. This paper will propose structures and mechanisms which can help ensure that we remain in control. While the word “algorithm” is the word most often used in this paper, the term “artificial intelligence” or AI will be used throughout this paper to refer specifically to its application to decision-making of the type being examined.
Literature review
Problems with AI and decision making
Concern about the potentially negative impacts of artificial intelligence have long been a staple of science fiction, both in literature and in the cinema. Many of these works have been prescient. To take just one example, Primo Levi (1966) in one of his amusing short science fiction stories, Pieno Impeigo (Full Employment), describes the development of a machine that today we would call a search engine and briefly speculates on its implications. Academic research on this topic has, however, been more scarce largely because, while writers of fiction can indulge freely in speculation about the future, scholars have to be more measured – particularly when it comes to predicting the impacts of a new technology and the risks that such technologies may bring with them.
One of the first researchers to conceptualise risk in terms of its dimensions was Cunningham (1967). The distinct dimensions that he proposed were uncertainty and consequences – concepts today widely used in the form of expected outcomes. Both of these dimensions are salient when considering AI, as the many uncertainties regarding the potential impact AI on society are increasingly juxtaposed with an awareness that the consequences of unbridled AI could wreak havoc on individuals and on society more generally (see, for example, Bostrom, 2014). Away from such dystopian discourses, there are many practical risks arising from the use of AI in day to day operations that need to be managed (as would be the case with any new technology). As will be shown below, there is growing evidence of AI-related damage occurring, albeit for now only in a localised way. Rather than assuming a perspective that enumerates only potential positive or only negative AI scenarios and outcomes, a more realistic position is to acknowledge the uncertainty that exists and conceptualise the risks of AI as a set of potential trade-offs that need to be carefully evaluated and managed.
The work of Floridi et al. (2018) is particularly valuable in this regard. These authors identify four types of risk associated with AI that are relevant to society and consider how we can balance the opportunities that AI offers against the potential risks that it presents.
The first relates to the enabling of human self-realisation without devaluing human abilities. Floridi et al. point out that although smart automation may free us from the tedium of monotonous tasks (just as did invention of the washing machine) and allow us to spend our time more intelligently, the risks that arise are not just those of making older skills obsolete with the emergence of new ones, but rather the pace at which this is happening and the resulting unequal distribution of costs and benefits. For example, employment serves economic, social and even cultural purposes in our society and the devaluation of old skills has the potential to disrupt the nature of employment at the level of both the individual and society. Beyond its obvious economic implications, jobs are central to an individual’s sense of personal identity and meaning, their self-esteem and their sense of self, whilst at a societal level de-skilling has potential to result in loss of human abilities and skills, thereby creating vulnerabilities in the event of AI malfunction. In this context, an extensive body of research has examined the effects of automation on the labour market (e.g. Manyika & Bughin, 2018; Acemoglu & Restrepo, 2017; Arntz et al., 2016; Karoly & Panis 2004), but much of that work has focused on robotic systems with limited intelligence which follow pre-programmed instructions to complete specific manual tasks. Emerging AI systems in contrast do not require such specific, bounded guidance and are capable of acting independently on insights obtained from data.
The second risk that Floridi et al. identify, one that is of particular relevance here, relates to the need to ensure enhancement of human agency without removing human responsibility. AI augments our intelligence in the same way as machines have previously augmented our abilities and outputs. However, they point to the need for responsibility in deciding what sort of AI we develop, how it is used and whether it should be shared with everyone. The question becomes to what degree do we abdicate responsibility for decision making to machines? The use of AI in parole decisions or creditworthiness decisions are just two of hundreds of possible examples. This question has a number of consequences that are discussed in subsequent sections.
In theory, an AI decision-making system can be objective or neutral (assuming that we can reach a consensus on what those terms mean – not a given by any means), but in most social decision-making contexts AI is almost never neutral and decisions made by complex algorithms are, almost by definition, never objective. To enable an algorithm to make a decision, values must be embedded in its code or indirectly in the data that it uses. Consequently, the use of AI in, for example, surveillance and military applications as well as use in social media and election result manipulation raises understandable concerns. As Manyika and Bughin (2018) note, AI has the potential to introduce new biases or perpetuate and institutionalise existing social and procedural biases. A recent report from the AI Now Institute (Myers West et al., 2019) found that the AI field, which is overwhelmingly white and male, is at risk of replicating historical biases and power imbalances. Because of its capabilities, AI is also at risk of amplifying biases and discriminatory practices. Examples of this abound, including use of image recognition services that make offensive classifications of minorities (as happened with Google Photos), chatbots that adopt hate speech (such as happened with Microsoft’s Twitter bot Tay and systems like Amazon’s facial recognition technology Rekognition which was marketed to law enforcement agencies, but has been shown to fail to recognise the gender of individuals with darker skin colours. According to Singer (2019) failures of this nature raise fundamental questions regarding whether facial recognition technology is suitable for use in important decision-making situations including job interviews, in drones and in police body cameras. Designed correctly, AI can amplify individual agency and strengthen moral systems, but as the Floridi et al. note, the degree of agency that we delegate to autonomous system is not a zero-sum game either pragmatically or ethically and it would be particularly unwise to ignore this fact.
The third risk they cite relates to the need to increase societal capabilities without reducing human control. AI presents a multitude of possibilities for augmenting and enhancing the collective capabilities of humans, assisting us to find solutions to enduring problems with significant individual and social impact, including early identification or even prevention of disease. But these authors note that it is precisely because of this, the fact that such technologies have the potential to be so powerful and disruptive, that they introduce proportionate risks. They argue that if we rely on AI to augment our own abilities in the wrong way, we may delegate important decisions to autonomous systems, systems that should remain at least partly subject to human supervision and choice. Worse, these potential harms could accumulate and become entrenched as more functions are delegated to machines. There is therefore a need to strike a balance between pursuing the opportunities offered by AI to improve human life on one hand and on the other hand ensuring that we remain in control of these developments and their effects
The final risk identified by Floridi et al. (2018) is how we can cultivate social cohesion without eroding human self-determination? In the face of current global societal problems such as climate change, terrorism, antimicrobial resistance and other similar problems, there is a need for high degrees of co-ordination between the stakeholders if these problems are to be successfully addressed.1
This article is being written during the middle of COVID-19 pandemic, but we have resisted the temptation to bring pandemics into the discussion.
An episode of the TV series Black Mirror (Nosedive) explored this theme.
Eggers and Beyer (2019) suggest that AI has potential to bring significant improvements in efficiency or cost savings to government by replacing existing decision-making processes and as the basis for policy determinations. The rationale behind this view is that government agencies are characterised by voluminous data and AI can both process and learn from these data much more efficiently than humans, thereby making it possible to tackle administrative problems in ways that were not previously possible and lowering costs. It is therefore understandable that, even outside of periods of austerity governments will seek to procure and deploy AI. Examples of current applications include:
Use of AI driven chatbots to assist patients with chronic conditions in the UK; Use of AI in Japan to analyse medical test results and recommend appropriate treatments; AI-enabled traffic lights in Pittsburgh; The use of analytics in Singapore to reduce traffic congestion; The use of chatbots to answer queries from human services case processing officers in Australia; The use of machine learning in the Netherlands to detect fraud in social benefit programmes; In Chicago the use of spatial algorithms on 911 call data to predict where and when violent crimes are most likely to happen; Use of AI to predict how disease will spread over time. Dynamic neural networks were used to predict spread of the Zika virus in 2015 and (at the time of writing) these were being re-trained using Covid-19 data.
These examples are only the tip of a large iceberg. These types of application are attractive at a number of levels, but there are potentially serious problems with the use of AI in many situations. For example, AI currently cannot generally take account of nuances that human agents may detect. Failure by machines to do this may have detrimental outcomes for many vulnerable populations including, for example, those requiring affordable housing, food assistance benefits or children at risk. The Nash report (2017) into the death of a child in Los Angeles found that an autonomous decision system failed to identify the child as being at risk. Instead, reflecting encoded bias, the system focused on poor children from working class communities, especially communities of colour. Similarly, automated essay scoring systems have been shown to encode bias against certain ethnic groups (Eubanks, 2018). These exacerbate existing inequalities in relation to further education and employment opportunities.
The literature on the impact of algorithmic decision making on people, organizations and societies is relatively new, but growing rapidly. In recent years, a number of books have explored this phenomenon. One influential contributor to this debate is O’Neil (2016). O’Neil uses the word “model” rather than algorithm. The words do not have the same meaning, but in this context they can be considered as interchangeable. “Models”, writes O’Neill, “are opinions embedded in mathematics” (p. 12). Although her focus is on the private sector, she also argues that the algorithms built into models and the data that they use can create biases and distortions in public policy. For example, she points out how algorithms can be biased against certain types of college or job applicant. Using other examples O’Neil discusses how machine learning systems that use historical data can learn the wrong things and end up embedding the prejudices, or even the now illegal practices, of an earlier era. A further shortcoming of some machine learning systems is that there are no feedback mechanisms that tell them whether the decisions that they made were good or bad. Even with supervised learning, there is a risk that any biases the supervisor has will be absorbed by the machine. Consider a system which recommends whether or not to hire somebody for a job (such systems exist – see Stanojevic (2020)). If a poor applicant is accepted, then there is at least the possibility of feedback that the decision was not a good one because the employee’s performance will be monitored and that performance can in theory (if not necessarily always in practice) be fed back to the software. However should the computer reject a potentially outstanding applicant, the system will never know. As a consequence, it never learns from this type of mistake.
A further problem is that it is often impossible to question or contest the decisions made by certain systems. This particularly the case when, as Whittaker et al. (2018) note, the systems are developed by private vendors and are not subject to public testing, review or auditing to ensure their fitness for a domain. Such systems may be built without explicit public oversight or accountability and are often opaque, a problem that makes discovery of erroneous automated outcomes more difficult. In support of this fact, Whittaker et al. reference the use of a third-party automated decision-making system by Houston Independent School District that used student test data to make decisions regarding teacher employment, promotions and terminations. It transpired that no employee in the School District was able to explain the determinations made by the system despite legal attempts to make them do so. The vendor refused to provide any access to enable experts to determine how the system worked. This type of lack of transparency and explicability, the growing accountability deficit, favours those who create the decision-making technologies at the expense of those who are affected.
Such cases highlight the need for greater transparency, oversight, and accountability in the adoption, development, and implementation of automated decision-making systems. They also highlight the need to reflect on the economic and policy factors that underpin their usage by government, specifically whether their usage is for the benefit of society. Whittaker et al. propose that automated decision systems fairness research needs to be expanded beyond a focus on mathematical parity and statistical fairness towards issues of justice, a view that we strongly endorse. Finally, it is worth noting that this problem is now widely recognised. For example, the EUHIEG Guidelines on Trustworthy AI (EUHLIG 2019) propose seven key requirements that AI needs to meet namely:
Human agency and oversight; Technical robustness and safety; Privacy and data governance; Transparency; Diversity, non-discrimination and fairness; Societal and environmental well-being; Accountability.
In considering each of these areas, the question of risk is important.
In summary, when it comes to the use of AI in public decision making, significant concerns remain regarding lack of due process, accountability, community engagement and auditing. Whittaker et al. (2018) reference another case where an individual in Arkansas in the United States was assessed for disability carer supports. The initial nurse-led assessment in 2008 determined that the woman in question required 56 hours of home care help per week. However, in 2016 the automated decision system that was now employed in Arkansas to make these assessments, (using a proprietary algorithm) calculated that the same woman only required 32 hours of carer support per week, a drop in the level of care that had a detrimental impact on her quality of life. By the time use of this algorithmic allocation programme was subsequently successfully subjected to legal challenge and determined to be erroneous and unconstitutional, irreparable damage to the lives of many citizens had already been done.
From the perspective of this paper, a particularly powerful and relevant critique is provided by Eubanks (2018).3
There is a talk by Virginia Eubanks available on YouTube in which she elaborates on the ideas in her book at:
Decision making algorithms have many uses and many advantages. Indeed, such algorithms are at the heart of what has been called digital-era governance (Margetts & Dunleavy, 2013) and similar ideas about what Williamson calls knowing public services (Williamson, 2014). However, as ever more decisions are delegated to, or influenced by, machines, so grows the need for systems of regulating, controlling and monitoring those machines. Leadbeater (2011, p. 18) describes a system of “government by algorithm” that uses “systems to mine and analyse data, to make automated decisions about allocating resources.” Demos researchers Wind-Cowie and Lekhi (2012, p. 63) argue that big data:
“…should be viewed as a transformative agent that has the potential to revitalise, reinvigorate and renew public services.”
As Williamson (2014) puts it (p. 18):
“Algorithms that can be taught to anticipate and predict how people act, or machine learning, have been positioned as a major component of the new techniques of governing promoted by these think tankers and intermediaries.”
On the other hand, these government databases do not merely represent the individual. Instead, as they continually join together different pieces of data, they constantly remake the identity of the individual, anticipate likely future behaviours and requirements, and continually recalculate decisions about the best lines of intervention. As goes the private sector (Zubhoff, 2019), so goes the public (Fredridksson et al., 2017; Klievink et al., 2017), although the public sector’s use of big data is more likely to be constrained by legal requirements (such as the General Data Protection Regulation in Europe). The individual is a composite enactment made up by software out of data rather than a fixed and stable identity. Williamson (2015, p. 100) draws attention to the point made by Ruppert (2013) that:
“…such software systems or ‘algorithm machines’ do not merely implement a policy or programme but are generative of both their subjects of governing and modes of intervention” (Ruppert 2013: p. 100.).
The literature on AI or algorithmic decision making in complex public sector problems is still embryonic, but a number of clear messages emerge from it. Six important points to emerge from the current literature which are important in the context of this paper are:
Faced with complex human or social problems, it is almost impossible for AI decision-making systems to be objective. Feedback mechanisms are often absent or partial, so systems do not learn or alternatively may learn the wrong things because they use historical data which may not be relevant to the current situation. Values need to be embedded in the code of many systems which means that existing human biases and values can also become embedded in the code – sometimes consciously, but also subconsciously. There are significant negative and damaging consequences for people/citizens currently occurring as a result of the use of machine decision-making. There is a large incentive to use such systems to displace more expensive human decision makers. In addition to problems with equity and fairness, there are problems with transparency, explicability and accountability, particularly when such systems are outsourced to the private sector.
To this we would add, as already noted that the risks are not yet always well understood. Addressing these problems is not easy. In the following sections we put forward a framework and some ideas about how to limit the potential damage caused by such systems.
We first propose a framework for categorising decision-making algorithms based on two elements:
A two by two typology; A three-dimensional risk spectrum.
Both of these are described in the following sections.
All algorithms can first be classified by whether or not they use subjective data. This statement immediately raises the question: what do we mean by ‘subjective’ data?
All data used by algorithms, whatever its source or whatever form or format it takes, can be classified as objective or subjective. This has parallels with what is generally referred to in the decision support literature as “hard” and “soft” data. As Saaty (2000) puts it hard measurement must be consistent with what is known in nature, whereas soft measurement must be consistent with what is known in the mind. Of course much data depend on scales and categories created by humans, but the distinction we will use is that objective data is not dependent on, or arrived at using human judgement be that of designer programmer or (with some qualifications – see below) user.
An objection to this definition is to claim that all data involves a degree of subjectivity (an argument that has parallels in the longstanding debate about constructivist and positivist research). Almost everything physical that we measure, height, weight, temperature, speed and so on is based on scales or criteria designed by humans and some of these scales are only analogues for more complex phenomena (temperature being a good example). Nonetheless we argue that there is a tractable difference between such data and personal or social or psychological variables such as happiness, poverty, tolerance, risk aversion and so on. In fact, many of the latter are constructs rather than variables in the formal sense and can only be measured indirectly or by proxy. They are much more complicated to measure and dependent on scales that are typically fuzzy and often arbitrary. We argue that not only is this a valid distinction, but that, for all practical purposes, it is binary.
There is, to take one example, a clear distinction between, say, the airspeed of an aircraft as measured by a sensor and, say, a police officer’s assessment, in a bail application, of a suspect’s flight risk. This is not just a distinction between something measured by an instrument and something “measured” by a human. Headcounts are objective data even when the counting is done by a person, but a headcount is fundamentally different from data based on an opinion or a judgement, no matter how informed that opinion may be. A simple test is whether presented with the same environment or set of circumstances, a properly calibrated measuring device will always give exactly the same answer. In this context, how the scale is arrived at is unimportant. There is a categorical difference between measuring a patient’s body temperature and asking a patient to rank her pain on a scale of one to ten. Most of the data collected by sensors fall into the objective category, but so do other data such as (as already mentioned) counts of various sort or financial data such as cash paid. Subjective data encompasses a wide range of types including constructs, assessments, value judgements, human estimates, opinions and so on. It should be noted that the term “data” is not confined to data that is input by the user when an algorithm is run, it can be, and frequently is, embedded in the algorithm itself.
We acknowledge that concept of objective data is not problem free. A specific problem in AI for example is unstructured data (such as a photograph or video). A photograph can be objective in the sense meant here if it does not directly involve human judgement, but an AI system seeking to make “sense” of a photograph may derive, for example, from supervised learning that may have been be biased to seek out particular features. Of particular relevance in this context is facial recognition.
Secondly, algorithms can be passive in the sense that they solely inform human decision-making or active in the sense that machines make decisions whose consequences are enacted without further human review or involvement or approval. Historically, most decision-making algorithms have been passive. This is what is meant by the term “decision support system” – in itself a large field of both academic research and commercial products (Power, 2008).4
A search for “Decision Support Systems” in Google Scholar yields over 750,000 hits.
The above analysis leads to the two by two framework illustrated in Fig. 1.
Classification of algorithms.
The following are some illustrative examples of each classification. Further examples and their implications are discussed elsewhere below.
A simple example of an objective/passive algorithm used in our country is one to compute a citizen’s pay related pension entitlement. This is based on a number of factors, primarily on the number and type of “stamps” that a person’s various employers have paid for during their working life. There are also technical adjustments that can be made in certain circumstances. The computation is based on purely objective data with no human judgements involved. The rules are explicit. This is a simple example, but public administration is replete with such algorithms. Much of management science/operations research falls into this category e.g. techniques like linear programming or the North-West corner algorithm. While the outputs or results of such algorithms can be implemented automatically, they are typically inputs into human decisions.
Objective/active (OA)
Objective/Active algorithms are widely used in a variety of areas including the operation of machinery. The example of the airspeed sensor given above is just one of numerous sensors on which modern aircraft rely. In normal conditions, most modern commercial aircraft can, for example, land without any input from or action on the part of the pilots. This process depends on numerous algorithms and sensors as well as data transmitted to the aircraft from systems on the runway and air traffic control. Algorithms of this type are found everywhere. A lift/elevator will not move if the weight of the passengers is greater than its certification permits; a police camera will take a photograph only if a vehicle exceeds a certain speed an automatic car will change gear when appropriate and so on.
Subjective/passive (SP)
Subjective passive algorithms are based on human judgements or decisions which may be quite arbitrary or, in some cases, deliberately biased. Consider a Google search. The results of a search request returned by Google are determined by a whole host of factors built into the ranking algorithm – something that has been decided by Google’s engineers. When the algorithm is changed, ‘before and after’ searches may present a different ordering of results or even present different results entirely. Google changes its search engine all the time, 3,234 times in 2018 alone (Moz, 2019). Search algorithms are passive because they pass their output back to the user and how the user uses that information (for example to book a hotel, to buy a product, to find out how to get somewhere, etc.) is up to the user. Most expert systems fall into this category. Recent discussion of medical diagnostics being undertaken by computers has provided evidence that computers are already more accurate than most human doctors in the diagnosis of certain illnesses such as skin cancer (Budd, 2019). So far however, use of such systems requires presenting human doctors with the diagnostic findings and allowing the doctors to take it from there. This will typically take the form of various possible diagnoses with probabilities attached. Basic forms of such expert systems are available on the Web. WebMD is one such example. Such systems are becoming steadily more sophisticated.5
And the bane of physicians lives as more and more patients arrive in surgery convinced that they know what their problem is.
While all four types of algorithm are of concern, the fourth category, subjective/active is the most significant from the point of view of risk in public administration. This is where a computer makes a decision based on subjective data supplied by humans and executes a policy decision based on those data. These data often include criteria, thresholds or scores that the system uses to make a decision and to act accordingly. A good example is so-called job pre-screening. When you apply for a job in certain companies today, they will ask you to go online and fill in a form and/or questionnaire. Based on your answers, you may, or may not, be called for an interview. Many such systems today also gather data not supplied by the candidate directly, but extracted from social media and other on-line sources. This may not only be used to rank a candidate, but to decide whether the candidate is a good “fit’ for the job (Berkelaar & Buzzanell, 2014). Such algorithms are being used in situations where the consequences are more serious, for example in determining whether a child should be taken into care or the length of a prison sentence. They are also being developed for complex judgement tasks such as examination marking of textual answers (Ab Aziz et al., 2009; Shermis & Burstein, 2013; Nwosu, 2017; RM Results, 2018).
All of the four types have certain risks associated with them though those associated with objective/passive algorithms are probably fairly modest. In section four we examine some of the risks that the other three types, and in particular SA algorithms present in a public sector context. Before doing that, we first consider the risk spectrum.
A three-dimensional risk metric
In considering the risk(s) presented by an algorithm, three dimensions need to be considered.
The first is transparency. Transparency has long been the subject of much discussion (Bannister & Connolly, 2011; Grimmelikhuijsen, 2009). Transparency can refer to data or to process or in the case of algorithms, to both. Meaningful transparency in this context is the ability to see the code/steps in the algorithm so that that people can understand how does its job and how it arrives at its conclusion and/or its decision(s). A critical distinction needs to be made between the ability to see what an algorithm is doing and the ability to understand or explain what it is doing (see below).
Google’s page ranking algorithm is a good case in point. What determines the order of the results that you see when you do a Google search? Google does not disclose this although it makes certain information available. Obviously, the main reason Google does not publish details of the algorithm is that the Page Rank algorithm is proprietary and valuable commercial property. Another reason is that it wishes to prevent web site owners gaming the system (a common practice in the past and still tried today). A public system is, or should be, different and yet public bodies have refused, often for what seem like spurious reasons, to publish the code in the algorithms that they use (Carlson, 2017; Whittaker et al., 2018; Eubanks, 2018). It is also important that the code or logic underlying the algorithm be made available in a readable form. The more opaque an algorithm, the higher the risk of unforeseen consequences. Unfortunately some algorithms are black boxes where those affected have no way of knowing how a decision or recommendation is reached. In extreme cases, the authors of the algorithm may not even know (Veale et al., 2018).6
An interesting example is the Irish Tax Authorities who, when they started implementing an entirely new computer system in the 1990s, had to leave one part of the system on legacy machines because nobody understood the code and trying to re-write it was too risky. Eventually, not only did nobody understand the code, nobody still working in the Authority was familiar with the tax laws encoded in the system (though the legislation was still on the books).
A second criterion is comprehensibility. Users (or experts acting on their behalf) should be able to understand what an algorithm does and explain how it arrived at its decision. Transparency is of limited value if the code is readable, but indecipherable. There is an important distinction between the algorithms used inside machines, those used in business and those used in the public sector. For example, in the case of avionics or diagnostic machines it is not expected that users (pilots or doctors) will understand the computer code, but they need to understand what the system does and how it behaves in different conditions. To ensure that such systems behave properly and as expected, organisations like the Federal Aviation Authority and the Federal Drugs Administration and their counterparts in other countries certify these systems.7
It is worth noting that the delegation of this role by some authorities to companies has been much criticised, most recently in the case of Boeing, but reliance on self-regulation was a factor in the lead up to the banking crisis in 2007/8.
The third criterion is impact. The impact or influence of an algorithm on individuals, groups or even societies varies from unimportant to (literally) existential. Negative impacts range from being a source of minor irritation to (again, literally) life-endangering. Table 1 contains is a list of examples of impacts roughly ranked from the marginal to the highly significant.
Range of risk
If one goes to a government website and searches for a particular service or document, then the efficiency and effectiveness of that search does matter, but the order in which search results are presented is rarely of importance. At the other extreme, where algorithms are used to target specific electoral messages to certain voters, there is a potential for major distortions to democracy.
Defining a scale of this nature will require some thought and debate. Similar scales are used in other areas where direct measurement is not meaningful. Two well-known examples are subjective pain scales (Pesudovs & Noble, 2005) or the hunger scale (Derby Community Heath Service 2020). Table 2 is not proposed as a definitive scale, but is indicative of what such a scale might look like.
Outline possible structure for a risk scale
Governance has been a topic of growing interest for quite some time and, as is seen all too frequently, where governance is poor or absent it almost invariably results in undesirable consequences. The use of decision-making algorithms (and AI generally) requires proper governance and this topic has recently started to attract academic attention in the legal and philosophical literature (Coglianese & Lehr, 2018; Lepri et al., 2018; Fink, 2018; Valentine, 2019).
There are three components necessary for such governance:
Risk assessment and management Accountability and Regulation.
We discuss each of these in turn.
The framework whose elements have been described in the preceding section can be used to assess the risk associated with a given algorithm or AI system. Risk is assessed under four headings:
Type of algorithm; Transparency; Comprehensibility; and Impact.
The type of algorithm affects risk. OP algorithms, while not always risk free, are generally low risk. Such algorithms can be faulty like any other, but subjective bias will only occur when the user’s own internal bias(es) affects a decision on the basis of what the algorithm has told the user, though there is an increased risk when a bias in the machine coincides with a user’s own bias.
OA algorithms are somewhat higher risk, but the risk lies in possible design and programming flaws rather than in built in bias.
SP algorithms are higher risk again as they depend both on the risk of internal bias within the machine and on the human user’s ability to use what the machine tells him or her effectively and to use it to inform, but not override his or her own judgement. A system recommendation of a sentence for somebody found guilty can be modified by a judge for example. The difficulty is that many humans may be reluctant to override a machine recommendation. One reason for this may be legal or personal exposure; it can be safer to hide behind the algorithm. An analogous problem is over-testing in medicine. Some doctors order large numbers of tests, not because they are necessary for diagnosis or action, but because of the fear that if something goes wrong, they will be sued for having failed to explore every possibility. By relying on the computer, responsibility can be transferred to the computer and by implication to the system designers and coders. Where, as seen in the literature review, the system provider is a private company that refuses to disclose details of how its decision-making algorithms work, then the responsibility falls into a black hole – similar to the excuse that is often used in the public sector of ‘systemic failure’ where the avoidance of individual blame is key.
SA algorithms present the greatest risk. In such systems, the decision is entirely abdicated to the machine.
It would not be difficult to develop a scoring system based on these, but it would be better if expert actors looked at a given decision-making system under each of these headings and assessed risk on a spectrum. A black box system based on subjective scoring which can affect people’s lives in fundamental ways would be high risk; a support system which offers powerful ‘what if’ type exploration of the potential consequences of different decisions is likely to be lower risk (though not risk free).
One does not have to go far to find systems that, by any standards, are medium to high risk. One case which has been in the news recently is the UK Home Office’s system for processing visa applications (Warrell, 2019). This is a SP type system; the algorithm streams and ranks applicants into red, amber and green according to their level of perceived risk and then passes this information to humans to decide whether or not to approve the application. The Home Office has refused to disclose details of which factors are used to assess this risk or how they are weighted. Such an algorithm is ineluctably based on a large amount of subjective judgement as, of its nature, statistical data on such risks will be incomplete. Commenting on the system, Christine Blacklaws, president of the British Law Society observed that such a system:
“…may well disadvantage certain groups of people based on generic markers such as age, country of origin or whether they have travelled before”.
But the public does not know. This raises the question; should the public know? There are circumstances when transparency is clearly not in the public interest (Moon & Adams, 2015). In this case, transparency might enable people to game the system. This possibility therefore has to be factored into any risk assessment.
In other words, just because an algorithm is an impenetrable black-box or has its code available on-line does not, of itself, make the system high or low risk. It depends on the purpose of the algorithm – if an error in such a system does not have major consequences the risk might be ranked low. Nevertheless, while risk is determined by a combination of factors, use of opaque algorithms is, with rare exceptions, not a good policy.
Algorithms require policing (Veale et al., 2018) and that means that there needs to be an accountability framework. Accountability can be a problematic concept at the best of times (Dubnick, 2002). It is based on the principle that individuals and/or organisations are answerable for their actions and that failure or bad behaviour will be identified, exposed, where possible corrected and, where appropriate, punished. In practice, organizations frequently seek to hide problems, are reluctant to expose themselves to full scrutiny even when problems are identified and are adept at avoiding punishments that should be imposed on those responsible. True accountability is often lacking in the public sector (Guerin, 2018) and even where it exists, sanctioning of the individuals responsible for failure frequently does not happen. Politicians often suffer as a result of administrative failures over which they have little or no control or influence while those actually culpable for such failures hide behind the shield of public service anonymity
Ignoring for the moment the many real-world shortcomings of accountability in practice, accountability means, in principle at least, that some person or persons is answerable for a decision and if that decision is bad then sanctions can be applied to a human being. This may range from public criticism to demotion, firing, a fine or, in extreme cases, a jail sentence (or, in the case of politicians, having to resign or losing their seat at the next election).
In any bureaucracy, the standard way to avoid personal sanction is to blame the system. The term ‘systemic failure’ has already been mentioned and is a commonplace in public life where numerous failures of public policy or practice have successfully been blamed on the ‘the system’ by individual public servants and managers (Anderson, 2009; O’Brien, 2012; Leon & Orriols, 2019). In some countries, it is common for government ministers or senior managers to resign and take the blame for failure on the part of their subordinates. In others it is rare, although it occasionally does happen (Bardon, 2017).
But what happens when the failure or scandal is caused by an algorithm? In such circumstances, the term ‘systemic failure’ acquires new possibilities. Who should carry the can? Consider the Boeing 800 Max disasters. It was a software failure (or a combination of software and hardware failures) that led to the two tragedies. If we want to attribute blame for this to a human (or humans), who should that be? There is a long list of candidates including:
The Federal Aviation Authority (who delegated testing to Boeing); Boeing senior management (for deciding to rely on software to fix an inherently unstable configuration rather than re-designing the aircraft); The aircraft’s designers (who, inter alia, specified only one speed sensor creating a single point of failure); The managers who decided not to have a second sensor in order to reduce costs; The software designers; The programmers who wrote the code; The project managers who failed to test the system adequately; The people in Boeing who failed to put in place proper training for the pilots.
There are undoubtedly others. The only people who were not responsible were the pilots (who of course were the first to be blamed by Boeing).8
It is worth noting that, at the time or writing, nobody in Boeing has resigned as a result of these disasters. One senior executive has ‘retired’ (
It is not hard to translate this catalogue of possible people responsible into any one of a number of public service contexts.
One way of dealing with this would be to define a clear structure of what might be called responsibility points for any public sector decision-making system. At the moment, the single point of accountability is usually a politician (or a carefully chosen scapegoat – often well down the line). Responsibility points would assign both responsibility and accountability and clarify the borders/parameters of that responsibility for everyone concerned with a decision-making system and with the decisions it makes. A possible structure might include:
Those who determine policy; Those who specify the requirements to implement the policy; Project owners; Project managers; Designers; Coders/programmers; Testers; Managers/administrators; Final Decision makers;9 There will, of course, be decisions made by people in all of these groups. People should be responsible for the decisions that they make, but in a proper system, to adapt the words of Harry Truman, the buck has to stop with one person or maybe group.
Front line staff.
A critical point is that there should be a person or persons responsible or answerable for every decision and for every stage in the process that leads to the final decision. Furthermore, it should never be the case where blame for failure cannot be pinned on one or more people, i.e. there must be no gaps in the accountability chain. Where a human makes a decision, whether based on machine ‘advice’ or not, the decision maker is identifiable. However where a machine makes the decision, then if a decision turns out to be wrong or harmful, there should be an clear audit trail back to the person or persons responsible for this whether it be the politician who signed off on the policy or the programmer who wrote the code.
It is essential that it be clear from the outset where responsibility lies. An analyst or programmer cannot be expected to be aware of potential nuances in a wicked social problem or even a relatively simple problem like visa approval. The responsibility of the programmer is to code whatever the specification specifies. Furthermore, it needs to be clear that there are real sanctions for errors that are appropriate to the nature of the error and its cause as well as allowing for the degree of discretion or authority that the person in question has. For example, if a system recommends a decision to a street level bureaucrat, can that bureaucrat reject the advice or seek more information or does she have no choice but to implement what the machine tells her? In addition, there need to be processes in place where humans can query and refer such decisions upward without fear of reprimand.
Finally, sanctions need to be appropriate up to and including criminal indictment in extreme cases.
The principles outlined are pragmatic and, if done properly, effective. Given human nature, they are also likely, like many attempts to introduce good governance, to encounter resistance, but they are essential if there is to be public trust in such systems and processes.
Thirdly we propose that there should be a regulatory agency responsible for approving all algorithms which affect people’s lives in a material way. This is simply the extension of a principle and praxis that applies in many other areas of life. No medical drug may be prescribed or sold without the approval of the appropriate authorities (e.g. in the USA, the Federal Drugs Agency, in the EU, the European Regulatory System for Medicines). The same is true for all sorts of other products from aircraft to zoos. Often such regulation is multi-layered. Drug regulators not only ensure that new drugs have been extensively tested, they also supervise their production and increasingly today the supply chain that delivers them to avoid forged or fake drugs entering the system.
What is needed is an agency to monitor and approve the use of algorithms, AI or not, in routine public decision making, say an Artificial Intelligence/Algorithm Regulatory Agency (AIRA). The remit of such an agency would be decision-making algorithms which have the potential to materially affect citizens, business or other organisations on a continuing basis. It would not include, for example, decision support systems or models built for one-off decisions. Such an agency could certify algorithms before they are used, certifying that:
They have designed and developed according to established software engineering standards; They are properly documented; They are appropriately transparent; An appropriate responsibility framework is in place; They have been thoroughly tested; Proper training courses are in place.
An AIRA would also have an appeal function whereby citizens, unhappy with a decision or the explanation for that decision or in the case of a dispute, could appeal a decision to the agency for arbitration.
In addition to risk assessment, accountability and regulation, there are other actions that should be considered. One is to mandate that all public sector decision-making software other than that related to areas of state security and policing, should be open source. Another possibility is a charter of rights with regard to machine decision-making. How does a citizen appeal against a decision made by a machine? These, and other possibilities, require further exploration.
In some ways, the problems presented by contemporary AI and algorithm-based decision-making are new; in others they are not. There have always been problems with (non-AI ) algorithms in “traditional” ICT systems; the history of e-government is littered with computer generated problems (and disasters). However, a key difference with AI is that it “learns” and this creates the possibility that it will learn the wrong things and/or that it will amplify errors in the original design. Existing faulty algorithms repeat the same mistakes. AI algorithms may have the capability to make new ones. AI and the drive to reduce costs and increasing efficiency in the public sector is going to make the use of this technology increasingly pervasive in the public sector and whatever the provenance of the problems that are ineluctably going to ensue, it is preferable that they should be anticipated rather than reacted to in retrospect.
Conclusion
Artificial intelligence (AI) takes many forms and it is not difficult to think of developments in AI that have or will have implications, and often major implications, for public administration, public governance and/or democracy as well as for individual citizens.
One manifestation of AI that is already generating extensive debate is the increasing use of algorithmic decision-making for implementing public policy in a wide range of areas including, for example, medicine, social work education justice and public housing. A growing number of scholars and commentators have expressed concerns not just about machines making bad decisions resulting from poor design or coding, but about more serious longterm risks such as inbuilt bias and the permanent embedding of existing social prejudices and power structures in software. Decisions may also be influenced by the subconscious prejudices of the individuals who formulate, clean and structure the data that is used in such systems.
At another level, there are questions about the ability of such systems to adjust to unforeseen circumstances or to be able to exercise the types of discretion that human actors often do when faced with complex and nuanced social situations. The absence of human factors such as sympathy, empathy, insight, experience and flexible thinking can lead to rigidity and potentially harmful consequences as machines work with hard coded (and what may be poorly understood) rules as well numeric ranking systems that try to reduce complex decisions to simple weights or scores.
Being able to detect and categorise these risks is essential in order to develop a systematic and calibrated response. In this paper we have proposed a risk management framework comprising a two-by-two typology and a three-dimensional risk spectrum that is designed to assist scholars and practitioners interested in calibrating and managing such risks in the absence of structured accountability and legal regulation of AI in the public sphere. Society needs to move fast on this. The speed of development of AI is accelerating and there may come a tipping point whereby control of AI becomes impossible. We are, just about, still at a stage where this can be controlled. We need to act now.