Abstract
The inability of current machines to expose biases induced by programmers and data scientists is leading towards the creation of a new religion, where machines are mystic oracles whose pronouncements have to be believed, and computer users are their servants.
This has to change.
In this paper we discuss the issues that can raise from biases introduced in autonomous systems, with specific care of the case of machine learning systems, and their impact on our society. In the light of the (current and future) exploitation of autonomous systems for law enforcement and war – fighting, we emphasise the importance of issues related to discrimination and safety. We also support the bold claim that artificial intelligence can help artificial intelligence in overcoming those issues: by enabling artificial intelligence to record every single step that lead to a given inference, and to argue with humans, we can unveil the mystic oracle and trust its services.
Introduction
Indisputably, Artificial Intelligence (AI) is already strongly affecting our lives and our society. AI approaches are exploited in a wide range of applications, such as: for understanding and driving consumer behaviour,1
This has to change.
Research has already been carried out for unveiling part of such a mystic oracle, thanks also to events like the Workshop on Human Interpretability in Machine Learning,5
Fundamental components of an autonomous agent: A kite-level picture.
Research like [5, 4] is clearly more than welcome, but it does not solve all the issues associated to using such a mystic oracle. As we will develop further in Section 2, the mystic oracle relies on various components, each of those plays a specific role, and may very well be affected by human biases or mischievous actions. For example, what would happen if an autonomous system for detecting suspicious activities, that might lead to searches and seizures by the police, is trained on a specific dataset where most of the training cases labelled as suspicious activity show a male, black individual? Would the autonomous system infer that the gender and the skin colour are relevant features? And what would happen if such a system becomes widely used in policing actions? Under the UK law, every individual can be stopped and searched if policemen have “reasonable grounds” for suspicious activities.7
Indeed, the goal of this paper is twofold: on the one hand, we want to raise the awareness of these issues within the ACM community and with the general public. On the other hand, we aim to discuss a reasonable, methodological pathway for addressing these issues. Our belief is that artificial intelligence can help artificial intelligence to overcome (some of) those issues, and we discuss it at length in Section 4. Finally, we conclude the paper with general discussions and conclusion in Section 5.
Diakopoulos [6] describes a generic autonomous agent on the basis of five main components: human involvement and accountability; data used as input; model for manipulating the input; inferencing and issues associated to accuracy; and algorithmic presence as the impact that might have on the ultimate user knowing that the given inference has been curated by an autonomous agent. Although these elements are relevant for many types of AI approaches, in this paper we will consider mostly the case of machine learning systems because of their massive diffusion and exploitation.
Formally, a computational system is said to learn from experience
Practically, machine learning approaches generate predictive models on the basis of experience gained by analysing a provided training data set. Training instances are described through some available observed characteristics (features). The training set is, by assumption, representative of the population on which the trained model will be exploited. There exists a wide range of techniques for generating predictive models – such decision trees, neural networks, support vector machines, etc. – but they all try to identify some sort of patterns in the values of the features of the given training examples, that allows to provide accurate predictions. It shall be noted that the type of features (e.g., continuous values, classes, etc.), as well as the type of prediction, has a strong impact on the algorithms for generating predictive models.
Figure 1 depicts the above mentioned five components introduced by Diakopoulos [6], and the interactions among them that we will consider in this paper. It is beyond doubt that, in machine learning approaches, a model is derived from data, and leads to a series of inferences. Moreover, human is usually – but not always – involved in the process of selecting relevant data/formalism to be used, as well as the general model to be used in a specific scenario, e.g. Bayesian network vs neural network, etc. Finally, algorithmic presence is clearly determined by the use or not of a given model.
The veiled oracle
With reference to Fig. 1, we argue that biases and mischievous actions can affect each of the five components. However, given the venue for this paper, we will abstain from discussing how biases can be originated in humans, therefore we will look more closely at the selection of data, the choice of model, the quality of inferencing, and at the effect of algorithmic presence.
Data and model
First of all, it is worth remember here that each autonomous system “is limited in making its predictions to analysis of the data within its dataset, and it cannot consider other facts that might be relevant but that were not included. In contrast, human beings are always at least potentially capable of including a new piece of relevant information in an analysis” [8].
Bearing in mind such caveat, Citron and Pasquale [1] discuss predictive algorithms assessing risks, desirable employees, reliable tenants, etc. However, they specifically highlight how datasets may contain “inaccurate and biased information.” Indeed, as noted by van Eijk [9], criminologists and legal scholars have criticised risk assessment tools for their potential bias against racial/ethnic minorities and women, which could result in sentencing disparities, e.g. [10, 11, 12]. As noticed by Skeem and Lowenkamp [12], criminal history mediates the relationship between race and future arrest.
Those are evidence of possible implicit discrimination, which can be of three types: masking, subconscious discriminatory motivations, and relying upon tainted datasets or tools [13]. The first type collects all mischievous activities aimed at making discriminatory practices in a way in which discrimination might prove undetectable or at least defensible [14].
With regards to the second type, Citron and Pasqua-le [1] highlight how “human beings […] biases and values are embedded into the […] predictive algorithms,” thus giving rise to subconscious discrimination. For instance, as reported by Friedman and Nissenbaum [15], the National Resident Match Program (NRMP).9
Finally, Zarksy [13] discusses at length how discriminatory outcomes are generated by “recklessly – or perhaps, mere negligently – relying upon tainted datasets.” It is beyond the scope of this paper to discuss the subtle distinction between reckless and negligence. However, this is perhaps the simplest case of bias a computer or data scientist encounters. Is the dataset representative enough? Are we applying a model violating the assumptions behind its training?
This naturally raises questions about accountability of the human involvement. Discussions around this theme now dominate chronicles, and examples include autonomous cars, and more dangerously autonomous weapons. On 28th July 2015, at the opening of the International Joint Conference on Artificial Intelligence in Buenos Aires, Argentina, an open letter was published10
When faced with the result of an algorithmic procedure, people often experience “automation bias”, which is “the tendency to trust an automated system, in spite of evidence that the system is unreliable, or wrong in a particular case” [17]. Moreover, Kirkpatrick [18] discusses legal implications that can arise when an autonomous system is used in risk assessment of a suspect. In July 2016 the Wisconsin Supreme Court sentenced that warnings must be attached to the scores to flag such a system’s “limitation and cautions”.12
The way Rich [8] analyses the result of an inference produced by an autonomous agent is to consider it at the same level of an informant, in that they are outside of a law enforcement and provide information about criminal activity. Informants can be divided in: (1) criminal informants; (2) anonymous tipsters; and (3) citizen-informants [19]. Criminal informants generally ask in exchange money or leniency. Anonymous tipsters do not disclose identifying information. Citizen-informants provide information by virtue of being the victim of or witness to a crime.
Rich [8] argues that an autonomous system would not seat in any of these three traditional categories. In one sense, autonomous systems’ designers are like citizen-informants. However, like anonymous informants, the designers almost certainly will not be subject to criminal prosecution if a prediction is wrong. Moreover, a designer is motivated (at least in part) by money.
Finally, even if a legal framework for handling information coming from an autonomous system was available, a more serious issue still stands, namely the difficulty that most humans have to fully understand uncertainty. For instance, Gigerenzer [20] discusses at length the fact that even physicians are often confused about the notions of sensitivity and specificity,
Our main claim is that most of – if not all – the issues highlighted in Section 3 can be addressed building on top of existing artificial intelligence techniques. At the same time, it is beyond the scope of this limited paper to show complete solutions to those issues. Rather, we choose to consider a sub-field in artificial intelligence, namely argumentation theory, and look at how it would help in addressing those issues.
Argumentation theory sits in the intersection of three major lines of research: informal logic, philosophy, and non-monotonic reasoning. Stephen Toulmin in his book The Uses of Argument criticised traditional studies in deductive reasoning and logic. This lead him to introduce a warrant for each line of reasoning allowing for rebutting such a line of reasoning when specific circumstances happened thus affecting the warranty. Other works on rebuttals were influential for John Pollock, who migrated from philosophical studies to artificial intelligence in order to formalise the intuitive concept of defeasible reasoning, now at the hearth of argumentation theory. Finally, argumentation theory borrows largely from the field of non-monotonic reasoning, especially for what concerns reasoning systems: the first systems were heavily based on logic programming.
The field is now start testing the claim that argumentation theory “captures naturally the way humans argue to justify their solutions to many social problems” [22]. Recentely, an argumentation-based system has been used to aggregating clinical evidence on lung cancer treatments [23], taking into account subjective criteria such as preferences over outcome indicators. Moreover, experiments with human participants are carried out, and projects are proposed to apply argumentation theories to a large variety of aspects related to human cognition. For instance, Framework for Computational Persuasion13
Since the oracle we want to unveil is, unquestionably, a logical one, let us focus on logical aspects of argumentation brilliantly summarised by Chesñevar et al. [24] in their survey paper to which we refer interested readers.
Formal logic of arguments emerged in the last 30 years as one style of formalising non-monotonic reasoning which differentiates from classical logic for its ability to deal with incomplete and potentially inconsistent information [25]. Among other founding fathers of argumentation theory, Pollock [26] moved his theory of defeasibily – originally used within epistemology to address questions of justification – into computer science, developing a theory of defeasible reasoning. In his view [26], reasoning proceeds by constructing arguments based on reasons that can be either conclusive – i.e. non defeasible – or prima facie – i.e. that create a presumption in favour of their conclusion but they can be defeated. Pollock was a strenuous supporter of statistical generalisation and thus, there should be little surprise in reading that “a rational agent must be equipped with rules (1) enabling it to form beliefs in statistical generalisations, and (2) enabling it to make inferences from those statistical generalization to beliefs about individual matters of fact” [26, p. 59].
Pollock’s work was very influential for a large part of the argumentation community, who started building rule-based systems incorporating the distinction between defeasible and strict, i.e. non defeasible, rules, e.g. [27] among others. Although a logical language is often used to express such rules, Walton et al. [28] considered the more informal concept of argumentation scheme as a reasoning pattern aimed at supporting the acceptance of a conclusion. This line of research benefited from Dung’s theory [22] that abstracts from the inner structure
Argumentation and data
From our perspective, issues highlighted in Section 3 with regards to data belong to two classes that can be summarised by two questions: are we using the right information? And, are we using quality data?
As per the first question, the issue is far from being settled. In more computational terms it can be rephrased as: what are the criteria of adequacy of logical formalisation of natural language statements [29]? In this respect, there are two main schools of thoughts, one promoting atomism, the other reflective equilibrium. The former one, advocated for instance in [30], suggests that the adequacy of a formalisation of a sentence does not depend on formalisation of other sentences. The latter, advocated for instance in [29], argues that formalisation is inherently a holistic endeavour. We believe that this question should be settled for the sake of artificial intelligence, leaving aside benefits for the philosophy of logic. Having clear criteria to use for testing the quality of formalisation and thus answering the question are we using the right information? is of primary importance.
The second question – i.e. are we using quality data – highlights the need of confidence in evidence-based reasoning.14
Unless alternative facts
It should follow an argumentation process constructing reasons for/against competing claims; Evidential arguments should increase/reduce confidence in claims;
Ceteris paribus, the more independent and sound arguments for a given claim, the greater our confidence in such a claim; A single argument can be conclusive for confirming or refuting a claim; Arguments and theories can themselves be questioned; Some arguments can be stronger than others; In the absence of information about relative stren-gth, contradictory arguments still play an important role in decision making; It is desirable to develop systems using sound, formal languages for argumentation but that can be translated to and from intuitive natural language interfaces; A rational agent can choose the hypothesis that has the greatest confidence among all the competing hypotheses, unless there are grounds to argue against such a confidence; A rational agent not forced to choose may defer a decision on the grounds that the arguments are unwarranted.
To this end, the CISpaces tool [32] is, to our knowledge, the most advanced attempt to satisfy most of those properties. It supports intelligence analysts in their sense-making activities exploiting argumentation schemes [28] for building arguments, thus satisfying the first property, as well as supporting critical thinking. Moreover, CISpaces allows analysts to inspect the provenance of data, recorded using the PROV – DM model,15
Letham et al. [5] present a machine learning system where the inferencing step is based on a decision list that is mined from data. This reflect, in spirit, where Holte argues that there are cases in which the accuracy of rules that classify examples on the basis of a single attribute is sometime higher than other more complex approaches [34]. Letham et al. [5] show that their Bayesian Rule Lists (BRL) aims at hitting “the sweet spot between predictive accuracy, intrepretability, and tractability” by providing (1) more accurate results than SVM, Random forests, and others systems, on the task of predicting stroke risk on real data; and (2) and interpretable model of the form of the one in Listing 1.
An example of Bayesian Rule Lists from [5, Fig. 3]. In parentheses is the 95% credible interval
if hemiplegia and age > 60
stroke risk 58.9% (53.8%-63.8%)
else if cerebrovascular disorder
stroke risk 47.8% (44.8%-50.7%)
else if …
An example of Bayesian Rule Lists from [5, Fig. 3]. In parentheses is the 95% credible interval
if hemiplegia and age > 60
stroke risk 58.9% (53.8%-63.8%)
else if cerebrovascular disorder
stroke risk 47.8% (44.8%-50.7%)
else if …
Although Listing 1 already provides a fairly understandable prediction model, there are two main issues that need to be addressed. The first one is to explain the meaning of those percentages, and to this aim we believe that the discussion on provenance we had in Section 4.2 is very relevant in this context as well. For instance, knowing the details of the chosen training dataset would be extremely relevant to understand whether the prediction can actually apply to the given case.
The second issue is how an expert can interact with the system in order, perhaps, to correct the mined model or to ask questions such as “what if
While we are not aware of any approach in argumentation aimed at addressing what – if analysis – an area that we believe should be definitely subject of future studies – argumentation community addressed other elements of interaction with machine learning systems. Notably, Možina et al. [35] combine machine learning with experts’ arguments or reasons for some of the learning examples thus constraining the combinatorial search among possible hypotheses. We believe that it would be very beneficial to further investigate connections with [5]. Moreover, in the context of the RoboCup, Gao and Toni [36] present a way for experts to provide knowledge in the form of heuristics to a system for reinforcement learning. It would also be clearly very beneficial to investigate further how this might link in the context of active learning [37] – i.e. where an active learner may question a human expert that already understands the nature of the problem.
To exemplify an inferencing case, let us consider the case that a Bayesian network – a directed acyclic graph where nodes represents stochastic variables, and arrows dependencies that can be specified by the means of conditional probabilities – has been used by an autonomous agent as a model to make predictions. That is, the posterior probability of a specific variable in a Bayesian network is derived on the basis of evidence. The question is how such an inferencing step can be explained to a human agent.
In a recent paper, Timmer et al. [38] discuss an approach to derive explanatory arguments from a Bayesian network. Pivotal in their work is the notion of a support graph constructed for a variable of interest that captures the support the variable of interest receives from the other variables. Then, once evidence is provided, the support graph is used to derive arguments that describe the logical steps needed to interpret the Bayesian judgement of the variable of interest.
Despite Timmer et al. [38] seem not to be aware of [5], it is almost immediate to draw parallels between the two approaches: the Bayesian Rule List derived using [5] seems to represent a very specific type of support graph. The main difference is that in [5] the inferencing step is provided by the Bayesian Rule List itself, rather than an underlying Bayesian network.
However, both [38] and [5] suffer from the lack of empirical evaluation of the quality of their explanations with human judges. This has been the subject of [39], where it is shown that Causal Explanation Tree [40] and Most Relevant Explanation [41] models provide better fits to human data. This is encouraging especially because, intuitively, Causal Explanation Trees seem expressive enough to capture Bayesian Rule Lists.
Argumentation and algorithmic presence
In Section 3.2 we reported that Rich [8] would consider autonomous agents at the same level of informants. Assuming that an autonomous agent can also provide a confidence interval (cf. for instance Listing 1), then this should be the case when the confidence in the answer is low.16
How to label a numeric value as low is not trivial. A confidence interval of 20%–30% might be low if we are contemplating the risk of rain in London; but it might be very high if it is the risk of a terrorist attack in New York.
For instance, we can build on top of Walton et al. [28] schemes for argument from witness testimony and from expert opinion to derive the following scheme for evidencing and questioning the algorithmic manipulation of a piece of information we received:
A is an autonomous system trained in subject domain S containing proposition P.
A asserts that proposition P is true (or false).
P is true (or false).
What are A’s maker interests?
Is A’s assertion internally consistent?
Is A training adequate to make a judgement about P?
Is the provenance of A’s judgement about P sound?
Is A’s assertion consistent with the known fact of the case (based on evidence independent from A)?
Is A’s assertion consistent with other, independent autonomous systems’ assertions?
In particular, to minimise the “automation bias” highlighted by Asaro [17] and to ensure accountability, we strongly believe that no information provided by an autonomous system should be accepted until all the critical questions have been successfully addressed. This will then require to record: eventual conflict of interest with the individual of company that created the autonomous system; the internal consistency of the system; the dataset(s) used for training; all the steps that went from the data collection of the dataset, to the generation of the model, to the collection of data specifically used for prediction, to the inferencing step. Finally, we believe it should be enforced the general principle that every piece of information requires confirmation from multiple independent sources before being accepted.
The great challenge of Artificial Intelligence is “to understand the nature of intelligence and cognition so well that computers can be made to exhibit human-like abilities”([42], Preface).
We argue that Artificial Intelligence can go beyond enabling computer to exhibit everyman-like abilities such as driving a car: it can enable computers to collaborate with us at the highest scientific standards. In fact, research is currently being carried out for investigating how AI can be exploited for evolving some of the most pivotal human society processes, such as the democratic process,18
To successfully achieve that, we believe an holistic, homogeneous approach that addresses all the components depicted in Fig. 1 of an autonomous system is not only desirable: it is necessary. Several pieces of research already addressed many of those components separately: it is now necessary to connect the dots, to draw the big picture, and to evaluate it by the means of extensive experimentation with a large variety of human subjects.
There lies before us, if we choose, a continual progress and benefit in this fourth industrial revolution with true human-machine collaborations. Otherwise, we risk the rise of a new religion where machines are mystic, veiled, ubiquitous oracles, and their pronouncements and decisions will have just to be accepted until a new revolution.