Abstract
The more intelligent systems based on sub-symbolic techniques pervade our everyday lives, the less human can understand them. This is why symbolic approaches are getting more and more attention in the general effort to make AI interpretable, explainable, and trustable. Understanding the current state of the art of AI techniques integrating symbolic and sub-symbolic approaches is then of paramount importance, nowadays—in particular in the XAI perspective. This is why this paper provides an overview of the main symbolic/sub-symbolic integration techniques, focussing in particular on those targeting explainable AI systems.
Introduction
In the era of the fourth industrial revolution, we are witnessing a fast and widespread adoption of AI in disparate domains. Nowadays, new AI applications often leverage on sub-symbolic approaches – e.g., deep learning [1–3] – to provide sophisticated features that would be hard for human developers to implement otherwise.
However, despite the unprecedented performance arising from the exploitation of sub-symbolic approaches, experts and practitioners nowadays widely acknowledge that state-of-the-art sub-symbolic models lack key features such as inspectability, interpretability, or explainability, given that they are inherently designed, trained, and used as black boxes [4]. In other words, sub-symbolic techniques perform very well in learning complex relationships and tasks from data, but they struggle in easing humans’ understanding of how a particular relationship or task could or should be performed. This is potentially troublesome since, especially in critical domains – such as the medical, financial, or legal ones, where sub-symbolic intelligent systems are exploited to support or automate decision making –, ethical and legal issues are likely to arise. In fact, in all the cited contexts, it is not enough for intelligent systems to support human decisions: instead, they are also expected to provide motivations and explanations giving insights about how and why decisions are attained.
Broadly speaking, the “black box” expression is used to refer to models where knowledge is not explicitly represented, but rather it is distributed among tensors of real numbers—whose complexity seldom fits our cognitive capabilities as humans. This is why human beings can rarely understand what a black box actually knows, or how it achieved a particular decision. In turn, troubles in understanding black-box behaviour is what prevents people from fully trusting – and thus accepting – them. In other words, the “black box” expression stresses the intrinsically opaque nature of most sub-symbolic techniques—which is one of the main limitations of the current state of the art.
New research efforts towards eXplainable artificial intelligence (XAI) [5] are aimed at mitigating the opacity issue, and pursue the ultimate goal of building understandable, accountable, and trustable intelligent systems—although still with a long way to go.
In this context, it is increasingly recognised that symbolic approaches to machine intelligence may have a critical role to play in overcoming the current limitations arising from the intrinsic opacity of sub-symbolic approaches [6–9]. Symbolic and sub-symbolic approaches are somewhat complementary to each others: while the latter ones are inherently opaque, fuzzy, and data-eager, the former ones are human-intelligible, exact, and parsimonious in terms of data. At the same time, while symbolic approaches often require human experts to manually encode symbolic knowledge, sub-symbolic approaches typically support some form of automatic learning from data.
Accordingly, it becomes fundamental to understand whether and to what extent a combination of symbolic and sub-symbolic approaches may contribute in making AI more inspectable, interpretable, or explainable. As a first step in this direction, in this paper we assess the current state of the art of integration of symbolic and sub-symbolic techniques under the XAI perspective. We provide an overview of the main models, methods, and technologies from the literature that simultaneously (i) propose a combination of symbolic and sub-symbolic AI approach (or vice versa), and (ii) explicitly target – or have the potential to be exploited for pursuing – XAI tasks or goals.
We categorise the surveyed works according to an original taxonomy discriminating contributions depending on how the integration of symbolic and sub-symbolic approaches is realised. For each approach we provide a brief description, a technological assessment of the software tools possibly available, and we discuss how the synergic exploitation of both symbolic and sub-symbolic techniques can bring benefits w.r.t. the aforementioned properties, as well as the kind of explanation the approach can provide.
Our assessment aims at (i) providing a way to compare, analyse, and evaluate the different capabilities of existing models and techniques w.r.t. XAI; (ii) identifying possible directions to advance the state of the art of AI, as far as the integration of symbolic and sub-symbolic approaches to automatic reasoning and machine learning is concerned; (iii) providing designers and developers with methods, techniques, and tools for knowledge extraction towards transfer learning; (iv) identifying a road map for principles and applications of explainable AI.
Accordingly, the paper is organised as follow. Section 2 provides some context information along with some definitions upon which the classification of the works is based. Section 3 presents and organises the main symbolic and sub-symbolic techniques for XAI devised out according our taxonomy. Section 4 collects and displays information from the selected approaches in form of tables, graphs, and word clouds. Finally, Section 5 provides for an overall view on symbolic and sub-symbolic techniques for XAI, offering some interpretation criteria on their current state of the art and perspectives; conclusive remarks are reported as well.
Context and definition
Explainable AI is an emerging field, focussing on the design and development of intelligent systems that allow humans to understand the reasons behind their recommendations or decisions, making it possible to know the data, the rationale, and the arguments leading to a particular outcome, other than to question or correct them 1 .
The XAI research area has gained momentum in recent years, probably due to the occurrence of a number of related phenomena, including, but not limited to: (i) the pervasive adoption of data-driven AI which characterised the last decade, with associated growing expectations; (ii) the consequently-increased sensitivity of the world’s public opinion w.r.t. data-management issues; (iii) the appearance of data-related regulations in most developed countries—as the European Union’s GDPR 2 . However, many contributions in the XAI field are rooted on works published in the early 90s, as witnessed by some of the contributions surveyed in this paper. It is indeed well recognised that not all approaches to machine intelligence are equally understandable and interpretable in the eyes of human observers. In particular, the opacity of most techniques exploited by modern sub-symbolic AI – like neural networks – is a well-understood issue, in the same way as the interpretability of other approaches laying at the edge between the symbolic and sub-symbolic worlds—like decision trees, or rules.
In order to avoid confusion on the meaning of terms, in this section we provide a definition for fundamental notions – namely, interpretation, and explanation –, as well as the classification of the kind of explanation we can find in the surveyed techniques and the purposes for which an explanation is sought. Notions and distinctions are useful for understanding the discussion of models and their classification in the next section.
Interpretation vs explanation
Interpretability and explainability are desirable properties for intelligent systems. We may briefly and informally define XAI as the corpus of literature and methods aimed at making sub-symbolic AI either more interpretable for humans, possibly by automating the production of explanations. However, despite the many definition attempts, there is still no common agreement on a shared, unambiguous definition for the two terms.
Based on the preliminary work by Ciatto et al. [10], and by drawing inspiration from computational logic, we let the term interpretation indicate the subjective relation that associates each representation with a specific meaning in the domain of the problem. In AI algorithms, interpretability refers to the cognitive effort required by human observers to assign a meaning to the way the algorithm works, or motivate the outcomes it produces. Indeed, in those contexts, the notion of interpretability is often coupled with properties as algorithmic transparency (characterising approaches which are not opaque), decomposability, or simulatability.
As far as the term explanation is concerned, we trace back its meaning to the Aristotelian thought, other than the Oxford dictionary definition. In particular, the Oxford dictionary defines explanation as a set of statements or accounts that makes something clear, or, alternatively, the reasons or justifications given for an action or belief. Thus, an explanation is an activity aimed at making the relevant details of an object clear or easy to understand to some observer.
Accordingly, the concepts of explainability and interpretability are basically orthogonal. However, since they are not unrelated notions, it may happen that an explanation could be given by exploiting the interpretability feature. This is the case of explanation by model simplification [11], where a poorly-interpretable model is translated into another – a more interpretable one –, having “high fidelity” [12] w.r.t. the first. The translation process of the first model into the second one can be considered as an explanation. For instance, as we further discuss in the next section, there exists a number of methods for extracting decision trees out of sub-symbolic classifiers. In this case, the extraction method is technically an explanation. Accordingly, we say that a system is explainable if there is a way to explain any of its opaque components.
Kind of explanation
According to the main impact surveys in the XAI area [6, 12], the explainability feature of a system can be classified as either explainability by design or post-hoc explanability.
Explainability by design. Methods in this category aims at creating interpretable or explainable intelligent systems by construction. This category can be further decomposed into two sub-categories:
Post-hoc explanability. Methods in this category aims at making intelligent systems interpretable or explainable ex-post, i.e., by somehow manipulating some poorly interpretable pre-existing system. This category can be further decomposed into the following sub-categories:
Goals of explanation
XAI research so far has highlighted several goals that can be achieved by an explainable model. Once again, there is no shared definition of what XAI goals are, and, unfortunately, just a handful of contributions have attempted to define those goals by adopting a conceptual perspective. However, in the reminder of this work we stick on the following XAI goals and corresponding definition, also proposed in [6]:
These goals can help discriminating the many diverse purposes for which different explainability techniques for sub-symbolic approaches are developed.
Symbolic vs Sub-Symbolic AI
As discussed above, the focus of this survey is not on the whole spectrum of XAI methods, but rather on the ones laying at the intersection between symbolic and sub-symbolic AI—sometimes referred as hybrid in the literature. Note that not all hybrid methods are directly targeting the needs of XAI.
More precisely, elaborating on how the hybridisation of the model is performed actually allows us to build the taxonomy for the surveyed works. Two main hybridisation schemes are currently leading the research in the field:
In both schemes, the adjective “symbolic” is often an alias for “logic-based”. In fact, as it is further discussed in the next section, most symbolic approaches actually leverage on some sort of logic—there including simple if-then rules. This is unsurprising, as logic-based approaches already have a well-understood role in building intelligent declarative systems [13].
On the other side, the term “sub-symbolic” is used to refer to numerical, statistical, and distributed representation of machine learning models.
Symbolic and sub-symbolic integration: main approaches
The integration of symbolic and sub-symbolic approaches is a research field already active in the last decades, which has acquired even more relevance with the recent momentum gained by explainable AI systems. In the following we assess the state of the art according to the original taxonomy depicted in Fig. 1, which classifies contributions based on how the blending of symbolic and sub-symbolic approaches is realised—integration vs combination, see Subsection 2.4.
Taxonomy of hybrid (symbolic + sub-symbolic) techniques for XAI.
Works are further discriminated inside the model integration category depending on the kind of logics they leverage upon: “logic and numerical” integration vs “numerical, statistical, and logic”. On the other side, the model composition category is split depending on the kind of composition: symbolic knowledge extraction vs symbolic knowledge injection. The former sub-category includes those approaches where some sort of symbolic knowledge is somehow extracted from sub-symbolic models – namely, rules, and tree extractors –, whereas the latter includes those approaches where some sort of symbolic knowledge is injected into sub-symbolic models.
For each selected approach we provide a brief description, as well as a technological assessment and an analysis under the XAI perspective.
In this category we review the main attempts to integrate symbolic models (such as the logic ones) with sub-symbolic ones (such as statistical and numerical). The main research lines here are those related to the neural-symbolic computing – the study of logics and connectionism as well as statistical approaches working on the integration of computational learning and symbolic reasoning – and relational learning—focused on learning expressive logic / relational representations. Neural-symbolic integration [46] aims at building a bridge between the symbolic and sub-symbolic sides of the divide. Relational learning [47] is the union of inductive logic programming, statistical relational learning, and multi-relational data mining; it constitutes a general class of techniques and methodology for learning from structured data – such as graphs, networks, relational databases – and background knowledge.
Note that in the following we differentiate statistical and numerical approaches as follows. In statistical models, relevant aspects are either explicitly or implicitly modelled via random variables or probability distributions, and decisions are taken via Bayesian inference—e.g., graphical models, hidden Markov models, Markov networks. Conversely, in numerical models, relevant aspects are represented through one or more functions over real numbers that approximate the phenomena of interest, and decisions are taken by either minimising error or maximising the likelihood w.r.t. data—e.g., support vector machines, decision trees, random forests, or (deep) neural networks.
Accordingly, in this category we can further distinguish among approaches that integrate logic and numerical models (e.g., Logic Tensor Networks), and approaches that integrate numeric, statistical and logic models (e.g., DeepProbLog, Markov Logic Networks).
Generally speaking, the key advantage of these approaches lies in the blended integration of different models, thus allowing efficiency, on the one hand, and possibilities for interpretability and explainability, on the other.
Logic and numerical models integration
Here we present a number of works integrating logic and symbolic knowledge with numeric predictors such as (deep) neural network.
Integration exploits logic rules expressed via first-order logic (FOL) – or some subset of it – which are used to constrain the behaviour of one or more numeric predictors. In these cases, constraining is performed by extending the loss function used by most numeric learning algorithms – there including the back-propagation algorithm used for neural networks – with an additive, regularisation term constructed from the logic constraints. The numeric predictor is then trained “as usual”, via optimisation—i.e. minimising the loss function. However, thanks to the regularisation term, the training process is more likely to select a set of parameters for the numeric predictor, which are consistent with the logic rules.
Accordingly, works in this sub-category are interesting from the XAI perspective as they support the creation of trustworthy hybrid systems, which provide an higher confidence to their users as they will behave as expected in all possible scenarios.
Knowledge-Based Artificial Neural Networks, (KBANN) 1990. KBANN [14] is one of the earliest attempts of exploiting symbolic AI to govern the structure and the behaviour of neural networks.
Differently from the other approaches described in this category, the main idea behind KBANN is not to alter the loss function used to train a neural network. Rather, KBANN is capable of devising the structure of a neural network from a symbolic knowledge base containing the user-defined, symbolic background knowledge. More precisely, KBANN assumes a stratified, Prolog-like, logic theory is available, encoding the background knowledge. Under this assumption, the KBANN algorithm aims at creating a neural network semantically reflecting the symbolic knowledge from which it was created. This step essentially sets the network structure and weights in order to reflect the rules contained into the logic theory. The resulting neural network can then be trained over data via back-propagation, in order to refine or generalise its functioning over (possibly novel) data.
According to the authors, the KBANN algorithm has proven to be useful in the area of molecular biology. In particular, neural networks attained via KBANN has been used to detect promoters in strings of nucleotides, with superior performance w.r.t. randomly initialised neural networks or other sorts of numeric predictors.
XAI perspective. KBANN is essentially a technique aimed at exploiting symbolic background knowledge to bootstrap a neural network. For this reason, we argue that KBANN provides explainability by design by exploiting logic as a constraint.
As far as XAI goals are concerned, KBANN improves the trustworthiness, confidence, and fairness of neural networks, thanks to its strong reliance on prior symbolic knowledge. However, transferability is not guaranteed in this case, as the KANN algorithm operates in a one-way fashion: apparently, there is no way to add further symbolic rules to some network built via KBANN after it has been trained on data.
Technological perspective. The KBANN algorithm is presented only in [14]: even though the authors present some experimental results, no link is provided to any sort of software implementation.
Deep Neural Networks (DNN) with Logic Rules, 2016. The recent work by [15] (no concise name is given) introduces another method for constraining a (deep) neural network behaviour via FOL rules. The proposed framework enables neural networks to be simultaneously trained on labelled data or logic rules, via an iterative distillation procedure aimed at transferring the symbolic knowledge encoded in the logic rules into the network parameters.
Unlike the main approaches described in the same category, this work is not only based on the use of logic constraints as regularisation terms. Rather, the authors propose the exploitation of two networks: a teacher and a student one. The teacher network is rule-regularised via an ad-hoc term added to the loss function, meaning that it is trained by keeping into account the user-provided logic rules. In particular, logic constraints are encoded into the loss function via soft logic [48]. Conversely, the student network is trained to balance between emulating the teacher network output and predicting the expected outcomes of the dataset.
In other words, this method leverages on a metaphor rooted in human education, where teachers possess the background knowledge for a given topic, and instruct students by providing them with informed suggestions helping them answer to a number of questions, until the students success rate is high enough.
According to the authors, the proposed technique has been successfully exploited in some tasks related to text analysis—namely, sentiment classification and named entity recognition. The former task consists of identifying whether a particular sentence (free text) is characterised by an underlying positive sentiment or a negative one. Conversely, the latter task consists of identifying (i.e. locating and tagging) well-know entities (characterised by a name) possibly mentioned in some portion of text, taking into account the unavoidable subtleties or ambiguities of the natural language.
XAI perspective. Similarly to other approaches in the same category, this work targets the construction of explainable intelligent systems by exploiting logic as a constraint. In particular, DNN with Logic Rules support XAI by providing trustworthiness, confidence, and fairness. Here, the distillation of logic rules increases the expectation that the student network will behave as intended, and rules may be exploited as a countermeasure biases possibly buried in the training dataset. It is worth to mention that this method represents a notable example of transferability. In fact, the teacher network is essentially in charge of transferring its background knowledge to the student network.
Technological perspective. An implementation based on Theano 3 is mentioned in [15], even if we were not able to find any software technology related to this work.
Logic Tensor Networks (LTN), 2017. LTN [18, 19] integrate learning based on tensor networks [49] with reasoning based first-order many-valued logic [50]. The work enables a range of knowledge-based tasks using rich knowledge representation in FOL to be combined with efficient data-driven machine learning based on the manipulation of real-valued vectors.
Given data available in the form of real-valued vectors, logic soft and hard constraints, and relations that apply to certain subsets of the vectors can be specified in a compact way using FOL. Reasoning about the constraints can help improve learning, and, viceversa, learning from new data can revise the constraints thus affecting the reasoning task. The model integrates symbolic and sub-symbolic approaches and it is defined upon the Real Logic [19].
Intuitively, the logic formulæ are used to build a loss function that aims at training a network capable of approximating the truth value (in the [0,1] interval) of the formulæ given as input. This is done by searching for the best possible representation for symbolic constructs in a vector space (grounding of atoms, functions, predicates), so that the satisfiability of the network is as close as possible to 1 on the test dataset. The resulting network is able to learning from the rightly-labelled real examples, but keeps the logic imprint given in the training phase. For the sake of simplicity, the approach can be seen as placing a logic network on top of a deep neural network in order to learn the relations between abstractions, so to enable the system to self-explain.
In addition to the interesting contribution at the foundational level of AI, one of the main application of LTN has been their evaluation in a challenging yet unsolved AI task: the Semantic Image Interpretation [51]. In particular, LTN have been exploited to infer part-of relations between the portions of an image, to improve the effectiveness of object recognition tasks on graphical data.
XAI perspective. LTN enable the representation of relational knowledge to be infused into neural networks, in addition to the completion and distillation of knowledge through network queries. More precisely, LTN learn from numerical data and logic constraints, enabling approximate reasoning on unseen data to predict new facts which are consistent with the logic knowledge used to constrain the learning process. The predicted facts can then be used to provide inferences and reasoning mechanisms over data.
LTN are of course an example of explainability by design, where logic is exploited to constrain the sub-symbolic model. Accordingly, we argue that LTN supports XAI by providing trustworthiness, transferability, confidence, and, potentially, fairness. In fact, the symbolic knowledge (exploited as a constraint and as background knowledge) strongly reduces the risk of neural networks learning unexpected or undesired behaviours from data, thus increasing both the trustworthiness and the confidence of the system. Furthermore, by supporting the prediction of novel facts from data, LTN support transferability of knowledge. Finally, LTN may provide fairness by letting developers express constrains aimed at preventing the learning of biased or unfair behaviours from data, assuming that constraints can be expressed via FOL fourmlæ.
Technological perspective. LTN come with a technological framework based on Python 3 and TensorFlow [16], available on GitHub 4 . It consists of a full fledged library for experimenting with the LTN model. The software is actively maintained and well documented. Furthermore, it includes examples and tests.
Semantic Loss Function (SLF), 2018. Semantic Loss [20] is another attempt of bridging neural networks and symbolic constraints via loss-function manipulation, similarly to what already described for LTN. In other words, the main idea behind Semantic Loss is to constrain the training process of a neural network via some propositional logic formulæ which are then encoded as part the loss function exploited by the training algorithm. Such formulæ consist of boolean variables representing input and output neurons of the networks to be constrained, possibly combined via classical logic connectors.
Many differences exists, however, both at the theoretical and the technological level, between Semantic Loss and LTN (or Lyrics). In fact, Semantic Loss mostly focuses on propositional logic and, therefore, it does not support quantifiers or predicates within the logic formulæ used as constraints. This trait greatly simplifies both the model and its implementation. In particular the embedding schema adopted by Semantic Loss – which maps the propositional constraints in the loss functions – simply interprets neurons activation values as probabilities and accordingly computes the semantic loss via simple algebraic operations.
According to the authors, the main goals of Semantic Loss are (i) to improve the predictive performance of neural networks – by allowing the training process to take background knowledge into account –, and (ii) to support semi-supervised learning.
XAI perspective. Similarly to LTN (and Lyrics), the Semantic Loss approach targets the construction of explainable intelligent systems by exploiting a very simple logic (the propositional one) as a constraint. In particular, Semantic Loss supports XAI by providing trustworthiness, transferability, confidence, and possibly fairness. Motivations are analogous to the ones discusses above. However, we expect Semantic Loss to be less effective than the previously-discussed solutions in supporting XAI goals such as fairness, given the reduced expressibility of the logicon which it is based.
Technological perspective. The Semantic Loss technology is available on GitHub 5 and contains the code exploited by the authors for the experiments proposed in [20]. Source code consists of a few scripts targetting Python 3, coming with no documentation except for a brief description of the project. The code has not been updated since 2019. Thus, we argue that the available code is not a full fledged technology ready for general purpose usage, but rather a successful – yet concise – experiment shared with the research community.
Lyrics, 2019. At the conceptual level, the Lyrics framework is an extension of LTN, improving the way symbolic knowledge is declaratively enforced while training the sub-symbolic part of an intelligent system. Like LTN, Lyrics transforms FOL clauses into a set of constraints that are jointly optimised during learning. However, Lyrics focuses on a more declarative approach where users can model and implement their hybrid system via an handy interface that fruitfully mixes Python and FOL.
Thanks to its declarativeness, Lyrics can combine one or more neural networks into a single computational graph. Each neural network is mapped onto a logic predicate, when necessary, while (possibly global) constrains over the outcomes of the networks are mapped into logic formulæ. The resulting computational graph is then optimised against the available data via TensorFlow.
Similarly to LTN, the Lyrics framework leverages on fuzzy logic to create a differentiable way of measuring how much the output of a sub-symbolic predictor violates the enforced logic constraints. This measure is then added as an additional loss component in the loss function to be minimised during training, therefore acting as a regulariser. In other words, Lyrics lets the background knowledge to be enforced via regularisation in sub-symbolic black-box systems.
According to the authors, the major applications of Lyrics are related to predictive model verification, semi-supervised learning with background knowledge, collective classification [52], and text chunking. While model-checking-related applications concerns XAI, and are therefore discussed in the next paragraph, semi-supervised learning can be briefly defined as the task of learning from examples where only a subset of the examples have been labelled. Collective classification, in turn, is the task of classifying data based not only on their features but also on contextual data—such as correlations or functional relations among classes. Finally, text chunking is the task of semantically recognising sentences in textual data by tagging each word with its logic role in the sentence (noun, verb, adjective, etc). In all the discussed tasks, the performance achieved by the Lyrics technology deserve to be highlighted, as they are related to the fruitful combination of symbolic and sub-symbolic techniques.
XAI perspective. Similarly to LTN, the Lyrics approach is another means to build explainable intelligent systems by construction. In particular, it falls under the “logic as constraint” sub-category. Arguably, Lyrics supports XAI by providing trustworthiness, transferability, confidence, fairness, and accessibility. Motivations are analogous to the LTN framework, except for accessibility which is a peculiarity of Lyrics due to its declarativeness.
Technological perspective. The Lyrics framework is implemented in TensorFlow. In practice, it consists of a few Python scripts targetting Python 2—which is no longer supported by the Python Software Foundation since January 2020. The code is stable and it is available on GitHub 6 . It comes with a number of usage examples, but no actual documentation is provided apart from some brief description of the project. Furthermore, it seems that the code has not been updated since 2018. For all these reasons, we classify Lyrics as legacy software.
Logic and numerical & statistical models integration
Here we present a number of works integrating logic and symbolic knowledge with sub-symbolic black boxes, via statistic- or probability-based approaches.
Integrating logics and probability has a long story in AI and machine learning: the integration allows to express complexity and uncertainty of complex systems—thanks to the probability component. Moreover, complex models can be learned efficiently from examples and powerful inference algorithms can be used to answer queries about the world, or verify and improve the pre-existing symbolic knowledge. Following this purpose, most methods presented in this category exploits some form of inductive logic programming or (statistical) relational learning playgrounds, fruitfully integrated with some sub-symbolic component. The way this integration is performed, however, is what characterises each surveyed technique.
Markov Logic Networks (MLN), 2006. MLN [22] is an approach for blending FOL and probabilistic graphical models (a.k.a. Markov networks) in a single model. MLN is a first-order knowledge base with a weight attached to each formula (or clause). Together with a set of constants representing objects in the domain, it specifies a ground Markov network containing one feature for each possible grounding of each clause in the knowledge base, and the corresponding weight. Weights are efficiently learned from (pre-existing) relational databases by iteratively optimising a pseudo-likelihood measure via the L-BFGS algorithm.
A MLN can be queried by users, and answers are provided via probabilistic inference. In particular, inference in MLN is performed via the Markov chain Monte Carlo (MCMC) algorithm over the minimal subset of the ground network required for answering the query.
It is worth to be mentioned that, despite being a foundational framework, MLN are at the base of many approaches adopted towards explainable AI [5, 54].
XAI perspective. The weighted rules – representing the underlying model of this category – are exploited by a meta-interpreter to generate an explanation in the form of a proof for a why-question. Indeed, Markov Logic supports a special type of deductive inference for MLN, known as maximum a-posteriori inference, which can be exploited to find the most probable world given some evidence. Thanks to the logic formulæ exploited to describe dependencies and risk in the network the most probable explanation can be provided following the step of the inference process. Moreover, extension to abductive reasoning can be exploited, as in [55–57], to find an explanation for a given observation in the light of some background knowledge: the explanation refers to the root cause for a given behaviour, and the background knowledge refers to the dependency graph of the network.
For all these reasons, we classify MLN as a technique explainable by design—since there is no black-box component. It is particularly suitable to design transparent boxes out of interpretable blocks such as relational databases and graphical models. Accordingly, we argue that intelligent systems based on MLN may be exploited to pursue XAI goals such as causality, informativeness, and interactivity as they help users in understating the causes of contextual situations via interactive why and what-if queries.
Technological perspective. The authors proposes a quantitative assessment of MLN based on the Alchemy framework 7 . Alchemy is a software package providing a series of algorithms for statistical relational learning and probabilistic logic inference, based on Markov theory. The Alchemy homepage and software distribution are not updated since 2013, and we were not able to access the code, thus we are not able to provide any further technological assessment.
CILP++, 2014. CILP++ [23] is a model aimed at performing inductive logic programming (ILP) via bottom clause propositionalisation and neural networks.
Similarly to relational learning, ILP aims at inducing novel rules out of example data. However, in ILP, example data consist of logic clauses, describing (i) both positive and negative examples of the to-be-learned rule, and (ii) the background knowledge, containing a number of rules the to-be-inducted ones may exploit in their bodies.
CILP++ leverages on (i) neural networks to make ILP faster, and on (ii) propositionalisation to make the construction of neural networks out of arbitrary logic theories possible. More precisely, propositionalisation is a preliminary step, which is necessary to convert the example clauses into real vectors and the background knowledge into a multi-layered neural network to be fed with those vectors. Of course, the structure of this network reflects the rules contained in the background knowledge, and the input layer contains a neuron for each possible atom used in the background knowledge.
The resulting network is then trained against the positive and negative examples. In this way, the weights of the network are optimised to select the atoms of the to-be-induced rule—which are thus encoded in the structure of the trained network, in a sub-symbolic form. The extraction rule proposed in [58] is finally used to extract the induced rule from the neural network, bringing it back in symbolic form.
XAI perspective. CILP++, like other works focussing on logic induction or relational learning, can be conceived as explainable by construction exploiting logics as a constraint technique—despite it also exploits rule-extraction techniques similar to the ones used for “model simplification”, which we describe later in this section. We argue that CILP++ can be exploited to pursue XAI goals such as transferability, informativeness, fairness, and accessibility. Transferability derives from the possibility of transferring both the source and the induced logic clauses from one two different instantiations of CILP++, possibly targetting similar problems. Informativeness and fairness derive from the exploitation of the induced information as a means to revel hidden information of biases, possibly buried in data. Accessibility derives from the declarativeness of NTP, where the potential of sub-symbolic AI is encapsulated behind a logic-based front end.
Technological perspective. The CILP++ technology consist of a C++ project hosted by SourceForge 8 , and, more recently, by GitHub 9 . We only analysed the GitHub-hosted one, as it is the most recent one. The project consists on bare, undocumented C++ sources, coming with no release, instruction, or build automation support. The codebase has not been updates since 2017.
Neural Theorem Prover (NTP), 2017. NTP [25] are neural networks acting as logic reasoners (a.k.a. theorem provers). They are built by taking inspiration from backward-chaining-based reasoning algorithms, as in Prolog. In particular, the neural network is recursively constructed to encapsulate the knowledge encoded in some logic theory, and trained to correctly answer to all possible queries on such theory. Of course, the structure of the resulting network reflects the structure of the clauses contained into the source logic theory. However, differently from the other techniques presented in this sub-category, both theories and queries supported by NTP can contain logic variables, as NTP is able to calculate, at the neural network level – i.e., in the sub-symbolic model –, the logic unification. In other words, NTP perform symbolic reasoning on top of sub-symbolic and distributed representations of knowledge.
Thanks to their peculiarities, NTP overcome a number of limitation characterising similar approaches such as LTN and CILP++. For instance, logic theories require no preliminary grounding of clauses to be compatible with NTP. Furthermore, NTP support multi-hop reasoning.
Moreover, by exploiting the distributed representation of knowledge, the NTP tend to place the vector representations of similar symbols in the close proximity of their corresponding vectorial space. For this reason, NTP can induce novel rules to be added to the symbolic theory, given prior assumptions about the admissible structures of the logic relationships to be induced. Thus, NTP can also be considered as a tool for performing inductive logic programming (and related relational learning).
According to the authors, NTP can be exploited to perform link prediction in knowledge bases. Big handcrafted logic theories may contain some holes, in terms of missing facts or rules, and NTP may help in filling them by inducing the missing information.
XAI perspective. From the XAI point of view, we classify NTP as a means to reach explainability by construction, by using logics as a constraint. Similarly to CILP++, NTP can be exploited to pursue XAI goals such as transferability, informativeness, fairness, and accessibility, for similar reasons.
Technological perspective. Despite some experiments on NTP (and their results) are mentioned in [25], no link is provided to any sort of software artefact, and we were not able to find any software resource on this topic.
Differentiable Inductive Logic Programming (∂ILP), 2017. ∂ILP [24] is another means for ILP leveraging on neural networks. As an hybrid ILP system, it aims to combine the advantages classical symbolic induction (e.g., data-efficiency, comprehensibility, transferability, and generalisation capabilities) with the advantages of the sub-symbolic systems (e.g., robustness to noisy or ambiguous data in training).
The main idea behind ∂ILP is to mimic logic deduction on definite clauses via a neural network. However, differently from NTP, ∂ILP perform deduction using forward chaining, instead of backward chaining. Briefly speaking, the authors re-interpret ILP as a binary-classification problem. As for other similar approaches discussed in this category, a neural network is constructed in such a way that its structure reflects a grounded version of the background knowledge. The resulting network is then trained to minimise the cross-entropy with respect to positive and negative examples.
∂ILP is essentially a foundational work. However, a number of experiments are discussed in [24] showing how ∂ILP outperforms previous ILP system in term of the number of tasks it can induce.
XAI perspective. The classification of ∂ILP under the XAI perspective is quite straightforward. Similarly to other hybrid approaches to ILP or relational learning, we classify ∂ILP as explainable by construction, exploiting logics as a constraint technique. Similarly to CILP++, and NTP, ∂ILP can be exploited to pursue XAI goals such as transferability, informativeness, fairness, and accessibility, for similar reasons.
Technological perspective. Even though some experiments on ∂ILP (and their results) are mentioned in [24], no link is provided to any sort of software artefact, and we were not able to find any software resource on this topic.
DeepProbLog, 2018. DeepProbLog [26] is another attempt of blending neural networks with logic programming, and in particular probabilistic logic programming.
Probabilistic logic programming is an extension of logic programming where facts and rules are enriched with probabilities, and queries are solved by not only stating if they are satisfied or not, but also to what extent—i.e., solution include probability values. As Prolog is the main language used in logic programming, ProbLog 10 is the main language used in probabilistic logic programming.
DeepProbLog is an extension of ProbLog exploiting neural networks for (i) computing the probabilities of facts, and (ii) letting neural classifiers be used as logic predicates—defined as “neural predicates” by the authors. In particular, each DeepProbLog program is translated into a tensorial computational graph – possibly including one ore more neural classifiers as sub-graphs – to be optimised via gradient descend. The structure of the computational graph reflects the structure of the rules contained into the DeepProbLog program. The optimisation step is aimed at simultaneously setting all the possible parameters regulating the behaviour of the computational graph, including the probabilities of facts and the internal weights of neural predicates. The resulting sub-symbolic system is then exploited to draw probabilistic inferences. In other words, hybrid systems based on DeepProbLog fruitfully combine probabilistic reasoning and sub-symbolic classification in a single, unified, coherent framework.
It is worth to mention that, unlike NTP, DeepProbLog requires the source logic theories to be grounded before any network can be constructed. This is why we consider DeepProbLog as a less-general approach than NTP.
According to the authors, DeepProbLog can also be used to induce new rules for a given probabilistic logic theory, exploiting its sub-symbolic representation.
XAI perspective. From the XAI point of view, DeepProbLog can be conceived as a means to reach explainability by construction by using the (probabilistic) logics as a constraint. Arguably, DeepProbLog can be exploited to pursue XAI goals such as transferability, informativeness, fairness, and accessibility. Motivations are analogous to the NTP, and CILP++ cases.
Technological perspective. DeepProbLog is available on BitBucket along with some open-source software 11 . It consists of a well-organised library targetting Python 3, ProbLog, and PyTorch 12 , coming with some basic documentation. The code based was lastly update in 2019.
Lifted Relational Neural Networks (LRNN), 2018. LRNN [27] aim at performing relational learning from data via neural networks.
Similarly to DeepProbLog, LRNN exploit sets of weighted first-order formulæ as structural templates for building a neural network to be trained over the available data. The resulting network is exploited to infer latent rules buried in data and to estimate the weights of the existing clauses. Like DeepProbLog, and unlike NTP, LRNN requires the logic knowledge base to be grounded before the network construction.
According to the authors, LRNN has been successfully exploited in the context of molecular biology.
XAI perspective. From the XAI point of view, LRNN, too, can be conceived as a means to reach explainability by construction by using the (probabilistic) logics as a constraint. Again, similarly to NTP and DeepProbLog, LRNN can be exploited to pursue XAI goals such as transferability, informativeness, fairness, and accessibility, for similar reasons.
Technological perspective. Even though some experiments on LRNN (and their results) are mentioned in [27], no link is provided to any sort of software artefact, and we were not able to find any available software resource.
Model composition
In this category we review the main attempts to combine symbolic models – such as rule sets, decision threes, or ontologies – with sub-symbolic ones. Among the surveyed works, we identify two major research lines. Thus, we split our categorisation and our discussion in as many sub-categories.
The first sub-category deals with the translation in symbolic terms of the sub-symbolic knowledge that most numeric predictors attain from data. A number of works from the past three decades converge in this line of research. Consequently, a plethora of names have been proposed during the years for this sub-category. To avoid any confusion or bias, we stick to the name “symbolic knowledge extraction”.
The second sub-category includes works and methods from the literature enabling sub-symbolic predictors to accept symbolic information as input. In fact, while most symbolic approaches can represents data structure of variable size, most sub-symbolic predictors can only work with vectorial and fixed-sized data. However, as all works in the second category focus on how to inject ontologies (a.k.a. knowledge graphs) into sub-symbolic predictors, we refer to this sub-category as “knowledge graph injection”.
Symbolic knowledge extraction
Here we discuss the main approaches for extracting symbolic knowledge out of sub-symbolic predictors.
The main underlying assumption behind most works in this category is that, once a sub-symbolic system has been trained over large amounts of data reaching some good predictive performance, then it must have attained a distributed representation of the knowledge contained in the data. Even though unintelligible to human beings, the distributed representation is still somehow buried in the internals of that sub-symbolic systems. Assuming this is the case, then a knowledge extraction technique is a means for making the distributed representation explicit and intelligible.
It is worth to mention that the idea of extracting decision rules or trees from sub-symbolic predictors is not new: it has been introduced several times, in many forms, and with different names and methods, since the late 80s. In fact, generally speaking, systems supporting symbolic knowledge extraction have a number of appealing features. In particular, they support a full exploitation of sub-symbolic techniques, which are the best choice when information must be mined from large amounts of data, and are usually better suited in terms of precision, robustness, and predictive performance. However, thanks to the knowledge extracted, those systems retain desirable XAI-related properties which would otherwise be lost.
Knowledge extraction techniques can be described and discriminated according to a number of orthogonal dimensions, including: the structure of the symbolic knowledge they extract (e.g., decision rules, decision trees, etc) the type of constraints they exploit for decision-making (e.g., linear constraints, M-of-N rules, etc) the sort of sub-symbolic predictor(s) they can deal with (e.g., neural networks, support vector machines, etc)
In the reminder of this section we partition the surveyed works according to dimensions 1, and 3, then for each approach we discuss the sort of the constraints exploited. In particular, in the same way as other impactful surveys on the topic [12, 59], we distinguish between techniques extracting rule lists and techniques extracting decision trees. Then, we further distinguish between pedagogical and decompositional approaches. In doing so we borrow the terminology from [59], where pedagogical techniques are those capable of extracting symbolic knowledge from any sort of sub-symbolic predictor – as they do not exploit any internal detail of the predictor under study to perform the extraction –, whereas decompositional techniques are those only capable of extracting symbolic knowledge from some particular sort of sub-symbolic predictor (e.g., neural networks, in most cases)—as they perform the extraction by looking at the internals of the predictor at hand.
Rules extraction. Here we focus on methods for extracting flat list of rules in the form
⋮
out of sub-symbolic predictors, where each condition is can be a conjunction or disjunction of (i) boolean predicates, (ii) linear constraints, or (iii) M-of-N rules over the attributes of the data used to train the sub-symbolic predictor.
We categorise the surveyed techniques for rule extraction depending on whether they are decompositional or pedagogical; then we provide some details for each technique; finally, we analyse them from the XAI perspective in an aggregate manner, given the huge similarity characterising the surveyed techniques from the XAI perspective.
Pedagogical approaches. We identified three main pedagogical approaches for rule extraction:
In particular, [28] extracts M-of-N rules out of any black-box classifier, regardless of whether it is a neural network or not. Apparently, however, this method does not support regression, and it only supports categorical attributes as conditions in the extracted rules. In spite of its limitations, this work has been proven to be effective in expert systems for diagnosis support.
On the contrary, [29] an [30] extract if-then-else rules out of arbitrary classifiers. In particular, RxREN supports datasets with mixed mode attributes (i.e., either categorical or numerical). The algorithm is based on a reverse-engineering algorithm that essentially discards insignificant attributes and discovers the variation range of input attribute for each possible outcome of the classification. For this reason, the rules extracted by RxREN are composed by linear constraints. Conversely, the ALPA rule extraction technique is the first that is applicable to any black-box model with no limitations on the nature of constraints.
It is worth remarking that pedagogical approaches are not based on the structure of the network, therefore they also work with other sub-symbolic models—even though oldest papers tend to mention neural networks more than other sorts of predictors.
Finally, it is worth to be mentioned that pedagogical extraction algorithms can essentially be described as oracle-based algorithms. In fact, in most cases the extraction algorithm works by querying the black box (which is therefore considered as an oracle), and by using the corresponding responses to build the rule list. This behaviour is repeated until the set of rules given by the white-box model converges to that of the black box. In other words, the extraction procedure terminates when the rule set as whole reaches an high fidelity w.r.t. the original black box.
Decompositional approaches. We identify some main decompositional approaches for rule extraction:
Generally speaking, most approaches here explicitly target a particular sort of sub-symbolic predictor. In particular, all approaches except [37] target neural networks, whereas [37] target support vector machines (SVM).
Some approaches [31–33] exploit some strict assumptions that limit the kind of neural networks they can manage, thus reducing their generality. For instance, the RuleNet technique described in [31] can only handle neural networks aimed at computing endomorphisms on n-sized strings of characters, and it aims at making explicit the condition-action rules exploited by such sorts of networks. At the same time, the MofN technique [32] can only handle neural networks attained via the KBANN algorithm described above. As suggested by its name, this method extracts M-of-N-like rules. Finally, the method proposed in [33] focuses on neural networks trained to act as recognisers for regular languages, and it is capable of extracting rules in the form of finite state automata for parsing these languages.
Other approaches – e.g., [34–36] – target general purpose neural networks. Briefly speaking, they compile networks into sets of rules with equivalent structure. There, each processing unit (neuron) is mapped into a separate rule – or a small set of rules –, and the in-going connections are interpreted as preconditions to that rule. The particular shape of preconditions – e.g., linear constraints, M-of-N constraints, etc. –, is then inferred by taking into account the weights of a neuron in-going connections, and its activation function. For instance, the KT algorithm [34] is capable of learning if-then-else rules with linear constraints out of general neural-network classifier. Similarly, the VI-Analysis [35] and RX [36] algorithms perform the same task via different procedures.
Finally, a different and noteworthy approach is described in [37]. There, the authors propose a method for extracting if-then-else rules with linear constraints out of SVM classifiers.
XAI perspective. Generally speaking, rule extraction techniques provide post-hoc explainability via model simplification. In fact, all the surveyed extraction procedures aims at creating a list of rule having an high-fidelity w.r.t. the source black-box predictor. This rule list can then be considered a symbolic, intelligible explanation of the source predictor. Accordingly, we argue that all these techniques may contribute to the pursuit of XAI goals as: trustworthiness, causality, transferability, informativeness, confidence, and possibly fairness. In fact, by making the inner function a black-box predictor explicit and intelligible, these techniques may increase the trustworthiness and confidence of intelligent systems. Furthermore, by providing an overview of the all the possible context-decision situation an intelligent system may face, and by making it possible to inspect which particular rule lead to a particular decision, rule extraction techniques may provide informativeness and causality. Moreover, the symbolic knowledge extracted can be translated into several forms, possibly making it compliant with symbolic intelligent systems. This of course provides for transferability. Finally, rule extraction techniques may help with fairness as well, by highlighting the biases possibly learned by sub-symbolic predictors.
It is worth to mention, however, that rule extraction technique are not the silver bullet of XAI. Issues related to accuracy, fidelity, and consistency, may easily arise in this kind of approaches, because the extracted rule list may not perfectly reflect insights of the original one. We argue that this is essentially unavoidable: the extracted rule lists are essentially approximated models, which are attained by removing (i.e. loosing) information from the source black-box. Moreover, interpretability of the extracted rule list may easily deteriorate as the amount of rules (or the amount of predicates per rule) increases—a situation which may easily arise as the complexity or the dimensionality of the black-box become non-trivial. Finally, it is worth to be noted that virtually all rule extraction techniques only focus on black-box predictors acting as classifiers. Not so much attention has been devoted by the academic community to the extraction of rules out of sub-symbolic regressors, as well as black boxes aimed at performing unsupervised learning tasks.
As a side note concerning SVM-based rule extraction techniques, it is worth to be mentioned that, although they have been known to produce classifiers that are easily comprehensible, they often approximate secondary models of worse accuracy [60]. Moreover, even though these models may be reasonably understandable from an expert perspective, they still lack the simplicity and familiarity to an individual user that often intelligent systems have to provide, as in the case of recommendation.
Technological perspective. In spite of the many surveyed papers for rule extraction algorithm, only one of them comes with some actual implementation: ALPA. The implementation of ALPA is available for download on the web page of the Applied Data Mining Research Group at the University of Antwerp 13 . It consists of a plugin for the Weka data-mining framework 14 . Thus, it is JVM-based software coming with a detailed manual. Apparently, the ALPA source and binary code was published in 2013 and never been updated since then.
Decision trees extraction. Here we focus on methods for extracting hierarchical decision tree out of sub-symbolic predictors.
Generally speaking, extracted decision trees are ordinary decision trees whose nodes are represented by rules consisting of a conjunction or disjunction of (i) boolean predicates, (ii) linear constraints, or (iii) M-of-N rules over the attributes of the data used to train the sub-symbolic predictor, similarly to the aforementioned decision rules. In other words, the main difference with decision rules lays in the hierarchical nature of decision choices.
Given the small amount of techniques for decision tree extraction surveyed in this section, we do not split our discussion any further to distinguish between pedagogical or decompositional approaches. Rather we provide this information as part of the details description of each method, provided below. We provide a joint discussion of decision tree extraction methods from the XAI perspective, at the end of the paragraph.
Surveyed methods. We identify three main approaches for decision tree extraction: the method by Krishnan et al. (1999, pedagogical) [39] the method by Schetinin et al. (2007, decompositional: random-forest-specific) [40]
Along the same line, [39] proposes decision tree extraction from neural networks. Unlike
Finally, [40] proposes an approach for the probabilistic interpretation of Bayesian decision trees ensembles (a.k.a. random forests) as a single decision tree. Classification confidence for each tree in the forest is calculated by exploiting training data: the decision tree covering the maximum number of correct training examples is selected, keeping the amount of classification errors in the remaining examples minimal. Unlike the previous ones, this method of explanation does not extend the input data set with random additional data and cannot be generalised to other types of sub-symbolic black boxes.
XAI perspective. From the point of view of XAI, decision tree extraction methods are quite similar to rule extraction ones, thus similar concerns fit their case. Accordingly, we argue that decision tree extraction techniques provide post-hoc explainability via model simplification, and help in pursuing XAI goals such as trustworthiness, causality, transferability, informativeness, confidence, and fairness.
In spite of the many similarities with rule extraction techniques, a remarkable peculiarity of decision trees extractors is worth to be mentioned: as hierarchical models, they are less prone to interpretability issues when the complexity or dimensionality of the source predictor grows.
Other critical aspects remain however unresolved w.r.t. rule extraction techniques. These include issues arising from the potential lack of fidelity, as well as the concentration of practically all the decision tree extraction techniques on the sole case of classification tasks—leaving others kinds of tasks in machine learning essentially uncovered.
Technological perspective. As far as implementations of decision tree extraction facilities are concerned, we were only able to find an ancient C-based implementation of
Knowledge graph injection
Inspired by the benefits of logic background knowledge that can lead to (more) interpretable system, knowledge injection is a field of research in some way specular to knowledge extraction. It deals with the injection of symbolic knowledge into sub-symbolic models.
Even though knowledge injection is in some sense already provided by many works described in Subsection 3.1, where symbolic and sub-symbolic techniques are integrated in an unique model – for instance, by using logics a constraint to inject background knowledge into neural systems [51] –, in this category we focus on works performing a particular sort of knowledge injection—namely, knowledge graph embedding. The key idea is to embed components of an ontology (a.k.a. knowledge graph, or simply KG) – including entities and relations – into continuous vector spaces, to allow neural networks to accept such a type of structured information as input and take advantage of its background knowledge to perform ordinary machine learning tasks.
Most of the currently-available techniques perform the embedding task only on the basis of observed facts. Given a KG, knowledge graph injection techniques first represent entities and relations in a continuous vector space, and then measure facts plausibility exploiting some scoring function. Entity and relation embeddings can be obtained by maximising the total plausibility of observed facts.
During this whole procedure, the learned embeddings are only required to be compatible within each individual fact, and hence might not be predictive enough for downstream tasks [41, 42]. As a result, more and more researchers have started to add other types of information, including logic rules [41, 44], in order to learn more predictive embeddings.
The noteworthy approaches that we deem significant for the purpose of this survey – as they combine symbolic and sub-symbolic models – are: Low-rank Logic Embeddings, LLE (2015) [43]
In particular, [41, 42] exploit rules to refine embedding models aimed at KG completion. KG completion is formulated as an integer linear programming problem, where the objective function is generated from embedding models and constraints are generated from rules. Facts inferred in this way are the most preferred by the embedding models and comply with all the rules. By incorporating rules, these approaches can greatly reduce the solution space and significantly improve the inference accuracy of embedding models.
Along this line, other works – e.g., [43, 44] – propose approaches that embed KG facts and logic rules simultaneously in a unified framework. In particular, in
In [45] the authors propose a method,
It is worth noting that in all these approaches rules are modelled separately from embedding models, serving as post-processing steps: this is why we classify these work as combination and not integration. Furthermore, all these works share a common drawback, in that they have to instantiate universally-quantified rules into ground rules before learning their models. This is called grounding procedure, and can be time- and space-inefficient—especially when dealing with big data scenarios or in case of rules complexity.
XAI perspective. From the point of view of XAI, knowledge graph embedding methods are aimed at providing explainability by design using logics as a constraint. Arguably, they help in pursuing XAI goals such as trustworthiness, informativeness, confidence, and fairness. In fact, the background knowledge injected by embedding knowledge graphs may increase the confidence and the trustworthiness of users by reducing the risks related to sub-symbolic systems acquiring undesired or biased information from data. Furthermore, the same background knowledge may be used to contextualise decisions and suggestions provided by sub-symbolic part of an intelligent system, thus improving informativeness. Finally, background knowledge may be used to prevent or compensate biased data, thus potentially improving fairness.
Technological perspective. Among the five surveyed methods for KG injection, only two works include with some public software resource. These are
Statistics, Data Collections & Discussion
In this section we collect and discuss information from the sources and the selected approaches, and display it in form of tables and word clouds.
Word cloud
Fig. 2 shows the word cloud generated from all the papers this survey is based on. In order to give some meaning to this limited yet synthetic view of the literature, the ten most evident words in the cloud could be divided into three subcategories: explanation, systems, model learning, neural, network, training, value, logic, rule.
In short, the picture could be interpreted as suggesting that integrated symbolic and sub-symbolic techniques for XAI are mostly exploited to give an explanation of sub-symbolic systems exploiting the expressive power of logics in modelling knowledge into rules, formulæ and states. Neural networks are chosen as typical representatives of sub-symbolic models, while logic rules are chosen to represent the symbolic approaches.
The cloud of words obtained by the main keywords extracted from the papers subject of the survey.
The cloud also highlights the relatively-high frequency of other words, related to the logic one (representative of the symbolic approaches), such as: knowledge, formula, variable, clauses, interpretation, ontology, fuzzy—all of them basically reinforcing the (quite obvious) idea that the knowledge to be discovered should be represented exploiting some kind of logic rules to formalise the systems and its properties. On the other side, the cloud shows some high-frequency words related to the machine learning domain (which includes most of the sub-symbolic approaches), such as: function, output, input, classification, space, features—somehow hinting at the main features of sub-symbolic approaches, such as the efficiency in finding solutions and learning in a very large data space.
Further scrutiny reveals that terms understanding, black, hidden, information, and trust also appear quite often. This hints at the main XAI goals these hybrid approaches aspire to achieve.
Other interesting words emerging from the cloud are prediction, reasoning, inference, and decision, possibly reflecting one of the main purposes of the symbolic / sub-symbolic integrated approaches for XAI—namely, predicting and reasoning over the system knowledge in order to take autonomous decision, and supporting the design and development of intelligent systems as human-centred AI technologies and applications. Finally, words like social, human, and interactions spotlight the social component of the intelligent systems target for the surveyed hybrid approaches.
Fig. 3 presents the papers in this survey on a timeline. At first glance, it can be seen that forerunners works – which characterised the decades between the 1990s and 2010s – are those related to knowledge extraction—in the form of both rules and trees, precisely in this order, to underline the influence of the former on the latter.
Surveyed works timeline.
In 2015, research works on model integration debuted on the XAI playground, and soon gained momentum, reaching the high concentration of works that we see today—although a couple of (theoretical) precursors can be found from the early 90s (e.g., [14]). This underlines two basic concepts – perhaps obvious, but worth remembering – of today’s AI scenario: i) the large amount of environmental data and the availability of increasingly-advanced technologies have made it possible to exploit integrated symbolic / sub-symbolic approaches in real applications of intelligent systems; ii) the need for these systems to be explainable is today more fundamental than ever.
It is also worth noting how the progress of technology and techniques has led to the exploration of approaches for the injection of knowledge, however dominated – in the same years – by approaches always linked to “logics as constraints”, yet based on model integration. Model integration has several advantages, discussed in the next sub-section: first of all, that of being explainable by design.
Tables 1 and 2 summarise the XAI perspective analysis provided in Section 3 for each approach.
Symbolic/sub-symbolic techniques for XAI: kind of explanation
Symbolic/sub-symbolic techniques for XAI: kind of explanation
Symbolic/sub-symbolic integrated techniques for XAI: main goals
Kind of explanation. At a glance, Table 1 shows symbolic/sub-symbolic integrated/combined techniques for XAI are actually concentrated in just two categories w.r.t. the type of explanation—namely, “logics as constraint” and “explanation by simplification”.
About the techniques that integrate symbolic and sub-symbolic approaches, there is actually one item labelled as transparent box design – the Markov Logic networks – since they are the only ones without components classifiable as black boxes. Apart from this exception, other techniques that integrate symbolic and sub-symbolic approaches exploit the use of “logics as constraints”: the integration is indeed implemented by binding the network (via logic constraints) during the training phase in order to produce a model that comply with the background knowledge (logic constraints).
The use of knowledge in the form of logic declarations or constraints in the knowledge base has shown not only to improve explainability, but, in some approaches, also to improve performance compared to approaches based exclusively on data [19, 51]. The main positive effect of these approaches is that hybrid and integrated approaches provide robustness to the learning system when errors occur in the training data labels.
Furthermore, many of these approaches have proven to be able to learn and reason jointly with both symbolic and sub-symbolic representations and inferences, sometimes enabling a probabilistic expressive logic reasoning [26].
A different perspective on XAI hybrid models is to enrich the knowledge of the black box models with symbolic knowledge, as proposed by the techniques categorised as model composition. In particular, this can be done by binding the neural network thanks to semantic KB and the like.
This is the case of composition of symbolic and sub-symbolic techniques that can be “explained by design” through the “logics as a constraint” technique. Unlike the integrated models, in this case the logic constraints (i.e. the knowledge graphs) are injected into the symbolic sub-model as input (once appropriately translated into numerical vectors). In this case the output of the sub-symbolic model is bounded by the constraints imposed as input.
Another hybrid approach is to map a system of black boxes onto a white box, or, a more interpretable twin. Along this line, any technique that reduces the complexity of the model or simplifies the results should be considered as a XAI approach. The size of this translation, in terms of complexity or simplicity, should correspond to how explainable the resulting model is. An underlying problem that remains unsolved is that the gain of interpretability provided by such XAI approaches may not be easy to quantify. The definition of general and shared metrics allowing XAI approaches to be evaluated remains an open challenge.
About the sort of explanation, all the approaches categorised as composition of techniques via knowledge extraction are included in the model simplification category. The approaches are those related to the extraction of rules or trees from the sub-symbolic model.
Decision trees have always been one of the most used categories of transparent models. Also based on the number of approaches collected in this survey, the literature on simplification and generation of the decision tree is very broad. The fact that a large majority of the application of these models does not fall within the field of AI (and not even in information technologies) mostly means that experts from other sectors usually feel comfortable when interpreting the results of these models. In fact, decision trees are models that perfectly achieve the goal of trustworthiness. However, they have poor scalability properties when compared to other models: therefore, they are difficult to apply in scenarios where predictive performance is of paramount importance.
Overall, Table 1 shows that obtaining a model as a transparent box is almost unlikely (due to the need of integrating the two techniques). However, the table also highlights that there are unexplored research directions that it might be worth exploring—such as textual, visual, local, by example and by feature explanations. All these directions become worthy of consideration and may encompass promising research perspectives to be taken into account. It is worth noting that textual or visual explanations are easy to obtain, once the model is simplified, from models such as decision trees: some attempts in this direction have already been made [9].
Generally speaking, methods from the “post-hoc explanation” category are usually preferred, as they do not prevent the exploitation of black-box predictors—commonly considered as better-performing in the general case. Conversely, the methods from the “explainable by design” category are best suited for critical contexts where interpretability is a major concern, and predictive performance can be sacrificed in its favour.
XAI goals. As already mentioned above, XAI definition brings together two concepts – namely, understanding and trust – that need to be addressed by the XAI model for AI. However, other purposes motivate the need for interpretable AI models—such as causality, transferability, informativeness, fairness, confidence, accessibility, interactivity, and privacy awareness. Table 2 focuses on XAI goals, based on the papers that state them explicitly as their goal. A checkmark between brackets denotes that the analysis of Section 3 has highlighted the XAI goal although it is not explicitly mentioned by the authors in their works.
Trustworthiness. Several authors agree on trustworthiness as the primary goal of an XAI model of AI. However, not all models addressing this goal explicitly talk about it in their works. According to the definition given here, trustworthiness amounts to ensure that a model will act as expected when facing a specific problem. This is why the works categorised as those pursuing this goal are the ones exploiting any form of symbolic knowledge (exploited as a constraint, or as background knowledge, or as knowledge extracted / injected into a model). Indeed, this feature strongly reduces the risk of black-box model to learn unexpected or undesired behaviours from data, thus increasing both the trustworthiness and the confidence of the system. Part of the works reviewed here explicitly mention the concept of trust when stating their purpose for achieving explainability. However, as one may observe in Table 2, they represent more or less half of the contributions.
Causality. Another goal in XAI is to find the causality between the data and the variables that represent them. Several authors argue that XAI models could facilitate the task of finding relationships that could be further tested for a stronger causal link between the variables involved. A sub-symbolic model discovers only the correlations between the learning data, therefore it may not be enough to reveal the general cause-effect relationships. However, causality implies correlation, therefore a sub-symbolic model integrated with symbolic techniques could validate the results provided with inference techniques, or, provide a first intuition of possible causal relationships. Once again, Table 2 shows that causality is neither among the most important objectives nor among those mostly addressed by hybrid models. Indeed, abduction-based approaches in Markov Logic Networks fall into this category. Abduction aims at finding the root cause, by drawing a graph of the causes, thus providing an explanation for a given observation.
Transferability. The model constraints that delimit and harness the model behaviour should be easy to transfer. Explainability may also be expressed under the guise of transferability, since it can facilitate the task of clarifying the boundaries that could influence a model, thus allowing a better understanding and implementation. For this reason, an explainable model should also be easily transferable—yet again, not viceversa. According to Table 2, the number of articles that include transferability among the features that make a model explainable is quite high—indeed transferability is the second most-mentioned goal. Generally speaking, the works that integrate ILP or relational learning do not address this goal: in fact, those techniques exploit the symbolic component to induce new rules, yet they do not represent the global knowledge of the system in order to make transferability possible.
Informativeness. Intelligent systems – and therefore models that support them – are mostly used with the intention of supporting the decision-making process of agents—here intended both as software and human ones. However, the problem solved by the model is never the same as the one humans would face. Therefore, a lot of information is always required in order to be able to relate the decision that a user would take to the solution provided by the model so as to avoid misunderstandings and misleading outcomes. To this end, explainable models should provide information on the problem addressed. Most of the articles surveyed use the expression provide information to refer to the process of disclosing relationships between data and the corresponding variables. All knowledge extraction techniques (regardless of whether they extract rules or trees) essentially provide post-hoc explainability via model simplification. In fact, sub-symbolic knowledge is then expressed and presented to the users via symbolic proxies – such as list or trees of rules – which can be considered and explanation of the original black box model. This is why all those techniques have informativeness as their main goal. Even at the level of model integration, the achievement of this goal is mentioned almost everywhere. This is certainly the most widely-used topic among the articles reviewed to support what they expect to achieve from XAI models.
Confidence. Confidence is basically mandatory in the models where reliability is expected—even more in the XAI perspective. Only half of the surveyed works deals with this feature, which seems reasonably to go hand in hand with trustworthiness. In fact, as mentioned above, some models – such as those based on ILP and RL, which are mainly used to learn latent rules and relationships – do not aim at making systems more trustworthy or confident—their (other) purposes are detailed in Section 3.
Fairness. Explainability, from a social perspective, can be considered as the ability to achieve and ensure absence of biases in the models for intelligent systems. There are strands of literature – outside the scope of this survey – that enable for an ethical analysis of the model in order to measure the level of equity achieved. Accordingly, a XAI objective should be enable the detection of possible distortion in the data. In fact, XAI could actually work as a bridge to avoid the unfair or unethical use of the model results. Among the works accounted here, almost no one explicitly mentions fairness, which instead is nowadays perceived as one of the most relevant issues. However, in Section 3 we have shown how, by increasing the degree of information, therefore of confidence and trust, many techniques already take a first step towards fairness.
Accessibility. A small portion of the surveyed contributions also aims at improving accessibility, always in order to promote explainability of a model. This is quite obvious, since the ability for humans to map their own cognitive models upon software models is surely a fundamental step towards explainability.
Interactivity. Only one contribution include the model ability to interact with the user—namely, Markov Logic Networks, which help users understanding the causes of contextual situations via interactive why and what-if queries. Yet, interaction with users should be one of the main XAI goals, so it seems likely that many future research efforts will be devoted to that.
Privacy awareness. Almost forgotten in the surveyed works, the ability to evaluate conditions relating to privacy should be accounted for by explainable models. In fact, the lack of understanding of what has been acquired, memorised, and learned by the model can lead to violations of privacy norms, thus leading to legal issues. This is a critical problem, given the main social contexts where XAI is going play a crucial role. As a result, researches in the hybrid legal/IT field look both promising and needed.
AI systems nowadays work on large amounts of data, learning from experience and making predictions with the goal of supporting human decisions or taking autonomous ones—applications range from clinical decision support to autonomous driving and predictive policing [61]. Nevertheless, concerns about the intentional and unintentional negative consequences of AI systems are legitimate, as well as ethical and legal concerns, mostly related to darkness and opaqueness of AI decision algorithm. For that reason, recent work on explainability and interpretability in machine learning and AI has focussed on simplified models that approximate the true criteria used to make decisions.
Bridging the gap between symbolic and sub-symbolic representations is a key obstacle along the path from the present state of AI achievement towards human-level artificial intelligence—in particular in the explainable AI perspective.
Symbolic intelligence, based on the use of symbolic rules to represent, explore, and infer knowledge, has many advantages, such as naturalness, interpretability, and easiness in giving explanations. However, it has two major disadvantages: acquiring rules and learning from a huge amount of data is difficult, and scalability is poor. On the other hand, sub-symbolic intelligence has the great advantage of being able to acquire knowledge from examples, to be highly scalable, and to succeed in representing complex and highly-inaccurate knowledge bases. The benefits and drawbacks of the two approaches compensate each other: this has lead to the emergence of techniques combining and integrating them so as to get strengths and benefits from both.
In this paper we focus on explainable AI, and discuss the main techniques for the integration of symbolic and sub-symbolic approaches on the general perspective of explainability of intelligent systems. On the one hand, we categorise the works that deal with model integration, where the symbolic and sub-symbolic components are no longer two separate parts of the system. On the other hand, we instead classify techniques and systems that combine the two approaches to obtain hybridisation.
Overall, in spite of the heterogeneity of the techniques and the variety of the research directions, all works demonstrate the feasibility and the potential benefits of the integration of symbolic techniques with sub-symbolic ones towards explainable AI.
Funding
This work has been partially supported by the H2020 Project “AI4EU” (G.A. 825619).
One of the authors, Roberta Calegari, has been supported by project “CompuLaw”, funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (G.A. 833647).
