On the role of knowledge graphs in explainable AI

2.1. Machine learning (except neural netwok)

∙ Research Question : ML algorithms [68] aim at elaborating a mathematical model based on sample data, known as“training data”, in order to make predictions or decisions on unseen data, known as “test data” without being explicitly programmed to perform the task. Five main tasks of learning are studied: (i) supervised learning if data contains both input and labeled data, (ii) unsupervised learning to derive some structures in data if labels are not exposed, (iii) semi-supervised learning if labelled data is small compared to unlabelled data, (iv) distant learning [34] which exploits relational data of unlabelled data from existing knowledge bases, and (v) reinforcement learning if further information could be captured through interaction with the environment.

∙ XAI Challenge : All tasks of ML expose mathematical models through an appropriate, but somehow abstract representation of data. XAI in ML [30] is about explanation of (i) models, known as global explanation, and (ii) a prediction, known as local explanation.

∙ Approaches : Some models are naturally designed to explicit their rationale e.g., linear regression, decision trees, generalized linear (or additive), naive Bayes models. In case of more complex models, some of their representative elements, such as feature importance, partial dependency plot or individual conditional expectation can be used for capturing high level representation of the ML model for global explanation. State-of-the-art approaches [52,64] go further by revisiting feature importance for local explanation.

∙ Limitations : Most approaches limit explanation to features involved in the data and model, or at best to examples, prototypes [44] or counterfactuals [55]. Explanation should go beyond correlation (which is what feature importance is about) and numerical similarity (which is what local explanation is about).

∙ Opportunity : Knowledge Graphs do encode contexts, do expose connections and relations, and support inference and causation natively. Existing XAI approaches in ML consider a flat representation of data, and context is out of the loop of the explanation process. Knowledge Graphs could be used for encoding better representations of data, structuring an ML model in a more interpretable way, adopt semantic similarity for local explanation. For instance we could envision linking knowledge graphs extracts to input data of a Machine Learning task to solve some distant learning tasks [34]. In addition we could envision approaches relying on Knowledge Graphs to compact large trees in decisions trees or even random forest. For instance combinations of nodes could be captured as a unique (probabilistic) concept or property in Knowledge Graphs. Machine Learning and Knowledge Graphs have great potential to be combined, and benefit from each other strength [26].

2.2. Artificial (deep) neural networks

∙ Research Question : Similarly to other ML approaches, Artificial Neural Networks (ANNs) aim at learning representation. The main differentiator with other approaches is its scalability and performance with a high number of features and instances, which better fit images and texts.

∙ XAI Challenge : Both local and global explanations are a strong focus of the ANN community.

∙ Approaches : Contrary to other ML approaches, there is no easy way around explanation of ANN models or predictions. Existing techniques either encode feature importance through attribution [73,79], attention mechanism [62], or obtain a more interpretable approximation through surrogate models [24], such as decision tree.

OPEN IN VIEWER

Fig. 3.

On the role of knowledge graphs for explainable artificial (deep) neural networks. (What is the causal relationship between the input/output/training data?) – Extension of Fig. 8 in [83] and https://fortune.com/longform/ai-artificial-intelligence-deep-machine-learning/.

∙ Limitations : Explanations are artificially built, for instance by forcing the network to focus on some group of features or correlations at best. In addition they do not represent any logic of the learning task, making explanation a very difficult task to achieve. The latter is due to the foundational theory of ANN, which consists in deriving a mathematical model through local optimizations.

∙ Opportunity : Novel ANN architectures need to be designed to natively encode explanation. Some recent approaches which aim at capturing better model hierarchical relationships [36], or causality mechanism [10] are promising. However, they could be polished further by (i) adding logic representation layers in ANN, such as [38] using network dissection approaches [8], (ii) encoding the semantics of inputs, outputs and their properties cf. Fig. 3. Knowledge Graphs could play a central role in such a new design, particularly as novel architectures should embed causation and feature reasoning. This is the case of [53] which introduced a layered graph model representation of (RDF-type) graphs in the ANN architectures for reasoning purpose. The layer is representing the semantics of predicates in Knowledge Graphs, and is captured as 3 D adjacency matrices. Other approaches from the neural-symbolic reasoning community [37] are worth investigating as they combine ANNs with probabilistic logic [54] or first order fuzzy logic [27]. Knowledge graph embeddings [33,72] are also Machine Learning artifacts where explanations could be elaborated their a latent representations. Such design could advance ANN further by supporting integration, discovery, fragmentation, composition and even reasoning.

∙ Research Question : Computer Vision relies on ANN architectures due to the nature and size of its data. Tasks range from semantic segmentation, object detection, scene reconstruction to visual question answering.

∙ XAI Challenge : The main XAI task in Computer Vision is identification of pixels, or group of pixels responsible for triggering a shape detection, an uncertainty or an error. Explanation is often referred to as visual inspection due to the nature of data processed.

∙ Approaches : Saliency maps [2] are classic methodologies in Computer Vision. They include many variants of gradient modification for capturing representative features. Network dissection [8] is another approach segmenting ANN to derive interpretable units and layers.

∙ Limitations : Although saliency maps expose interesting visualization artifacts, they do not capture any semantics. At best those artifacts capture a disentangled representation, which remain subject to human interpretation. Knowledge Graphs could expose the semantics of such disentangled representation. However, integrating semantics in ANN, hidden units of feature space remain open challenges.

∙ Opportunity : Adding semantics through context and Knowledge Graphs could help answering open questions, such as: What is a disentangled representation, and how can its factors be quantified and detected? Do interpretable hidden units reflect a special alignment of feature space, or are interpretations a chimera? All are open questions discussed in [8], and not yet resolved. Other open questions are: What conditions in state-of-the-art training lead to representations with greater or lesser entanglement? What is the semantics of a group of hidden units in neural networks? Interesting avenues aim at combining detection with reasoning to improve, and potentially explain semantic segmentation [3].

2.4. Constraint satisfaction and search

∙ Research Question : Constraint Satisfaction and Search aims at finding a solution to a set of constraints that impose conditions that the variables must satisfy. A solution is a set of values for the variables that satisfies all constraints. Constraints are defined on a finite domain.

∙ XAI Challenge : The main challenge is to identify which constraints to relax for conflict resolutions. Explanations are usually a subset of variables which satisfies a set of constraints.

∙ Approaches : Constraint Satisfaction and Search problems on finite domains are typically solved using a form of search. Backtracking, constraint propagation, local search are examples of such approaches. Even though the problem is known to be an NP complete problem with respect to the domain size, research has shown a number of tractable sub-cases with promising approaches [40,59].

∙ Limitations : Even though optimal structures and search spaces have been largely introduced in the community, complexity remains one of the main limitations.

∙ Opportunity : It has been demonstrated that any structure in problem representation has largely benefited search [49]. We could envision more knowledge-driven structure, inspired from Knowledge Graphs, which could dynamically adapt to variables, constraints, search space. Knowledge Graphs could even drive search through semantic and logical relations among constraints, which could be modelled as entities in a graph. In such cases constraints will be augmented with distant data from Knowledge Graphs.

2.5. Game theory

∙ Research Question : Game Theory [70] is the study of mathematical models of strategic interaction between rationale decision-makers. Examples of games include zero-sum games [56], in which one person’s gains result in losses for the other participants.

∙ XAI Challenge : Game Theory has been dealing with XAI from its inception as one of its main challenge is to identify and to understand the underlying mathematical model as well as its properties. Game theory is applied to a wide range of behavioural relations, and is now an umbrella term for the science of logical decision making in humans, animals, and computers, in which explanation is the core question driving the modelling.

∙ Approaches : The Shapley value [69] is a solution concept in game theory, which inspired recent research in Machine Learning to address the problem of explanation [52]. The Shapley value is characterized by a collection of desirable properties, and is used to capture the influence of a player in a game settings (or a feature in a machine learning setting). Such properties characterize the explanation.

∙ Limitations : Similarly to the domain of Constraint Satisfaction and Search, complexity is a challenge for explainability in game theory. Only an approximate solution is feasible, usually identified through some randomization of coalition in feature values.

∙ Opportunity : As recently explored, structured representation of the models as its features [18] has shown better scalability, while not necessarily improving explainability. Knowledge Graphs could be considered to better structure models, organize features, then reducing the search space and potentially improve understanding and readability of explanation, particularly when embedded in a structured set of connected entities. Recent examples [19] have demonstrated that graph structures do reduce the complexity of search.

2.6. Uncertainty in AI

∙ Research Question : The field of Uncertainty in AI is at the frontier of various AI fields, namely knowledge representation, learning and reasoning. Bayesian probability is one of the core fundamental, and Probabilistic Graphical Models (PGMs) [47] are usually central for representing and reasoning with uncertainty as they encode probability distributions.

∙ XAI Challenge : Graphical models are often used to model multivariate data, since they allow to represent high-dimensional distributions compactly. The explanations draw their attention on the compact distributions and their underlying data. Explanation is then naturally embedded through those relationships, usually through interdependencies and decomposition in data.

∙ Approaches : Some approaches [9] are formulating PGMs as weighted logical formulas [43] to tightly decouple the constraints and dependencies from the probabilistic parameters. Reasoning can then be performed on the logic representations. Other approaches analyzes latent spaces and its direct connections with the underlying data [81]. The strength of existing approaches is the underlying reasoning capabilities that PGMs and other probabilistic and logic systems offer.

∙ Limitations : Even though PGMs are appropriate representations to connect inter-dependable data, dependencies remains probabilistic. Therefore humans are required to remain in the loop to interpret any dependencies. Even embedded in logical formulas there is little gained as we are still embedded in the framework of standard probability theory.

∙ Opportunity : Semantic representations and connections through Knowledge Graphs could be used to disambiguate and force latent variables to represent interpretable content. This is particularly relevant as PGMs fit naturally in graph representations, in contextual information such as knowledge graphs could extend reasoning functionalities. Interesting avenues are Probabilistic Knowledge Graphs [57] or knowledge expansion over probabilistic knowledge bases [20].

2.7. Robotics

∙ Research Question : Robotics is an interdisciplinary branch of engineering and AI science, which deals with the design, construction, operation, and use of robots, as well as computer systems for their control, sensory feedback, and information processing. The underlying technologies are used to develop machines that can replicate human actions. They usually combine and integrate many of the technologies in the AI field.

∙ XAI Challenge : XAI is required in Robotics mainly for debugging and resolving discrepancy between a solution and an expected answer. Some of the XAI challenges are (1) the rationale of coordination in multi-robots Systems and swarms, (2) the fusion of explanation coming from many underlying AI systems, such as Planning and Scheduling, Computer Vision, or Knowledge Representation and Reasoning. They are unique challenges for robotics with many interesting opportunities as explanation is multi-modal, could be complementary but also conflicting, is spatial and temporal, is driven by goals but also initial conditions.

∙ Approaches : Narration of autonomous robot experience [66] together with approaches of summarization [15] have been recently introduced as a succinct way of presenting the decision process of robots. Various levels of granularity in the decision process are provided. [61] combine a robotics ontology with linguistic elements to expose the rational of robots’ actions.

∙ Limitations : Although the latter models extract information from a large poll of data, such systems do not explain their actions and justify their decisions [71]. Explanation is usually too fine-grained to be properly integrated by humans. Seamless integration of multi-modal explanation is also not addressed in the literature.

∙ Opportunity : The level of abstraction in explanation together with its multi-modal fusion are net opportunities for Knowledge Graphs. Some semantics could deeply support in exposing appropriate and personalized representations of explanations while fusing explanation content in a compact and comprehensible representation [60]. Knowledge Graphs have been designed to capture knowledge from heterogenous domains, making them a great candidate to achieve explanation per se in robotics.

2.8. Distributed AI

∙ Research Question : Distributed AI is the field of AI dedicated to the development of distributed solutions for problems. It is related to Multi-Agent Systems but also to any representation, structure, system which could make AI scalable.

∙ XAI Challenge : Main XAI challenges are focusing on explaining and resolving agent conflicts, based on their intentions and beliefs [80]. State-of-the-art approaches aim at identifying the best strategy, through explanation, to achieve a goal. More recent works focus on human comprehension of agent behaviour, its strategy, and its convergence in case of conflicting intentions and beliefs of agents [4,5].

∙ Approaches : Approaches, such as [39] determines the motivation for a decision by recalling the situation in which the decision was made, and replaying the decision under variants of the original situation. In such scenario they are able to discover what factors led to the decisions, and what alternatives might have been chosen had the situation been slightly different. Approaches tend to be very close to counterfactual [14] and case-based reasoning [1].

∙ Limitations : Even though ontology is a core representation layer for agents to communicate and negotiate, it is rarely used for explaining agent behaviour, its strategy and success. Lighter knowledge representations might be envisioned.

∙ Opportunity : The dynamics of agents interaction should be captured more formally, and embedded with broader common sense knowledge to identify human interpretable explanation. Formalization does not need to be complex. For instance some dedicated Knowledge Graphs could be used to contextualize the agents environment. Some recent works are going towards this direction of formalizing agent interactions [21].

2.9. Automated planning and scheduling

∙ Research Question : Automated Planning and Scheduling [29] is a branch of Artificial Intelligence that is about the realization of strategies or action sequences, typically for execution by intelligent agents, autonomous robots and unmanned vehicles. Unlike classical control and classification problems, the solutions are complex and must be discovered and optimized in multi-dimensional space. It could be done in real-time, i.e., on-line, or at design-time, i.e., off-line. Solutions usually resort to iterative trial and error processes.

∙ XAI Challenge : XAI challenges in AI planning [28] are as follows: explaining (i) causal relationships of actions, (ii) why some actions are chosen in particular situations, (iii) why plans are better than some, (iv) why plans could not be computed, (v) why replanning might be required.

∙ Approaches : Past work on explanations primarily involved the AI system explaining the correctness of its plan and the rationale for its decision in terms of its own model [17].

∙ Limitations : Existing approaches fail in exposing human-understandable explanation, as it is usually limited to the planner’s domain e.g., in term of actions and initial situation. This strongly limits the comprehension to experts in the given tasks.

∙ Opportunity : Knowledge Graphs could be a way forward to better contextualize complex terms, and even better summarize complex actions in more succinct and meaningful way.

2.10. Natural language processing

∙ Research Question : Natural Language Processing is concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data. Research questions includes (visual [6], multi-turn [51]) question answering [48], conversational agents with broader questions related to Speech Recognition, Natural Language Understanding and Generation.

∙ XAI Challenge : Similarly to machine learning, identifying importance of feature or entity is critical, as it aims at identifying which part of speech is driving the most relevant information. Other core XAI tasks include: explaining the rationale of questions sequencing in dialogue, debugging a plan-based dialogue system [45] or explaining the utterances which were intended to achieve [23]

∙ Approaches : The problem of identifying the most representative entities in a text classification task is addressed by [64] with many variants. Some works [22] extract plan-based model to understand intention and explain rationale of the discourse.

∙ Limitations : On the one hand ML-based approaches, which focus on important entities in text, suffer from having statistics-based explanation only, i.e., mainly based on co-occurrence and correlation. Pioneering work [76], relying on tree like structure in form of dependency trees, have been first steps towards structuring text processing tasks. On the other hand plan-based models have not been deeply explored, and many research questions related to their representation, rationale in questions sequencing remain open.

∙ Opportunity : Semantic descriptions, exposing meaningful representations, have demonstrated to have a positive impact on tasks such as relation extraction [32,41], event extraction [63] or text classification [82]. Similar representations, inspired from Knowledge Graphs could provide the semantic layer missing from brute-force machine learning approaches on text, aiming at exposing explanation [65]. They could also drive or at least guide sequencing of questions by refining, abstracting or instantiating obscure terms in questions. Challenges and approaches from neural language models for the semantic web are also interesting avenues of exploration [31].