Engineering user-centered explanations to query answers in ontology-driven socio-technical systems

2.1. Explainable cybersecurity: Questions guiding R&D

In [69], the authors highlight several research directions by describing different aspects related to explanations in the cybersecurity context such as who gives/receives the explanations, what constitutes a good explanation, when the explanation should be given, how to give explanations, where the explanations should be made available, and why the explanations are needed. We now briefly discuss each of these questions.

Who? – As we have mentioned, cybersecurity is closely related to socio-technical systems because it combines human efforts with those of more traditional information systems. On the human side, participants occupy roles that range from novice, barely computer-literate users, to experts who can write their own code. In particular, malicious actors also fall somewhere in this range, since nowadays there are tools that allow to launch attacks with very little knowledge. Thus, the level of expertise of the user plays an important role in providing an explanation.

What? – Another key related question is to identify the object to be explained; in this context, the level of detail will depend on the user’s goals. For example, novice users will likely need explanations that improve their trust in system answers, or that help them to understand how to use the system correctly, whereas further details will be necessary for power users. So, different components will need to be explained, such as automated mechanisms, policies, threat models, vulnerabilities, and countermeasures.

Where? – Location is another issue that needs to be addressed in order to decide where it is best for explanations to be made available. Possibilities include different levels, as part of specific workflows, detached from system functionalities, or incorporating dedicated modules to handle features related to explanations.

When? – Explanations can be provided at various instances, such as before, during, or after a task is carried out. Explainability may be essential for the system to begin functioning (for instance, to explain aspects of input data), it may be crucial to support users’ trust at runtime, or the goal may be to support future decisions, making it important for explanations to be available after the execution.

Why? – One of the most important questions regarding explainability is the reason why we need an explanation, since it points to the importance of the users’ understanding of the way the system produces its outputs. Some examples include increasing confidence, trust, transparency, usability, verifiability, or testability. Moreover, the importance and need for explanations generally fall into two classes: critical and beneficial. In the former, the system’s outputs will not be accepted without adequate explanation, while in the latter such explanations are beneficial, but not crucial.

How? – Finally, a delivery style suitable for the intended audience must be chosen. Styles can be categorized according to language, including:

natural language to produce possibly informal text or sentences that follow a template-based approach;

2.2. Explanations in intelligent socio-technical systems

As we began to argue in the previous section, a key requirement for the success and practical adoption of Intelligent Socio-Technical Systems (ISTSs, for short) is that users must have confidence in the system’s outputs and automated decisions. In this sense, automatically generated explanations have been recognized as a fundamental mechanism to increase user trust in AI applications. With this motivation, here we will analyze several kinds of explanations that have been proposed in the literature and are now well-established, and that will be used in the rest of the paper as basic styles. For a more comprehensive review of the literature on XAI, we refer the reader to [31,41,64].

Contrastive: Refers to approaches where explanations are based on a strategy that focuses on establishing some type of contrast [23,44,54,64] seeking to answer the question “why was one decision made instead of another”. The contrast is then presented between the resulting decision and one or more alternative ones. The goal of this approach is to provide users with sufficiently comprehensive information on the reasoning that led to the system’s outputs by answering questions focusing on possible alternatives that may be more intuitive than the ones received.

Counterfactual: As a sub-class of contrastive, counterfactual explanations identify the required changes on the input side that would have significant impact on the output [14,42,44]. People are used to considering counterfactuals in daily life; for example, adding new information to incomplete knowledge, making assumptions to evaluate possible scenarios, etc. Counterfactuals have become relevant to a variety of AI applications (e.g., improve training in generative adversarial networks (GANs)), and especially in XAI; the strategy has been particularly successful in systems that do not rely on data-driven techniques [8]. Although this explanation type is often seen as human-friendly, one drawback that has been identified [5] is the “Rashomon effect”, where each counterfactual explanation tells a different (but still true) story to reach a prediction.

Justification-based: In the area of ontological reasoning, there has been a significant amount of work on methods and techniques for computing and working with justifications as a dominant and popular form of explanation [35]. In this context, a justification for an entailment is a minimal subset of the ontology that is sufficient for the entailment to hold. The set of axioms corresponding to the justification is minimal in the sense that if an axiom is removed from the set, the remaining axioms no longer support the entailment. Advantages to this approach include, for example, that they are conceptually simple, they have a clear relationship with the underlying ontology, and there are simple presentation strategies that work well in many settings.

Data-driven: This kind of explanations have recently received much attention because they are very useful in complex AI data-driven models [1,4], which in general seek to derive a mathematical function such that the input is a set of values for features or attributes of an entity, and the output is the value of a particular feature of that entity. Typically, this function is obtained by fitting an example entity set (training set), and its main usefulness is the ability to make predictions on the value for the output feature of the entity, provided the process to build the function is carried out successfully. Data-driven explanations are most valuable when the model is complex, in which case the explanation-building process is based on deriving another simpler function that approximates the original one.

Statistical: The most likely explanation is not always the best explanation [44], and users benefit most from the identification of causes to explain events, not mere associative relationships; however, these models based on probabilities or statistics are still useful and commonly used in XAI.

Simulation-based: Leverages the implementation of a system to imitate a process of interest in order to analyze the results that arise from running it on one or more input configurations. Some works [29,57] have focused on exploring the role that these virtual simulations play in social explanations, for example by explaining real-world social phenomena. In general, because simulations play the role of a data-generating experiment, they do not directly provide an explanation; however, they do provide data to evaluate hypotheses within a theoretical framework, which ultimately provides the explanation. The field of simulation-based inference [22] has recently gained traction, and is a promising tool to be leveraged in XAI.

Contextual: Context typically refers to domain information about objects other than the ones explicitly participating as inputs to or outputs generated by the system, such as information about users, situations, and broader environment affecting the computation [7,12,25]. In [13], the author defines context as a “collection of relevant conditions and surrounding influences that make a situation unique and comprehensible”. Thus, context-aware explanations often include extra information that is not contained in the current described situation but is part of the users’ context.

2.3. HEIST: An application framework for XAI in socio-technical systems

As mentioned, in this paper we continue the line of work where we presented the Hybrid Explainable and Interpretable Socio-Technical Systems application Framework (HEIST, for short), adopting it as the central architecture for tackling explainability in ISTSs. HEIST is a framework for guiding the implementation of hybrid XAI-based socio-technical systems via a combination of data-driven and KR-based models. Figure 1-(A) shows an overview of the architecture proposed in [50] (in turn, this model is a generalization of the one first introduced in [48]), which has six components:

OPEN IN VIEWER

Fig. 1.

The HEIST (Hybrid Explainable and Interpretable Socio-Technical systems) application framework (A), and its instantiation into the NetDER-HS system for detection of online hate speech (B).

Having introduced the HEIST framework, in the next section we show how a particular instantiation can be derived to produce a system for a specific use case.

3.1. The NetDER model

Figure 1 shows how the HEIST architecture generalizes NetDER as proposed in [48]; note that NetDER does not make explicit reference to sub-symbolic services as we do here. HEIST’s Symbolic Reasoning module is instantiated in NetDER by choosing a particular knowledge representation language;. Furthermore, part of the knowledge base is assumed to encode a network of users in which diffusion processes of interest take place. This leads to two sub-modules:

Ontological Reasoning: Stores the knowledge database both for background knowledge (ontological data, rules, and constraints) as well as for network knowledge (nodes, edges, and their attributes). The general HEIST module in this case is instantiated with the language chosen of existential rules (also commonly referred to as Datalog+/–) [17].3

This combination of data-driven and KR-based models is very powerful for achieving explainability, since it allows to understand the logic behind the outputs that the query answering component can provide. There are many ways in which explanations can be derived; here, we consider two main approaches, which can also be combined. First, as mentioned above, having access to the ontological knowledge is often a useful starting point in understanding the system’s outputs. Second, it may be necessary to access dynamic aspects of the network diffusion process, such as the network state. In Section 4, we go into further detail by proposing several types of explanations that can be generated based on these components, particularly taking advantage of combining both knowledge- and data-driven explanation techniques.

In this section, we instantiate HEIST/NetDER to derive a system for detecting hate speech in social platforms (an overview of this instantiation is shown in Fig. 1-B); we call the resulting system NetDER-HS. We describe a simple use case where we have a typical social platform with a set of users that share information by posting and reposting content, or replying to posts from other users. In particular, we define a set of rules in the Ontological Reasoning module to work in a hate speech detection setting. Towards this end, we need to provide the details of the main components; as a first step, we assume that the Data Ingestion module feeds data into the ABox (ontological database) as well as the encoding of the network based on data about:

We assume that the Data Ingestion module feeds the database available in the Symbolic Reasoning module with data in the following schema:

Note that hs_level / 2 , hs_context_level / 2 , and offensive_language / 2 are atoms derived from external data-driven classifiers (such as the tools developed by [59] and [28]). Hence, these can be seen as wrappers for a call to a sub-symbolic service. This easily allows to tune different thresholds required in the rules to be instantiated, replace one classifier with another, combine different classifiers (ensembles), and use machine learning-based predictors to solve specific tasks.

OPEN IN VIEWER

Fig. 2.

Rules in the Ontological Reasoning Module. Underlined predicates correspond to wrappers for an underlying sub-symbolic service.

The Ontological Reasoning sub-module contains the logical rules shown in Fig. 2 (underlined predicates correspond to wrappers for sub-symbolic services, as discussed above); we provide intuitive explanations for each rule next. Roughly speaking, rule r 1 states that given a post that the classifier detects as offensive with confidence level L (greater than the threshold 0.5) and is targeted to a person or group of people, a hypothesis is created that P is hate speech with high danger level (red); on the other hand, r 2 expresses that a hypothesis that P is hate speech with medium danger level (orange) is created if P is an offensive post (that is, the targeting condition is removed) with a lower threshold for its confidence level. Rule r 3 generates a hypothesis that P is hate speech with danger level red if P is detected as hate speech with confidence L > 0.5 ; rule r 4 is a variant that uses a context-based classifier when L > 0.2 and L < 0.5 . If the user who posted P is suspected to be a hater, then rule r 5 generates hypotheses for the existence of hate speech with lower danger level (yellow). Observe that the sixth and eighth rules produce hypotheses stating the existence of haters; being the user UID responsible for disseminating two hate speech posts is enough for generating the hypothesis that UID is a hater, whereas having been repeatedly banned is also a reason for generating the hypothesis that UID is a hater. Finally, the seventh and ninth rules can be read as: “If post P is suspected to contain hate speech (with any danger level), and user UID is considered to be an early poster of P that is flagged as viral on the network (with confidence at least 0.7), then there exists a hypothesis that UID is responsible for disseminating hate speech”, and “the hypothesis that UID is repeatedly banned is created if the number of bans for UID ’s account is above the average number of bans per user”.

For the Network Diffusion sub-module, we could implement a set of logical diffusion rules with the purpose of guiding the evolution of (uncertain) knowledge about the network; an example of a language for expressing such rules is MANCaLog [60]. For this particular example, Fig. 2 shows two diffusion rules, loc_rule 1 and glob_rule 1 , where the former is a local diffusion rule that intuitively states “users that are not popular with certainty and have popular neighbors who reposted P with certainty, will update the belief that P should be reposted, according to influence function if ”. Here, we assume that function if always returns the interval [ 1 , 1 ] , meaning that P is likely to be reposted by the node in the next time point. Likewise, glob_rule 1 is a global diffusion rule that allows us to flag a post as viral based on the forecasting of reposts for each node. Note that this rule involves a function avg that will be used to forecast the viral state of each post and it returns the required value by computing an average for the values associated with each of the node’s reposts. We refer the reader to [48] and [49] for further discussions about diffusion rules.

Both the ontological KB and network database can be useful for the construction of complex explanations for the outputs generated by NetDER-based systems. In the next section, we will study several explanation types, with a special focus on explainability in NetDER. The NetDER-HS system will be adopted as the use case to show concrete examples; in Section 4.6, we extend this use case to cyberbulllying scenarios.

4.1. Build your own explanation

OPEN IN VIEWER

Fig. 3.

Wireframe of the interface for the proposed explanation customization process.

Figure 3 illustrates in a wireframe layout a general graphical representation of a process for choosing custom-built explanations. This process can be outlined as a two-step sequence:

Select the information sources to be used in building the explanation. This choice may include a single source or all of them. This is a first filter over the explanation styles that will be available in Step 2.

We now study the range of possibilities for leveraging the different components of an ISTS based on the HEIST application framework. We consider four possible knowledge sources and their associated explanations: Terminological-based (T), Assertional-based (A), Process-focused (P), and Result-focused (R); a fifth option, which combines the first two, is the Full Ontological (FO) source. Next, we describe them in more detail.

Terminological-based (T) explanations Explanations may benefit from extracting information and insights contained in the ontological rules/axioms; leveraging this source allows us to understand why the system provides an answer for a specific query by highlighting the rules involved in its derivation. For example, in a social platform setting like the one in the previous section, pointing to the specific rules used in reaching a conclusion that a user is suspected to be a “hater” may be a useful way to convey to users why this inference was made.

Assertional-based (A) explanations Atomic formulas (or assertional axioms) are basic objects that in our context refer to elements present in both the ontological and network databases. Choosing this source thus allows to incorporate elements from this basic set which, as we will see, is often connected to several explanation styles arising in data-driven AI techniques. For example, an explanation may use the data available on a social platform user to show why they are classified as potentially malicious.

Full Ontological-based (FO) explanations These explanations are fed from both atomic formulas and ontological rules. In this context, this source allows leveraging the application of a greater number of explanation styles since it is possible to exploit more information to be presented to users. For example, a trace-based explanation providing information (in form of rule trace) about inference process or network knowledge-based explanations may benefit from this type of knowledge source.

Process-focused (P) explanations In this case, explanations are designed with the aim of shedding light on how the network model evolves as the execution of the diffusion process progresses from the initial state to the final state (when the model is ready to make a prediction). For example, if a user posts a comment containing offensive language, a rule may be fired incrementing the probability that the user is a hater (as in the use case above), producing an evolution of the network model modifying the initial state. Then, having a user with this probability above a certain threshold may fire another rule issuing a prediction that the user will likely post such content again in the future, and should be flagged with a first warning. Explanations of this kind would then highlight the rules used to update the network model during the process; in general, if the diffusion process is not rule-based, explanations can highlight the transitions that the model goes through.

Result-focused (R) explanations These explanations focus on the final result of the diffusion process over the network, which can be seen as a forecast of the network’s evolution over time. For example, an explanation as to why a post has been flagged may highlight how the post will “go viral” by showing the number of users it is expected to reach in the next three days.

In the next two sections, we describe in detail how single-style and combined-style explanations arise in the context of the seven basic styles4

OPEN IN VIEWER

Fig. 4.

Sources available to each primitive explanation style. Checkmarks indicate availability, crosses indicate non-availability, and question marks are used when sources are potentially available (exploring the specific conditions is outside the scope of this work).

We now present additional details regarding the primitive explanation styles discussed in Section 2.2, providing initial details on each primitive style and examples in each case depending on the sources that are available, as presented in Fig. 4.

Contrastive explanation Consider the following elements involved in this explanation style:

A NetDER knowledge base KB = ( D , G , Σ , P ) .

OPEN IN VIEWER

Fig. 5.

Combined explanation styles against the different types of knowledge sources that can be used in explanations. Check marks indicate that it is possible to build the explanation applying the styles in any order, while “≺” indicates that a specific order must be followed. Finally, × and ? indicate that the combination of styles and source is not possible (or not fully explored), respectively.

Note that function similar ( Q 1 , Q 2 ) requires implementation in order to yield a similarity value given two queries (cf. Class ContrastiveStyle in Section 5).

Example: Consider the application domain introduced in Section 3.1, and the user query hyp_hater ( john ) , which has answer Yes. Assuming that similar ( hyp_hater ( john ) , hyp_hater ( paul ) ) holds, the contrastive explanation is: ⟨ ( hyp_hater ( john ) , Yes ) , ( hyp_hater ( paul ) , Yes ) ) ⟩ , which can intuitively be interpreted as “user John is a hater because user Paul is similar to John, and he is a hater, too”.

Counterfactual explanation This style yields explanations built using knowledge that is not included in any knowledge source. Concretely, consider a Boolean query Q, and a NetDER knowledge base KB = ( D , G , Σ , P ) . A counterfactual explanation for an answer to Q over KB is based on both available and not available knowledge; that is, it produces KB ′ = ( D ′ , G ′ , Σ ′ , P ′ ) by changing elements in KB (such as data and/or rules) to be used to answer Q.

Example: Continuing with the example illustrated in Fig. 2, let hyp_hatespeech ( post 1 , yellow ) be a query, which we assume has answer “No”. A counterfactual explanation to this answer can be generated by adding new information to the knowledge base, such as the fact that a user who posted post 1 is identified as a hater. Furthermore, suppose that with this new information rule r 5 can now be applied and the answer for our query now becomes “Yes”.5

Justification-based explanations Suppose we have a Boolean NetDER query Q, a NetDER knowledge base KB = ( D , G , Σ , P ) , and that the answer for Q over KB is “Yes”. Then, a justification-based explanation for this answer is another NetDER knowledge base KB ′ = ( D ′ , G ′ , Σ ′ , P ′ ) such that:

D ′ ⊆ D ,

Example: Consider the ontological rules presented in Fig. 2, and let hyp_hatespeech ( post 1 , red ) be a Boolean query. Suppose we can create a hypothesis for hyp_hatespeech ( post 1 , red ) , which can be obtained by applying rule  r 1 . From our definition of justification-based explanation, we can generate an explanation with r 1 and the set of atoms { post 1 , offensive_language ( post 1 , 0.6 ) , targeted ( post 1 ) } , which we assume holds. From the point of view of an ontology engineer, this type of explanation would for instance help carry out debugging tasks by pinpointing what causes hyp_hatespeech ( post 1 , red ) to hold.

Data-driven explanations In this context, we have the following central elements:

A NetDER knowledge base KB = ( D , G , Σ , P ) over the relational schema R .

As shown in Fig. 4, this primitive style is only available for the assertional source.

Example: We can assume that a data-driven explanation request is posed in order to know the reasons behind the answer “Yes” for the query offensive_language ( post , level ) ∧ ( level > 0.5 ) , where post is a value based on textual content using offensive language with confidence value level predicted by the sub-symbolic service s 1 . Hence, a data-driven explanation could rely on a function f obtained from variants of the entity offensive_language and the result of predictions for these variants made by s 1 ; in this case, it can be used in the provided explanation in order to unveil the most important words involved in the assertion “ offensive_language is used in post ”.

Statistical explanation This kind of explanation is likely the most general and flexible of the primitive styles we consider, since it allows to summarize in a numeric way the reasons involved in obtaining a specific output, making it possible to apply them in a variety of situations. Consider a Boolean NetDER query Q and a NetDER knowledge base KB = ( D , G , Σ , P ) ; we say that a statistical explanation of an answer to Q over KB is an explanation based on numeric indicators from functions that summarize the occurrence of data related to the answers of Q over KB . Examples of such functions required for these explanations are the well-known aggregation functions such as max, min, classical statistical functions such as mean, median, standard deviation, as well as more complex ones.

Example: Consider again the query hyp_hater ( paul ) , and assume that the answer is “Yes”. Examples of these explanations can include outputs of functions used to calculate the number of occurrence of posts, bans, comments, and specific connections between users such as follow, friendship, likes, etc. More concretely, network atoms associated with the current network state (encoded by atoms as node, edge, posted, among others) offer interesting alternatives when computing such statistics.

Simulation-based explanation Assume a Boolean NetDER query Q and a knowledge base KB = ( D , G , Σ , P ) ; a simulation-based explanation of an answer to Q over KB is built based on imitating the evolution over time of a network diffusion process that provides insight as to why it is possible to answer Q using KB . As indicated in Fig. 4, this primitive style is only available with the process- and result-focused sources.

Example: An explanation based on a forecasting model about the evolution of content in the network may explain why a post has been flagged as viral. Consider the diffusion rule diff_rule 1 presented in Fig. 2; a possible explanation may be to show how a diffusion process models the future spread of hate content after three days, indicating estimates of the percentage of users that will repost such content.

Contextual explanation Consider a Boolean NetDER query Q and a NetDER knowledge base KB = ( D , G , Σ , P ) ; a contextual explanation of an answer to Q over KB is built including knowledge that does not participate in the computation. The explanation may thus contain knowledge that, while not used to answer Q over KB , is otherwise related to the set of answers.

Example: Considering the previous explanation scenario, the information about users that reposted hate content could serve as a contextual explanation. This may include additional information about them (such as location, date of last post, etc.), and the users’ closest network relations.

In requesting an explanation, our proposal provides facilities for generating explanations that could combine several styles and knowledge sources. In order to better understand these combinations, we carry out here a preliminary analytical study about which explanations can be built using the styles and sources addressed in this paper. Our main findings are summarized in Fig. 5. Note that we do not consider all possible combinations between available sources and explanation styles; in these first steps towards understanding combined explanations, we analyze how to combine two single-style explanations at a time.

Different approaches could be developed for combining explanation styles; this is in general not a simple endeavor due to several characteristics, such as contemplating the combined explanation’s goal, stakeholders, application domain, among others. For the purposes of this work, we identify three basic options:

Policy 1 can be applied by taking the information from Fig. 4 and checking if all the chosen styles are compatible with the selected sources; this is the policy we include in our initial implementation (cf. Section 5). We now provide a preliminary analysis of the feasibility of applying Policy 2 with two styles; Policy 3 requires an analysis at a lower level of abstraction, and will be addressed in future work.

4.5. Combining two different styles

In order to apply Policy 2, we need to understand what combinations are useful and comprehensible to make sense of the resulting explanation. In the following, we discuss the central aspects that emerge from considering the pairwise combinations shown in Fig. 5; in some cases, the scope of the discussion may be limited by the fact that concrete implementation details of each particular style, as well as the explanation generation process, are domain-specific. Our discussion will be organized according to the blocks in which the table in Fig. 5 is separated.

We begin with an analysis of three groups of combinations that share an important feature that causes order to be important; in particular, in each case there is a primitive style that generates elements to be used in the explanation that can later naturally be used as inputs to other styles:

Counterfactual + another primitive style (Styles 8–13): The counterfactual style involves deriving another knowledge base, which can then be taken as input for the other style.

Styles 23–27: In this group, we have four possibilities. In the first two, we have Data-driven with either Statistical or Contextual styles; for the former, the most natural order is to first apply the Data-driven style, which can then be fed to the Statistical style, while for the latter, both orders could in principle be applied: Data-driven ≺ Contextual, or the other way around. Similarly, for Style 26, the Statistical style is most naturally applied last, while in the case of Style 27 both orders are possible. For an illustrative example of style 23 (Data-driven + Statistical), consider the example from Section 4.3 (cf. Page 1003) where the user requests a data-driven explanation for offensive_language ( post , level ) ∧ ( level > 0.5 ) and is shown the set of most relevant words involved in that assessment. If an additional Statistical explanation is requested, the system may provide the information regarding the percentage of times in which posts containing those words were correctly classified as offensive.

Finally, in Style 28 the most sensible order is Simulation-based ≺ Contextual, given that a simulation may be enhanced by contextual information.

Having presented the set of primitive and combined explanation styles, we now come back to the running example and present another possible application in the domain of detecting unwanted behavior in social platforms.

4.6. A cyberbullying detection use case

In order to further illustrate how the different explanation styles offered in HEIST can be leveraged in a real-world setting, we now develop an additional example to show how the NetDER-HS system can be extended to be used in a simple cyberbullying detection case. More concretely, we present a set of additional rules and then discuss how several explanations can be derived so that domain experts can get a clearer understanding of outputs generated by HEIST-based systems.

As a first step, we extend our hate speech example already presented in Section 3.2 with new rules. The following rules drive the generation of hypotheses for online cyberbullying behavior:

Having introduced how the knowledge base in the system’s Symbolic Reasoning Module is set up, we now focus on how explanation sources and styles (as presented in Fig. 3) can be used to build different explanations for a specific query. We contemplate how a range of such different explanation requests can be addressed; as discussed in Section 4.4, we assume that the system follows Policy 2, which solves the problem of combining explanation styles by sequentially applying the chosen ones (the output of one becomes the input to the next).

The first step is to choose the information source to be used; independently of the selected styles, in this particular example we assume that users always choose to use all of the available sources in building explanations. The second step is to choose the specific explanation style(s) that will be presented. We now briefly discuss how several different explanations can be generated by combining specific pairs of styles.

Counterfactual + another primitive style Suppose that we are interested in knowing whether user Paul is a victim of bullying posts, and let’s assume that the answer to the corresponding query hyp_bullying ( paul ) is “No”. As shown in Section 4.3, building counterfactual explanations involves considering information that is not present in the knowledge base. In this case, the negative answer to our query implies that rules r 10 – r 12 are not applied, so possible counterfactual explanations would require satisfying at least one of their bodies; for instance, a user hiding behind a fake identity could post a fake news article with content about Paul, leading to rule r 12 firing and the answer changing to Yes. An explanation arising in this manner could be:

“There would be a hypothesis for hyp_bullying ( paul ) if it were the case that there is a user suspected of having a fake profile and posting a fake news article about Paul”.

Finally, we may wish to obtain a contrastive explanation in connection with our counterfactual explanation. In this case, ⟨ ( fake_profile ( mary ) , Yes ) , ( fake_profile ( john ) , Yes ) ) ⟩ could be considered a contrastive explanation that is read as follows: “User Mary often uses fake profiles because user Paul is similar to Mary, and he uses fake profiles, too”.

Contrastive + another primitive style Suppose the user selects combined explanation style 9 (Contrastive + Statistical). Considering the contrastive explanation presented above, statistical explanations for fake_profile ( mary ) can include for instance outputs of functions based on the number of posts by Mary and how many followers she has. If a simulation-based explanation is required, it may be generated via the use of the network diffusion simulation model to forecast who will participate in the diffusion of Paul and Mary’s posts for the next 5 days.

Justification-based + Data-driven Suppose a hypothesis hyp_bullying ( mary ) is derived by applying rule r 10 , and so the answer to our query is “Yes”. In the case that a justification-based explanation built with solely atomic formulas was required, the system can build an explanation with the atoms6

Data-driven, Statistical, Simulation-based, and Contextual Consider again the data-driven explanation presented above; if explanation style 25 (Data-Driven + Contextual) is required, the system may provide information related to post post 1 as a network knowledge-based contextual explanation, such as number of reposts, number of humiliating words, and relationship between users reposting post 1 . In the case of contextual explanations derived from a statistical explanation (style 27), an interesting example may include information regarding the users that arise from analyzing the percentage of anonymous users reposting hostile messages (statistical explanation). Finally, suppose we create a simulation-based explanation that is built leveraging information regarding a diffusion forecasting model of Paul and Mary’s posts (cf. the example of Contrastive + Simulation-based presented above); then, the system may show the number of suspected anonymous users as contextual explanatory information.

Statistical + Simulation-based As discussed above, the most natural order when combining these primitive styles involves first obtaining a simulation-based explanation. Suppose we generate a hypothesis for hyp_bullying ( mary ) , which is derived by applying rule r 11 . In this case, we assume that there exists a post targeting Mary that contains hateful comments. Suppose we create the simulation-based explanation illustrated in Section 4.3; then, the percentage of suspected anonymous users that have reposted hate speech posts may serve as contextual explanation.

Finally, with this use case illustrating additional examples, we have shown how our approach can be used in a different domain requiring supporting information for different query answers, which highlights the capacity of our approach to flexibly embed different types of explanations. Note that concerns regarding the design of interfaces to present explanations to users are outside of scope of this work; nevertheless, it is important to note that there have been many R&D efforts towards this direction. While there are common approaches that communicate explanation structures with textual explanations or simple lists of elements, complex concepts are often better explained (and deeper insights are obtained) with richer visualization techniques, or via a combination of visualizations and simpler presentations.

Figure 6 shows a high-level view of the class diagram for the HEIST framework (we include the detailed version of this diagram in the Appendix); this diagram corresponds to a more detailed view of Fig. 1 (A). In the following, we describe the most important links between the classes, which appear as relations in the diagram. For the purposes of this work, we assume that the symbolic reasoning module will be implemented as a rule-based system (i.e., the knowledge base is comprised of a set of rules and a set of atomic formulas referred to as a database).

One of the main components in HEIST is the Data Ingestion module, which is in charge of creating the knowledge base and also communicates with the Sub-symbolic Services module in order to both train new data-driven tools and make use of others already in place. The Symbolic Reasoning module implements a series of reasoning tasks, each of which involves a specific inference mechanism that functions by leveraging specific knowledge sources. Note that the examples in this paper are based on a use case in which existential rules are used as the ontology language; however, HEIST allows the use of other languages, such as the well-known Web Ontology Language (OWL) or others by instantiating the relevant classes accordingly (cf. Figs 6 and 9). The Query Answering module is in charge of delivering the main system functionality, which depends primarily on the Symbolic Reasoning module (which, in turn, is supported by the Sub-ssymbolic Services). The main client of the Query Answering module is the User, who issues queries and may also request explanations for their answers. In that case, the Explanations module interacts with: (i) the Query Answering module, in order to obtain the available sources of knowledge; (ii) the Sub-symbolic Services module, to have access to the sub-modules that are related to the query answers; and (iii) the User, to allow them to choose the source(s) and style(s) for the requested explanation.

Finally, in designing this class structure, the Strategy design pattern was applied for the ReasoningTask, Explanation, Source, and Style classes. For the StyleVisitorFactory class, a combination of the AbstractFactory and Visitor patterns were applied.

OPEN IN VIEWER

Fig. 7.

High-level view of the class structure of the NetDER instantiation of HEIST.

Figure 7 shows the high-level class diagram for the instantiation of HEIST that produces the NetDER model; this diagram corresponds to a more detailed view of Fig. 1 (B). Compared to the previous section’s diagram, here we can see that this model includes the result of making several implementation choices:

Reasoning Tasks: Chase (for query answering with existential rules) and Diffusion (which is used to represent and query diffusion processes in networked data).

5.3. The explanation request workflow

Figure 8 shows interaction diagrams for the main activity of interest for this paper: the explanation request issued by a user. The diagram at the top shows the main steps: a user carries out an explanation request for a query answer, which implies that a User object receives a message that invokes the request(ans,q) method, and then the sources are chosen in order to provide the necessary ingredients to build the explanations. Then, the explanation styles are chosen, which allow us to define the particular nature of the required explanation.

OPEN IN VIEWER

Fig. 8.

Interaction diagram for the explanation request activity.

The diagram at the bottom of Fig. 8 shows a more detailed view of what goes on in the explain(ans, q) method in ExplanationsModule: after sources and styles are chosen, an ExplanationsModule object receives an invocation of its explain(ans,q) method associated with a query and an answer. The StyleVisitorFactory classes relevant to the user’s choices are then in charge of creating the necessary style objects; the chosen knowledge sources also affect the process by linking the relevant source objects; this essentially implements the source-style associations illustrated in Fig. 4. After these steps, the ExplanationsModule object will request from each style object the associated explanations, which are returned to the user as a collection of Explanation objects.

6.1. Knowledge-based explanations

Knowledge-based systems were conceived as tools to support human decision-making, and generally consist of a knowledge base encoding the domain knowledge, usually modeled as a set of rules, a database, and a reasoner that makes use of these components to answer user queries. Though traditionally logic-based reasoning formalisms have been touted as more closely related to the way humans reason, and therefore more inherently explainable, different kinds of explanations are essential to the interaction between users and knowledge-based systems, as discussed in this paper. Thus, questions like what a system does, how it works, and why its actions are appropriate may not be immediately answerable even for logic-based approaches.

Many works, such as [11,35,71], focus on explaining a system’s decisions from a range of several different explanations types, providing various levels of assistance. In this regard, when requiring domain information to explain a decision, structured forms of domain information like ontologies are in many cases particularly well suited. For example, there has been an important body of research directed towards the area of explanation and ontology debugging related to the Web Ontology Language (OWL); in particular, work has focused on a specific type of explanation called justifications [35,37,58]. Approaches like [36] have focused on implementation aspects by presenting libraries and computational tools for working with justification-based explanations of entailments in OWL ontologies. The main focus of research in this area has been on explanation for the sake of debugging problematic entailments, such as class unsatisfiability or ontology inconsistency. Other approaches regard proofs, which can contain intermediate steps between a justification and the conclusion, as a means of providing more detailed explanations [3,9,10]. Recent successes in machine learning technologies [21] have brought more attention to explainability, especially on how to integrate and use domain knowledge to drive the explanation delivery process and how this influences the understanding of explanations by users. Our proposal is part of the same general effort to enhance logic-based formalisms with knowledge obtained from the use of machine learning tools. In this regard, examples of advantages of using HEIST include adaptability and interoperability with data models implemented as a service in the sub-symbolic module. Other specific works on knowledge-based explainability include approaches based on argumentation [2] and answer set programming [15].

6.2. Data-driven and hybrid explanations

In the last decade or so, with the availability of vast volumes of data, machine learning algorithms have seen a boom in popularity; however, these models are typically seen as difficult to understand, and there has thus been a significant focus on making them explainable. While some models are considered explainable by design (such as decision trees), others do not reveal sufficient details about how their outputs are generated, resulting in an opaque (also referred to as black box) decision model. This has led to the development of a variety of approaches for deriving explanations of opaque models, for both fully autonomous and human-in-the-loop systems. The literature on data-driven XAI is vast, and although there is no clear general consensus on what constitutes a good explanation, many taxonomies commonly used to classify explainability methods cover the following dimensions:

Method: There are two main types of methods to provide explanations. Local ones give explanations via individual instances or groups of nearby instances, while global ones describe the behavior of models as a whole.

Artificial Intelligence efforts have recently evolved to hybrid systems that employ both data-driven and logical reasoning techniques. With increasing adoption of these systems, there is a need for approaches that tackle explainability as a primary consideration, fitting end user needs to specific explanation types and the system’s AI capabilities. Recent works have focused on developing approaches that focus on combining data-driven and symbolic reasoning methods for capturing explainability of outputs [16,70]. Following this idea, [19] presents an Explanation Ontology that treats explanations as a primary component of AI system design, providing a semantic representation that can help system designers to capture and structure the various components necessary to enable the generation and composition of hybrid user-centric explanations in their systems. The ontology captures several classes and properties from [66], where the authors proposed an Explanation Ontology Design Pattern to represent explanations. Note that most of the primitive styles that we adopt in this paper are captured in that ontology; incorporating other styles from that model is a possibility that is being explored for future versions of HEIST. Another interesting work is that of [47], which shows an extension of the Explanation Ontology to model explanations in the food domain. Other works in this vein include [24], in which the authors provide a hybrid explanation method based on decision trees to explain predictions that applies hidden-layer-clustering [27] on a deep neural network. Recently, [38] describes a hybrid neuro-symbolic reasoner, and their implementation in a distributed task-based framework, and explores building of deductive proofs/explanation graphs for questions regarding story understanding. Examples of other research efforts on integrating symbolic methods with Neural Networks include [40,56].

The closest work in spirit to our approach is the work mentioned above by Chari et al. [19]; we therefore now discuss the relationship between their work and our own in more detail. Chari et al. also aim at combining hybrid explanation types by using different forms of knowledge; however, whereas their approach develops an ontology at a higher level of abstraction (which covers the AI system, the users, and the interface between the two, where the explanations lie), we on the other hand leverage ontologies at the object level of the system – i.e., we assume that the software being developed has a knowledge base comprised of an ontology (ABox + TBox in Description Logic or other ontological language), and explanations are built using elements from those sets as well as possibly other sources in a sub-symbolic services module. Our contributions center on the systems side, since we develop an application framework (HEIST) for knowledge-based intelligent systems that incorporate XAI capabilities; as discussed above, part of the design of HEIST incorporates primitive explanation types that are also used by Chari et al. In summary, the main difference with our proposal is that our work does not focus on introducing a user-centered general-purpose explanation approach, but rather on: (i) analyzing several explanation types for NetDER, which is an architecture designed for the development of systems capable of detecting malicious behavior in social platforms; and (ii) implementing HEIST (proposed up to the design stage in [50]), a general application framework that helps system designers build explainable socio-technical systems in any domain.

Finally, there are several similarities and differences between our work and the other approaches mentioned above. In this setting, our approach can be valuable for building systems with hybrid reasoning mechanisms capable of generating different explanation types, including combined types like the ones explored, and allowing human-in-the-loop interactions. We provide the basic building blocks for the implementation of explainable intelligent socio-technical systems based on NetDER, a particular case of the HEIST application framework, which are documented here via class and interaction diagrams and have the corresponding publicly-available source code. Despite their direct application to real-world social platforms, mechanisms to deal with network knowledge-based explanations have not been extensively studied in the literature.

6.3. XAI in cybersecurity

Explanations are essential to decision-making support, and more importantly when these systems handle critical information (such as in cybersecurity, health, or financial systems, among others). However, XAI in cybersecurity has not received much attention in the literature; we now briefly review differences and similarities between the most salient works related to our approach.

Several topics and future challenges to be explored were discussed by Viganò and Magazzeni [69]; their focus is on stressing questions that should be taken into consideration, such as who gives/receives the explanations, what constitutes a good explanation, when the explanation should be given, how to give explanations, where the explanations should be made available, and why the explanations are needed (as discussed in Section 2). In [33], Holder et al. present a use case for capturing these issues, seeking a guide for the development of future explainable systems that users can leverage. Other works are part of a recent surge of research on enhancing the explainability of machine learning techniques; in our domain of interest in particular, Mahdavifar and Ghorbani [43] propose to do this via rule extraction from a trained DNN, which is then used to add explanation capabilities for the detection of cyber threats. In [65], Szczepanski et al. develop and evaluate an Explainable Intrusion Detection System based on a combination of ML tools via a microaggregation heuristic. A particularly closely related work is the one reported by Kuppa et al. in [39], which centers on understanding the security robustness of explainable methods by introducing a taxonomy for XAI methods, capturing various security properties and threat models relevant to the cybersecurity domain. Motivated by this, the authors then present a black-box attack to study the consistency, correctness, and confidence security properties of gradient-based XAI methods. Reviews of recent developments in the area of XAI in cybersecurity are presented in [32,63].

Our line of research takes initial steps towards the implementation of explainable software tools based on the HEIST application framework in general, and the NetDER model in particular, providing decision support via the leveraging of both ontological knowledge and network diffusion processes. To the best of our knowledge, there have not been other works of this kind to date.

Finally, though in this paper we focus on applications related to cybersecurity, the HEIST framework (and NetDER-based implementations) can find application in many other domains that are quite far from the ones explored here, such as digital history and cultural heritage; we refer the reader to [55] and [68] for recent works in these areas. In [68], the authors state that the explainability of heterogeneous AI-driven art-related systems is a challenging and promising task notably because this functionality would allow for a better understanding of the system’s outputs by artists who could also interact with the knowledge base in order to tune the performance of the underlying machine learning mechanisms. Furthermore, due to ever-increasing adoption of intelligent systems in critical fields such healthcare and finance, researchers have recognized that symbolic reasoning approaches have potential in the general effort to mitigate opacity and to make data-driven AI models more explainable [18]. There is clearly much work to be done in this direction, but preliminary results are promising.

7.1. Limitations

There are several limitations that can be identified in our work. The version of HEIST being presented here aims to be as general as possible, but the code that we developed in this first version has been developed as a generalization of our previous work on NetDER, which uses Datalog+/– as the underlying language; therefore, further work needs to be done in order to gather experience with other languages.

Focusing on the XAI functionalities, the current version of HEIST only supports multi-style explanations applying Policy 1 (i,e„obtaining each style independently – cf. Section 4.5). As we have shown in our preliminary analysis of combining styles in pairs, the problem is quite challenging; developing more complex multi-style explanation delivery mechanisms requires further study. Another aspect that we have not addressed in this work is that of how to best present explanations to users depending on their type. In recent work [50], we have taken first steps in analyzing this problem in terms of three general types of users: basic, power, and analyst, showing possible ways in which functionalities depend on user capabilities. Such functionalities have not yet been incorporated into HEIST; this involves further developing several of the components of the model illustrated in Fig. 6.

Other limitations are more closely related with the scope of the tool, given that there are many challenges associated with developing large-scale intelligent socio-technical systems. In the following we briefly describe and discuss such challenges.

7.2. Challenges

Seen from a software engineering perspective, intelligent socio-technical systems are information systems that can range in complexity from very simple (like a mobile app that applies filters to pictures and posts them to social media) to highly complex (like a monitoring system that combines feeds from many social media platforms and detects a range of possible malicious activities). There are many well-known software engineering challenges involved in building complex systems, and the level of hurdle that such challenges pose depends on many factors, such as number and types of users, number and type of data sources, impact of failures, and performance requirements, among others. In order to discuss such challenges in greater detail, we continue adopting as a general setting the cybersecurity domain, as we have done in this work and others in this line of research [48,50,61] – this domain exhibits all such characteristics since it covers a wide range of applications. Some of the main challenges that arise as systems become more complex along the dimensions described above are the following; note that they are often interrelated in real-world applications:

Data and knowledge management: There are many challenges associated with adequately manipulating data at large scale, such as cleaning, curation, and integration. Dealing with multiple sources of potentially incomplete, inconsistent, or otherwise noisy data requires efforts that are closely related to the specific application being developed and the impact on users that decisions have. Regarding knowledge management, such activities often require the involvement of experts that are familiar with the application domains and who must work with other experts and automated tools in order to ensure that knowledge is manipulated in a principled manner.