Towards semantically enriched embeddings for knowledge graph completion

1. Introduction

Knowledge Graphs (KGs) have recently gained attention due to their applicability to diverse fields of research, ranging from knowledge management, representation, and reasoning to learning representations over KGs. KGs represent knowledge in the form of relations between entities structured as (head, relation, tail) triples, referred to as facts, along with schematic information in the form of ontologies. KGs have been used for various downstream tasks such as web search, recommender systems, and question answering.

KGs, however, suffer from incompleteness because of manual or automated generation. Manual generation leads to limited knowledge represented by the curator and contains curator bias [56], while automated generation may lead to erroneous or missing information. KG completion in particular includes the tasks of (i) triple classification, i.e., deciding if the triple is true or not, (ii) Link Prediction (LP) to complete the head, tail, or relation in a triple, and (iii) entity classification, also known as entity type prediction. To perform KG completion, various rule-based as well as embeddings based models have been proposed. These algorithms are computationally expensive and are transductive: they only perform predictions based on triple information involving known entities and relations. This is not readily usable when the inference has to be performed on unseen entities and relations. To this end, inductive KG completion allows for predicting triples that involve unseen entities and relations. These transductive and inductive LP algorithms are mostly based on factual information contained in KGs. Various studies leveraging language models as an external source of knowledge have been proposed for KG completion. These algorithms lag behind in terms of performance w.r.t. KG embedding based methods because of ranking-based metrics such as Hits@k since existing KG embedding-based algorithms operate under the Closed World Assumption. This led to the need for human evaluation since Large Language Models (LLMs) contain more general knowledge and may generate correct answers that are different from what is expected by the ground truth with the highest score. Apart from the factual information (i.e., Assertion Box, ABox), another source of information is the ontological information (i.e., Terminology Box, TBox) contained in the KG. The current methods almost completely ignore this TBox information. To rectify this situation various attempts have been made to include type hierarchies and ontological information with different expressivity levels such as E L + + , A L C , etc. In some cases, additional representational capabilities are utilized to capture this information such as box embeddings.

Related work Several studies have surveyed the state of the art (SoTA) in KG completion. The work by Paulheim [59] provides a survey of the articles related to KG refinement including various classical and rule-based approaches for KG completion, yet KG embeddings are not discussed. Other surveys specifically focus on KG embedding-based methods for KG completion. Wang et al. [79] organises the algorithms for embedding-based KG completion according to their scoring functions such as translational models, semantic matching models, etc. However, this survey does not discuss the methods proposed for KG completion using multimodal information related to an entity or relation such as images, text, and numeric literals. These aspects are targeted in the survey by Gesese et al. [29], which categorizes these methods based on their scoring function (inspired by the work by Wang et al. [79]) as well as based on multiple modalities. The survey shows theoretical and experimental comparisons of the existing approaches. Still, none of the aforementioned works cover the aspects of ontological statements and semantics in KG embeddings. Recent works [57,58] discuss the interplay of KGs and LLMs. The study by S. Pan et al. [58] describes methods that integrate LLMs for KG embeddings and KG completion, yet the role of schematic information is not considered in the anlaysis. Complementary, J. Z. Pan et al. [57] describe how LLMs can be used to complete triples and perform different ontology engineering tasks (e.g., completion or refinement) in KGs. In contrast to these works, our work focuses on KG embeddings and how LLMs can be used as external sources to improve the performance of KG completion tasks. In summary, as compared to these existing studies, the current article targets the semantic aspects of knowledge graph embeddings by summarizing and discussing the approaches that have been proposed so far for leveraging semantics provided in the KG.

Contributions This position paper provides an overview of the evolution of methodologies proposed for KG completion, starting from the embedding-based algorithms and LLM-based approaches to the various categories of algorithms proposed for incorporating schematic information within KG embeddings for performing different kinds of completion tasks. It further discusses the limitations of the approaches in each of these categories and concludes with critical reflections and future research directions.

2. Preliminaries: Semantics in knowledge graphs

Commonly, KGs are defined as a set of head-relation-tail triples ( h , r , t ) , where h and t are nodes in a graph, and r is a directed edge from h to t (e.g., [8,17,37,54]). In this way, a KG corresponds to a directed, labeled graph, where triples in the KG are labeled edges between labeled nodes, as captured in the following definition.

Definition 1 captures the graph structure of KGs, yet aspects concerning the meaning of nodes and edges in a KG are not explicitly defined. First, there is no distinction about the kind of nodes that can exist in a KG, i.e., whether nodes correspond to entities or text descriptions. This distinction exists in data models like the Resource Description Framework (RDF), where nodes can represent entities, labeled with IRIs or blank nodes, or literals that are used for values such as strings or different types of numbers like integers, float, etc. Distinguishing between entities and literals impacts both the structure and connectivity of nodes in the graph as well as the meaning of things in the graph. As we will see in Section 3.1, specialised KG embedding models are required to handle both kinds of nodes and different types of literals in a KG. Second, the semantics (formal meaning) of classes or relations in the KG are not provided in Definition 1. This is typically the role of ontologies in KGs, which specify the definitions for classes (or concepts) and relations (or roles) using labels or symbols coming from logic. Different logical languages introduce different levels of expressivity (cf. Table 1). Expressivity here refers to the allowed complexity of statements that go beyond simple assertions about individuals (e.g., class assertions) and their relations to other individuals or literals, that are comprised in the ABox. More expressive statements included in the TBox comprise class and role definitions. The constructs and axioms defined in a logic language L can be transformed into labels and triples to be encoded in a KG. Based on this, we provide a definition for KGs that takes into account the semantics of statements.

Typical logic languages used in KGs are in the family of Description Logics, due to their convenient trade-off between expressivity and scalability [65]. For example, the existential language E L defines the notions of concept intersection and full existential quantification. The extension E L + + introduces the additional notions of concept intersection and concept subsumption; the latter is necessary to model class hierarchies in KGs. A L C provides additional expressivity w.r.t. E L + + as it includes concept union, negation, and universal quantification. Higher levels of expressivity are captured in KGs modeled with formalisms defined for the semantic web like the Web Ontology Language (OWL). OWL is based on the S H language, which includes more complex definitions for roles, including role subsumption and transitivity. Another important aspect of KG semantics is the concept of data types captured in the ( D ) extension. ( D ) allows for modeling the meaning of literals in KGs which can be part of statements in the ABox. It is important to note that KGs based on OWL can achieve different levels of expressivity beyond the ones described in this section, e.g., OWL-Lite is based on S H I F ( D ) , OWL-DL on S H O I N ( D ) , and OWL2 on S R O I Q ( D ) , the latter providing definitions for role reflexivity, irreflexivity, and disjointness, inverse properties, enumerated classes, and qualified cardinality restrictions.

3. Knowledge graph completion using embeddings

The vast majority of embedding based algorithms for KG completion treat KGs as graph structures as presented in Definition 1. Most of these algorithms perform transductive link prediction (LP) [13] where the inference is performed on the same graph on which the model is trained. On the contrary, inductive LP is performed over the unseen graph, i.e., (parts of) the graph are not seen during training. This section gives an overview of KG embedding algorithms for transductive LP (Section 3.1), inductive LP algorithms (Section 3.2), and entity type prediction (Section 3.3) since the previous two categories mostly focus on head, tail, relation, and triple prediction.

4. Large language models for knowledge graph completion

LLMs can further be categorized into encoder-decoder models, encoder-only, and decoder-only models. The encoder-decoder models and encoder-only models, such as BERT [19], RoBERTa [44], etc., are masked language models which are discriminative models and are pretrained for predicting a masked word. These models have achieved SoTA performance on the NLP tasks such as entity recognition, sentiment analysis, etc. On the other hand, decoder-only models, such as LLaMa [75], ChatGPT [3], and GPT-4 [10], are autoregressive models which are generative and are pretrained on the task of predicting the next word. The rest of the subsection focuses on how these two categories of models have been utilized in the context of KG completion.

One of the pioneers of LLM-based approaches for KG completion is KG Bidirectional Encoder Representations from Transformer (KG-BERT) [98] which is fine-tuned on the task of KG completion and represents the triple as textual sequences. It takes the entity-relation-entity sequence and computes the scoring function using KG-BERT. The model represents entities and relations by their names or descriptions and takes the name/description word sequences as the input sentence of the BERT model for fine-tuning. Despite being an LLM-based approach, KG-BERT does not outperform models considering the structural information of a KG in terms of the ranking-based metrics, i.e., hits@k. Kim et al. [40] associates the shortcomings of KG-BERT with the fact that KG-BERT does not handle relations properly and it has difficulties in choosing the correct answer in the presence of lexically similar candidates. Therefore, Kim et al. propose a multitask learning based model [40] that combines the task of relation prediction and relevance ranking with LP to lead to better learning of the relational information.

The previously described methods still suffer from high overheads because of costly scoring functions and a lack of structured knowledge of the textual encoders. A Structured Augmented text Representation (StAR) model [77] was proposed where each triple is partitioned into two asymmetric parts, similar to a translation-based graph embedding approach. Both parts were encoded into contextualized representations with the help of a Siamese-style textual encoder. Yet, KG completion methods based on pretrained language models (PLM) lag in performance w.r.t. structure-based algorithms. Lv et al. [45] highlight that the reason underlying this lag is the evaluation setting, which is currently based on a Closed World Assumption (CWA), where an absent fact in the KG is considered false. Under the CWA, the performance of an LP algorithm is measured by its capability to predict a set of links that were removed from the KG. In contrast, LLMs introduce external knowledge which may lead to the prediction of new links that are semantically correct, but which were not in the original KG (and therefore not in the evaluation set) and, hence, not counted in the success metric. Additionally, the pretrained language models (PLMs) are utilized in an inappropriate manner, i.e., when triples are used as sentences it leads to incoherence in the generated sentences. Therefore, Lv et al. [45] present a PLM-based approach dubbed PKGC. This work targets the first problem by proposing manual annotation as an alternative. However, a medium-sized dataset containing 10,000 entities and 10,000 triples in the test set will lead to true labels of at most 200 million triples precluding human annotation. This observation leads to a new evaluation metric called C R @ 1 , where the triples are sampled from the test set and the missing entities are filled with the top-1 predicted entity. Manual annotation is then performed to measure the correct ratio of these triples. PKGC then addresses the second problem by converting each triple and its support information (i.e., the definition and the attribute information) into natural prompt sentences, which are fed to the PLM. PKGC outperforms structural and LLM-based methods in various modalities, i.e., with the attribute and definition.

GenKGC [88] converts the KG completion task to a sequence-to-sequence (Seq2Seq) generation task, where the input to the model is ⟨ h e a d , r e l a t i o n , ? ⟩ . Following the “in-context learning” paradigm of GPT-3, the authors select some samples consisting of several triples with the same relation as in the input from the training set which is termed a relation-guided demonstration. It introduces an entity-aware hierarchical decoder during generation for better representation learning and reduced time complexity. On traditional datasets such as WN18RR and FB15k-237, GenKGC underperforms as compared to structural SotA models on hits@k (confirming the hypothesis made By Lv et al. [45]) and outperforms masked language model based approaches demonstrating the capabilities of generative models for KG completion. The Seq2Seq Generative Framework KG-Completion algorithm (KG-S2S) [11] takes into account the aspect of emerging KGs. Given a KG query, KG-S2S directly generates the target entity text by fine-tuning the pre-trained language model. KG-S2S learns the input representations of the entities and relations using entity descriptions, soft prompts, and Seq2Seq dropout. The KG elements, i.e., entity, relations, and timestamp are considered flat text, enabling the model to generate novel entities for KGs. The method is further evaluated in static, few-shot, and temporal settings.

Discussion LLMs have revolutionized the natural language processing (NLP) landscape by their ability to generate coherent and relevant text. However, they are not without limitations. One of the major concerns with LLMs is their tendency to hallucinate [69], which means they may generate coherent yet entirely fictional information or associations. This poses a significant challenge in knowledge graph completion tasks, where the goal is to accurately predict missing links or facts.

Moreover, LLMs typically require extensive fine-tuning and adaptation to specific domains to perform effectively. This is prevalent in specialized domains such as biomedical, legal, or scientific fields, where terms and context can vary significantly from general language use. The LLM performance is also lagging in torso and tail entities, where the LLM has limited or sparse knowledge about certain entities during learning [70]. Therefore, while LLMs show promise in improving the quality of KG completion, they need to be augmented with additional information to cope with knowledge scarcity or with domain-specific resources to provide accurate and reliable results in such domains.

Regarding the evaluation of KG completion methods, it is important to recognize that existing benchmarking methodologies often rely on the Closed World Assumption (CWA). However, CWA-based evaluations may not be adequate for assessing methods that integrate external knowledge into the prediction process, such as LLMs. This is because such approaches could produce accurate predictions that may not be represented in the KG itself, leading to spurious false positives under the CWA. Thus, there is a growing need for novel experimental configurations and evaluation metrics that can accommodate more complex and nuanced assessments, such as considering the confidence or uncertainty levels of predictions or incorporating additional external knowledge sources to provide context.

In conclusion, while LLMs present a valuable resource for knowledge graph completion tasks, their inherent limitations, particularly in terms of hallucination and domain-specific adaptation, need to be addressed. Additionally, the development of more sophisticated evaluation methodologies that go beyond the CWA will be crucial to ensure a reliable understanding of the LLM capabilities for KG completion in various domains.

5. Towards capturing semantics in knowledge graph embeddings

This section provides an overview of the evaluation protocol including the benchmark datasets as well as the used evaluation metrics. Table 3 shows an overview of the evaluation tasks based on the categories of algorithms introduced in the rest of the paper along with the benchmark datasets.

Benchmark datasets The benchmark datasets vary from task to task, depending on the requirements of the algorithms. Most datasets are based on WordNet, YAGO, and Freebase which utilize only the triple information and do not incorporate the ontology by default. A detailed comparison of the datasets for LP in a transductive setting is provided by Gesese et al. [27]. The datasets used for link prediction (LP) typically provide training/validation sets; still, these splits are computed arbitrarily. Therefore, a fine-grained analysis of how the methods perform on different relations or classes is not always possible with these datasets. ELEm has been evaluated on the Protein-Protein Interaction (PPI) dataset for the LP task. However, the successor algorithms introduced datasets that consider more expressive description logic axioms such as Gene Ontology (GO) [41] or FoodOn [21]. In addition, some works [50,90] for ontology embedding models have further classified the test sets into normal forms (NF) to evaluate the models on subsumption prediction between atomic concepts (NF1), atomic concepts and conjunctions (NF2), and atomic concepts and existential restrictions (NF3 and NF4).

Metrics For the task of LP, evaluation protocols typically involve rank-based metrics. Most of the algorithms use hits@k where k varies from 1 to 100, and typically values are 1, 3, 10, 50. Other ranking metrics are Mean Rank (MR) and Mean Reciprocal Rank (MRR). Besides rank-based metrics, for the task of entity classification or entity type prediction, some works (e.g., [83]) report on metrics used for information retrieval, i.e., precision, recall, and their corresponding aggregates such as F1 scores. In addition, works on ontology embedding models [42,90,102] also report on the area under the curve (AUC) of the ROC curve.

Evaluation and reported results Most of the algorithms are evaluated on the task of LP since they consider only the triple structure in the KG. Exceptions to this are the tasks that are used to evaluate the algorithms, incorporating description logic axioms. In the case of transductive LP, studies have shown that the efficiency of these algorithms greatly depends on the choice of hyperparameters. For example, Ruffinelli et al. [66] show that RESCAL, one of the earliest models, outperforms all the later approaches by training the models with proper hyperparameters. A similar study by Ali et al. [2] performs a large-scale evaluation of 21 KG embedding models using PyKeen ¹ and provides the best set of hyper-parameters. The results of the study show that the performance of the models is not only determined by their architecture. The training approach, loss function, and the explicit modeling of inverse relations are also crucial. This leads to the conclusion that a unified benchmarking is needed which allows for comparing the approaches under the same conditions. This will reveal interesting insights, e.g., some models considered not competitive anymore can still outperform more sophisticated approaches under certain configurations. While the evaluation setting for transductive LP algorithms has been well established, the other three categories still lack a unified evaluation setting. Lastly, most of the evaluations present reports on the models’ performance over the entire test set. These analyses fail to provide a detailed understanding of the limitations of the embeddings on different aspects of the KG, e.g., tail elements (entities, classes, or relations) or relation types (e.g., transitive, symmetric, etc.). Few studies have addressed this issue and present specific results, e.g., per normal form in the ontology [36], or aggregate metrics like F1-macro [83] that take into account the weights of classes. Still, there is a notable discrepancy in the literature regarding the manner datasets, benchmarks, and analyses are reported.

In this section, we take a critical perspective on the SoTA over the past decade, based on the overview given in the preceding sections. We end with recommendations for fruitful future directions of work.