Two flexible translation-based models for knowledge graph embedding

Abstract

Knowledge Graph Embedding (KGE), which aims to embed the entities and relations of a knowledge gxraph into a low-dimensional continuous space, has been proven to be an effective method for completing a knowledge graph and improving the quality of the knowledge graph. The translation-based models represented by TransE, TransH, TransR and TransD have achieved great success in this regard. There is still potential for improvement in dealing with complex relations. In this paper, we find that the lack of flexibility in entity embedding limits the model’s ability to model complex relations. Therefore, we propose single-directional-flexible (sdf) models and multi-directional-flexible (mdf) models to increase the flexibility and expressiveness of entity embeddings. These two methods can be applied to the TransD model and its variant models without increasing any time cost and space cost. We conduct experiments on benchmarks such as WN18 and FB15k. The experimental results show that the models significantly surpasses the classical translation models in both tasks of triplet classification and link prediction. In particular, for Hits@1 of link prediction of WN18, we get 71.7% after applying our method to TransD, which is much better than 24.1% of TransD.

Keywords

Knowledge graph embedding translation model complex relation single-directional-flexible model multi-directional-flexible model

1 Introduction

Incorporating human knowledge is one of the important research directions of artificial intelligence (AI). Knowledge representation and reasoning, inspired by human problem solving, are to represent knowledge for intelligent systems to gain the ability to solve complex tasks [13]. However, how to effectively represent knowledge has always been a difficult point in knowledge engineering. In 1988, Stokman and Vries proposed a method to represent structured knowledge by using graph data. Subsequently, many real-world knowledge graphs (such as WordNet, Freebase, Yago, etc.) were constructed, published and played an important role in AI applications such as relation extraction [30], question answering systems [8], recommendation systems [13], and disease recognition [4].

A knowledge graph usually contains huge amounts of structured data as the form of triplets (head entity, relation, tail entity) (denoted as (h,r,t)). Since knowledge graphs are usually incomplete, a basic problem of knowledge graphs is to predict the missing edges. However, knowledge graphs are usually very large and mainly consist of symbolic representation, which makes the knowledge graphs difficult to calculate. In order to solve these issues, many Knowledge Graph Embedding (KGE) models were introduced to embed entities and relations into a low-dimensional continuous space to facilitate downstream tasks.

TransE is the earliest and simplest translation-based KGE model. Groundbreakingly, the model proposes that the relation is a translational vector from the head entity to the tail entity, i.e., h + r ≈ t. For example, for the triples (Steven Spielberg, directors, Jurassic Park), TransE believes that the vector of Steven Spielberg entity can approach the vector of Jurassic Park entity after being translated by the vector of the directors relation, which means that Steven Spielberg directed the movie Jurassic Park.

However, TransE encountered obstacles in dealing with the complex relations of 1-N, N-1 and N-N (shown in Fig. 1) in the knowledge graph, as mentioned in [12, 28]. For example, (Steven Spielberg, directors, Jurassic Park) and (Steven Spielberg, directors, E.T.) exist in a knowledge graph at the same time. The directors relation can be simply viewed as a 1-N relation. However, there are some flaws in the above models when dealing with such relations. Take TransE’s modeling process of these two facts as an example. Model need to satisfy both Steven Spielberg + directors = Jurassic Park and Steven Spielberg + directors = E.T. It will cause Jurassic Park and E.T. to be equal which is unexpected.

Fig. 1

Proportion of complex relations in FB15k.

In this paper, we find that the lack of flexibility and expressiveness of entity embeddings can lead model to a dilemma when modeling complex relations. By exploring the efforts and thinking of the classical translation models, we find that there is still space for improvement in this regard. In order to improve the representation ability of the model to the complex relations, we add an adjustment vector for each entity embedding, so that the entities have different representations in the context of different triples. For example, Steven Spielberg will have different embeddings when directing different movies, which can distinguish the embeddings of all the movies he directed. The adjustment vectors integrate the information of the triples where the entities are located, it can more flexibly adapt to the restrictions on the entity embeddings by complex relationships.

We propose sdf and mdf methods to increase the flexibility and expressiveness of entity embeddings, and apply these two methods to the TransD model and its variant models. The experiments are carried out on two benchmark data sets WN18 [2] and FB15k [3]. We report the accuracy of triplets classification task and the MRR, MR, Hits@1, Hits@3 and Hits@10 of the link prediction task. Finally, we compare the results with TransE [3], TransH [28], TransR [17], TransD [11], lppTransD [33], DistMult [32] and ComplEx [27].

The contributions of this paper are as follows:

Through the exploration and analysis of the classical translation models, we find that the flexibility and expressiveness of entity embedding are factors that affect the modeling of complex relations.

In order to improve the modeling ability of the model for complex relations, we propose two methods named single-direction-flexible (sdf) model and multi-direction-flexible (mdf) model and implement these two methods separately on TransD and lppTransD.

We conduct experiments on two benchmark data sets WN18 and FB15k, and the results show our method significantly outperforms classic translation models in link prediction and triplet classification tasks on multiple data sets.

The rest of this paper is organized as follows. Section 1 briefly introduces translational distance models and other state-of-the-art methods. Section 1 describes our proposed framework and its application, and discusses the time and space complexity. Experimental studies are presented in Section 1. We conclude the paper in Section 1.

2 Related work

KGE aims to embed the entities and relations in the knowledge graph into a low-dimensional continuous vector space while maintaining semantic information as much as possible. In a knowledge graph, triplets are represented as (head entity, relation, tail entity) (abbreviated as (h,r,t)). For each model, there is a scoring function, also named energy function, to measure the plausibility of the triplet, denoted as S = f_r (h, t). The goal of each model is to enforce the score of the golden triplet (h, r, t) as much higher or lower as possible than the corrupted triplets ((h, r, t′) or (h′, r, t)). According to the different forms of the score function, we can roughly divide the existing KGE models into translation-based models and semantic matching models.

2.1 Translation-based models

The translation-based models regard a relation as a translational vector operation. After the translation, models obtain a score of the triplet based on the distance. Under the condition of using less resources, this method not only obtains better results, but also gives certain interpretability. Before proceeding, we define some notations. A column vector $e \in ℝ^{m}$ is used to represent the embedding of an entity e, and the column vectors $h \in ℝ^{m}, r \in ℝ^{n}, t \in ℝ^{m}$ for a triplet.

The unified score function form of the translation model is as Formula (1): $f_{r} (h, t) = ∥ h_{⊥} + r - t_{⊥} ∥_{p}$ (1) where ∥ · ∥ _p is p-norm (used to be l₁-norm or l₂-norm), and the vectors e_⊥ represents some transformation of entity e.

TransE [3] assumes that the head entity vector can approach the tail entity vector by the translation of the relation vector. Formally, h + r ≈ t. Despite its simplicity and efficiency, TransE has flaws in handling complex relations, such as 1-N, N-1 and N-N relations (N means ‘many’) [17 , 29].

TransH [28] allows entities to have different representations under different relations, so it maps vectors of entities to a relation hyperplane through the unit normal vector of the corresponding relation, i.e. $e_{⊥} = e - w_{r}^{⊤} e w_{r}$ . The score function is same as TransE (Formula (1)).

TransR [17] extends the mapping of entities from the relational hyperplane of TransH to the relation space, thereby obtaining a more flexible entity mapping, i.e. e_⊥ = M_re. However, TransR suffers from the high complexity of matrix operations, so it is difficult to apply to large-scale knowledge graphs.

To alleviate this problem, Ji, et al. [12] proposed TranSparse. Transparse greatly reduces the computational cost by replacing the matrix in TransR with an adaptive sparse matrix. In addition, the model also consider the heterogeneity and imbalance,which are in fact caused by complex relationships, in KGs.

TransD [11] also makes a lot of efforts to reduce the computational complexity of model. The authors believe that the transformation of entity is associated not only with the relation but also with the entity itself. Therefore, it constructs dynamic mapping matrices through the corresponding information vectors of entities and relations for each entity: $M_{r} = r_{p} e_{p}^{⊤} + I$ . The corresponding entity vector after mapping is $e_{⊥} = M_{r} e = r_{p} e_{p}^{⊤} e + e$ . Replacing time-consuming matrix operations with vector operations makes it possible to deploy TransD on large-scale knowledge graphs.

It would be nice to simplify the model so that it can be applied to real problems. There are also some other models that improve the performance by slightly increasing the complexity of the models. lppTransX [33] divides the projection matrix of the relation in above models into the head projection relation matrix and the tail projection relation vector, so that the entities have different embedding at the head and tail positions. TransD-pa [36] improves the performance of the original models by adding regularization to the objective function.

Recently, RotatE [25] is presented to model symmetric, antisymmetric, inverse and combined relationships simultaneously by regarding relationships as rotation operations on entities in complex space. AutoETER [19] combines RotatE with previous classical models, such as TransR, to capture different attributes of entities. Inspired by RotatE, HAKE [37] embeds entities and relations into polar coordinate space, expecting to capture the hierarchy between entities while retaining the properties of rotation operation.

2.2 Semantic matching models

Unlike the translation-based models, the semantic matching models use similarity-based score functions instead of distance-based score functions. SME [2] proposes to semantically match separate combinations of entity-relation pairs of (h, r) and (r, t). DistMult [32] simplifies the calculation and prevents overfitting by constraining the relation matrix in the SME to a diagonal matrix. ComplEx [27] expands DistMult to the complex space to improve the model’s ability to capture semantic information, and expands the model’s ability to reason about other logical relations, such as symmetry, antisymmetric and inversion. HolE [20] uses a circular correlation of embedding to learn compositional relations representations (another logic rule).

The factorization models can be regarded as another form of semantic matching models, which aims to capture semantic information through tensor decomposition. Typically, RESCAL [21] gets the tensor decomposition by slicing the relation. With the rise of neural networks, a lot of work focused on the application of neural networks on KGE.

2.3 Other models

With the popularity of deep learning, many researchers try to apply deep neural network to KGE. NTN [24] takes entity embeddings as input associated with a relational tensor to train neural network parameters. ConvE [7] is the first model to introduce convolutional neural networks into KGE.

A lot of work [14, 16] improves the performance of embedding through additional information such as paths. There are also many translation-based models that embed entities and relations into other spaces, such as complex space [25], Gaussian space [10, 31], hyperbolic space [1, 5], time domain space [35] and so on. But this paper only focuses on the discussion of Euclidean space. There is also a lot of work to re-evaluate previous work [22, 23].

3 Our method

First, we analyze in detail the process of entity embedding of TransE, TransD [11] and lppTransD [33]. Through our analysis, we find that the TransE is in a dilemma when dealing with complex relationships. TransD and its variant lppTransD make up for this defect by increasing the flexibility and expressiveness of entity embeddins, but there still exists space for improvement. Subsequently, we propose two methods to improve the ability of the model to deal with complex relations by increasing the flexibility and expressiveness of entity embeddings. Finally, we elaborate on the training and optimization process of the model in detail.

3.1 Analysis of TransD and lppTransD

Before analyzing, we give a definition:

Definition 1. Perfect embedding: For a triplet (h, r, t) and its embeddings (h, r, t), if the distance-based score function f_r (h, t) =0 by the embeddings after training. We call the embeddings of the entities and the relation are perfect embeddings for the triplet.

TransD prepares two vectors for each entity e and relation r, which are called information vector and projection vector, respectively. We denote the information vector as $e \in ℝ^{n}, r \in ℝ^{m}$ and the projection vector as $e_{p} \in ℝ^{n}, r_{p} \in ℝ^{m}$ . TransD uses the projection vectors construct a projection matrix ( $r_{p} e_{p}^{⊤} + I$ ) to achieve an effect similar to TransR. From subsection 1 TranslationModels, we can expand the final score function to: $f_{r} (h, t) = ∥ r_{p} h_{p}^{⊤} h + h + r - r_{p} t_{p}^{⊤} t - t ∥_{p}$ (2) lppTransD is a classic variant of TransD, which also assumes that an entity should have corresponding embeddings in different situations. For this reason, it separates the relational projection vector $r_{p} \in ℝ^{m}$ into a head projection vector $r_{ph} \in ℝ^{m}$ and a tail projection vector $r_{pt} \in ℝ^{m}$ on the basis of TransD. The final score function is expressed as: $f_{r} (h, t) = ∥ r_{ph} h_{p}^{⊤} h + h + r - r_{pt} t_{p}^{⊤} t - t ∥_{p}$ (3)

Note that the score functions can be seen as consisting of two parts. One part is the same as TransE: h + r - t. The other part can be regarded as an adjustment part for entities, i.e. $r_{p} h_{p}^{⊤} h$ for head and $r_{p} t_{p}^{⊤} t$ for tail or $r_{ph} h_{p}^{⊤} h$ for head and $r_{pt} t_{p}^{⊤} t$ for tail. Note that $e_{p}^{⊤} e$ is a scalar, which means that TransD actually adds a specific-relation adjustment vector for entities on the basis of TransE. The sizes of adjustment vectors are determined by the inner product ( $e_{p}^{⊤} e$ ) of corresponding information vectors e and projection vectors e_p of the entity. For lppTransD the adjustment part consists of r_ph and r_pt, the sizes of the vector are also determined by $e_{p}^{⊤} e$ .

Figure 2 shows the perfect embedding of different models in two dimensional Euclidean space. Suppose there are two triplets (h, r, t₁) and (h, r, t₂) in a knowledge graph (It can be thought that h is the director Steven Spielberg in section 1, t₁ is the movie Jurassic Park and t₂ is E.T.). In Fig. 2(a), we can see that TransE, which aims to get perfectly embedding for both triplets, embeds t₁ and t₂ as a same point (t₁ = t₂).

Fig. 2

Illustration of translation-based models.

For TransD (Fig. 2(b)), it can be seen that the head entity vector h is translated to h_c by adjustment vector αr_p, then through the relation vector r to t_c, and finally through the adjustment vectors β_ir_p to tail entity vectors. Among them, α is determined by the head entity, and β_i are determined by the tail entity, i.e. $α = h_{p}^{⊤} h$ and $β_{i} = t_{pi}^{⊤} t_{i}$ . More generally, we can regard h_c as conceptual head entities (conception of director) and t_c as conceptual tail entities (conception of movie), respectively. It should be noted that although TransD can perfectly embed these two triplets, the tail entities (t₁ and t₂) are limited to a straight line in the r_p direction with t_c as the origin. Similarly, the perfect embeddings of the tail entities in lppTransD (Fig. 2(c)) are limited to the same straight line. The difference is that the directions of the projection vectors for the head entity (r_ph) may be different from the tail entities (r_pt).

From the above analysis, we find there exist unnecessary limitations on entity embeddings in TransD and lppTransD. These limitations affect the ability of the model to model complex relations. Therefore, how to remove these restrictions has become our focus.

3.2 Single-direction-flexible models and multi-direction-flexible models

Our motivation is to enhance the model’s ability to handle complex relationships by increasing the flexibility of entity embedding. From the analysis of subsection 1, we find that the adjustment vectors of TransD or lppTransD for entities are mainly composed of two parts: sizes and directions. The sizes of adjustment vectors are determined by corresponding entities ( $e_{p}^{⊤} e$ ) and the projection vectors of relations (r_p, r_ph and r_pt). So we start with these two aspects and propose single-direction-flexible TransD (TransD-sdf) and multi-direction-flexible TransD (TransD-mdf).

On the one hand, we achieve our goal by increasing the flexibility of the adjustment vector sizes. The sizes are determined jointly by the head entities h and tail entities t instead of themselves. Figure 2(d) shows the embedding of triplets by TransD-sdf, where $α_{i} = t_{pi}^{⊤} h$ and $β_{i} = h_{p}^{⊤} t_{i}$ . From Fig. 2(d), it can be seen that TransD-sdf expands the directors’ conception h_c and movies’ conception t_c from a point to a line range. Since the directions of the adjustment vectors, which are determined by the projection vectors of relations r_p, are still single in model, we call it single-direction-flexible model. We implement our ideas on TransD and lppTransD, called TransD-sdf and lppTransD-sdf respectively. The score functions of TransD-sdf and lppTransD-sdf are as follows: $f_{r} (h, t) = ∥ r_{p} t_{p}^{⊤} h + h + r - r_{p} h_{p}^{⊤} t - t ∥_{p}$ (4) $f_{r} (h, t) = ∥ r_{ph} t_{p}^{⊤} h + h + r - r_{pt} h_{p}^{⊤} t - t ∥_{p}$ (5) Note that compared to the original models, the new models do not add any parameters. TransD-sdf does not essentially alleviate the limitation that the tail entities are embedded in a fixed vector direction. However, the model increases the ability of the model to represent semantic information by incorporating the information of the head and tail entities. Therefore, the entity embeddings of TransD-sdf can express richer entity semantics.

On the other hand, we hope to break through the limitation of projection direction. Therefore, we use the projection vectors of entity e_p as their translation directions instead of the projection vectors of the relations r_p. Figure 2(e) shows the embedding of TransD-mdf. It can be seen that t₁ and t₂ have adjustment vectors in different directions. Therefore, we call this model multi-direction-flexible (mdf) model. We also apply mdf to TransD and lppTransD, and their score functions are as follows: $f_{r} (h, t) = ∥ h_{p} r_{p}^{⊤} t + h + r - t_{p} r_{p}^{⊤} h - t ∥_{p}$ (6) $f_{r} (h, t) = ∥ h_{p} r_{pt}^{⊤} t + h + r - t_{p} r_{ph}^{⊤} h - t ∥_{p}$ (7) Similar to TransD-sdf, the conceptions of head entity h_c and tail entity t_c are also embedded from a point to a line range. Since tail entities have respective directions of adjustment vectors, the tail entity instances t_i can be embedded in a wider space instead of a straight line in Fig. 2(d).

3.3 Train and complexity analysis

3.3.1 Training details

The general optimization goal of the translation models [3 , 29] is usually as follows: $L = \sum_{S} \sum_{S^{'}} {[f_{r} (h, t) + γ - f_{r} (h^{'}, t^{'})]}_{+}$ (8) where [x] ₊≜ max(0, x), and γ is a hyperparameter used to control overfitting. Jia et al. [15] analyzed the deep meaning and function of γ in detail. S is a collection of golden triplets, i.e. (h, r, t) ∈ S. S′ is a collection of corrupted triplets generated by corrupting the head or tail of triplets, i.e. (h, r, t′) ∉ S and (h, r, t′) ∈ S′, so does (h′, r, t). For the generation of S′, we use two mainstream negative sampling methods, ^′unif′ [3] and ^′bern′ [28].

We are inspired by RotatE [25]: every negative samples should contribute its corresponding weight to the optimization goal. Therefore, we modify the margin-loss to the following form: $L = \sum_{S} \sum_{S^{'}} ({[f_{r} (h, t) + γ - α_{i} * f_{r} (h^{'}, t^{'})]}_{+})$ (9) where α_i is the weight of the i^th negative sample contributed. The weight values are softmaxed out as follow: $α_{i} = \frac{exp (f_{r} (h_{i}^{'}, t_{i}^{'}))}{\sum_{i = 0}^{k} exp (f_{r} (h_{j}^{'}, t_{j}^{'}))}$ (10)

Algorithm 1

Training process.

Input:
Training set S = {(h, r, t)}
Entity set E and Relation set R
margin γ and embeddings dim m, n
1: initialize r, r_ph, r_pt for each r ∈ R;
2: initialize e, e_p for each e ∈ E;
3: LOOP
4: normalize r, r_ph, r_pt for each r ∈ R;
5: normalize e, e_p for each e ∈ E;
6: sample a minibatch S_batch of size b;
7: for (h, r, t) ∈ S_batchdo
8: sample corrupted triplets $S_{batch}^{'}$ ;
9: update embeddings according to Formula(9)
10: end for
11: end loop

The whole optimization process uses Stochastic Gradient Descent (SGD) to optimize the objective function. In order to speed up the convergence and prevent overfitting, we explicitly restrict the sizes of the entity vectors and relation vectors, i.e., make ∥e ∥ ₂ ≤ 1 and ∥r ∥ ₂ ≤ 1 by setting e = e/∥ e ∥ ₂ and r = r/∥ r ∥ ₂. We summarize the process of training embeddings in Algorithm 1 alg.

3.3.2 Complexity analysis

Table 1 Comparisonof model time complexity and space complexity shows the time complexity and space complexity of our models and all comparison models. Given a certain knowledge graph, we use N_e to represent the number of entities, N_r to represent the number of relations, and N_t to represent the number of triplets. Meanwhile, the dimension of entity and relation are denoted by m and n, respectively. Note that KGE models are expected to embed entities and relations into low-dimensional spaces. So m and n are usually in the tens to hundreds of magnitude.

Table 1
Comparison of model time complexity and space complexity

Model #Parameters #Operations

TransE O (N_em + N_rn) (m = n) O (nN_t)

TransH O (N_em + 2N_rn) (m = n) O (2nN_t)

TransR O (N_em + N_r (m + 1) n) O (2mnN_t)

TransD O (2N_em + 2N_rn) O (2nN_t)

lppTransD O (2N_em + 3N_rn) O (2nN_t)

TransD-pa O (2N_em + 3N_rn + 12n²) (m = n) O (5nN_t)

TransD-sdf O (2N_em + 2N_rn) (m = n) O (2nN_t)

TransD-mdf O (2N_em + 2N_rn) (m = n) O (2nN_t)

lppTransD-sdf O (2N_em + 3N_rn) (m = n) O (2nN_t)

lppTransD-mdf O (2N_em + 3N_rn) (m = n) O (2nN_t)

Model	#Parameters	#Operations
TransE	O (N_em + N_rn) (m = n)	O (nN_t)
TransH	O (N_em + 2N_rn) (m = n)	O (2nN_t)
TransR	O (N_em + N_r (m + 1) n)	O (2mnN_t)
TransD	O (2N_em + 2N_rn)	O (2nN_t)
lppTransD	O (2N_em + 3N_rn)	O (2nN_t)
TransD-pa	O (2N_em + 3N_rn + 12n²) (m = n)	O (5nN_t)
TransD-sdf	O (2N_em + 2N_rn) (m = n)	O (2nN_t)
TransD-mdf	O (2N_em + 2N_rn) (m = n)	O (2nN_t)
lppTransD-sdf	O (2N_em + 3N_rn) (m = n)	O (2nN_t)
lppTransD-mdf	O (2N_em + 3N_rn) (m = n)	O (2nN_t)

Note that our improvements to TransD and lppTransD do not increase any time complexity and space complexity. Therefore, both the sdf methods and mdf methods perfectly inherit the advantages of TransD and its variant that can be deployed on a large-scale knowledge graph.

4 Experiments and results analysis

In this section, we first introduce the commonly used public data. Then we evaluate our method on two tasks: triplet classification and link prediction. Finally, we analyse of the experimental results in detail. OpenKE [9] is an open source tool for knowledge graph embedding, which integrates the classic KGE model. Our experiment was conducted on OpenKE. Experimental environment: CPU: Intel Xeon Gold 6248, RAM: 376GB, GPU: NVIDIA GV100GL [Tesla V100 PCIe 32GB].

4.1 Data sets

WordNet is a large lexical knowledge graph, which structures the concepts and logic in human language into entities and relations respectively (e . g . (folding _ chair, has _ part, armchair)). In this paper, we use two subsets of WordNet: WN11 [3] and WN18 [2].

Freebase is a large number of the real world facts such as (anthony _ asquith, location, london). Similarly, we also use two mainstream subsets: FB13 [3] and FB15k [2].

The statistics of the data set used in the experiment are shown in Table 2 Datasetsused in the experiments. In the table, Avg. #Train/#Ent represents the average number of linked triplets regarding with each entity in the training set. With a small average linked triplets, it is more challenging to learn the semantic of the entities [36]. For this reason, the link prediction task will only be performed on WN18 and FB15k, while the triplet classification task will be performed on all data sets in Table 2 Datasetsused in the experiments. For most knowledge graphs, the number of entities N_e is often much larger than the number of relations N_r.

Table 2
Datasets used in the experiments

Dataset #Rel #Ent #Train #Valid #Test Avg.#Train/#Ent

WN11 11 38,696 112,581 2,609 10,544 2.909

WN18 18 40,943 141,442 5,000 5,000 3.455

FB13 13 75,043 316,232 5,908 23,733 4.214

FB15k 1,345 14,951 483,142 50,000 59,071 32.315

Dataset	#Rel	#Ent	#Train	#Valid	#Test	Avg.#Train/#Ent
WN11	11	38,696	112,581	2,609	10,544	2.909
WN18	18	40,943	141,442	5,000	5,000	3.455
FB13	13	75,043	316,232	5,908	23,733	4.214
FB15k	1,345	14,951	483,142	50,000	59,071	32.315

4.2 Experimental details and implementation

4.2.1 Tasks

This paper focuses on two tasks: triplet classification and link prediction. Next we will introduce two tasks and their details.

Triplet Classification: The classification of triplets is a binary classification problem to determine whether a triplet (h, r, t) is a fact triplet. This task associates the scores f_r (h, t) with the class labels by setting a score threshold δ_r for each relation. Specifically, a triplet (h, r, t) will be classified as positive if f_r (h, t) > δ_r, otherwise negative. δ_r is obtained by maximizing the accuracies on the validation set. For convenience, we only use accuracy (acc) as our evaluation indicator.

Link Prediction: Link prediction is essentially a rank problem. Note that the sum of the number of entities in the training set, the verification set and the test set is N_t. For a golden triplet (h, r, t), if we replace its head entity or tail entity with all other entities, we will get N_t - 1 corrupted triplets. The purpose of link prediction is to give the ranking of the score of the fact triplet in the score set, which is composed of the scores of the fact triplet and all corrupt triplets. In other words, the ranking range of fact triplets is [1, N t - 1]. We assume that there are N triplets in the test set, and the rank corresponding to the score of the i-th triplet is rank_i. The main evaluation indicators are Mean Rank (MR), Mean Reciprocal Rank (MRR) and Hits @ n, whose calculation method are as follows: $MR = \sum_{i = 1}^{N} {rank}_{i}$ (11) $MRR = \sum_{i = 1}^{N} \frac{1}{{rank}_{i}}$ (12) $\begin{matrix} Hits @ n = \frac{\sum_{i = 1}^{N} x_{i}}{N} \times 100 %, x_{i} = {\begin{matrix} 0 & {rank}_{i} > n \\ 1 & otherwise \end{matrix} \end{matrix}$ (13)MR may be vulnerable to triplets with extremely low scores. MRR is a relatively comprehensive measure. Hits @ n pays more attention to the situation where the triplet score ranks high.

It is worth to pay a special attention that corrupted triplets may also be fact triplets. In order to make a more appropriate comparison, these fact triplets should be filtered out during the evaluation. We will report more reasonable results which have filtered other fact triplets.

4.2.2 Experimental settings

Hyperparameter settings. We use SGD [34] to optimize Formula (8). The hyperparameters involved in the models are: embedding dimension m and n, number of batches b, fixed margin γ, number of negative samples k and norm p. It is obviously difficult to find the optimal parameter combination with grid search. For a fair comparison, we use the optimal m, n, k and p which used in other compared models. Benefit from the development of deep learning tools, we can also unify the learning rate to 1. The ranges of others for the grid search are set as follows: b ∈ {50, 75, 100, 125, 150}, γ ∈ {1, 2, 3, 4, 5, 6}.

Negative sampling. In classification algorithms, the distribution of positive and negative samples often has a great impact on the performance of the model. Since there only exists positive samples in a knowledge graph, how to generate negative samples has become an important issue of concern. Inspired by word embedding [18], TransE [3] uses a similar unified negative sampling method called “unif ". Subsequently, Wang et al. [28] discovered that there are a large number of complex relations in the knowledge graph, which cannot be represented by the general negative sampling method alone. Therefore, they use the Bernoulli method to perform negative sampling to greatly improve the performance of the model. We call this method “bern " for short. This paper reports the experimental results of both two negative sampling methods. The specific methods of the two negative sampling methods are as follows:

unif : For a fact triplet (h, r, t), the corrupt triplet ((h, r, t′) or (h′, r, t)) is obtained by randomly replacing the head entity or the tail entity from all entities. This method is simple and effective, but it will generate a large number of false negative labels during training.

bern : Sampling after analyzing the Bernoulli distribution of the sample. Before the model starts training, count the average number of tail entities per head entity under a specific relation, and record it as t_ph. Symmetrically, record the average number of head entities per tail entity as h_pt. Then replace the tail entity with probability $\frac{t_{ph}}{t_{ph} + h_{pt}}$ and the head entity with probability $\frac{h_{pt}}{t_{ph} + h_{pt}}$ to generate a corrupted triplet.

4.3 Results analysis and disicussion

In this section, we report and analyze the experimental results of triplet classification task and link prediction. Since our method is based on translation model, we mainly compare with classical translation models: TransE, TransH, TransR and TransD. In addition, we compare two improved models based on TransD. In the link prediction task, we also report the results of DistMult and ComplEx for comparison. All the best experimental results are marked in bold. The optimal results of our method are underlined for comparison with the optimal results.

Due to differences in experimental equipment, parameter settings and other details, the experimental results may be different. To be fair, excluding lppTransD, we use the results of translation-based models reported by [36] in recent related work that are better than original papers. And the results of DistMult and ComplEx are derived from the report of [7].

4.3.1 Result of triplet classification

We compare our methods with the classic translational distance models on all datasets. The experimental results are shown in the Table 3 Resultof Triplet Classification. Note that most of the comparison models only perform this task on WN11, FB13 and FB15k. Their original texts do not contain the results of WN18, so we quote the results of Zhang et al. [36].

Table 3
Result of Triplet classification

WN11 WN18 FB13 FB15k

TransE (unif) 75.9 95.5 70.9 77.3

TransE (bern) 75.9 95.7 81.5 79.8

TransH (unif) 77.7 95.0 76.5 80.2

TransH (bern) 78.8 95.7 83.3 87.7

TransR (unif) 85.5 95.8 74.7 81.7

TransR (bern) 85.9 95.8 82.5 83.9

TransD (unif) 85.6 95.9 85.9 86.4

TransD (bern) 86.4 95.8 89.1 88.0

lppTransD (unif) 86.1 - 86.6 84.7

lppTransD (bern) 86.2 - 88.6 85.3

TransD-pa (unif) 86.9 96.8 89.6 89.9

TransD-pa (bern) 86.8 96.8 89.5 90.3

TransD-sdf (unif) 77.9 98.2 82.0 95.0

TransD-sdf (bern) 79.4 98.2 84.9 96.5

TransD-mdf (unif) 78.6 98.2 82.6 95.5

TransD-mdf (bern) 79.2 98.2 83.4 96.7

lppTransD-sdf (unif) 78.4 98.1 81.8 94.7

lppTransD-sdf (bern) 76.5 98.2 85.2 96.6

lppTransD-mdf (unif) 79.2 98.2 82.8 95.4

lppTransD-mdf (bern) 80.2 98.3 83.6 96.6

WN11	WN18	FB13	FB15k
TransE (unif)	75.9	95.5	70.9	77.3
TransE (bern)	75.9	95.7	81.5	79.8
TransH (unif)	77.7	95.0	76.5	80.2
TransH (bern)	78.8	95.7	83.3	87.7
TransR (unif)	85.5	95.8	74.7	81.7
TransR (bern)	85.9	95.8	82.5	83.9
TransD (unif)	85.6	95.9	85.9	86.4
TransD (bern)	86.4	95.8	89.1	88.0
lppTransD (unif)	86.1	-	86.6	84.7
lppTransD (bern)	86.2	-	88.6	85.3
TransD-pa (unif)	86.9	96.8	89.6	89.9
TransD-pa (bern)	86.8	96.8	89.5	90.3
TransD-sdf (unif)	77.9	98.2	82.0	95.0
TransD-sdf (bern)	79.4	98.2	84.9	96.5
TransD-mdf (unif)	78.6	98.2	82.6	95.5
TransD-mdf (bern)	79.2	98.2	83.4	96.7
lppTransD-sdf (unif)	78.4	98.1	81.8	94.7
lppTransD-sdf (bern)	76.5	98.2	85.2	96.6
lppTransD-mdf (unif)	79.2	98.2	82.8	95.4
lppTransD-mdf (bern)	80.2	98.3	83.6	96.6

From Table 3 Resultof Triplet Classification we can see that our models have the optimal results on both WN18 and FB15k, and the improvement is significant. It can be seen that the accuracy on WN18 is improved to 98.3% from 96.8%. Significantly, the accuracy of the model on FB15k is increased to 96.7% from 90.3%, which far exceeds the result of the baseline model. However, our models have only baseline performance on the WN11 and FB13 data sets. We think it might be that there exist few logical rules or complex relations in FB13 and WN11. The original intention of our models is to make up for the flaws in the process of modeling these complex data with traditional translation-based models. Therefore, our methods are better at dealing with complex relations.

4.3.2 Result of link prediction

For the link prediction task, we give the experimental results of WN18 and FB15k data sets. The results of MRR, MR, Hits@10, Hits@3, and Hits@1 are given respectively. Since there are multiple evaluation indicators, we select the best results and parameters based on the best MRR. Table 4 LinkPrediction results on FB15k and WN18 shows the experimental results of link prediction for FB15k and WN18, respectively.

Table 4
Link prediction results on FB15k and WN18

models FB15k WN18

MR MRR Hits@10 Hits@3 Hits@1 MR MRR Hits@10 Hits@3 Hits@1

translation TransE (unif) 83 .394 70.6 55.7 32.5 319 .591 93.5 87.0 10.3

TransE (bern) 101 .457 71.1 56.1 33.8 315 .589 93.7 88.7 11.0

TransH (unif) 75 .519 75.9 60.6 38.6 305 .617 93.5 88.3 18.2

TransH (bern) 92 .530 64.4 62.6 39.5 300 .601 92.5 84.2 17.8

TransR (unif) 78 .394 67.3 52.3 33.9 312 .601 93.3 88.8 22.0

TransR (bern) 79 .450 69.8 53.6 32.1 304 .617 93.4 87.6 21.5

TransD (unif) 72 .511 76.8 62.2 35.9 312 .623 93.5 87.9 21.3

TransD (bern) 95 .539 76.2 63.6 40.6 306 .617 92.4 84.8 24.1

lppTransD (unif) 69 - 77.4 - - 328 - 93.6 - -

lppTransD (bern) 78 - 78.7 - - 270 - 94.3 - -

TransD-pa (unif) 63 .583 81.1 67.5 45.3 312 .667 93.9 90.9 35.4

TransD-pa (bern) 67 .591 80.7 68.1 46.5 295 .675 94.2 90.8 37.4

semantic DistMult 97 .654 82.4 73.3 54.6 902 .822 93.6 91.4 72.8

match ComplEx - .692 84.0 75.9 59.9 - .941 94.7 94.4 93.9

our models TransD-sdf (unif) 393 .650 81.2 73.3 54.4 444 .820 95.0 93.3 71.0

TransE-sdf (bern) 214 .657 82.3 74.1 55.0 419 .816 95.2 93.4 70.2

TransD-mdf (unif) 141 .555 79.7 68.4 39.4 305 .824 95.2 93.6 71.5

TransD-mdf (bern) 95 .642 83.0 73.5 52.1 327 .825 95.3 93.7 71.7

lppTransD-sdf (unif) 471 .622 79.2 71.1 51.1 445 .810 95.3 93.5 68.7

lppTransD-sdf (bern) 248 .670 82.1 73.9 57.8 444 .808 95.2 93.1 68.7

lppTransD-mdf (unif) 155 .559 79.5 68.6 40.4 370 .819 95.1 92.9 71.0

lppTransD-mdf (bern) 96 .646 83.0 73.8 52.7 300 .821 95.3 93.1 71.2

models	FB15k	WN18
translation	TransE (unif)	83	.394	70.6	55.7	32.5	319	.591	93.5	87.0	10.3
TransE (bern)	101	.457	71.1	56.1	33.8	315	.589	93.7	88.7	11.0
TransH (unif)	75	.519	75.9	60.6	38.6	305	.617	93.5	88.3	18.2
TransH (bern)	92	.530	64.4	62.6	39.5	300	.601	92.5	84.2	17.8
TransR (unif)	78	.394	67.3	52.3	33.9	312	.601	93.3	88.8	22.0
TransR (bern)	79	.450	69.8	53.6	32.1	304	.617	93.4	87.6	21.5
TransD (unif)	72	.511	76.8	62.2	35.9	312	.623	93.5	87.9	21.3
TransD (bern)	95	.539	76.2	63.6	40.6	306	.617	92.4	84.8	24.1
lppTransD (unif)	69	-	77.4	-	-	328	-	93.6	-	-
lppTransD (bern)	78	-	78.7	-	-	270	-	94.3	-	-
TransD-pa (unif)	63	.583	81.1	67.5	45.3	312	.667	93.9	90.9	35.4
TransD-pa (bern)	67	.591	80.7	68.1	46.5	295	.675	94.2	90.8	37.4
semantic	DistMult	97	.654	82.4	73.3	54.6	902	.822	93.6	91.4	72.8
match	ComplEx	-	.692	84.0	75.9	59.9	-	.941	94.7	94.4	93.9
our models	TransD-sdf (unif)	393	.650	81.2	73.3	54.4	444	.820	95.0	93.3	71.0
TransE-sdf (bern)	214	.657	82.3	74.1	55.0	419	.816	95.2	93.4	70.2
TransD-mdf (unif)	141	.555	79.7	68.4	39.4	305	.824	95.2	93.6	71.5
TransD-mdf (bern)	95	.642	83.0	73.5	52.1	327	.825	95.3	93.7	71.7
lppTransD-sdf (unif)	471	.622	79.2	71.1	51.1	445	.810	95.3	93.5	68.7
lppTransD-sdf (bern)	248	.670	82.1	73.9	57.8	444	.808	95.2	93.1	68.7
lppTransD-mdf (unif)	155	.559	79.5	68.6	40.4	370	.819	95.1	92.9	71.0
lppTransD-mdf (bern)	96	.646	83.0	73.8	52.7	300	.821	95.3	93.1	71.2

WN18: The best MRR, Hits@10, Hits@3 and Hits@1 are distributed over TransD-mdf (bern) and lppTransD-sdf (bern), respectively. To facilitate discussion, we will discuss the best results of ’Filter’ more. TransD-mdf (bern) improves MRR to 0.825 which obviously surpassed 0.623 of TransD (bern) and 0.675 of TransD-pa (bern). Significantly, the Hits@1 of TransD-mdf (bern) (71.7%) is almost twice as much as TransD-pa (bern) (37.4%) and three times as TransD (bern) (24.1%). Note that these improvements do not increase any time cost and space cost from the beginning to the end. Hits@10 (95.3%) and Hits@3 (93.7%) of TransD-mdf (bern) are also promoted to the best results, even if the improvements are not satisfactory. It may be that model pursuit better loss, which caused some triplets to be ranked far behind. And results of MR would be suffered from these triplets ranked behind seriously. In other words, the model carries out a more refined sorting for the results of TransD and lppTransD, and upgrades the original entity from the top ten ranking to the first ranking.

FB15k: Compared to the results of WN18, the best results of different indicators are distributed in our method and applied to different models. In general, TransD-pa has obtained the best results of various indicators under ‘Raw’ condition. In order to reflect the advantages of our methods, we compute the average of the results of the models (bern). We show the average results in Table 5. It can be seen that MRR, Hits@3, and Hits@1 of sdf method are improved remarkably. The best Hits@10 are got by mdf method. Unfortunately, both of two methods compromise on the MR. The possible reasons have been discussed above, so they won’t be repeated here.

Table 5

Average results of sdf and mdf on FB15k

MRR	MR	Hits@10	Hits@3	Hits@1
TransD-pa (unif)	.583	63	81.1	67.5	45.3
TransD-pa (bern)	.591	67	80.7	68.1	46.5
sdf (bern)	.663	231	82.2	74.0	56.4
mdf (bern)	.644	96	83.0	73.7	52.4

In addition, in order to show the superiority of our method more intuitively, we average all the results of sdf and mdf methods, and compare them with the results of recent model TransD-pa. The results of the comparison are shown in Table 5.

Comparison with semantic matching models: Compared with translation models, the performance of semantic matching models are better. Our method is better than DistMult model in general. For FB15k data set, our method is close to ComplEx model on every index. However, on the WN18 dataset, our method is only better than complex on Hits@10. For other indicators, especially MRR and MRR Hits@1, ComplEx is significantly better than all other models.

Complex relation analysis in FB15k: We also report the average Hits@10 of 1-1, 1-N, N-1, and N-N relations in FB15k (Table 6) to analyze the model’s ability to model complex relations. Bordes et al.[3] gave the type definition of a relation based on the average number of head-to-tail relations in the data set. According to the following steps, the type of a relation can be derived:

For a relation r, traverse the two-tuple (h, r,) composed of the head entity and the relation (or the two-tuple (, r, t) composed of the tail entity and the relation).

Count the average number of correct entities under (h, r,) denoted as t_r. Similarly, Let h_r correspond to (, r, t).

If this number is less than 1.5, we mark the position corresponding to this number as 1 otherwise N. For example, if h_r > 1.5 and t_r < 1.5, the type of the relation is N-1.

Back to Table 6, it can be seen that the sdf models handle the relations between 1-1, 1-N and N-1 very well. The mdf models have more advantages when dealing with N-N relations. Recall Fig. 1 complexrelations, N-N relations account for about three-quarters of all FB15k relations. It explains why the mdf method can achieve better results on Hits@10. Overall, both sdf method or mdf method have a much better improvement in dealing with complex relations compared with original models.

Table 6

Experimental results on FB15k by mapping properties of relations (%)

Prediction Head (Hits@10)				Prediction Tail (Hits@10)
1-1	1-N	N-1	N-N	1-1	1-N	N-1	N-N
TransE	43.7	65.7	18.2	47.2	43.7	19.7	66.7	50.0
TransH (unif)	66.7	81.7	30.2	57.4	63.7	30.1	83.2	60.8
TransH (bern)	66.8	87.6	28.7	64.5	65.5	39.8	83.3	67.2
TransR (unif)	76.9	77.9	38.1	66.9	76.2	38.4	76.2	69.1
TransR (bern)	78.8	89.2	34.1	69.2	79.2	37.4	90.4	72.1
TransD (unif)	80.7	85.8	47.1	75.6	80.0	54.5	80.7	77.9
TransD (bern)	86.1	95.5	39.8	78.5	85.4	50.6	94.4	81.2
lppTransD (unif)	82.7	89.5	53.2	81.6	82.3	52.6	84.8	81.5
lppTransD (bern)	86.0	94.2	54.4	82.2	79.7	43.2	95.3	79.7
TransD-sdf (unif)	91.1	85.9	63.4	83.3	91.6	79.1	72.7	83.6
TransD-sdf (bern)	90.9	95.3	49.2	84.1	91.0	60.7	90.6	86.0
TransD-mdf (unif)	85.5	82.8	45.5	79.4	82.3	61.2	81.0	85.1
TransD-mdf (bern)	86.8	95.2	50.1	84.2	86.4	60.0	93.3	87.3
lppTransD-sdf (unif)	91.1	79.6	57.7	82.7	90.9	67.4	70.4	82.4
lppTransD-sdf (bern)	90.1	94.9	49.5	84.0	90.7	62.0	89.7	84.9
lppTransD-mdf (unif)	85.0	82.2	43.7	82.9	83.4	59.5	78.0	85.1
lppTransD-mdf (bern)	85.9	95.4	49.9	84.2	84.5	60.1	93.7	87.3

Ablation Experiment: In section 1, we introduced Formula (9) as a loss function to optimize the model. In order to verify the effectiveness of the models, we provide the original margin-loss (Formula8) as the loss function of ablation experiments:

Table 7 Ablationexperiment shows the experimental results of TransD-sdf and TransD-mdf under different loss function optimizations. The “original” represents the models using the original margin-loss for optimization, and the “optimized” represents the models using the optimized margin-loss with weight. For comparison, we only report the “Filter” results under the “bern” negative sampling method. Experimental results show that our sdf method and mdf method can significantly improve the performance of TransD even under the original margin loss function optimization. Although the results of Hits10 are improved, the improvement is limited. TransD-sdf and TransD-mdf mainly improve the MRR results (MRR indicators are highly dependent in Hits@1). This is consistent with our previous result analysis, sdf and mdf methods carry out a more refined sorting for the results of original TransD, and upgrade the original entity from the top ten ranking to the first ranking. Using the optimized loss function further strengthen the refining process.

Table 7

Results of models under different loss function optimizations

models	WN18		FB15k
	MRR	Hits@10	MRR	Hits@10
TransD	0.617	92.4	0.539	76.2
TransD-sdf (original)	0.793	95.3	0.598	80.0
TransD-sdf (optimized)	0.816	95.2	0.657	82.3
TransD-mdf (original)	0.817	95.1	0.598	80.9
TransD-mdf (optimized)	0.825	95.3	0.642	83.0

Supplementary experiment: According to [7], there are different degrees of test set leakage problems in FB15k and WN18. Therefore, we conduct supplementary experiment on the more challenging data set WN18RR to verify the effectiveness of our method.

Table 8

Results of supplementary experiment of WN18RR

models	WN18RR
MRR	Hits@10
TransE	22.6	50.1
DistMult	43.0	49.0
ComplEx	44.0	51.0
TransD-sdf	21.1	47.5
TransD-mdf	21.4	48.9

The results of TransE are given by [6]. The results of DistMult and ComplEx are given by [7]. Hits@10 of TransD-sdf and TransD-mdf are closed to DistMult and ComplEx. However, MRR results are not so good. Although result of TransE looks better, [23] points out that experiment in [6] exists a test set leak problem. In addition, [22] points out that adding new training tricks can make the early KGE models perform well on FB15k-237 and WN18RR. We have shown in theory that our models are superior to TransE and close to Complex and DistMult. In the future, we will try to add some training tricks to enhance our results.

5 Conclusion and future work

In this paper, we found that the lack of flexibility and expressiveness of entity embeddings limits the abilities of the classical translation models in modeling complex relations. Through in-depth exploration of the classic translation models, we discovered the unnecessary limitations of these models in process of entity embedding. Specifically, we proposed single-direction-flexible models and multi-directional-flexible models to increase the flexibility and expressiveness of entity embeddings. Such improvement enables the models to improve the modeling ability of complex relations between 1-N, N-1 and N-N. However, our method only pays attention to the flexibility of entity embedding and does not make corresponding improvements to relation embedding. This will be our future research direction. Meanwhile, the model can also embed more triplets perfectly. Experiments proved that our models greatly improve the prediction accuracy of triplet classification and the results of link prediction without increasing any time cost and space cost. Our methods can be applied not only to TransD, but also to its variants. Our models inherit the advantages of TransD that can be deployed to large-scale knowledge graphs, and at the same time improve the ability to model complex relations. In addition, we did not pay enough attention to training tricks, which may limit our results, so we will explore the impact of different training tricks and experimental settings on the model in future work.

Footnotes

Acknowledgments

This work is supported by the National Key Research and Development Program of China (Grant No. 2019YFA0706200), the National Natural Science Foundation of China (Grant Nos. 61632014, 61627808 and 61802158), the Natural Science Foundation of Gansu Province (Grant No. 20JR10RA605), and the Fundamental Research Funds for the Central Universities (lzujbky-2021-66). Thanks for the computing resources provided by the Supercomputing Center of Lanzhou University.

References

Balazevic

, Allen

and Hospedales

, Multi-relational poincaré graph embeddings, Advances in Neural Information Processing Systems 32 (2019), 4463–4473.

Bordes

, Glorot

, Weston

and Bengio

, A semantic matching energy function for learning with multi-relational data, Machine Learning 94(2) (2014), 233–259.

Bordes

, Usunier

, Garcia-Duran

, Weston

and Yankhnenko

Translating embeddings for modeling multirelational data, Neural Information Processing Systems (NIPS), 2013.

Cai

, Qu

, Li

, Zhang

, Hu

and Hu

, Feature-level fusion approaches based on multimodal EEG data for depression recognition, Information Fusion 59 (2020), 127–138.

Chami

, Wolf

, Juan

D.C.

, Sala

, Ravi

and Ré

Low-dimensional hyperbolic knowledge graph embeddings, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.

Dai

Q.N.

, Tu

D.N.

, Nguyen

D.Q.

and Phung

, A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2 (Short Papers) (2018).

Dettmers

, Minervini

, Stenetorp

and Riedel

Convolutional 2D knowledge graph embeddings, Proceedings of the AAAI Conference on Artificial Intelligence, 2018.

Dong

, Wei

, Zhou

and Xu

Question answering over freebase with multi-column convolutional neural networks, Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2015.

Han

, Cao

, Lv

, Lin

, Liu

, Sun

and Li

OpenKE: An open toolkit for knowledge embedding, Proceedings of the 2018 conference on empirical methods in natural language processing: system demonstrations, 2018.

10.

, Liu

, Ji

and Zhao

Learning to represent knowledge graphs with gaussian embedding, Proceedings of the 24th ACM international on conference on information and knowledge management, 2015.

11.

, He

, Xu

, Liu

and Zhao

Knowledge graph embedding via dynamic mapping matrix, Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: Long papers), 2015

12.

, Liu

, He

and Zhao

Knowledge graph completion with adaptive sparse transfer matrix, Proceedings of the AAAI Conference on Artificial Intelligence, 2016.

13.

, Pan

, Cambria

, Marttinen

and Yu

P.S.

A Survey on knowledge graphs: representation, acquisition, and applications, IEEE Transactions on Neural Networks and Learning Systems, 2021.

14.

Jia

, Wang

, Jin

and Cheng

, Path-specific knowledge graph embedding, Knowledge-Based Systems 151 (2018), 37–44.

15.

Jia

, Wang

, Lin

, Jin

and Cheng

Locally adaptive translation for knowledge graph embedding, Proceedings of the AAAI Conference on Artificial Intelligence, 2016.

16.

Lin

, Liu

, Luan

, Sun

, Rao

and Liu

Modeling relation paths for representation learning of knowledge bases, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015.

17.

Lin

, Liu

, Sun

, Liu

and Zhu

Learning entity and relation embeddings for knowledge graph completion, Proceedings of the AAAI Conference on Artificial Intelligence, 2015.

18.

Mikolov

, Chen

, Corrado

and Dean

Efficient estimation of word representations in vector space, International Conference on Learning Representations, 2013.

19.

Niu

, Li

, Zhang

, Pu

and Li

AutoETER: Automated Entity Type Representation for Knowledge Graph Embedding, Findings of the Association for Computational Linguistics: EMNLP 2020.

20.

Nickel

, Rosasco

and Poggio

Holographic embeddings of knowledge graphs, Proceedings of the AAAI Conference on Artificial Intelligence, 2016.

21.

Nickel

, Tresp

and Kriegel

H.P.

A three-way model for collective learning on multi-relational data, Proceedings of the 28th International Conference on International Conference on Machine Learning, 2011.

22.

Ruffinelli

, Broscheit

and Gemulla

You can teach an old dog new tricks! on training knowledge graph embeddings, In International Conference on Learning Representations.

23.

Sun

, Vashishth

, Sanyal

, Talukdar

and Yang

A Re-evaluation of Knowledge Graph Completion Methods, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 5516–5522).

24.

Socher

, Chen

, Manning

C.D.

and Ng

Reasoning with neural tensor networks for knowledge base completion, Proceedings of the 26th International Conference on Neural Information Processing Systems-Volume 1, 2013.

25.

Sun

, Deng

Z.H.

, Nie

J.Y.

and Tang

RotatE: Knowledge graph embedding by relational rotation in complex space, International Conference on Learning Representations, 2018.

26.

Toutanova

, Chen

, Pantel

, Poon

, Choudhury

and Gamon

Representing text for joint embedding of text and knowledge bases, Proceedings of the 2015 conference on empirical methods in natural language processing, 2015.

27.

Trouillon

, Riedel

, Gaussier

and Bouchard

Complex embeddings for simple link prediction, Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48, 2016.

28.

Wang

, Zhang

, Feng

and Chen

Knowledge graph embedding by translating on hyperplanes, Proceedings of the AAAI Conference on Artificial Intelligence, 2014.

29.

Wang

, Mao

, Wang

and Guo

, Knowledge graph embedding: A survey of approaches and applications, IEEE Transactions on Knowledge & Data Engineering 29(12) (2017), 2724–2743.

30.

Weston

, Bordes

, Yakhnnenko

and Usunier

Connecting language and knowledge bases with embedding models for relation extraction, Conference on Empirical Methods in Natural Language Processing, 2013.

31.

Xiao

, Huang

, Hao

and Zhu

TransG: 826 A generative model for knowledge graph embedding, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016.

32.

Yang

, Yih

W.T.

, He

, Gao

and Deng

Embedding entities and relations for learning and inference in knowledge bases, International Conference on Learning Representations, 2014.

33.

Yoon

H.G.

, Song

H.J.

, Park

S.B.

and Park

S.Y.

A translation-based knowledge graph embedding preserving logical property of relations, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016.

34.

Zeiler

M.D.

ADADELTA: An adaptive learning rate method, arXiv preprint arXiv:1212.5701, 2016.

35.

Zhang

, Wang

, Yang

and Xue

, Knowledge graph embedding by translating in time domain space for link prediction, Knowledge-Based Systems 212 (2021), 106564.

36.

Zhang

, Sun

and Zhang

, Improve the translational distance models for knowledge graph embedding, Journal of Intelligent Information Systems 55(3) (2020), 445–467.

37.

Zhang

, Cai

, Zhang

and Wang

, Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction, Proceedings of the AAAI Conference on Artificial Intelligence 34(3) (2020), 3065–3072.

Two flexible translation-based models for knowledge graph embedding

Abstract

Keywords

1 Introduction

2.1 Translation-based models

2.3 Other models

3 Our method

3.1 Analysis of TransD and lppTransD

3.3.1 Training details

4.1 Data sets

Table 2 Datasets used in the experiments Dataset #Rel #Ent #Train #Valid #Test Avg.#Train/#Ent WN11 11 38,696 112,581 2,609 10,544 2.909 WN18 18 40,943 141,442 5,000 5,000 3.455 FB13 13 75,043 316,232 5,908 23,733 4.214 FB15k 1,345 14,951 483,142 50,000 59,071 32.315

4.2.1 Tasks

4.3 Results analysis and disicussion

4.3.1 Result of triplet classification

Footnotes

Acknowledgments

References

Table 2
Datasets used in the experiments

Dataset #Rel #Ent #Train #Valid #Test Avg.#Train/#Ent

WN11 11 38,696 112,581 2,609 10,544 2.909

WN18 18 40,943 141,442 5,000 5,000 3.455

FB13 13 75,043 316,232 5,908 23,733 4.214

FB15k 1,345 14,951 483,142 50,000 59,071 32.315