KnowMore – knowledge base augmentation with structured web markup

2.1. Knowledge base augmentation (KBA)

The main goal of KBA is to discover facts pertaining to entities and augmenting Knowledge Bases (KBs) with these facts [15,38].

Some previous works have proposed to augment KBs through inference on existing knowledge, i.e. KB statements. Such works typically focus on predicting the type [19] of an entity or finding new relations based on existing data [5,6,34]. Other prior works are more closely relevant to our problem setup; in that they focus on predicting relations with external data. Notable works propose the use of Wikipedia as a text corpus annotated with entities, search for patterns based on existing KB relations, and further apply the patterns to find additional relations for DBpedia [1] or Freebase [25]. News corpora have also been used for augmenting DBpedia through similar approaches [14]. Paulheim et al. [26] proposed to identify common patterns of instances in Wikipedia list pages and apply the patterns to add relations to the remaining entities in the list. Dong et al. proposed ‘Knowledge Vault’ [10], a framework for extracting triples from web pages, aimed at constructing a KB from Web data. Dutta et al. [13] focus on the mapping of relational phrases such as facts extracted by ‘Nell’ and ‘Reverb’ to KB properties. Furthermore, they group the same semantic relationships represented by different surface forms together through Markov clustering. Recent works by Ritze et al. use relational HTML tables available on the Web to fill missing values in DBpedia [30,31]. The authors propose to first match the tables to the DBpedia entities, and then compare several data fusion strategies such as voting and the Knowledge-Based Trust (KBT) score to identify valid facts.

Ristoski et al. proposed an approach to enrich product ads with data extracted from the Web Data Commons [29]. The approach extracts attribute-value pairs from plain text and matches them to database entities with supervised classification models. The notable methods described in previous works are tailored to specific data sources, which have different characteristics compared to markup data. Hence, merely adopting the existing methods to cater for markup data is not sufficient. However, we have revised and adopted some of the features in our proposed approach.

Other works suggest using the whole Web as a potential data corpus through search engines [16,39]. QA-based approaches are often designed to facilitate the filling of values of a specific set of properties, and rely on manually created templates. This limits their application to a constrained sets of properties.

Existing works typically assume that, there is only one true value for a property when resolving conflicts. In contrast, we aim for higher recall by allowing multiple (non-redundant) correct values, catering for the fact that multiple-cardinality properties are wide-spread. Another limitation of existing KBA works is that the novelty of the discovered facts is ignored; there is an overlap between the result and the facts existing in a KB. On the contrary, our approach aims at providing correct and novel results that are of immediate value to the KB.

2.2. Data fusion

Data fusion is defined as “the process of fusing multiple records representing the same real-world object into a single, consistent, and clean representation” [2]. In the context of the Semantic Web, previous works on data fusion can be categorized into two classes – heuristic-based and probability-based.

Heuristic-based Methods. Schultz et al. introduced ‘LDIF’, that uses user provided heuristics to find duplicate real-world entities [33]. Mendes et al. proposed ‘Sieve’, which resolves conflicts in Linked Data from different sources by selecting one value for each property based on quality measures such as recency and frequency [4,20]. ‘ODCleanStore’ provides heuristic-based mechanisms such as max frequency for resolving conflicts during the fusion of Linked Data [17,24]. One of the limitations of such heuristics-based approaches is that they rely on the observation of a specific dataset, which is often not generalizable for other datasets. Furthermore, the used heuristics usually focus on a single aspect of the quality, e.g. recency or frequency, while the quality of a resource is typically influenced by multiple factors to varying degrees.

Probability-based Methods. Zhao et al. [43] proposed an unsupervised probabilistic graphical model to infer true records and source quality based on the false-positives and false-negatives of the data source. Dong et al. [11] introduced data fusion techniques which identify true subject-predicate-object triples, that are extracted by multiple extractors and originate from multiple sources [10]. Pochampally et al. proposed to use joint precision and joint recall to indicate correlation between sources in order to penalize the copying between sources [28]. In later work, the authors proposed a probabilistic model to compute the Knowledge-Based Trust (KBT), i.e., a score for measuring the trustworthiness of the resources [12]. KBT focuses on the general quality of a resource, and is computed based on the overlap of the extracted data and the knowledge base.

Whereas previous works focus only on the correctness of the source and assign equal weights to all the facts from the same source, in contrast, we not only consider the source quality but also features of the predicates, facts and entities. Hence, distinct facts from the same source are classified differently, depending on multiple feature dimensions. Thus, through a more fine-grained classification, our data fusion approach is able to improve both precision and recall. Another difference is that our query-centric data fusion approach does not require the fusion of the entire dataset after partial changes to the corpus, but can be applied iteratively over specific subsets.

OPEN IN VIEWER

Fig. 2.

Proportion of book and movie instances per KB that include selected popular predicates.

Our recently published work presents an entity summarization approach that retrieves entities from WDC and selects distinct facts to build entity descriptions based on clustering [42]. Additional recent work [41] proposes a data fusion approach focused on ensuring correctness of facts obtained from noisy entity descriptions in Web markup. While the focus of the former work was on deduplication, the main focus of the latter was correctness. In contrast, this paper proposes a complete two-step pipeline aiming at obtaining correct and non-redundant facts which augment existing KBs.

3.1. Motivation

Previous works [9,32,40] have investigated the nature of several type-specific subsets of Web markup, namely bibliographic data, metadata about learning content, books and movies. These works assess markup data on several dimensions, such as data quality, the source distribution and the schema usage. Results show the complementary nature of markup data when compared to traditional knowledge bases, where the extent of additional information varies strongly between types.

For a preliminary analysis of DBpedia, Freebase and Wikidata, we randomly select 30 Wikipedia entities of type Movie and Book and retriev the corresponding entity descriptions from all three KBs. We select the 15 most frequently populated properties for each type and provide equivalence mappings across all KB schemas as well as the schema.org vocabulary manually.7

In addition, coverage varies heavily across different properties, with properties such as editor or translator being hardly present in any of the KBs.

Tail entities/types as well as time-dependent properties which require frequent updates, such as the award of a book, are prevalent in markup data [23], yet tend to be underrepresented in structured KBs. Hence, markup data lends itself as data source for the KBA task. However, given the specific characteristics of markup data [40], namely the large amount of coreferences and near-duplicates, the lack of links and the variety of errors, data fusion techniques are required which are tailored to the specific task of KBA from Web markup.

For the purpose of this work, an entity description is considered a semi-structured representation of an actual entity, where the latter is either a physical (e.g. a person) or an abstract notion (e.g. a category or theory). Entity descriptions which represent the same entity are considered to be coreferences.

In particular, our work is concerned with entity descriptions extracted from structured Web markup of Web documents. We refer to such a dataset as M, where the WDC dataset is an example. Data in M consists of entity descriptions e i , each consisting of a set of RDF quads, i.e. a set of ⟨ s , p , o , u ⟩ quadruples which are referring to entities. The elements ⟨ s , p , o , u ⟩ of the quadruple represent subject, predicate, object and the URL of the document from which the triple ⟨ s , p , o ⟩ has been extracted, respectively. An example of an entity description consisting of 3 quadruples is shown in Table 1. Here, for instance, _:node1 corresponds to subject s, <http://schema.org/author> corresponds to predicate p, “Evelyn Waugh” corresponds to object o and <http://example.url>8

There exist n ⩾ 0 subjects { s 1 , s 2 , … , s n } , and consequently, n entity descriptions e i = ⟨ s i , p i , o i ⟩ ∈ E which represent a particular query entity q in M. Here, E is the set of all entity descriptions which (co)refer to entity q. We define a property-value pair ⟨ p , o ⟩ describing the entity q as a fact of q. Note that we explicitly consider multi-valued properties, i.e. a particular predicate p might be involved in more than one fact for a particular entity q.

We define the task of augmenting a particular entity description e q , representing a query entity q within a particular KB from data in a markup corpus M as follows:

Note that F nov represents the final output of the KnowMore pipeline. We consider a fact valid for augmentation, if it meets the following criteria:

As an illustrative example, let q be the book Brideshead Revisited. In a given KB, such as DBpedia, there is an entity description9

Our approach (KnowMore) for addressing the KBA problem defined above consists of two steps as shown in Figure 3, namely (i) entity matching, and (ii) data fusion. We introduce the intuition behind each step below and describe the actual method in the following sections.

OPEN IN VIEWER

Fig. 3.

Overview of pipeline.

Entity matching. The first step, KnowMore match , aims at obtaining candidate facts by collecting the set E of coreferring entity descriptions e i ∈ E from M which describe q and corefer to the entity description e q in a given KB. We use a three step approach in order to efficiently achieve high accuracy results.

Data cleansing to improve general data quality.

Data fusion. During the data fusion step, Know More class , we aim at selecting a subset F nov ⊂ F that fulfills the criteria as listed in Section 3.2. More specifically, we introduce data fusion techniques based on supervised classification to ensure the correctness (Section 5.1) and two deduplication steps to ensure novelty (Section 5.2), namely deduplication with respect to M ( KnowMore ded ) and deduplication with respect to the KB ( KnowMore nov ). The latter is a prerequisite for the KBA task.

For clarity, we summarize the notations used for identifying each step of the KnowMore pipeline and the corresponding in- and outputs in Table 2.

We describe each step of the approach in the following two sections in detail.

4.1. Data cleansing

This initial data cleansing step aims at (i) resolving object references and (b) fixing common errors [21] to improve overall usability of the data.

While schema.org property range definitions are not bound in a strict way but constitute mere recommendations, previous studies [9] observe a strong tendency towards statements which refer to literals rather than objects, i.e. URIs. For instance, within a markup corpus of 44 million quads, 97% of transversal properties referred to literals rather than URIs/objects, despite the fact that only 64% of quads involved properties where schema.org recommends literals as property range [9]. Given this prevalence of literals in Web markup and the need to homogenise entity descriptions for further processing, we resolve object references into literals by replacing object URIs with the labels of the corresponding entity.

In addition, based on earlier work [21] which studied common errors in Web markup, we implement heuristics and apply these to E as a cleansing step thereby improving the quality of the data. In particular, we implemented the heuristics as proposed in [21] to:

Applying these heuristics improves the performance of the subsequent step by providing a wider and higher quality pool of candidates.

4.2. Blocking

Blocking is typically used as a pre-processing step for entity resolution to reduce the number of required comparisons by placing potentially relevant entity descriptions, i.e. potential coreferences, into the same block so that the entity resolution algorithm is applied to entity descriptions within the same block only [7].

Related work [37] shows that string comparison between labels of markup entities is an efficient way for obtaining potential coreferences, whereas the Lucene BM25 retrieval approach has been used successfully by previous works on entity resolution as summarized by [7].

Therefore, we implement the blocking step through entity retrieval using the BM25 retrieval model, i.e. a probabilistic ranking function used to rank matching documents according to their relevance to a given search query, to reduce the search space. We created an index for each type-specific subset using Lucene, and then use the label of e q to query the field name within a type-specific index. Hence, queries for a specific type/label-combination which represents q result in a set of candidate entity descriptions e i 0 ∈ E 0 that potentially describe the same entity as e q .

Given that the name10

For instance, considering the query “Brideshead Revisited” (of type Book), as part of the blocking step we query the Lucene index and obtain 1,657 entity descriptions consisting of 15,940 quads. An excerpt of the result set is shown in Table 3.

When attempting to match entities, one can build on the observation that particular property-value pairs can be considered near-unique identifiers for a specific entity, so-called pseudo-key properties. For instance, taxID can be considered one of the pseudo-key property for instances of type Person, and isbn for instances of type Book. However, as studied by Meusel et al. [23], resolving coreferences simply through pseudo-key properties does not produce sufficient results when applied on sparsely described and heterogeneous entity descriptions obtained from Web markup.

Thus, we adapt the entity matching approach described in [29] to filter out noise in E 0 , for instance, entity descriptions which are not relevant to q but fetched through the initial blocking step due to ambiguous labels. Our matching approach builds on the assumption that the importance of distinct properties differs when computing entity similarity, with pseudo-key properties being the most decisive ones. For example, instances of type Book which share the same author have a higher probability to be equivalent than books which share the same bookFormat.

In order to compute the similarity for each property, we consider all properties as attributes of the feature space A → = { a 1 , a 2 , … , a n } , so that each entity description e can be represented as a vector of values v → = { o a 1 , o a 2 , … , o a n } which represent the objects of the considered ⟨ p , o ⟩ tuples. We construct a similarity vector sim → ( v KB → , v → ) between e q and each entity description e i 0 ∈ E 0 as in Equation 1. (1) sim → ( v KB → , v → ) = { λ a 1 , λ a 2 , … , λ a n } (2) λ a i = sim ( o a i KB , o a i )

In order to compute sim ( o a i KB , o a i ) , we employ datatype-specific similarity metrics, i.e., we implemented one similarity measure for each schema.org datatype,11

We then train a supervised classification model, to make the decision whether or not e i 0 is a match for e q . We experimented with several state-of-the-art classifiers (SVM, Logistic Regression and Naive Bayes). Since Naive Bayes achieves a F 1 score that is 0.08 higher than the best SVM (linear kernel), and 0.123 higher than the Logistic Regression (LR), throughout the remaining paper we rely on a trained Naive Bayes classifier unless otherwise stated. More details about classifier performance are provided in Section 7.1. The classification and clustering implementation in our approach is built on top of the Java-ML toolkit.14

The final result of the entity matching step is the set of coreferring entity descriptions e i ∈ E which constitute candidate facts f i ∈ F for the following steps.

Returning to our running example “Brideshead Revisited”, after removing the unmatched entity descriptions from the blocking result through the entity matching step, there are 44 matched entity descriptions remaining in the result set. Some examples are shown in Table 4, where, for instance, _:node3 had been removed since it refers to the stage play rather than the book and does not match entity q.

5.1. Correctness – supervised classification

The first step ( KnowMore class ) aims at detecting correct facts by learning a supervised model that produces a binary classification for a given fact f ∈ F into one of the labels {‘correct’, ‘incorrect’}. For the classification model, we have experimented with several different approaches, namely Naive Bayes classification, SVM with different kernel functions, Logistic regression, kNN with varying k’s and Random Forest. We rely on a Naive Bayes classification since our experiments have shown superior performance over the second best performing approach Logistic Regression with an increase in F 1 score of 0.016, and over SVM (linear kernel) with an increase in F 1 score of 0.044. Details about the performance of the three best performing approaches (Naive Bayes, SVM and Logistic regression) are provided in Section 7.2. We introduce the features used for our supervised learning approach in Table 5 and describe them in detail below. Through an initial data analysis step, all features have been identified as potential indicators of fact correctness.

While we aim to detect the correctness of a fact, we consider characteristics of the source, that is the Pay-Level-Domain (PLD, i.e. the sub-domain of a public top-level domain, which Website providers usually pay for), from which a fact originates, the entity description, the predicate term as well as the fact itself. The four different categories are described below.

Source level. As has been widely studied in previous works, source quality is an important indicator for data fusion [12,28,43]. Features t 1 r , … , t 3 r are generated from the PageRank score as an authority indicator of the PLD from which a fact is extracted, assuming a higher PageRank indicates higher authority and hence quality. Based on the intuition that more errors across the markup from of respective PLD indicate a higher potential of this PLD to provide incorrect facts, we consider the rate of common errors detected based on previously identified heuristics [21] to compute features t 4 r , … , t 6 r . Finally, we use precision of a PLD computed based on our ground truth (Section 6.2) as quality indicators in features t 7 r , … , t 9 r .

Entity level. Based on the data analysis, entity descriptions containing a large number of facts are usually of higher quality. Thus, we use the size of entity descriptions, reflected through features t 1 e , … , t 3 e , as additional indicator of quality.

Property level. The quality of facts strongly varies across predicates, as identified in previous studies [23,40], with some properties being more likely to be part of a correct fact than others. One example of a predicate often included in incorrect statements is datePublished of a movie, that is often mistakenly used to describe the publishing time of the Web document. Following this observation, we extract features t 1 p , … , t 5 p to consider characteristics of the involved predicate terms, such as their frequency.

Given that our candidate set contains vast amounts of near-duplicate facts, we approach the problem of identifying semantically equivalent statements through clustering of facts which use varied surface terms for the same or overlapping meanings. We employ the X-Means algorithm [27], as it is able to automatically determine the number of clusters. This clustering step aims at grouping or canonicalizing different literals or surface forms for specific object values. For instance, Tom Hanks and T. Hanks are equivalent surface forms representing the same entity. To detect duplicates and near-duplicates, we first cluster facts that have the same predicate p into n clusters ( c 1 , c 2 , … , c n ) ∈ C . In this way, considering string similarity, we can canonicalize equivalent surface forms. The performance of the clustering on removing near duplicates is discussed in Section 7.3. Another challenge considered here is the cardinality of predicates. Depending on the predicate, the number of potentially correct statements varies. For example, actor is associated with multiple values, whereas duration normally has only one valid statement. This is reflected in the cluster amount n for a given predicate ( t 3 p ). The intuition behind feature t 4 p is that the average size of clusters is an indicator of the frequency of facts in p which usually correlates with the quality. Feature t 5 p is extracted based on the observation that in most cases, wrong facts have lower frequency than average, thus the variance is larger if there are wrong facts among the facts of p.

Fact level. Fact frequency [20] has been used in previous data fusion works and is shown to provide efficient features for determining the correctness of facts. Based on these insights, we extract features t 1 f and t 2 f . We consider the size of a cluster as feature t 2 f indicating the frequency of a fact, where the normalized size of cluster c i is ( | c i | / ( ∑ j = 1 n | c j | .

From the computed features we train the classifier for classifying the facts from F into the binary labels {‘correct’, ‘incorrect’}. More details about the training and evaluation through 10-fold cross-validation are presented in Section 7.2. The ‘correct’ facts form a set F class that is the input for the next steps.

Again returning to our running example, after removing wrong facts from the candidate facts, such as datePublished: 1940 through the classification step, we obtain 37 correct facts in the result set for the query Brideshead Revisited, type:(Book). An excerpt of the resulting facts are shown in Table 6.

A fact f is considered to be novel with respect to the KBA task, if it fulfills the conditions: i) is not duplicate with other facts selected from our source markup corpus M, ii) is not duplicate with any facts existing in the KB. Each of these two conditions corresponds to a deduplication step.

Deduplication with respect to M ( KnowMore ded ). As introduced in Section 5.1, we detect near-duplicates via clustering. For each predicate p, all the facts f = ⟨ p , o i ⟩ corresponding to p are clustered into n clusters { c 1 , c 2 , … , c n } . Each cluster c i , i = 1 , … , n contains a set of near-duplicates. To fulfill i), we select only one fact from each cluster by choosing the fact that is closer to the cluster’s centroid. This results in the fact set F ded that is the input for next deduplication step.

Deduplication with respect to KB ( KnowMore nov ). We compute the similarity sim ( f i , f KB ) between a fact f KB in a respective KB for a particular predicate p and a fact f i for the same (mapped) predicate p in F ded with the datatype-specific similarity metrics as introduced in Section 4. If sim ( f i , f KB ) is higher than a threshold τ, we remove the fact along with its near-duplicates, i.e. the facts in the same cluster from the candidate set F nov . We explain τ and its configuration during the experimental Section 6.3. The facts selected from F nov in this step are the final result for augmenting the KB.

In our running example, the fact author: Waugh, Evelyn is removed during the deduplication with regard to M as it is a duplicate of fact author: Evelyn Waugh, which has been selected as more representative.

With respect to the KBA task, consider the augmentation of the example entity “Brideshead Revisited” (of type Book) as illustrated in Table 7. The example facts #2 and #3 would be valid results of the KBA task for DBpedia since they are novel, while only fact #2 is a valid augmentation for Freebase and Wikidata as it is the only fact that is novel.

Note that our deduplication step considers and supports multi-valued properties. By relying on the clustering features, computed during the fusion step, we select facts from multiple clusters (corresponding to multiple predicates) as long as they are classified as correct. As documented by the evaluation results (Section 7), this does not negatively affect precision while improving recall for multi-valued properties.

6.1. Data

Dataset. We use the WDC2015 dataset,15

Entities & KBs to Augment. As input for the KBA task, we randomly select 30 entities from Wikipedia for each type Book and Movie. We evaluate the performance of our approach for augmenting entity descriptions of these 60 entities obtained from three different KBs: DBpedia (DB), Freebase (FB) and Wikidata (WD). For DBpedia, we retrieve entity descriptions through the SPARQL endpoint16

The full list of entities can be found online ¹³ . An analysis of the completeness of these obtained entity descriptions is shown in Fig. 2 in Section 3.1.

Properties to Augment. To simplify the schema mapping problem between WDC data and the respective KBs while at the same time taking advantage of the large-scale data available in our corpus, we limit the task to entities annotated with the http://schema.org ontology for this experiment. Previous works have shown that schema.org is the only vocabulary which is consistently used at scale [23]. We manually create a set of schema mappings that maps the schema.org vocabularies to the DB, FB, WD vocabularies. For this, we first select all the schema.org predicates appearing in F. We identify the ones that have equivalent properties within all involved vocabularies and create equivalence mappings (owl:equivalentProperty). The list of predicates and the mapping statements can be found online ¹³ .

This section describes the ground truth used for training and testing together with the evaluation metrics used for assessing performance in different tasks.

6.3. Configuration & baselines

Configuration. We deploy our approach as described in Section 4.3 and 5. For the entity matching step, we use Lucene for indexing and BM25 retrieval with the Lucene default configuration where k 1 = 1.2 , b = 0.75 . For the deduplication with respect to KBs, we report the evaluation result of KnowMore nov using different τ = { 0.3 , 0.5 , 0.7 } in Section 7.3.

Baselines. We compare ( KnowMore class ) with PrecRecCorr that is proposed by Pochampally et al. [28] and CBFS  [42]. To the best of our knowledge, the CBFS approach is the only available method so far directly geared towards the challenges of markup data, while PrecRecCorr represents a recent and highly related data fusion baseline. We also present the results of KnowMore match and KnowMore class using different classifiers.

KnowMore class : facts selected based on the KnowMore class pipeline from F, using Naive Bayes as classifier for selecting correct facts.

7.1. Entity matching

As the entity matching step ( KnowMore match ) is a precondition for the subsequent fusion step, we provide evaluation results for this step and compare it to entity descriptions obtained through BM25@k as baseline. The BM25 configuration is the same as used in our approach (Section 6.3). Since we obtain the corresponding URIs of Freebase and Wikidata entities through the sameAs link in DBpedia, here we present only the result of matching entity descriptions to DBpedia. Table 8 shows the evaluation results of the standard precision P, recall R and F 1 scores.

As presented in Table 8, our supervised matching approach KnowMore match (using NB as classifier) achieves high F 1 scores of 0.83 and 0.887 respectively, thereby outperforming the BM25@20 and BM25@50 baselines and providing a sound set of candidates for the subsequent step. One reason for the poor precision of the baseline BM25@k is the introduction of false positives by relying on the value of property name, which is inherently ambiguous for the matching task. Since KnowMore match uses BM25 retrieval as a blocking step and then applies a classifier that takes all the properties into consideration, precision is significantly improved. Among different configurations of the KnowMore match approach, the Naive Bayes classifier achieves the highest precision and F 1 score compared to the Logistic Regression (LR) classifier KnowMore match ( L R ) and SVM KnowMore match (SVM) on both types.

7.2. Correctness – data fusion

The results for KnowMore class as well as the baselines are shown in Table 9. As shown in the table, our chosen configuration, i.e. using a Naive Bayes classifier ( KnowMore class ) achieves highest F 1 scores among all the different configurations. Compared to the different configurations of our approach deploying different classification models, the precision is 0.025 (0.089) higher, and the F 1 score is 0.016 (0.044) higher than using LR (SVM). Potential reasons for the strong performance of the NB classifier could be 1) the assumed independence of features, which is one of the expectations of NB classifiers and 2) that, compared to other models, NB tends less to overfitting when the amount of training data is limited.

The presented F1 score of the PrecRecCorr baseline is the best possible configuration for our given task, where we experimented with different thresholds ( [ 0 , 1 ] , gap 0.1 ) as discussed in [28] and identified 0.5 experimentally as the best possible configuration. We observe that the F 1 score of our approach is 0.141 higher than PrecRecCorr and 0.119 higher than CBFS on average across datasets. This indicates that our approach provides the most efficient balance between precision and recall across the investigated datasets, when applied to the novel task of data fusion from Web markup. Although, the precision of the baseline approach PrecRecCorr is 0.013 higher than the one from KnowMore class on the Book dataset, the baseline fails to recall a large amount of correct facts, where the recall of KnowMore class is approximately 0.388 higher. This also is reflected in the average size of entity descriptions obtained through both approaches, where the entity descriptions from PrecRecCorr consist of 4.88 statements on average, and the ones from KnowMore class are 8.83, indicating a larger potential for the KBA task.

The reason that the baseline PrecRecCorr has lower recall compared to KnowMore class is that the PrecRecCorr approach relies on the assumption that the facts extracted from high quality sources (i.e. PLDs that provide larger percent of correct facts) are more likely to be true. Although this is a reasonable assumption that results in a high precision result, it penalises facts that originate from sources (PLDs) which contributed wrong facts as part of the training set. This leads to a large number of false negatives. KnowMore class , however, considers the source quality (source level features), yet uses several other features in the decision making process and is more robust to the source quality and thus results in higher recall.

The CBFS approach is built based on the intuition that the more frequent a fact is, the higher the chance that it is true. Based on the results, this assumption does not necessarily lead to higher precision compared to the assumption used by the PrecRecCorr approach. Since KnowMore class utilizes multiple features extracted from several different dimensions, including fact frequency, the classifier trained on these multi-dimensional features results in better performance.

A more detailed discussion of the potential impact on the KBA task is provided in Section 9, investigating the KBA potential beyond the narrow definition of the investigated task of this setup, e.g. by augmenting additional predicates not already foreseen in a given KB schema or to populate KBs with additional entities.

We ran 20 iterations of 10-fold cross validation for different baselines and our approach in order to test the statistical significance of our results, as suggested in [8]. We conducted paired T-tests and employed Bonferroni’s correction for Type-I error inflation. We found that all comparisons were statistically significant at the 95% confidence interval ( p < 0.05 ), with some comparisons significant at the 99% confidence interval ( p < 0.01 ). Tables 8 and 9 reflect the average values over 20 runs of the corresponding algorithms.

7.3. Novelty

This section presents the evaluation results for the deduplication steps introduced in Section 5.2.

Diversity. Table 10 presents the evaluation result before ( Dist % ( F class ) ) and after ( Dist % ( F ded ) ) the step KnowMore ded .

The Dist % of facts improves by 1.3 percentage points for the Movie dataset and by 13.5 percentage points for the Book dataset (Table 10). The less improvement gain for the Movie dataset presumably is due to the nature of the randomly selected Movie entities. As these appear to be mostly tail entities, candidate facts in our markup corpus M are fewer and less redundant. Hence, the amount of duplicates and near-duplicates is smaller, reducing the effect of the deduplication step. Hence, deduplication is of particular importance for popular and well-represented entities.

Novelty with respect to KB. The results before ( Nov ( F ded ) ) and after ( Nov ( F nov ) ) the deduplication for specific KBs using different similarity thresholds (τ) are presented in Table 11.

Since our approach is not aware of the total number of novel facts for a particular entity description on the Web a priori, in this evaluation, we consider all the novel facts in F ded as the gold standard, and compute the recall of F nov after applying the KnowMore nov accordingly. We evaluate the performance of KnowMore nov using τ in { 0.3 , 0.5 , 0.7 } since (1) 0.5 is widely used as threshold for identifying duplicate text using cosine similarity, (2) for computing DateTime similarity between facts, the possible similarities according to our metric are: { 0 , 0.33 , 0.5 , 0.67 , 1 } . These 3 selected thresholds are most influential on the selection of DateTime facts, thus can produce most conclusive results for showing the trade-off between novelty and recall. As shown in Table 11, even though there is a trade-off between novelty and recall, different values of τ do not have a strong influence on the evaluation metrics. One of the reasons is that, a large proportion of facts have non-literal (e.g. numeric) values. While our datatype-specific similarity computes a binary (0 or 1) score in these cases, it is not influenced by the selection of τ.

Consider τ = 0.5 as an example. The KnowMore nov step improves novelty by 0.282 on average across datasets and KBs compared to the result of the KnowMore ded step ( F ded ), and is able to recall over 90% of the novel facts. Our final result F nov shows a novelty of over 90% on average, what translates to a minor amount of near-duplicates and a sufficient novelty for augmenting the target KBs.

OPEN IN VIEWER

Fig. 4.

Proportion of augmented entity descriptions with KnowMore . Only predicates which were augmented in at least one KB are shown.

This section discusses the coverage gain, as an indicator of the KnowMore performance in the particular KBA task described in Section 3.2. Since the addressed task is fairly narrow, i.e. dealing with the augmentation of a selected set of attributes and entities existing in all three investigated KBs, we discuss the overall KBA potential of Web markup data beyond our ground truth dataset in Section 9.1.

OPEN IN VIEWER

Fig. 5.

P, R and F1 score using different size of the training data for KnowMore match . X-axis shows the percent of training data, Y-axis shows the P/R/F1 value.

Figure 4 shows the coverage gain on the previously empty slots as shown in Fig. 2 per predicate and KB for our selected 30 entities (per type). Based on the evaluation result, the KnowMore pipeline shows a coverage gain of 34.75% on average across different properties for DBpedia, 39.42% for Freebase and 36.49% for Wikidata. We observe that the obtained gain varies strongly between predicates and entity types, with a generally higher gain for book-related facts. For instance, within the Movie case, for property actor we were able to gain 100% coverage in both DBpedia and Freebase, while the property award shows a coverage gain of 10% or less for all three KBs. Reasons behind low coverage gain for a particular property are 2-fold: 1) the lack of data in the Web markup data corpus, and 2) the lack of true facts in the real world for a particular attribute, e.g. only a small proportion of movies have won an award. On average, we obtained 2.8 (6.8) facts for each movie (book) entity in our experimental dataset.

For a more thorough discussion of the KBA potential of the KnowMore approach, i.e. on tasks beyond our ground truth dataset, we refer the reader to Section 9.1.

As described in Section 6.2, our ground truth consists of judgments for 217 (298) entity descriptions for Movie (Book) entities for the entity matching step KnowMore match , and 456 (371) facts of Movie (Book) entities for the data fusion step KnowMore class . The experimental evaluation in Section 7 is based on the averaged P/R/F1 scores from a 10-fold cross validation (90% training and 10% testing). To evaluate how performance is affected by the scale of the training data, we have conducted our experiments with subsets of the data which vary in size. In particular, for each type, we run 10 experiments where the subset of the training data uses n percent of the original training data set and n is in the range of [10, 100]. 10-fold cross-validation is performed for each n.

Figure 5 presents the results for KnowMore match , where the X-axis indicates the size of the training data and the Y-axis shows the P/R/F1 scores. The F1 score reaches 0.8 (0.88) at 60% (40%) percent of training data, and the P, R and F1 curves become steady with training data sets of the size of 80%(50%) for the Movie (Book) type, i.e. training data sets of at least 130 (119) entity descriptions for the Movie (Book) type.

Similar characteristics can be observed for the KnowMore class approach in Fig. 6, where the F1 score reaches 0.89 (0.87) at 70% (70%) of training data for Movie (Book). Hence, results suggest that even with comparably limited amounts of training data, reasonable performance can be achieved, thereby supporting the application across types and datasets. For instance, the cost for retrieving 80% (70%) of our entity matching (data fusion) ground truth from CrowdFlower with the approach as described in Section 6.2, is less than 15 USD and the time required is less than 24 hours for each type.

8.2. Model performance across types

In this section, we assess the performance of KnowMore across different types, i.e. without a type-specific training phase. Thus, we merge the aforementioned type-specific datasets Book and Movie and perform a 10-fold cross-validation using the query sets for both types. The averaged performance of KnowMore match on this cross type dataset is P = 0.782 , R = 0.892 , F 1 = 0.833 , where the precision is lower than the type-specific results (Section 7), but the overall performance and F1 score is still comparable, the latter being slightly above the F1 score of the type-specifically trained Movie model (0.83). The result of KnowMore class is P = 0.902 , R = 0.825 , F 1 = 0.862 . Here, the precision is higher than the type-specific result for Book model (0.886) and lower than the one from the Movie model (0.967). This suggests that our approach can work on models trained on cross-type data. In order to fully validate this finding, further studies are required with more diverse query sets as well as datasets involving larger amounts of types.

9.1. Potential of KBA from web markup

Beside the specific KBA task evaluated in this paper, where we aim at (i) augmenting existing entities in a given KB by (ii) populating a given set of properties from a given KB schema for these entities, markup data shows large potential to augment KBs with properties and entities not yet present in KBs. Investigating the data from our two datasets (Movie, Book) and another set of 30 randomly selected entities of type Product, we observe that a large proportion of statements in the WDC dataset involve properties not yet present in any of the KB schemas. For instance, for movies (books), 62.5% (66.8%) of entity descriptions in F contain facts not yet present in our set of mapped predicates. Comparing product descriptions from F, we detect 20.6% statements containing properties not yet present in the DBpedia ontology at all (verified through manual inspection).

In order to better highlight the potential of Web markup data to support knowledge base augmentation, we apply KnowMore on all the movie and book entities from DBpedia. Out of all the 106,613 movie entities in DBpedia (10 Jan., 2017 version), we found coreferences in our markup corpus for 101,069 distinct entities (94.8%). Out of 35,577 book entities in DBpedia, we found coreferences for 34,964 entities in our markup corpus M (98.3%). In total, this resulted in 4,412,337 (1,783,231) instances, i.e. markup nodes, and 42,624,281 (16,580,862) candidate facts for the selected set of movies (books). On average, we found 5.06 (7.98) facts for each movie (book) instance. Based on the experimental results, our KnowMore approach obtained 511,409 (279,013) new facts for all DBpedia movie (book) entities. Note that this includes only entities already present in DBpedia. Whereas there is a wide variety of instances of both types which are not present in DBpedia but in markup, our approach could be used to populate DBpedia, in particular with less popular and long-tail entities. Thus, we observe a considerable potential for augmentation of KBs with new entities, as opposed to augmenting existing ones.

To assess performance in such cases, we randomly select 30 names of products under the requirement that each appears in at least 20 different PLDs in WDC, to ensure that there is sufficient consensus on the name being a legitimate product title. Manual inspection confirmed that none of such randomly selected products is represented in DBpedia.

By running the KnowMore class approach on 30 selected product entities, we found 136 correct facts in total, resulting in 4.53 facts for each entity on average. Table 12 shows the performance of KnowMore class and our baselines on this dataset.

Results indicate that the performance gain of our approach is particularly evident on such long-tail entities as represented in our Product dataset.

Results demonstrate that KnowMore is able to exploit Web markup data for KBA tasks. Further improvement can be gained by applying our approach on a focused crawl, targeted towards a specific KBA task, such as movie enrichment, rather than a cross-domain Web crawl such as the WDC/Common Crawl.

In contrast to related KBA approaches such as [16] or [39], it is worth noting that our approach is trained for particular entity types only, not towards particular properties, as is the case with the aforementioned approaches. Hence, KnowMore can be adapted to a wider range of scenarios with less effort than previous KBA approaches. In addition, we have demonstrated in Section 8.2 that our models can potentially generalise across types.

Performance strongly differs between query sets, and hence, type-specific markup datasets, what presumably is caused by the variance in quality and quantity of facts in the WDC corpus between distinct types. Particular challenges arise from entities with a large amount of coreferences, where data usually originates from a wide variety of sources with varying degrees of quality. Compared to the baselines, our results indicate a particular strong performance gain of our approach in such cases.

Another limitation is our exclusive focus on schema.org statements. This constraint is motivated by the costliness of providing high-quality schema mappings between markup statements and three KBs and the fact that schema.org is the vocabulary of most widespread use [23]. While schema.org adopters usually are motivated by the goal to improve their search result rankings, one assumption is that other vocabularies might show a different distribution of types and predicates, due to distinct motivations. This deserves deeper investigation as part of future work.

In this context, it is worth noting that our KBA task setup ignored a large part of the markup data, i.e. 49.3% of facts in our type-specific subsets do not involve any of our selected schema.org properties. To consider other vocabularies, we are currently aiming at including a preliminary schema matching step with the intention of improving recall further.

Another important aspect concerns the temporal nature of fact correctness, specifically for highly dynamic predicates, such as the price tag of a particular product. While we do not consider temporal features as such, we argue that the dynamic nature of markup annotations is well-suited to augment particularly dynamic statements. This suggests particular opportunities for updating or complementing KBs with dynamic knowledge sourced from Web markup.