Linked data quality of DBpedia,Freebase,OpenCyc,Wikidata,and YAGO

3.2.1. Accuracy

Definition of dimension. Accuracy is “the extent to which data are correct, reliable, and certified free of error” [45].

Discussion. Accuracy is intuitively an important dimension of data quality. Previous work on data quality has mainly analyzed only this aspect [45]. Hence, accuracy has often been used as synonym for data quality [37]. Bizer [9] highlights in this context that Accuracy is an objective dimension and can only be applied on verifiable statements.

Batini et al. [6] distinguish between syntactic and semantic accuracy: Syntactic accuracy describes the formal compliance to syntactic rules without reviewing whether the value reflects the reality. The semantic accuracy determines whether the value is semantically valid, i.e., whether the value is true. Based on the classification of Batini et al., we can define the metric for Accuracy as follows:

Definition of metric. The dimension Accuracy is determined by the criteria

Syntactic validity of RDF documents,

Syntactic validity of RDF documents The syntactic validity of RDF documents is an important requirement for machines to interpret an RDF document completely and correctly. Hogan et al. [27] suggest using standardized tools for creating RDF data. The authors state that in this way normally only little syntax errors occur, despite the complex syntactic representation of RDF/XML.

RDF data can be validated by an RDF validator such as the W3C RDF validator.14

Syntactic validity of literals Assessing the syntactic validity of literals means to determine to which degree literal values stored in the KG are syntactically valid. The syntactic validity of literal values depends on the data types of the literals and can be automatically assessed via rules [20,32]. Syntactic rules can be written in the form of regular expressions. For instance, it can be verified whether a literal representing a date follows the ISO 8601 specification. Assuming that L is the infinite set of literals, we can state: m synLit ( g ) = | { ( s , p , o ) ∈ g ∣ o ∈ L ∧ synValid ( o ) } | | { ( s , p , o ) ∈ g ∣ o ∈ L } |

In case of an empty set in the denominator of the fraction, the metric should evaluate to 1.

Semantic validity of triples The criterion Semantic validity of triples is introduced to evaluate whether the statements expressed by the triples (with or without literals) hold true. Determining whether a statement is true or false is strictly speaking impossible (see the field of epistemology in philosophy). For evaluating the Semantic validity of statements, Bizer et al. [9] note that a triple is semantically correct if it is also available from a trusted source (e.g., Name Authority File), if it is common sense, or if the statement can be measured or perceived by the user directly. Wikidata has similar guidelines implemented to determine whether a fact needs to be sourced.15

We measure the Semantic validity of triples based on empirical evidence, i.e., based on a reference data set serving as gold standard. We determine the fulfillment degree as the precision that the triples which are in the KG g and in the gold standard G S have the same values. Note that this measurement is heavily depending on the truthfulness of the reference data set.

Formally, let no g , G S = | { ( s , p , o ) ∣ ( s , p , o ) ∈ g ∧ ( x , y , z ) ∈ G S ∧ equi ( s , x ) ∧ equi ( p , y ) ∧ equi ( o , z ) } | be the number of triples in g to which semantically corresponding triples in the gold standard G S exist. Let no g = | { ( s , p , o ) ∣ ( s , p , o ) ∈ g ∧ ( x , y , z ) ∈ G S ∧ equi ( s , x ) ∧ equi ( p , y ) } | be the number of triples in g where the subject-relation-pairs ( s , p ) are semantically equivalent to subject-relation-pairs ( x , y ) in the gold standard. Then we can state: m semTriple ( g ) = no g , G S no g

In case of an empty set in the denominator of the fraction, the metric should evaluate to 1.

Definition of dimension. Trustworthiness is defined as “the degree to which the information is accepted to be correct, true, real, and credible” [47]. We define it as a collective term for believability, reputation, objectivity, and verifiability. These aspects were defined by Wang et al. [45] and Naumann [37] as follows:

Discussion. In summary, believability considers the subject (data consumer) side; reputation takes the general, social view on trustworthiness; objectivity considers the object (data provider) side, while verifiability focuses on the possibility of verification.

Trustworthiness has been discussed as follows:

Definition of metric. We define the metric for the data quality dimension Trustworthiness as a combination of trustworthiness metrics on both KG and statement level. Believability and reputation are thereby covered by the DQ criterion Trustworthiness on KG level (metric m graph ( h g ) ), while objectivity and verifiability are covered by the DQ criteria Trustworthiness on statement level (metric m fact ( g ) ) and Indicating unknown and empty values (metric m NoVal ( g ) ). Hence, the fulfillment degree of a KG g w.r.t. the dimension Trustworthiness is measured by the metrics m graph , m fact , and m NoVal , which are defined as follows.

Trustworthiness on KG level The measure of Trustworthiness on KG level exposes a basic indication about the trustworthiness of the KG. In this assessment, the method of data curation as well as the method of data insertion is taken into account. Regarding the method of data curation, we distinguish between manual and automated methods. Regarding the data insertion, we can differentiate between: 1. whether the data is entered by experts (of a specific domain), 2. whether the knowledge comes from volunteers contributing in a community, and 3. whether the knowledge is extracted automatically from a data source. This data source can itself be either structured, semi-structured, or un-structured. We assume that a closed system, where experts or other registered users feed knowledge into a system, is less vulnerable to harmful behavior of users than an open system, where data is curated by a community. Therefore, we assign the values of the metric for Trustworthiness on KG level as follows: m graph ( h g ) = 1 manual data curation, manual data insertion in a closed system 0.75 manual data curation and insertion, both by a community 0.5 manual data curation, data insertion by community or data insertion by automated knowledge extraction 0.25 automated data curation, data insertion by automated knowledge extraction from structured data sources 0 automated data curation, data insertion by automated knowledge extraction from unstructured data sources Note that all proposed DQ metrics should be seen as suggestions of how to formulate DQ metrics. Hence, other numerical values and other classification schemes (e.g., for m graph ( h g ) ) might be taken for defining the DQ metrics.

Trustworthiness on statement level The fulfillment of Trustworthiness on statement level is determined by an assessment whether a provenance vocabulary is used. By means of a provenance vocabulary, the source of statements can be stored. Storing source information is an important precondition to assess statements easily w.r.t. semantic validity. We distinguish between provenance information provided for triples and provenance information provided for resources.

The most widely used ontologies for storing provenance information are the Dublin Core Metadata terms16

Indicating unknown and empty values If the data model of the considered KG supports the representation of unknown and empty values, more complex statements can be represented. For instance, empty values allow to represent that a person has no children and unknown values allow to represent that the birth date of a person in not known. This kind of higher explanatory power of a KG increases the trustworthiness of the KG. m NoVal ( g ) = 1 unknown and empty values are used 0.5 either unknown or empty values are used 0 otherwise

Definition of dimension. Consistency implies that “two or more values [in a dataset] do not conflict each other” [35].

Discussion. Due to the high variety of data providers in the Web of Data, a user must expect data inconsistencies. Data inconsistencies may be caused by (i) different information providers, (ii) different levels of knowledge, and (iii) different views of the world [9].

In OWL, restrictions can be introduced to ensure consistent modeling of knowledge to some degree. The OWL schema restrictions can be divided into class restrictions and relation restrictions [11].

Class restrictions refer to classes. For instance, one can specify via owl:disjointWith that two classes have no common instance.

Relation restrictions refer to the usage of relations. They can be classified into value constraints and cardinality constraints.

Value constraints determine the range of relations. owl:someValuesFrom, for instance, specifies that at least one value of a relation belongs to a certain class. If the expected data type of a relation is specified via rdfs:range, we also consider this as relation restriction.

Cardinality constraints limit the number of times a relation may exist per resource. Via owl:Functionalproperty and owl:Inverse FunctionalProperty, global cardinality constraints can be specified. Functional relations permit at most one value per resource (e.g., the birth date of a person). Inverse functional relations specify that a value should only occur once per resource. This means that the subject is the only resource linked to the given object via the given relation.

Definition of metric. We can measure the data quality dimension Consistency by means of (i) whether schema constraints are checked during the insertion of new statements into the KG and (ii) whether already existing statements in the KG are consistent to specified class and relation constraints. The fulfillment degree of a KG g w.r.t. the dimension consistency is measured by the metrics m checkRestr , m conClass , and m conRelat , which are defined as follows.

Check of schema restrictions during insertion of new statements Checking the schema restrictions during the insertion of new statements can help to reject facts that would render the KG inconsistent. Such simple checks are often done on the client side in the user interface. For instance, the application checks whether data with the right data type is inserted. Due to the dependency to the actual inserted data, the check needs to be custom-designed. Simple rules are applicable, however, inconsistencies can still appear if no suitable rules are available. Examples of consistency checks are: checking the expected data types of literals; checking whether the entity to be inserted has a valid entity type (i.e., checking the rdf:type relation); checking whether the assigned classes of the entity are disjoint, i.e., contradicting each other (utilizing owl:disjointWith relations). m checkRestr ( h g ) = 1 schema restrictions are checked 0 otherwise

Consistency of statements w.r.t. class constraints This metric is intended to measure the degree to which the instance data is consistent with the class restrictions (e.g., owl:disjointWith) specified on the schema level.

In the following, we limit ourselves to the class constraints given by all owl:disjointWith statements defined on the schema level of the considered KG. I.e., let C C be the set of all class constraints, defined as C C : = { ( c 1 , c 2 ) ∣ ( c 1 , owl:dis jointWith , c 2 ) ∈ g } .18

Consistency of statements w.r.t. relation constraints The metric for this criterion is intended for measuring the degree to which the instance data is consistent with the relation restrictions (e.g., indicated via rdfs:range and owl:FunctionalProperty) specified on the schema level. We evaluate this criterion by averaging over the scores obtained from single metrics m conRelat , i indicating the consistency of statements w.r.t. different relation constraints: m conRelat ( g ) = 1 n ∑ i = 1 n m conRelat , i ( g )

In case of evaluating the consistency of instance data concretely w.r.t. given rdfs:range and owl:FunctionalProperty statements,19

Let R r be the set of all rdfs:range constraints, R r : = { ( p , d ) ∣ ( p , rdfs: range , d ) ∈ g ∧ isDatatype ( d ) } and R f be the set of all owl:FunctionalProperty constraints, R f : = { ( p , d ) ∣ ( p , rdf:type , owl:FunctionalProperty ) ∈ g ∧ ( p , rdfs:range , d ) ∈ g ∧ isDatatype ( d ) }

Then we can define the metrics m conRelatRg ( g ) and m conRelatFct ( g ) as follows: m conRelatRg ( g ) = | { ( s , p , o ) ∈ g ∣ ∃ ( p , d ) ∈ R r : datatype ( o ) ≠ d } | | { ( s , p , o ) ∈ g ∣ ∃ ( p , d ) ∈ R r } | m conRelatFct ( g ) = | { ( s , p , o ) ∈ g | ∃ ( p , d ) ∈ R f : ¬ ∃ ( s , p , o 2 ) ∈ g : o ≠ o 2 } | | { ( s , p , o ) ∈ g ∣ ∃ ( p , d ) ∈ R f } |

In case of an empty set of relation constraints ( R r or R f ), the respective metric should evaluate to 1.

3.3.1. Relevancy

Definition of dimension. Relevancy is “the extent to which data are applicable and helpful for the task at hand” [45].

Discussion. According to Bizer [9], Relevancy is an important quality dimension, since the user is confronted with a variety of potentially relevant information on the Web.

Definition of metric. The dimension Relevancy is determined by the criterion Creating a ranking of statements.20

Creating a ranking of statements By means of this criterion one can determine whether the KG supports a ranking of statements by which the relative relevance of statements among other statements can be expressed. For instance, given the Wikidata entity “Barack Obama” (wdt:Q76) and the relation “position held” (wdt:P39), “President of the United States of America” (wdt:Q11696) has a “preferred rank” (wdo:PreferredRank) (until 2017), while older positions which he holds no more are ranked as “normal rank” (wdo:NormalRank). m Ranking ( g ) = 1 ranking of statements supported 0 otherwise

Note that this criterion refers to a characteristic of the KG and not to a characteristic of the system that hosts the KG.

Definition of dimension. Completeness is “the extent to which data are of sufficient breadth, depth, and scope for the task at hand” [45].

We include the following two aspects in this dimension, which are separate dimensions in Wang et al.’s framework:

Discussion. Pipino et al. [39] divide Completeness into

Schema completeness, i.e., the extent to which classes and relations are not missing,

Definition of metric. We follow the above-mentioned distinction of Pipino et al. [39] and determine Completeness by means of the criteria Schema completeness, Column completeness, and Population completeness.

The fulfillment degree of a KG g w.r.t. the dimension Completeness is measured by the metrics m cSchema , m cCol , and m cPop , which are defined as follows.

Schema completeness By means of the criterion Schema completeness, one can determine the completeness of the schema w.r.t. classes and relations [39]. The schema is assessed by means of a gold standard. This gold standard consists of classes and relations which are relevant for the use case. For evaluating cross-domain KGs, we use as gold standard a typical set of cross-domain classes and relations. It comprises (i) basic classes such as people and locations in different granularities and (ii) basic relations such as birth date and number of inhabitants. We define the schema completeness m cSchema as the ratio of the number of classes and relations of the gold standard existing in g, no clatg , and the number of classes and relations in the gold standard, no clat . m cSchema ( g ) = no clatg no clat

Column completeness In the traditional database area (with fixed schema), by means of the Column completeness criterion one can determine the degree by which the relations of a class, which are defined on the schema level (each relation has one column), exist on the instance level [39]. In the Semantic Web and Linked Data context, however, we cannot presume any fixed relational schema on the schema level. The set of possible relations for the instances of a class is given “at runtime” by the set of used relations for the instances of this class. Therefore, we need to modify this criterion as already proposed by Pipino et al. [39]. In the updated version, by means of the criterion Column completeness one can determine the degree by which the instances of a class use the same relations, averaged over all classes.

Formally, we define the Column completeness metric m cCol ( g ) as the ratio of the number of instances having class k and a value for the relation r, no k p , to the number of all instances having class k, no k . By averaging over all class-relation-pairs which occur on instance level, we obtain a fulfillment degree regarding the whole KG: m cCol ( g ) = 1 | H | ∑ ( k , p ) ∈ H no k p no k

We thereby let H = { ( k , p ) ∈ ( K × P ) ∣ ∃ k ∈ C g ∧ ∃ ( x , p , o ) ∣ p ∈ P g imp ∧ ( x , rdf:type , k ) } be the set of all combinations of the considered classes, K = { k 1 , … , k n } , and considered relations, P = { p 1 , … , p m } .

Note that there are also relations which are dedicated to the instances of a specific class, but which do not need to exist for all instances of that class. For instance, not all people need to have a relation :hasChild or :deathDate.21

Population completeness The Population completeness metric determines the extent to which the considered KG covers a basic population [39]. The assessment of the KG completeness w.r.t. a basic population is performed by means of a gold standard, which covers both well-known entities (called “short head”, e.g., the n largest cities in the world according to the number of inhabitants) and little-known entities (called “long tail”; e.g., municipalities in Germany). We take all entities contained in our gold standard equally into account.

Let G S be the set of entities in the gold standard. Then we can define: m cPop ( g ) = | { e | e ∈ G S ∧ e ∈ E g } | | { e | e ∈ G S } |

Definition of dimension. Timeliness is “the extent to which the age of the data is appropriate for the task at hand” [45].

Discussion.Timeliness does not describe the creation date of a statement, but instead the time range since the last update or the last verification of the statement [37]. Due to the easy way of publishing data on the Web, data sources can be kept easier up-to-date than traditional isolated data sources. This results in advantages to the consumer of Web data [37]. How Timeliness is measured depends on the application context: For some situations years are sufficient, while in other situations one may need days [37].

Definition of metric. The dimension timeliness is determined by the criteria Timeliness frequency of the KG, Specification of the validity period, and Specification of the modification date of statements.

The fulfillment degree of a KG g w.r.t. the dimension Timeliness is measured by the metrics m Freq , m Validity , and m Change , which are defined as follows.

Timeliness frequency of the KG The criterion Timeliness frequency of the KG indicates how fast the KG is updated. We consider the KG RDF export here and differentiate between continuous updates, where the updates are always performed immediately, and discrete KG updates, where the updates take place in discrete time intervals. In case the KG edits are available online immediately but the RDF export files are available in discrete, varying updating intervals, we consider the online version of the KG, since in the context of Linked Data it is sufficient that URIs are dereferenceable. m Freq ( g ) = 1 continuous updates 0.5 discrete periodic updates 0.25 discrete non-periodic updates 0 otherwise

Specification of the validity period of statements Specifying the validity period of statements enables to temporally limit the validity of statements. By using this criterion, we measure whether the KG supports the specification of starting and maybe end dates of statements by means of providing suitable forms of representation. m Validity ( g ) = 1 specification of validity period supported 0 otherwise

Specification of the modification date of statements The modification date discloses the point in time of the last verification of a statement. The modification date is typically represented via the relations schema:dateModified and dcterms:modified. m Change ( g ) = 1 specification of modification dates for statements supported 0 otherwise

3.4.1. Ease of understanding

Definition of dimension. The ease of understanding is “the extent to which data are clear without ambiguity and easily comprehended” [45].

Discussion. This dimension focuses on the understandability of a data source by a human data consumer. In contrast, the dimension Interoperability focuses on technical aspects. The understandability of a data source (here: KG) can be improved by things such as descriptive labels and literals in multiple languages.

Definition of metric. The dimension understandability is determined by the criteria Description of resources, Labels in multiple languages, Understandable RDF serialization, and Self-describing URIs. The fulfillment degree of a KG g w.r.t. the dimension Consistency is measured by the metrics m Descr , m Lang , m uSer , and m uURI , which are defined as follows.

Description of resources Heath et al. [24 ,28] suggest to describe resources in a human-understandable way, e.g., via rdfs:label or rdfs:comment. Within our framework, the criterion is measured as follows: Given a sample of resources, we divide the number of resources in the KG for which at least one label or one description is provided, (e.g., via rdfs:label, rdfs:comment, or schema:description) by the number of all considered resources in the local namespace: m Descr ( g ) = | { u | u ∈ U g local ∧ ∃ ( u , p , o ) ∈ g : p ∈ P lDesc } | / | { u | u ∈ U g local } | P lDesc is the set of implicitly used relations in g indicating that the value is a label or description (e.g., P lDesc = { rdfs:label , rdfs:comment } ).

Beschreibung). Darüber hinaus ist das Ergebnis der Evaluation auf Basis der Entitäten interessant -> DBpedia weicht deutlich ab, da manche Entitäten (Intermediate-Node-Mapping) keine rdfs:label haben. Folglich würde ich die Definition der Metrik allgemein halten (beschränkt auf proprietäre Ressourcen, d.h. im selben Namespace), die Evaluation jedoch nur anhand der Entitäten machen.

Labels in multiple languages Resources in the KG are described in a human-readable way via labels, e.g., via rdfs:label or skos:prefLabel.22

Understandable RDF serialization RDF/XML is the recommended RDF serialization format of the W3C. However, due to its syntax RDF/XML documents are hard to read for humans. The understandability of RDF data by humans can be increased by providing RDF in other, more human-understandable serialization formats such as N3, N-Triple, and Turtle. We measure this criterion by measuring the supported serialization formats during the dereferencing of resources. m uSer ( h g ) = 1 Other RDF serializations than RDF/XML available 0 otherwise

Note that conversions from one RDF serialization format into another are easy to perform.

Self-describing URIs Descriptive URIs contribute to a better human-readability of KG data. Sauermann et al.23

Interoperability is another dimension of the representational data quality category and subsumes Wang et al.’s aspects interpretability, representational consistency, and concise representation.

Definition of dimension. We define Interoperability along the subsumed dimensions of Wang et al.:

Discussion regarding interpretability. In contrast to the dimension understandability, which focuses on the understandability of RDF KG data towards the user as data consumer, interpretability focuses on the representation forms of information in the KG from a technical perspective. An example is the consideration whether blank nodes are used. According to Heath et al. [24], blank nodes should be avoided in the Linked Data context, since they complicate the integration of multiple data sources and since they cannot be linked by resources of other data sources.

Discussion regarding representational consistency. In the context of Linked Data, it is best practice to reuse existing vocabulary for the creation of own RDF data. In this way, less data needs to be prepared for being published as Linked Data [24].

Discussion regarding concise representation. Heath et al. [24] made the observation that the RDF features (i) RDF reification,25

Definition of metric. The dimension Interoperability is determined via the following criteria:

The fulfillment degree of a KG g w.r.t. the dimension Interoperability is measured by the metrics m Reif , m iSerial , m exVoc , and m propVoc , which are defined as follows.

Avoiding blank nodes and RDF reification Using RDF blank nodes, RDF reification, RDF container, and RDF lists is often considered as ambivalent: On the one hand, these RDF features are not very common and they complicate the processing and querying of RDF data [24,28]. On the other hand, they are necessary in certain situations, e.g., when statements about statements should be made. We measure the criterion by evaluating whether blank nodes and RDF reification are used. m Reif ( g ) = 1 no blank nodes and no RDF reification 0.5 either blank nodes or RDF reification 0 otherwise

Provisioning of several serialization formats The interpretability of RDF data of a KG is increased if besides the serialization standard RDF/XML further serialization formats are supported for URI dereferencing. m iSerial ( h g ) = 1 RDF/XML and further formats are supported 0.5 only RDF/XML is supported 0 otherwise

Using external vocabulary Using a common vocabulary for representing and describing the KG data allows to represent resources and relations between resources in the Web of Data in a unified way. This increases the interoperability of data [24,28] and allows a comfortable data integration. We measure the criterion of using an external vocabulary by setting the number of triples with external vocabulary in predicate position to the number of all triples in the KG: m extVoc ( g ) = | { ( s , p , o ) | ( s , p , o ) ∈ g ∧ p ∈ P g external } | | { ( s , p , o ) ∈ g } |

Interoperability of proprietary vocabulary Linking on schema level means to link the proprietary vocabulary to external vocabulary. Proprietary vocabulary are classes and relations which were defined in the KG itself. The interlinking to external vocabulary guarantees a high degree of interoperability [24]. We measure the interlinking on schema level by calculating the ratio to which classes and relations have at least one equivalency link (e.g., owl:sameAs, owl:equivalentProperty, or owl:equivalentClass) to classes and relations, respectively, of other data sources. m propVoc ( g ) = | { x ∈ P g ∪ C g | ∃ ( x , p , o ) ∈ g : ( p ∈ P eq ∧ ( o ∈ U ∧ o ∈ U g ext ) ) } | / | P g ∪ C g | where P eq = { owl:sameAs , owl:equivalentProperty , owl:equivalenClass } and U g ext consists of all URIs in U g which are external to the KG g which means that h g is not responsible for resolving these URIs.

3.5.1. Accessibility

Definition of dimension. Accessibility is “the extent to which data are available or easily and quickly retrievable” [45].

Discussion. Wang et al.’s definition of Accessibility contains the aspects availability, response time, and data request. They are defined as follows:

Definition of metric. We define the metric for the dimension Accessibility by means of metrics for the following criteria:

The fulfillment degree of a KG g w.r.t. the dimension Accessibility is measured by the metrics m Deref , m Avai , m SPARQL , m Export , m Negot , m HTMLRDF , and m Meta , which are defined as follows.

Dereferencing possibility of resources One of the Linked Data principles [7] is the dereferencing possibility of resources: URIs must be resolvable via HTTP requests and useful information should be returned thereby. We assess the dereferencing possibility of resources in the KG by analyzing for each URI in the sample set (here: all URIs U g ) the HTTP response status code and by evaluating whether RDF data is returned. A successful dereferencing of resources is given if HTTP status code 200 and an RDF document is returned. m Deref ( h g ) = | dereferencable ( U g ) | | U g |

Availability of the KG The Availability of the KG criterion indicates the uptime of the KG. It is an essential criterion in the context of Linked Data, since in case of an integrated or federated query mostly all data sources need to be available [37]. We measure the availability of a KG by monitoring the ability of dereferencing URIs over a period of time. This monitoring process can be done with the help of a monitoring tool such as Pingdom.26

Provisioning of public SPARQL endpoint SPARQL endpoints allow the user to perform complex queries (including potentially many instances, classes, and relations) on the KG. This criterion here indicates whether an official SPARQL endpoint is publicly available. There might be additional restrictions of this SPARQL endpoint such as a maximum number of requests per time slice or a maximum runtime of a query. However, we do not measure these restrictions here. m SPARQL ( h g ) = 1 SPARQL endpoint publicly available 0 otherwise

Provisioning of an RDF export If there is no public SPARQL endpoint available or the restrictions of this endpoint are so strict that the user does not use it, an RDF export dataset (RDF dump) can often be used. This dataset can be used to set up a local, private SPARQL endpoint. The criterion here indicates whether an RDF export dataset is officially available: m Export ( h g ) = 1 RDF export available 0 otherwise

Support of content negotiation Content negotiation (CN) allows that the server returns RDF documents during the dereferencing of resources in the desired RDF serialization format. The HTTP protocol allows the client to specify the desired content type (e.g., RDF/XML) in the HTTP request and the server to specify the returned content type in the HTTP response header (e.g., application/rdf+xml). In this way, the desired and the provided content type are matched as far as possible. It can happen that the server does not provide the desired content type. Moreover, it may happen that the server returns an incorrect content type. This may lead to the fact that serialized RDF data is not processed further. An example is RDF data which is declared as text/plain [24]. Hogan et al. [27] therefore propose to let KGs return the most specific content type as possible. We measure the Support of content negotiation by dereferencing resources with different RDF serialization formats as desired content type and by comparing the accept header of the HTTP request with the content type of the HTTP response. m Negot ( h g ) = 1 CN supported and correct content types returned 0.5 CN supported but wrong content types returned 0 otherwise

Linking HTML sites to RDF serializations Heath et al. [24] suggest linking any HTML description of a resource to RDF serializations of this resource in order to make the discovery of corresponding RDF data easier (for Linked Data aware applications). For that reason, in the HTML header the so-called Autodiscovery pattern can be included. This pattern consists of the phrase link rel=alternate, the indication about the provided RDF content type, and a link to the RDF document.27

Provisioning of KG metadata In the light of the Semantic Web vision where agents select and make use of appropriate data sources on the Web, also the meta-information about KGs needs to be available in a machine-readable format. The two important mechanisms to specify metadata about KGs are (i) using semantic sitemaps and (ii) using the VoID vocabulary28

Definition of dimension. Licensing is defined as “the granting of permission for a consumer to re-use a dataset under defined conditions” [47].

Discussion. The publication of licensing information about KGs is important for using KGs without legal concerns, especially in commercial settings. Creative Commons (CC)29

Noteworthy is that most data sources in the Linked Open Data cloud do not provide any licensing information [29] which makes it difficult to use the data in commercial settings. Even if data is published under CC-BY or CC-BY-SA, the data is often not used since companies refer to uncertainties regarding these contracts.

Definition of metric. The dimension License is determined by the criterion Provisioning machine-readable licensing information.

The fulfillment degree of a KG g w.r.t. the dimension License is measured by the metric m macLicense , which is defined as follows.

Provisioning machine-readable licensing information Licenses define the legal frameworks under which the KG data may be used. Providing machine-readable licensing information allows users and applications to be aware of the license and to use the data of the KG in accordance with the legal possibilities [24,28].

Licenses can be specified in RDF via relations such as cc:licence,33

Definition of dimension. Interlinking is the extent “to which entities that represent the same concept are linked to each other, be it within or between two or more data sources” [47].

Discussion. According to Bizer et al. [10], DBpedia established itself as a hub in the Linked Data cloud due to its intensive interlinking with other KGs. These interlinking is on the instance level usually established via owl:sameAs links. However, according to Halpin et al. [22], those owl:sameAs links do not always interlink identical entities in reality. According to the authors, one reason might be that the KGs provide entries in different granularity: For instance, the DBpedia resource for “Berlin” (dbo:Berlin) links via owl:sameAs relations to three different resources in the KG GeoNames,34

Definition of metric. The dimension Interlinking is determined by the criteria

The fulfillment degree of a KG g w.r.t. the dimension Interlinking is measured by the metrics m Inst and m URIs , which are defined as follows.

Interlinking via owl:sameAs The forth Linked Data principle according to Berners-Lee [7] is the interlinking of data resources so that the user can explore further information. According to Hogan et al. [28], the interlinking has a side effect: It does not only result in otherwise isolated KGs, but the number of incoming links of a KG indicates the importance of the KG in the Linked Open Data cloud. We measure the interlinking on instance level38

Validity of external URIs The considered KG may contain outgoing links referring to RDF resources or Web documents (non-RDF data). The linking to RDF resources is usually enabled by owl:sameAs, owl:equivalentProperty, and owl:equivalentClass relations. Web documents are linked via relations such as foaf:homepage and foaf:depiction. Linking to external resources always entails the problem that those links might get invalid over time. This can have different causes. For instance, the URIs are not available anymore. We measure the Validity of external URIs by evaluating the URIs from an URI sample set w.r.t. whether there is a timeout, a client error (HTTP response 4xx) or a server error (HTTP response 5xx). m URIs ( g ) = | { x ∈ A ∣ resolvable ( x ) } | | A | where A = { y ∣ ∃ ( x , p , y ) ∈ g : ( p ∈ P eq ∧ x ∈ U g ∖ ( C g ∪ P g ) ∧ x ∈ U g local ∧ y ∈ U g ext ) } and resolvable ( x ) returns true if HTTP status code 200 is returned. P eq is the set of relations used for linking to external sources. Examples for such relations are owl:sameAs and foaf:homepage.

In case of an empty set A, the metric should evaluate to 1.

5.1.1. Triples

Ranking of KGs w.r.t. number of triples. The number of triples (see Table 2) differs considerably between the KGs: Freebase is the largest KG with over 3.1B triples, while OpenCyc resides the smallest KG with only 2.4M triples. The large size of Freebase can be traced back to the fact that large data sets such as MusicBrainz have been integrated into this KG. OpenCyc, in contrast, has been built purely manually by experts. In general, this indicates a correlation between the way of building up a KG and its size.

Size differences between DBpedia and YAGO. As both DBpedia and YAGO were created automatically by extracting semantically-structured information from Wikipedia, the significant difference between their sizes – in terms of triples – is in particular noteworthy. We can mention here the following reasons: YAGO integrates the statements from different language versions of Wikipedia in one single KG while for the canonical DBpedia dataset (which is used in our evaluations) solely the English Wikipedia was used as information source. Besides that, YAGO contains contextual information and detailed provenance information. Contextual information is for instance the anchor texts of all links within Wikipedia. For representing the anchor texts, the relation yago:hasWikipediaAnchorText (330M triples in total) is used. The provenance information of single statements is stored in a reified form. In particular, the relations yago:extractionSource (161.2M triples) and yago:extractionTechnique (176.2M triples) are applied therefore.

Influence of reification on the number of triples. DBpedia, Freebase, Wikidata, and YAGO use some form of reification. Reification in general describes the possibility of making statements about statements. While reification has an influence on the number of triples for DBpedia, Freebase, and Wikidata, the number of triples in YAGO is not influenced by reification since data is here provided in N-Quads.62

In case of DBpedia, Freebase, and Wikidata, reification is implemented by means of n-ary relations. An n-ary relation denotes the relation between more than two resources and is implemented via additional, intermediate nodes, since in RDF only binary statements can be modeled [15,25]. In Freebase and DBpedia, data is mostly provided in the form of plain N-Triples and n-ary relations are only used for data from higher arity.63

Methods for counting classes. The number of classes can be calculated in different ways: Classes can be identified via rdfs:Class and owl:Class relations, or via rdfs:subClassOf relations.64

Ranking of KG w.r.t. number of classes. Our evaluations revealed that YAGO contains the highest number of classes of all considered KGs; DBpedia, in contrast, has the fewest (see Table 2).

Number of classes in YAGO and DBpedia. How does it come to this gap between DBpedia and YAGO with respect to the number of classes, although both KGs were created automatically based on Wikipedia? For YAGO, the classes are extracted from the categories in Wikipedia, while the hierarchy of the classes is deployed with the help of WordNet synset relations. The DBpedia ontology, in contrast, is very small, since it is created manually, based on the mostly used infobox templates in Wikipedia. Besides those 736 classes, the DBpedia KG contains further 444,895 classes which originate from the imported YAGO classes and which are published in the namespace yago:. Those YAGO classes are – like the DBpedia ontology classes – interconnected via rdfs:subClassOf to form a taxonomy. In the evaluation of DBpedia, the YAGO classes are ignored, as they do not belong to the DBpedia ontology given as OWL file.

OPEN IN VIEWER

Fig. 1.

Coverage of classes having at least one instance.

Coverage of classes with at least one instance. Figure 1 shows for each KG the extent to which classes are instantiated, that is, for how many classes at least one instance exists. YAGO exhibits the highest coverage rate (82.6%), although it contains the highest number of classes among the KGs. This can be traced back to the fact that YAGO classes are chosen by a heuristic that considers Wikipedia leaf categories which tend to have instances [42]. OpenCyc (with 6.5%) and Wikidata (5.4%) come last in the ranking. Wikidata has the second highest number of classes in total (see Table 2), out of which relatively little are used on instance level. Note, however, that in some scenarios solely the schema level information (including classes) of KGs is necessary, so that the low coverage of instances by classes is not necessarily an issue.

OPEN IN VIEWER

Fig. 2.

Distribution of classes w.r.t. the number of instances per KG.

Correlation between number of classes and number of instances. In Fig. 2, we can see a histogram of the classes with respect to the number of instances per class. That is, for each KG we can spot how many classes have a high number of instances and how many classes have a low number of instances. Note the logarithmic scale on both axes. The curves seem to follow power law distributions. For DBpedia, the line decreases consistently for the first 250 classes, before it decreases more than exponentially beyond class 250.

All considered KGs are cross-domain, meaning that a variety of domains are covered in those KGs. However, the KGs often cover the single domains to a different degree. Tartir [43] proposed to measure the covered domains of ontologies by determining the usage degree of corresponding classes: the number of instances belonging to one or more subclasses of the respective domain is compared to the number of all instances. In our work, however, we decided to evaluate the coverage of domains concerning the classes per KG via manual assignments of the mostly used classes to the domains people, media, organizations, geography, and biology.65

The manual assignment of classes to domains is necessary in order to obtain a consistent assignment of the classes to the domains across all considered KGs. Otherwise, the same classes in different KGs may be assigned to different domains. Moreover, in some KGs classes may otherwise appear in various domains simultaneously. For instance, the Freebase classes freebase:music.artist and freebase:people.person overlap in terms of their instances and multiple domains (such as music and people) might be assigned to them.

As the reader can see in Table 1, our method to determine the coverage of domains, and, hence, the reach of our evaluation, includes about 80% of all entities of each KG, except Wikidata. It is calculated as the ratio of the number of unique entities of all considered domains of a given KG divided by the number of all entities of this KG.66

OPEN IN VIEWER

Fig. 3.

Number of entities per domain.

OPEN IN VIEWER

Fig. 4.

Relative number of entities per domain.

Figure 3 shows the number of entities per domain in the different KGs with a logarithmic scale. Figure 4 presents the relative coverage of each domain in each KG. It is calculated as the ratio of the number of entities in each domain to the total number of entities of the KG. A value of 100% means that all instances reside in one single domain.

The case of Freebase is especially outstanding here: 77% of all entities here are located in the media domain. This fact can be traced back to large-scale data imports, such as from MusicBrainz. The class freebase:music.release_track is accountable for 42% of the media entities. As shown in Fig. 3, Freebase provides the most entities in four out of the five domains when considering all KGs.

In DBpedia and YAGO, the domain of people is the largest domain (50% and 34%, respectively). Peculiar is the higher coverage of YAGO regarding the geography domain compared to DBpedia. As one reason for that we can point out the data import of GeoNames into YAGO.

Wikidata contains around 150K entities in the domain organization. This is relatively few considering the total amount of entities being around 18.7M and considering the number of organizations in other KGs. Note that even DBpedia provides more organization entities than Wikidata. The reason why Wikidata has not so many organization entities is not fully comprehensible to us. However, we can point out that for our analysis we only considered Wikidata classes which appeared more than 6,000 times67

Evaluation method. In this article, we differentiate between relations and predicates (see also Section 2):

Relations – as short term for explicitly defined relations – refers to (proprietary) vocabulary defined on the schema level of a KG. We identify the set of relations of a KG as the set of those links which are explicitly defined as such via assignments (for instance, with rdfs:Property) to classes. In Section 2 we used P g to denote this set.

Evaluation results.

Relations

Ranking regarding relations. As presented in Table 2, Freebase exhibits by far the highest number of unique relations (around 785K) among the KGs. YAGO shows only 106 relations, which is the lowest value in this comparison. In the following, we point out further findings regarding the relations of the single KGs.

DBpedia Regarding DBpedia relations we need to distinguish between so-called mapping-based properties and non-mapping-based properties. Mapping-based properties are created by extracing the information from infoboxes in Wikipedia using manually created mappings. These mappings are specified in the DBpedia Mappings Wiki.68

Freebase The high number or Freebase relations can be explained by two facts: 1. About a third of all relations in Freebase are duplicates in the sense that they are declared by means of the owl:inverseOf relation as being inverse of other relations. An example is the relation freebase:music.artist.album and its inverse relation freebase:music.album.artist. 2. Freebase allowed users to introduce their own relations without any limits. These relations were originally in each user’s namespace. So-called commons admins were able to approve those relations so that they got included into the Freebase commons schema.

OpenCyc For OpenCyc we measure 18,028 unique relations. We can assume that most of them are dedicated to statements on the schema level.

Wikidata In Wikidata a relatively small set of relations is provided. Note in this context that, despite the fact that Wikidata is curated by a community (just like Freebase), Wikidata community members cannot insert arbitrarily new relations as it was possible in Freebase; instead, relations first need to be proposed and then get accepted by the community if and only if certain criteria are met.71

YAGO For YAGO we measure the small set of 106 unique relations. Although relations are curated manually for YAGO and DBpedia, the size of the relation set differs significantly between those KGs. Hoffart et al. [26] mention the following reasons for that:

OPEN IN VIEWER

Fig. 5.

Frequency of the usage of the relations per KG, grouped by (i) zero occurrences, (ii) 1–500 occurrences, and (iii) more than 500 occurrences in the respective KG.

Frequency of the usage of relations. Figure 5 shows the relative proportions of how often relations are used per KG, grouped into three classes. Surprisingly, DBpedia and Freebase exhibit a high number of relations which are not used at all on the instance level. In case of OpenCyc, 99.2% of the defined relations are never used. We assume that those relations are used only within Cyc, the commercial version of OpenCyc. In case of Freebase, only 5% of the relations are used more than 500 times and about 70% are not used at all. Analogously to the discussion regarding the number of Freebase relations, we can mention again the high number of defined owl:inverseOf relations and the high number of users’ relation proposals as reasons for that.

Predicates Ranking regarding predicates. Freebase is here – like in case of the ranking regarding relations – ranked first. The lowest number of unique predictes is provided by OpenCyc, which exhibits only 165 predicates. All KGs except OpenCyc provide more predicates then relations. Our single observations regarding the predicate sets are as follows:

DBpedia DBpedia is ranked third in terms of the absolute numbers of predicates: about 60K predicates are used in DBpedia. The set of relations and the set of predicates varies considerably here, since also facts are extracted from Wikipedia info-boxes whose predicates are considered by us as being only implicitly defined and which, hence, occur only as predicates. These are the so-called non-mapping-based properties. Note that in the studied DBpedia version 2015-04 the set of explicitly defined relations (mapping-based properties) and the set of implicitly defined relations (non-mapping-based properties) overlaps. An example is dbp:alias with dbo:alias.

Freebase We can observe here a similar picture as for the set of Freebase relations: With about 785K unique predicates, Freebase exceeds the other KGs by far. Note, however, that 95% of the predicates (around 743K) are used only once. This relativizes the high number. Most of the predicates are keys in the sense of ids and are used for internal modeling (for instance, freebase:key.user.adrianb).

OpenCyc In contrast to the 18,028 unique relations, we measure only 164 unique predicates for OpenCyc. More predicates are presumably used in Cyc.

Wikidata We measure more Wikidata predicates than Wikidata relations, since Wikidata predicates are created by modifying Wikidata relations. An example are the following triples, which express the statement “Barack Obama (wdt:Q76) is a human (wdt:Q5)” by an intermediate node (wdt:Q76S123, abbreviated):

The relation extension “s” indicates that the RDF term in the object position is a statement. The “v” extension allows to refer to a value (in Wikidata terminology). Besides those extensions, there is “r” to refer to a reference and the “q” extension to refer to a qualifier. In general, these relation extensions are used for realizing reification via n-ary relations. For that, intermediate nodes are used which represent statements [15].

YAGO YAGO contains more predicates than DBpedia, since infobox attributes from different language versions of Wikipedia are aggregated into one KG,72

Evaluation method. We distinguish between instances I g and entities E g of a KG (cf. Section 2).

OPEN IN VIEWER

Fig. 6.

Number of instances per KG.

Ranking w.r.t. the number of instances. Table 2 and Fig. 6 show the number of instances per KG. We can see that Wikidata comprises the highest number of instances (142M) in total and OpenCyc the fewest (242K).

Ranking w.r.t. the number of entities. Table 2 shows the ranking of KGs regarding the number of entities. Freebase contains by far the highest number of entities (about 49.9M). OpenCyc is at the bottom with only about 41K entities.

Differences in number of entities. The reason why the KGs show quite varying numbers of entities are the information sources of the KGs. We illustrate this with the music domain as example:

OPEN IN VIEWER

Fig. 7.

Average number of entities per class per KG.

Average number of entities per class. Figure 7 shows the average number of entities per class, which can be written as | E g | / | C g | . Obvious is the difference between DBpedia and YAGO (despite the similar number of entities): The reason for that is that the number of classes in the DBpedia ontology is small (as created manually) and in YAGO large (as created automatically).

Comparing number of instances with number of entities. Comparing the ratio of the number of instances to the number of entities for each KG, Wikidata exposes the highest difference. As reason for that we can state that each statement in Wikidata is modeled as an instance of wdo:Statement, leading to 74M additional instances. In other KGs such as DBpedia, statements are modeled without any dedicated statement assignment. OpenCyc exposes also a high ratio, since it contains mainly common sense knowledge and not as many entities as the other KGs. Furthermore, for our analysis we do not regard 100% of the entities, but only a large fraction of it (more precisely, the classes with the most frequently occurring instantiations), since entities are not consistently instantiated in OpenCyc (see beginning of Section 5.1.5).

Evaluation method. The number of unique subjects and unique objects can be a meaningful KG characteristic regarding the link structure within the KG and in comparison to other KGs. Especially interesting are differences between the number of unique subjects and the number of unique objects.

We measure the number of unique subjects by counting the unique resources (i.e., URIs and blank nodes) on the subject position of N-Triples: S g : = { s ∣ ( s , p , o ) ∈ g } . Furthermore, we measure the number of unique objects by counting the unique resources on the object position of N-Triples, excluding literals: O g : = { o ∣ ( s , p , o ) ∈ g ∧ o ∈ U ∪ B } . Complementary, the number of literals is given as: O g lit : = { o ∣ ( s , p , o ) ∈ g ∧ o ∈ L } .

Ranking of KGs regarding number of unique subjects. The number of unique subjects per KG is presented in Fig. 8. YAGO contains the highest number of different subjects, while OpenCyc contains the fewest.

OPEN IN VIEWER

Fig. 8.

Number of unique subjects and objects per KG. Note the logarithmic scale on the axis of ordinates.

OPEN IN VIEWER

Fig. 9.

Ratio of the number of instances to the number of entities for each KG.

Ranking of KGs regarding number of unique objects. The number of unique objects is also presented in Fig. 8. Freebase shows the highest score in this regard, OpenCyc again the lowest.

Ranking of KGs regarding the ratio of number of unique subjects to number of unique objects. The ratios of the number of unique subjects to the number of unique objects vary considerably between the KGs (see Fig. 9). We can observe that DBpedia has 2.65 times more objects than subjects, while YAGO on the other side has 19 times more unique subjects than objects.

The high number of unique subjects in YAGO is surprising and can be explained by the reification style used in YAGO. Facts are stored as N-Quads in order to allow for making statements about statements (for instance, storing the provenance information for statements). To that end, IDs (instead of blank nodes) which identify the triples are used on the first position of N-Triples. They lead to 308M unique subjects, such as yago:id_6jg5ow_115_lm6jdp. In the RDF export of YAGO, the IDs which identify the triples are commented out in order to facilitate the N-Triple format. However, the statements about statements are also transformed to triples. In those cases, the IDs identifying the reified statements are in the subject position, leading to such a high number of unique subjects.

DBpedia contains considerably more owl:sameAs links to external resources than KGs like YAGO (29.0M vs. 3.8M links), leading to a bias of DBpedia towards a high number of unique objects.

Based on the evaluation results presented in the last subsections, we can highlight the following insights:

Triples: All KGs are very large. Freebase is the largest KG in terms of number of triples, while OpenCyc is the smallest KG. We notice a correlation between the way of building up a KG and the size of the KG: automatically created KGs are typically larger, as the burdens of integrating new knowledge become lower. Datasets which have been imported into the KGs, such as MusicBrainz into Freebase, have a huge impact on the number of triples and on the number of facts in the KG. Also the way of modeling data has a great impact on the number of triples. For instance, if n-ary relations are expressed in N-Triples format (as in case of Wikidata), many intermediate nodes need to be modeled, leading to many additional triples compared to plain statements. Last but not least, the number of supported languages influences the number of triples.

5.2.1. Accuracy

The fulfillment degrees of the KGs regarding the Accuracy metrics are shown in Table 3.

Syntactic validity of RDF documents, m synRDF  Evaluation method. For evaluating the Syntactic validity of RDF documents, we dereference the entity “Hamburg” as resource sample in each KG. In case of DBpedia, YAGO, Wikidata, and OpenCyc, there are RDF/XML serializations of the resource available, which can be validated by the official W3C RDF validator.76

Evaluation result. All considered KGs provide syntactically valid RDF documents. In case of YAGO and Wikidata, the RDF validator declares the used language codes as invalid, since the validator evaluates language codes in accordance with ISO-639. The criticized language codes are, however, contained in the newer standard ISO 639-3 and actually valid.

Syntactic validity of literals, m synLit  Evaluation method. We evaluate the Syntactic validity of literals by means of the relations date of birth, number of inhabitants, and International Standard Book Number (ISBN), as those relations cover different domains – namely, people, cities, and books – and as they can be found in all KGs. In general, domain knowledge is needed for selecting representative relations, so that a meaningful coverage is guaranteed.

Note that OpenCyc is not taken into account for this criterion: Although OpenCyc comprises around 1.1M literals in total, these literals are essentially labels and descriptions (given via rdfs:label and rdfs:comment), i.e., not aligned to specific data types. Hence, OpenCyc has no syntactic invalid literals and is assigned the metric value 1.

As long as a literal with data type is given, its syntax is verified with the help of the function RDFDatatype.isValid(String) of the Apache Jena framework. Thereby, standard data types such as xsd:date can be validated easily, especially if different data types are provided.78

Evaluation results. All KGs except YAGO performed very well regarding the Syntactic validity of literals.

Date of birth For Wikidata, DBpedia, and Freebase, all verified literal values (1M per KG) were syntactically correct.79

Number of inhabitants The data types of the literal values regarding the number of inhabitants were valid in all KGs. For DBpedia, YAGO, and Wikidata, we evaluated the syntactic validity of the number of inhabitants by checking if xsd:nonNegativeInteger, xsd:decimal, and xsd:integer were used as data types for the typed literals. In Freebase, no data type is specified. Therefore, we evaluated the values by means of a regular expression which allows only the decimals 0–9, periods, and commas.

ISBN The ISBN is an identifier for books and magazines. The identifier can occur in various formats: with or without preceding “ISBN,” with or without delimiters, and with 10 or 13 digits. Gupta81

Semantic validity of triples, m semTriple  Evaluation method. The semantic validity can be reliably measured by means of a reference data set which (i) contains at least to some degree the same facts as in the KG and (ii) which is regarded as some kind of authority. We decided to use the Integrated Authority File (Gemeinsame Normdatei, GND),85

Evaluation result. We evaluated up to 400 facts per KG and observed only for a few facts some discrepancies. For instance, Wikidata states as death date of “Anton Erkelenz” (wdt:Q589196) April 24, whereas GND states April 25. For DBpedia and YAGO we encountered 3 and for Wikidata 4 errors. Hence, those KGs were evaluated with 0.99. Note that OpenCyc has no values for the chosen relations and thus evaluates to 1.

During evaluation we identified the following issues:

In conclusion, the evaluation of semantic validity is hard, even if a random sample set is evaluated manually. Meaningful differences among the KGs might be revealed only when a very large sample is evaluated, e.g., by using crowd-sourcing [2,3,46]. Another approach for assessing the semantic validity is presented by Kontokostas et al. [32] who propose a test-driven evaluation where test cases are created to evaluate triples semi-automatically: For instance, an interval specifies the valid height of a person and all triples which lie outside of this interval are evaluated manually. In this way, outliers can be easily found but possible wrong values within the interval are not detected.

Our findings appear to be consistent with the evaluation results of the YAGO developer team for YAGO2, where manually assessing 4,412 statements resulted in an accuracy of 98.1%.86

The fulfillment degrees of the KGs regarding the Trustworthiness criteria are shown in Table 4.

Trustworthiness on KG level, m graph  

Evaluation method. Regarding the trustworthiness of a KG in general, we differentiate between the method of how new data is inserted into the KG and the method of how existing data is curated.

Evaluation results. The KGs differ considerably w.r.t. this metric. OpenCyc obtains the highest score here, followed by Wikidata. In the following, we provide findings for the single KGs, which are listed by decreasing fulfillment score:

Cyc is edited (expanded and modified) exclusively by a dedicated expert group. The free version, OpenCyc, is derived from Cyc and only a locally hosted version can be modified by the data consumer.

Wikidata is also curated and expanded manually, but by volunteers of the Wikidata community. Wikidata allows importing data from external sources such as Freebase.87

Freebase was also curated by a community of volunteers. In contrast to Wikidata, the proportion of data imported automatically is considerably higher and new data imports were not dependent on community approvals.

DBpedia and YAGO The knowledge of both KGs is extracted from Wikipedia, but DBpedia differs from YAGO w.r.t. the community involvement: Any user can engage (i) in mapping the Wikipedia infobox templates to the DBpedia ontology in the DBpedia mappings wiki88

Trustworthiness on statement level We determine the Trustworthiness on statement level by evaluating whether provenance information for statements is used in the KGs. The picture is mixed:

DBpedia uses the relation prov:wasDerivedFrom to store the sources of the entities and their statements. However, as the source is always the corresponding Wikipedia article,89

YAGO uses its own vocabulary to indicate the source of information. Interestingly, YAGO stores per statement both the source (via yago:extractionSource; e.g., the Wikipedia article) and the used extraction technique (via yago:extractionTechnique; e.g., “Infobox Extractor” or “CategoryMapper”). The number of statements about sources is 161M, and, hence, many times over the number of instances in the KG. The reason for that is that in YAGO the source is stored for each fact.

In Wikidata several relations can be used for referring to sources, such as “imported from” (wdt:P143), “stated in” (wdt:P248), and “reference URL” (wdt:P854).90

Freebase uses proprietary vocabulary for representing provenance: via n-ary relations, which are in Freebase called Compound Value Types (CVT), data from higher arity can be expressed [38].94

OpenCyc differs from the other KGs in that it uses neither an external vocabulary nor a proprietary vocabulary for storing provenance information.

Indicating unknown and empty values, m NoVal  This criterion highlights the subtle data model of Wikidata and Freebase in comparison to the data models of the other KGs: Wikidata allows for storing unknown values and empty values (e.g., that “Elizabeth I of England” (wdt:Q7207) had no children). However, in the Wikidata RDF export such statements are only indirectly available, since they are represented via blank nodes and via the relation owl:someValuesFrom.

YAGO supports the representation of unknown values and empty values by providing explicit relations for such cases.95

The fulfillment degrees of the KGs regarding the Consistency criteria are shown in Table 5.

Check of schema restrictions during insertion of new statements, m checkRestr  

The values of the metric m checkRestr , indicating restrictions during the insertion of new statements, are varying among the KGs. The web interfaces of Freebase and Wikidata verify during the insertion of new statements by the user whether the input is compatible with the respective data type. For instance, data of the relation “date of birth” (wdt:P569) is expected to be in a syntactically valid form. DBpedia, OpenCyc and YAGO have no checks for schema restriction during the insertion of new statements.

Consistency of statements w.r.t. class constraints, m conClass  Evaluation method. For evaluating the consistency of class constraints we considered the relation owl:disjointWith, since this is the only relation which is used by more than half of the considered KGs. We only focused on direct instantiations here: if there is, for instance, the triple (dbo:Plant, owl:disjointWith, dbo:Animal), then there must not be a resource which is instantiated both as dbo:Plant and dbo:Animal.

Evaluation results. We obtained mixed results here. Only Freebase, OpenCyc, and Wikidata perform very well.96

Freebase and Wikidata do not specify any constraints with owl:disjointWith. Hence, those two KGs have no inconsistencies w.r.t. class restrictions and we can assign the metric value 1 to them. In case of OpenCyc, 5 out of the 27,112 class restrictions are inconsistent. DBpedia contains 24 class constraints. Three out of them are inconsistent. For instance, over 1,200 instances exist which are both a dbo:Agent and a dbo:Place. YAGO contains 42 constraints, dedicated mainly for WordNet classes, which are mostly inconsistent.

Consistency of statements w.r.t. relation constraints, m conRelat  Evaluation method. Here we considered the relations rdfs:range and owl:FunctionalProperty, as those are used in more than every second considered KG. rdfs:range specifies the expected type of an instance on the object position of a triple, while owl:FunctionalProperty indicates that a relation should only be used at most once per resource. We only took datatype properties into account for this evaluation, since consistencies regarding object properties would require to distinguish Open World assumption and Closed World assumption.

Evaluation results. In the following, we consider the fulfillment degree for the relation constraints rdfs:range and owl:FunctionalProperty separately. In Table 5, we show the average of the fulfillment scores of each KG regarding rdfs:range and owl:FunctionalProperty. Note that the numbers of evaluated relation constraints varied from KG to KG, depending on how many relation constraints were available per KG.

Range. Wikidata does not use any rdfs:range restrictions. Within the Wikidata data model, there is wdo:propertyType, but this indicates not the exact allowed data type of a relation (e.g., wdo:propertyTypeTime can represent a year or an exact date). On the talk pages of Wikidata relations users can indicate the allowed values of relations via “One of” statements.97

DBpedia obtains the highest measured fulfillment score w.r.t. consistency of rdfs:range statements. An example for a range inconsistency is that the relation dbo:birthDate requires a data type xsd:date; in about 20% of those relations, the data type xsd:gYear is used, though.

YAGO, Freebase, and OpenCyc contain range inconsistencies primarily since they specify designated data types via range relations which are not consistently used on the instance level. For instance, YAGO specifies proprietary data types such as yago:yagoURL and yago:yagoISBN. On the instance level, however, either no data type is used or the unspecific data type xsd:string.

FunctionalProperty. The restriction indicated by owl:FunctionalProperty is used by all KGs except Wikidata. On the talk pages about the relations in Wikidata, users can specify the cardinality restriction via setting the relation to “single”; however, this is not part of the Wikidata data model. The other KGs mostly comply with the usage restrictions of owl:FunctionalProperty. Noteworthy is that in Freebase 99.9% of the inconsistencies obtained here are caused by the usages of the relations freebase:type.object.name and freebase:common.notable_for.display_name.

The fulfillment degrees of the KGs regarding the Relevancy criteria are shown in Table 6.

Creating a ranking of statements, m Ranking  

Only Wikidata supports the modeling of a ranking of statements: Each statement is ranked with “preferred rank” (wdo:PreferredRank), “normal rank” (wdo:NormalRank), or “deprecated rank” (wdo:DeprecatedRank). The “preferred rank” corresponds to the up-to-date value or the consensus of the Wikidata community w.r.t. this relation. Freebase does not provide any ranking of statements, entities, or relations. However, the meanwhile shutdown Freebase Search API provided a ranking for resources.98

The fulfillment degrees of the KGs regarding the Completeness criteria are shown in Table 7.

Schema completeness, m cSchema  Evaluation method. Since a gold standard for evaluating the Schema completeness of the considered KGs has not been published, we built one on our own. This gold standard is available online.99

Evaluation results. Generally, Wikidata performs optimal; also DBpedia, OpenCyc, and YAGO exhibit results which can be judged as acceptable for most use cases. Freebase shows considerable room for improvement concerning the coverage of typical cross-domain classes and relations. The results in more detail are as follows:

DBpedia DBpedia shows a good score regarding Schema completeness and its schema is mainly limited due to the characteristics of how information is stored and extracted from Wikipedia.

Freebase Freebase shows a very ambivalent schema completeness. On the one hand, Freebase targets rather the representation of facts on instance level than the representation of classes and their hierarchy. On the other hand, Freebase provides a vast amount of relations, leading to a very good coverage of the requested relations.

OpenCyc In total, OpenCyc exposes a quite high Schema completeness scoring. This is due to the fact that OpenCyc has been created manually and has its focus on generic and common-sense knowledge.

Wikidata According to our evaluation, Wikidata is complete both with respect to classes and relations.

YAGO Due to its concentration on modeling classes, YAGO shows the best overall Schema completeness fulfillment score among the KGs.

Column completeness, m cColumn  Evaluation method. For evaluating KGs w.r.t. Column completeness, for each KG 25 class-relation-combinations102

Evaluation results. In general, no KG yields a metric score of over 0.43. As visible in Table 8, KGs often have some specific class-relation-pairs which are well represented on instance level, while the rest of the pairs are poorly represented. The well-represented pairs presumably originate either from column-complete data sets which were imported (cf. MusicBrainz in case of Freebase), or from user edits focusing primarily on facts about entities of popular classes such as people. We notice the following observations with respect to the single KGs:

Population completeness, m cPop  

OPEN IN VIEWER

Fig. 10.

Population completeness regarding the different domains per KG.

Evaluation method. In order to evaluate the Population completeness, we need a gold standard consisting of a basic entity population for each considered KG. This gold standard, which is available online,104

Selecting four entities per class and five classes per domain resulted in 100 entities to be used for evaluating the Population completeness.

Evaluation results. All KGs except OpenCyc show good evaluation results. Since also Wikidata exhibits good evaluation results, the population degree apparently does not depend on the age or the insertion method of the KG. Figure 10 additionally depicts the population completeness for the single domains for each KG. In the following, we firstly present our findings for well-known entities, before we secondly go into the details of rather unknown entities.

Well-known entities: Here, all considered KGs achieve good results. DBpedia, Freebase, and Wikidata are complete w.r.t. the well-known entities in our gold standard. YAGO lacks some well-known entities, although some of them are represented in Wikipedia. One reason for this fact is that those Wikipedia entities do not get imported into YAGO for which a WordNet class exists. For instance, there is no “Great White Shark” entity, only the WordNet class yago:wordnet_great_white_shark_101484850.

Not-well-known entities: First of all, not very surprising is the fact that all KGs show a higher degree of completeness regarding well-known entities than regarding rather unknown entities, as the KGs are oriented towards general knowledge and not domain-specific knowledge. Secondly, two things are in particular peculiar concerning long-tail entities in the KGs: While most of the KGs obtain a score of about 0.88, Wikidata deflects upwards and OpenCyc deflects strongly downwards.

Wikidata exhibits a very high Population completeness degree for long tail entities. This is a result from the central storage of interwiki links between different Wikimedia projects (especially between the different Wikipedia language versions) in Wikidata: A Wikidata entry is added to Wikidata as soon as a new entity is added in one of the many Wikipedia language versions. Note, however, that in this way English-language labels for the entities are often missing. We measure that only about 54.6% (10.2M) of all Wikidata resources have an English label.

OpenCyc exhibits a poor population degree score of 0.14 for long-tail entities. OpenCyc’s sister KGs Cyc and ResearchCyc are apparently considerably better covered with entities [34], leading to higher Population completeness scores.

The evaluation results concerning the dimension Timeliness are presented in Table 9.

Timeliness frequency of the KG, m Freq  Evaluation results. The KGs are very diverse regarding the frequency in which the KGs are updated, ranging from a score of 0 for Freebase (not updated any more) to 1 for Wikidata (updates immediately visible and retrievable). Note that the Timeliness frequency of the KG can be a crucial point and a criterion for exclusion in the process of choosing the right KG for a given setting. In the following, we outline some characteristics of the KGs with respect to their up-to-dateness:

DBpedia is created about once to twice a year and is not modified in the meantime. From September 2013 until November 2016, six DBpedia versions have been published.107

Freebase had been updated continuously until its close-down and is not updated anymore.

OpenCyc has been updated less than once per year. The last OpenCyc version dates from May 2012.109

Wikidata provides the highest fulfillment degree for this criterion. Modifications in Wikidata are via browser and via HTTP URI dereferencing immediately visible. Hence, Wikidata falls in the category of continuous updates. Besides that, an RDF export is provided on a roughly monthly basis (either via the RDF export webpage110

YAGO has been updated less than once per year. YAGO3 was published in 2015, YAGO2 in 2011, and the interim version YAGO2s in 2013. A date of the next release has not been published.

Specification of the validity period of statements, m Validity  Evaluation results. Although representing the validity period of statements is obviously reasonable for many relations (for instance, the president’s term of office), specifying the validity period of statements is in several KGs either not possible at all or only rudimentary performed.

DBpedia and OpenCyc do not realize any specification possibility. In YAGO, Freebase, and Wikidata the temporal validity period of statements can be specified. In YAGO, this modeling possibility is made available via the relations yago:occursSince, yago:occursUntil, and yago:occursOnDate. Wikidata provides the relations “start time” (wdt:P580) and “end time” (wdt:P582). In Freebase, Compound Value Types (CVTs) are used to represent relations with higher arity [38]. As part of this representation, validity periods of statements can be specified. An example is “Vancouver’s population in 1997.”

Specification of the modification date of statements, m Change  Evaluation results. The modification date of statements can only be specified in Freebase but not in the other KGs. Together with the criteria on Timeliness, this reflects that the considered KGs are mostly not sufficiently equipped with possibilities for modeling temporal aspects within and about the KG.

In Freebase the date of the last review of a fact can be represented via the relation freebase:freebase.valuenotation.is_reviewed. In the DBpedia ontology the relation dcterms:modified is used to state the date of the last revision of the DBpedia ontology. When dereferencing a resource in Wikidata, the latest modification date of the resource is returned via schema:dateModified. This, however, does not hold for statements. Thus Wikidata is evaluated with 0, too.

The evaluation results of the dimension Ease of understanding are presented in Table 10.

Description of resources, m Descr  Evaluation method. We measured the extent to which entities are described. Regarding the labels, we considered rdfs:label for all KGs. Regarding the descriptions, the corresponding relations differ from KG to KG: DBpedia, for instance, uses rdfs:comment and dcelements:description, while Freebase provides freebase:common.topic.description.112

Evaluation result. For all KGs the rule applies that in case there is no label available, usually there is also no description available. The current metric could therefore (without significant restrictions) be applied to rdfs:label occurrences only.

YAGO, Wikidata, and OpenCyc contain a label for almost every entity. In Wikidata, the entities without any label are of experimental nature and are most likely not used.113

Surprisingly, DBpedia shows a relatively low coverage w.r.t. labels and descriptions (only 70.4%). Our manual investigations suggest that relations with higher arity are modeled by means of intermediate nodes which have no labels.114

Labels in multiple languages, m Lang  Evaluation method. Here we measure whether the KGs contain labels (rdfs:label) in other languages than English. This is done by means of the language annotations of literals such as “@de” for literals in German.

Evaluation results. DBpedia provides labels in 13 languages. Further languages are provided in the localized DBpedia versions. YAGO integrates statements of the different language versions of Wikipedia into one KG. Therefore, it provides labels in 326 different languages. Freebase and Wikidata also provide a lot of languages (244 and 395 languages, respectively). Contrary to the other KGs, OpenCyc only provides labels in English.

Coverage of languages. We also measured the coverage of selected languages in the KGs, i.e., the extent to which entities have an rdfs:label with a specific language annotation.115

Understandable RDF serialization, m uSer  The provisioning of understandable RDF serializations in the context of URI dereferencing leads to a better understandability for human data consumers. DBpedia, YAGO, and Wikidata provide N-Triples and N3/Turtle serializations. Freebase, in contrast, only provides a Turtle serialization. OpenCyc only uses RDF/XML, which is regarded as not easily understandable by humans.

Self-describing URIs, m uURI  We can observe two different paradigms of URI usage: On the one hand, DBpedia, OpenCyc, and YAGO rely on descriptive URIs and therefore achieve the full fulfillment degree. In DBpedia and YAGO, the URIs of the entities are determined by the corresponding English Wikipedia article. The mapping to the English Wikipedia is thus trivial. In case of OpenCyc, two RDF exports are provided: one using opaque and one using self-describing URIs. The self-describing URIs are thereby derived from the rdfs:label values of the resources.

On the other hand, Wikidata and Freebase (the latter in part) rely on opaque URIs: Wikidata uses Q-IDs for resources (“items” in Wikidata terminology) and P-IDs for relations. Freebase uses self-describing URIs only partially, namely, opaque M-IDs for entities and self-describing URIs for classes and relations.116

The evaluation results of the dimension Interoperability are presented in Table 11.

Avoiding blank nodes and RDF reification, m Reif  Reification allows to represent further information about single statements. In conclusion, we can state that DBpedia, Freebase, OpenCyc, and YAGO use some form of reification. However, none of the considered KGs uses the RDF standard for reification. Wikidata makes extensive use of reification: every relation is stored in the form of an n-ary relation. In case of DBpedia and Freebase, in contrast, facts are predominantly stored as N-Tripels and only relations of higher arity are stored via n-ary relations.117

Blank nodes are non-dereferencable, anonymous resources. They are used by the Wikidata and OpenCyc data model.

Provisioning of several serialization formats, m iSerial  DBpedia, YAGO, and Wikidata fulfill the criterion of Provisioning several RDF serialization formats to the full extent, as they provide data in RDF/XML and several other serialization formats during the URI dereferencing. In addition, DBpedia and YAGO provide further RDF serialization formats (e.g., JSON-LD, Microdata, and CSV) via their SPARQL endpoints. Freebase is the only KG providing RDF only in Turtle format.

Using external vocabulary, m extVoc  Evaluation method. This criterion indicates the extent to which external vocabulary is used. For that, for each KG we divide the occurrence number of triples with external relations by the number of all relations in this KG.

Evaluation results. DBpedia uses 37 unique external relations from 8 different vocabularies, while the other KGs mainly restrict themselves to the external vocabularies RDF, RDFS, and OWL.

Wikidata reveals a high external vocabulary ratio, too. We can mention two obvious reasons for that fact: 1. Information in Wikidata is provided in a huge variety of languages, leading to 85M rdfs:label and 140M schema:description literals. 2. Wikidata makes extensive use of reification. Out of the 140M triples used for instantiations via rdf:type, about 74M (i.e., about the half) are taken for instantiations of statements, i.e., for reification.

Interoperability of proprietary vocabulary, m propVoc  Evaluation method. This criterion determines the extent to which URIs of proprietary vocabulary are linked to external vocabulary via equivalence relations. For each KG, we measure which classes and relations are linked via owl:sameAs,118

Evaluation results. In general, we obtained low fulfillment scores regarding this criterion. OpenCyc shows the highest value. We achieved the following single findings:

Regarding its classes, DBpedia reaches a relative high interlinking degree of about 48.4%. Classes are thereby linked to FOAF, Wikidata, schema.org and DUL.119

Freebase only provides owl:sameAs links in the form of a separate RDF file, but these links are only on instance level. Thus, the KG is evaluated with 0.

In OpenCyc, about half of all classes exhibit at least one external linking via owl:sameAs. Internal links to resources of sw.cyc.com , the commercial version of OpenCyc, were ignored in our evaluation. The considered classes are mainly linked to FOAF, UMBEL, DBpedia, and linkedmdb.org, the relations mainly to FOAF, DBpedia, Dublin Core Terms, and linkedmdb.org. The relative high linking degree of OpenCyc can be attributed to dedicated approaches of linking OpenCyc to other KGs (see, e.g., Medelyan et al. [36]).

Regarding the classes, Wikidata provides links mainly to DBpedia. Considering all Wikidata classes, only 0.1% of all Wikidata classes are linked to equivalent external classes. This may be due to the high number of classes in Wikidata in general. Regarding the relations, Wikidata provides links in particular to FOAF and schema.org and achieves here a linking coverage of 2.1%. Although this is low, frequently used relations are linked.121

YAGO contains around 553K owl:equivalentClass links to classes within the DBpedia namespace dby:. However, as YAGO classes (and their hierarchy) were imported also into DBpedia (using the namespace http://dbpedia.org/class/yago/), we do not count those owl:equivalentClass links in YAGO as external links for YAGO.

The evaluation results of the dimension Accessibility are presented in Table 12.

Dereferencing possibility of resources, m Deref  

Evaluation method. We measured the dereferencing possibilities of resources by trying to dereference URIs containing the fully-qualified domain name of the KG. For that, we randomly selected 15K URIs in the subject, predicate, and object position of triples in each KG. We submitted HTTP requests with the HTTP accept header field set to application/rdf+xml in order to perform content negotiation.

Evaluation results. In case of DBpedia, OpenCyc, and YAGO, all URIs were dereferenced successfully and returned appropriate RDF data, so that they fulfilled this criterion completely. For DBpedia, 45K URIs were analyzed, for OpenCyc only around 30K due to the small number of unique predicates. We observed almost the same picture for YAGO, namely no notable errors during dereferencing.

For Wikidata, which contains also not that many unique predicates, we analyzed around 35K URIs. Note that predicates which are derived from relations using a suffix (e.g., the suffix “s” as in wdt:P1024s is used for predicates referring to a statement) could not be dereferenced at all. Furthermore, the blank nodes used for reification cannot be dereferenced.

Regarding Freebase, mainly all URIs on subject and object position of triples could be dereferenced. Some resources were not resolvable even after multiple attempts (HTTP server error 503; e.g., freebase:m.0156q). Surprisingly, server errors also appeared while browsing the website freebase.com, so that data was partially not available. Regarding the predicate position, many URIs are not dereferencable due to server errors (HTTP 503) or due to unknown URIs (HTTP 404). Note that if a large number of Freebase requests are performed, an API key from Google is necessary. In our experiments, the access was blocked after a few thousand requests. Hence, we can point out that without an API key the Freebase KG is only usable to a limited extent.

Availability of the KG, m Avai  Evaluation method. We measured the availability of the officially hosted KGs with the monitoring service Pingdom.122

Evaluation result. While the other KGs showed almost no outages and were again online after some minutes on average, YAGO outages took place frequently and lasted on average 3.5 hours.123

Availability of a public SPARQL endpoint, m SPARQL  The SPARQL endpoints of DBpedia and YAGO are provided by a Virtuoso server,124

Especially regarding the Wikidata SPARQL endpoint we observed access restrictions: The maximum execution time per query is set to 30 seconds, but there is no limitation regarding the returning number of rows. However, the front-end of the SPARQL endpoint crashed in case of large result sets with more than 1.5M rows. Although public SPARQL endpoints need to be prepared for inefficient queries, the time limit of Wikidata may impede the execution of reasonable queries.

Provisioning of an RDF export, m Export  All considered KGs provide RDF exports as downloadable files. The format of the data differs from KG to KG. Mostly, data is provided in N-Triples and Turtle format.

Support of content negotiation, m Negot  We measure the support of content negotiation regarding the serialization formats RDF/XML, N3/Turtle, and N-Triples. OpenCyc does not provide any content negotiation; only RDF/XML is supported as content type. Therefore, OpenCyc does not fulfill the criterion of supporting content negotiation.

The endpoints for DBpedia, Wikidata, and YAGO correctly returned the appropriate RDF serialization format and the corresponding HTML representation of the tested resources. Freebase does currently not provide any content negotiation and only the content type text/plain is returned.

Noteworthy is also that regarding the N-Triples serialization YAGO and DBpedia require the accept header text/plain and not applicationn-triples. This is due to the usage of Virtuoso as endpoint. For DBpedia, the forwarding to http://dbpedia.org/data/[resource].ntriples does not work; instead, the HTML representation is returned. Therefore, the KG is evaluated with 0.5.

Linking HTML sites to RDF serializations, m HTMLRDF  All KGs except OpenCyc interlink the HTML representations of resources with the corresponding RDF representations by means of <link rel="alternate" type="{content type}" href= "{URL}"/> in the HTML header.

Provisioning of metadata about the KG, m meta  For this criterion we analyzed if KG metadata is available, such as in the form of a VoID file.126

The evaluation results of the dimension License are shown in Table 13.

Provisioning machine-readable licensing information, m macLicense  

DBpedia and Wikidata provide licensing information about their KG data in machine-readable form. For DBpedia, this is done in the ontology via the predicate cc:license linking to CC-BY-SA128

The evaluation results of the dimension Interlinking are shown in Table 14.

Linking via owl:sameAs, m Inst  Evaluation method. Given all owl:sameAs triples in each KG, we queried all those subjects thereof which are instances but neither classes nor relations133

Evaluation result. OpenCyc and YAGO achieve the best results w.r.t. this metric, but DBpedia has by far the most instances with at least one owl:sameAs link. We can therefore confirm the statement by Bizer et al. [10] that DBpedia has established itself as a hub in the Linked Data cloud.

In DBpedia, there are about 5.2M instances with at least one owl:sameAs link. Links to localized DBpedia versions (e.g., de.dbpedia.org ) were counted as internal links and, hence, not considered here. In total, one-fourth of all instances have at least one owl:sameAs link.

In Wikidata, neither owl:sameAs links are provided nor a corresponding proprietary relation is available. Instead, Wikidata uses for each linked data set a proprietary relation (called “identifier”) to indicate equivalence. For example, the M-ID of a Freebase instance is stored via the relation “Freebase identifier” (wdt:P646) as literal value (e.g., "/m/01x3gpk"). So far, links to 426 different data sources are maintained in this way.

Although the equivalence statements in Wikidata can be used to generate corresponding owl:sameAs statements and although the stored identifiers are provided in the Browser interface as hyperlinks, there are no genuine owl:sameAs links available. Hence, Wikidata is evaluated with 0. If we view each equivalence relation as owl:sameAs relation, we would obtain around 12.2M instances with owl:sameAs statements. This corresponds to 8.6% of all instances. If we consider only entities instead of instances (since there are many instances due to reification), we obtain a coverage of 65%. Note, however, that, although the linked resources provide relevant content, the resources are not always RDF documents, but instead HTML web pages. Therefore, we cannot easily subsume all “identifiers” (equivalence statements) under owl:sameAs.

YAGO has around 3.6M instances with at least one owl:sameAs link. However, most of them are links to DBpedia based on common Wikipedia articles. If those links are excluded, YAGO contains mostly links to GeoNames and would be evaluated with just 0.01.

In case of OpenCyc, links to Cyc,134

Validity of external URIs, m URIs  Regarding the dimension Accessibility, we already analyzed the dereferencing possibility of resources in the KG namespace. Now we analyze the links to external URIs.

Evaluation method. External links include owl:sameAs links as well as links to non-RDF-based Web resources (e.g., via foaf:homepage). We measure errors such as timouts, client errors (HTTP response 4xx), and server errors (HTTP response 5xx).

Evaluation result. The external links are in most of the cases valid for all KGs. All KGs obtain a metric value between 0.89 and 0.96.

DBpedia stores provenance information via the relation prov:wasDerivedFrom. Since almost all links refer to Wikipedia, 99% of the resources are available.

Freebase achieves high metric values here, since it contains owl:sameAs links mainly to Wikipedia. Also Wikipedia URIs are mostly resolvable.

OpenCyc contains mainly external links to non-RDF-based Web resources to wikipedia.org and w3.org.

YAGO also achieves high metric values, since it provides owl:sameAs links only to DBpedia and GeoNames, whose URIs do not change.

For Wikidata the relation “reference URL” (wdt:P854), which states provenance information among other relations, belongs to the links linking to external Web resources. Here we were able to resolve around 95.5% without errors.

Noticeable is that DBpedia and OpenCyc contain many owl:sameAs links to URIs whose domains do not exist anymore.136

We now summarize the results of the evaluations presented in this section.

Syntactic validity of RDF documents: All KGs provide syntactically valid RDF documents.