Evaluating the quality of the LOD cloud: An empirical investigation

5.2.1. RDF data model

RDF is a graph-based data model, in which the basic structure is a triple t : = ( s , p , o ) . Each element of a triple t is restricted to a set of RDF constants [33]: URIs ( U ), literals ( L ), and blank nodes ( B ). More specifically, a triple t can be defined as a cartesian product t ∈ ( U ∪ B ) × U × ( U ∪ B ∪ L ) . In this study, the notation U refers to the set of all URIs in dataset D. A finite set of triples t make up an RDF dataset D. In this article, whenever we refer to triple t, it implies t ∈ D , unless otherwise stated. When required, we refer to particular parts a triple, in this article we use t s as a reference to the subject part of the triple, t p for predicate, and t o for object.

A quad q is an ordered pair of ( t , c ) , where c is the context for t : = ( s , p , o ) .

5.2.2. Linked Data entities

A Linked Data entity e : = { ( s , p 1 , o 1 ) , … , ( s , p n , o n ) } , where n ⩽ size ( D ) , is a set of triples sharing the same subject URI. In this paper, we refer to a set of Linked Data entities e as E .

5.2.3. Data-level constants

A data-level constant is defined by Hogan et al. in [33] to be the subject or the object of a triple or quad, when the predicate is not rdf:type. Similar to Hogan et al., in this paper we use the same notation of d l c ( D ) to refer to the set of possible data-level constants in dataset D.

We define a function class : P ( D ) → U such that class ( D ) : = { o ∈ U ∣ ( s , rdf : type , o ) ∈ D } , that given a dataset D it returns a set of URIs of the classes used in the dataset to define Linked Data entities. Similarly, we define a function prop : P ( D ) → P ( U ) such that prop ( D ) : = { p ∈ U ∣ ( s , p , o ) ∈ D } that returns a set of all predicates used in the dataset.

5.2.5. External vocabularies

A vocabulary dereferencer [52] function V : U → S is defined as the mapping from the URI to a dereferenceable schema s, where s is a set of domain specific concepts (classes) and relationships (properties) between these concepts, and S is the set of all schemas.

5.2.6. Metrics terminology

In Table 2 we list and explain the terms and symbols commonly used throughout the 27 metrics. Some specific one-off terms are explained directly where used.

Classified in the representational-conciseness dimension, this metric observes the length of URIs. In the Cool URIs document [52], the editors remarked that apart from providing descriptions for people and machines, the best URIs are simple, stable, and manageable.

This metric focuses on the simplicity aspect of this definition, where by simplicity the editors of the same document mean that having short and mnemonic URIs are easier for humans to remember (e.g. http://danbri.org/foaf vs. https://w3id.org/lodquator/resource/826514e9-e34a-40a2-bc8d-9e6b8bd54770), whilst serving the purpose of being machine processable. Hogan et al. [33] remarked that short URIs have other benefits such as allowing for smaller sized datasets and indexes.

Metric Computation: The metric computation is based on the W3C best practices for URIs, where the editor suggests that a URI should not be longer than 80 characters [58, §1.1]. Furthermore, URIs with appended parameters are considered as bad, irrelevant of their length. The metric can be quantified as follows: RC 1 ( D ) : = size ( { u ∈ U ∣ ( len ( u ) ⩽ 80 ) ∧ ( ′ ? ′ ∉ u ) } ) size ( dlc ( D ) ∩ U ) where u is a URI that has a length (defined by function len ) of 80 or less and is not parameterised (URI contains no ‘?’). Therefore, this metric value measures the ratio of short URIs in a dataset.

Discussion: A box plot with the quality values is illustrated in Fig. 5. The box plot for this metric (RC1) suggests that publishers tend to have quite different inclinations on how long the URI identifiers should be. The sample over the LOD Cloud is centred at a median of 97.92% with a standard deviation of 24.90%. A number of outliers (around 13% of the datasets), were detected. These are datasets that scored lower than 54.55%. We also notice that the population is negatively skewed (i.e the median is closer to the third quartile). The median quality value across the assessed datasets is around 84.07%, with 69% of the datasets scoring more than 90%, and 28% of the datasets scoring 100%. From the sample problem report we extracted during the assessment, 14.65% of the URIs were parameterised whilst the rest were URIs longer than 80 characters. This metric has two drawbacks. First, our metric takes into consideration external URIs, however, we acknowledge that the length of such external URIs cannot be influenced by the datasets’ publisher. A solution for this is that the metric looks only at locally minted URIs. The second drawback of this metric is the lack of discriminative power, since URIs with 80 characters are adequate, whilst longer ones are deemed to be non-conformant with the guidelines set in [58, §1.1]. In order to avoid this discriminatory power problem, Hogan et al. [33, §5.1 – Issue IV] calculate the metric value based on the mean length of the URIs in a dataset, rather than relying on the 80 character limit, promoting those datasets that have short URIs.

The editor of the Common HTTP Implementation guidelines [58] states that 80 character limit is not a technical limitation but rather a practical goal one should pursue. There might be various reasons for publishers to use longer URIs. For example, URIs can comprise some structure, such as a directory scheme. Moreover, in a recent article by Szász et al. [57], the authors discussed the idea of self-unfolding semantic URIs, where such URIs follow a specific pattern and template that would result into a set of RDF triples “on-demand”. These self-unfolding URIs might be longer than the 80 character limit, but the benefits of such URIs include frequent complex insertion in RDF triple stores, for example in sensor network streams. Therefore, we deem that this 80 character practical limit suggested in [58, §1.1] has its own limitations and could affect the quality value of datasets where publishers cannot do without lengthy URIs, for example having resources with a multi-level domain name such as in the case of university URIs.

5.3.2. (RC2) Minimal usage of RDF data structures

The usage of RDF data structure features, more specifically reification, containers, and collections, is discouraged due to their syntactic/semantic complexity. Despite the fact that a number of efforts were made in order to facilitate the use of such data structures (e.g. the introduction of property paths25

Metric Computation: This metric detects the use of standard RDF data structure features. More specifically, this metric checks quads as suggested in [33, §5.3 – Issue VIII]:

if the predicate is rdf:type and the object is one of rdf:Statement, rdf:Alt, rdf:Bag, rdf:Seq, rdf:Container, or rdf:List;

Discussion: Similar to the findings of Hogan et al. [33], most publishers do not use RDF data structures. In our assessment 87.2% of the publishers use none, compared to the 78.7% reported by Hogan et al. This is reflected in the short box plot illustration for this metric (RC2 – Fig. 5), with the interquartile ranges and whiskers all being close to 100%. The mean quality value of this metric is 99.44% and the calculated standard deviation is σ s = 2.86 % (median value is 100). The σ s value confirms our findings that most publishers try to minimise the use of such undesired RDF features, with 97% of the datasets ranking within the standard deviation (i.e. having a quality value of at least 96.369%). Similar to Metric RC1, a relatively small number of outliers (around 12% of the datasets) were detected.

Nonetheless, this metric punishes datasets where ordering is essential. For example, the dataset with the lowest quality value (70.25%) for this metric is http://bibsonomy.org, which contains resources on scientific papers amongst others. Upon further inspection of this dataset, we found that the dataset’s publisher used rdf:Seq and rdf:Bag to represent the editors and authors of publications in their order, which is essential in this domain. However, since quality is termed as fitness for use, practitioners that require datasets where ordering is important might not use this metric at all. In general, the RDF collections were the most common issue (95.23%), followed by RDF containers (3.09%) and RDF reification (1.67%).

Vocabulary re-use is widely advocated. For instance, Bizer and Heath [13] argue that re-using terms from known vocabularies makes it easier for applications to process Linked Data, thus increasing interoperability between agents. Schemas for different domains are publicly available; also via registries such as the Linked Open Vocabulary (LOV) portal.26

Metric Computation: This metric assesses if a dataset re-uses relevant terms in a particular domain. In particular, this metric checks if a property or a class (in case the predicate is rdf:type) used in a triple refers to a term in another vocabulary. Identifying vocabularies for different domains can be achieved in various ways. For example a dataset might have a set of potentially relevant vocabularies manually assigned prior to the assessment. This ‘assignment’ could also be done in an automatic fashion, for example, by looking at vocabularies used by datasets in the same domain, or even using services such as LOV. Similarly, a dataset domain can be assigned to a dataset manually or automatically, for example, from its metadata, as in the case of datasets manifested in the LOD Cloud (e.g. Lexvo is tagged as linguistics). Furthermore, based on the usage study conducted in [54], we included the following vocabularies by default for all datasets: RDF, RDFS, FOAF, DCMI Terms, OWL, GEO, SIOC, SKOS, VoID, DCAT.

We identify overlapping classes and properties in the same manner as defined in [33, §5.3 – Issue IX], with the set of known vocabularies. The metric counts the number of external classes and properties for a particular domain: IO 1 ( D ) : = size ( c l e x s ‾ ) + size ( p r e x s ‾ ) size ( class ( D ) ) + size ( prop ( D ) ) c l e x s ‾ : = v c ‾ ∩ class ( D ) p r e x s ‾ : = v p ‾ ∩ prop ( D ) where v c ‾ and v p ‾ are the sets of all classes and properties respectively, gathered from the identified external vocabularies for the particular dataset.

Discussion: In this empirical study, we used the LOD Cloud classification in order to identify the domain of an assessed dataset. The LOV service was used in order to fetch vocabularies related to the datasets’ domain. With the pre-defined tags associated to each dataset, we ensured that each dataset is assessed solely based on its domain; relying on the LOV service to provide us with relevant public vocabularies. Nonetheless, using the LOV service comes with two potential pitfalls that can affect the results of this metric:

The box plot for this metric (IO1 – Fig. 5) is comparatively (against the other metrics in this category) long and positively skewed, suggesting that most values are small with some larger values. This also suggests that there is a lack of conformity on the principle of re-use; only few publishers rely actively on the re-using vocabularies (≈10% of datasets have a quality value of > 90%), with 8.8% of the datasets being outliers in this case as they have a quality value larger than 92.32% (i.e. the upper whisker value). The sample is centred at a median of 24.00% with a standard deviation of 29.10%, and a mean value of 34.01%, indicating low overall re-use. One possibility is the fact that publishers (such as DBpedia) use local terms and properties with few external properties (e.g. rdfs:label). Our values are comparable to those in Hogan et al. [33, §5.3 – Issue IX], where the authors also suggest that the amount of re-used terms and properties in their sample is widely distributed.

Our current implementation of this metric does not take into account the best practice of introducing user-defined terms by linking them existing terms using predicates such as owl:sameAs, owl:equivalentClass, or owl:equivalentProperty; such terms are not currently recognised as valid re-used existing terms.

The invalid usage of undefined classes and properties metric is classified under the interpretability dimension [60], targetting the technical representation of the data itself. Using classes and properties without a formal definition is undesirable, as agents would not be able to understand how the data should be interpreted, for example, during reasoning. Errors leading to such invalid usage include capitalisation errors (e.g. foaf:person vs. foaf:Person), syntactic errors (e.g. foaf:img vs. foaf:image), and schema dereferencability issues.

Metric Computation: This metric measures the number of undefined classes and properties in the assessed dataset: IN 3 ( D ) : = 1.0 − size ( c l u n d e f ‾ ) + size ( p r u n d e f ‾ ) size ( class ( D ) ) + size ( prop ( D ) ) c l u n d e f ‾ : = { c ∈ class ( D ) ∣ c l u n d e f ‾ : = { ( c , rdf:type , C ) ∣ C ∈ C } c l u n d e f ‾ : = ∩ V ( ns ( c ) ) = ∅ } p r u n d e f ‾ : = { p ∈ prop ( D ) ∣ p r u n d e f ‾ : = { ( p , rdf:type , P ) ∣ P ∈ P } c l u n d e f ‾ : = ∩ V ( ns ( p ) ) = ∅ } An undefined class is a term c that is used as a class in the dataset, as defined above for Metric IO1, but that is not defined to be an instance of a class type C in the vocabulary V ( ns ( c ) ) retrieved by dereferencing its namespace URI.27

Discussion: The box plot for this quality metric (IN3 – Fig. 5) covers a range of 99.58%. This suggests that data publishers are using a wide range of defined and undefined classes and properties. Furthermore, the quality value is centred at a median of 53.33% with a standard deviation value of 32.18%, whilst the mean quality value is 54.48%.

A higher value means that publishers were using fewer undefined terms in their dataset. From our assessment, 30.80% of properties used were undefined. Some of the undefined terms were possibly previously defined. For example, for the rkbexplorer datasets, the publishers use terms from the aktors.org namespace, which now resolves to a personal blog. As another common pattern of undefined terms, we noticed the use of the wrong namespace for a term that did exist in a similar namespace, for example, rdfs:Property as opposed to rdf:Property. Other datasets had schemas that were unavailable during the assessment, thus resulting in undefined terms.

Blank nodes are undesirable in Linked Data because they cannot be externally referenced, which conflicts with the two Linked Data best practices interlinking and re-using. In simple terms, the scope of blank nodes is “limited to the document in which they appear” [29]. Moreover, the existence of blank nodes can cause a number of problems during Linked Data consumption and when performing certain tasks, such as deciding whether two RDF graphs are isomorphic.

Metric Computation: This metric assesses the usage of blank nodes within the subjects and objects. The metric value is assessed as suggested in [33, §5.1 – Issue I]: IN 4 ( D ) : = size ( dlc ( D ) ∖ B ) size ( dlc ( D ) )

Discussion: The box plot (IN4) illustrated in Fig. 5, is relatively short, suggesting that most data publishers agree to avoid blank nodes. The median value is 100%, the standard deviation is 12.15%, whilst the mean quality metric value of 96.01% confirms the generally high conformance with this metric.

Whilst the majority of data publishers use blank nodes sparsely or not at all (around 85% of the datasets score higher than 94.93%, which is the lower whisker limit), there are a number of datasets marked as outliers consequently stretching the standard deviation. In particular, the prefix.cc dataset uses blank nodes in almost every triple. This dataset affected the standard deviation significantly, which otherwise would be considerably lower than in [33, §5.1 – Issue I]. One should note that the corpus in [33] contained FOAF profiles, which traditionally, according to Hogan et al. [33], may contain many blank nodes. In certain situations, the usage of blank nodes is due to RDF data structure features and OWL axioms, as these structures and axioms use blank nodes as the encoding, though in general avoiding them means that resources in a dataset are more likely to be re-used for linking.

5.3.6. (V1) Different serialisation formats

An RDF data model can be serialised using a variety of formats, including RDF/XML, RDFa, Turtle, N-Triples, N-Quads, and JSON-LD. For example, Web applications prefer the JSON-LD format, rather than having to use some parser, as the JavaScript environment handles JSON data internally. The different characteristics of each serialisation brings about different pros and cons, as described in [29, §2.4.2]. The rationale of this metric is to assess whether various consumption methods are supported. Ensuring that a dataset is available in multiple serialisation formats facilitates its use. The metric is classified under the versatility dimension [60].

Metric Computation: This metric checks whether a dataset has multiple serialisation formats defined in its metadata, by verifying that multiple quads having void:feature as a predicate exist in the assessed dataset. The void:feature predicate (range:void:TechnicalFeature) is used to express the technical features of a dataset, such as the serialisation formats the dataset is available in [3, §2.6]. Data publishers can serialise their data in up to 23 different formats.28

Discussion: In most cases, the publishers did not define any serialisation format in the metadata of their datasets. In total, only nine datasets had a serialisation format following our guideline. The standard deviation is 0.71 formats whilst the mean value is 0.18 formats per dataset. In total we had one dataset with four different serialisation formats, five datasets (four of which had a pay-level domain of psi.enakting.org) had three different serialisation formats listed, two datasets had two formats listed, whilst one dataset had only one serialisation format listed. The most common formats were N-Triples, RDFa, RDF/XML, Turtle, N-Quads, and SPARQL query results in XML format.

To compute this metric, we have assumed that datasets use the predicate void:feature to express the format of the dump. However, this metric can be extended to include the DCAT specific predicate dcat:mediaType, given that a finite set of media types for each serialisation format is specified in the metric.

A dataset serialised in different formats, widens possible uses in different scenarios. In order to encourage multiple format serialisation, tools such as Raptor29

Catering for multiple languages ensures that the dataset reaches a wider global audience. For example, a dataset with literals having only a Maltese language tag is not suitable for Chinese speaking users. On the other hand, if the dataset has literals in both Maltese and Chinese, then the dataset is likely to be re-used more. A plain (textual) literal string can be combined with a language tag (e.g. @mt) Furthermore, the Data on the Web Best Practices document suggests that locale parameters should be provided in metadata:

“making the language explicit allows users to determine how readily they can work with the data and may enable automated translation services.” – [38]

Metric Computation: This metric checks the number of languages a dataset supports. Specifically, the metric checks whether the data (in this case string literals) is evenly available in different languages. For example if resource R 1 has two labels with different language tags and R 2 , R 3 , R 4 , R 5 have only one label with one language tag each, then on average the dataset has resources defined in 1.2 languages, which is rounded down to one language. l : = { s ∣ ∃ o . o = “lexical form” @ lang ∧ ( s , p , o ) ∈ D } triple s lang : = { t ∣ t o ∈ L ∧ hasLangTag ( t o ) } V 2 ( D ) : = round ( size ( triple s lang ) size ( l ) ) where triples lang is the set of all triples in the dataset D whose object is a literal with some language tag, and l is the set of unique resources having at least one property with a literal object that has a language tag.

Discussion: In most cases, publishers describe their textual literals using only one language (≈83% of the datasets). One possible reason is that publishers target a particular audience, or do not have the resources to create multilingual datasets. The mean value for this metric is 1.72 languages, whilst the standard deviation is 2.71 languages (median: 1 language). Overall, this metric shows positive skewness as the standard deviation is almost twice of the mean and median values. In total, we had four datasets having fifteen different languages (these had a pay-level domain of psi.enakting.org), languages including Arabic, Greek, English, Italian and Spanish. Furthermore, resources where described in thirteen, eight and four different languages, in three different datasets. Finally, eleven datasets had at least resources described in two different languages for human consumption.

Figure 5 shows a box plot illustration of the aggregated quality value compared with the category’s metrics (V1 and V2 are missing as the quality value are integers used to count things, whilst the rest are float values between 0.0 and 1.0). The overall aggregated box plot shows a population which is slightly skewed to the left, with a median of 60.70% conformance. This shows that there is more variety amongst higher quality values amongst the sample. Nevertheless, the standard deviation is 14.50%, which indicates a moderate distribution, whilst the mean score is 63.60%.

Data provenance is considered as one of the main assets in a Linked Data.

“Data provenance becomes particularly important when data is shared between collaborators who might not have direct contact with one another either due to proximity or because the published data outlives the lifespan of the data provider projects or organisations.” – [38, §9.4]

Metric Computation: At the very least, a dataset should have a dc:creator or dc:publisher within their VoID or DCAT metadata. We focus on searching for triples with the predicates dc:creator or dc:publisher in the dataset having a type void:Dataset or dcat:Dataset. The metric can be formally defined as follows: P 1 ( D ) : = ∑ d ∈ ds ( D ) basic ( d ) size ( ds ( D ) ) where ds ( D ) is the set of resources having a type of void:Dataset or dcat:Dataset in the assessed dataset D, whilst basic ( d ) is a function that returns ‘1’ if d ∈ ds has a triple corresponding to d × (dc:creator ∥ dc:publisher) × object.

OPEN IN VIEWER

Fig. 6.

Contextual category box plot. Outliers are represented by dots.

Discussion: A box plot with the quality values for the contextual dimension metric is given in Fig. 6. The box plot for this metric (P1) is, qualitatively speaking, positively skewed, suggesting that most of the sampled datasets contain no basic provenance information in their VoID or DCAT metadata (when available). Nonetheless, this metric has a number of outliers, amounting to around 16.27% of the datasets. From this 16.27%, 71% of the datasets have a quality value of 100%. The σ s value stands around 32.89% (median 0%), whilst the mean is 12.78%. One possibility of this low score was due to the fact that the metrics did not consider dcterms:creator and its equivalent foaf:maker. Considering these two predicates in this metric might give higher overall values.

Publishers might add basic provenance triples directly in a dataset rather than in the metadata, which is a drawback in terms of “understand(ing) the meaning of data” [38], as the provenance will be unknown to an automated agent looking for this information within the metadata before consuming the actual data. For example, europeana.eu attaches a dc:creator to every resource (which is understandable considering that it is a dataset about cultural heritage) rather to specific metadata resources. Hence, we encourage publishers to use dataset profiles specifications such as DCAT-AP32

In the Data on the Web Best Practices recommendation, the editors note that

“consumers need to know the origin or history of the published data, [...], data published should include or link to provenance information” – [38, §9.4]

Metric Computation: This metric checks whether each resource has provenance information related to the origin of data. With regard to the quality metric survey in [60], this metric can be related to the “trustworthiness of statements”. More specifically, this metric checks for provenance entities with the following characteristics:

Identification of an agent of an entity (quads having a predicate prov:wasAttributedTo);

Discussion: Similar to Metric P1, this metric (P2 – Fig. 6) is also very positively skewed. Unlike Metric P1, the granularity level of the metadata in this case can even reach a triple level. This means that the size of the overall dataset can grow very large, therefore publishers might not be willing to trade-off size for better metadata coverage. In fact, we noticed that there is only one publisher (270a.info datasets) with metadata enabling users to identify the origin of data. The overall median value is 0%. The standard deviation stands around 10.06%, whilst the mean is 2.17%.

The practice of tracking the origin of data is often ignored by data publishers, possibly for a myriad for reasons, such as the inflating the size of the dataset, or modelling issues. We suggest that publishers add provenance information on the activities undertaken when creating resources in their dataset, and possibly separating this metadata from the data itself by using named graphs.

5.4.3. (U1) Human readable labelling and comments

Data on the Web is meant to be exposed to both humans and machines. Therefore, a human information consumer should be able to comprehend and understand a Linked Data resource. Apart from human understandability, labels and comments can be used in various applications, such as keyword-based and natural-language based search [20]. A Linked Data application is dependent on labels and comments provided with each resource, as the application itself is not yet intelligent enough to try to map a resource to its real-world description. Labels can possibly be extracted from a human readable URI, e.g. extracting the fragment ‘Dublin’ from http://dbpedia.org/resource/Dublin.

Heath and Bizer suggest that predicates such as rdfs:label, foaf:name, skos:prefLabel, or dcterms:title should be used to label resources as they are widely supported by Linked Data applications, whilst dcterms:description and rdfs:comment should be used for a textual description of a resource [29]. Nevertheless, there are a number of vocabularies having terms to describe human readable labels and comments.34

Metric Computation: The aim of this metric is to calculate a dataset’s completeness in terms of human-readable labels and descriptions. The metric measures the percentage of local entities that have a label or a description. More specifically, each resource should have one (or more) of the following predicates, extracted from the top 50 vocabularies used in the LOD Cloud [54]:

rdfs:label;

Discussion: The box plot for this quality metric (U1 – Fig. 6) is relatively tall, showing a uniform distribution. This suggests that data publishers follow varying practices with regard to the conformance of human-readable labels and comments. Furthermore, one must keep in mind that similar to the metric related to the usage of multiple languages (Metric V2), one might not need to label or textually describe a resource. For example, in data cube36

The quality value is centred on 33.33% with a standard deviation of 40.93%, whilst the mean quality value is 43.76%. This quality metric displays the highest dispersion from all contextual metrics as evidenced by a wide bar. Moreover, around 29.29% of the assessed datasets have a completeness value of more than 90%, whilst in total around 43% of the datasets have a value of more than 50%. This metric is similar to the one presented in Hogan et al. [33, §5.3 – Issue XI], however, in this assessment we analysed a larger variation than in that of the study in 2012. We can also draw parallels between our assessment results and the results presented in [20], as both assessments show that the community needs to work harder to ensure the completeness of human readable labels and descriptions in Linked (Open) Datasets.

One of the main purposes of the Web of Data is to be queried and explored. Structural metadata enables consumers to understand the underlying structure of a dataset. Having a regular expression defining the URI structure of a dataset enables agents to interpret resources better, for example, extracting fragments of a resource URI such as its local name, or querying a dataset to retrieve local resources according to the specified URI structure. The regular expression definition of a URI metric is classified under the understandability dimension [60].

Metric Computation: This metric checks for the identification of a URI regular expression in the dataset’s metadata, and can be quantified as follows: U 3 ( D ) : = 1.0 if has pattern , 0.0 otherwise , where by has pattern , the metric is looking for a triple subject × void:uriRegexPattern × object in the assessed dataset.

Discussion: This metric reports 100% if the assessed dataset has a URI regular expression pattern defined. Our assessment showed that only ten of the datasets had such an expression, giving a total mean value of 7.75%, and a standard deviation of 26.84%. The box plot for the metric U3 in Fig. 6, illustrates this positively skewed quality indicator.

5.4.5. (U5) Indication of used vocabularies

Vocabularies play an important role in the structure of a dataset, since one or more of these vocabularies describe the dataset’s resources. Similar to Metric U3, indicating the vocabularies used is part of the structural metadata of a dataset. Knowing the vocabularies used in a dataset, a human consumer can query the data. This metric is also classified under the understandability dimension [60].

Metric Computation: This metric checks whether vocabularies used in the datasets, either in the predicate position or in the object position if the predicate is rdf:type, are indicated in the dataset’s metadata, specifically using the void:vocabulary predicate. The RDF, RDFS and OWL vocabularies are not taken into account in this metric. This metric value can be computed as follows: U 5 ( D ) : = size ( vocabularies ( D ) ) size ( { ns ( t ) | t ∈ class ( D ) ∪ prop ( D ) } ) where vocabularies ( D ) is the set of vocabularies, identified by the object in a triple subject × void:vocabulary × object. The metric’s value represents the ratio of the defined vocabularies in the dataset’s VoID description vs. the actual vocabularies used in a dataset, identified by the unique namespaces of the terms.

Discussion: Similar to most of the contextual metrics, the box plot for this metric (U5) is, also positively skewed, suggesting that most of the population datasets have no indication of the vocabularies used. Despite having a median value is 0%, this metric has a number of outliers, amounting to around 11% of the population dataset. These outliers pushed the standard deviation to 10.62%, whilst the mean is 2.71%. This metric might also have values that are > 100%, especially when a lot of vocabularies are declared, but not all of them are used in the dataset’s triples. Nonetheless, we have not encountered this during our assessment.

From our assessment, around 2,800 different (not unique) vocabularies were used throughout the assessed dataset, whilst only 128 (around 4%) vocabularies were identified by the void:vocabulary predicate. Moreover, only 63 of those 128 defined vocabularies (around 63%, and around 2% of the total number of vocabularies used) were actually used in the dataset. This means that around 37% of the defined vocabularies were not used in their respective datasets. Using VoID generators as part of their publishing methods (mentioned in Metric P1), such issues can be easily rectified by the publishers.

5.4.6. Aggregated results

Figure 6 shows a box plot illustration of the aggregated quality value compared with the category’s metrics. The overall aggregated box plot shows a population with a median of 8.66%. The standard deviation is 13.84%, whilst the mean score is 13.04%. Five datasets from the whole population are “positive outliers” (since their overall quality value in this category is superior to rest of the population). These quality scores shed light on the real problems related to the contextual category. More worrying is the fact that provenance information is not given the same importance as other quality metrics. Data consumers might look at such provenance information to make informed decisions on whether to trust a particular dataset or data publisher prior to using a dataset. Lacking such information might make it hard for data consumers to re-use and adopt some dataset.

In [14], Bleiholder and Naumann define the conciseness metric as “measure(ing) the uniqueness of object representations”. Undoubtedly, from a database point of view, data redundancy causes a dataset to be large. This issue might not be that significant anymore because of large storage devices, or distributed storage. Moreover, data redundancy can be challenging in terms of data curation. For example, a data curator has to ensure that all “replicated” resources are updated accordingly. Nonetheless, data redundancy is not always a bad thing, for example, such redundancies can lead to improvements in query rewriting in Ontology-based Data Access, although it should be avoided if the publisher does not understand how to maximise its utility [59].

At the Linked Data level, a linked dataset is concise if there are no redundant instances [41]. By redundancy, Mendes et al. [41] mean that there are two local instances with different identifiers but with the same set of properties and corresponding data values. The extensional conciseness metric is classified under the conciseness dimension in [60].

Metric Computation: The extensional conciseness metric checks for redundant resources in the assessed dataset, and thus measures the number of unique instances found in the dataset. We follow the definition of Mendes et al. [41] in order to identify where two instances with a different URI are the same or not. In [19, §5.2], we showed that a naïve implementation of this metric leads to great time complexity, therefore we suggest the use of Bloom Filters [10] as an approximation technique. Using the Bloom filter for identifying possible duplicate instances during the assessment process, we quantify this metric as: CN 2 ( D ) : = 1.0 − size ( r b f ) size ( E ) r b f : = { r ∈ E ∣ isSet ( hash ( r ) ) } where hash is the hashing function used in the bloom filters, E is the set of entities in dataset D and isSet is the function that checks if the resource was seen or not before. r is a resource whose hash bits might have been set before, thus indicating a possible duplicate resource. This metric is discussed in more detail in our previous publication [19, §5.2].

Discussion: Our assessment estimated that overall the assessed datasets had a mean of around 7.6% redundant resources. Nevertheless, this does not mean that there is low redundancy on the whole Web of Data, since the standard deviation stands at 13.22% (median 99.34%), which suggests a moderately varied quality value overall. Around 13% of the datasets had a quality value less than the lower whisker, i.e. 78.55%. The range of quality values, including outliers, is 62.31%.

For this estimate value, we used 13 filters with a size of 5,500,000 each, ensuring efficient runtime with a low loss in precision (cf. [19, §6]). Around 76% of the datasets scored a value of 90% or more, meaning that the level of redundancy in these datasets is on the low side. Publishers should keep redundancy at a low level, and ensure that identical resources are not recurrent throughout the dataset. This can be done by creating owl:sameAs links between identical resources, without repeating property-value triples.

5.5.2. (CS1) Entities as members of disjoint classes

The Web Ontology Language (OWL) extends the RDFS expressivity by modelling primitives that are otherwise difficult to express in RDFS. Generally, the OWL axioms deal with restrictions that can be placed on an otherwise open world assumption. On the other hand, incorrect usage of OWL features results in inconsistencies and thus jeopardizes reasoning.

The owl:disjointWith property is used to “guarantee(s) that an individual that is a member of one class cannot simultaneously be an instance of a specified other class” [55, §5.3]. One of the most popular examples of disjoint classes can be found in the FOAF vocabulary, where foaf:Person and foaf:Document are defined disjoint, which means that the resource HarryPotter (as an example) cannot be both a person and a document.38

Metric Computation: Metric CS1 checks for disjointness between types in multi-typed resources. Moreover, each assessed type has its super-types inferred39

Discussion: The assessment shows that, in the presence of the simple inferencing explained above, almost all of the assessed datasets observe the owl:disjointWith property and their entities do not violate this property’s restriction. In total around 98% of the datasets score a value of 100% for this metric, whilst the other two datasets score a value of more than 99.9%, therefore still considered as of high quality. The mean quality value for this metric is 100%, whilst the standard sample deviation ( σ s ) is 0% (median is 100). Such low values in OWL inconsistencies were also reported in [31], were the authors attribute inconsistency problems caused by various incompatible exporters, such as FOAF exporters.

RDF Schema provides property-centric mechanisms for defining classes (rdfs:Class) and properties (rdf:Property) in vocabularies [16]. This means that:

“instead of defining a class in terms of the properties its instances may have, RDF Schema describes properties in terms of the classes of resource to which they apply.” – [16, §2]

The RDF data model is represented by a triple form (subject, predicate, object), where the predicate is expected to be a property that describes a resource in the subject position and its value in the object position. On the other hand, a class URI defining the type of a resource is usually in the object position when rdf:type is in the predicate position. The RDF data model is flexible allowing any resource URI to be in the predicate position. Therefore, whilst in OWL this practice is prohibited (unless OWL 2 punning is used), the data model does not prohibit publishers to have a defined class in the predicate position and a property in the object position, but this could cause problems when agents are interpreting the data. In this metric triples having the OWL axioms owl:equivalentProperty or owl:inverseOf are excluded from the assessment, as these require a property to be in the object position. This metric is classified under the consistency dimension in [60].

Metric Computation: The misplaced classes or properties metric assesses the datasets’ statements in order to check the correct usage of classes and properties. More specifically, this quality indicator checks if the assessed dataset has defined classes placed in the triple’s predicate and defined properties in the object position. We quantify this metric as follows: CS 2 ( D ) : = 1.0 − size ( c misp ‾ ) + size ( p misp ‾ ) size ( D ) c misp ‾ : = { c ∈ class ( D ) ∣ { ( c , rdf:type , P ) c misp ‾ : = ∣ P ∈ P } ∩ V ( ns ( c ) ) ≠ ∅ } p misp ‾ : = { p ∈ prop ( D ) ∣ { ( p , rdf:type , C ) ∣ c misp ‾ : = C ∈ C } ∩ V ( ns ( p ) ) ≠ ∅ }

In other terms, this metric checks for triples using a class c that is in the set of properties of the vocabulary of c (as defined in Metric IN3), which would mean that c is wrongly placed as a resource type, and similarly for properties p. A high value of this metric is interpreted as conformance to usage of classes and properties in a dataset.

Discussion: The usage of classes as properties and vice-versa are not common in the assessed datasets. Overall, 83% of the datasets score a value of 100% whilst the rest score 99.99%. The σ s value for this metric is 0.01% (median 100%), which shows a very low deviation, whilst the mean is 99.99%. Upon further inspection, we saw that no properties were used in the object position of an rdf:type triple, although classes such as http://creativecommons.org/ns#License were used infrequently (two instances in this case) as properties. Figure 7 shows the box plot for metric CS2.

We noticed that in the iteration of this metric we did not exclude triples with properties that are also expecting a property in the object position of the triple, such as rdfs:subPropertyOf and owl:onProperty. These might change the quality value of some datasets, however, it will not change the computation of the metric.

OPEN IN VIEWER

Fig. 7.

Intrinsic category box plot. Outliers are represented by dots.

OWL differentiates between properties referring to individuals (owl:ObjectProperty) and properties referring to data values (owl:DatatypeProperty). Incorrect usage of properties in this regard might lead to inapt functioning of an agent, for example, if a Linked Data viewer is using owl:ObjectProperty and owl:DatatypeProperty characteristics in order to hyperlink objects or not. Zaveri et al. [60] classify this metric under consistency.

Metric Computation: This quality indicator assesses a dataset’s statements for the correct usage of the predicate in terms the owl:DatatypeProperty and owl:ObjectProperty axioms. Therefore, this metric detects “erroneous” triples where a data value (literal) object is attached to an owl:ObjectProperty, and an entity (individual) to an owl:DatatypeProperty. Following this description, the metric can be formalised as follows: CS 3 ( D ) : = 1.0 − size ( { t ∣ misusedOWL ( t ) } ) size ( D ) misusedOWL ( t ) : = ( isLiteral ( t . o ) ∧ isOP ( t . p ) ) ∨ ( isIndividual ( t . o ) ∧ isDP ( t . p ) ) where isLiteral is a function that returns true if the assessed triple’s object is a literal (i.e. data value), isIndividual is a function that returns true if the assessed triple’s object is a URI or a blank node, isOP and isDP are functions that check if the assessed triple’s predicate has explicitly been defined by its vocabulary as an owl:ObjectProperty or owl:DatatypeProperty respectively (in analogy to checking for classes or properties as introduced in the context of Metric IN3). A high value of this metric indicates a low amount of misused properties.

Discussion: Fig. 7 shows the box plot for metric CS3. Similar to the previously discussed metrics for this dimension, the datasets adhere to a high quality score (mean 98.88%) and a considerably low standard deviation value of 5.17% (median 100%). Overall, around 87% of the datasets scored 100% whilst in total 95% of the datasets scored 90% or higher. Nonetheless, the box plot shows that around 12% of the assessed datasets are outliers. The dataset with the lowest quality value scored 52.60%. Comparing this against the study on Microdata formats on the Web in [42], it is confirmed that on average datasets on the LOD cloud misuse OWL datatype and object properties less.

From our assessment the following datatype properties (top five) were used with resources:

http://swrc.ontoware.org/ontology#series (28,269 times),

5.5.5. (CS4) Usage of deprecated classes or properties

Removing classes and properties from schemas renders data using them incoherent. OWL introduces the two classes owl:DeprecatedClass and owl:DeprecatedProperty for such situations. Any class or property that is declared an instance of these is no longer recommended to be used in published data. This metric is classified under the consistency dimension in [60].

Metric Computation: This metric checks whether deprecated terms are used in a dataset. More specifically, all used classes and properties are checked if they are members of owl:DeprecatedClass or owl:DeprecatedProperty respectively: CS 4 ( D ) : = 1.0 − size ( c dep ‾ ∪ p dep ‾ ) size ( class ( D ) ∪ prop ( D ) ) c dep ‾ : = { c ∣ ( c , rdf:type , owl:DeprecatedClass ) c dep ‾ : = ∈ V ( ns ( c ) ) } , p dep ‾ : = { p ∣ ( p , rdf:type , owl: p dep ‾ : = DeprecatedProperty ) ∈ V ( ns ( p ) ) } .

Discussion: With around 97% of the datasets scoring a quality value of 100%, data publishers tend to avoid using deprecated classes and properties. The LOD Cloud sample that was assessed used the minimal deprecated terms in most cases, with the lowest quality score of 97.41% marked as an outlier in the box plot (CS4) in Fig. 7. The standard deviation, as in the other consistency metrics, is very low (0.23%), and the mean value is 99.97%.

5.5.6. (CS5) Valid usage of the inverse functional property

In the real world, a public key used for encryption is unique to every individual. If we want to represent this public key in a Linked Data document, then there should be one exactly one resource (possibly an individual of the type foaf:Agent) describing this public key, in order to represent this uniqueness between the key and the individual. Such properties are termed as inverse functional, meaning that if two different resources share the same value for that property, during reasoning or smushing40

Metric Computation: This quality indicator checks for incoherent values within the assessed dataset’s values. More specifically, this metric checks if a value attached to a property member of owl:InverseFunctionalProperty (IFP) is shared by two or more different resources. With regard to the definition of different resources, this metric follows the local unique name assumption. In this metric, we only consider those statements that have an inverse functional property. We quantify this metric as follows: CS 5 ( D ) : = 1.0 − size ( v IFP ‾ ) size ( p IFP ‾ ) v IFP ‾ : = { t ∣ ifp ( t . p ) ∧ ∃ t ‾ ∈ D . φ ( t , t ‾ ) } p IFP ‾ : = { t ∣ ifp ( t . p ) } φ ( t , t ¯ ) : = t . s ≠ t ‾ . s ∧ t . p = t ‾ . p ∧ t . o = t ‾ . o where ifp ( t . p ) checks whether the predicate of a triple is a member of owl:InverseFunctionalProperty, φ ( t , t ¯ ) returns true if the two triples t and t ¯ violate the IFP characteristic. This definition does not work in the absence of inverse functional properties, as we would then divide by zero. In this case, we define CS 5 ( D ) : = 1 .

Discussion: The box plot for this metric (CS5) in Fig. 7 shows the trend in this metric where a large part of the assessed datasets have no varying quality, bar a few number of datasets that are considered as outliers. These outliers, around 18% of the assessed datasets, increased the standard deviation to 12.29%, whilst the calculated median is 100%.

One should keep in mind that not all assessed datasets made use of inverse functional properties and were thus given a 100% score (since there was no triple breaking the IFP constraint); nevertheless, these where included in the assessment. From the assessment, around 3% of the datasets got a quality score of less than 50%.

Triples with the following IFP properties (top 5) where identified with violations during our assessment:

http://xmlns.com/foaf/0.1/homepage (violated in 2861 triples)

Hogan et al. defined ontology hijacking as the “re-definition or extension of a definition of a legacy concept […] in a non-authoritative source” [32]. s being an authoritative source for concept c means that the namespace of c coincides with that of s. For example, http://xmlns.com/foaf/0.1/ is the authoritative source for the concept foaf:Person. Ontology hijacking may lead to incorrect inferencing throughout the data [32]. Nevertheless, ontology hijacking can be seen as restricting the Linked Data principle of an open world assumption, in a sense that discouraging ontology hijacking restricts what one can say about some concept. Zaveri et al. [60] classify this metric under consistency.

Metric Computation: This metric assesses a dataset for its redefinition of third party external classes and properties. More specifically, this metric identifies if a dataset is the authoritative document for all classes and properties it defines, following the axioms identified in [32].

Along these lines, we quantify the metric as follows: CS 6 ( D ) : = 1.0 − size ( { t ∈ tdef ( D ) ∣ H ( t ) } ) size ( tdef ( D ) ) where tdef ( D ) is the set of triples in dataset D having one of predicates or objects for which hijacking rules have been defined (i.e. all triples in dataset D that can potentially trigger a rule violation), and the H ( t ) function checks whether triple t violates its corresponding hijacking rule. Therefore, this metric calculates the ratio between the violating hijacking triples, and all triples that can potentially trigger a hijacking violation.

Discussion: Similar to the Metric CS5, the variation in quality within most of the assessed datasets (≈86% of the datasets) is very low, though a number of outliers (shown in Fig. 7 Metric CS6) causes a standard deviation of around 19.99% (median is 100%). Furthermore, the mean value is 93.64%. Overall, publishers tend to avoid redefining terms that they are not authoritative to do so, with around 85% scoring a quality value of 100%. In general, publishers should try to avoid redefining terms, but instead they should extend existing terms (if needed), thus avoiding the confusion that can be caused by term cross-definition.

5.5.8. (CS9) Usage of incorrect domain or range types

In a schema, a property can optionally have domain and range defined. The domain is the expected type (class) of the subject of a triple using the given property. The range is the expected type (class of a resource, or datatype of a literal) of the object of such a triple. Using the incorrect domain or range types makes the data incoherent, as consumers who know the underlying schemas could query the data without looking at it, making it harder to retrieve the right or all results. Zaveri et al. [60] classify this metric under consistency.

Metric Computation: This metric assesses a dataset for the compliance of the types of the subjects and objects of its statements with the domains and ranges of its predicates according to the schema of the respective predicate. In particular, the predicate of each triple is dereferenced to identify the expected domain and range types, for the subject and object respectively. Furthermore, the subject’s type should not be disjoint with any type identified as the domain of the predicate in the triple. Similarly, this applies to the object’s type with regard to the predicate’s range. At the typical scale of Linked Open Datasets it is, however, prohibitively expensive to infer all types of a resource and all disjointnesses between classes. For these reasons, we define this metric to efficiently compute the practically most relevant sub-case: checking whether the type of the subject/object of a triple or any of its superclasses matches the domain/range. For each triple being assessed, hierarchical inferencing is done on the subject and object in order to determine the parents of their.41

This is required as we might have predicates with an abstract domain or range; for example, the foaf:mbox predicate has foaf:Agent as its domain. Therefore, if we assume a triple :j foaf:mbox <...> and :j is typed as foaf:Person, then we need to know that foaf:Person is a subclass of foaf:Agent, the domain type required for the foaf:mbox predicate. We define this metric as follows: CS 9 ( D ) : = 1.0 − size ( dom ‾ ( D ) ) + size ( ran ‾ ( D ) ) 2 × size ( D ) dom ‾ ( D ) : = { t ∈ D ∣ dom ( t . p ) ∈ T ( t . s ) } ran ‾ ( D ) : = { t ∈ D ∣ ran ( t . p ) ∈ T ( t . o ) } where T ( r ) is a function that returns the type of the local resource42

Discussion: This metric is implemented as a probabilistic metric using reservoir sampling as explained in [19]. Our assessment shows that data publishers tend to use incorrect domain and range types in the triples. Around 4% of the assessed datasets had a quality score of 90% or more, with the highest score being 99.51%. On the other hand, around 13% of the datasets scored less than 50%. The mean score for this metric is 60.11% whilst the standard deviation is around 13.43%. The box plot for Metric CS9 in Fig. 7 is symmetrical with the median standing at 57.14%. It also depicts a set of outliers over the top whisker and one dataset marked as outlier under the bottom whisker. It is also lower than the rest of the consistency metrics (Metrics CS1 to CS6), suggesting that Linked Data publishers might be more complacent with using the right datatypes when creating resource triples. Linked Data publishers should be aware of the domain and ranges of the properties used in their datasets by consulting with the relevant vocabularies. Furthermore, simple on-the-fly type checking scripts can be created and used throughout the publishing activities, inspecting for such schema-to-data inconsistencies.

Linked datasets often mix terms from different vocabularies to describe its resources, which might lead to predicates being used incorrectly. For example, assigning a foaf:Person to a dc:creator would lead to a violation. Nonetheless, such assignments to “generic”43

Ranges with a data value (i.e. literal) are usually constrained to be of a certain datatype, for example, a property ex:age would have an xsd:integer. Being an important component in the RDF data model, literals are used to represent values for properties, whilst the datatype attached to the value can be used to interpret the value concisely. Beek et al. describe four benefits of having good quality literals including efficient computation [8]. This means that having a canonical representation of the datatype ensures a unique representation of a literal across the Web of Data, and thus actions such as comparing two literals of the same type would be easy [8]. It is recommended that publishers add explicit datatype information to literals. This metric is classified under the syntactic validity dimension [60].

Metric Computation: This quality indicator assesses the lexical form of the data values against the data type attached with the literal itself. Consider “10”^^xsd:integer, the value 10 is what is known as the lexical form, whilst xsd:integer (i.e. http://www.w3.org/2001/XMLSchema#integer) is its datatype. Along these lines we quantify this metric as follows: SV 3 ( D ) : = size ( { v ∈ lit t ( D ) ∣ ϑ ( v lf , v dt ) } ) size ( lit t ( D ) ) where lit t ( D ) is the set of all typed literals, ϑ ( v lf , v dt ) is a function that checks the validity of the value’s lexical form v lf against the value’s datatype v dt .44

Discussion: The box plot for metric SV3 in Fig. 7 shows that most of the datasets assessed adhere to a 100% quality value, though there were also a number of datasets that scored less and thus are marked as outliers. On average, the quality score of the assessed dataset is around 96.80% whilst the standard deviation is a high 14.16% (median 100%). Datasets that had no literal values were omitted from this assessment.

Similar to Meusel’s and Paulheim’s findings [42], our assessment identified xsd:date as the most problematic datatype, present in around 37.24% of the problematic triples. A literal typed with the xsd:date datatype should have the ISO 8601 lexical form of CCYY-MM-DD, where CC represents the century, YY year, MM month, and DD day [12]. Nonetheless, we found errors that can be grouped as follows:

literals with a lexical form of xsd:dateTime;

In order to reduce incompatible datatypes vis-à-vis the lexical form of a data value, publishers could publish and serialise their data using the latest Turtle 1.145

Figure 7 shows a box plot illustration of the aggregated quality value compared with the category’s metrics. The overall aggregated box plot shows a population with little dispersion (most of which results from Metric CS9) having a standard deviation of 12.89%, a median of 80.94% and mean of 77.36%. The majority of the metrics shows that a relatively high quality is adhered to by Linked Data publishers.

Dereferenceability is one of the main principles of Linked Data. HTTP URIs should be dereferenceable, i.e., HTTP clients should be able to retrieve the resources identified by the URI. According to the LOD principles, a typical web URI resource would return a 200 OK code indicating that a request is successful and a 4xx or 5xx code if the request is unsuccessful. In Linked Data, a successful request should return an RDF document containing triples that describe the requested resource. Resources should either be hash URIs or respond with a 303 See Other (redirect) code [52].

Metric Computation: The aim of this metric is to check the number of valid dereferenceable URIs used (according to these LOD principles) in a data source. More specifically, an HTTP GET request is performed on a URI defining a concept, together with a header accepting a variety of MIME types valid for Linked Data (including application/rdf+xml, text/n3, text/turtle, etc.). Unless a resource’s URI is a hash URI, a correct server-side dereferencing mechanism should identify that the requested resource is a real-world object or abstract concept (a “non-information resource”) and thus reply with a 303 See Other and a redirect location where the data object in the desired format is (an “information resource”). Heath and Bizer explain that “where URIs identify real-world objects, it is essential to not confuse the objects themselves with the Web documents that describe them” [29, §2.3.1].

This 303 redirection is handled automatically by the client, with the server responding with a 200 OK together with the semantically described object in the requested format. This metric checks all local and non-local URIs for dereferencability. Along these lines we adapt the metric from Hogan et al. [33, §5.1, Issue III]: A 3 ( D ) : = size ( { u ∈ dlc ( D ) ∩ U ∣ deref ( u ) } ) size ( dlc ( D ) ∩ U ) where deref ( u ) is a function that returns true if the URI u being examined follows the dereferencability rules.

Discussion: In [19, §5.1] we describe a probabilistic technique for this metric using reservoir sampling. In this technique, each resource URI is split into two parts: (1) the pay-level domain (PLD), and (2) the path to the resource. This is analogous to a dictionary data structure. For this metric we employ a “global” reservoir sampler for the PLDs. Furthermore, for each PLD we employ another sampler holding an evenly distributed sample list of resources to be dereferenced. However, such sampling might lead to an unbalanced representative sample. We therefore adopt a hybrid of the reservoir technique used in [19, §5.1] and the stratified sampling idea as described in [27]. Stratified sampling is a technique that can be used when the data can be partitioned into a number of disjoint subgroups [27]. The idea is that the sample is chosen per proportion of these subgroups, therefore improving the representative sample. Therefore, for each domain in the higher-level sampler, we keep track of the total number of items encountered during assessment, to simulate the size of the strata groups. A final sampler is then drawn from all lower-level sampled resources using the proportionate allocation method. The parameters used were 5,000 as the global reservoir size (i.e. the number of possible different pay-level domains (PLD) in a dataset), and a PLD size of 10,000. Nevertheless, one must keep in mind that these parameters introduce a bias in our results in a way that the sample might be under-represented.

OPEN IN VIEWER

Fig. 8.

Accessibility category box plot. Outliers are represented by dots. Machine-Readable License, Human-Readable License, and Links to External Data Providers metrics are excluded from the plot, but their values are included in the aggregated result box plot.

The box plot for metric A3 in Fig. 8 shows a large variance with values ranging 100% to 0%. The mean quality value of this metric is 36.86%, which is 33.44% lower than the mean recorded in [33, §5.1 – Issue III]. There are two reasons for this difference. First, in our study we do not just study local dereferenceable URIs, but also take into consideration the dereferenceability of external resources the publishers use. This might mean that publishers are punished if the external URIs used are not dereferenceable. Secondly, we noticed that certain hosts blacklisted our IP address during this assessment following numerous HTTP requests. The box plot for metric A3 in Fig. 8 is positively skewed, meaning that the assessment shows a high concentration of low quality values. Similar to [33, §5.1 – Issue III], our assessment shows a high variability between data producers on the dereferencability of resources. We report a standard deviation of 36.54%, with a median of 31.11%. In total our assessment attempted to dereference a total of 709,356 resources, out of which only 233,127 where valid dereferenceable resources. The rest of the resources resulted in the following problems:

Surprisingly, not a lot of publishers abide by the dereferenceability guideline. Our assessment shows that only 33% of the assessed datasets have a dereferenceability value of 50% or more. Whilst this guideline is one of the Linked Data principles, one should understand the extra costs this mechanism requires, including the maintenance of content-negotiation and re-direction schemes. However, one must investigate if the need of the dereferenceability mechanism is a must in Linked Data, or if agents can be adapted to understand Linked Data URIs automatically, for example by introducing catalog mechanisms similar to those used in XML to resolve URI references. In the meantime, a possible solution is that data publishers make use of Linked Data-based content management systems (such as OntoWiki48

Open licences, as defined by the Open Definition [45], are the heart of open data. They specify whether third parties can re-use or otherwise, and to what extent. In Linked Open Data, one would expect that such licences are either machine-readable using predicates such as dct:license, dct:rights and cc:licence, or at most human-readable (e.g. within dc:description). Such a license specification should be included in a dataset’s metadata.

5.6.3. (L1) Machine-readable license

Having machine-readable license definitions (such as those in the http://purl.org/NET/rdflicense dataset [51]), agents would be able to consume (for example to visualise) different parts of the license, such as the jurisdiction and duties (e.g. share-alike or attribution). Furthermore, agents would be able to understand the limitations of a license, and make informed decisions (e.g., if resources can be used within paid services) with less human interaction.

Metric Computation: The aim of this metric is to check if a dataset has a valid machine-readable license. By valid we mean that a license can be retrieved from a semantic resource (e.g. http://purl.org/NET/rdflicense/.*) or with an owl:sameAs link to one of the following URLs:

http://(www.)?opendatacommons.org/licenses/odbl.*

Discussion: In Section 3.2 we discuss the licences and rights in the LOD Cloud datasets’ metadata. We show that around 41% of the whole LOD Cloud datasets have license or rights metadata, using the predicates dct:license, and dct:rights. In this metric we assessed the acquired data dumps and SPARQL endpoints for machine-readable licenses. The quality assessment resulted in just 17 datasets (≈13%) that contained at least one machine-readable license. 14 out of the 17 datasets identified during this assessment had licence metadata in the LOD Cloud metadata. The other three datasets had licence information directly in the data itself. Machine-readable license statements can be easily included in a dataset by using other linked open datasets such as [51].

5.6.4. (L2) Human-readable license

In contrast to Metric L1, a human-readable license enables human agents to read and understand a license in textual format, rather than in terms of triple statements.

Metric Computation: The aim of this metric is to verify whether a human-readable license text, stating the licensing model attributed to the dataset, has been provided as part of the dataset itself. The difference from Metric L1 is that this metric looks for objects containing literal values and analyses the text searching licensing related terms. More specifically, we check for the following:

A license description triple, identified by a predicate dct:description, rdfs:comment, rdfs:label, or schema:description and a literal object matching the following regular expression: .*(licensed?|copyrighte?d?).*(under|grante?d?|rights?).*;

Discussion: Similar to Metric L1, the assessment shows a low overall level of conformance to this metric. We detected human-readable licenses in eleven (≈8.46%) datasets, four of which also had a machine-readable license. Whilst it is understandable that publishers are less inclined to having statements with large textual literals containing licensing data, we suggest that publishers should at least define the license name in the datasets’ metadata. Licenses are of utmost importance to open data [45, §1], therefore, publishers should define the license or rights either as machine-readable (preferable) or at least human-readable.

5.6.5. (I1) Links to external linked data providers

One of the main Linked Data principles is to “include links to other URIs, so that [agents] can discover more things.” [11]. Furthermore, Berners-Lee states that linking your data to external sources would earn the dataset the fifth star, given that the rest of the four guidelines are satisfied. Having external links in a dataset would enable data consumers to explore and understand better the data in question. Additionally, Heath and Bizer [29] emphasize the importance of external RDF links in the Web of Data since:

“they are the glue that connects data islands into a global, interconnected data space and as they enable applications to discover additional data sources in a follow-your-nose fashion.” – [29, §2.5]

Metric Computation: The aim of this metric is to identify the total number of external RDF links used within the assessed dataset. An external link is identified if the object’s resource URI in a triple has a PLD different from the assessed dataset’s PLD. Furthermore, the external link should be a semantic resource that can be dereferenced and parsed by an RDF parser. For this metric we use a reservoir sampling approach [19] to estimate the number of external Linked Data providers in a dataset.49

Discussion: Similar to Metric A3, this metric was assessed using a sampling technique. In this case, each external PLD has a sampler of maximum 25 items. Estimation techniques create a bias since the parameters might create an under-represented sample. In this case, we might miss out possible Linked Data documents with an external PLD. Furthermore, datasets using non-dereferenceable external PLDs will be punished by this metric. Dereferenceability is essential in this metric in order to abide by the follow-your-nose principle described by Heath and Bizer in [29, §2.5]. Table 7 shows the top five assessed datasets, the number of unique dereferenceable external PLDs linked in the dataset, and the total number of unique PLDs. In the sample acquired from the LOD Cloud, only 9 datasets had no external PLDs, whilst around 88% of the datasets had less than 50 unique external PLDs linked. In total, the number of external PLDs extracted from the assessed 3.7 billion triples amounted to 977,609. Three datasets, namely dbpedia.org , kent.zpr.fer.hr, and www.pokepedia.fr accounted for around 97% of these PLDs. However, the actual number of PLDs with dereferenceable resources is 3,086, which is around 0.31% of the linked external PLDs.

Considering the Linked Data principles, one would have expected a higher ratio of external RDF links. However, there is no set number of external Linked Data PLDs each dataset should have. The assessed datasets feature a large standard deviation of 183.3 Linked Data PLDs, and a mean of 27.01 Linked Data PLDs. Nevertheless, one should consider that these two statistical measures are highly influenced by the top two datasets.

Ideally, a Linked Data host can accommodate a large number of requests without affecting the consumers’ productivity. That is, a consumer is not left waiting “in a queue” until other agents have been served. Therefore, in an ideal situation, a host has the capacity to handle a large number of parallel requests.

Metric Computation: Adapting a definition by Flemming [23], the high throughput metric measures the efficiency with which a system can bind to the data source by measuring the number of HTTP requests answered by the source of the dataset per second. From the dataset we use reservoir sampling to “randomly” choose a maximum of ten local resources (i.e. whose namespace is the same as the data source namespace) that will be used for this metric. The metric estimates the number of requests served per second, computed as the ratio to the total number of requests sent to the dataset’s host. We define this metric adopting [23] as follows: PE 2 ( D ) : = 1.0 ⩾ 5 requests answered in ⩽ 1 s servedRequestsPerSec 200 ms otherwise where servedRequestsPerSec is number of requests that the host served per second. If five or more requests can be answered in a second or less, then the metric’s value is defined as 100%, otherwise a percentage is calculated as the ratio of the number of served requests against the ideal time (200 ms) taken to serve one request.

Discussion: The box plot for this metric (PE2) in Fig. 8 shows a large varying quality with values ranging from 100% to 0%. The standard deviation stands around 45.60% (median value is 29.67%) whilst the mean value is 47.78%. The box plot is positively skewed, suggesting that observations at the low end are concentrated. Around 38% of the assessed datasets gave a result of 100%, which means that more than five requests were answered in 1 second or less. Around 8.52% of the datasets scored a quality value between 50% (inclusive) and 100% (not inclusive). All quality results are dependent on the data host during the time of the assessment, therefore, such a quality assessment should be performed more frequently.

5.6.7. (PE3) Low latency

Latency is the amount of time an agent has to wait until the host responds with the particular request. The time taken largely depends also on how big the HTTP request is, and the number of HTTP round-trips the server has to make before serving the request. Therefore, the choice of Hash URIs and 303 redirects (i.e. Slash URIs) is also an important factor for latency [23,29]. Hash URIs would reduce the number of HTTP round-trips, as the document with the requested fragment resource description would contain descriptions of other resources in the same document. Therefore, the client would end up receiving unnecessary resources that would eventually increase the latency (since the document size will be larger). On the other hand, Slash URIs require the client to go through the whole dereferencing process for every resource, but then the client will receive exactly the required resource. Ideally, the data source should serve resource requests with the lowest possible latency, which in turn means that data publishers should choose the right strategy for publishing data (hash URIs vs. slash URIs).

Metric Computation: The low latency metric measures the efficiency with which a system can bind to the data source by measuring the delay between submitting a request for that very data source and receiving the respective response. Similar to Metric PE2, a reservoir sampler is used to sample a maximum of ten local resources from the dataset under assessment. This metric is defined as the mean time taken for ten requests to respond, normalised to a percentage value between 0 and 100 by dividing by an ideal response time defined as one second [23]. Along these lines, Metric PE3 is defined as follows: PE 3 ( D ) : = 1.0 ⩾ 1 requests answered in ⩽ 1 s 1 , 000 ms meanResponseTime otherwise where meanResponseTime is the mean response time of the 10 sampled resources. A 100% low latency means that the data source can respond to a resource within up to a second, otherwise, the percentage value is calculated as a ratio of the number of possible requests served in one second.

Discussion: Similar to Metric PE2, results of these metrics rely on the data host at time of assessment. The box plot for this metric (PE3) in Fig. 8 confirms the large range varying quality, as in Metric PE2. The standard deviation is 47.12% with a mean value of 57.55%. However, unlike PE2, the metric’s values are negatively skewed, with a median value of 99.23%. This shows that there is a large concentration of very high quality values. Around 49.61% of the datasets have a quality value of 100%, meaning that at least one request is answered in one second or less.

5.6.8. Aggregated results

Figure 8 shows a box plot illustration of the aggregated quality value compared with the category’s metrics. The overall aggregated box plot shows a population that is moderately varied having a standard deviation of 19.00% and a median of 29.96%. The mean aggregated quality score is 33.12%, with only 19% of the assessed datasets scoring 50% or more. The aggregated value is affected by the low values of the licenses metrics (L1 and L2), which is a concerning matter considering that the assessed datasets are part of the Linked Open Data cloud. Not having a defined license might make the adoption of linked dataset more difficult.

Most of the “failing” datasets are SPARQL endpoints, whilst others contained syntactic errors. In Luzzu, quality metrics are not written and executed on SPARQL endpoints, but instead triples are streamed from the endpoint51