Comparison and evaluation of ontologies for units of measurement

2.1. Ontologies for measurement units

Over the last years, several projects were initiated to create ontologies modeling the domain of measurement units. The following selection focuses on ontologies that model a reasonable number of relevant individuals and classes in an OWL compatible format. There are, of course, several other unit ontologies within and beyond the OWL world, but they do not meet this basic condition. In addition, the selection contains an OWL compatible knowledge base which provides data on measurement units.

The selection evaluated in this work includes:

We are aware of the current efforts towards a second version of QUDT. At the time of writing, however, the ontology has only been partly published. The available parts do not allow a meaningful comparison. Thus, the analysis of QUDT 2 is left open for future work.

DBpedia extracts structured data from Wikipedia11

On the other hand, Wikidata – a sister project to Wikipedia – aims at collecting factual knowledge for Wikipedia in a central repository [26]. This information can then be linked directly from different Wikipedia articles and thus provide a consistent view even across several language versions. Wikidata also includes a large number of measurement units and their relations. As this data can also be accessed through a SPARQL endpoint, WD is included in the selection.

In the following “OM” will refer to both OM 1 and OM 2, when a statement applies to both ontologies alike. If only one ontology is affected, we will use “OM 1” or “OM 2” respectively.

The selected ontologies use different terms to refer to the same concepts. For example, the concept “kind of quantity” is denoted as “physical quality” (MUO), “quantity kind” (QU, QUDT), and “physical quantity” (WD). OBOE, OM, SWEET, and UO do not provide an explicit entity for this concept, but model the respective entities as subclasses of “physical characteristic” (OBOE), “quantity” OM, “property” (SWEET), and “quality” (UO). Therefore, a clear specification of the terminology used in this survey is needed. This is provided in Appendix. Where possible, it is compliant with the International Vocabulary of Metrology [10].

Motivated by the lack of published evaluation processes as well as examination results for well known ontologies, an early evaluation approach was developed by Gómez Pérez [6]. This approach consists of two steps: In a first analysis step, the ontology is inspected regarding consistency, completeness, conciseness, expandability, and sensitiveness. Consistency is defined as the absence of contradictions between formal definitions and the modeled world, contradictions between informal definitions and the modeled world, contradictions between formal and informal definitions, and contradictions between formal definitions (“inferential consistency”). Completeness is informally defined as “each definition is complete” and “all that is supposed to be in the ontology” is included. Conciseness is the absence of redundancies and “unnecessary or useless definitions”. Expandability is an assessment of the effort to add new definitions. Sensitiveness describes the impact of small changes to the whole ontology. In a second synthesis step the ontology is then corrected.

This approach was applied to the Standard Unit Ontology (SU), which contains 61 individuals and one class, by two ontology engineers and two domain experts. They used a checklist of eight possible errors in taxonomies to decide on its consistency, completeness and conciseness. As a result, Gómez Pérez concluded that SU is consistent and concise, but incomplete. Further, they identified seven problems by inspecting the ontology’s source code. These errors were related to completeness, conciseness, expandability, and missing code conventions. However, the theoretical criteria were not consistently applied in the evaluation process. The choice of SU indicates a particular need for evaluation and improvement in the domain of measurement units.

Based on Gómez Pérez’s criteria, Rijgersberg et al. evaluated five ontologies concerned with units of measurement [17]. They used a checklist of 34 requirements related to “completeness of the modeled scope”, “quality of formal definitions”, “understandability and extensibility”, and “completeness in the natural language documentation”. The results indicated that none of the ontologies contained all important concepts of this domain.

Consequently, they developed OM 1. Subsequently, Rijgersberg et al. published a comparison of OM 1 with QUDT 1.0.0 [16]. They compared the schemata of both ontologies and outlined the differences in the modeling. Further, they compared the number of kinds of quantities and measurement units as well as the support of five use cases. Overall, the results indicated that OM 1 outperformed QUDT 1.0.0.

Another domain specific comparison approach was introduced by Marcus P. Foster. He suggested a ranking of measurement ontologies in five levels regarding the supported concepts and use cases [4]. Ontologies of the first level “standardize characters for unit symbols and operators”. On the second level they “additionally represent the concepts of unit and dimension” and “support the operations of unit conversion and unit conformance”. On the third level they “additionally include the concept of quantity and system of quantities”. On the fourth level they additionally provide a “full descriptions of quantities, prefixes, ordinal and nominal measurement scales, and systems of quantities and units”. On the fifth level they “also incorporate the concepts (precision, accuracy, distribution etc.), relations and operations associated with calculating and representing measurement uncertainty”. QUDT was classified as a third level and OM 1 as a fourth level ontology.

This ranking provided a fast overview regarding scope and level of development of ontologies. However, the order of requirements for each ranking-level seems biased by the authors background in parts. For example, an ontology might provide concepts of quantity and system of quantities, as required by the third level, without supporting unit conversion, as required by the second level.

Chau Do and Eric J. Pauwels applied MathML to map unit ontologies [3]. They manually created a predefined mapping of the seven SI base units. Then the definitions of unit composition and unit conversion were used to map the remaining measurement units. This approach was applied to OM 1, QUDT, and SWEET. During the evaluation of their mapping approach, they discovered several problems caused by “incorrect information in the ontologies”. Some examples were provided for each ontology. However, as the work was not focused on ontology evaluation, they did not continue to systematically search for errors. Unfortunately, not all ontologies in our selection provide definitions of unit composition and unit conversion. Therefore, this mapping approach is not applicable in our analysis.

Samadian et al. compared the Ontology for Engineering Mathematics (EngMath), MUO, OM, and QUDT [18]. Tailored to the specific requirements of a medical data integration project, the comparison was focused on the conversion of measurement units, the modeling of their mathematical relations, and the extensibility of the ontology. OM was the most suitable ontology for their purpose. But, due to the narrow focus, the benefit for other projects is limited.

In [23] a comprehensive list of 16 use cases in the field of unit ontologies was presented. For each use case, the required concepts and relations were identified and categorized into necessary and optional requirements. While necessary requirements are mandatory for a use case, optional requirements “simplify the implementation of a use case or increase its usefulness”. A metric was developed to measure the suitability of a given ontology with respect to those use cases. Finally, a selection of seven ontologies (MUO, OBOE, OM 1, QU, QUDT, SWEET, and UO) was analyzed and compared using this metric. OM 1 surpassed the other ontologies in each use case with the closest overall contenders being QUDT, QU, and SWEET. This evaluation, however, remained on a conceptual level. The individuals of the ontologies have not been considered.

Zhang et al. tried to compare the four unit ontologies OM 1, QUDT, QUDV, and UO [27]. They focused on the OWL standard vocabulary used for modeling and possible reuse of the ontologies. However, their approach lacks a general methodology, which could be extended to include other ontologies in the comparison. Furthermore, the used OWL language features in an ontology do not enable any conclusions regarding its usefulness in other projects.

A domain independent approach of ontology comparison is the usage of general quality criteria as implemented by, e.g., OntoClean [7] and the OntOlogy Pitfall Scanner! (OOPS!) [14]. OntoClean provides a “methodology for validating the ontological adequacy and logical consistency” and “is based on highly general ontological notions” like identity and unity. It can be used to check for common pitfalls related to taxonomic relationships. On the other hand, OOPS! (semi)-automatically checks for many common pitfalls in OWL ontology development. For instance, this includes unconnected ontology elements or missing human readable annotations. It is easy to use and can give an initial impression of the quality. However, a detailed analysis of domain specific issues is out of the scope for both methods.

Although the comparison in [16] includes some statistics, all mentioned evaluation efforts are manual assessments almost exclusively remaining on a conceptual level or only cover general quality issues. Thus, there is a lack of evaluation concerning the individuals of unit ontologies.

Despite an extensive search, we are not aware of a comprehensive reference dataset in the field of measurement units. There are some datasets [8,19,24,25], but those either cover only a subset of the units in use or omit some of the major concepts as listed in Appendix. Further, some of these datasets have been used to create some of the selected ontologies. Therefore, the results would likely be biased in favor of these ontologies.

For a suitable replacement, we turned to the information contained within the selected ontologies. We extracted relevant entities from the ontologies into an ontology-agnostic relational model (cf. Fig. 1) and subsequently mapped them semi-automatically to one another. This resulted in a reference dataset for the further analysis.

OPEN IN VIEWER

Fig. 1.

Ontology-agnostic model of the created reference dataset.

The created dataset can be used to detect deviations between the ontologies. In most cases, these deviations can be backtracked to issues within at least one ontology.13

OPEN IN VIEWER

Fig. 2.

Schematic of evaluation approach and the respective results.

Following the previous discussion, we decided against a strictly manual assessment and for a semi-automatic approach as shown in Fig. 2. To apply it to the selected ontologies we created an implementation in the form of a collection of scripts. It is publicly available for free use and extension [20]. To easily be able to run just a subset of scripts, but yet preserve dependencies, each script will store its results into a set of JSON files, which will then be used by its successors.

In the following, we will describe each step in detail. The intermediate results of each step are available at [21]. Note that the received results were not only used to assess the given ontologies, but also to improve upon our analysis process. The results of the first analysis phase, e.g., allow to verify the extraction process and hint towards erroneous queries, which can then be corrected.

4.1. Statistics

For an initial overview the supported concepts and relations in the ontologies as well as the respective number of individuals are given in Table 1. Also included are the number of non-prefixed and prefixed units, which incorporate the results of the aforementioned heuristic to identify prefixed units.

The first notable observation is the absence of a connection between prefixed units and prefixes in MUO, QUDT and UO, even though both exist. First of all, despite containing both prefixed units and prefixes MUO, QUDT and UO do not model the connection between both. This hinders attempts to retrieve multiples or submultiples of a given unit. Furthermore, individuals for the concepts of field of application, dimension and system of units are just included in small subset of the ontologies.

OM is the only ontology with two versions present in this analysis. Most improvements between both OM versions are concerned with the simplification of the ontology structure, which is outside of the scope of this paper, but is documented in the SPARQL queries provided [20]. An important observation in the comparison of both versions is the absence of almost 40% of the individuals for kinds of quantities and systems of units in OM 2. This also has a major impact on the number of app ⋈ qk and qk ⋈ unit relations. Remarkably, the numbers for system ⋈ unit remain almost the same. This suggests, that the removed individuals for systems of units were poorly connected anyways. In contrast to the absence of many kinds of quantities and systems of units, there is a considerable increase of app ⋈ unit relations, whose number has more than doubled.

The number of measurement units in WD exceeds the number of measurement units in the other ontologies by far. Manual inspection reveals a large number of historic measurement units to be present in WD, which might explain this large number. Another reason are possible duplicates that were not recognized due to the reasons mentioned in Section 3.2.3. The second place is taken by OM, which does not suffer from these limitations. If the prefixed units are discarded, however, the situation changes in favor of QUDT. A closer look at the actual individuals in the ontologies suggests that a systematic way has been used to create the prefixed pendants for all SI units in OM, which vastly increased the total number of units.

In the next step, the individuals for each concept were mapped to one another. Table 2 compares the number of individuals present in the ontologies and the number of mapped distinct units overall. The number of found mappings especially for measurement units is surprisingly low. To eliminate shortcomings of the mapping process itself the list of mapped units was carefully examined by hand, which led to a number of mappings added manually, but to no substantial overall change. Due to the wide range of languages we were not able to perform this manual duplicate search for WD to the same extent as for other ontologies. Therefore in some of the further analysis the values reported for WD are not as reliable as the ones reported for the other ontologies.

Following up on differences in the number of units included and the lack of mappings, we analyzed the frequency in which units appear in the different ontologies. Table 3 lists the number of prefixed and non-prefixed units which are present in a certain number of ontologies. The actual number of widely used measurement units was less than anticipated as over 85%19

Also notable is the fact that there seems to be little consensus which units are essential to an ontology. Only 17 units are present in all ontologies.20

To explore this even more, the overlap between the ontologies with respect to the included units is given in Table 4. It displays the fraction of both prefixed and non-prefixed units from one ontology, that are included in another.21

As can be seen, for any two given ontologies less than three quarters of the units of one ontology are part of the other. Even the smaller ontologies with respect to the number of units include units, that are not covered by the larger ones. There are different possible reasons for this fact. The ontologies were created for different projects or domains and hence needed to meet different demands regarding the included units. This is especially noticeable in the different sets of compound units included.22

WD provides a wealth of languages for its labels – totaling over 300 different language tags including multiple regions for some of the languages. Following this observation we were interested in the frequency of each individual language. An excerpt of this is given in Table 5. Note that just the five most frequent languages for each type are shown for WD, while all present languages are listed for the other ontologies. Most ontologies choose to only add English labels. When the language tags were missing, the label turned out to be in English in all cases examined (QU, QUDT, UO). The only exception for domain ontologies is OM which also added Dutch and Chinese labels for a subset of individuals. In WD English is not as dominant as in other ontologies: Considering just the subset of analyzed individuals English is surpassed by German and rivaled at least by Russian in numbers. On the other end of the spectrum are OBOE and SWEET which omit labels altogether for most of their individuals.

The final examined aspect is the coverage of relations. It is important not only to include individuals for each type, but also to link them together in an appropriate manner. All measurement units, e.g., could be linked to at least one kind of quantity to represent what physical property can be measured using a particular unit. That statement, however, does not hold true for all relations: Not all measurement units can be linked to a matching prefix as there is no prefix for non-prefixed measurement units. This also shows that both types within a single relation have to be considered individually in terms of coverage. Hence, in the example for each prefix there should at least be one unit be linked to it. The current state of the ontologies is shown in Table 6.

Note that a frequency of 1.00 does not suggest that all possible links are included, but only that each individual is part of at least one link. To illustrate this restriction, assume an ontology includes just two measurement units EX→meter and EX→foot, as well as two kinds of quantities EX→Breadth and EX→Height. To achieve a frequency value of 1.00 this ontology only needs to include two relations like, e.g., (EX→meter, EX→Breadth) and (EX→foot, EX→Height). The other two possible relations would not be needed, and their absence would not be detected by this metric. An optimal value of 1.00 was only achieved in a few cases. On the contrary, the relation coverage within the different ontologies varies quite a lot. Noteworthy are especially those entries that suggest no link at all, despite both types being modeled. MUO, QUDT and UO, although having both prefixes and measurement units, did not choose to link them at all. Similarly, QU and SWEET did not link field of application and measurement units. This begs the question why prefixes or fields of application were included in these ontologies in the first place. Other relations are usually given for just a subset of individuals with just few exceptions of a full coverage.

The analysis revealed several classes of issues affecting the ontologies. Some are relevant to all kinds of ontologies, others are specific to unit ontologies. In the following we describe common issue classes encountered, except the most obvious classes like missing, wrong or duplicated values, assignments or entities. Note that each of these classes only applies to a subset of ontologies. An ontology specific discussion will follow in the next section.

Entity IRIs contain ontology version. The IRIs of entities contain the version number of the ontology. This is in contrast to the idea of Linked Data [1] as an update of the ontology will break references by other ontologies or systems. The version number should be contained in an additional version-specific IRI of the ontology [2].

Label concatenation. A single label contains the concatenation of multiple alternative names of a single entity, e.g., the label “tonne (metric ton)” actually represents two labels: “tonne” and “metric ton”. This concatenation might become a problem in automation and should be replaced by multiple or alternative labels.

Mix-up of equivalent and identical units. A conversion of factor one and offset zero does not imply two unit individuals to be the same. A unit also consists of other properties like the use within a system of units or the assignment to a kind of quantity. Consider the following:

This would erroneously imply that liter is an SI unit, which is wrong [8]. Similarly, a fraction is not identical to its simplified form (e.g., m 3 m 2 is not identical to m) as discussed in [9].

Namespace proliferation. An ontology or ontology family contains multiple namespace definitions for the same or similar namespaces or the same prefix is used for multiple namespaces. Assume the following two ontology files:

The namespaces in b.owl differ only in whether or not to include the tailing #. Furthermore ref is used for two completely different namespaces.

SPARQL incompatible namespace. The namespace may be separated from the local name of an entity by using #, but the namespace definition does not include the #. Thus the namespace can not be reused in SPARQL as # is reserved for comments in SPARQL and has to be escaped each time a entity is referred using the namespace prefix.

Vague designation of units. The existence of several units of the same name in different systems of units has been ignored. For example, acre might refer to acre (international) as well as acre (U.S. survey), which represent different sizes of an area [19]. The respective systems of units should be explicitly declared or alternatively, the conversions of a unit should be defined with respect to a specific system of units as done in WD. The disadvantage of the second option would be the lack of an IRI for a unit of a certain system of units and therefore impedes referring them.

Unspecified language. An ontology provides literals containing natural language without identifying the used language with a language tag. In automated processing this may cause language mixups if ignoring those literals is not an option.

Depending on the usage context of an ontology or the specific task it was created for, some of the listed issues might not be considered problems at all. For example, concatenated labels might not do harm or even be an advantage, if the ontology is just used for manual labeling. However, if the same ontology is used for automatic annotation, it is hard to work around that issue.

4.3. Ontology assessment

A full list of issues found was sent to the authors of the respective ontologies. They were also asked about the status of the development, the intended handling of the issues and were asked for permission to report on their answer. Helpful responses were provided by authors of OM, QUDT, SWEET, and UO. They affirmed the ongoing development of their ontologies. The authors of OM fixed the reported issues immediately. Furthermore, the new community project maintaining SWEET has also started to handle the reported issues. Similarly, the authors of UO have fixed most issues, but so far did not provide a new release. All the other ontologies at the time of writing still contain the reported issues. As WD is a collaborative project we did not report discovered issues to specific authors. However, a more detailed verification and correction will be subject of future work.

In the following we discuss each ontology with regard to encountered issues. If not stated otherwise, the ontologies turned out to be inferential consistent.

MUO is one of the smaller ontologies and almost exclusively includes non-prefixed measurement units. Since the data was imported from UCUM [19], most of the issues seem to be shortcomings of the import process or UCUM itself:

In six labels the Unicode symbol #160 (“no break space”) was used instead of a space (e.g. in “calorie at 15␣°C”). This might cause problems when comparing labels to different inputs. We found two most likely unintended unit duplicates (MUO→liter-2 and MUO→the-number-ten-for-arbitrary-powers-2). Although UCUM defines both MUO→calorie and MUO→British-thermal-unit just as aliases to other units, the ontology includes them as independent units. A typing error has been recognized in the IRI and the label of MUO→pound-per-sqare-inch. The entity MUO→horsepower is a case of vague designation.

OBOE is the smallest unit ontology included in the evaluation. Nevertheless, we detected a substantial number of issues. Most critical, ten conversions contained wrong factor or offset values. In eight of these cases the direction of the conversion has been mixed up resulting in an inverse conversion factor. In case of OBOE→NanogramToKilogram, this would, e.g., result in a calculation error of factor 10 24 . Less critical, one conversion (OBOE→HectopascalToPascal) has been defined without any conversion values. A typing error occurred in the IRI of OBOE→MilliliterPerLIter. Further, we found 14 measurement units with a vague designation (e.g., OBOE→Acre and OBOE→Gallon). One conversion occurred twice (OBOE→AcreToMeterSquared and OBOE→ AcreToSquareMeter). Three times equivalent and identical units have been mixed up, e.g. OBOE→MeterCubed and OBOE→Kiloliter have been connected using owl:sameAs but should only have a conversion with factor one and offset zero.

OM is the largest unit ontology. Thus, a larger number of issues is to be expected. However, OM stands out as most reported issues below have been fixed immediately. Due to further development and the resolving of reported issues, updated releases have been published by the authors (1.8.6 and 2.0.6 respectively).

We started our analysis with version 1.8. As of the latest version, 2.0.6, the following issues remain to be addressed:

All IRIs contain the version number of the ontology. Four entities classified as units should be scales instead. For example, OM→_1-10 should by a OM→Ordinal_scale. Five temperature scale individuals were connected to the according temperature scale class by owl:equivalentClass which makes them individuals of themselves. Six temperature quantity classes erroneously contained the according temperature scale classes in their definition. For example, the definition of OM→CelsiusTemperature contained the following statement:

That caused the temperature scale classes to be a subclass of OM→Unit.

Further, we found the following issues that have been fixed in the current version (2.0.6). Most of these issues have also been fixed in the current version of OM 1 (1.8.6).

OM 2 was inferential inconsistent due to missing datatype declarations of property values of OM2→hasFactor. Therefore, OM 2 was not usable with a reasoner. Two conversions contained wrong factors. For example, the factor between OM1→metre_of_mercury and OM1→pascal was 1.33322e 2 instead of 1.33322e 5 . Three dimension individuals had wrong dimension vector values. For example OM1→thermal_resistance-dimension used the exponent − 2 instead of 0 for the length component. Nine typing errors in labels, comments, symbols, and alternative symbols have been found like “reciprocal metre kilogram” instead of “kilogram second to the power −2” in OM1→kilogram_second-time_to_the_power_-2. Seven entity IRIs contained typing errors like in OM1→SI_luminous_intensity_dimension _dimension _exponent a duplicated “dimension” breaking references to the entity. Four duplicated units were included like OM1→metre_to_the_power_-2, which is a duplicate of OM1→reciprocal_square_metre. We recognized 18 units with a vague designation like OM1→foot. The unit OM1→g · m⁻² · m⁻¹ was missing m⁻¹ in its composition definition. Two measurement units contained a wrong dimension like OM1→reciprocal_kelvin with dimension OM1→thermodynamic_temperature-dimension. Nine labels were falsely tagged English instead of Dutch like “Gaussische eenhedenstelsel” on OM1→Gaussian_system_of_units. The property OM2→hasFactor contained the class OM2→Prefix as an additional domain that caused the most units to be a prefix too. The property OM2→hasUnit instead of OM2→commonlyHasUnit was used to assign the unit OM2→are and the quantity kind OM2→Area.

QU is one of the medium sized ontologies. We found in QU→xo15degCCalorie one wrong conversion factor. We also recognized six measurement units with concatenated labels, e.g., in the label “tonne (metric ton)” of QU→tonneMetricTon. The entity QU→micrometreMicron contains a mix-up of equivalence and identity of units and should be separated, since “micrometre” is a SI unit, but “micron” is not [24]. Three units (QU→reciprocalSquareMetre and QU→perSquareMetre, QU→dryQuartUs and QU→quartUsDry, QU→dryPintUs and QU→pintUsDry) occurred twice in QU. Further, we found 14 units with vague designation.

QUDT is also one of the medium sized ontologies. The most important issue of QUDT is its inferential inconsistency. For example, it contains the following definitions:

This means that every individual of the class FinancialUnit has the type prefix “UF”. Moreover, because it is also an individual of the class Unit, it has the type prefix “U”. As the number of type prefixes is, however, limited to one, this results in a conflict. As a result QUDT can not be used with a reasoner. Also critical, three conversions and two prefixes contained wrong factors. For example, the factor of the binary prefix QUDT→Pebi was 125899906842624 instead of 1 125899906842624 ( 2 50 ). We found four duplicates of measurement units, like QUDT→Dalton2 and QUDT→AtomicMassUnit for QUDT→Dalton [24]. Further, QUDT → ind . AtomicMassUnit was classified as QUDT→AtomicPhysicsUnit but should be a QUDT → class AtomicMassUnit.23

SWEET contains only a few measurement units more than the smallest ontology in our test, but likewise contains a lot of issues:

The prefix SWEET→micro has no symbol. There are two duplicates like SWEET→FTU for SWEET→FormazinTurbidityUnit. Three conversions contain a wrong factor or a wrong offset. Two of them used the inverse values. The measurement units SWEET→FTU and SWEET→FNU are marked as identical but are not. Further, a mix-up of equivalence and identity occurred in case of SWEET→hertz and SWEET→perSecond as well as SWEET→micrometer and SWEET→micron.

Further, we found the following general issues, that have been fixed in the current version (3.1): Most adversely, the SWEET unit module file reprSciUnits.owl did not contain valid RDF,24

UO contained the following issues: The most important issue regarding UO is the lack of a stringent version management. Although most of the issues have been fixed or the erroneous concepts have been removed in the development repository25

WD Please note that our analysis of WD was not as detailed as the analysis of the other ontologies. This will be part of future work. However we found the following issues:

WD contains links to entities from OM 1 and QUDT. The links to QUDT, however, point to the URLs of an HTML representation of QUDT, but do not refer the entities themselves. The references to OM 1 contain one wrong entry: The unit WD→Q734439 (“hour angle”) points towards OM→Hour_angle, which is a kind of quantity, but should point to OM→hour-hour_angle. The conversion of “1 E-16 s” to “second” contained a wrong precision. Eight conversions contained a wrong factor like from WD→Q21014455 (“metre per minute”) to WD→Q182429 (“meter per second”) with the factor 0.166666667 instead of 0.0166666667. Further, the conversion offset in the conversion of “degrees Celsius” to “Kelvin” was missing.