Visual notations for viewing RDF constraints with UnSHACLed

3.1.1. Shape

We reuse classes (rectangles) from UML [47] to represent both node and property shapes, redefine the meaning of rectangle’s compartments for RDF constraint specifics, introduce data shape stereotypes to indicate a data shape’s type and distinguish it from other UML rectangles representing other concepts.

We use the graphical primitive shape to represent the fundamental construct data shapes and its subclasses node and property shape thus adhering to ODM [46]. Data shapes are represented using a rectangle (Fig. 3 (1)), and describe constraints applying on subjects and objects from the data graph. Node shapes describe constraints on individual focus nodes, while property shapes describe constraints for reachable nodes via a property path.

OPEN IN VIEWER

Fig. 3.

Constraints visualized using ShapeUML: a subject valid to the person data shape should have an IRI (1), at least one but maximum two ex:address properties (2) of class schema:PostalAddress (3) and the object of at least one ex:address property should comply with the existing data shape ex:validAddress (4). Additionally, the subject valid to person should either have exactly one ex:fullName or at least one schema:givenName (5) and at least one schema:familyName all of datatype xsd:string. The value of ex:fullName must not comply with the data shape ex:organizationShape (6). Addresses must only have values for the property postalAddress with an exception for rdf:type (7). Constraints of the ex:organizationShape are not considered for validation (8).

In UML “a class is drawn as a solid-outline rectangle with three compartments separated by horizontal lines” [47] which we redefine for data shapes. The upper compartment contains the data shape’s type and name (Fig. 3). We determine the data shape’s type by reusing UML concepts similar to the UML profile for OWL and RDF [46], i.e. we define UML “stereotypes” to signify what the rectangles represent: node shapes declared as ≪NodeConditions≫, property shapes declared as ≪PropertyConditions≫ and (if the data shape type is not specified) data shapes as ≪Conditions≫. The name of the data shape is displayed as bold text to support the user in the identification and differentiation of data shapes. This name may be populated from rdfs:label values of the data shape, thus following best practices in labeling RDF concepts for humans. Both the middle and lower compartment list text-based key-value pairs, therefore we stay compliant to UML. Additionally, constraint language independent labels (Figs 1, 2) are used to convey meaning and support users. The middle compartment lists information about the data shape’s identification and validation (Fig. 3). Thus, data shapes are similar to UML where the middle compartment usually contains the attributes of classes, i.e. what characterizes them. The lower compartment contains actual constraints as a key-value list (Fig. 3).

We reuse directed solid edges from ODM/UML [46] to represent relationships, reuse dashed edges overlaying individual edges from UML [47] to represent one-to-many relationships, and redefine directed dashed edges for RDF constraint specifics.

Directed edges represent different relationships between data shapes and, thus, ShapeUML is able to represent relationships between different types of data shapes. Directed edges have a label at the center of the edge and possibly cardinalities next to the ends of the association (Fig. 3). These edges associate a data shape with another data shape or set of data shapes.

We introduce solid and dashed directed edges to visually distinguish between different types of relationships. We indicate the edges from node shapes to property shapes as a directed solid edge (Fig. 3) as it represents relationships between subjects and objects of the data graph. The label of such a connection is the property path of the connected property shape which supports readability as humans can read the label while processing the edge and do not have to look for this label elsewhere in a rectangle; annotating an edge with a label also follows UML. A dashed directed edge with the label complyWith indicates that the source data shape needs to comply with the constraints of the destination data shape (Fig. 3). Therefore such connections can be distinguished from property shape connections both via a visual difference and a different label. Similarly, a dashed directed edge with the label NOT indicates that the source data shape must not comply with the destination data shape (Fig. 3). A dashed line vertically over individual edges with label next to the dashed line indicates one-to-many relationships between a data shape to a set of data shapes, following the UML specification [47] (Fig. 3).

3.1.3. Text

We reuse text from UML to represent different concepts and introduce striked through text for data shape stereotypes to indicate a deactivated data shape.

Text represents constraints stated by a data shape and provides additional information where necessary. Text is added to the upper, middle and lower compartment of a data shape and as label on edges. The type of a data shape in the upper compartment can be struck through, showing that the constraints of this data shape are not used for validation, i.e. the data shape is deactivated (Fig. 3). This visual aid aims in the quick identification of deactivated data shapes which does not introduce any visual symbol and thus does not deviate too much from the UML specification. Values referring to RDF terms can be shortened with a prefix, therefore the tool implementing the visual notation has to provide a prefix list.

3.1.4. Border

We reuse solid borders from UML, they are used for data shapes. According to the UML standard, stylistic details, such as line thicknesses, are not material to the specification. So, all data shapes are rendered using solid borders.

3.1.5. Position

We reuse positions at the beginning and end of directed edges from UML to represent cardinality-related constraint types. Within UML, association ends are among others specified by their cardinality.

In ShapeUML, cardinality constraints referring to properties are visualized next to the arrow head of a directed edge, i.e. minCount and maxCount (Fig. 3); cardinality constraints referring to data shapes are visualized next to the source of a directed edge, i.e. qualifiedMinCount and qualifiedMaxCount (Fig. 3). Thus, the visualization reflects the reading direction, for example: the person data shape requires the property ex:address at least 1 but maximum 2 times (Fig. 3) vs a valid address property requires that at least 1 property value need to comply with the address node shape (Fig. 3).

OPEN IN VIEWER

Fig. 4.

Constraints expressed using ShapeVOWL: a subject valid to the person data shape should have an IRI (1), at least one but maximum two ex:address properties (2) of class schema:PostalAddress (3) and the object of at least one ex:address property should comply with the existing condition set ex:validAddress (4). Additionally, the subject valid to person should also either have exactly one ex:fullName or at least one schema:givenName (5) and at least one schema:familyName all of datatype xsd:string. The value of ex:fullName must not comply with the data shape ex:organizationShape (6). Addresses must only have values for the property postalAddress with an exception for rdf:type (7). Constraints of the ex:organizationShape are not considered for validation (8). ShapeVOWL also visualizes optional accompanying logos for constraint types (9).

The visual vocabulary of ShapeUML defined in the last section, can be used to represent SHACL shape graphs. We present and discuss an example (Fig. 3).

ShapeUML defines visual elements for data shapes (Fig. 3). Data shapes of different types (≪Conditions≫, ≪NodeConditions≫ and ≪PropertyConditions≫) can be uniquely identified with an IRI but can also have a human readable label. For example, a node shape uniquely identified (ex:personConditions, middle compartment) can have the human readable name Person (bold label in upper compartment) (Fig. 3). Such a node shape can by default be applied on resources, e.g. ex:bob, or all instances of a class, e.g. schema:Person, both indicated by the key appliesOn in the middle compartment of a ShapeUML data shape.

Constraints are listed in the lower compartment of a data shape rectangle. A node could be constrained to be of a specific type using the nodeKind constraint. Similarly, constraints on property values are placed in the lower compartment of the corresponding ≪PropertyConditions≫ property shape. A fictive person node shape can represent the constraint that data valid to this data shape must have a unique identifier. And in the same fashion, the value of an ex:address property can be constrained to be of a specific class (Fig. 3).

Cardinality constraints are represented using text and position. Therefore a constraint to express that a person must have at least one but maximum two addresses will be denoted with the (inclusive) cardinality specification 1..2 next to the arrow head of the directed edge which connects the person node shape with the address property shape (Fig. 3).

Dashed directed edges can be used to indicate reuse of data shapes. To denote that the value of at least one of the aforementioned ex:address properties must comply with the ex:validAddress data shape, a dashed relationship with corresponding cardinalities 1..* is drawn at the source property shape (Fig. 3). In case every address should comply with the provided data shape, the qualified cardinalities at the source of the dashed arrow need to be removed. Such a removal would mean for a SHACL implementation that the two constraints sh:qualifiedValueShape and related sh:qualifiedMinCount are replaced by a single sh:node constraint. However, this is transparent in the visualization and users are not bothered with this specific terminology.

Data shapes can be connected with logical operators to build more complex constraints (Fig. 3): subjects valid to the Person node shape should have either exactly one ex:fullName property, or at least one schema:givenName and at least one schema:familyName: dashed vertical OR edge overlaying individual edges.

OPEN IN VIEWER

Fig. 5.

Correspondence between semantic constructs and ShapeVOWL: SHACL core constraints (left) and graphical notations (right).

OPEN IN VIEWER

Fig. 6.

Correspondence between semantic constructs and ShapeVOWL: other relevant SHACL concepts besides core constraints (left) and graphical notations (right).

3.2.1. Shape

We reuse blue ellipses and blue and yellow rectangles from VOWL to represent subjects, predicates and objects of the data shape graph, introduce white note-elements to represent constraints and introduce blue rectangles with rounded corners to represent node shapes.

The graphical primitive shape distinguishes the fundamental constructs node shapes, property shapes and constraints, and represents one-to-many relationships. This follows VOWL where nodes in the graphs as well as specific restrictions such as disjointness are represented with dedicated nodes. Node shapes, subjects of triples, are represented as blue rectangles with rounded corners (Fig. 4), property shapes, the predicate and object of a triple, as rectangular label on a directed edge (Fig. 4) and either a ellipse or rectangle at the end of the edge representing the object (Fig. 4, ), and constraints as rectangle with the upper right corner bent (note element) (Fig. 4, ). Thus, node and property shapes align with VOWL as the data shapes appear like the RDF graph on which they define constraints on.

The note-element, containing constraints as text, is visually attached at the node shape or property shape indicating the constraints applying on the represented subjects, predicates or objects of a triple; constraints are visualized where they apply to facilitate the processing of the visualization by users. We also introduce ellipses as intermediate element to denote one-to-many relationships (see edges).

3.2.2. Edge

We reuse directed solid edges with rectangular labels from VOWL to represent properties and redefine directed dashed edges for RDF constraint specifics.

Edges represent relationships between data shapes which makes ShapeVOWL able to represent different kind of constraints in a visual fashion. Directed dashed edges (Fig. 4) refer to relationships between data shapes and denote their label directly as text on top of the relationship. They indicate that the source data shape needs to comply with the constraints of the destination data shape.

Directed solid edges are part of a property shape and indicate their label in a rectangle above the edge (Fig. 4), following VOWL. The label of directed solid edges is the property path of the represented property shape; relationships between data shapes are visually distinguished from property shapes due to the use of different edges.

Similar to VOWL, cardinalities are denoted next to the arrow head (Fig. 4), but additionally data shape related qualified cardinalities are denoted at the start of a directed dashed edge (Fig. 4). Node and property shapes may refer to multiple other node and property shapes in a one-to-many relationship to represent logical relationships. We represent such relationships using additional ellipses, representing the meaning of individual one-to-many relationship, i.e. conjunction and disjunction (Fig. 4), similar to certain restrictions in VOWL, e.g., disjointness.

3.2.3. Text

We reuse text from VOWL to represent labels, redefine datatype to represent datatype constraints, introduce text to represent constraints and italic text to represent the unique identifier of data shapes.

We use text to represent constraints stated by data shapes, unique identifiers, and labels. Text is added in constraint note elements, node shapes and property shapes. Constraint note elements contain constraints in the form of text where the constraint’s name is listed followed by its value in parentheses. This allows a consistent representation of different constraints without introducing a new visual variable for each of possibly more than 80 constraint types [7]. Values referring to RDF terms can be shortened with a prefix, therefore the tool implementing the visual notation has to provide a prefix list. Data shapes may have an optional human readable name which is denoted as bold text in the upper part of the data shape to facilitate the distinction of data shapes. This name may be populated from rdfs:label values, and, thus following best practices for labeling RDF concepts. Additionally, the unique identifier of node shapes is visualized as text in italics in the center of the rectangle with rounded corners representing the node shape (Fig. 4). Users can also identify node shapes without a human readable label present. The italic type distinguishes the unique identifier from other text.

3.2.4. Border

All visual shapes have a border, we reuse solid borders from VOWL, redefine dashed borders to accommodate for validation-specific characteristics regarding deactivation and introduce thick solid borders to represent the constraint type closed.

VOWL uses dashed borders for specific OWL classes and literals without datatype. However, we use dashed borders to indicate which data shapes are not considered for validation (deactivated), because in contrast to an ontology visualization, we do not consider specific OWL classes but RDF constraints for validation, and our visualization of literals has a different meaning as we visualize constraints (Fig. 4). For deactivated node shapes both the rectangle with rounded corner representing the node shape as well as a possibly attached note element with constraints will get a dashed border (Fig. 4). Similarly, for deactivated property shapes the rectangle of the relationship label, the object and potentially attached note elements get a dashed border.

We introduce thick solid borders for node shapes, indicating that for validation only the explicitly linked properties are allowed (closed data shape, Fig. 4).

The thick borders aim to represent the closeness whereas dashed borders aim to represent inactiveness. As the thickness and style of the edges are two different visual features, possible combinations of deactivated and closed data shapes can still be represented.

3.2.5. Position

We reuse cardinality positions at directed edge endings for property-based cardinality constraints, introduce cardinalities at the beginning of a directed edge to represent data shape related cardinality constraints, introduce positions for logical constraints within dedicated nodes and introduce positions for datatype and class constraints within the objects of visualized triples.

We use specific positions for cardinality, datatype, class and logical constraints utilizing the graph visualization to support users in the parsing of information. In ShapeVOWL, cardinality constraints referring to properties are visualized next to the arrow head of a directed edge, i.e. minCount and maxCount; cardinality constraints referring to data shapes are visualized next to the source of a directed edge, i.e. qualifiedMinCount and qualifiedMaxCount (Fig. 4). The visualization reflects the reading direction, for example: the person data shape requires the property ex:address at least 1 but maximum 2 times (Fig. 4) vs a valid address property requires at least 1 property value to comply with the address node shape (Fig. 4).

Datatype and class constraints are not visualized in a note element, but directly as text in the graphical element representing the object, i.e. a yellow rectangle for datatype constraints (Fig. 4) or a blue ellipse for class constraints (Fig. 4). VOWL visualizes datatypes as text within the yellow rectangle representing a literal. We reuse this visualization to denote a datatype constraint of a property value and add an additional datatype icon in front of the name of the datatype to indicate that a constraint exists (Fig. 4). This icon is an orange D in a black circle (Fig. 4). Consistently with datatypes, class constraints are denoted as text within the ellipse representing the property value. Class constraints have an additional class icon in front of the name of the class. This icon is an orange C in a black circle (Fig. 4).

Logical constraints are not represented in a note element, but as dedicated nodes or as labels on dashed edges which enables ShapeVOWL to represent relationships between different types of data shapes. Conjunction and (exclusive) disjunction constraints are visualized as ellipse with respective labels on the upper part of the ellipse (Fig. 4). Additionally, icons representing Venn diagrams are used to distinguish the different logical constraint types. These icons represent Venn diagrams, similar to certain VOWL constructs. Negation constraints are represented as text label “NOT” on top of dashed edges connecting data shapes (Fig. 4).

3.2.6. Color scheme

We reuse the VOWL base color to represent subjects, predicates and objects of the data shape graph, reuse the VOWL literal color to represent literals and introduce border colors for data shapes’ severity.

A color scheme is applied on the border color of data shapes and note elements to express different severities (Fig. 4). VOWL uses a color scheme for a better distinction of the different elements [41]. We reuse the base color and literal color of VOWL.

Additionally, for ShapeVOWL colors on borders are used to express the severity of data shapes. For the severities violation, warning and information from the SHACL specification we recommend the respective colors red, yellow and green. Green is chosen instead of blue so the severity colors for data shapes are not confused with the VOWL general color.

3.2.7. Visual example

The visual vocabulary of ShapeVOWL defined in the last section, can be used to represent SHACL shape graphs. We present and discuss an example (Fig. 4).

ShapeVOWL defines visual elements for data shapes (Fig. 4). Our color scheme is applied; data shapes are colored with respect to their severity.

Node shapes can be uniquely identified with an IRI but can also have a human readable label. For example, a node shape uniquely identified with the IRI ex:personConditions (center of rectangle with rounded corners representing a subject node) can have the human readable name Person (bold label in upper part of the rectangle with rounded corners) (Fig. 4). Such a node shape can by default be applied on resources, e.g. ex:bob or all instances of a class such as schema:Person, both is indicated by the appliesOn annotation in the attached white note-element of a ShapeVOWL data shape.

Constraints have a special position or are listed in white note-elements attached to a data shape; depending on the rendering either overlapping an ellipse/rectangle with rounded corners (Fig. 4, ) or next to a rectangle (Fig. 4). A fictive person node shape can represent the constraint that persons must have a unique identifier (Fig. 4, nodeKind constraint). The value of an ex:address property can be constrained to be of a specific class whereas value type constraints are listed within the shape representing the object together with an icon (Fig. 4). Cardinality constraints are represented using text and position. Thus, a constraint to express that a person must have at least one but maximum two addresses will be denoted with the (inclusive) cardinality specification 1..2 next to the arrow head of the directed edge which connects the person node shape with the address property shape (Fig. 4).

Dashed directed edges with the label complyWith indicate reuse of data shapes. To denote the constraint that the value of at least one of the aforementioned ex:address properties must comply with the ex:validAddress data shape, a dashed relationship with corresponding cardinalities 1..* is drawn at the source property shape (Fig. 4). In case every address should comply with the provided data shape, the qualified cardinalities at the source of the dashed arrow have to be removed.

Data shapes can be connected with logical operators to build more complex constraints (Fig. 4): subjects valid to the Person node shape should have either exactly one ex:fullName property, or at least one schema:givenName and at least one schema:familyName: disjunction node with label “OR” and Venn diagram icon. The logical operator negation only takes one data shape as argument and not a whole data shape list, therefore it is visualized with the label NOT on a dashed connection (Fig. 4).

With respect to validation data shapes may be closed or deactivated. The ex:validAddress data shape is closed, visually indicated by a thick border: valid addresses are only allowed to have the property postalCode and an exception is made for rdf:type denoting the class (Fig. 4). The data shape ex:organizationShape is deactivated, visually indicated by dashed border: its constraints are not considered during validation (Fig. 4). Constraint types can be accompanied with a logo displayed before the constraint in the note element (Fig. 4).

5.2.1. Architecture

UnSHACLed is a web-based RDF constraint editor independent from specific data formats, visual notations or validation engines.

Framework UnSHACLed is implemented with the web framework Vue.js following the model-view-viewmodel (MVVM) design pattern introduced by John Gossman in 2005.16

It therefore can run in modern Browsers and no additional server infrastructure such as databases are required.

Intermediate format UnSHACLed uses the state management pattern and library Vuex to store RDF constraints using an intermediate data format which allows all application components to access the RDF constraints in a controlled manner. Therefore other constraint languages can be supported by providing a mapping to the intermediate format without the need to change other parts of the implementation.

Visual notations UnSHACLed uses the VueKonva library to draw canvas graphics. Several components for both ShapeUML and ShapeVOWL were developed to render the two notations. New visual notations can be added in the form of new components which also read and write data to the intermediate format of Vuex store.

Validation For validation the intermediate format is transformed to a representation of a concrete RDF constraint language (currently supported is SHACL) and is passed together with the data to a separate validation engine. Another constraint language and validation engine can be used which only leads to adjustments in UnSHACLed with respect to transformations of the intermediate representation or invocation of another validation engine, no adjustments to the GUI are required.

In this section we discuss the graphical user interface of UnSHACLed, namely the different existing panels and interactions elements with which users can interact using visual notations.

Panels The GUI consists of three panels representing different parts of a Linked Data validation workflow: a data panel, modeling panel and validation result panel.

The Data panel shows data which should be constrained or described (left panel in Fig. 7). RDF is currently supported in different serializations, such as turtle and JSON-LD. This is raw data and can also be edited. UnSHACLed is modular and the functionality can be extended to also visualize data of other kind to support other editing approaches.

OPEN IN VIEWER

Fig. 7.

The user interface of our tool UnSHACLed consisting of several panels supporting different editing approaches.

The Modeling panel shows RDF constraints in the visual notation chosen in the menu, both ShapeUML and ShapeVOWL are supported (middle panel in Fig. 7). Elements in the modeling panel are denoted visually and scalability is addressed with geometric zooming.

The Validation result panel shows the validation result of applying the RDF constraints of the modeling panel on the data of the data panel as reported by a validation engine for RDF constraints. The validation result panel is implemented as a modal dialog, i.e. it appears after clicking the validation button. UnSHACLed is independent of concrete RDF constraint languages, it can be extended with different validation engines.

Interactions Visual notations specify how RDF constraints are visualized, but UnSHACLed also allows to interact with the visualizations. Most notably nodes in the graph can be dragged and dropped inside the modeling panel. When hovering over an element a red and a green button appear representing actions for delete and editing. In the latter case a modal dialog opens in which users can change or add constraints. Thus, users can also edit RDF constraints using the visual notations and do not have to learn a specific textual syntax.

6.1.1. Constraint concepts questionnaire

We derived questions from the SHACL specification relevant to RDF constraints and validation, which were used in a user study to validate our hypothesis.

We created questions to test (i) at least one constraint type per core constraint category of the SHACL specification, and (ii) other RDF constraint concepts relevant for validation, i.e. the targeting mechanism, property paths, severity and deactivation. The SHACL specification lists eight core constraint categories:

We selected at least one constraint type for each category and created an associated question, for example “How many datatype constraints can you see?” for the constraint type datatype of value type category. The last two categories mix several relevant constraint types, so, we selected 2 constraints types for each.

Additionally we created one question for each of the aforementioned other relevant concepts, such as “How many property conditions with the severity ‘information’ can you see?” for the concept severity or “How many zero-or-more property paths can you see?” for the concept property paths.

6.1.2. Follow-up questionnaire

We created six questions to cover more diverse visualization tasks compared to the initial user study questionnaire and one question to test participants’ understanding of property paths for which we observed the highest error rate in the initial user study.

The questions cover the following six visualization tasks from ten Amar et al. [2] tasks, which we have chosen based on a taxonomy alignment from Brehmer and Munzner [9], see also Fig. 10 and the Task description paragraph in the next section.

To keep the follow-up questionnaire short, we have chosen to select maximum one task per taxonomy leaf node. Therefore, no question was asked for the tasks Find anomalies, Find clusters, Find correlation and Characterize distribution as the first three belong to the same taxonomy leaf node explore such as Find extremum, and the last belongs to the same taxonomy leaf node identify such as Determine range.

Additionally we asked the question “Do you see any ‘property path’ which is not just a single property?” because in the initial user study participants identified property paths in test cases when in fact no were shown. In case they answered yes, we also asked the question “Which is the property path and where do you see it, please elaborate” to obtain more information.

6.3.1. External validity threats

External validity threats occur when wrong inferences from sample data are made beyond the studied population or experimental setup [12]. We identified two external threats: participants familiarity with Linked Data and experiment environment.

Participants familiarity with linked data This threat concerns the generalization to individuals outside the study [12]. All our participants were recruited from Ghent University, Belgium and RWTH Aachen, Germany and were familiar with Linked Data, thus the findings might not be generalizable to a more general population. However, this was intentional as we aimed to study users already familiar with RDF graphs, a prerequisite to understand RDF constraints which are the semantic constructs our visual notations represent.

Experiment environment This threat concerns the generalization to individuals outside the experiment’s setting [12]. The experiment was an online questionnaire. Participants could use any browser and computer, thus, they participates from a well-known environment. No specific experimental setup prevents generalizations to individuals outside our study.

6.3.2. Internal validity threats

Internal validity threats concern the experimental setup or experience of participants which threaten the ability to draw correct conclusions about the population in the experiment [12]. We identified three internal threats: selection bias, sample size and order effects.

Selection bias This threat concerns the selection of biased participants, i.e. participants with certain characteristics that predispose them to have certain outcomes [12]. Our participants were all recruited from Ghent University and RWTH Aachen and have similar demographics. All participants have knowledge about Linked Data, but this is intentional as it is a prerequisite of the user study. To mitigate a selection bias all participants were assigned in a round-robin fashion to one of two groups, i.e. groups were not assigned based on specific characteristics. Some participants might be more familiar with one of the underlying visual notations of ShapeUML or ShapeVOWL. However, they self-assessed their familiarity with UML class diagrams and the WebVOWL tool in the pre-questionnaire, therefore any bias is visible. Please note that familiarity with one of the notations is considered positive as the design rationale of both visual notations is to build upon the underlying visual notation.

Sample size A small sample size may not have sufficient statistical power to detect an effect. Our sample size is relatively small. To mitigate this threat, we chose a within-subject study design [12]. It reduces errors associated with individual differences without requiring a large pool of participants.22

Order effects When participants perform tasks several times certain effects like learning can occur. To counterbalance potential order effects when presenting ShapeUML and ShapeVOWL, we assigned participants in a round-robin fashion to two different groups. The first group (group A) started with the first example in ShapeUML, the second in ShapeVOWL, the third in ShapeUML and so forth. Participants of the second group (group B) were presented the first example in ShapeVOWL, the second in ShapeUML and so forth.

6.4.1. ShapeUML/ShapeVOWL error rate

Based on the correct answers, we calculated the error rates of all questions to compare ShapeUML and ShapeVOWL: initial questions for general examples and the 4 test cases (Section 6.1.1), as well as for questions in the follow-up study covering different tasks (Section 6.1.2).

OPEN IN VIEWER

Fig. 12.

The error rates for both visual notations are relatively similar. Higher error rates for both visual notations were observed for property paths and maximum value.

There is no significant difference in the mean error rates of ShapeUML and ShapeVOWL. We first tested the normality of the error rates’ distribution using a distribution plot and a Shapiro–Wilk test [59] with α = 5 % to determine which statistical test to choose. The data was not normally distributed, thus we performed a Wilcoxon signed-rank test [66] with α = 5 % . The calculated p-value of 0.856 is bigger than α so we fail to reject the null hypothesis, which means there is no significant difference in the mean error rates.

The questions of Section 6.1.1 represent tasks identifying fundamental concepts and core constraints of RDF constraint languages. With both visual notations more than 81% of questions were answered correctly. We elaborate for each constraint concept why mistakes possibly happened by qualitatively analyzing provided answers (Figs 12 and 13) and optionally given free text feedback answers. We elaborate on (i) the tasks themselves, (ii) constraint concepts which have similar error rates between both notations, and (iii) constraint concepts which show more error variation between notations. Finally, we discuss overall findings. The analysis is further enriched with findings from a follow-up user study covering different questions, yet also involving certain constraint concepts.

Visualization tasks Questions of the main questionnaire combine the tasks to retrieve a value and compute a derived value, i.e. retrieving constraint types and compute the sum as aggregated value. If tasks resulted in wrong results it is usually because participants retrieved the value wrongly, for example because they did not understand a constraint concept or were unaware of default values (see following discussion); similarly for errors in the other tasks covered in the follow-up user study. However, for the main questionnaire a small possibility that participants retrieved the value correctly and just computed the sum wrongly cannot be ruled out completely.

OPEN IN VIEWER

Fig. 13.

The error rates for the different questions across the 4 real world test cases of the main questionnaire. Most RDF constraint concepts related questions were answered correctly. Participants made the most errors for property paths, maximum value, specific value and disjunction.

Best and worst recognized constraint concepts There was (almost) no difference in error rates between ShapeUML and ShapeVOWL for 8 out of 13 constraint concepts. Most notably this covers the questions with the least and most errors, meaning that certain constraint concepts are equally good/bad recognized.

The least errors in the main questionnaire, only 4%, were observed for deactivated data shapes with both notations. This constraint type is indicated by struck-through text in ShapeUML and by dashed borders in ShapeVOWL. One participant who was not sure whether deactivation is a transitive constraint, pointed out that the visualization helped to make clear which data shapes are deactivated. The participant raised the same point for the severity constraint for which around 10% errors were made. Regarding visualization tasks, these constraint concepts had to be retrieved and then counted. In the follow-up study one question asked to only retrieve the correct value of a nodeKind constraint with multiple choice answers and there no errors were observed.

The most errors, more than 40%, were observed for property paths. In both notations this concept is encoded as an atomic label for property shapes. The provided answers suggest that participants have confused property paths with a combination of logical relationships and cardinalities on properties. To further investigate this issue, we performed a follow-up study where we explicitly asked the participants to indicate if they can identify property paths and if so where. No property path was present in the examples, yet three participants identified property paths. One wrongly identified property path was a property constrained with a literal pattern containing a pipe symbol (logical disjunction) separated list of URIs, hence probably identified as an alternative path. In another example the value of one property shape contains a class constraint, the same class is mentioned as target by another shown node shape – without any explicit link. According to the provided feedback, this implicit link appeared to be a property path for at least one participant, another participant identified the same property but did not provide explicit reasoning. Finally, one participant mentioned to not understand the question “Do you see any ‘property path’ which is not just a single property?”, which may indicate issues in understanding the underlying semantic construct (property paths were explained in the user introduction to both visual notations users had to read before the study).

Constraint concepts with similar error rates between visual notations Other constraint concepts with similar error rates between ShapeUML and ShapeVOWL were datatype, disjunction, property, closed and comply with. Whereas comply with constraints are encoded in the same way in both notations, the other constraint concepts are encoded differently, usually with more visual features in ShapeVOWL.

Datatype constraints were mostly identified correctly. Based on the provided answers it seems that one participant once wrongly counted a nodeKind literal constraint as a datatype constraint in a retrieve value task. Within an ordering task of the follow-up study participants had to alphabetically order properties with datatype constraints, but 40% of participants wrongly ordered a property with literal value without datatype constraint. However, they correctly excluded a property with class constraint in another example which at least for ShapeVOWL could indicate issues in understanding a datatype constraint if the value is visualized as a literal.

Two real world test cases contained disjunction constraints, one of the test cases additionally contained an exclusive disjunction constraint which was not supposed to be counted. However, one participant counted both and another participant indicated that in fact a second (exclusive) disjunction is present but was not counted by the participant. We acknowledge that our question leaves room for interpretation. One participant seemed to have counted properties instead of disjunctions and two participants identified this constraint when in fact it was not present, a zero-or-more property path constraint and an associated cardinality might have been counted as these were the only other constraint types shown in the example.

Property constraints were mostly correctly recognized. Mistakes were mainly made when the property shape or the corresponding node shape were deactivated, in this case some participants did not count the deactivated properties.

Two test cases contained closed constraints, participants also counted node shapes in other test cases when no closed constraint was present and did not count it when it was present.

Constraint concepts with error variation between visual notations Even though not significant, there is more variation between ShapeUML and ShapeVOWL error rates for 5 out of 13 constraint concepts. This includes the constraint concepts target, less than, specific value as well as constraints related to a minimum or maximum namely min length, max cardinality, and max value.

In 3 out of 4 real world test cases, there was 1 target concept encoded which was correctly recognized. However, in the last test case where no target concept was present, 4 participants probably counted the number of node shapes, which was 2, or the single node shape which had constraints attached. No additional feedback was provided, therefore we cannot interpret why these participants failed to recognize this constraint concept in the last test case.

A few participants miscounted the constraint type less than or equals, but the provided answers do not indicate they were mistaken for other constraint concepts.

Participants counted specific value constraints mostly correct. However, most errors occurred in the only test case which actually contained specific value constraints. One valueIn and one hasValue constraint were present which we both intended as “specific value” according to the question phrasing. Yet, 5 out of 12 participants only count 1 which suggests that they either counted valueIn or hasValue. This indicates the question was too ambiguous, and indeed a participant also pointed out for another test case that even a range constraint might be considered “specific”.

Error rates related to minimum and maximum values are explained by participants’ misconception or misinterpreted questions. The provided answers suggest that participants counted cardinality constraints when in fact they were supposed to count min length or max value. One participant even pointed out to not understand the difference between max cardinality and maximum value. This indicates issues in understanding the underlying semantic constructs and not necessarily issues with the visual notations or questions. Furthermore, provided answers suggest that cardinalities were miscounted because firstly min/max pairs were counted instead of only minimum respectively maximum cardinalities in the general example, and secondly, wrong default cardinalities were assumed by the participants for the 4 real world test cases; “For me, an arrow without cardinalities means ‘1..1”’ said by one participant is actually the wrong default, our introduction mentions a default cardinality of 0..* if no values are provided, i.e. no minimum or maximum constraints present.

Discussion of findings Based on the qualitative analysis of the results, we discuss findings related to default values, needed understanding of semantic constructs, and clarity of questions.

Default values should be encoded explicitly. On the one hand, according to provided feedback, encoded severities and constraint deactivation helped a participant to correctly interpret these concepts and discard the wrong assessment that these concepts are transitive. On the other hand, missing defaults lead participants to assume wrong cardinality defaults.

Clear documentation and/or tooltips are necessary to support users in understanding constraint concepts, because some constraint types are conceptually similar and need clarification. We noticed that users mistook e.g. different minimum and maximum constraint concepts with each other and property paths with a combination of logical relationships and cardinalities. A visual notation represents semantic constructs, the used visual notations do not suffer from symbol redundancy, symbol overload or symbol excess, thus they provide semiotic clarity (see Section 4). However, if underlying semantic constructs are not clear to a user, visual notations can only support to a small extent to alleviate misunderstandings, e.g. by providing semantic transparency.

We acknowledge that a limited number of questionnaire questions leave room for interpretation which negatively influences the analysis of results. For the follow-up user study we relied on adapted questions from related work, increasing consistency. Furthermore, for further research on RDF constraint visualization, we recommend a pilot study with a smaller number of participants – ideally from different backgrounds – to reveal possible ambiguities in questions.

The post-questionnaire contained three questions in which the participants could self assess how confident they are with their answers, if they prefer ShapeVOWL over ShapeUML and if they would like to use ShapeVOWL also for RDF constraint editing. These three questions were asked using a 7-point Likert scale from 1 (not agree at all) to 7 (fully agree).

All participants were asked if they are confident that their provided answers are correct. Their average value is 3.6 and median is 3, thus in a self assessment participants are not very confident. Participants could also provide feedback for each test case via a text field. Considering the provided feedback, some participants had trouble interpreting the asked questions which could relate with their low confidence.

All participants were asked if ShapeVOWL is preferred and the average value is 4.6 and median is 5, thus in a self assessment participants prefer ShapeVOWL. Similarly the average is 4.8 and median is 5 for the question if the participants would like to use ShapeVOWL to edit RDF constraints.

6.5.1. Method

A general procedure for a qualitative analysis involves the process of “coding” [12], a commonly used technique for reducing qualitative data to meaningful information by assigning labels to chunks of data [53]. Following common guidelines [12] we read answers provided in the post questionnaire and thus were able to identify 5 high level codes: advantages, disadvantages, uncertainty, suggestion and preference. These codes are further detailed in a hierarchy, for example the high level code advantages is further specified as easier comprehensible, display of sparse constraints and space efficiency. In a similar fashion the other high level codes are further specified to be used as annotation for the qualitative data.

6.5.2. Interpretation and meaning

Based on the created annotations we interpret the feedback provided by the participants by discussing the high level codes such as advantages and which information specifically was provided.

Participants preference and suggestions Findings regarding preferences and provided suggestions correspond with our analysis of cognitive effective design principles in both notations, i.e. ShapeVOWL adheres to more design principles. In total 4 participants explicitly indicated which visual notation they prefer, 3 of them prefer ShapeVOWL and 1 ShapeUML. To improve ShapeUML one participant suggested to remove potential redundancies in ShapeUML (see disadvantages) and “a more user-friendly visualisation of UML (eg: colors, option to hide parts)”.

Advantages Slightly more advantages were pointed out for ShapeVOWL, whereas both notations have their own advantages mostly related to how comprehensible they are for certain use cases. In total 5 participants provided feedback with respect to advantages, 3 of them for ShapeUML and 4 for ShapeVOWL. ShapeUML was recognized more space efficient by 1 participant whereas the same participant mentioned that for sparse constraints ShapeVOWL “looks cleaner”. For both notations 3 participants indicated that the respective notation is easier comprehensible. For ShapeUML 2 participants pointed out that its list representation allows to condense more constraints of a single node and 1 participant expressed that ShapeUML is more intuitive. For ShapeVOWL 2 participants pointed out that it is easier to spot constraints due to visual features and 1 participant explicitly mentioned the familiarity to VOWL as reason.

Disadvantages Although ShapeVOWL was preferred, more disadvantages were explicitly pointed out for it compared to ShapeUML. In total 3 participants provided feedback with respect to disadvantages, 1 for ShapeUML and 3 for ShapeVOWL. ShapeUML was perceived redundant by 1 participant in a negative sense, i.e. the repetition of property paths both in the data shape rectangle and on the relationship between node and property shape. For ShapeVOWL, 2 participants reported possible complications when interacting with it, namely “many small comment boxes” for constraints which “would be less orderly” and the different geometrical shapes and colors as “things” which are “more of a hassle to work with (more clicking and less typing involved)”. Additionally, 1 participant noted that ShapeVOWL “looks very simplistic, but needs more understanding to apply”.

Uncertainty Corresponding with Likert-scale answers regarding confidence and our quantitative analysis, participants explicitly mentioned unclear terminology. In total 3 participants provided feedback with respect to unclarity, whereas 2 participants mentioned an unclear terminology and 1 participant ambiguous questions. This corresponds also with observations from the quantitative analysis, i.e. wrong answers for conceptually similar constraint types.