Eye tracking the user experience – An evaluation of ontology visualization techniques

2.1. Ontology visualization techniques

Katifori & Halatsis [27] review 34 ontology visualization tools and classify them into six categories including indented list, node-link and tree, zoomable, space-filling, focus with context or distortion, and 3D information landscapes, based on the presentation, interaction, functionality and dimensions used. The predominant functionalities of the tools are used in the categorization when some tools contain overlapping features of other categories, e.g., when some space-filling techniques are also zoomable. A key observation from the review is that among the surveyed visualizations, 24 techniques1

Given a readily available set of visualization techniques, recent research has focused on improving the underlying technical algorithms and platforms. For instance, indented lists are used in WebProtégé [68] and FlexViz [13], which have moved the ontology editing and browsing process to an online environment to facilitate collaboration. Recent improvements on graphs include reducing the number of nodes displayed at once to avoid cognitive overload in KC-Viz [41]; improving real-time interaction with force-based layouts in OLSVis [71]; as well as supporting web-based collaborative visualization in BioMixer [16]. Using both indented lists and graphs, the NeOn toolkit [22] aims to support networked ontology engineering, while others aim to present multiple coordinated visualizations such as those presented by Kuhar & Podgorelec [32], BioPortal [55], ALViz [35], and BioMixer [16]. A key observation from these recent developments is that the majority of recent advancement in ontology visualization has incorporated or is building upon indented lists and graphs, which makes these two most commonly used visualization techniques relevant and interesting subjects for our evaluation.

Evaluating visualizations is a challenging task as discussed in [46], as ideally we need to use realistic tasks that echo real-world problems, cope with incomplete datasets, and report on long term use in natural settings. Over the years, a range of evaluation approaches have been developed to improve our understanding of usability issues associated with different visualization types. Komlodi et al. [30] identify four themes of evaluation approaches including i) controlled experiments that compare design elements, ii) usability evaluation of tools, iii) controlled experiments that compare tools, and iv) case studies in realistic settings. Lam et al. [33] reviewed over 800 papers with 345 evaluations and identified seven guiding scenarios based on the goal of the evaluation, including i) evaluating environments and work practices in order to elicit formal requirements for design, ii) evaluating visual data analysis and reasoning in order to assess how a visualization tool supports users in generating actionable and relevant knowledge in specific domains, iii) evaluating communication through visualization in order to assess how messages are conveyed to relevant audiences, iv) evaluating collaborative data analysis in order to study how a visualization tool support joint conclusion and/or decision-making, v) evaluating user performance in order to investigate how specific features affect measurable user performance objectively, vi) evaluating user experience in order to understanding how people react to visualizations either in a short or a long time span, and vii) automated evaluation of visualizations to test certain aspects that can be measured automatically through a computerized procedure. Carpendale [7] groups visualization evaluation techniques into quantitative and qualitative approaches. Quantitative approaches typically include controlled experiments that measure the effect of (a set of) independent variable(s) on (a set of) dependent variable(s), and the application of statistical tests. Examples of qualitative evaluations include interviews, observations, and think-aloud protocol. Regardless of the specific evaluation approach applied, there is always a trade-off between the generalizability, precision, and realism of the results [40]. For instance, controlled experiments can produce precise results, but may lose realism when tasks are overly simplified to suit the needs of the study and fail to mirror real-world problems. Interviews and observations may collect in-depth results from realistic tasks in natural settings, but lack precision and are difficult to generalize when the sample size is too small to minimize bias in individual subjectivity.

User studies or user-centered evaluations have become the predominant method in recent years and are well documented in terms of experimental design, data collection and analysis [14,15,67]. Using eye tracking as a method to infer cognitive processes in user-centered evaluation has long been practiced in InfoViz, and early eye movement studies date back many decades in cognitive and perceptual psychology [23]. As eye tracking hardware and algorithms mature, various reviews on tracking systems [10,72], as well as guidelines on experimental designs and metrics have been proposed [20,49,50], which have been applied in a range of research fields such as neuroscience, psychology, marketing, and computer science [10]. Compared to the aforementioned evaluation approaches that indirectly measure visualizations (i.e. through interviews and controlled experiments), eye tracking is able to provide direct information on the eye movements of a participant during an experiment. In other words, while task time and task success tell us what has happened as a result of using a particular tool, eye tracking results help us understand why certain design elements lead to an increase/decrease in speed and accuracy. This makes eye tracking a suitable method for the evaluation of visualizations, as it offers a “more focused tool to investigate micro-level design issues” [18].

2.3. Evaluation of ontology visualizations

Compared to the advances in developing ontology visualization techniques, efforts to evaluate existing and new visualizations have been relatively scarce. One evaluation approach is to benchmark the proposed visualization against one or a set of existing tools in terms of supported features and use cases [21,71]. The evaluation typically outlines the extra features and functions that are supported by the proposed tool but not by other existing tools. It is often difficult to argue against bias in the choice of the benchmarks and scenarios. For instance, what makes some tools reliable benchmarks but not others, or how to ensure that the use cases are selected objectively and are not chosen to simply make the proposed tool appear superior.

Another evaluation approach is to elicit qualitative feedback through a range of techniques such as interviews, observations and think-aloud protocols [31]. As discussed previously, it is often challenging to argue for representativeness and generalizability of the results due to individual preferences and subjectivity. This is especially difficult when the sample size is very small.

OPEN IN VIEWER

Fig. 1.

Sample Task Screen. In this example, graphs are used to visualize two biomedical ontologies. Mappings to be evaluated are presented in a spreadsheet. Interacting with the visualizations, participants must use drop-down lists containing either “yes” or “no” responses to evaluate the correctness of existing mappings (in part 1) and add missing mappings by typing class names (into part 2 of the spreadsheet).

A third evaluation approach that is most often practiced is controlled experiments that typically measure the speed and accuracy such as [17,28,41,65,73]. While typically, these experiments can evidently show better results of some visualizations in terms of their effectiveness and efficiency, but they do not necessarily provide insight to why that is happening. Building upon these existing evaluation strategies, we aim to provide a micro-level understanding of ontology visualizations through combining eye tracking in controlled experiments.

3.1. Study setup

We presented each participant with a set of mappings between two ontologies, and asked them to identify correct and incorrect mappings as well as to add missing mappings. The participants were assisted by visualizations of the ontology pair. The participant needs to: 1) identify correct and incorrect results among an existing set of mappings (i.e., determining the correctness of the given mappings); and ii) create new mappings that are absent from the existing set (i.e., determining the completeness of the given mappings). Figure 1 shows an example of what a participant saw on her (or his) screen.

We used a Tobii 21505

To successfully evaluate a mapping, the participant must understand the semantics of the mapped entities in their respective ontologies and must use this knowledge to determine whether a mapping relation exists. Since it is only possible to complete the tasks after exploring and interacting with the ontology visualizations, this setup ensures the evaluation focuses on how well the indented lists and graphs support ontology understanding. In particular, the participant must first search for relevant information in the visualizations, and only after processing this information can she (or he) use this knowledge to complete the tasks. This experimental design is thus well suited to answer our research questions (discussed in Section 1).

OPEN IN VIEWER

Fig. 2.

Tobii 2150 Eye Tracker used in the Study. The eye tracker was set up in an office environment. There were indoor lighting but no direct sunlight or other high intensity near-infrared light.

We used two pairs of ontologies in the study. Each accompanied by a set of mapping standards, taken from the Ontology Alignment Evaluation Initiative (OAEI) 2012 conference7

The conference ontologies describe the organization of conferences with a total of 74 and 110 entities respectively, with at most 3 or 6 classes on the longest path to root, at most 8 or 9 subclasses for a class, and without any multiple inheritance. The conference task represents a less difficult scenario, because the ontologies involved have fewer classes, the number of subclasses per class is fewer and the paths to root are shorter.

The BioMed task involves ontologies describing concepts in the organism domain. We reduced the size of the original ontologies and gold standards. In our study, the BioMed ontologies have a total of 89 and 181 entities respectively, at most 11 or 12 classes on the longest path to root, at most 6 or 10 subclasses for a class, with at most 4 occurrences of multiple inheritance. The BioMed task illustrates a more difficult scenario as the ontologies contain more entities, the number of children per entity is increased, the paths to root are longer, and multiple inheritance is present.

Based on the OAEI gold standards, for each ontology pair, we randomly removed correct mappings from its gold standard and added incorrect mappings in order to create two mapping sets to present to the participants. The conference task and the BioMed task both required the participants to identify 13 correct results, 3 incorrect results and to add 7 missing mappings in each task scenario. This setup thus ensures the study outcome (in particular, time on task) is not affected by the number of mappings to be evaluated, but rather a result of ontology and visualization complexity. In addition, we aim to reduce the possible impact of a participant’s background on the study outcome by using two different knowledge domains.

OPEN IN VIEWER

Fig. 3.

Visualization Techniques Investigated in the Study. The examples shown here are taken from the SNOMED ontology.

We presented indented list visualizations to the participants by loading ontologies into Protégé and asked participants to interact with the indented lists exclusively but not with any other features in Protégé. We implemented graph visualizations10

Figure 3 presents visualization snippets of indented list (Fig. 3a) and graph (Fig. 3b). In the indented list visualization, is-a relationships are illustrated by indentation and the expanders allow users to toggle children of a node. Participants can use horizontal and vertical scroll bars to adjust the viewing area. The presence of the expander itself illustrates whether a node is expandable. Given the similarities that indented lists share with computer file directories, it is challenging in practice to recruit participants who have never browsed a file directory prior to taking part in the study. Thus in an effort to minimize bias and achieve a study environment simulating novel users, we believe the indented list in Protégé is sufficient for the purpose of our study since none of the participants have used Protégé previously.

In the graph visualization, classes are illustrated by nodes and is-a relationships are illustrated by directional edges with arrowheads pointing to the subclasses. The coloring of a node denotes whether it is expandable (i.e., dark-colored nodes illustrate the existence of subclasses whereas light-colored nodes illustrate nonexpendable classes). Clickable nodes allow users to toggle children of a particular class. In addition to using scroll bars to adjust the viewing area, users can manipulate the visualization by dragging and dropping nodes to any location on the screen. In both visualization techniques, we presented only the ontology root initially and participants must expand the root to view the rest of the classes.

There can be various implementations to both indented lists and graphs, therefore, it is necessary to ensure that the visualizations used in our study are representative of current techniques. Since the visualizations used in this study each contain all the key characteristics shown in the literature (e.g. [9,22,43,55,68]), i.e. indented lists use indentation to illustrate super/sub-class relationships with one single path between any pair of ontological entities, and graphs are node-link diagrams that have nodes with connecting edges (discussed in Section 2.1), we believe they are sufficient for the purpose of the study. Also, as Shneiderman points out [58], visualizations should provide high-level overviews, and support zoom-ins on specific parts and filter-outs of irrelevant parts. In the indented list visualization, collapsed expanders allow an overview of the ontological structure with irrelevant information remaining hidden, while further expansion allows the user to focus on specific parts of the ontology. Similarly in the graph visualization, collapsed color-coded nodes present an overview of the ontology at a high level while keeping irrelevant details hidden, and expanded nodes allow the user to concentrate on specific parts of the ontology. Since the two visualizations used in our study each satisfy all of the above, we believe that the evaluation setup is reasonable and fair, and that bias is minimized since the indented list and the graph visualization are highly comparable in terms of functionalities.

We recruited a total of 36 participants in our study. The participants were undergraduate and graduate students enrolled in disciplines including computer science, biomedical, biochemistry, and mechanical, electrical, software engineering. All participants were novice users of semantic technologies and they were new to ontologies and ontology mapping. As users of ontologies and visualizations increasingly include people with little knowledge of semantic technologies (e.g., BioPortal [55] users are clinical and biomedical researchers who are new to ontologies and mappings), we were interested in studying the visualization support for novices. This group of users is of particular interest to us especially when a true expert should be able to successfully and accurately evaluate mappings regardless of the visualization support.

We carried out one-on-one sessions with participants, where a session lasted approximately two hours. In each study session, we asked the participant to first complete an online tutorial on ontologies and ontology mapping.12

Each participant began a study session by first calibrating her (or his) eyes. The calibration lasted 6 seconds and required the participant to look at a dot that appeared in different locations on the display. This process allows the eye tracker to accurately track a subject’s eye movements in the subsequent task recording, based on the calibration data that was collected while the fixation locations were known to the eye tracker. In addition, we asked each participant to look at an empty task screen (i.e., Fig. 1 without any visualization or data in the spreadsheet) for 10 seconds to collect her (or his) baseline pupil data, which was later compared to the pupil data generated during the task recording for pupil dilation analysis (discussed in Section 4.3). After calibration and baseline data collection, the participant was then given a task to complete, where the eye tracker recorded her (or his) eye movements from the beginning to the end of the task.

Each participant sat on a stationary chair, i.e. without wheeling, leaning, or swirling functions, during the calibration and throughout the task recording. This ensured that there was no sudden distance change from the eyes of a subject to the eye tracker. Participants were told to use the display as how they normally would with a computer screen. They could blink as usual and look down on their keyboards if necessary during a recording. Although the eye tracker is able to tolerate small changes in head positions (i.e. is able to track eye movements if the head of the participant moves no faster than approximately 10 cm/s), the participants were encouraged to reduce sudden large head movements to improve tracking accuracy. Furthermore, participants were asked to raise any questions before they began a task and no interactions were allowed with the researcher during a task.

4.1. Measures of search

Measures of search include fixation and saccade count, saccade length, convex hull, and scanpath length.

Fixations are those moments when the eyes are relatively stationary. Saccades occur between fixations, which are the quick eye movements from one fixation to the next. A scanpath is the complete saccade-fixate-saccade sequence. Figure 4 illustrates these concepts with a diagram. In good visualization designs, it is likely that the user will find relevant components quickly, where they only need to sample a small amount of objects to complete a task, consequently leading to a lower fixation count [20]. As the saccade count is essentially the number of fixations minus one, naturally, given low fixation counts, there will also be small saccade counts. In contrast, poor designs are likely to produce more extensive search behaviors, i.e. leading to higher number of fixations and saccades.

OPEN IN VIEWER

Fig. 4.

Fixation, Saccade, and Scanpath. Dots illustrate fixations, and lines illustrate saccades. The saccade-connected fixations illustrate the scanpath.

Given the X and Y coordinates of each fixation, we can determine the saccade length, the absolute angle that is associated with each saccade as well as the relative angles between saccades. An absolute angle (measured in degree) indicates the inclination of the saccade with respect to the X-axis. A relative angle indicates the gradient between two saccades. An example is presented in Fig. 5. Given the positions of fixation points n and k, coupled with the absolute degree β, we can determine the length L of the saccade, as shown in Fig. 5. A well-designed visualization is likely to provide useful cues to direct users’ scanning to desired targets rapidly with few interim fixation, meaning that it would generate larger saccadic amplitudes, i.e. longer average saccade length for a participant [20]. In the case of poorly designed visualizations that provide misleading or unhelpful cues, it is likely that the distances between saccades are smaller as the user continues to search near or around a component until a meaningful cue is found.

OPEN IN VIEWER

Fig. 5.

Saccade Length, Absolute and Relative Angle. L illustrates the saccade length between fixations n and k. α illustrates an example of relative angle, and β illustrates an example of absolute angle.

Knowing each saccade length, the scanpath length therefore can be determined as the sum lengths of all saccades. It is important to note that lengthy scanpaths are an indication of inefficient scanning behavior, but do not directly measure search efficiency since scanning does not distinguish between searching activities and processing activities. Thus, scanpath length should be understood in the context of the search area, or convex hull. Figure 6 shows an example of convex hull area. It is likely for a well-designed visualization to have a smaller search area coupled with a shorter scanpath. In contrast, a wildly distributed search pattern in poorly designed visualizations is likely to lead to a larger search area with longer scanpath [20].

OPEN IN VIEWER

Fig. 6.

The highlighted area shows an example of convex hull. The convex hull area is determined by the bounding fixations in the scanpath.

Measures of information processing include average fixation duration and saccade-to-fixation ratio.

Longer fixations indicate the user is spending more time interpreting or processing the objects presented [20]. In a well-designed visualization, it is likely that the user will have a relatively short average fixation duration, i.e. faster processing time. Knowing a sequence of fixation durations of a participant { f k } , her (or his) average fixation duration is simply the sum of all fixation durations divided by the fixation count n, as ∑ k = 1 n f k / n . Poorly designed visualizations are likely to yield longer average fixation durations, as the cues are not as helpful.

Given the timestamps of fixations, the duration of each saccade can be determined, as the time between two succeeding fixation timestamps. Knowing the time a participant spent searching for information, i.e. a sequence of saccade durations { s j } and the time she (or he) spent processing information, i.e. a sequence of fixation durations { f k } , we can determine the search-to-process ratio for this participant, as the sum of saccade durations divided by the sum of fixation durations: ∑ j = 1 n − 1 s j / ∑ k = 1 n f k . The higher this ratio, the more time the participant spent processing information, or the less time the participant spent searching for information. When comparing two visualizations’ support for information processing, the more effective visualization would have faster processing time combined with a low search-to-process ratio, as the users are likely to not only quickly find relevant information but also spend less time processing them, i.e. lower total fixation durations coupled with lower saccade-to-fixation ratio.

4.3. Measures of cognitive workload

Measures of cognitive workload include pupil dilation, absolute and relative saccade angles.

The changes in pupil sizes and more specifically, the increase of pupils are an indication of more cognitive effort [35], i.e. the more difficult or demanding it is to complete the task using a particular visualization, the larger the pupils will be. It is likely for a well-designed visualization to generate none or very small increase in pupil sizes compared to a poorly designed visualization. To determine the pupil size for a participant, we used the average of both eyes. To analyze pupil dilation, for each participant, we first calculated her (or his) baseline average pupil size of both eyes s while looking at an empty task screen. We then calculated her (or his) average pupil size of both eyes s ′ during a task. Finally, her (or his) pupil dilation is calculated as a percentage: ( s ′ − s ) / s .

Recall that relative angles (see Fig. 5) express the successive eye movements of a participant. Within a scanpath, the sum of absolute values of relative angles indicates the directness of the scanning [18]. In a well-designed visualization, it is likely that the participant will have a certain and confident scanpath throughout, as opposed to randomly scanning around with a great deal of uncertainty. Hence, when comparing two visualizations, the better-designed one is likely to lead to smaller sum relative angles. Absolute saccade angles (see Fig. 5) express the inclination of scan directions, and similar to relative angles, a confident scanpath with certainty would lead to consistent and less dispersed absolute angles. It is likely when the visualization is more demanding to work with (i.e. requiring more cognitive effort), the sum of relative angles would be higher and the standard deviation of absolute angles would be higher as well.

4.4. Effectiveness & efficiency

Measures of a participant’s speed (indirectly measuring the visualization’s efficiency) and accuracy (indirectly measuring the visualization’s effectiveness) include calculating her (or his) task success, error rate, and time on task.

We calculated success scores for a participant to reflect identification success (i.e., the activity focusing on evaluating the correctness), creation success (i.e., the activity focusing on evaluating the completeness) and overall success (i.e., combing both type of activities). For example, suppose a task presents a set of existing mappings between ontology O and O ′ , among which are n 1 number of correct mappings, n 2 number of incorrect mappings and n 3 number of missing mappings. If a participant successfully identifies x number of correct mappings and y number of incorrect mappings, then her (or his) identification success = ( x + y ) / ( n 1 + n 2 ) . If a participant correctly creates z number of new mappings, then her (or his) creation success = z / n 3 . Her (or his) overall success = ( x + y + z ) / ( n 1 + n 2 + n 3 ) . Her (or his) error rate is recorded as the number of incorrect answers divided by her (or his) total number of answers. Success scores range between 0 and 1; the higher the score the more successful the participant was at the task. Error rates also range between 0 and 1; the lower it is, the fewer mistakes the participant made in the task. Finally, time on task is the length of time it took a participant to complete the task (i.e. after the participant has finished both identification and creation activities).

4.5. Data cleaning, validation & statistical tests

As discussed earlier, raw data is associated with validity codes, which are an indication of how confident the eye tracker was that it had recorded the correct data. In an effort to reduce noisy data, we filtered out the raw data that were associated with poor validity codes. For example, when the eye tracker could not detect both eyes (e.g., data of one eye is associated with a poor validity code), we would then discard that entire row of raw data in our analysis.

Eye trackers can be sensitive to participants with eyewear, such as glasses and contact lenses. They can also have difficulty tracking eyes with certain characteristics such as large pupils, or when eyelids obscure pupils. We thus discarded incomplete data sets that had lost traces of some participants’ eyes during the recordings. We discarded 5 participants’ data during the data cleaning procedure, which led to a total of 31 valid data sets. Among which, 16 participants used indented lists to solve the conference task and graphs to solve the BioMed task, and another 15 participants used graphs to solve the conference task and indented lists to solve the BioMed task.

Given both fixed and random effects in the study, when differences are found between the two visualizations, we used MIXED models to validate statistical significances of our findings. For a comprehensive review on mixed-effects models, see [39]. In addition, we measured covariance between variables using the Pearson correlation coefficient [44] to determine the degree of associations among these variables. Following the guidelines presented by Taylor [66], in absolute values, we interpret weak correlations to be r ⩽ 0.35 , moderate correlations to be 0.36 ⩽ r ⩽ 0.67 , high correlations to be 0.68 ⩽ r ⩽ 1.00 , and very high correlations to be r ⩾ 0.90 .

Figure 7 presents box plots of the fixation counts that were generated for each participant. In both tasks, fixation counts for participants who used indented lists vary less than those who used graphs. The aggregated group mean fixation counts are shown in Fig. 8, where the average number of fixations is always lower for participants who used indented lists. This difference is statistically significant. Similarly, as expected, we found smaller number of saccades for those who used indented lists (see Fig. A1). These results suggest that participants needed to sample a fewer number of visualization components when using indented lists. In other words, indented lists were more effective at assisting participants to efficiently search for information compared to graphs.

When comparing the average saccade length, we did not find statistically significant differences between the participants who used indented lists and those who used graphs. It is important to note that non-statistically significant differences do not imply a lack of difference between the two user groups (i.e. they are the same). It simply means that the results are inconclusive, in that a difference may exist although it was not shown in the present sample.

OPEN IN VIEWER

Fig. 8.

Aggregated Group Mean Fixation Count. Mean values are labelled on each T-bar. The differences between the two visualization groups are statistically significant with p < .05 , effect size Ω 2 = 0.0656 .

OPEN IN VIEWER

Fig. 9.

Distribution of Convex Hull Areas. We calculated the convex hull area for each participant.

Figure 9 presents an overview of the convex hull areas found for each participant. The variations of convex hull were greater for participants who used indented lists. Figure 10 shows the aggregated group mean convex hull, where indented lists had a smaller area in the conference task, and graphs had a smaller area in the BioMed task. Convex hull areas should be understood in the context of scanpaths, as discussed in Section 4.1, i.e. more efficient search would have smaller convex hull area companied by shorter scanpath, we thus compared the scanpath lengths of the two groups next.

OPEN IN VIEWER

Fig. 10.

Comparison of Aggregated Group Means found in Convex Hull Areas. The differences are statistically significant with p < .02 , effect size Ω 2 = 0.1900 .

OPEN IN VIEWER

Fig. 11.

Distribution of Scanpath Lengths. For each participant, we generated the length of individual saccade. The sum of all saccades’ lengths is the scanpath length.

Figure 11 presents an overview of the data generated for scanpath lengths, and Fig. 12 presents the aggregated group means. Participants who used indented lists had consistently shorter scanpaths with less variations compared to those who used graphs. These results suggest that indented lists were generally more effective at supporting participants searching for information, since participants generally searched within smaller areas with shorter scanpaths. In contrast, even though in the BioMed task, participants searched within a slightly smaller area using graphs, but since they had much lengthier scanpaths, it suggests they had inconsistent and random searches and felt lost during the task.

OPEN IN VIEWER

Fig. 12.

Comparison of Aggregated Group Means of Scanpath Lengths. The differences are statistically significant with p < .002 , effect size Ω 2 = 0.0680 .

Figure 13 presents an overview of the distribution of the average fixation durations that were generated for each participant. We found that graphs had less variation of average fixation durations. Figure 14 presents the aggregated group means, which shows that the group who used graphs had shorter fixation durations on average. Since participants who used graphs generally spent less time interpreting visualization components, this result suggests that participants were consistently more efficient at processing information with less variations using graphs.

OPEN IN VIEWER

Fig. 13.

Distribution of Average Fixation Duration. The Y-axis illustrates the average fixation duration of a participant during a recording.

OPEN IN VIEWER

Fig. 14.

Aggregated Group Mean Fixation Duration. The differences between the two groups are statistically significant with p < .01 , effect size Ω 2 = 0.0265 .

When comparing the sum of fixation durations, we did not find statistically significant differences between the user groups. An average search-to-process ratio of 1.88 was found for the participants who used indented lists, and an average ratio of 1.83 was found for those who used graphs, although this difference is not statistically significant either. These results suggest inconclusive findings, however, since both visualizations yield ratios larger than 1.0, we can conclude that regardless of the visualization used, all participants spent more time searching for information than processing information (as otherwise, the ratios would have been smaller than 1.0).

When comparing pupil dilations, we found an average increase in pupil sizes from approximately 3.6077 mm to 3.6923 mm for participants who used indented lists, and to 3.6568 mm for participants who used graphs. However, the difference of these changes, 4.99% for indented list users and 4.93% for graph users, are not statistically significant. It was not clear whether one particular visualization technique was less demanding to use than the other.

OPEN IN VIEWER

Fig. 15.

Distribution of Standard Deviations found in Absolute Angles. For each participant, the standard deviation indicates how dispersed were her (or his) absolute angles.

OPEN IN VIEWER

Fig. 16.

Comparison of Aggregated Group Means of Standard Deviations found in Each Participant’s Absolute Angles. The differences are statistically significant with p < .0001 , effect size Ω 2 = 0.6650 .

When comparing absolute and relative angles, we found that those participants who used graphs consistently produced less varied absolute angles, as shown in Fig. 15. In addition, the variations are always smaller for those who used graphs, as shown in Fig. 16. However, we did not find statistically significant differences in the sums of relative angles. As discussed earlier (in Section 4.3), sum of relative angles should be understood in the context of dispersions of absolute angles, these results are therefore inconclusive. It is not clear whether cognitive workload was reduced using one particular visualization technique.

OPEN IN VIEWER

Fig. 17.

Distribution of Time on Task. We recorded the time each participant needed in order to complete a given task.

OPEN IN VIEWER

Fig. 18.

Comparison of Aggregated Group Mean Task Time. These differences are statistically significant with p < .02 , effect size Ω 2 = 0.1121 .

We found that participants who used indented lists were consistently faster when completing the tasks. As shown in Fig. 17, there were fewer variations in task time for those who used indented lists, and the aggregated group means were always shorter, as shown in Fig. 18.

However, participants were not necessarily more accurate with their tasks when using either visualization, as we did not find statistically significant differences in the correctness success (i.e. identifying an pre-existing incorrect mapping), completeness success (i.e. create a correct mapping that was previously missing in the spreadsheet), the overall success (i.e. taking both identification and creation success into account), or error rates. These results suggest that although task accuracy was not improved, those who used indented lists were faster at completing the given tasks.

5.5. Correlations

We found several moderate to high correlations, some of which are to be expected. For instance, fixation count is highly correlated with scanpath length (with r = 0.967 , p < .0001 ) and the sum of fixation duration (with r = 0.898 , p < .0001 ). Theses results are not surprising, considering more fixations will most likely increase their duration sum and add to the length of a scanpath. In this section, we present two most interesting correlations found.

OPEN IN VIEWER

Fig. 19.

Correlation between Average Saccade Length and Average Fixation Duration. Each scatter illustrates the average saccade length and the average fixation duration found for each participant. The brown dotted line indicates the overall trend of the correlation between these two variables irrespective of the visualization, where r = − 0.291 and p < .03 . The blue and green lines illustrate the correlation of the two variables found in the intended list user group and the graph user group respectively.

OPEN IN VIEWER

Fig. 20.

Correlation between Convex Hull Area and Fixation Count. Each scatter illustrates the convex hull area and the total number of fixations for each participant. The brown dotted line indicates the overall trend of the correlation between these two variables irrespective of the visualization, where r = 0.343 and p < .01 . The blue and green lines illustrate the correlation of the two variables found in the intended list user group and the graph user group respectively.

We found a weak negative correlation between the average saccade length and the average fixation duration. As shown in Fig. 19, as the participant fixated on visualization components that were further apart from one another, she (or he) also spent less time fixating on these points of interest. As longer saccade lengths suggest more useful cues that direct one’s attention with few interim fixations (discussed in Section 4.1), and since Fig. 19 shows the correlation to be stronger for graphs, it suggests that graphs were more effective at providing useful cues and supporting information processing than indented lists.

We found that as the number of fixations increased, the convex hull area also increased, as shown in Fig. 20. This correlation is stronger for participants who used indented lists. Note that the trend lines of the two visualizations cross at approximately (6000, 1200000), which suggest that for a total fixation count less than 6000, participants focused on a smaller area of interest when using indented lists. For those who had fixation counts greater than 6000, graphs generated smaller areas of interest. These results suggest that typically, graphs led to more widespread and less efficient searches and indented lists were more effective at assisting participants with concentrated and localized searches.