Visualization model validation via inline replication

Abstract

Data visualizations typically show a representation of a data set with little to no focus on the repeatability or generalizability of the displayed trends and patterns. However, insights gleaned from these visualizations are often used as the basis for decisions about future events. Visualizations of retrospective data therefore often serve as “visual predictive models.” However, this visual predictive model approach can lead to invalid inferences. In this article, we describe an approach to visual model validation called Inline Replication. Inline Replication is closely related to the statistical techniques of bootstrap sampling and cross-validation and, like those methods, provides a non-parametric and broadly applicable technique for assessing the variance of findings from visualizations. This article describes the overall Inline Replication process and outlines how it can be integrated into both traditional and emerging “big data” visualization pipelines. It also provides examples of how Inline Replication can be integrated into common visualization techniques such as bar charts and linear regression lines. Results from an empirical evaluation of the technique and two prototype Inline Replication–based visual analysis systems are also described. The empirical evaluation demonstrates the impact of Inline Replication under different conditions, showing that both (1) the level of partitioning and (2) the approach to aggregation have a major influence over its behavior. The results highlight the trade-offs in choosing Inline Replication parameters but suggest that using $n = 5$ partitions is a reasonable default.

Keywords

Visual analytics information visualization replication validation prediction

Introduction

Visualization tools are most often designed to depict the entirety of a data set—subject to a set of filters applied to focus the analysis—as accurately as possible. In this typical pattern, the goal is to provide the user with an accurate understanding of all of the data in the underlying data set that matches the active set of filters. This ethos was captured, perhaps most famously, in Shneiderman’s Visual Information Seeking Mantra: overview first, zoom and filter, then details-on-demand.¹ Variations of this basic approach have since been adopted in most modern visualization systems.

The foundations for these systems are visual mappings that specify a graphical representation for the underlying data. For small and low-dimensional data sets, these mappings can be direct (e.g. a scatterplot for a small two-diemsional (2D) data set). As problems grow in data size or dimensionality, algorithmic data transformation methods can be used to filter, manipulate, and summarize raw data into a more easily visualized form.

On top of these mappings, interactive controls are often provided to give users even more flexibility to filter or zoom to specific subsets of data. These interactions can be linked to more detailed information about data objects, such as via levels-of-detail or multiple coordinated views. The result, when well designed, is an effective visual interface for data exploration and insight discovery.

For this reason, these steps form the core stages of the canonical visualization pipeline.^2,3 This approach can be enormously informative and has led to advancements in how people seek to understand information across a wide range of domains, such as helping computer users navigate through large file systems, visualizing medical records to help doctors understand patient histories, and visualizing weather data to identify regions most impacted by a recent storm.

Critically, these visualization use cases are all retrospective in nature. More specifically, they employ visualizations that attempt to faithfully report data exactly as it was observed. Users aim to see an overview of the entirety of a given data set. If a user applies constraints to focus the visual investigation (e.g. via zoom and filter), the visualization is expected to show the full set of data that satisfies the applied constraints.

In many visualization scenarios, however, users are in fact more interested in conducting prospective analysis: using visualizations of these historical data to reason about future or not-yet-observed data. For example, medical experts examining data for a cohort of patients might be most interested in what treatments would work best for a future patient with similar characteristics. Visualizations of historical sports statistics are often used to inform strategic decisions that are used in upcoming competitions. Financial visualization tools are often used to inform future investment decisions. In each of these use cases, visualizations of historical data are used to inform future decisions. For such prospective analysis tasks, these retrospective visual depictions of the data are often used, in essence, as naïve visual predictive models, with the assumption that visualizations of historical data can be used to infer or predict future observations. In many ways, these visual models are analogous to statistical models developed during statistical analyses, from which users often also attempt to predict future observations.

In many cases, retrospective representations are indeed very informative. However, just like the underlying descriptive statistics that such visualizations often depict,⁴ traditional retrospective visualizations often provide insufficient evidence for making predictive inferences, even as the visual depiction itself might be especially suggestive for making such inferences. In many cases, the trends and patterns that a visualization of retrospective data present to a user may be artifacts of noise and expected randomness within the underlying data. For users to make valid inferences or predictions based on historical data, a more nuanced understanding of the data being visualized is required.

This critical gap between (1) retrospective visualization designs and (2) the predictive requirements of many users has been recognized within the visualization community.⁵ Some have attempted to bridge this gap by adding support for inferential statistics within the visualization. Typically, this approach combines carefully designed statistical models with visualizations of the model’s results. For example, visualizations can be instrumented to estimate and display uncertainty, confidence intervals, or statistical significance. Alternatively, predictive modeling methods can be used to generate additional data, with the predictions themselves being incorporated into the visualization. These systems go beyond traditional descriptive reporting, but typically require a careful and sometimes onerous focus on modeling, including estimating underlying statistical distributions, which is incompatible with many applications for which a more accurate assessment of the repeatability of a given visualization would be useful.

This article presents Inline Replication (IR), an alternative approach to enabling inferential interpretation that is designed to overcome the above challenges. We introduce a partition function within the visualization pipeline to produce multiple folds for each visualized data subset. Metric functions are applied to each fold, and an aggregation function combines the individual measures prior to visualization. Interaction techniques enable users to examine both aggregate and individual fold metric values.

Our method is motivated by the bootstrap sampling and cross-validation techniques used widely in the statistics and machine learning communities. The IR approach is non-parametric, making it easy to apply and use generically within a visualization system without arduous modeling or assumptions about distribution parameters. IR integrates easily with the standard visualization pipeline and is also ideal for use in large-scale visualization systems where progressive or sample-based approaches are required. Finally, our method provides users with validation information that is both intuitive and easy to interpret. (This article includes material described in a pre-print that has been released by the authors via arXiv.⁶ This version of the article is heavily revised and contains an entirely new evaluation section describing and analyzing results from a set of empirical experiments to test the behavior of IR under various conditions.)

The remainder of this article is organized as follows. It begins with a review of related work, then describes the details of the IR methodology. After that we present an experiment that aims to test the accuracy of such method against traditional statistical analysis. We then share example results from a variety of proof-of-concept systems that include the IR technique. These examples range from simple bar charts to more sophisticated interactive visualizations of large-scale event data collections.⁷ The article concludes with a discussion of limitations and outlines key areas for future work.

Related work

The IR approach to visual model validation is informed by advances in several different areas of research. These include the topics of uncertainty, predictive visualization, and progressive or incremental visualization. Also relevant are visualization systems that utilize inferential statistics methods and conceptual models of the visualization pipeline.

Visualization of uncertainty

The visualization of uncertainty has been an active research area within the visualization community for many years. Studies have explored the problem from many perspectives, including developing taxonomies that have examined types of uncertainty⁸ as well as visualization methods for conveying uncertainty.⁹ In addition, there have been many efforts to formally study alternative methods for depicting uncertainty measures^10–12 through user studies that explore the perceptual understanding of various uncertainty representations. However, these studies focus on the visual representation rather than methods for determining the degree of uncertainty.

Perhaps most relevant to the IR approach proposed in this article is work that has focused on estimating uncertainty via measures of entropy within a data set rather than using carefully constructed statistical models.¹³ Compatible with IR, this work proposes using entropy as a non-parametric measure of uncertainty for categorical data which do not require formal modeling nor make assumptions about specific distributions within the data. IR provides a broader and more general framework for this approach, within which the entropy metric could be easily adopted.

In other work, the distinction between the “visualization of uncertainty” and “the uncertainty of visualization” has been highlighted.¹⁴ The latter is a related but separate concept from traditional uncertainty visualization. Such work highlights that the rendered graphics of a visualization can convey a sense of authority which may not be warranted, even when the underlying data itself are considered to be beyond reproach. This challenge is a key motivation for IR, especially with respect to the confidence of the user in making predictions based on this unwarranted authority, as outlined in the discussion presented in section “Visualization as a predictive model.”

Finally, we distinguish IR’s focus on variation of data over partitions to challenges related to non-representative samples. Non-representiveness and selection bias are important threats to visualization validity,¹⁵ and recent research has proposed methods to address these challenges within the context of interactive exploratory visualization.^16,17 IR can complement these methods but focuses on issues of variation rather than representativeness.

Predictive visual analytics

Visualization has long been used to support predictive analysis tasks. However, most often, the “prediction” is performed by users reviewing historical data and making assumptions about what might happen in the future for similar situations. In fact, the relatively limited history of work on visualizations that incorporate more formal predictive modeling methods was the topic for a workshop at a recent IEEE VIS Conference.⁵

The work that does exist in this area has often focused on model development and evaluation rather than supporting end users’ predictive analysis tasks. For example, BaobabView¹⁸ supported interactive construction and evaluation of decision trees. More recent work has focused on building and evaluating regression models.¹⁹ This method, like ours, adopts a partition-based approach to avoid making structural assumptions about the data. However, the focus on building regression models leads to an overall workflow that is very different from the proposed IR approach.

Others have focused on visualizing the output produced by predictive models. For example, Gosink et al.²⁰ have visualized prediction uncertainty based on formalized ensembles of multiple predictors. This approach, however, requires careful modeling to develop the predictors, including the specification of priors that enables the Bayesian method that they propose.

Outside the visualization literature, where novel visual or interaction methods are not a concern, predictive features are typically visualized using traditional statistical graphics, for example, systems that visually prioritize and threshold p values to rank features for prediction.²¹ Such methods are fully compatible with the IR process proposed in this article.

Progressive/incremental visualization

Model overfitting and other sampling challenges are common to “Big Data” visualizations that rely on progressive or incremental techniques.^22,23 Initial samples are small, grow over time, and can change in distribution as time proceeds. Some have addressed this challenge by including confidence intervals along with partial sets of query results.²⁴ However, relying on the query platform to assess confidence in data subsets does not easily support interactive zoom and filter operations after the query because these changes in visual focus do not necessarily result in new queries that generate new result sets. Moreover, these papers do not propose methods for computing confidence intervals, but rather assume that such data will be provided by the the database.

Inferential statistics

Statistical inference is a discipline with a very long and distinguished history. Most relevant to the IR method described in this article are challenges related to statistical significance and null hypotheses, and in particular, Type 1 and Type 2 errors. Type 1 errors refer to improper rejections of the null hypothesis which lead to conclusions that are not real effects, while Type 2 errors refer to falsely retaining the null hypothesis which can lead to assumptions that a true effect is false.²⁵

These types of errors are of critical concern in high-dimensional exploratory visualization where computational methods can quickly assess vast numbers of dimensions for statistical significance. Statistical correction methods have been proposed to reduce Type 1 errors,²⁶ but arguments have also been made against such approaches. Those arguments suggest that parameterized models or assumptions of “default” null hypotheses do not match real-world situations where distributions are rarely straightforward or independent. Suggesting that these correction methods are the wrong approach for exploratory work, Rothman²⁷ argues that “scientists should not be so reluctant to explore leads that may turn out to be wrong that they penalize themselves by missing possibly important findings.”

This tension is present in many interactive exploratory systems which make it easy to generate vast numbers of potential hypotheses. As a result, a wide range of methods have been proposed for modeling measures of confidence or significance.^28–30 These efforts, however, typically rely on formal statistical methods that make assumptions about distributions and variable independence. For example, confidence intervals have some conceptual similarities to IR. However, calculating a confidence interval requires assumptions about underlying distributions of the data and knowledge of key distribution parameters such as mean and variance. Such approaches are problematic for exploratory visualizations which enable users to rapidly apply filters or constraints that can quickly change the underlying assumptions. The IR method we propose enables users to visually assess the reliability of hypotheses, providing a high degree of flexibility. Similar approaches that rely on user judgment have been shown to be quite effective.^31,32

Models of the visualization pipeline

The traditional visualization pipeline model describes the process of transforming raw data first to an analytical abstraction, then to a visualization abstraction, and finally to a rendered graphic for interaction.^2,3 We add partitioning and aggregation stages to support the IR approach. As we will describe, a special case of the IR model (with just one partition) is equivalent to the traditional model. By extending the canonical pipeline, our work has similarities with Correa et al.’s³³ paper describing pipeline extensions for an uncertainty framework focused on the data transformation process. However, unlike Correa at al., IR proposes a different set of extensions which adopt an empirical approach rather than assuming a Gaussian or any other formal error model.

Visualization as a predictive model

Visualization design is often conceptualized as a mechanism for reporting. This retrospective approach is so ubiquitous that terms such as prediction, forecast, and inference cannot be found within the indices of many leading visualization texts from the past 25 years.^34–36

Many visualization consumers, however, use graphical representations of historical data as the basis for decisions about future performance. This is done even when the underlying data and transformations do not support such prospective conclusions. Despite potentially fatal flaws in terms of generalizability and repeatability, retrospective visualizations are in essence being used as predictive models.

The tendency to assume predictive power in visualization can be seen, for example, in modern casinos. Roulette wheels commonly include an electronic display³⁷ which shows the table’s recent history. Assuming a fair table, “red” and “black” numbers should be equally likely to appear. However, as illustrated in Figure 1, the history provided to gamblers is not sufficiently long to learn if the table has any systemic bias.

Figure 1.

Roulette wheels allow users to bet on “black” or “red” squares. Casinos often display a simple visualization of “recent spins” to provide gamblers with a false sense of predictive knowledge. In this example, the display shows a recent preponderance of black numbers with the implication to gamblers that this may influence future spins of the wheel.

Why then is the gambler presented with a simple visualization of the history? The data are visualized to provide gamblers with a false sense of knowledge, to suggest to a hesitant gambler that a bet is an informed decision rather than a random choice. A gambler may infer that the recent streak of black suggests more black spins will soon appear. Alternatively, the gambler may infer an imminent return to red. To the casino, it does not matter what predictive inference is drawn as long as it provides a false confidence that leads to increased betting.

It is tempting to dismiss this scenario as one in which the gambler should be more informed about basic statistics. The small sample size and the independence of each roulette spin should make it clear that the display is not especially informative. However, relatively sophisticated users performing visual analysis of data from more complex systems can make similarly poor predictive assessments on the basis of visual representations that do not properly convey the underlying limits of their predictive power.

For example, consider a business analyst attempting to learn about why sales are declining or a physician using historical patient data to compare treatment efficacy. In these complex real-world cases, in which it is essentially impossible to fully understand the underlying statistical processes, it is natural for analysts to turn to visualization as a predictive model for their problem. Visualization enables these users to see what has happened and, based on trends or patterns in the visual representation, make assumptions about what will happen in the future.

However, just as the casino gambler draws inference from a not-so-meaningful visualization, these power users can be led to make poor predictions on the basis of visualizations that are essentially “overfit” models based on poor visual representations of the underlying process. This problem has even been documented in highly quantitative fields such as epidemiology, where public health analysts have had trouble discounting statistics from small sample sizes when visualized.³⁸

Issues of poor sampling and overfitting are especially problematic during exploratory visualization in which users can interactively apply arbitrary combinations of filters to produce new ad hoc subsets of data for visualization. Such systems are at greater risk of generating misleading visualizations that occur “by chance” rather than due to real properties of the underlying problem.³⁹ The same is true for visualization systems that use sampled or progressive queries to address issues of scale.

The potential for this sort of “visual model overfitting” is analogous to the overfitting problem in more traditional modeling tasks. In the machine learning community, this is addressed in part by cross-validation, a widely used technique for assessing the quality and generalizability of a model.⁴⁰ Rather than relying on a single model solution, cross-validation methods create and compare multiple solutions, one for each of several partitions of a data set (often called “folds”). This enables an assessment of model repeatability, with models that work consistently across partitions considered more trustworthy. Similarly, bootstrap sampling techniques in statistics⁴¹ produce multiple samples from a single set of observations in order to derive better estimates of the original sample’s statistical properties.

If one considers—as we argue here—that a visualization is often used as a form of predictive model, then validation becomes a critical guard against problems associated with visual model overfitting. When a visualization is zoomed and filtered to focus on a specific subset, is the visual representation repeatable, or is it due to chance variation? Are the conclusions drawn from the visualization generalizable? The IR method outlined in the next section seeks to help answer such questions by embedding an approach similar to cross-validation and bootstrap sampling within the visualization pipeline so that each new view produced during user interaction can be evaluated for validity.

IR

IR is an approach to visualization in which the data set associated with each visualized measurement is partitioned into multiple subsets, or folds, processed independently to calculate derived statistics or metrics, then aggregated back together to be rendered in a visualization. This partitioned approach embeds an automated and non-parametric workflow for data replication within the visualization pipeline, as illustrated in Figure 2. The result is that visualizations based on IR can provide users with important information about the repeatability of observed visual trends, reducing the likelihood of certain types of erroneous conclusions.

Figure 2.

The Inline Replication (IR) visualization pipeline sends each derived measure’s subset of data $(m_{i})$ through a partition function to create multiple folds $(f_{j})$ prior to mapping and visualization. A metric function is applied to each fold independently, and an aggregation function recombines the folds to form an aggregate measure $(m'_{i})$ for subsequent visualization and interaction.

The IR pipeline begins with the same initial step as a traditional visualization pipeline. A set of query or filter constraints is first applied to a primary data source D to produce a focused data set $d \subset D$ . The data in d are then organized into subsets for which statistical measurements are calculated, creating measure-specific data subsets, $m_{i}$ . For example, a visualization pipeline configured to generate the inset bar chart in Figure 1, showing the distribution between black and red spins for a roulette wheel, would include the subset $m_{recent}$ containing data for the most recent spin results (black or red). If the visualization included multiple bar charts (e.g. past 10 spins, past 100 spins, and past 1000 spins), then multiple subsets $m_{i}$ would be defined because each requires the calculation of a distinct set of measurements.

Traditionally, the data for each $m_{i}$ would immediately be processed to compute the measurements required for visualization (e.g. the fraction of spins resulting in black and the fraction of spins resulting in red). Those measures would then be mapped to visual properties of objects within the visualization (e.g. the size of each bar in the bar chart).

The IR pipeline, however, behaves differently. Each $m_{i}$ is first partitioned into a set of distinct folds, $F_{i}$ , where each fold $f_{i}^{j} \in F_{i}$ is analyzed independently via a metric function. The results are then aggregated to form a merged data set $m_{i'}$ . It is this merged representation of the measures that is mapped to the visual representation and rendered to the screen for interaction using methods designed to convey the repeatability of the visual model across each of the folds.

This section provides an overview of the IR pipeline, focusing on the three functions at the core of the design: the partition function, the metric function, and the aggregation function. It then describes the IR approach to visual display and interaction and concludes with a discussion of useful variations to the core design.

Partition function

Conceptually, the partition function is designed to subdivide the data in a given measure-specific subset $m_{i}$ into multiple independent partitions or folds. These folds are used as the basis for calculating each measurement. Later in the IR process, derived measures (e.g. proportions or statistical significance) will be calculated for each fold.

Formally, we define the partition function as an operator that subdivides a measure-specific set of data $m_{i}$ into n folds such that each fold $f_{i}^{j} \subset m_{i}$

Partition (m_{i}, n) \to {f_{i}^{0}, f_{i}^{1}, \dots, f_{i}^{j}, \dots, f_{i}^{n - 1}}

(1)

This function is applied to the raw data in $m_{i}$ , prior to any other aggregating transformations (such as the summation in the roulette example). Following an approach inspired by k-fold cross-validation,⁴⁰ the baseline partition function creates n folds that are disjoint, approximately equal in size, and randomly partitioned such that

⋃_{j = 0}^{n - 1} f_{i}^{j} = m_{i}

(2)

As discussed previously, multiple folds are created with the goal of supporting repeated calculations for each measure. Increasing the value of n to produce more folds increases the replication factor. However, higher n values also produce smaller $| f_{i}^{j} |$ . If n is too large for a given $m_{i}$ , the folds may be too small to compute useful measures. Therefore, n can be dynamically determined so as to require a minimum fold size. If $m_{i}$ represents a “large enough” subset of data, it will produce a full set of folds. If, however, $m_{i}$ is too small for the minimum fold size, fewer than n folds will be produced. The threshold for “large enough” depends on many factors, including the specific metrics that will be calculated.

Partitioning with $n = 1$ results in the identity partition function where $f_{i}^{0} = m_{i}$ regardless of the size of $m_{i}$ . Because no partitioning is performed, an IR process using the identity partition function produces results that are identical to a traditional visualization pipeline: a single metric is calculated and visualized. In this way, the traditional approach to visualization can be seen as a special case of the IR process containing only onefold.

Choosing an appropriate n is necessarily a compromise between increased replication and smaller sample size. We can look to the machine learning community for guidance, however, where empirical studies have shown that there is no meaningful benefit for values of n over 10.⁴⁰ Moreover, as data sets grow larger in many fields, smaller sample size becomes less of a concern.

Finally, there are certain conditions (e.g. very small data sets with little data to partition or very large data sets where sampled queries are required) where the basic formulation for the partition function can be problematic. Variations to the partitioning process, designed to help address these challenges of scale, are discussed in section “Variations.”

To illustrate the partitioning process, consider the roulette example from earlier in this article. The example bar chart showing the fraction of spins resulting in black or red is based on a single measure-specific subset of data $m_{recent}$ . The partition function would be applied to this subset to create a set of multiple folds, each of which would contain a subset of the recent spin results. For example, $Partition (m_{recent}, 5)$ would produce a set of fivefolds, each containing results from roughly one-fifth of the overall set of recent spins.

Metric function

The folds produced during partitioning are sent to a metric function which is applied independently to each fold as illustrated in Figure 2. The metric function computes derived statistics $s_{i}^{j}$ for each fold $f_{i}^{j}$ . These derived statistics (one $s_{i}^{j}$ for each of the n folds $f_{i}^{j}$ ) can then be aggregated and compared during the visualization rendering process.

The specific measures computed by the metric function are application specific but could range from simple descriptive statistics (e.g. sums, averages) to more complex analyses (e.g. classification, regression). Generally speaking, metric functions produce the same derived values that would normally be computed as part of a more traditional visualization process. The key difference in IR is that the metrics are computed n times for each $m_{i}$ (once for each fold), where traditionally such values would be computed just once. We also note that multiple statistics might be computed for each fold. For example, a system might compute both the average and a linear regression for each fold.

For example, consider the roulette use case described earlier. The metric function in this example would compute the fraction of spins resulting in black and red in each fold $f_{i}^{j}$ . This fraction is the same measure that the original bar chart is designed to display. However, with the IR approach, the metric is calculated for each of the fivefolds produced by $Partion (m_{black}, 5)$ .

An actual implementation of IR using a similar “fraction of the population” metric function is discussed in section “Use cases.” However, more sophisticated systems may adopt more advanced measures. For example, correlation statistics, p values, metrics of model “fit,” and regression lines are all compatible with the IR approach. Examples of IR using linear regression, correlation, and statistical significance testing are also described in section “Use cases.”

Aggregation function

The metric function produces a set of statistical measures $s_{i}^{j}$ corresponding to the set of n folds $f_{i}^{j}$ that are produced by the partition function. These multiple measures could, in principle, be visualized directly. However, this would introduce complexity to the visual representation. Therefore, prior to visualization, the multiple $s_{i}^{j}$ metrics can be aggregated to a single representation $m'_{i}$ , inverting the partition process to produce a simple aggregate summary. We note that the detailed $s_{i}^{j}$ metric values are retained in this process and can be made visible through the visualization via unfolding as described in the next section. As illustrated in Figure 2, the values are combined via an aggregation function which we define as follows

Aggregate (s_{i}^{j}) \to m'_{i}

(3)

A variety of aggregation algorithms can be employed, with different approaches appropriate to different types of metrics. For example, for count-based metrics which capture the frequency of data items in each fold, a summation across all folds might be the most appropriate because a sum of counts for each fold provides an accurate total for the overall data subset $m_{i}$ . For a metric that captures a mean or rate, averaging the values across all folds may be most appropriate. For categorical metrics, meanwhile, such as those produced by classification algorithms, a “majority-vote” aggregation method⁴² can be applied to capture the most frequently assigned category. The same voting approach can be used when aggregating thresholded measures (e.g. tests of statistical significance) across all folds. This approach is demonstrated in section “DecisionFlow2.”

The aggregation function produces a single aggregate measure $s_{i}$ which is derived from the set of fold-specific measures $s_{i}^{j}$ . These are combined to form a merged data representation $m'_{i} = (s_{i}, {s_{i}^{j}})$ which is then used in the mapping and rendering process of the final visualization.

As a concrete example, consider again the roulette scenario. The metric function described previously computed the fraction of spins resulting in black and red numbers for each of the fivefolds created by $Partition (m_{recent}, 5)$ . The aggregate rates for both colors are obtained by averaging the fivefold-specific rates. The overall average, along with the individual values for each fold, is combined to form $m'_{recent}$ .

Visual display and unfolding of partition data

Once aggregation has been performed, the merged data $m_{i'}$ are mapped to their corresponding visual marks and displayed as part of the visualization, as shown in the final step in Figure 2. The IR approach to visualizing $m_{i'}$ has two components, which correspond to the two distinct types of information in the merged data structure: (1) the aggregate statistical measure, $s_{i}$ , and (2) the individual fold measures, ${s_{i}^{j}}$ .

First, an initial visualization is created using only the aggregate measures. The process for this stage is similar to a traditional visualization pipeline. The aggregate measures are mapped to visual properties of the corresponding graphical marks, which we call aggregate marks. These marks are then rendered to the screen for display and interaction. In the roulette example, for instance, the aggregate data for black and red spin rates (produced by the Aggregate function) can be used to generate a basic bar chart that is identical to what is shown in Figure 1.

Second, an IR visualization enables aggregate marks to be unfolded. An unfolding operation—typically triggered by a user interaction event such as selection or brushing—augments the aggregate marks with a visualization of the individual fold statistics that contribute to the aggregate measures. In the ongoing roulette example, the fold data would show the variation in proportion of spins that result in black and red numbers across each of the $n = 5$ independent folds. Additional examples from our experimental prototypes are described in section “Use cases.”

Discussion

The ability to unfold aggregate measures into repeated measurements is a central contribution of the IR approach. By graphically depicting the repeatability of a particular measure across multiple folds, IR provides users with important and easy-to-interpret cues as to the variability of a given measure. Traditional visualization methods do not convey this information, meaning it is often not considered when predictive conclusions are made by users.

Another benefit of IR comes from the aggregation function. In particular, embedding within the visualization pipeline an ability to aggregate categorical values such as statistical significance classification can lead to more accurate results. Repeated measures combined with voting-based aggregation can, for instance, reduce the exposure to Type 1 errors when looking for statistically significant p values. For example, a statistically significant $(p < 0.05)$ run of black spins on the roulette wheel is less likely to occur “by chance” across a majority of n folds than it is across a single group of spins. This is a major benefit for exploratory visualization techniques that enable users to visually “mine” through large numbers of variables searching for meaningful correlations. The results of an experiment exploring how IR can help reduce Type 1 errors with respect to correlation are presented in section “Empirical experiments.”

Variations

Following the traditional approach to k-fold cross-validation, the baseline Partition function defined in section “Partition function” specifies that the constructed folds are disjoint, randomly partitioned, and exhaustive (equation (2)). However, relaxing these constraints leads to several valuable variations to the baseline IR procedure.

Partial partitioning

Relaxing the requirement of equation (2) enables the creation of partitions that do not contain all data points within $m_{i}$ . For very large data sets, this can enable approximate analyses that use only a subset of the available data. This approach provides significant performance benefits for metric functions that have poor scaling properties and enables IR to work directly with recently proposed techniques for progressive visualization.^22,23

Partitioning with replacement

Relaxing the requirement that all folds are disjoint enables partitioning with replacement. Similar to bootstrap resampling,⁴¹ this approach enables the same data point to be included in multiple folds (or even multiple times within the same fold). When using replacement, the data set in $m_{i}$ becomes a sample distribution from which the partitioning algorithm can generate a larger population. This is especially useful for small data sets—a frequent occurrence in exploratory visualization in which multiple filters can be quickly applied—because the larger generated population can enable the IR process to run with less concern about producing fold sizes that are too small.

Incremental partitioning

A number of progressive or sampled methods have been proposed in recent years to address the challenges of “Big Data” visualization.^23,24 In these approaches, the full data set $m_{i}$ is often never fully retrieved. To utilize an IR approach in these cases, an incremental partitioning process is needed. During this process, data points should be distributed to folds as they are retrieved such that all n folds are kept roughly equal in size. This will enable IR to work with continuing improvement in metric quality as more data arrive. However, we note that IR will not overcome selection bias that may be introduced as part of the progressive query process. Therefore, the determination of a progressive sampling order that is both representative and balanced remains a critical concern.

Empirical experiments

A series of simulated data experiments were conducted to empirically measure the behavior of IR in conditions where—unlike many real-world situations—accurate ground truth information about data distributions was available. The experiments focused on evaluating the impact of IR with respect to Type 1 and Type 2 errors for a common type of analysis: correlations between two data sets.

Data generation

To simulate empirical experiments under varying conditions, two different types of base data distributions were created as illustrated in Figure 3. These base distributions include a unimodal test case with simple unimodal normal distributions—the typical assumption for traditional statistics—and a multi-modal test case designed to test how IR behaves when the unimodal assumption is broken—a common occurrence in real-world data sets.

Figure 3.

Empirical experiments measured IR behavior under two different conditions: (a) data drawn from simple unimodal normal distributions and (b) data drawn from a more complex condition of two non-aligned multi-modal Gaussian distributions. This figure illustrates the distributions from which the eight data sets used in the empirical experiments were drawn.

A second factor in the empirical experiments was the amount of known correlation in a data set. The experiment tested four levels of correlation: strong correlation (Pearson’s r = 0.61), moderate correlation (r = 0.25), weak correlation (r = 0.1), and completely random data with no correlation (r = 0.0).

This two-by-four experimental design (two types of base distributions, four levels of correlation) resulted in the creation of eight artificial data sets—eight D in the IR nomenclature—as summarized in Table 1. Each D consists of 5000 $(x, y)$ data points drawn from the corresponding base distributions (unimodal versus multi-modal) to achieve the pre-determined level of correlation. These data sets served as the starting point for the empirical experiments described below.

Table 1.

The empirical evaluation utilized eight artificially generated data sets corresponding to the two-by-four experimental condition design of the experiments (two distribution types, four correlation levels).

Pearson correlation	Unimodal	Multi-modal
Strong correlation (r = 0.61)	$D_{uni}^{strong}$	$D_{mult}^{strong}$
Medium correlation (r = 0.25)	$D_{uni}^{med}$	$D_{mult}^{med}$
Weak correlation (r = 0.1)	$D_{uni}^{weak}$	$D_{mult}^{weak}$
No correlation (r = 0.0)	$D_{uni}^{none}$	$D_{mult}^{none}$

Experiments

For each D, 10,000 randomized IR experimental trials were conducted for each of the $4 \times 5 \times 3 = 60$ experimental conditions (four settings for the number of IR folds n, five settings for the p value threshold in the metric function, and three settings for the type of aggregation function). The results of these trials were then analyzed to characterize how IR behaves under the various conditions.

In each trial, 1000 $(x, y)$ data points were randomly sampled from D to create a focused data subset $d_{i}$ where $i \in [0, 9999]$ . We note that while each D has a known correlation value between x and y, the correlation for each random subset $d_{i}$ varies due to the random sampling process. Figure 4 shows a summary of the correlations found in the 10,000 $d_{i}$ for each of the eight D in our experiments. As expected, the mean correlation for each set of 10,000 $d_{i}$ was aligned with the correlation value present in the corresponding D. Moreover, because we aim to test only a single statistic (correlation) in each trial, we treat each $d_{i}$ as a single measurement-specific subset $m_{i}$ in the IR pipeline.

Figure 4.

For each of the eight artificially generated data sets D, we created 10,000 randomly sampled subsets $(d_{i})$ of size 1000 to use in our experiments. Due to the random sampling process, the correlation present in each $d_{i}$ varied slightly from the correlation in D. Each plot in this figure shows a histogram of the correlation values for the 10,000 $d_{i}$ , with the correlation for the corresponding D shown as a dotted black line.

A first set of experiments was designed to explore three different parameters of the IR process. First, IR was applied to each $m_{i}$ with four different n values (1, 3, 5, and 7) during partitioning. When IR is used with $n > 1$ , the data are randomly split into n disjoint folds. Second, the metric function applied in the experiments was a significance test for Pearson correlation for the x and y values in each $m_{i}$ . This was chosen because it is a commonly used metric in real-world systems, including the DecisionFlow2 system described in section “Use cases.” The metric calculated the Pearson correlation for each fold and computed a p value against the null hypothesis, $h_{0}$ , that the data points in the fold are not correlated. Tests were performed at five different threshold levels (0.005, 0.01, 0.02, 0.03, and 0.04). Finally, three different aggregation methods were used to generate $m'_{i}$ for each $m_{i}$ : (1) any pass, in which we reject $h_{0}$ if any fold passes the p value threshold test; (2) majority vote, in which we reject $h_{0}$ if more than half of the folds pass the test; and (3) all pass, in which we reject $h_{0}$ only if all folds pass the test.

A second set of experiments was designed to characterize IR’s behavior in response to different numbers of data points in $d_{i}$ . The previously described experiments maintained a constant size of 1000 for each $d_{i}$ while (1) varying the number of folds and (2) drawing samples for D with different levels of correlation. In the second round of experiments, the number of folds was kept constant at $n = 5$ and the samples were all drawn from a D with a correlation of $r = 0.25$ . However, in these experiments, the size of $d_{i}$ was varied with sizes of 100, 500, 1000, 1500, and 2000. A similar set of any-pass, majority-vote, and all-pass experiments were performed for both the unimodal and multi-modal distributions.

Results and discussion

The results of the experimental trials outlined above are illustrated in Figures 5 and 6 for the unimodal and multi-modal distributions, respectively. The charts reflect the 4 × 5 × 3 experimental design: the number of folds $(n)$ is shown along the x-axis of each individual chart, the p value threshold is represented by differently colored lines, and the three aggregation methods are represented by the three columns. The eight D map to the eight rows (four in each figure). The y-axis in all charts shows the number of the 10,000 trials that are flagged as having significant correlation in the aggregate measure.

Figure 5.

Results of significance tests on correlation for 10,000 trials of subsets of 1000 $(x, y)$ data points sampled from data sets of 5000 $(x, y)$ data points from unimodal normal distributions with differing correlations (0.61, 0.25, 0.1, and 0). The number of trials with p values below different threshold values (0.04, 0.03, 0.02, 0.01, and 0.005) for inline replication using different numbers of folds (1, 3, 5, and 7) are shown for three different aggregation methods (any pass, majority vote, and all pass).

Figure 6.

Results of significance tests on correlation for 10,000 trials of subsets of 1000 $(x, y)$ data points sampled from data sets of 5000 $(x, y)$ data points from multi-modal normal distributions with differing correlations (0.61, 0.25, 0.1, and 0). The number of trials with p values below different threshold values (0.04, 0.03, 0.02, 0.01, and 0.005) for inline replication using different numbers of folds (1, 3, 5, and 7) are shown for three different aggregation methods (any pass, majority vote, and all pass).

The charts in Figure 5 show the results for the unimodal experiments. As might be expected, when the correlation was high $(r = 0.61)$ , all trials are shown to pass the significance threshold regardless of aggregation method or number of folds. This demonstrates that high correlation is successfully found with zero false negatives. When the level of correlation in the data set is moderate $(r = 0.25)$ , the number of trials passing the significance threshold for a given aggregation methods decreases as the number of folds $(n)$ increases. This demonstrates that the n parameter serves as a control over the sensitivity of the system to flag results as significant.

There is a similar reduction as n increases for correlations of 0.1 and 0 for all conditions except under the the “any pass” aggregation method. This result may seem counter-intuitive at first. However, in this case, the only correlation present in a given trial is the result of random noise since the data set from which the sample is drawn has zero correlation by design. Increasing the number of folds results in an increase in the number of statistical tests. This in turn allows for a higher chance of noise producing a result that appears significant. Because the “any pass” aggregation method marks any result as significant as long as at least onefold was marked as significant, an increase in n results in an increase in false positives. This result is critical. Using an aggregation method such as “any pass” essentially eliminates the benefits of IR because it enables any single fold result to determine the aggregate measure result. This means that results need not replicate across folds, increasing the chance for false-positive results.

As expected, trials that used the stricter “majority-vote” and “all pass” aggregation methods had fewer samples exceed the significance threshold when compared to “any pass.” As the correlation level decreased, the “all pass” aggregation method returned the fewest significant results. This reflects the stricter replication requirements for that approach. This trend is evident across all permutations of the experiment, suggesting that IR behaves predictably, providing system designers with multiple controls (the number of folds n and the aggregation method) over the trade-off between false positives and false negatives.

We note that the results of these experiments confirm that the “any pass” aggregation method should not be used for IR as it results in an increase in false positives. The results are reported to contrast with the other aggregation options tested in the experiments (“all pass” and “majority vote”) and to highlight that some aggregation functions are counter-productive.

Comparing the data from the unimodal experiments to the multi-modal results in Figure 6, the results are nearly identical. All of the observations noted above with respect to the unimodal results also apply in the multi-modal case. This suggests that one of the primary design goals for IR has been met: that IR is robust to the underlying data distribution, without any assumption of normality.

The results of the second set of experiments, which focused on the impact of sample size on IR behavior, are shown in Figure 7. As expected, larger sample sizes produced more reliable detection of the correlations at all correlation levels. Using a stricter p value thresholds (e.g. $p < 0.01$ ), a stricter aggregation functions (e.g. “all pass”) resulted in fewer trials producing significant results. This result confirms that IR behaves as desired, exhibiting the same response to increases in samples size as traditional statistical tests.

Figure 7.

Results of significance tests on correlation for 10,000 trials of subsets of data points sampled from unimodal and multi-modal distributions with correlation of $r = 0.25$ . The number of trials with p value thresholds (0.04, 0.03, 0.02, 0.01, and 0.05) for inline replication using a constant number of folds $(n = 5)$ is shown for different sample sizes of $d_{i}$ (100, 500, 1000, 1500, and 2000).

Overall, the experimental results present a clear and familiar picture of the trade-offs that exist in the parameterization of IR (e.g. the number of folds $(n)$ and the choice of aggregation functions) when it comes to limiting the number of false positives versus false negatives. Higher n and stricter aggregation methods result in a higher threshold for reporting meaningful results. This trade-off suggests that there is no “always optimal” set of parameters for IR. However, the results suggest that using $n = 5$ seems to provide some balance between the two extremes, while “majority-vote” aggregation seems to offer a similar compromise. In particular, this combination exhibited desired behavior in both low- and high-correlation experiments as illustrated in Figure 6.

Use cases

The IR approach is compatible with a broad range of visual metaphors and interaction models, from basic charts to more sophisticated exploratory visual analysis systems. To demonstrate this flexibility and to explore the impact of adopting an IR pipeline, we developed two prototype IR systems: (1) a reference prototype to study IR in isolation and (2) a sophisticated visual analysis system to examine IR in the context of a more complex analysis environment.

Reference prototype

We developed a reference IR implementation as part of a simplified visual analysis prototype with the goal of exploring the IR parameter space in isolation, without concern for the more complex interactions that are part of a real-world application such as the one described in section “DecisionFlow2.” The prototype supports two basic visualization metaphors: (1) bar charts and (2) scatterplots with linear regression lines. The prototype was tested using a data set of electronic medical data containing over 40,000 intensive care unit (ICU) stays.⁴³

The prototype interface, shown in Figure 8, contains three panels. In the center is the visualization canvas itself. The left panel enables users to issue queries and control key parameters of the IR process. Options include the number of folds $(n)$ for the partition function, the use of sampling with replacement, support for random or ordered incremental sampling, and controls to unfold the merged statistics and show individual folds within the visualizations. The right panel shows detailed descriptive statistics for both the individual folds $(s_{i}^{j})$ and the aggregation $(s_{i})$ .

Figure 8.

The IR-based prototype shown here was developed to test the proposed pipeline and to explore the parameter space with two baseline visualization types: bar charts and linear regression lines. The left panel shows the query and IR controls, the middle panel shows the visualization space, and the right panel shows detailed descriptive statistics computed for both the aggregate representation and the individual folds.

Figure 9 shows a series of bar charts rendered using the IR prototype to visualize the gender distribution across three subpopulations from the ICU database. This example is directly analogous to the roulette wheel bar chart example introduced in section “Visualization as a predictive model,” as both summarize the distribution of a binary variable in a given population.

Figure 9.

Six charts produced by the IR prototype system. (a–c) The top 3 charts show the gender distribution for three different sets of ICU patients. The relatively similar bar charts suggest that the underlying populations are comparable. However, when the same populations are visualized with (d–f) fivefolds, a different story appears. The charts now clearly demonstrate that we know less about the population visualized in the left column than we do about the population on the right. In this case, the difference is due largely to the size of the respective populations.

The top row of charts in Figure 9 shows the aggregate gender distribution for each of the three populations. The charts show a relatively similar distribution across all three populations, with a moderate increase in female representation moving from Figure 9(a) to (b) to (c). The bar chart shows the gender breakdown in each population quite clearly. However, there is no indication of the distribution’s stability across different groups of patients. Consumers of the visualization are left to assume that the bar charts provide an accurate depiction.

Figure 9(d–f) shows the exact same populations as Figure 9(a–c), respectively. However, these views incorporate measures computed for multiple folds $(n = 5)$ using IR. These unfolded views provide a more accurate picture regarding the repeatability of the gender distributions. In particular, we see from Figure 9(d) that the population visualized in the left column of the figure is not very predictable. Meanwhile, far less variation across folds is visible in Figure 9(f). In this case, the major difference is the size of the respective populations, which are 100, 1000, and 10,000 for Figure 9(d)–(f), respectively. The expected variance within each group is a critical factor for interpretation but is invisible in the original bar charts and easily overlooked even by expert users.³⁸ The IR approach makes this information more visible without requiring assumptions about the underlying distributions.

Figure 10, meanwhile, shows three screenshots of the IR prototype displaying data from the ICU data set using scatterplots with linear regression lines. In this case, the examples show data for populations of neonates, with weight mapped to x position and height mapped to y position. A linear regression model was calculated in all three cases using the IR pipeline with $n = 5$ . The five regression lines, one for each fold, are visible (“unfolded”) in the visualizations as red lines. In addition, an aggregate best-fit linear model is shown in blue.

Figure 10.

Weight versus height distribution for patients admitted to a neonatal intensive care unit. Simulating the results from a progressive visualization system, this figure shows both the raw data and best fit regression line (shown in blue) for (a) 500 patients, (b) 1000 patients, and (c) 2500 patients. In all three cases, the IR pipeline has computed a regression across fivefolds, shown in red. The decreasing spread across the red regression lines conveys the expected—but often overlooked—change in variation between folds as the sample size increases. The gray band across all three charts has been added to this figure to emphasize these differences and reflects the variation across folds in (c) at the maximum observed weight.

These screenshots show how IR helps convey uncertainty during progressive analysis, using the incremental sampling feature of the prototype to vary the number of samples while keeping all IR parameters constant. In Figure 10(a), only 500 patients are included in the scatterplot. As captured by the varying slopes between the five red lines, there is relatively large disagreement across folds in the linear models they produce. This uncertainty would be invisible in a similar plot rendered without the folds.

As expected, the spread between the individual fold regression lines decreases as more patients are retrieved by the incremental query feature. For example, Figure 10(b) shows the same visualization with the same $n = 5$ folds. However, this version includes data for 1000 patients. The larger sample size results in increased stability across the folds. Figure 10(c) shows the same visualization with 2500 patients. We see little improvement in agreement across folds compared to 1000 patients, suggesting that the rate of further gains in agreement will slow down.

As previously stated, the improvement in agreement as sample size increases is expected. However, as evidenced by the “recent history” charts at casino roulette tables and the other examples referenced throughout this article, visualizations are often assumed to be accurate, without taking into account issues of sample size or variation. This use case shows that IR can effectively convey this variation in the data without the need for careful modeling and in a non-parametric way that avoids assumptions about the underlying distributions.

DecisionFlow2

To test IR within a more fully featured exploratory visual analysis environment, we developed DecisionFlow2, a new IR-based version of our existing visual analysis system for high-dimensional temporal event sequence data.⁷ A screen capture of the DecisionFlow2 interface is shown in Figure 11.

Figure 11.

The DecisionFlow2 visual analytics system is shown here displaying medical event data using the Inline Replication (IR) process outlined in this article. The data in this example have been analyzed using fivefolds, without replacement. The inset subfigures show (a) an initial visualization of the aggregation function’s results for a particular medical event, and (b) a more detailed “unfolded” representation showing the variation in positive and negative support as observed across the fivefolds produced by the partition function. Figures 12 and 13 show how differences in the unfolded representation can help inform users during an analysis.

Original DecisionFlow design

The original version of DecisionFlow made heavy use of p values to help users identify event types that had a statistically significant correlation to a user-specified outcome measure. When visualizing medical data, for example, this approach enables users to find types of medical events (such as specific diagnoses, medications, and procedures) that—when appearing in a particular pattern in a patient’s history—are associated with better or worse medical outcomes.

An interactive timeline at the top of the screen enables users to segment a cohort of event sequences based on the presence of so-called “milestone” events. For a given subgroup, DecisionFlow visualizes statistics for the potentially thousands of different event types that occur between milestones, with the goal of helping users identify good candidates for new milestones. DecisionFlow conveys the event type statistics via an interactive bubble chart similar to the one seen in Figure 11.

In the bubble chart, each event type is represented by a circle whose x-axis position is determined by its positive support (the fraction of “good outcome” event sequences that contain the event type). Similarly, each circle’s y-axis position is determined by its negative support (the fraction of “bad outcome” sequences with the event type). Circle size and color encode correlation and odds ratio, respectively. Importantly, circles representing event types whose presence correlates significantly $(p < 0.05)$ with outcome are drawn with a distinct border to make it easier for users to visually distinguish between expected variation and potentially meaningful associations.

Design adaptation for IR

In the IR-based DecisionFlow2 system developed for this article, a similar bubble chart design is used to visualize the event-type statistics. However, rather than showing data for measures computed for the overall population $m_{i}$ , the circles encode aggregate measures computed by IR. For example, Figure 11 shows the system with a bubble chart focused on a subset of data containing 45,278 individual events with 1148 distinct event types. The support values (used to position the circles) and other measures were all computed across fivefolds.

The aggregate view (without showing the unfolded data) in Figure 11(a) looks essentially identical to the original DecisionFlow design. This is as intended, with the goal of making IR compatible with typical visualization designs. However, while the visual encoding is similar, the number of statistically significant correlations scores is reduced. In particular, a number of event types that were labeled as statistically significant in the original design were no longer found to be significant once majority-voting across the fivefolds was used to determine which event types were significant. This makes the visualization system more selective in rejecting the null hypothesis. The result is a reduction in the likelihood of Type 1 errors, which are a common problem in high-dimensional exploratory analysis. More detailed results and discussion are provided in section “Results and analysis.”

Another important part of the IR-based DecisionFlow2 is the ability to unfold the aggregate statistics for each event type. Users can unfold an event type by hovering the mouse pointer over the corresponding circle. For example, after hovering the mouse pointer for a few seconds over the circle shown in Figure 11(a), the unfolded representation shown in Figure 11(b) is added to the visualization.

As this example shows, the DecisionFlow2 displays the unfolded data as a convex region drawn around the original circle and outlined with a dashed border. This region corresponds to the convex hull determined by the $(x, y)$ locations for each of the n folds that contribute to the aggregate measures that determine the position of the original circle. In other words, the size and position of the unfolded region represent the variation across folds in both the positive and negative support measures. Smaller unfolded regions indicate that the values have little variation across folds. Larger unfolded regions, such as the one shown in Figure 11(b), suggest a high degree of variation between folds and therefore lower confidence in the repeatability of the aggregate measure.

The typical behavior observed when utilizing the IR-based implementation of DecisionFlow2 is shown in Figure 12. Figure 12(a) shows an event type from a very large subset of data that shows very limited variation across folds. This is represented by the very small unfolded region located near the center of the red circle just above the mouse pointer. Figure 12(b) of the figure, meanwhile, shows an event type with much higher variation (the dashed unfolded region surrounding the mouse pointer). This figure, visualizing data from a smaller sample size, demonstrates what one might expect: findings based on smaller sample sizes tend to have more variability and therefore should typically be given less weight in a decision-making process.

Figure 12.

In general, (a) smaller differences between folds are seen when sample sizes are larger, while (b) higher levels of variation are seen for smaller sample sizes.

However, this very critical difference is not observable via the original bubble chart. The size of the data corresponding to each bubble is made available elsewhere in the user interface for users who consciously seek it out, but the implications of the differences in data size are left to the user’s imagination. It is only through the unfolding process that the visualization itself conveys the difference in confidence that users should place in one view versus the other.

Moreover, it is critical to note that the size of the data set is not the sole determinant of repeatability for a given measure across folds. Major differences in measure values can be seen even for similarly sized data sets. For example, Figure 13 shows three different event types from the exact same subset of event sequences. While the number of event sequences was the same for each type, the association between ACE Inhibitors (center panel) and the user-defined outcome (eventual diagnosis with heart failure) was far more consistent across folds.

Figure 13.

Even with the same sample size, different measures can have different levels of repeatability across folds. In this example, both (a) and (c) show relatively high levels of variability, while the small unfolded region in (b) suggests that the relationship between outcome and ACE Inhibitors was fairly consistent across all fivefolds. All three views were calculated using identical sample sizes.

Results and analysis

The IR-based DecisionFlow2 prototype provides visual feedback regarding the variation in positive and negative support. As previously described, the system also uses IR to assess the statistical significance of each event type’s correlation with patient outcome. For a given event type, correlation coefficients and p values are computed for each fold, then aggregated via majority vote. Event types with more than $n / 2$ folds showing statistically significant correlation are displayed in the visualization with a distinct visual encoding.

To better understand the impact of IR and the choice of n on the visualized results, we conducted a quantitative experiment in which we compared performance for a sample user interaction sequence under various conditions. More specifically, we experimented repeatedly by performing the exact same exploratory analysis steps using DecisionFlow2, using the exact same input data, varying only the number of folds. The experiment was conducted at three partition settings: $n = {1, 3, 5}$ .

In all three cases, the input data set consisted of event data from the medical records of 2899 patients containing 1,074,435 individual medical events. These timestamped events contained 3631 distinct medical event types: specific diagnoses, lab tests, or medication orders that were present in the patients’ records. Of the 3631 distinct event types, 381 were deemed prevalent enough by the DecisionFlow2 system to be the target of correlation analysis within the metric function. The same threshold was used across all three partition settings, enabling us to compare analysis results across the exact same control conditions.

The results of our analysis are shown in Table 2. With $n = 1$ , the DecisionFlow2 system flagged statistically significant results in the same way as in the original paper.⁷ Using a threshold of $p < 0.05$ , 144 statistically significant event types were detected. When n was increased to three, the numbers were reduced dramatically. Only 50 of the original 144 statistically significant event types remained after applying a majority-vote aggregation algorithm. Of those 50, only 43 were significant across all threefolds. For $n = 5$ , the number of significant event types was even smaller. The stricter requirements for replication resulted in just 24 event types being flagged as significant given a majority-vote aggregation algorithm, and just 15 event types were significant across all fivefolds.

Table 2.

A comparison of statistically significant findings in three different IR configurations with DecisionFlow2 applied to the same data.

Number of folds	$n = 1$	$n = 3$	$n = 5$
Unanimously significant	144	43	15
Majority significant	144	50	24
At least one significant	144	56	29
Total number of measurements made	381
Total number of event types	3631

The number of event types flagged as significantly associated with outcome was largest for $n = 1$ . This setting corresponds to a traditional visualization approach with no partitioning. Larger n values dramatically reduced the number of significant findings.

As expected—and as intended—the number of statistically significant findings is reduced as n grows from one to five. There are two primary reasons for this reduction. First, because each condition is applied to the same set of event sequences for the same patients, the partition size is smaller as n increases. The smaller number of patients reduces the statistical power for each partition. The expected impact of this is higher p values and fewer statistically significant findings. With the ever-growing size of data sets in many applications, however, the impact on statistical power due to partitioning should be minimal in many use cases. At the same time, the majority-vote aggregation function requires that a significant level be repeatedly observed across multiple partitions ( $2$ for $n = 3$ or $3$ for $n = 5$ ). This reduces the likelihood of random variation being misinterpreted.

While statistical significance based on p value thresholds has known limitations to medical research and beyond,⁴⁴ it is a widely used metric in exploratory visualization because it enables a rough filtering of data to manage visual complexity and the user’s analytic attention. Follow-up analysis of any discovered insights is required. For this reason, reducing Type 1 errors becomes critical for modern visual analysis applications where vast numbers of data points can be tested and prioritized for user analysis. As the results presented here show, IR applies a higher bar for statistical significance, which has the potential to limit unsupported conclusions from the data in cases where users make quick predictive assessments directly from a visualization. It can also save significant effort in cases where follow-up analysis is performed by reducing the number of falsely generated hypotheses.

Discussion of limitations

The IR approach is designed to embed the process of replication directly within the visualization pipeline, providing a non-parametric approach to calculating and visualizing the repeatability of derived measures. As the examples in section “Use cases” demonstrate, the approach can be effective when applied to a variety of different measures and visual metaphors. However, there are limitations to IR that must be acknowledged.

First, the proposed approach does nothing to combat selection bias or other problems in the creation of the original data set. Any systemic sampling biases in the original data will be present across all folds created by the partitioning algorithm. Therefore, even measures that generalize well across multiple partitions are not necessarily generalizable to entirely new data sets.

Second, the IR approach is not truly predictive in nature. While information about the ability of various measures to replicate across multiple folds can be useful in vetting potential conclusions, findings uncovered via IR should be considered hypotheses that require testing using more rigorous methods when important decisions are to be made.

In particular, hypothesis testing often requires the collection and analysis of new data to fully understand the conditions under which a given insight holds true. Our method does not replace this step. Instead, IR helps reduce the number of Type 1 errors, which can lower the number of conclusions that need testing. However, IR does not eliminate the necessity of a post-hypothesis validation process.

Conclusion

Traditional data visualizations show retrospective views of existing data sets with little to no focus on prediction or generalizability. However, users often base decisions about future events on the findings made using these visualizations. In this way, visualization can be considered to be a visual predictive model that is subject to the same problems of overfitting as traditional modeling methods. As a result, visualization users can often make invalid inferences based on unreliable visual evidence.

This article described an approach to visual model validation called IR. Similar to cross-validation and bootstrap resampling techniques, IR provides a non-parametric and broadly applicable approach to visual model assessment and repeatability. The IR pipeline was defined, including three key functions: the partition function, the metric function, and the aggregation function. In addition, methods for visual display and interaction were discussed. The article reported results from empirical experiments that capture how IR performs under different conditions, providing insights into how the choice of IR parameters impacts performance. Finally, two uses cases were described, including a new IR-based implementation of a previously published exploratory visual analytics system. The use cases demonstrated the successful compatibility of IR with a variety of visual metaphors and derived measures.

While the results presented in this article are promising, they represent only one step in a growing effort to bring high repeatability and predictive power to visualization-based analysis systems. There are many areas for future work including improved techniques for detecting and conveying issues related to missing data, techniques for addressing and visually warning users regarding selection bias, and improved methods for conveying the degree of compatibility between a given statistical model’s assumptions and the actual underlying data.

Footnotes

Acknowledgements

The authors thank Brandon A Price for his contributions to the software development process for the reference prototype described in the “Use Cases” section of this article.

Funding

This article is based in part upon work supported by the National Science Foundation under grant no. 1704018.

ORCID iDs

David Gotz

David Borland

References

Shneiderman

. The eyes have it: a task by data type taxonomy for information visualizations. In: Proceedings of IEEE symposium on visual languages, Boulder, CO, 3–6 September 1996, pp. 336–343. New York: IEEE.

Card

Mackinlay

Shneiderman

Readings in information visualization: using vision to think. San Francisco, CA: Morgan Kaufmann Publishers, 1999.

Chi

. A taxonomy of visualization techniques using the data state reference model. In: IEEE symposium on information visualization, Salt Lake City, UT, 9–10 October 2000, pp. 69–75. New York: IEEE.

Ostle

Statistics in research. 2nd ed. Ames, IA: The Iowa State University Press, 1963.

Perer

Bertini

Maciejewski

, et al. An IEEE VIS 2014 workshop on visualization for predictive analytics, 2014, http://predictive-workshop.github.io/

Gotz

Price

Chen

AT.

Visual model validation via inline replication. arXiv:1605.08749, 2016, http://arxiv.org/abs/1605.08749

Gotz

Stavropoulos

DecisionFlow: visual analytics for high-dimensional temporal event sequence data. IEEE T Vis Comput Gr 2014; 20: 1783–1792.

Skeels

Lee

Smith

, et al. Revealing uncertainty for information visualization. Inform Visual 2010; 9(1): 70–81.

Potter

Rosen

Johnson

CR.

From quantification to visualization: a taxonomy of uncertainty visualization approaches. IFIP Adv Inf Commun Technol 2012; 377: 226–249.

10.

MacEachren

Roth

O’Brien

, et al. Visual semiotics and uncertainty visualization: an empirical study. IEEE T Vis Comput Gr 2012; 18(12): 2496–2505.

11.

Sanyal

Zhang

Bhattacharya

, et al. A user study to compare four uncertainty visualization methods for 1D and 2D datasets. IEEE T Vis Comput Gr 2009; 15(6): 1209–1218.

12.

Tak

Toet

van Erp

The perception of visual uncertainty representation by non-experts. IEEE T Vis Comput Gr 2013; 20: 935–943.

13.

Potter

Gerber

Anderson

EW.

Visualization of uncertainty without a mean. IEEE Comput Graph 2013; 33(1): 75–79.

14.

Brodlie

Osorio

Lopes

. A review of uncertainty in data visualization. In: Dill

Earnshaw

Kasik

, et al. (eds) Expanding the frontiers of visual analytics and visualization. Berlin: Springer, 2012, pp. 81–109.

15.

Borland

Wang

Gotz

Contextual visualization: making the unseen visible to combat bias during visual analysis. IEEE Comput Graph 2018; 38(6): 17–23.

16.

Gotz

Sun

Cao

Adaptive contextualization: combating bias during high-dimensional visualization and data selection. In: Proceedings of the 21st international conference on intelligent user interfaces, Sonoma, CA, 7–10 March 2016, pp. 85–95. New York: ACM.

17.

Gotz

Sun

Cao

, et al. Adaptive contextualization methods for combating selection bias during high-dimensional visualization. ACM T Interact Intell Syst 2017; 7(4): 17-1–17-23.

18.

van den Elzen

van Wijk

. BaobabView: interactive construction and analysis of decision trees. In: IEEE conference on visual analytics science and technology (VAST), Providence, RI, 23–28 October 2011, pp. 151–160. New York: IEEE.

19.

Muhlbacher

Piringer

A partition-based framework for building and validating regression models. IEEE T Vis Comput Gr 2013; 19(12): 1962–1971.

20.

Gosink

Bensema

Pulsipher

, et al. Characterizing and visualizing predictive uncertainty in numerical ensembles through bayesian model averaging. IEEE T Vis Comput Gr 2013; 19(12): 2703–2712.

21.

Sipes

Martin

Reif

, et al. Predictive models of prenatal developmental toxicity from ToxCast high-throughput screening data. Toxicol Sci 2011; 124: 109–127.

22.

Fisher

Drucker

Konig

AC.

Exploratory visualization involving incremental, approximate database queries and uncertainty. IEEE Comput Graph 2012; 32(4): 55–62.

23.

Stolper

Perer

Gotz

Progressive visual analytics: user-driven visual exploration of in-progress analytics. IEEE T Vis Comput Gr 2014; 20(12): 1653–1662.

24.

Fisher

Popov

Drucker

, et al. Trust me, I’m partially right: incremental visualization lets analysts explore large datasets faster. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’12), Austin, TX, 5–10 May 2012, pp. 1673–1682. New York: ACM.

25.

Sheskin

DJ.

Handbook of parametric and nonparametric statistical procedures. 3rd ed. Boca Raton, FL: CRC Press, 2003.

26.

Stoline

MR.

The status of multiple comparisons: simultaneous estimation of all pairwise comparisons in one-way ANOVA designs. Am Stat 1981; 35(3): 134–141.

27.

Rothman

KJ.

No adjustments are needed for multiple comparisons. Epidemiology 1990; 1(1): 43–46.

28.

Chen

Zhang

Chen

, et al. Uncertainty-aware multidimensional ensemble data visualization and exploration. IEEE T Vis Comput Gr 2015; 21: 1072–1086.

29.

Feng

Kwock

Lee

, et al. Matching visual saliency to confidence in plots of uncertain data. IEEE T Vis Comput Gr 2010; 16(6): 980–989.

30.

Yuan

KL.

Visualizing flow of uncertainty through analytical processes. IEEE T Vis Comput Gr 2012; 18(12): 2526–2535.

31.

Majumder

Hofmann

Cook

Validation of visual statistical inference, applied to linear models. J Am Stat Assoc 2013; 108(503): 942–956.

32.

Wickham

Cook

Hofmann

, et al. Graphical inference for infovis. IEEE T Vis Comput Gr 2010; 16(6): 973–979.

33.

Correa

Chan

. A framework for uncertainty-aware visual analytics. In: IEEE symposium on visual analytics science and technology, Atlantic City, NJ, 12–13 October 2009, pp. 51–58. New York: IEEE.

34.

Munzner

Visualization analysis and design. HAR/PSC ed. Boca Raton, FL: A K Peters/CRC Press, 2014.

35.

Tufte

ER.

Envisioning information. Cheshire, CT: Graphics Pr., 1990.

36.

Ware

Information visualization: perception for design (interactive technologies). Burlington, MA: Morgan Kaufmann Publishers, 2004.

37.

Cammegh Limited. Cammegh—the world’s finest roulette wheel, 2015, http://www.cammegh.com/product.php?product=displays

38.

Sutcliffe

Bruijn

Thew

, et al. Developing visualization-based decision support tools for epidemiology. Inform Visual 2014; 13(1): 3–17.

39.

Zgraggen

Zhao

Zeleznik

, et al. Investigating the effect of the multiple comparisons problem in visual analysis. In: Proceedings of the CHI conference on human factors in computing systems, Montreal, QC, Canada, 21–26 April 2018. New York: ACM.

40.

Kohavi

. A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the 14th international joint conference on artificial intelligence (IJCAI’95), Montreal, QC, Canada, 20–25 August 1995, vol. 2, pp. 1137–1143. San Francisco, CA: Morgan Kaufmann Publishers.

41.

Efron

Bootstrap methods: another look at the jackknife. Ann Stat 1979; 7(1): 1–26.

42.

Lam

Suen

SY.

Application of majority voting to pattern recognition: an analysis of its behavior and performance. IEEE T Syst Man Cy A 1997; 27(5): 553–568.

43.

Goldberger

Amaral

LAN

Glass

, et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 2000; 101(23): e215–e220.

44.

Goodman

SN.

Toward evidence-based medical statistics. 1: the P value fallacy. Ann Intern Med 1999; 130(12): 995–1004.