Providing visual analytics guidance through decision support

Abstract

Guidance in visual analytics aims to support users in accomplishing their analytical goals and generating insights. Different approaches for guidance are widely adopted in many tools and frameworks for various purposes – from helping to focus on relevant data subspaces to selecting suitable visualization techniques. With each of these different purposes come specific considerations on how to provide the needed guidance. In this paper, we propose a generic method for making these considerations by framing the guidance problem as a decision problem and applying decision making theory and models toward its solution. This method passes through three stages: (1) identifying decision points; (2) deriving and evaluating alternatives; (3) visualizing the resulting alternatives to support users in comparing them and making their choice. Our method is realized as a set of practical worksheets and illustrated by applying it to a use case of providing guidance among different clustering methods. Finally, we compare our method with existing guidance frameworks to relate and delineate the respective goals and contributions of each.

Keywords

Visual Analytics user guidance decision support

Introduction

Guidance in visual analytics (VA) has received increasing attention in recent years. Defined as “a computer-assisted process that aims to actively resolve a knowledge gap encountered by users during an interactive VA session,”¹ guidance aids users in producing analytic results, generating new insights, and eventually building new knowledge.²

Despite being a concept recently introduced to VA, the practice of using different forms of guidance to support users’ analysis processes has been widely adopted in many VA systems – from data³ and visualization exploration^4,5 to model building⁶ and reasoning.⁷ The wide and diverse usage of guidance validates the usefulness of the concept and provides a solid foundation for research on guidance mechanisms. With a clear characterization and a broad range of applications, recent research extends guidance beyond resolving knowledge gaps,⁸ discusses when guidance should be used,⁹ and lists the considerations to be made when designing guidance.² However, a generic guidance method for designing and providing guidance across its various applications and goals is still missing.

Facing this challenge, the main contribution of this paper is a step-by-step process to derive the practical “how” of providing guidance. To this end, we reformulate the guidance problem as a decision making problem and apply decision support models toward its solution. Our method passes through the following three stages:

Stage 1 identifies the decision points in VA processes where guidance is needed regarding the data, algorithms, visualizations, and reasoning.

Stage 2 utilizes multiple criteria decision analysis (MCDA)¹⁰ to evaluate the alternatives to choose from in these decision points in order to generate guidance.

Stage 3 uses composite visualizations of multiple alternatives for inspecting and comparing the resulting alternatives to guide the users in their decisions.

In addition to its application-agnostic coverage of process, goals, and conceptual levels, there are various benefits of utilizing this method to provide guidance. From the perspective of designing VA systems, our method allows for quickly realizing and testing if and how guidance might work for the system. Once realized, it can also be more easily adapted and re-used in other systems. Finally, any scoring or ranking metric can be directly included as a criterion in MCDA models, making it backwards compatible to existing guidance methods as well as to a wide range of utility and quality measures for data, algorithms, and visualizations. From the perspective of using VA systems, our MCDA-based guidance exposes the criteria and their weights to the users, which makes the generated guidance more explainable. It further allows for adjusting these weights to user- and scenario-specific needs, making the provided guidance flexible and adaptive. Finally, framing and presenting guidance consistently through criteria and weights provides for a uniform guidance experience across domains and applications.

Background and motivation

To explain the rationale behind our guidance method, we present the research background and motivate the reasoning behind our method from three perspectives: guidance for resolving knowledge gaps, generic guidance beyond knowledge gaps, and guidance through decision support.

Guidance for resolving knowledge gaps

Framed around the concept of knowledge gaps, guidance supports users in their VA processes to overcome hurdles and successfully proceed in their analyses. In the following, we present the research on VA guidance from this perspective and provide our motivation for a generic guidance method that goes beyond knowledge gaps.

Background

Formally characterized in VA by Ceneda et al. guidance is usually framed around resolving knowledge gaps encountered by users in VA processes.¹ These knowledge gaps can be of different types, either target-unknown, where the optimal solution of a VA problem is unclear (I-know-it-when-I-see-it), or path-unknown, where the sequence of actions to reach a known target is unclear. Guidance can also be of varying degrees, ranging among orienting, directing, and prescribing; and lie in different domains, from data and tasks to VA methods and knowledge management.¹ Further research has brought forth some considerations and tools for designing guidance. A decision tree was proposed for deciding if guidance is needed and to what degree.⁹ Five key requirements for effective guidance were established – namely for guidance to be available, trustworthy, adaptive, controllable, and non-disruptive.²

The conceptual space covers a wide range of approaches that have proven useful in assisting users to resolve their knowledge gaps.¹¹ These approaches come in various guidance degrees and guide users in different domains. Orienting users toward regions in view space, Gladisch et al. provide visual cues pointing to where potentially interesting data points lie based on a degree-of-interest function.¹²Orienting users toward different levels of granularity in the data, Luboschik et al. provide heterogeneity-based guidance that indicates hidden details at higher levels of granularity and thus guides users to zoom-in for a closer inspection.¹³Directing users in carrying out their analytic tasks, Streit et al.’s Stack’n’Flip approach provides a guided view of the user’s analytic workflow and offers subsequent analysis steps to be taken.⁷Orienting and directing users among VA methods, Müller et al.’s Morpheus guides users through multiple parameters to choose the best ones for subspace clustering.⁶Directing and in part prescribing user actions in the space of generated knowledge and insights, the Nugget Management System from Yang et al. suggests and refines valuable information (nuggets) based on user interest.¹⁴

Motivation

The outlined research provides high-level guidelines regarding guidance design principles as well as context-dependent examples of how to guide users in VA. However, for VA designers to effectively employ the concept of guidance, a concrete method to produce guidance from end to end – that is, from the specification of the guidance problem through requirements all the way to generating and presenting guidance to the end user – that is independent of the guidance domain or scenario is still lacking. Specifically – How to systematically identify the points where guidance is needed? How to compute and generate guidance with a unified underlying mechanism? How to present and adapt guidance according to the context? These questions motivate our guidance method that can be used in various guidance scenarios.

Generic guidance beyond knowledge gaps

The utility of guidance goes far beyond knowledge gaps. In the following, we present related research in guidance supporting this perspective and motivating a guidance design framework that caters to a variety of guidance aims including, but not limited to knowledge gaps.

Background

Collins et al. aptly observed that the goal of guidance can go beyond resolving knowledge gaps, including to inform, to mitigate bias, to reduce cognitive load, for training, for engagement, and to verify conclusions.⁸

Indeed, when VA experts have sufficient knowledge to conduct their analyses, guidance can nevertheless support them by providing important meta-information, keeping track of their analyses, and making suggestions to reduce their cognitive load and improve their efficiency.^4,15,16 Furthermore, no matter how knowledgeable a VA expert is, they might still be subject to various cognitive biases in their analyses – especially subconscious ones¹⁷– and guidance can also help to combat these biases through revealing them and providing suggestions when indicators for bias are detected.¹⁸ Other research has also discussed guidance goals, such as attention management by guiding users to views that are currently important,¹⁹ supporting user learning and training with onboarding guidance that walks users through visualization elements,²⁰ engaging users by prompting them when they are inactive,²¹ as well as aiding the verification of analyses through monitoring of analytical hypotheses.¹⁵

Motivation

To provide a generic guidance method that is applicable to a wide range of guidance goals, we need to base it on a concept that also includes the additional goals of guidance beyond resolving knowledge gaps, such as to reduce cognitive load, to mitigate bias, and to verify conclusions. However, critical challenges lie in not only finding such concept and establishing its connections with guidance, but also integrating its existing theories as well as applications with guidance in order to practically build our guidance method.

Guidance through decision support

To yield the missing end-to-end guidance framework that is inclusive of the extended set of guidance goals, a new perspective on user guidance in VA is needed. This is where decision making theory in general and decision support in particular come into the picture. In the following, we present related work in decision support, and motivate our method for using decision support to design and provide guidance.

Background

A large body of decision support research has focused on a similar set of goals as the one that Collins et al. proposed for guidance. These include, for example, to provide important information relevant for the decisions,²² to reduce errors and mitigate various biases,²³ and to alleviate mental workload when the decisions are complex.²⁴

Research on Decision Support Systems (DSS) provides us with useful tools to realize a generic guidance method. Decisional guidance has been conceptualized in previous DSS research with when and why to provide deliberate decisional guidance, how to build it into DSS, and what consequences it might bring.²⁵ To practically relate guidance in DSS to guidance in VA, the Handbook on Decision Support Systems provides a useful overview.²⁶ Its chapter on DSS architectures and types summarizes the four basic components of a DSS, including language (input), presentation (output), knowledge (database), and problem-processing (model) (sub-)systems. Relating these components to the goals of guidance, the knowledge components contain important information to inform users and store expertise that reduces their cognitive load²⁷; the problem-processing components structure and model users’ decision making processes, helping users to combat their biases, and ease their cognitive effort²³; the language and presentation components also aid users’ mental work, while promoting users to be engaged in the decision making processes through interactions.²⁸

Among these four components, those focusing on problem-processing are particularly relevant to guidance generation, as they provide a mechanism to analyze decisions and generate evaluations of alternatives, which is essentially what guidance does. Multiple criteria decision analysis (MCDA) is a commonly used approach for problem-processing.¹⁰ It takes multiple quantifiable metrics as evaluation criteria and evaluates alternative decisions based on these criteria. As the underlying criteria can be easily exposed to and manipulated by users, MCDA methods allow for communicating and potentially even changing the mechanism providing the guidance.

Research also exists on combining VA and MCDA. To orient decision makers among the multiple alternatives and criteria, visualizations are commonly used as a part of the presentation component in MCDA-based decision methods.^29,30 MCDA models have also been integrated with geographical information systems (GIS) to support spatial decision-making through encoding the metrics and results of MCDA models in geographic visualizations.^31,32 Particularly, Jankowski discussed the architecture and implementation of GIS and MCDA through either loosely coupling them with file exchange module or tightly integrating the two with a shared user interface and database.³¹

Motivation

The similar sets of goals of DSSs and guidance make DSSs a suitable method to provide guidance in support of any decision making problem – be it which data to look at, which algorithm to choose, or in which direction to pan. Particularly MCDA methods are promising in this regard. However, the critical challenge still lies in how exactly to map the decision support research to specific steps for constructing a generic yet practical guidance method. This is where our work contributes by connecting research in decision support and VA to build a generic method for guidance design with concrete steps to follow.

Overview of the method

This section gives an overview of our guidance method through decision support. We first conceptualize guidance as a decision making problem, before structuring our method based on the decision making process.

Guidance as supporting decision points

When observing situations in VA in which guidance is needed, it is noticeable that users are often faced with making decisions among multiple alternatives. These can be decisions regarding a particular analysis target to pursue – for example: Which data (sub)set to zoom into? What aspects of the data to inspect – outliers, clusters, trends? Which keywords to search for? It can also be decisions on the particular analysis path that gets the user to a desired target – for example: What algorithms and parameters to choose to yield the envisioned clustering? Which direction to pan to reach a sought data region? What visual encoding to use to bring out best the patterns in the data? These decisions are inherent in VA: if no human decisions were required, then the analyses could be fully automated and neither a human user in the loop, nor guidance would be needed. Therefore, we can see the existence of “decision points” as a prerequisite for needing guidance. Knowledge gaps can then be seen as a common issue, among many others, that might arise at these decision points – when users lack the knowledge to make their decisions in the analysis, guidance can provide it.

This perspective of an analysis workflow as a series of decisions among multiple analytic alternatives is echoed in recent research by Liu et al.^33,34 They studied how researchers experiment with different paths when analyzing data and identified the points where these alternative paths fork as “decision points.” Their work provides a fundamental understanding of what constitutes decision points in analytical work and how analysts reason in their decisions. However, how to support decisions with guidance is an open question. In addressing this question by connecting decision support systems with guidance, we provide a novel and tangible path for guiding users.

The concept of decision points also covers the realm beyond knowledge gaps. At decision points, the users’ preconceptions might lead them to choose certain analytical paths to confirm their hypotheses (confirmation bias), where guidance can suggest alternative solutions and mitigate such biases; or particular analytical decisions might be cognitively complex and demanding, where guidance can facilitate them among the space of alternatives and reduce the cognitive load. Therefore, decision points are a fitting concept to capture situations for which guidance is needed and can be provided through decision support.

In short, we extend the guidance concept to decision points and re-frame guidance as providing users with decision support when they are faced with decision points in VA.

Structure of the guidance method

Our method for designing and providing guidance is based on this re-framing into a decision making/decision support problem, whose overall structure is outlined in this section.

We look at decision points in VA from each of the three stages of decision making processes proposed by Herbert Simon – intelligence, design, and choice.³⁵ Simon’s model has been widely studied, including in the context of visualization tasks.³⁶ The intelligence stage refers to recognizing the conditions calling for decisions, the design stage refers to the development and evaluation of the alternatives, and the choice stage refers to choosing the desired alternative(s) based on the evaluation results. These three stages are visually presented in Figure 1.

Figure 1.

The three stages in decision making process by Simon³⁵ with illustrated explanations of the three stages.

The focus of our method in the intelligence stage is to detect and assess the decision points where guidance is needed. This stage starts with an inspection of the context in which the VA system is being used. This helps VA designers to systematically understand the conditions under which guidance might be needed. With the context recognized, we can then systematically identify the decision points calling for guidance. Finally, to evaluate if guidance is actually needed and prioritize these identified decision points, we assess the need for guidance at each decision point.

In the design stage, the focus is on constructing the mechanism for guidance generation through developing and evaluating the alternatives for each decision point that calls for guidance. Continuing with the assessment of decision points produced in the intelligence stage, we first aim to recognize the space of alternatives for each decision point by specifying the number of alternatives along with a list of examples to be evaluated. This allows VA designers to more tangibly consider how to evaluate and present these alternatives in the later steps. Thereafter, we discuss how evaluation criteria can be produced for the alternatives. Finally, to evaluate them, we introduce MCDA and how it can be applied and adapted to varying guidance degrees.

The main challenge at the choice stage of our guidance method is to present and adapt the guidance output for each decision point. The first step in this stage carries over the results from the design stage by recognizing what data to present in the guidance output. Combining this data with the corresponding guidance degree and number of alternatives, we then consider how to compose the presentation of alternatives in order to produce a guidance output suitable for the given guidance scenario. Finally, to allow the guidance to be adaptive and flexible, we inspect how to adjust the produced guidance in different contexts.

With the three stages described above, we have outlined our generic guidance method framed around decision points on an abstract level. The conceptual connections between the key concepts in the initial characterization of VA guidance and our method in are illustrated in Figure 2. In the next three sections, we further detail the concrete steps in each of these three stages.

Figure 2.

Adapting the guidance model by Ceneda et al.^1,11 (left) to reflect the stages of decision making processes used in our method (right). For clarity, we subsumed the components of “history” and “domain” together with “data” and “knowledge” as “input,” and the three guidance degrees as “output.” Similar to the model by Ceneda et al.¹ the blue elements indicate the guidance components. Consistent with the colors used for the three stages of decision making in this paper, the orange elements indicate the intelligence stage, the green elements indicate the design stage, and the purple elements indicate the choice stage.

It is worth noting that the following design process formulates a middle ground that we expect to be applicable to most, but not necessarily all guidance scenarios. The reason is that guidance can vary greatly in its complexity depending on its context. In a simple analytical process with well-established “best practices,” guidance can be provided through a manually authored workflow with alternative paths without designing an elaborate evaluation model. Whereas in complex analysis settings where the alternatives’ evaluation criteria are challenging to define or compute, our design process may have to be reiterated multiple times and the evaluation model may include more complex considerations than can be expressed by weights alone. Therefore, the following description should not be used dogmatically as a fixed end point of all design considerations, but flexibly as a starting point from which to tailor a sensible design process for a guidance problem at hand.

Stage 1: Intelligence – Decision points calling for guidance

From the perspective of guidance as decision support, it is vital to first recognize the decision points for which to provide guidance.

In the following, we illustrate the steps in this intelligence stage by first articulating the context of use, then identifying the decision points in such context, and finally assessing the need for support in each of the decision points in order to prioritize them when implementing guidance. These three steps are listed in Figure 3.

Figure 3.

The steps in the Intelligence stage and the corresponding factors considered in each step.

Step 1.1 – Analyze the context of use

What

The context of use captures the conditions under which a product is used. Analyzing the context of use of a VA tool provides important information for devising effective guidance in later steps. For example, the offered guidance may differ depending on whether a VA tool is being used within or outside of its intended context of use.

Why

Context of use analysis is a prerequisite for designers to understand when, where, how, and by whom a system is being used to provide good usability.³⁷ This is also true for VA systems for which the context of use gives rise to the “conditions calling for decisions”³⁵– that is, the concrete situations in which guidance is needed. A thorough context of use analysis helps VA designers to more concretely identify the decision point (Step 1.2) and assess them (Step 1.3), as well as to build adaptive guidance generation (Stage 2) and presentation (Stage 3) grounded in knowledge about users, goals & tasks, resources, and environment.

How

Context of use is an important concept in HCI that can be interpreted from various perspectives.³⁸ Hence, the following considerations taken from Common Industry Format (CIF) for context of use descriptions (ISO/IEC 25063:2014) provide a least common denominator as a starting point for analyzing contexts of use,³⁹ but they should by all means be extended by additional, possibly domain-dependent considerations if these help to further pinpoint usage scenarios in which guidance may be needed.

• Users are persons, or decision makers, who directly interacts with a VA system and steers the analytical process. Decision makers are prone to use different decision making styles, depending on their culture, personality, and training.⁴⁰ This requires the provided decision support to accommodate these different styles. Their level of expertise should also be examined as decision makers often need support when they lack knowledge or experience.²

• Goals & Tasks relate to the motivation and execution of visual analyses. For example, different goals – explore versus confirm⁴¹– relate very much to different degrees of freedom in an interactive analysis and have thus implications for the degree of guidance needed – orienting versus directing.

• Resources capture the boundaries of the analyses to be run. They include technical limitations such as computing power and available visualization methods, as well as virtual limitations such as cognitive strain and time constraints. Resources delimit the space of viable alternatives among which to guide users.

• Environment is a multi-faceted factor that includes technical, physical, social, cultural and organizational environments. For guidance, characterizing the technical environment is particularly important – for example, describing how much visual support is already provided by a user interface.

Notably, context of use is often dynamic and subject to change according to the purpose of the system and the progress of development. Especially at an early stage of the system development, designers might not be able to articulate all the factors. Thus, the context of use analysis should be an iterative process and adapt to different usages.

Step 1.2 – Enumerate the decision points

What

Having established the context of use, we now enumerate the decision points within this context where multiple alternatives exist among which a user must choose.

Why

Decision points are essential for generating guidance (Stage 2) and presenting guidance (Stage 3), as these indicate where guidance may be needed. This step generates an overview of the various decisions users are facing in a VA tool without yet prioritizing among them, to ensure that decision points are not overlooked.

How

In the early stages of developing a VA solution, designers might not have direct access to users yet. In this case, decision points can be listed through a cognitive walk-through. At later design stages, decision points can be enumerated by involving users through interviews and contextual inquiries, possibly following structured protocols to assess challenging situations.⁴² In both cases, designers need to closely inspect each component in a VA tool in order to comprehensively recognize the decision points.

To do so in a structured way, we propose the use of suitable task taxonomies for VA. This makes sense, as fundamentally any task carried out by the user involves a decision – for example, Should I rather filter or sample the data to reduce it? Should I zoom-in here or there to see interesting details? If there was no decision to be made, the user would not have to carry out the task as the system could proceed by itself.

While many taxonomies exist for visualization tasks,⁴³ the literature is more sparse on VA tasks. After considering the taxonomies by Gotz and Zhou¹⁶ and Heer and Shneiderman,⁴⁴ we settled on von Landesberger et al.’s taxonomy of VA interactions for its universality, high level of abstraction, and inclusion of analytic reasoning.⁴⁵ In particular, the reasoning component captures the implicit decisions of what the analytical focus is and how to carry it out, which might change along the course of the analysis. The inclusion of reasoning component helps to consider these unexpected and less defined tasks that often occur in exploratory analytical processes. Slightly adapted to the context of decision making in VA, it breaks down into components and types. Components are the aspects of a VA system to which a task relates:

• Tasks relating to the data component deal with decisions on which data to use (e.g. subset selection, filtering) and how to use them (e.g. cleaning, transformation).

• Tasks relating to the algorithm component deal with decisions on how to process the data (e.g. which clustering algorithm) and how to parameterize the processing (e.g. distance metric, similarity threshold).

• Tasks relating to the visualization component deal with decisions on which visualization techniques to use and how to parametrize them (e.g. color mapping and axis scaling).

• Tasks relating to the reasoning component deal with decisions on which line of analytic reasoning to follow to yield insights (e.g. deductive reasoning to “detect the expected” or inductive reasoning for “discovering the unexpected”) and how to carry it out.⁴⁶

Types delineate between the fundamental What to do? and the subsequent How to do it? of a task. Concretely, these types are:

• Decisions on the scheme of a task – for example, What data subset to analyze? What algorithm to choose? What chart type to use? These are fundamental decisions for or against principal options.

• Decisions on the parameters of a task – for example, How to derive that subset? How to parametrize that algorithm? How to apply and fine-tune the chosen chart type? These are secondary decisions that follow from an already chosen scheme and that are needed to concretize and carry out that first decision.

Step 1.3 – Assess the need for guidance

What

To help VA designers to assess decision points that would benefit from guidance and prioritize them accordingly in the development process, we outline an assessment of the need for guidance among the decision points.

Why

Existing research on guidance emphasizes providing the right guidance at the right time and making sure the guidance is non-intrusive.^2,8 Hence, it is important to assess the need for support in each of the decision points to avoid providing unnecessary guidance that may distract from the analysis or even disrupt the analysis flow instead of enabling it. Furthermore, there can be a large number of possible decision points in a VA system, and to provide guidance for all of them can be an arduous task. Therefore, the decision points need to be prioritized to create clear priorities for the guidance generation (Stage 2) and presentation (Stage 3).

How

The need for guidance at the decision points is influenced by many contextual factors. Hence, such assessment is ideally done together with the end users to ensure that it reflects the real-life experience of users through qualitative methods such as interviews or workshops as well as quantitative ones like surveys or user performance. Previous studies in guidance also indicate the potentials of usage logs from user interactions for inferring the need for guidance.⁸ Without direct access to user information or logs, a cognitive walk-through or an internal expert review can be conducted to assess the need for support.^47,48

We formalize this process through an adapted version of risk assessment, which identifies and assesses potential risks at each decision point by quantifying the probability and impact of getting the corresponding analytic decisions “wrong.”⁴⁹ Seeing guidance as a support mechanism to mitigate the potential risks at each decision point, we utilize a risk assessment scheme to evaluate the need for guidance.

• The probability of a “wrong” analytic decision is often characterized by the lack of knowledge, which can be decided by the various factors in the context of use. Previous guidance research has also discussed how to identify such knowledge gaps.² Other factors, such as the likelihood of cognitive biases may also factor into this probability.⁵⁰

• The impact of getting it “wrong” is likewise a multi-faceted consideration based on how much the course and overall outcome of the remaining analysis workflow depends on this decision and how consequential a wrong result would be for the domain decision based on it – for example, a wrong treatment decision for a patient would be more dire than a wrong ad-buying decision for a marketing campaign.

• Evaluating the number of possible alternatives is another important factor. If there are only a handful of possible options for a decision points, modern UIs with Undo/Redo functionality allow to quickly try them out before deciding for one without the need for an elaborate guidance scheme. Yet if there are many possible options, this is no longer viable, increasing the need for guidance with the number of alternatives.

After analyzing these factors, they can then be combined to produce the decision points inventory with a priority ranking. In risk assessment, the factors are usually rated on a quantitative scale, multiplied together, and combined in the form of a risk assessment matrix or a risk inventory.⁴⁹

Stage 2. Design – MCDA to generate guidance

After identifying the decision points and assessing their respective need for guidance, we now specify the underlying mechanism that generates guidance for a decision point. To achieve this, we draw from the domain of decision support and propose to generate guidance through multiple criteria decision analysis (MCDA). MCDA integrates different criteria to evaluate the alternatives of decisions.⁵¹ MCDA is a useful method for our goal of a generic mechanism to generate guidance, as any algorithm (e.g. heterogeneity-based guidance) or metric (e.g. degree-of-interest functions) can be easily incorporated in an MCDA model as criteria.²⁹

For this stage, VA designers need to go over each decision point in the previously produced inventory according to the priority ranking. Additionally, as some of the decision points might be interconnected – for example, choosing a clustering algorithm and choosing its settings are often jointly decided, as they form scheme and parameters of the same decision – the guidance generation of such interconnected decision points can also be developed in conjunction.

In the following, we first recognize the space of alternatives that guidance generation should consider, then describe how the corresponding evaluation criteria can be produced, and finally discuss how an MCDA-based evaluation model can be built using these criteria to generate guidance. These three steps are listed in Figure 4.

Figure 4.

The steps in the Design stage and the corresponding factors considered in each step.

Step 2.1 – Recognize the space of alternatives

What

Although there might be a great number of available alternatives at each decision point, many of them may not be useful or possible to be considered. This step aims to recognize the space of alternatives that later feeds into the MCDA model in order to generate guidance.

Why

Recognizing the space of alternatives to be considered by guidance generation is essential for building the underlying mechanism that generates guidance. Specifically, the number of alternatives can influence how guidance should be generated and later presented in Stage 3. Moreover, recognizing some examples of the alternatives also helps VA designers to consider them in a more concrete manner and more easily identify the criteria to evaluate the alternatives.

How

To this end, we consider two elements that VA designers should inspect: the estimated number of alternatives and examples of alternatives. These two elements are considered under the constraints imposed by the identified context of use – users’ goals and tasks that govern how open they are to explore different alternatives; available resources such as the set of implemented algorithms or the time available for inspecting different alternatives before having to make a decision; and the environment such as the available user interfaces that influence how open-ended the exploration of alternatives is.

This step differs from the previous ones as we start to consider which of all the possible alternatives should feed into the MCDA model and be evaluated. Not all possible alternatives might be applicable or useful in a given context of use. And the subset of all those that are applicable may not be feasible to be evaluated due to time constraints or other limitations. And all those that are feasible to be evaluated may still be too many to then interactively inspect. For example, for a large multi-dimensional dataset, there can be almost infinite number of subsets to explore. Many processing algorithms also have numerous parameters that are continuous and need to be considered in conjunction. In these cases, corresponding sampling techniques can be used to produce a reasonable amount of alternatives under the limitations in the context of use, such as the computational and cognitive resources.^52,53 Additionally, decision strategies that utilize heuristics to eliminate alternatives, such as Elimination by Aspects, can already be applied to narrow the space of alternatives.⁵⁴ To yield a clearer understanding of the practically relevant subspaces of alternatives, we look at the following:

• The number of valid/useful alternatives for each decision point, as their number often differs from the number of all possible alternatives from Step 1.3. Particularly, previous research in decision making shows that a higher number of alternatives can significantly improve the decision making quality,⁵⁵ while too many alternatives can also decrease decision efficiency and even lead to decision paralysis.⁵⁶

• Examples of the alternatives illustrate what the alternatives look like for each decision point, such as names of different algorithms, different parameter range, or different encodings to be used for the visualization. These examples act as a concrete thinking tool for VA designers to consider potential alternatives under the contexts of use and later distill criteria that compare and evaluate them.

Step 2.2 – Produce the criteria

What

The aim of this step is to produce the criteria to evaluate the alternatives for each decision point. These criteria will then later feed into an MCDA model and help to rank the alternatives and generate guidance.

Why

To evaluate the alternatives for a decision point, a way to judge them is needed. As we adopt MCDA models to calculate the overall evaluation from a set of input measures in Step 2.3, we need some form of quantifiable metric or quality measure to do so. It is also consequential for the subsequent presentation of guidance how these criteria are produced, as different methods to generate evaluation metrics may incur varying degrees of uncertainty.

How

To yield suitable evaluation criteria, we consider three kinds of measures based on the context of use and the number and types of the alternatives for the decision points.

• Measures based on the full results can be used when they are not too time-consuming to precompute. A wide range of such measures are available for representative data selection,^53,57,58 evaluations of machine learning,^59,60 and quality metrics for data visualization.⁶¹

• Previews based on partial results for each alternative can be used to adapt to more time-sensitive contexts. Techniques like Progressive Visual Analytics can be helpful in such context of producing an early partial result and refining it over time.⁶²

• If the methods above are not applicable due to the limitations in the context of use, predictive metrics based on abstract features of the alternatives can also be used, including data coverage,¹⁸ algorithm runtime predictions,⁶³ and structure-oriented measures for visualizations.⁶⁴ This way, no precomputation of the result is necessary.

• Finally, when the metrics cannot be quantitatively and automatically produced or when they are too uncertain and imprecise, human-rated criteria and rankings can be used instead of computed measures.⁶⁵ Such human-rated criteria can be generated through expert-rating or literature review. Many techniques exist to elicit the criteria and their importance through comparisons of the alternatives.⁶⁶ Existing knowledge acquisition systems, such as AQUINAS, discussed the considerations of incorporating such comparisons as knowledge to elicit decision criteria.⁶⁷

Suitable criteria are chosen based on the identified contexts of the analysis (e.g. available time to generate and evaluate alternatives) and the decision point in question (e.g. algorithmic decision vs visualization decision). To not only communicate the alternatives and their computed “goodness,” but also their trustworthiness, their uncertainty may also be established and shown as meta-data for each alternative’s rating.^68,69 This is particularly important for criteria derived from partial or predicted results.

Step 2.3 – Construct the evaluation model

What

In this step, we combine the produced criteria for each decision point into an MCDA model to generate guidance.

Why

The evaluation model is an important element for generating guidance, as it forms the mechanism that produces the ranking among the alternatives that later enables guidance presentation (Stage 3). Here we consider how to build different evaluation models according to the varying degrees of guidance and user control, which allows the generated guidance to be adaptive and controllable.

How

MCDA methods come in various forms – Wątróbski et al. summarized 56 different MCDA methods and discussed how to choose the corresponding method to support different decisions.¹⁰ Overall, there are three types of models when it comes to combining criteria in MCDA:

• Functional approaches synthesize quantitative criteria into a single metric with assigned weights and optional value/utility functions for each criterion.⁷⁰

• Outranking methods choose, rank, or sort the alternatives through comparisons between them based on a set of quantitative or qualitative criteria and corresponding weights.⁶⁵

• Decision rules evaluate alternatives based on certain conditions and logic constraints that are often formulated as an “if…, then…” structure.⁷¹

To use MCDA models for generating guidance, the construction of them heavily depends on how the guidance should be used. Here we present two factors to consider – degree of guidance and level of user control.

MCDA models can adapt to varying degrees of guidance:

• As functional approaches produce the evaluation of alternatives as a single metric, they can be used to filter the alternatives with certain thresholds to provide orienting guidance, rank the alternatives by the produced metric to provide directing guidance, or select the highest ranked alternative to provide prescribing guidance. For example, the feature subset selection by May et al. filters and prioritizes features based on statistical ranking measures.⁷²

• For generating guidance, outranking methods can be used similarly to functional approaches, except for their possibility of directly taking in qualitative criteria. Depending on the specific model that either ranks or discards alternatives, they can also either produce a ranking among the alternatives to provide directing guidance with the ranking result and prescribing guidance with the highest ranked alternative, or evaluate if the alternatives are acceptable to provide orienting guidance by filtering unacceptable alternatives.

• Decision rules often have the form “if alternative $a$ is between $x$ and $y$ in criteria $c$ , then $a$ is a good enough alternative”. Hence, they can be used to filter alternatives for orienting guidance. For example, the underlying mechanism of “Show Me” in Tableau filters out visualizations not applicable to selected data based on a set of similar rules.⁷³ Decision rules can also be used to direct users along branching analysis workflows as directing or prescribing guidance, like the Stack’n’Flip approach does.⁷

Guidance can further be controlled explicitly or implicitly through user input.⁸ Here we discuss how MCDA models can be constructed to allow different levels of user control, including presets with no user control, inferences with implicit control, and direct input with explicit control.

• To construct a basic MCDA model, the appropriate weights, value/utility functions, and decisions rules for the criteria can be preset by the VA designers without user input, especially when users do not have detailed knowledge about these criteria and how to weight or constrain them.

• Furthermore, MCDA methods can also be controlled through implicit inference from user interactions. This is done through inferring some of the elements in the MCDA models based on user preferences produced by certain interaction patterns, such as mouse movement and user-generated materials.^8,74 In functional approaches, value/utility functions can be elicited from a partial ranking of alternatives,⁷⁵ and weights can be elicited from the users’ evaluation on the importance of the criteria.⁷⁶ Decision rules can also be flexibly modified, for example by inferring additional rules such as “if condition $c$ occurs, then alternative $a$ is more preferable than alternative $b$ ” from user choices made in the past.

• Experienced users who have abundant knowledge about these criteria and alternatives can also be exposed to the underlying evaluation mechanism and afforded with explicit and direct control of the elements in MCDA models. For functional and outranking approaches, this includes the criteria, their corresponding value entries, their weights, and optionally the value/utility function. For decision rules, this includes access to the rule set. Explicit control is also an important function for debugging guidance that does not work as expected.

Stage 3: Choice – Multiple alternative views for guidance output

After the alternatives have been evaluated, users need to closely inspect and compare them in the context of their own domain knowledge and make their choices. To this end, the generated guidance must be communicated to the users for them to interact with and provide feedback to.

In the following, we summarize the considerations in presenting guidance in ways that enable users to visually inspect, compare, and reason with these alternatives. While research has indicated a potential for providing guidance through other modalities than visual output (e.g. using vibrotactile feedback⁷⁷) these approaches are still experimental at this point and require special hardware. Therefore, we focus on the visual channel to communicate guidance in this work, as it currently stands as the main modality for guidance in VA. We first recognize what data regarding the alternatives to visualize. We then discuss how to compose the guidance presentation based on the previous considerations. Finally, we summarize how the provided guidance can be adapted to user interaction and feedback. These three steps are listed in Figure 5.

Figure 5.

The steps in the Choice stage and the corresponding factors considered in each step.

Step 3.1 – Recognize the data to present

What

This step aims to specify what information/data about each alternative is relevant for users’ choices and should thus be presented.

Why

What data to present depends on the available results produced at Stage 2 and decides the content and level of detail with which to present each alternative in Step 3.2. Hence, this step connects the guidance generation stage with the following step of guidance presentation.

How

In the previous design stage, we have generated and evaluated the alternatives through MCDA. This process produces different types of data:

• The main output of the design stage is the evaluation of the alternatives, which is the essential data element for presenting guidance. For orienting guidance, this is a list of all acceptable/valid alternatives for the concrete analysis decision at hand. For directing guidance, this is a subset of the top- $k$ best alternatives among the acceptable ones given as a ranked list to indicate the priorities among them. For prescribing guidance, this is the highest ranked alternative from that top- $k$ list.

• The different criteria on which the evaluation model was based can also be shown. This can help users in understanding the characteristics of each alternative and uncover how the evaluation was generated. Yet, VA designers will need to consider if the users have the relevant background to interpret these criteria to avoid confusion or information overload.

• Some of these evaluation criteria might have been produced from full or partial results precomputed for each alternative during Step 2.2. Presenting these results can help users to inspect the detailed differences between these alternatives. For example, visualizing the resulting subsets after data selections can help users to gain an overview of the differences among these selections. Techniques known from comparative visual analysis, such as algorithmically-enhanced visual comparison, can further aid this inspection by deriving difference metrics.⁷⁸

Step 3.2 – Compose the presentation of alternatives

What

Having described which data to present for each alternative, we then need to visually compose that data in order to display the guidance. In this step, we outline how to compose views of multiple alternatives based on the considerations we have made in the previous steps.

Why

Visualizing for guidance differs from other visualization in its fundamental characteristics of presenting and aiding the decision among multiple alternatives, such as the different levels of details and varying guidance degrees. Therefore, we outline the considerations for composing such visualizations that adapt to various characteristics of the guidance scenario at hand.

How

To visually present and compare the alternatives for VA decisions, multiple presentations of these alternatives need to be composed in a unified view. Techniques that combine multiple visualizations have been widely studied and given different names, such as coordinated and multiple views,^79,80 composite visualizations,⁸¹ and visual comparison techniques.⁸² However, to visualize for guidance, different data and numbers of alternatives need to be presented and varying guidance degrees call for dissimilar presentations. Therefore, additional considerations need to be made in terms of presenting multiple alternatives. Abstracted examples of the following considerations are illustrated in Figure 6.

Figure 6.

Abstracted illustrations of the guidance presentation according to varying prominence of guidance (VA – Visual Analytics, G – Guidance), signification of guidance, and level of detail.

The prominence of guidance refers to how visually noticeable the provided guidance is in comparison to the existing VA elements, which can range from being almost invisible and implied in the order of the options provided to users to a dedicated visualization of the options that is added to the existing VA system. Here we discuss the prominence of guidance by considering the visual relationship between guidance and the original VA system.

• With low prominence, guidance can be implicit without introducing any visual encoding or element. This can be done through prioritizing or constraining the alternatives provided to users – for example, re-ordering the alternatives in a drop-down menu or hiding undesirable alternatives.⁸³ To provide such implicit guidance, only the final evaluation from MCDA models is needed to rank the alternatives or eliminate alternatives under certain threshold.

• In existing techniques, guidance is most commonly embedded in VA systems through visual encoding, such as color, highlighting, and animation on top of the existing user interface.^12,84 The final evaluation produced by the MCDA models can be used to encode embedded guidance. The evaluation criteria and computation results can be provided on demand to help users understand the provenance of guidance.

• With high prominence, guidance can be expanded into additional views in VA systems to provide further details,^18,85 or to structure steps for users to follow.⁷ For MCDA models, the evaluation and criteria can be communicated through an individual visualization or table and helps users to fully understand the underlying mechanism of the provided guidance. The full or partial results computed for each alternative can also be visualized for users to compare them in detail.

To guide users through multiple alternatives, the signification of guidance can help to convey the relationships between alternatives and shift the focus to the important ones. For example, in Voyager, a “specified view” is put on top of other “related views,” signifying a higher relevance of the “specified view” to the context of use.⁴ Here we discuss the signification of guidance through the three guidance degrees – orienting, directing, and prescribing.

• For orienting guidance, the alternatives should be visualized with the same visual importance and avoid implying any preference. However, there might still exist some underlying relationships between the alternatives that can help users to orient among them. For example, when the users are branching out into different paths of analysis, the logical and chronological relationships between alternatives can be indicated through new alternatives branching out from previous ones.^85,86

• For directing guidance, the key consideration is the underlying preference among the alternatives. Such guidance can be encoded in the order of which the alternatives are ranked, especially when they are presented as a list, where the linear order already implies a ranking of the elements – whether this is intended or not (cf. position bias).⁸⁷ To further emphasize the ranking, color, textual information, size, and/or animation can be used to indicate the preferred alternatives and the ranking among them.⁸

• For prescribing guidance, users are guided through a process where they can only accept the suggested alternative and navigate back-and-forth between different steps. However, additional alternatives can still be useful to show so that users better understand the context in which the prescribed alternative is generated. In this case, the additional alternatives can be presented in a de-emphasized way without interactivity to indicate that they are not available to be chosen and only shown as contextual information.

The level of detail of each alternative can be influenced by many factors. For example, the available screen size and mental resource in context of use limits the level of details of presented alternatives. A higher number of alternatives may also limit how detailed each alternative can be shown. And the preferred alternative in directing guidance or the suggested alternative in prescribing guidance may be shown in greater detail. In the following, we discuss which options we have to accommodate different levels of detail.

At the highest level of detail, the alternatives can be individually instantiated and then combined. This is often used to present a list or a grid of alternatives, similar to visualization spreadsheets.⁸⁸

When there are more alternatives, especially with some form of underlying relationships between them, their individual visualizations can be combined into one visualization– either a larger visualization,⁸⁹ or an ensemble visualization for showing general patterns and trends among the alternatives.⁹⁰

At the lowest level of detail, each alternative can be abstracted to an individual data point. The relevant metrics of each alternative can be abstracted in a single visualization or be directly encoded in the display of the alternatives using scented widgets.⁸⁴

Additionally, these different levels of detail can also be combined to provide more adaptive and contextual guidance – the higher ranked or more important alternative(s) can be shown in greater detail with other alternatives abstracted into another visualization on the side, and more details about the abstracted alternatives can still be provided on demand when users hover over, zoom in, or select an alternative. In case the number of alternatives becomes too large to be shown at the desired level of detail, ensemble-based methods can be used to subsume multiple alternatives using a few representative exemplars.⁹¹

Step 3.3 – Adapt to user feedback

What

In this final step, we consider how to adapt the provided guidance and its underlying MCDA model to user feedback to ensure the guidance is adaptive and controllable.

Why

The MCDA models in our guidance generation come with different means of criteria production and allow for different types of user input, enabling them to take user feedback into account. Using this possibility of adapting the guidance to the specific demands of data, task, and user fully enables the benefits of MCDA models.

How

As users interact with VA systems, many of their interactions can be recorded and analyzed as user feedback to adapt the provided guidance accordingly. To this end, the different components of guidance design must take into account the possibility of implicit (inferred) or explicit (input) user feedback. Making users aware of adaptations due to feedback is essential for them to make full use of the provided guidance – the occurrence and origin of an implicitly inferred adaptation should be communicated to users to avoid confusion, and the availability of direct control should likewise be signified.

Here we discuss three general perspectives on how guidance can be adapted to user feedback corresponding to the overall goals of the three stages of our method – considering whether or not guidance should be present as determined during the intelligence stage, the generation of guidance at the design stage, and the presentation of guidance at the choice stage.

Presence of guidance: To ensure the provided guidance is non-disruptive, it should be possible for users to override the pre-determined need for guidance (Stage 1) by turning it off when not needed. Such mechanism can be directly accessed by the users through interactive elements on the user interface or prompted by user interactions.

Generation of guidance: To provide adaptive and controllable guidance, the evaluation model pre-determined in Stage 2 can be interactively adapted. This can for example be achieved by including interaction metrics as criteria in the models themselves (e.g. the interaction history)¹² to automatically update the evaluation results based on user actions. Another option is for users to directly manipulate the models when they have the expertise to do so.

To adapt the presentation of guidance to different contexts of use, the elements in the previous steps in this stage can also be made flexible – the guidance degree and detail level can be increased or decreased accordingly when users need more or less guidance. Such change can also be either made directly or inferred from the user interaction logs.

Worksheets for applying the method

Passing through the intelligence, design, and choice stages, we have provided a step-by-step method to design guidance for decision making in VA. In doing so, our method establishes a unified, reusable, and widely compatible guidance mechanism that generates transparent, adaptive, and consistent user guidance.

To provide concrete means that allow VA designers to put our method into practice, we developed a set of guidance design worksheets to accompany our method. We chose the format of worksheets, as they are accessible, flexible to be edited for different contexts, and well-suited for generating ideas and pen-and-paper prototypes especially in early design stages. Design worksheets are also a common in many aspects of visualization, such as visualization design,^92,93 teaching and learning,⁹⁴ and creative ideation.⁹⁵

To produce the worksheets, we went through a series of internal and external iterations to refine their design. In the beginning, we iterated on the worksheets three times internally. We started out by drafting the outline of each stage and step of our method on the worksheets. Then we went through a round of discussions that generated a list of improvements. After implementing them, we filled out the worksheets ourselves for a use case scenario of clustering analysis, and refined the worksheets based on this trial.

After the internal iterations, we went through a series of external iterations held in a workshop format. Three workshops were individually conducted with three experts and lasted around 90 min each. Two of the experts were VA researchers with experience in designing VA systems, and the third was a researcher in Data Visualization and Human-Computer Interaction. The workshops were semi-structured and consisted of three parts: introduction, method walk-through, and follow-up questions. We started the workshops with a short introduction to the overall concept and examples of guidance in VA, the structure of our proposed method, and a basic example of MCDA. In the method walk-through, we first asked participants to identify the context of a VA system that they designed or used. Thereafter, they were asked to identify the decision points that were relevant to the users’ workflows in the intelligence stage, develop a guidance generation mechanism for one important decision point in the design stage, and formulate the presentation of the alternatives in the choice stage. Finally, we ended the expert workshops with follow-up questions that we prepared and adapted with the observations from the method walk-through. In these questions, we focused on what could be improved in the worksheets and if the participants saw the benefits of using our method to formulate guidance.

Overall, the participants were all able to follow the steps in the worksheets, despite having some troubles articulating some of the key concepts. The two VA experts successfully identified a guidance problem in a VA system they previously designed and came up with a guidance solution with our worksheets, while the visualization expert was able to follow a set of pre-filled worksheets and provided some suggested improvements on the method and worksheets. In particular, they suggested to further clarify some of the key concepts in the method and emphasize the connections between the steps. Specifically, in the intelligence step, the meaning of the VA components –“data,”“algorithm,”“visualization,” and “reasoning,” as well as the types of knowledge gap –“which (scheme)” and “how (parameter)” can be challenging to delineate. We made revisions accordingly to produce our final worksheets. To clarify the key concepts, we added explanations to the worksheets. We emphasized the connections between the steps in the prompts to make the worksheets easier to follow.

The abstracted worksheets are shown in Figure 7. The detailed worksheets can be found at https://vis-au.github.io/guidance/.

Figure 7.

The abstracted worksheets accompanying our method correspond to its three stages, with three steps in each stage. Each worksheet consists of a title indicating the design progress, goal, and deliverable of the corresponding stage. For each step, the primary question to answer is presented with the keywords underlined. A prompt for each step outlines what the VA designers should achieve, and detailed instructions are provided to explain some of the key concepts.

Use case example and prototype

To provide a practical example, we present a use case based on our guidance method. To this end, we apply our worksheets on a scenario of cluster analysis and design an initial prototype through our method using existing visualizations and MCDA tools.

In the following, we describe how we apply each step in our guidance method in this use case, illustrate the developed prototype along the steps, and present the users’ workflow after implementing the guidance. A set of filled worksheets was completed along the process and can be found in the supplementary materials. For the prototype, we first drafted early iterations through pen-and-paper mockups and then implemented the prototype in Python. We utilized Bokeh for the visualizations, scikit-learn for the processing algorithms, and Scikit-Criteria for the MCDA methods. The prototype is available at https://vis-au.github.io/guidance/.

Stage 1 – Intelligence

We start our guidance design through a context of use analysis. The users in our context of use are epidemiologists who have domain knowledge about the diseases under study and knowledge about different VA methods. As the outcome of their analyses will inform future medical treatments and impact patients’ lives, they also tend to be less risk-taking in their decision making styles. Their goal is to explore and identify different types of patients for the same disease. To achieve such goal, the epidemiologists go through a series of tasks– they first clean the data and visualize them along some important features to first see if there are already some patterns in the patients. To further bring out these patterns, they then utilize dimension reduction and clustering algorithms to abstract the features and cluster the patients in different groups. During this process, they will need to experiment with different algorithms and parameters, then visually observe which set of results helps them to identify and determine the patient groups based on their expertise. As for resources, they often have limited manpower and computing power for a specific project before it proves promising. The environment that epidemiologists work in is usually a lab with desktop computers, where they can often be distracted with other ad-hoc tasks and experiments to run.

With the context analyzed, we then go through the four components of the users’ VA process in order to identify for which decision points guidance is needed. In the context of high-dimensional medical data, the epidemiologists would need to start with selecting the data dimensions to use and dealing with missing data that often appear in the medical context. With the data prepared, they then need to choose and parameterize the algorithms, such as dimension reduction and clustering, for grouping patients into different types. Finally, to uncover the patterns and insights from these computations, their results then need to be presented in visualizations with appropriate encodings and specifications to help epidemiologists visually observe these results and infuse their domain expertise. During these processes, the epidemiologists would also need to reason about how to combine the available data, algorithms, and visualizations together in order to uncover the insights they set off to seek. These reasoning decisions also govern exploratory analyses where the epidemiologists might go back and forth iteratively to try other alternatives, such as changing the included data dimensions after an initial exploration of the clustering results. The decision points are summarized in Table 1.

Table 1.

A list of decision points we recognize in the use case of clustering analysis through the 4 components – data, algorithm, visualization, and reasoning, as well as 2 types – which (scheme) and how (parameter) that we discussed in Step 1.2.

Type	Data	Algorithm	Visualization	Reasoning
Which	Data dimensions to use	Dimension reduction and clustering algorithms to apply	Encoding of visualization to apply	Insights to uncover
How	Deal with missing data	Parameterize the algorithms	Spec the visualization	Combine other components

Based on the context and the identified decision points, we then assess these decision points according to the three factors of guidance need on a scale of 0–3 – with “0” to signify the factor as “not important at all” and 3 to signify the factor as “extremely important.” We multiply the factors and rank them from high to low to yield the final assessment (see Table 2). We chose to include 0 as the lowest point on the scale and to multiply the factors, as we consider if any factor in the assessment is 0 (not important at all), then there is no need for guidance at the corresponding decision point. For the data component, which data dimensions to use is a particularly important decision, as it directly impacts the final results of the analysis and some of the following decisions, such as which clustering algorithm can be used for the corresponding data types. For the algorithm component, different clustering algorithms and different numbers of clusters produce various stratifications of the patient cohort – each capturing a different property or insight into the disease. However, which clustering algorithms to explore and which sets of results are likely to contain reasonable stratifications are not known beforehand, and the epidemiologists would have to experiment with many of them to find the “needle in the haystack.” This is where guidance becomes particularly helpful to point them in the directions of the most promising clustering results and achieve their goals of identifying patient types. For the visualization component, clustering results (i.e. cluster membership of data items) are often encoded through colors in scatterplots, while the specifications can be more complex with different color palettes, sizes, and inclusion of other facilitating elements, such as centroids and convex hulls. Finally, as the goal of the analysis is clearly defined as finding different groups in patients, the reasoning decisions are less of a concern in this context of use.

Table 2.

Example inventory ranking the decision points’ need for guidance for the use case of clustering analysis.

Decision points	Potential of wrong decision	Impact	Alternatives	Final assessment
Clustering Algorithm (Which)	3	3	2	18
Data Dimension (Which)	2	3	3	18
Clustering Parameters (How)	3	3	2	18
Visualization Specifications (How)	2	2	3	12
Dimension Reduction Parameters (How)	3	2	2	12
Dimension Reduction Algorithms (Which)	3	2	1	6
Visualization Encoding (Which)	2	2	1	4
Missing Data (How)	3	1	1	3
Order of the Algorithms (How)	3	1	1	3
Insights to Discover (Which)	1	3	0	0

We assessed the factors on a scale of 0–3 in their severity and multiplied the three factors to produce the final assessment.

Stage 2 – Design

From the inventory of decision points, we have ranked the decision points and obtained the following three decision points with highest priority: clustering algorithms, parameters, and data dimensions. For the purpose of exemplifying our method, we focus on the two decision points of clustering algorithms and parameters, as these two decisions are interconnected and their guidance should be developed in conjunction.

The first step in this stage is to recognize the space of alternatives. The number of alternatives among the clustering algorithms is often not very large. For example, in the overview of clustering methods in the Machine Learning package for Python, scikit-learn, 11 algorithms are listed.⁹⁶ In our context of analyzing clusters of patients, the epidemiologists need to directly manipulate the number of clusters to explore the different resulting stratifications of the cohort of patients. Therefore, we focus on the 5 algorithms with “number of clusters” as an available parameter – K-Means, Spectral Clustering, Ward Hierarchical Clustering, Agglomerative Clustering, and BIRCH. In our context, there are often not many different types of patients for the same disease, so the range for the number of clusters is also limited. Therefore, we set the range for the number of clusters from 2 to 10.

To produce the criteria, the evaluation of clustering algorithms and parameters are often based on similar metrics. In our case, the extrinsic measures based on a ground truth of actual class labels are not available. Therefore, intrinsic measures such as Silhouette Coefficient, Davies-Bouldin Index, and Calinski-Harabasz Index can be used to evaluate the separation and consistency of the clustering results.⁹⁷ These measures need to be calculated based on actual clustering results, but clustering the full dataset can take too much time due to the limited computing power. Therefore, we produce these metrics with a small but representative sample of the full dataset, while users can also change the sample size according to their needs. As we run these different algorithms on the sampled data, we can also measure their runtime.

To construct the MCDA model, we then consider the degree of guidance and level of control. As the users in our context have some knowledge regarding the algorithms and would like to explore alternative results, we decide to apply primarily directing guidance with additional orienting guidance to help. To provide directing guidance, the aforementioned criteria then need to be combined and some form of preference or recommendation should be derived from the evaluation. We combined the multiple criteria with through weighted sums. The weights of each criterion were set based on our experiences in working with epidemiologists. Additionally, as the users in our context are able to understand the generated results and metrics, we then orient the users by presenting the clustering results and corresponding criteria from each algorithm and parameter. This helps to trace how the directing guidance was generated and orient users among the alternatives with detailed information. For allowing users to adapt the guidance to their specific needs, we present the underlying weights of each criterion and enable the users to manipulate the weights when they are not content with the provided guidance.

Stage 3 – Choice

To present the guidance, relevant data to be presented need to be recognized first. We present the produced results based on sampled data for each alternative, including the data points and their predicted class label. Furthermore, we present the meta-data including the evaluation criteria – specifically Silhouette Coefficient, Calinski-Harabasz Index, Davies-Bouldin Index, and runtime.

Next, we consider the degree of guidance and level of detail to produce the guidance presentation. For both of the decision points of clustering algorithms and parameters, we primarily apply directing guidance. Therefore, the preferences among the alternatives need to be signified accordingly. In the view of the produced results, we signify the ranking of the alternatives by highlighting the highest ranked one. Additionally, the evaluation table and figure of the algorithms and number of clusters also help to orient users among the alternatives by listing the alternatives and the evaluation results. For the level of detail, the number of alternatives for clustering algorithms is limited to the 5 algorithms that have “number of clusters” as a parameter. Therefore, they could all be presented in detail with the produced results and criteria. For the “number of clusters” parameter, we have 9 alternatives ranging from 2 to 10 clusters. We decide to visualize their evaluation results as a line chart on the side, where the users can hover over the data points to inspect the underlying criteria. The prototype with guidance is presented in Figure 8.

Figure 8.

The annotated prototype for the use case with guidance produced based on our method: (a) prompts users for their analytic decision, (b) shows the criteria underlying the decision and how they are weighed, (c) and (d) show the results of computing the criteria for the different decisions – the different clustering algorithms and the different numbers of clusters, respectively, (e) previews the outcome of the highest-ranked decision, and (f) previews the outcomes of other alternatives.

Finally, we adapt the provided guidance to user feedback. For the presence of guidance, we allow users to manually enable or disable the guidance elements (evaluation and ranking of alternatives), especially when users do not need to inspect or do not understand these detailed criteria. This also influences the presentation of guidance – with the evaluation table and figure that provide orienting guidance hidden, the system only provides directing guidance by indicating the highest ranked algorithm with its enlarged size.

Use case example with guidance

After the guidance implementation, users enter their analysis with clear priorities and supporting information for the two decision points. For the algorithms, the enlarged view (Figure 8(e)) and highlight color (Figure 8(c)) guide users’ attention to the highest ranked algorithm at the first glance. The detailed evaluation metrics (Figure 8(c)) that generated the underlying guidance support the comparisons between the algorithms with important knowledge, while encouraging users to consider different alternatives. For the number of clusters, the evaluation chart (Figure 8(d)) also indicates how this decision point might influence produced results with the selected algorithm, guiding users to compare and consider different numbers of clusters for their analysis. Moreover, as users change the number of clusters, the guidance is updated accordingly to encourage users to consider how different combinations of algorithms and numbers of clusters might impact their analysis. With the implemented guidance for the two decision points, we bring out several benefits to users that address important challenges they might encounter.

First, the direct presentation of alternatives (Figure 8(e/f)) makes users aware of the decision points and their potential impacts on the results of the clustering analysis, encouraging users to explore the space of alternatives.

Second, the visual cues of color and size (Figure 8(c/e)) indicate the highest ranked alternative, guiding users toward it. Meanwhile, how the underlying mechanism generated this ranking (Figure 8 (b/c/d)) is shown to support the trustworthiness of the provided guidance.

In addition, the presented visualizations (Figure 8(e/f)) and evaluation criteria (Figure 8 (b/c/d)) allow the users to easily compare the alternatives, reason about them in a way that is grounded in the metrics, externalize their decision making process, and construct their evaluations, which in turn reduces their cognitive load. The possibility of manipulating the underlying criteria (Figure 8 (b)) enables users to adapt the guidance to their expertise and context of use, making the guidance design adaptive, flexible, and controllable.

Relations to existing guidance frameworks

To put our step-by-step design process in the context of existing research, this section discusses its commonalities and differences with four guidance frameworks that are closely related to ours:

Collins et al.’s paper collects a range of highly useful thoughts, arguments, models, and building blocks capturing guidance from its different perspectives of goals, requirements, roles, tasks, implementation, and evaluation.⁸ The common theme underlying this paper is that of an “intelligent guide” or an “artificial intelligence-guided visualization.”

Ceneda et al.’s paper introduces a framework for making design decisions on guidance functionality to be provided during visual-interactive data analysis.² It details aspects such as the requirements and goals of guidance, the knowledge gaps it addresses, the generation of guidance, and the users’ feedback.

Pérez-Messina et al.’s paper proposes a typology of guidance tasks that connects the concept of guidance – its degrees (orienting, directing, prescribing) and the knowledge gap it addresses (target unknown, path unknown) – with user tasks (mainly search tasks).⁹⁸ This typology is not a design framework in itself, but it supports guidance design with a nuanced abstraction of guidance tasks.

Sperrle et al.’s paper describes a syntax to specify guidance functionality on the implementation level.⁹⁹ Its compact notation and low overhead allow for rapid prototyping of guidance, which makes it a good fit for iterative guidance design.

In the following, we will discuss these frameworks and our guidance design method with regard to common requirements and different contributions to guidance in VA.

Common requirements for effective guidance

What makes for “good” guidance in Visual Analytics? Ceneda et al. state five requirements for “effective” guidance –available, non-disruptive, adaptive, trustworthy, and controllable.² Collins et al. includes similar requirements for “intelligent” guidance with different wording –effective, adapted to the context (contextual), white-box, and right timing and mode (see Figure 9).⁸ Note that effective means in this case that the provided guidance should be “easily accessible” and “avoid distraction or obscuring the current visualization.” This corresponds more closely to the requirements of available and non-disruptive from Ceneda et al. than to their overall goal of “effective” guidance.

Figure 9.

The requirements for guidance design from Ceneda et al.² (in blue) and Collins et al.⁸ (in red).

A requirement that has received a lot of attention in guidance research lately is that of providing adaptive guidance, as it is called by Ceneda et al. This requirement appears under the name contextual guidance in Collins et al. emphasizing that guidance “should be adapted to the context of the user analysis process.” Sperrle et al. cast this notion of adaptive guidance into a dedicated framework,¹⁰⁰ used it in complex guidance scenarios,¹⁰¹ and built their recent guidance syntax around it.⁹⁹

Furthermore, Ceneda et al. discuss the importance of non-disruptive guidance. This notion can also be found in Collins et al.’s requirement of providing guidance at the right time and in the right mode, which explicitly discerns between synchronous guidance that may disrupt and intervene with the analysis process, and asynchronous guidance that can be used or ignored as needed. The task typology by Pérez-Messina et al. makes prominent use of this distinction in their framework as well.⁹⁸ Interestingly, neither Collins et al. nor Pérez-Messina et al. connote disrupting or synchronous guidance necessarily as improper or inadvisable, but instead highlight the importance of making an informed and explicit choice about it. After all, guidance should never be followed blindly and it is thus important that guidance is provided explicitly and noticeably.

Our method

Many elements of the guidance design method presented here were purposefully included to meet these requirements. Hence, it is not surprising that a certain overlap exists, for example, between the considerations in Stage 1 – Intelligence and the first two steps in Ceneda et al.’s design framework. Through identifying and evaluating the decision points in a structured manner, we ensure the guidance is available when it is needed, yet non-disruptive. Furthermore, our underlying MCDA models can adapt to different guidance degrees and levels of user controls, making the produced guidance adaptive. As our method emphasizes the possibility of exposing the MCDA model to users, we also enable the produced guidance to communicate essential information about its trustworthiness and to act as a white-box through revealing the underlying mechanism. Finally, through adapting the guidance presence, generation, and presentation to user feedback, we further strengthen the ability of provided guidance to be easily controllable.

Different contributions to VA guidance

All mentioned frameworks address guidance from their own distinct perspectives and therein make different contributions to VA guidance that range from conceptual models to designing and ultimately implementing guidance. A summary of these contributions is given in Table 3.

Table 3.

Comparisons of our work with selected papers on guidance design in VA.

Paper	Contributions to VA Guidance
	Conceptualizing	Designing	Implementing
Collins et al.⁸	•		○
Pérez-Messina et al.⁹⁸	•
Ceneda et al.²		•
Sperrle et al.⁹⁹		•	○
our method	•	•

• Indicates the corresponding work includes the element as a main focus, while ○ indicates the work discusses the element without detailing specific processes or tools.

Conceptualizing guidance

From its inception, guidance in VA was likened to a car navigation system for visual-interactive analyses.¹ This metaphor is picked-up again by Collins et al. who point out the gap between this aspiration and the reality of current guidance functionality.⁸ As the main issues causing this gap, they identify that existing guidance models – particularly Ceneda et al.’s original guidance characterization – are rather abstract and too far removed from practice to be useful. The paper then goes on to alleviate this issue by extending and detailing different aspects of guidance – for example, task abstraction, user roles, implementation, and evaluation – to make the concept more actionable. In particular the aspect of task abstraction is then revisited in more detail by Pérez-Messina et al.’s typology.⁹⁸

Designing guidance

While the conceptual papers describe the tools for providing guidance (e.g. requirements, tasks, building blocks for implementation and evaluation), they do not detail how to use them to get from an identified knowledge gap to a suitable guidance solution. These more procedural concerns are addressed by Ceneda et al.’s four-step framework for guidance designers in which they propose a sensible order in which to make the different necessary considerations for arriving at guidance solutions.² As this four-step framework aims to anticipate user problems at design time, it has only limited possibilities to adapt to changing user needs as they emerge at runtime during exploratory, open-ended analysis sessions. Consequently, Sperrle et al. call this a theoretical design process and instead propose a strategy-centered guidance design in which various different guidance strategies are developed at design time and then chosen dynamically as needed at runtime.⁹⁹

Implementing guidance

Having a guidance design – that is, all questions regarding the why, when, what, and how of guidance are answered – still does not realize the thus specified guidance in code. To that end, Collins et al. provide a high-level discussion on guidance implementation (e.g. sources of information, computational processes, interaction modalities) using an input-compute-output structure.⁸ These can help to further detail the design considerations into implementation considerations, but it still does not lead to actual code. The framework from Sperrle et al. takes the next step in tackling this last mile bridging specification to implementation of VA guidance.⁹⁹ By expressing their guidance strategies through a declarative grammar, they are able to automatically generate guidance functionality from the specification. This narrows the gap between design and implementation considerably, leaving open only the implementation of the frontend for showing the guidance and interacting with it.

Our method

As can be seen, all existing frameworks have a particular focus in their contributions to VA guidance, and so does our method:

Conceptualizing Guidance: Our method reframes and extends the common notion of “guidance for knowledge gaps” to “support for decision points.”

Designing Guidance: This new perspective on guidance allows us to re-structure the guidance design process along the decision making process.

Implementing Guidance: Our framework is first and foremost a method for design and not for implementation. That being said, the use of existing decision support systems – in our case the MCDA models from Scikit-Criteria – may serve as a viable way to use off-the-shelf implementations to generate guidance.

In this way, our method combines a new way of thinking about guidance with a matching way of designing guidance. To bridge the gap between theory and practice, we provide the worksheets as practical tools to be used for guidance design and we root our method in the MCDA approach for which a wide range of libraries and packages in different programing languages are available to jumpstart guidance generation without much additional overhead.

Discussion of limitations

Throughout this paper, we have boldly called our design method a “generic” approach to providing guidance. In many regards, this statement holds true: Our method does not apply only to a single guidance domain or use case. It can be used to provide both types of guidance: assisting with unknown analysis targets and analysis paths. It is also not tailored to a particular guidance degree or constrained to a certain input.

Nevertheless, our guidance design method relies on a particular understanding of what guidance is and what it is not. It furthermore makes assumptions about the guidance problem that, if not met, limit the applicability of our method. These two aspects are briefly discussed in the following.

Scope of our method

The most common notion of guidance is that of an enhancement to an underlying unguided visual analysis process. Despite this agreed-upon intuition on what guidance means in the field of visualization and visual analytics, its delineation remains complicated. For example, it is not unreasonable to argue for any of the following to be forms of guidance:

overview and detail visualization – for example, a minimap providing information about the larger context of a currently inspected view region;

intrinsic emphasis effects¹⁰²– for example, directing the viewers’ attention to outliers in the data simply by choosing a scatterplot for showing the data;

context-sensitive help¹⁰³– for example, providing a tooltip with additional information when hovering with the mouse cursor.

Following this line of thought, any mechanism that provides additional information (minimap, tooltip, labeling, help texts, etc.) can be understood as an instance of informing guidance devised to resolve knowledge gaps. Any visual encoding that draws the reader toward the important aspects of the data can be understood as a form of directing guidance. Yet, if these are indeed instances of guidance then where does the plain and unguided interactive visualization end and where does guidance begin? After all, visualizations are in themselves tools to resolve knowledge gaps about the data. Otherwise, if everything about the visualized data was already known, why visualize it or even perform visual analytics on it? In that sense, any effective visualization design should be done in a way that guides the users’ attention toward the salient parts of the shown data and provides details on demand. Yet, our design framework is clearly not meant for general visualization design. Hence, it does not cover any of the above examples very well if one wants to consider them as guidance.

Assumptions underlying our method

Even if the scope of guidance to be provided fits the understanding of an enhancement to an unguided base visualization as discussed above, our method is not necessarily ideal for designing guidance in all cases. The reason is that our method is targeted to solve guidance problems characterized by three assumptions. The first fundamental assumption made by our framework is that all guidance problems can be framed as decisions in need of decision support. To back this assumption, we made the argument that all user tasks involve decisions as otherwise there was no need for the user in the first place. Hence, we reasoned that any task in need of guidance can be equated with a decision in need of decision support. This is true for many high-level analytic tasks. However, guidance for low-level tasks may not necessarily involve a decision making component. For low-level tasks, guidance is often provided to simply reduce interaction costs by automating particularly tedious aspects of the task in a prescribing manner. For example, interactive snap-to-grid brushing aids in precisely anchoring brushes at grid points, effectively taking over and guiding the hand in moving the mouse cursor toward the exact pixel spot once getting close to it.¹⁰⁴ Using the aforementioned analogy of a car’s navigation system,¹ this functionality could be likened to the self-parking feature of some cars as a means to navigate those last few meters. The second fundamental assumption of our design framework is that the decisions to be supported can be identified during the design phase of a VA system. Supporting this assumption, we have argued that our framework can and should be applied iteratively, so that crucial decisions in need of support are recognized as early as possible at different stages of the design cycle. Yet in particular for decisions identified in the very early stages of designing a VA system, it can be hard to already formally grasp evaluation criteria for an MCDA model as it is still unclear at this point what makes one alternative preferable over another. In these cases, the framework from Ceneda et al. might work better as a first design iteration to bootstrap the iterative process.² The last assumption is that a dedicated decision support mechanism can be put in place before a decision needs to be made. Yet, it is the very nature of exploratory visual analysis to try new analysis paths and approaches in order to discover the unexpected in the data. This is likely to lead to new and unforeseen analysis situations for which no preconceived guidance mechanism exists. Yet it is exactly these uncommon analysis situations in which guidance would be needed most. This conundrum is not specific to our design method, but shared with all existing design frameworks. The idea of guidance strategy templates proposed by Sperrle et al. that fit a wide variety of different concrete guidance situations – even those that were not explicitly anticipated when formulating them – might be a viable way to circumvent this problem in the future.⁹⁹

Conclusions

In this paper, we introduced a generic guidance design method through decision support. To this end, we re-framed guidance as supporting decision points, presented and detailed our generic guidance method through three steps for each of the three stages in the decision making process, and produced a set of flexible worksheets through internal iterations and expert workshops based on our method. Additionally, we developed an initial prototype with the produced worksheets to exemplify the usefulness and applicability of the method and discussed our method in the context of existing general research on VA guidance. With both theoretical and empirical insights, this work expands the concept of guidance in VA with decision making research and provides a practical guidance method covering the end-to-end process of producing guidance in VA.

As for future work, the discussion of relations between ours and existing frameworks make it obvious that a “nested model of guidance design” akin to Munzner’s nested model for visualization design is almost within reach.¹⁰⁵ By applying the different frameworks at the different levels of such a nested model – for example, Pérez-Messina et al.’s task typology for the task abstraction and Sperrle et al.’s Lotse guidance library or our MCDA models at the algorithm level – it should be possible to realize a nested design process specifically for VA guidance. Likewise, different aspects of Collins et al.’s considerations for evaluating guidance could then be related as validation measures to the different levels. For example, their suggestion to do user studies with “complex information seeking tasks” could be used to validate the guidance design, while the “metrics for automated monitoring” could be useful to benchmark the algorithmic implementation of the guidance generation. Filling in the still missing pieces to complete this picture – for example, a data abstraction for guidance in VA to complement the existing task typology – pose formidable research challenges in this direction.

Supplemental Material

sj-pdf-1-ivi-10.1177_14738716221147289 – Supplemental material for Providing visual analytics guidance through decision support

Supplemental material, sj-pdf-1-ivi-10.1177_14738716221147289 for Providing visual analytics guidance through decision support by Wenkai Han and Hans-Jörg Schulz in Information Visualization

Supplemental Material

sj-pdf-2-ivi-10.1177_14738716221147289 – Supplemental material for Providing visual analytics guidance through decision support

Supplemental material, sj-pdf-2-ivi-10.1177_14738716221147289 for Providing visual analytics guidance through decision support by Wenkai Han and Hans-Jörg Schulz in Information Visualization

Footnotes

Acknowledgements

We thank the participants of the expert workshops for their time and feedback, as well as the anonymous reviewers for their input on this work.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: the Innovation Fund Denmark (IFD) through the Grand Solution project “Hospital@Night”.

ORCID iDs

Wenkai Han

Hans-Jörg Schulz

Supplemental material

Supplemental material for this article is available online.

References

Ceneda

Gschwandtner

May

, et al. Characterizing guidance in visual analytics. IEEE Trans Vis Comput Graph 2017; 23(1): 111–120.

Ceneda

Andrienko

, et al. Guide me in analysis: A framework for guidance designers. Comput Graph Forum 2020; 39(6): 269–288.

Tominski

Abello

Schumann

CGV—An interactive graph visualization system. Comput Graph 2009; 33(6): 660–678.

Wongsuphasawat

Moritz

, et al. Voyager 2: Augmenting visual analysis with partial view specifications. In: Proceedings of the CHI conference on human factors in computing systems (CHI’17), pp. 2648–2659. New York: ACM. DOI:10.1145/3025453.3025768.

Leite

Arleo

Sorger

, et al. Hermes: guidance-enriched visual analytics for economic network exploration. Vis Inform 2020; 4(4): 11–22.

Müller

Assent

Krieger

, et al. Morpheus: Interactive exploration of subspace clustering. In: Proceeding of the 14th ACM SIGKDD international conference on knowledge discovery and data mining (KDD’08), pp.1089–1092. New York: ACM. DOI: 10.1145/1401890.1402026.

Streit

Schulz

Lex

, et al. Model-driven design for the visual analysis of heterogeneous data. IEEE Trans Vis Comput Graph 2012; 18(6): 998–1010.

Collins

Andrienko

Schreck

, et al. Guidance in the human–machine analytics process. Vis Inform 2018; 2(3): 166–180.

Ceneda

Gschwandtner

May

, et al. Guidance or no guidance? a decision tree can help. In: Proceedings of the EuroVis workshop on visual analytics (EuroVA’18), pp.19–23. Eurographics. DOI: 10.2312/eurova.20181107.

10.

Wątróbski

Jankowski

Ziemba

, et al. Generalised framework for multi-criteria method selection. Omega 2019; 86: 107–124.

11.

Ceneda

Gschwandtner

Miksch

A review of guidance approaches in visual data analysis: A multifocal perspective. Comput Graph Forum 2019; 38(3): 861–879.

12.

Gladisch

Schumann

Tominski

. Navigation recommendations for exploring hierarchical graphs. In: Proceedings of the international symposium on visual computing (ISVC’13), pp.36–47. Springer. DOI: 10.1007/978-3-642-41939-3_4.

13.

Luboschik

Maus

Schulz

, et al. Heterogeneity-based guidance for exploring multiscale data in systems biology. In: Proceedings of the IEEE symposium on biological data visualization (BioVis’12), pp.33–40. IEEE. DOI: 10.1109/BioVis.2012.6378590.

14.

Yang

Xie

Rundensteiner

, et al. Managing discoveries in the visual analytics process. SIGKDD Explorations Newsletter 2007; 9(2): 22–29.

15.

LrYi

Lee

, et al. ProReveal: Progressive visual analytics with safeguards. IEEE Trans Vis Comput Graph 2021; 27(7): 3109–3122.

16.

Gotz

Zhou

MX.

Characterizing users’ visual analytic activity for insight provenance. Inf Vis 2009; 8(1): 42–55.

17.

Dimara

Franconeri

Plaisant

, et al. A task-based taxonomy of cognitive biases for information visualization. IEEE Trans Vis Comput Graph 2020; 26(2): 1413–1432.

18.

Wall

Blaha

Franklin

, et al. Warning, bias may occur: A proposed approach to detecting cognitive bias in interactive visual analytics. In:Proceedings of the IEEE conference on visual analytics science and technology (VAST’17), pp.104–115. New York: IEEE. DOI: 10.1109/VAST.2017.8585669.

19.

Wang Baldonado

Woodruff

Kuchinsky

. Guidelines for using multiple views in information visualization. In: Proceedings of the Working conference on advanced visual interfaces (AVI’00), pp.110–119. New York: ACM. DOI: 10.1145/345513.345271.

20.

Stoiber

Ceneda

Wagner

, et al. Perspectives of visualization onboarding and guidance in VA. Vis Inform 2022; 6(1): 68–83.

21.

Conati

Hoque

Toker

, et al. When to adapt: Detecting user’s confusion during visualization processing. In: Proceedings of the workshop on user-adaptive visualization (WUAV’13), pp.17–24. CEUR Workshop Proceedings. https://ceur-ws.org/Vol-997/wuav2013_proceedings.pdf (accessed 30 September 2022).

22.

Walker

Harremoës

Rotmans

, et al. Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support. Integr Assess 2003; 4(1): 5–17.

23.

George

Duffy

Ahuja

Countering the anchoring and adjustment bias with decision support systems. Decis Support Syst 2000; 29(2): 195–206.

24.

Richardson

Fouquet

Kerns

, et al. Impact of mobile device-based clinical decision support tool on guideline adherence and mental workload. Acad Pediatr 2019; 19(7): 828–834.

25.

Silver

MS.

Decisional guidance for computer-based decision support. MIS Q 1991; 15(1): 105–122.

26.

Burstein

Holsapple

CW.

Handbook on decision support systems. London: Springer, 2008.

27.

Liu

Duffy

AHB

Whitfield

, et al. Integration of decision support systems to improve decision support performance. Knowl Inf Syst 2010; 22(3): 261–286.

28.

Lempert

Turner

Engaging multiple worldviews with quantitative decision support: A robust decision-making demonstration using the lake model. Risk Anal 2021; 41(6): 845–865.

29.

Bigaret

Hodgett

Meyer

, et al. Supporting the multi-criteria decision aiding process: R and the MCDA package. EURO J Decis Process 2017; 5(1-4): 169–194.

30.

Hakanen

Rados

Misitano

, et al. Interactivized: Visual interaction for better decisions with interactive multiobjective optimization. IEEE Access 2022; 10: 33661–33678.

31.

Jankowski

Integrating geographical information systems and multiple criteria decision-making methods. Int J Geogr Inf Syst 1995; 9(3): 251–273.

32.

Malczewski

GIS-based multicriteria decision analysis: a survey of the literature. Int J Geogr Inf Sci 2006; 20(7): 703–726.

33.

Liu

Althoff

Heer

. Paths explored, paths omitted, paths obscured: Decision points & selective reporting in end-to-end data analysis. In: Proceedings of the CHI conference on human factors in computing systems (CHI’20), pp.1–14. New York: ACM. DOI: 10.1145/3313831.3376533.

34.

Liu

Boukhelifa

Eagan

JR.

Understanding the role of alternatives in data analysis practices. IEEE Trans Vis Comput Graph 2020; 26(1): 66–76.

35.

Simon

HA.

The new science of management decision. The Ford distinguished lectures. New York: Harper & Brothers, 1960.

36.

Dimara

Stasko

A critical reflection on visualization research: Where do decision making tasks hide?

IEEE Trans Vis Comput Graph 2022; 28(1): 1128–1138.

37.

International Organization for Standardization. ISO 9241-11:2018 – ergonomics of human-system interaction; part 11: Usability: Definitions and concepts, 2018. https://www.iso.org/obp/ui/#iso:std:iso:9241:-11 (accessed 30 September 2022).

38.

Dourish

What we talk about when we talk about context. Pers Ubiquitous Comput 2004; 8(1): 19–30.

39.

International Organization for Standardization. ISO/IEC 25063:2014 – systems and software engineering; systems and software product quality requirements and evaluation (SQuaRE); common industry format (CIF) for usability: Context of use description, 2014. https://www.iso.org/obp/ui/#iso:std:iso-iec:25063 (accessed 30 September 2022).

40.

Urieta

Aluja

Garcia

, et al. Decision-making and the alternative five factor personality model: Exploring the role of personality traits, age, sex and social position. Front Psychol 2021; 12: 717705.

41.

Schulz

Nocke

Heitzler

, et al. A design space of visualization tasks. IEEE Trans Vis Comput Graph 2013; 19(12): 2366–2375.

42.

Cibulski

Dimara

Hermawati

, et al. Supporting domain characterization in visualization design studies with the critical decision method. In: Proceedings of the IEEE workshop on visualization guidelines in research, design, and education (VisGuides’22). New York, NY: IEEE.

43.

Kerracher

Kennedy

Constructing and evaluating visualisation task classifications: process and considerations. Comput Graph Forum 2017; 36(3): 47–59.

44.

Heer

Shneiderman

Interactive dynamics for visual analysis. Commun ACM 2012; 55(4): 45–54.

45.

von Landesberger

Fiebig

Bremm

, et al. Interaction taxonomy for tracking of user actions in visual analytics applications. In: Huang

(ed.) Handbook of Human Centric Visualization. New York: Springer, 2014, pp.653–670.

46.

Thomas

Cook

KA.

Illuminating the path: the Research and Development Agenda for Visual Analytics, chapter 2. New York: IEEE Computer Society, 2005. pp.33–68.

47.

Mahatody

Sagar

Kolski

State of the art on the cognitive walkthrough method, its variants and evolutions. Int J Hum Comput Interact 2010; 26(8): 741–785.

48.

Tory

Möller

Evaluating visualizations: do expert reviews work?

IEEE Comput Graph Appl 2005; 25(5): 8–11.

49.

Larson

Gray

Project Management: the Managerial Process. 8th ed. New York: McGraw Hill, 2020.

50.

Ellis

. (ed.) Cognitive biases in visualizations. Cham: Springer, 2018.

51.

Belton

Stewart

Multiple criteria decision analysis: an integrated approach. New York: Springer, 2002.

52.

Andres

TH.

Sampling methods and sensitivity analysis for large parameter sets. J Stat Comput Simul 1997; 57(1–4): 77–110.

53.

Daszykowski

Walczak

Massart

DL.

Representative subset selection. Anal Chim Acta 2002; 468(1): 91–103.

54.

Tversky

Elimination by aspects: a theory of choice. Psychol Rev 1972; 79(4): 281–299.

55.

Parks

Cowlin

Group discussion as affected by number of alternatives and by a time limit. Organ Behav Hum Decis Process 1995; 62(3): 267–275.

56.

Huber

Köcher

Vogel

, et al. Dazing diversity: investigating the determinants and consequences of decision paralysis. Psychol Market 2012; 29(6): 467–478.

57.

Gani

Limam

A kernel distance-based representative subset selection method. J Stat Comput Simul 2016; 86(1): 135–148.

58.

Chen

Yun

Wen

, et al. Representative subset selection and outlier detection via isolation forest. Anal Methods 2016; 8(39): 7225–7231.

59.

Japkowicz

Shah

Evaluating learning algorithms: a classification perspective. Cambridge: Cambridge University Press, 2011.

60.

Zaki

Meira

Jr.

Data mining and analysis: fundamental concepts and algorithms. 2nd ed. Cambridge: Cambridge University Press, 2020.

61.

Behrisch

Blumenschein

Kim

, et al. Quality metrics for information visualization. Comput Graph Forum 2018; 37(3): 625–662.

62.

Angelini

Santucci

Schumann

, et al. A review and characterization of progressive visual analytics. Informatics 2018; 5(3): 31.

63.

Hutter

Hoos

, et al. Algorithm runtime prediction: Methods & evaluation. Artif Intell 2014; 206: 79–111.

64.

Bolte

Bruckner

Measures in visualization space. In: Chen

Hauser

Rheingans

, et al. (eds) Foundations of Data Visualization. Cham, Switzerland: Springer, 2020, pp.39–59.

65.

Bouyssou

Outranking methods. In: Floudas

Pardalos

(eds) Encyclopedia of Optimization. New York: Springer, 2009, pp.2887–2893, Vol. 4.

66.

Shin

Ravindran

Interactive multiple objective optimization: Survey I—continuous case. Comput Oper Res 1991; 18(1): 97–114.

67.

Boose

Shema

Bradshaw

JM.

Recent progress in aquinas: a knowledge acquisition workbench. Knowl Acquis 1989; 1(2): 185–214.

68.

Bonneau

Hege

Johnson

, et al. Overview and state-of-the-art of uncertainty visualization. In: Hansen

Chen

Johnson

(eds) Scientific Visualization: Uncertainty, Multifield, biomedical, and Scalable Visualization. London: Springer, 2014, pp.3–27.

69.

Hägele

Schulz

Beschle

, et al. Uncertainty visualization: Fundamentals and recent developments. IT Inf Technol 2022; 64(4-5): 121–132.

70.

Winterfeldt

Fischer

. Multi-attribute utility theory: models and assessment procedures. In: Utility, Probability, and Human Decision Making: Selected Proceedings of an Interdisciplinary Research Conference, pp.47–85. D. Reidel Publishing. DOI: 10.1007/978-94-010-1834-0_3.

71.

Sałabun

Wątróbski

Shekhovtsov

Are MCDA methods benchmarkable? A comparative study of TOPSIS, VIKOR, COPRAS, and PROMETHEE II methods. Symmetry 2020; 12(9): 1549.

72.

May

Bannach

Davey

, et al. Guiding feature subset selection with an interactive visualization. In: Proceedings of the IEEE conference on visual analytics science and technology (VAST’11), pp. 111–120. New York: IEEE. DOI: 10.1109/VAST.2011.6102448.

73.

Mackinlay

Hanrahan

Stolte

Show me: Automatic presentation for visual analysis. IEEE Trans Vis Comput Graph 2007; 13(6): 1137–1144.

74.

Brown

Ottley

Zhao

, et al. Finding Waldo: Learning about users from their interactions. IEEE Trans Vis Comput Graph 2014; 20(12): 1663–1672.

75.

Siskos

Grigoroudis

Matsatsinis

NF.

UTA methods. In: Figueira

Greco

Ehrogott

(eds) Multiple Criteria Decision Analysis: State of the Art Surveys. Dordrecht: Springer, 2005, pp.297–334.

76.

Riabacke

Danielson

Ekenberg

State-of-the-art prescriptive criteria weight elicitation. Adv Decis Sci 2012; 2012: 276584.

77.

Han

Schulz

. Exploring vibrotactile cues for interactive guidance in data visualization. In: Proceedings of the 13th international symposium on visual information communication and interaction (VINCI’20), pp.1–10. New York: ACM. DOI: 10.1145/3430036.3430042.

78.

von Landesberger

. Insights by visual comparison: the state and challenges. IEEE Comput Graph Appl 2018; 38(3): 140–148.

79.

Roberts

. On encouraging multiple views for visualization. In: Proceedings the conference on information visualisation (IV’98), pp.8–14. New York: IEEE. DOI: 10.1109/IV.1998.694193.

80.

Roberts

. State of the art: Coordinated & multiple views in exploratory visualization. In: Proceedings of the 5th international conference on coordinated and multiple views in exploratory visualization (CMV’07), pp.61–71. New York: IEEE. DOI: 10.1109/CMV.2007.20.

81.

Javed

Elmqvist

. Exploring the design space of composite visualization. In: Proceedings of the IEEE pacific visualization symposium (PacificVis’12), pp.1–8. New York: IEEE. DOI: 10.1109/PacificVis.2012.6183556.

82.

Gleicher

Albers

Walker

, et al. Visual comparison for information visualization. Inf Vis 2011; 10(4): 289–309.

83.

Ritter

Berendt

Fischer

, et al. Virtual 3D jigsaw puzzles: studying the effect of exploring spatial relations with implicit guidance. In: Herczeg

Prinz

Oberquelle

(eds) Mensch & Computer 2002: Vom interaktiven Werkzeug zu kooperativen Arbeits- und Lernwelten. Wiesbaden: Vieweg+Teubner Verlag, 2002, pp.363–372.

84.

Willett

Heer

Agrawala

Scented widgets: improving navigation cues with embedded visualizations. IEEE Trans Vis Comput Graph 2007; 13(6): 1129–1136.

85.

Liu

Kale

Althoff

, et al. Boba: Authoring and visualizing multiverse analyses. IEEE Trans Vis Comput Graph 2021; 27(2): 1753–1763.

86.

Waser

Fuchs

Ribičič

, et al. World lines. IEEE Trans Vis Comput Graph 2010; 16(6): 1458–1467.

87.

Wood

Badawood

Dykes

, et al. BallotMaps: Detecting name bias in alphabetically ordered ballot papers. IEEE Trans Vis Comput Graph 2011; 17(12): 2384–2391.

88.

Chi

Barry

Riedl

, et al. A spreadsheet approach to information visualization. In: Proceedings of the IEEE Symposium on Information Visualization (InfoVis’97), pp.17–24. New York: IEEE. DOI: 10.1109/INFVIS.1997.636761.

89.

Marks

Andalman

Beardsley

, et al. Design galleries: A general approach to setting parameters for computer graphics and animation. In: Proceedings of the 24th annual conference on computer graphics and interactive techniques (SIGGRAPH’97), pp.389–400. ACM/Addison-Wesley. DOI: 10.1145/258734.258887.

90.

Wang

Hazarika

, et al. Visualization and visual analysis of ensemble data: a survey. IEEE Trans Vis Comput Graph 2019; 25(9): 2853–2872.

91.

Polikar

Ensemble based systems in decision making. IEEE Circuits Syst Mag 2006; 6(3): 21–45.

92.

McKenna

Mazur

Agutter

, et al. Design activity framework for visualization design. IEEE Trans Vis Comput Graph 2014; 20(12): 2191–2200.

93.

McKenna

Lex

Meyer

Worksheets for guiding novices through the visualization design process. arXiv preprint 2017; 1709.05723. DOI: 10.48550/arXiv.1709.05723.

94.

Byrd

Dwenger

Activity worksheets for teaching and learning data visualization. IEEE Comput Graph Appl 2021; 41(6): 25–36.

95.

Roberts

Headleand

Ritsos

PD.

Five design-sheets: creative design and sketching for computing and visualisation. Cham: Springer, 2017.

96.

scikit learn. Clustering, 2022. https://scikit-learn.org/stable/modules/clustering.html (accessed 30 September 2022).

97.

Kou

Peng

Wang

Evaluation of clustering algorithms for financial risk analysis using MCDM methods. Inf Sci 2014; 275: 1–12.

98.

Pérez-Messina

Ceneda

El-Assady

, et al. A typology of guidance tasks in Mixed-Initiative Visual Analytics Environments. Comput Graph Forum 2022; 41(3): 465–476.

99.

Sperrle

Ceneda

El-Assady

. Lotse: A practical framework for guidance in visual analytics. IEEE Trans Vis Comput Graph 2023; 29(1): 1124–1134.

100.

Sperrle

Jeitler

Bernard

, et al. Co-adaptive visual data analysis and guidance processes. Comput Graph 2021; 100: 93–105.

101.

Sperrle

Schäfer

Keim

, et al. Learning contextualized user preferences for Co-Adaptive Guidance in Mixed-Initiative topic model refinement. Comput Graph Forum 2021; 40(3): 215–226.

102.

Hall

Perin

Kusalik

, et al. Formalizing emphasis in information visualization. Comput Graph Forum 2016; 35(3): 717–737.

103.

Kang

Plaisant

Shneiderman

. New approaches to help users get started with visual interfaces: Multi-layered interfaces and integrated initial guidance. In: Proceedings of the annual national conference on digital government research, pp. 1–6. Digital Government Society of North America.

104.

Radoš

Splechtna

Matković

, et al. Towards quantitative visual analytics with structured brushing and linked statistics. Comput Graph Forum 2016; 35(3): 251–260.

105.

Munzner

A nested model for visualization design and validation. IEEE Trans Vis Comput Graph 2009; 15(6): 921–928.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.25 MB

0.30 MB