Interact: A visual what-if analysis tool for virtual product design

Abstract

Virtual prototyping is increasingly used by businesses to streamline operations, cut costs, and enhance daily operations. This often includes a variety of modeling techniques among which, complex, black-box models. The path from model development to utilization in applied contexts is yet long. Domain experts need to be convinced of the validity of the models and to trust their predictions. To be used in the field, model capabilities need to be affordable, that is, allow rapid and interactive scenario building, even for non-experts. Complex relations governed by statistical interactions must be unveiled for users to understand unexpected predictions. We propose Interact, a model-agnostic, visual what-if tool for regression problems, supporting (1) the visualization of statistical interactions between features, (2) the creation of interactive what-if scenarios using predictive models, (3) the evaluation of model quality and building trust, and (4) the externalization of knowledge through model explainability. While the approach applies in various industrial contexts, we validate the application purpose and design with a detailed case study and a qualitative user study with engineers in the tire industry. By unraveling statistical interactions between features, the INTERACT tool proves to be useful to increase the transparency of black-box machine learning models. We also reflect on lessons learned concerning the development of visual what-if tools for virtual product development and beyond.

Keywords

What-if analysis statistical interactions design study visualization

Introduction

Modern organizations seek to exploit the data they collect from their environment and activities, often aiming to consolidate, enrich or improve internal decision-making processes. Modeling techniques can be used to improve these often suboptimal processes, and thus become a companion technique guiding decision-making. Recently, we have seen many examples of the use of Machine Learning (ML) techniques, in various application contexts, to support different processes or tasks. In this article, we focus on the case where models support product design. Designing a product is a decision-making activity, consisting in setting a number of parameters, which will in the end determine the characteristics of the designed product. The decision can sometimes be difficult because desirable characteristics can contradict one another. Flexibility or ease of use, for example, is often the opposite of robustness or durability of a product. This antagonism is reflected at the parameter level: increasing the value of a parameter may affect another one, such as by limiting its range. Such effects may be hard to capture and describe, since parameter interdependencies are widespread across the design space. Human expertise is thus mandatory to assess the physical reality behind these effects, and to help to identify parameter settings that correspond to interesting and feasible solutions. In this article, we focus on the need to exploit many inputs governing a single-output model, therefore deliberately not overlapping with areas such as multi-objective optimization.

Much research has looked at combining state-of-the-art Machine Learning (ML) techniques with visualization tools in various application fields, including healthcare,^1,2 meteorology,^3,4 and social sciences.⁵ Some of this work falls under the umbrella of Explainable Artificial Intelligence (XAI) research,⁶ insofar that visualization can help to audit and to build trust in the predictions given by AI models. Techniques relying on model-specific and model-agnostic approaches were developed to allow explainability and interpretability.⁷ Most efforts in XAI, however, have focused on helping a user to understand a model. Aspects involving practical, but complex, decision-making problems in industrial contexts and, more specifically, in product design are less covered. In this context, application domain experts need to probe models to assess that they faithfully embrace the physical characteristics of the objects being designed. Additionally, they need help to comprehend why some input parameter combinations lead to unexpected outcomes and to recognize the underlying characteristic relationships.

Our approach is inspired by the industrial context of tire design. “Tires are highly engineered structural composites whose performance can be designed to meet the vehicle manufacturers’ ride, handling, and traction criteria, plus the quality and performance expectations of the customer.”⁸

Incidentally, the required characteristics of a tire for a sporty car differ very much from those required from a tire intended for a sedan or electric car. Sporty tires will have a high focus on handling⁹ while the others will focus more on acoustics¹⁰ and fuel efficiency.¹¹ However, from a physics perspective, these disciplines are quite diverse and might be considered as entirely independent sciences,^12,13 necessitating various modeling methodologies. This variety, in our opinion, may be encountered in other sectors and is not limited to the design of tires. In a broader context, our approach is relevant whenever a model needs to reflect the physical constraints inherent in a product, whether it is the soles of sports shoes or the casting of steel, for example, requiring experts to be able to assess the model’s ability to reflect physical reality.

As an example of an engineering activity in the tire design domain, an acoustics expert needs to understand how a specific tire interacts with the road surface and with the car, and how noise is propagated through air¹⁴ and car structures¹⁵ to the passenger. She needs to understand which design features can be tuned so that the final acoustic comfort inside the car is reached. This task is complex as these features interact, in the statistical sense¹⁶ as we shall explain. Put simply, the influence of a design feature on the outcome depends on one or multiple other features. For instance, a known mechanical relation such as an increase of noise as mass decreases¹⁷ might be invalidated when tire parameters related to stiffness are changed simultaneously. Such relations are at the border of recognized mechanical behaviors and are hard to process, especially in industrial contexts, in time-sensitive conditions. As building and testing physical products is often costly, the engineers need to be supported in virtual assessments.

The final design decision engineers will take often has high financial, production, testing and planning consequences. This is when state-of-the-art methods can help the specialist to de-risk this virtual assessment process.

The contributions of this work are: 1 –Interact, a what–if scenario system providing a novel combination of ML-based regression models and statistical interaction analysis. The system allows the user to: (a) Create and compare what-if scenarios together with a measure of confidence to support forecasting, (b) Understand model predictions by making statistical interactions more affordable; 2 –a case study and a qualitative user study with participants from the tire industry. 3 –a report of lessons learned concerning desirable characteristics of what-if systems.

Related work

Interact relates to various areas of research, including visualization and explainable machine learning, statistical interactions, what-if tools and decision making.

Visualization for model understanding

Recently, much work has focused on Explainable Artificial Intelligence (XAI).^18–22 Chatzimparmpas et al.⁷ organize prior work on visualization for interpreting ML models in six categories: Visual Analytics Pipelines, General ML models, Predictive Visual Analytics (PVA), Interactive ML (IML), Deep Learning and Dimensionality Reduction. Work in the PVA and IML categories is closer to Interact. In another extensive literature review, Chatzimparmpas et al.²³ stress the importance of trust in ML and introduce five levels of trust directed to various aspects of IML.

Adadi and Berrada²⁴ distinguish model-agnostic and model-specific XAI methods. Model-specific approaches are usually bound to the specific type of model used to fit the data. Examples of model-specific approaches are iForest²⁵ for Random Forests models, GAN Lab²⁶ for Generative Adversarial Networks or DQNViztool²⁷ for Deep Q Networks. Interested readers can refer to additional papers on the visualization of neural networks.^1,28–30 Model-specific solutions can be discounted with the advent of a new, more effective type of ML model.

In Interact, we take a model-agnostic approach for regression problems. Model-agnostic approaches are independent of the type of model used to fit the data, which makes them more likely to stand the test of time. As discussed by Spinner et al.,³¹ model-agnostic techniques can be distinguished according to their explanation coverage. An explanation is local when it is valid for a certain data sample, or global when it holds for the entire data set.

LIME³² uses local surrogate models to explain predictions, for example, by using words, or parts of an image, that lead to a certain model decision. SHAP³³ provides both local and global explanations. Its force plots depict which features contribute to a single model output, whether positively or negatively, and the global explanations are achieved by rotating these plots and concatenating them horizontally on a complete dataset. A feature overview helps to identify the most important features of the observed model. RuleMatrix³⁴ uses rule induction to provide explanations to non-ML practitioners who need to understand and use ML models. Going beyond the explanation of one model, the comparative analysis of multiple models has been the focus of Manifold³⁵ and Clustervision.³⁶ In Interact, we go beyond model-agnostic explainability by supporting the creation of what-if scenarios on the fly, together with an estimation of confidence for each prediction.

Visualization of statistical interactions

Statistical interactions, or interaction effects, are widely studied in statistics,^37–40 but rarely in ML-enabled visualization tools. They capture a non-additive influence of two variables to a model outcome. Typically, while variable X may vary monotonously to the dependent variable Y, the rate at which it varies may depend on a second variable Z. The H-statistic is a measure of interaction effects over the data distribution, proportional to the power of the interaction, but it may yield spurious interactions.⁴¹ Partial Dependence (PD) models are also used to evaluate interaction effects,⁴² without raising spurious interactions. SHAP³³ evaluates interactions using SHAP values. Save for a few exceptions,⁴³ the H-statistic and the PD-based method were not often used to uncover interaction effects in ML models. Interaction effects are visualized using line charts, such as Individual Conditional Expectation (ICE) lines⁴⁴ and SHAP plots.³³ Individual pairwise interaction effects are also visualized using 2D contour maps^43,45 2D surface curves in a 3D-coordinate system.^41,45 The VINE⁴³ tool uses clustering in ICE/PDP line charts to visualize interaction effects in black-box models. Variable Interaction Networks (VIN) use small node-link network diagrams⁴⁶ to provide an overview of a dozen of pairwise interaction effects. Unlike most tools, Interact provides an interactive and scalable visual overview of all pairwise interaction effects and their relative strength.

What-if tools

Amer et al.⁴⁷ and Harries⁴⁸ describe the scenario use as “the creation of the description of alternative future realities.” Golfarelli et al.⁴⁹ defines a scenario as a simulation of complex systems under a given hypothesis. What-if analyses are deemed useful in many domains, for example, healthcare,^50–52 social sciences,^53–55 military,^56,57 and were used heavily in the last couple of decades.^58,59 The most relevant to our work, visual what-if tools are Prospector,⁶⁰The What-If Tool,⁶¹ and CoFFi .⁶² All systems use partial dependence (PD) concepts⁶³ to support overall and/or localized model inspection. While Interact proposes PD lines, we extend the explainability part with Individual Conditional Expectation (ICE) lines and Accumulated Local Effects (ALE) lines.⁶³Interact differs also by a main focus on a fundamental new concept for what-if tools, which is the identification and support for understanding statistical interactions. Unlike previously cited tools, this aspect is required and useful in the context of virtual product design. Unlike other tools, Interact supports the comparative analysis of multiple alternatives (Figure 6).

Decision making

According to Edwards,⁶⁴ the scope of the Decision Making (DM) theory is to predict, given two states A and B, which state will be chosen by an individual. Slovic et al. studied DM with respect to the decision environment, the different theories of DM and information processing in DM.⁶⁵ Other fundamental work concerns DM and cognition.^66–69 Milkman et al.⁷⁰ stress that in the DM context, errors are costly and will get even costlier, hence the need to improve DM processes. It is believed that AI can improve human analytical skills, DM abilities and creativity.⁷¹ Along these lines, Bastani et al.⁷² devised an ML algorithm that helps human users in DM tasks by using tips. Khosravi et al.⁷³ focus on combining DM and ML for flood susceptibility modeling. Healthcare is also a key area for improving DM with AI.^74–77 Duan et al.⁷⁸ give an overview of AI for DM and propose 12 directions for future research in this area. Some of these directions are further enhanced by Dwivedi et al.⁷⁹ In both papers, the AI work is divided into supporting, augmenting, replacing or automating human tasks. Interact is designed to support and augment engineers’ analytical abilities in their DM processes.

Tasks, data, and users

Our research is based on frequent exchanges with domain experts, over a period of 2 years, leading us to address the formalization of user tasks as a primary step of our design. Also, two of the authors are domain experts (8 and 25 years in the field) and act as a “liaison,” a role highlighted by Simon et al.,⁸⁰ for a better definition of the tasks. The overarching goal of our target users is to design a product, that is, a tire that meets a specific requirement, for example a noise target. To do so, they need to factually decide on multiple characteristics governed by complex relationships.

Following Munzner’s visualization design methodology,⁸¹ we provide below a characterization of data and tasks.

Data and users

Our work was conducted in collaboration with domain experts in the tire industry. Throughout the paper, we will illustrate our concept with the application of Interact to support acoustics engineers in the tire industry. They mainly ensure that the tire design meets all requirements in terms of acoustic comfort. A typical example of an acoustic engineer’s work is to analyze why an existing tire does not meet the acoustic requirements and give tire design recommendations to reach the customer’s acoustic target.⁸² Among many possible recommendations, the engineer might propose to change the materials for a set of tire components, for example, the material of the tread band, or to optimize the tread pattern design. Only two people among our domain experts have created ML models in the past. For example, the predictive model used in the case study (see page 8) was created by a senior engineer from the acoustics team.

The data pipeline at hand starts from audio recordings from various noise acquisition systems. These files are processed by the engineers to extract specific acoustic metrics quantifying acoustic comfort. These indicators are stored in relational databases along with tire design features and will be the focus of our study. The data used in our work is therefore tabular. The models are essentially serialized Python objects, but Interact can broadly query any regression model callable from a command prompt.

Tasks

In this section, we present the Domain Specific Tasks (DST) supported by Interact. The tasks are inspired by existing task categorizations concerning predictive visual analytics,⁸³ trust in ML,²³ visualization for ML.⁸⁴ We retained four high-level tasks from the literature and propose a new one related to the inspection of statistical interactions (DST2).

DST1: Visualize the most important features

While senior engineers might know which features are the most important and which “knobs” to turn first, a novice engineer needs support in the more subtle parts of the design process.

DST2: Detect and analyze statistical interactions

Statistical interactions complexify the iterative process of reaching a target. Even senior engineers need assistance when interactions arise. The support for identifying and understanding how features interact in a statistical sense (see definitions on page 2) is a required functionality.

DST3: Assess trust in predictions and model as a whole

Firstly, the domain experts need to know how good the underlying model is. General goodness-of-fit statistics must be available. Secondly, the users need to know if an input they use for model probing is in the training data. If not, there is less evidence supporting the prediction. Uncertainty presentation and cognitive load are also factors that highly influence trust and which need to be considered when designing the system.⁸⁵

DST4: Create what-if scenarios

In the tire development context, building a physical tire is very costly and time consuming. A virtual tire development process allows many operations to be done in silico. Interact needs to support the creation of multiple what-if scenarios to reduce prototype manufacturing and testing cost and duration. For our domain experts and field, this includes the need to compare two or more virtual tires and make a factual decision on this basis.

DST5: Externalize knowledge through model explainability

A complex model can capture interesting and unknown forward paths that the engineer might explore to reach her targets. Revealing the potential effect of certain actions on reaching the target, like changing a specific tire design element, is also of high interest.

System description

In this section, we describe the coordinated multiple views of the Interact application (Figure 1). In line with the third level of Munzner’s nested model,⁸¹ we motivate the proposed visual encodings and interactions supporting the tasks listed above, and discuss alternative visual designs. We also highlight the current practices of the target population of tire engineers. A demo of the application with non-confidential data is available at: https://viana.list.lu/interact.

Figure 1.

The graphical user interface of Interact showing the visualization for the first vertical resonance mode frequency⁸⁶ model. (a) Data input panel: A set of scented widgets⁸⁷ for user control and display of the values of the most important model features. (b) Actual vs. predicted plot: a scatterplot augmented with a confidence (red) box and an histogram of the actuals (plotted on the y-axis) for a given prediction. The legend shows the most important model features. (c) Feature interaction matrix: Second degree feature interactions based on either the H-statistic or the Partial Dependence (PD) score. (d) Individual Conditional Expectation (ICE) plots: a set of linecharts showing the individual contribution of each feature. The yellow to red curves show the interaction between the selected pair of features from (c).

Input space visualization

In DST4, the application domain expert needs to probe the model with different candidate designs, by tuning any available input parameter. We offer scented widgets: sliders, dropdown menus, toggles, and calendar widgets to set numerical, categorical, binary and date values respectively (Figure 2). Scenting⁸⁷ the sliders was crucial, as data is mostly numerical in our use cases. Often, tire engineers run exploratory data analyses in commercial software like Minitab⁸⁸ and SAS JMP.⁸⁹ They usually plot data distributions as bar charts, a popular choice in many areas.⁹⁰ Thus, Interact uses scented sliders showing the data distribution of the training data as a bar chart. Instead of using an arbitrary number of equally spaced bins, Interact uses a Numpy function to compute the best binning for each feature. We tested density plots too. The interpolated segments of the line charts created spurious rise and fall patterns, when a feature had gaps in its domain.

Figure 2.

Control pane. (a) Two tabs to show either the inputs panel or the comparative analysis panel. The similarity score of the current set of inputs compared to training data is shown in the top-right corner. (b) Scented widgets⁸⁷ for various data types, sliders, dropdown menus, calendar widgets and toggles. (c) Widgets to reset, save combination for comparative analysis or add features.

Toward DST1, the sliders are sorted top-down by feature importance in the ML model. The top features are indeed the ones that affect the predicted outcome the most; varying their values may help in appraising the model. Interact supports any regression ML model as long as feature importance values are provided.

For DST5, the sliders carry a second scent as a (green) line chart on top of the bar chart showing the expected prediction trend. At the user’s discretion, the trend is computed using either a partial dependence plot (PDP) or an accumulated local effect (ALE) line.⁶³ These line charts share the same scale across all sliders to ease the comparison of the impact of features on the prediction. For boolean features, the toggle is scented with feedforward information⁹¹: a red or blue triangle shows the expected direction of change of the prediction, should the toggle be flipped.

In the top right corner of Figure 2, we display a score between 0 and 1 measuring the similarity of the current set of inputs to the training data based on a kernel method.⁹² The lower the score, the more cautious should the expert be regarding the prediction.

When the user finds a combination of input parameters yielding an interesting outcome, she may add it to the saved combinations for future reuse, for example comparative analysis. To support comparative analyses, the “saved combinations” tab (Figure 6) is laid out according to an augmented juxtaposition strategy. Techniques part of superposition or explicit encodings’ families were excluded due to their known drawbacks in terms of visual clutter and decontextualization.⁹³ The user must be able to easily identify an alternative (virtual tire/product), and the latter needs to use the same units and the same scales across all dimensions as the original product (for instance, a control tire).

Another argument in favor of juxtaposition is the common practice and familiarity of our users with such designs within the organization. We reached our final design (Figure 6) after two iterations, while optimizing the layout and the scents. We use the percentage of change of an input as a scent, depicted by a triangle every 25%. The direction of change is encoded by color and by the direction of the triangles. The comparative panel can be set to only show the modified inputs across all alternatives. Knowledge externalization (DST5) can occur through questions like: “Which alternative yields the best prediction?,” or “How do the inputs differ across the saved combinations?”

Prediction visualization

In DST3, the goal is to build trust in individual predictions and in the whole model. The trust we want to build with our system, in DST3, corresponds to the first, fourth and the fifth levels of trust of Chatzimparmpas et al.,²³ concerning the raw data, the concrete model and the evaluation/user expectation.

In our context, the domain experts need to appraise the model for their decision-making process. Statistical software like Minitab⁸⁸ and SAS JMP⁸⁹ use common metrics like RMSE, R2, and MAE, and actual versus predicted scatterplots to assess goodness of fit. In terms of alternate visual design, we wondered whether to plot predictions or actuals on the Y-axis of the scatterplot. It turns out that plotting the actuals on the Y-axis leads to a more correct model evaluation.⁹⁴ Hence, we use this design in Interact as in Figure 3, besides the popularity and suitability of scatterplots for regression models. In Interact, the prediction is displayed in textual form above the scatterplot and positioned as a red cursor (vertical line) on the plot. The prediction results from the current combination of inputs in Figure 2. Below the prediction, we display standard goodness-of-fit metrics, which provides the users with a way to gauge their confidence in the model. The legend reminds the top 10 most important features.

Figure 3.

Prediction panel matching Figure 2. (a) Prediction for current slider combination. (b) Actual versus predicted plot with linked cursor to current prediction. (c) Top 10 most important features of the model.

To trust a given prediction, the expert needs to situate it in the point cloud. When it falls in a dense area of the cloud near the identity line⁹⁵ the prediction is fairly reliable. If not, some caution may be needed. This calls for a confidence interval around the prediction.⁹⁶ To help the user assess such a confidence interval, Interact displays a red box enclosing 95% (box height) of the actual data around the prediction in the actual versus predicted plot (Figure 3). A tooltip shows the lower and upper bounds of the box. In addition, a histogram alongside the Y-axis (in red) shows the distribution of the actual data (Figure 3). This provides even more context to assess the trustworthiness and reliability of the prediction.

Overview visualization of feature interactions

In DST2, domain experts are wary of statistical interactions between model features. The response of the model to an input feature often depends on the value of another feature. Ignoring feature interactions would instigate a tedious trial and error strategy to reach a target, without understanding how features interact. Current practice consists in modeling with standard least squares fits⁹⁷ in JMP or Minitab. The user must specify manually which interactions, if any, to include in the model fit.⁹⁸ In contrast, Interact computes automatically all pairwise interactions using either the H-statistic⁴¹ or the PD-based metric.⁴²

Since pairwise feature interactions may be modeled as a weighted complete graph, possible visual designs include node-link diagrams⁴⁶ and matrix visualizations. By applying a threshold on edge weights, we generated node-link diagrams, which were yet too cluttered. Finally, we preferred matrix visualizations, as they are better suited for link lookup in dense graphs.⁹⁹ For example, Figure 4 shows the first four rows of the interaction matrix for a modeled dataset of the first vertical mode of a tire.⁸⁶ The darker the color, the stronger the interaction between the pair of features. The user can quickly spot the highest interactions and get details on demand by hovering over a cell. This visual design scales up to dozens of features on a regular desktop screen.

Figure 4.

First four rows of the Interaction matrix. The yellow cursor points to the strongest interaction. The name of the features and the interaction score are displayed.

Individual feature contribution visualization

In DST1 and DST5, domain experts need to understand the sensitivity of the model to each input feature. They often use software to generate X versus Y scatterplots to fit linear regression models between the predicted outcome and input features, along with statistical significance tests.¹⁰⁰ Even with a few features, the analysis of the resulting matrix of scatter plots is challenging.¹⁰¹ Another practical limitation results from the fact that the observed data is usually much sparser than all possible combinations of a feature set, which makes it difficult to clearly isolate the individual contribution of a feature of interest. An alternate method that allows the user to rapidly inspect the relation between a feature of interest and a target is the partial dependence (PD) plot (Figure 5). The equation corresponding to the first order fit of the PD line and the Pearson’s correlation coefficient¹⁰² are shown above the chart. In the presence of correlated features, the Accumulated Local Effects (ALE) can be used instead.⁶³Interact provides both PD and ALE line charts to give an overview of the full data set.

Figure 5.

Individual feature contribution panel. ICE lines in blue. PDP in green. A click by the green cursor in Figure 4 populates the plot with second order interaction visualizations – yellow/red lines.

Beyond global methods, finer phenomena can be inspected using the ICE lines (blue lines in Figure 5). For each point in the dataset, a line is created by varying the independent feature stepwise and fixing the values of all other features.

Figure 1(d) shows two ICE plots with the Partial Dependence (PD) line in bold green. The left plot presents a general descending trend (PD) for TIRE_WIDTH (green PD line). The ICE lines (in blue) tend to agree with this trend. This behavior is expected by experts and agrees with recent literature.¹⁰³ The wider the tire, the heavier the tread and the lower the first vertical resonance frequency. The vertical spread of the ICE lines also gives an idea of the range in which the prediction may lie with respect to the feature on the X-axis. Tightly bundled ICE lines point to the lack of interacting features and to a low influence of other features on the prediction. As ICE lines spread out and form crossing patterns, feature interactions are more likely.⁶³

User interaction

Interact is a coordinated multiple view system, designed to support what-if analyses built on top of a predictive ML model. More than a mere collage of the individual views described so far, the linked views support a cohesive analytic workflow driven by user interactions.

What-if query specification

Starting from Figure 1(a), the user can express a what-if query by pulling a few sliders to set the values of selected input features. The similarity score in the top right corner of the data input panel (Figure 1(a)) shows how close the new input configuration is to the training data. The underlying ML model is queried for a prediction. Figure 1(b) is updated to reflect the new prediction textually at the top and graphically, by adjusting the position of the red cursor, the related confidence box and histogram. The values of the input features are also cascaded to any related charts, for example, the red cursor in each ICE line chart in Figure 1(d). This helps the user to locate the prediction in the feature space. While only the top most important features are visible by default in Figure 1(a), the user can interactively display more features (Figure 2(c)). The interaction heatmap in Figure 1(c) is expanded accordingly to include additional rows and columns. Likewise, the related ICE line charts are added (Figure 1(d)). Since what-if analyses are often based on trial and error, Interact provides a save and reload functionality. The final decision making usually considers multiple saved candidates. We offer a simple yet effective visualization to compare the chosen alternatives.

Figure 6 shows three candidate tire designs, yielding the same prediction value (76.2). The user can decide which alternative suits best her needs, based on other, unmodeled criteria such as cost or manufacturing constraints.

Figure 6.

Three combinations showing the same prediction level (above red line) but with different solutions. The reference is the active combination (first column); the scents (triangles) and equality (>>) are with respect to the active combination.

Drilling down into statistical interactions

Continuing the analysis in Figure 1(c), the user’s attention is drawn to a few dark cells in the heatmap corresponding to pairs of strongly interacting features. By clicking on any cell, the individual feature contribution visualization in Figure 1(d) is overlaid with a set of new lines, colored using a yellow-to-red gradient. While the X-axis of the line chart corresponds to the selected row in the heatmap, the color encodes the level of the second feature, that is, the column in the heatmap. We chose to display feature names on demand only when more than five features are shown in the heatmap, to avoid label clutter. We also do so because feature names tend to be very long in our context of use, and truncating them is not helpful. The color scale is shown on the right side of the plot. These lines correspond to the second order partial dependence lines according to a predefined grid, for example, a decile-based grid. The user may hence inspect the trends of these lines and identify the regions where statistical interactions occur.

Technical implementation

Interact is implemented following a state-of-the-art Web architecture with a FastAPI server at the backend, a JavaScript middleware using Node.js, and a JavaScript frontend using React, Redux and D3. To ensure high availability of the application, the complete architecture is containerized using Docker¹⁰⁴ and orchestrated using Kubernetes and Terraform in a cloud environment.

Evaluation

Our evaluation methodology was inspired from existing efforts in categorizing and guiding evaluation in visualization research.^81,105–111 Referring to Lam et al.,¹⁰⁵ we evaluated Interact from two perspectives, namely Visual Data Analysis and Reasoning (VDAR) and User Experience (UE). We conducted a qualitative user study with a group of acoustic engineers in the tire industry (see User study) and a case study with two domain experts (see Case study). The evaluation procedure and hypotheses were preregistered at https://osf.io/-clickable. We also emphasize that user study participants and experts involved in the case study do not include any of this paper’s authors.

User study

Methodology

We ran a first pilot study with 11 engineers, aiming to assess the usability of interact. The first pilot study was also registered at https://osf.io/-clickable.

This study led to a small-scale deployment of Interact within the Goodyear organization, and many suggestions for improvement.

Six months later, we ran a second pilot study, preregistered at https://osf.io-clickable, to evaluate the improved software in a real use context, that is, with tire engineers and tire related data. We started with a training session aiming to introduce the main functionalities of Interact, train on solving tasks, and get familiar with the tool through concrete examples. Next, we organized the user study. User data and feedback were collected through: (1) a survey of demographic and occupational data, (2) a semi-structured focus group, and (3) an anonymous exit questionnaire. The training and user study took 2 h in total. The results of this second pilot study are discussed below.

Participants

The training was organized with 10 engineers, all of whom were domain experts in tire acoustics from the Goodyear organization. The median experience was 13 years with an inter-quartile range of 10 years. Two of them have previously created ML models and two more have participated in ML model validation. The remaining six have no experience with ML. All participants were co-located and used commodity 17″ laptops. More demographic information is provided in the Supplemental Material.

Focus group

After the training, the domain experts took part in a semi-structured focus group,¹¹² in the form of an open discussion around prepared questions in relation to our research objectives and domain-specific tasks. One of the co-authors facilitated the discussion while another one took notes. We also recorded participant voices, later transcribed and analyzed question by question. Each answer was split into one or multiple semantic units (one idea per unit text).¹¹³ Following a “peer debriefing” method,¹¹⁴ three independent coders labeled these units, and then met and agreed on the following set of tags: utility, UX, learnability, enjoyment, extensibility.

The utility tag was by far the most important tag in our coding effort. All comments were positive and highlighted how the features of the tool can support participants’ daily work. The tool was deemed an asset for discussing what-if scenarios with internal and external customers, for detecting and analyzing statistical interactions, for knowledge externalization and model explainability.

In what-if discussions, the tool can be used to answer customers on the spot rather than postponing the analysis. “It will help create a data-driven way of replying to customers.” Besides providing a prediction, “knowing the uncertainty is very valuable.” This will further allow to build trust in predictions (DST3). In this regard, the PD line visualization also was appreciated, with a participant saying: “Referring and following it would provide really good feedback in conversations with customers.” Regarding the what-if capabilities, the users liked the real-time model probing and its smoothness (“I like the speed of the tool”). This capability was even more interesting for younger engineers, who say they “don’t have the knowledge base to fall back on.” The visualization of statistical interactions was perceived as having a high added value as it allows going beyond guidelines. “More information on a specific variable can be extracted if a guideline rule is wrong.” Knowledge externalization and model explainability “helps from a fundamental understanding to better comprehend the sensitivities and which factors are most important.” Other comments on this topic include: “It provides a simple way to understand a model” or “Leveraging data that I never accessed expands my capabilities beyond my personal knowledge/experience.”

The UX tag encompassed mainly request for improvements of the user interface and requests for new functionalities. The latter often concerned open research questions, which we discuss thoroughly in the next section. Some bugs were also reported and fixed soon after.

In terms of learnability, the users asked for more contextual help and more training material. This was particularly the case with novice engineers who were somewhat new to statistical interactions. The concept is a bit more complex to grasp initially. Yet, all participants deemed Interact“intuitive after using it.”

The enjoyment tag was mainly assigned to comments regarding the application as a whole. The users liked the visualizations, the real-time model probing, but also the easy identification of the most important statistical interactions.

Extensibility in terms of the number of models that can be added to Interact was raised. Pending a formal assessment, the tool can offer many more models, owing to its cloud computing architecture and a caching mechanism.

Exit questionnaire

The post training questionnaire took the form of a 7-level Likert scale survey¹¹⁵ (see Supplemental Material):1-Strongly disagree; 2-Disagree; 3-Somewhat disagree; 4-Neutral; 5-Somewhat agree; 6-Agree; 7-Strongly agree. Its creation and analysis followed the best practices.^116–118 Where suitable, the question replaced (“Agree”) by (“Useful”). Each Likert scale was composed of several Likert items (sub-questions) that were closely related. We analyzed the results using the one-sample t-test¹¹⁹ and its non-parametric equivalent, the one-sample sign test.¹²⁰ This is due to conflicting opinions regarding whether Likert scales should be considered as ordinal or numerical.^116–118 We applied both strategies, and in each case, Interact obtained statistically significant positive outcomes ( $p < 0.05$ ) in support of our Domain Specific Tasks (see Tasks). The neutral point (mid-point of the Likert scale) served as the benchmark against which we calculated the p-value. Table 1 shows the results of the Likert scales only. More details about the individual Likert items are provided in Supplemental Material.

Table 1.

Summary of the anonymous exit questionnaire.

The columns comprise the Likert scale ratings: Strongly Disagree D–(1), Disagree D-(2), Somewhat Disagree D(3), Neutral N(4), Somewhat Agree A(5), Agree A+(6), Strongly Agree A++(7). Descriptive statistics are displayed in the following columns, including mean, median, standard deviation, interquartile range, one-sided t-test p-value (prob > t), sign test p-value, and the number of items answered for each corresponding Likert scale (sub-questions × number of participants). Heatmap annotations are expressed in percentage out of N. The descriptive statistics, such as the mean, are computed based on the rating values, that is, Strongly disagree = 1 and Strongly agree = 7.

Case study

This case study aims to observe and report on how Interact can be used by domain experts to address real business needs. We ran this study with two senior experts in tire acoustics, none of whom is an author of this paper. We used the First Vertical Mode Model, shown in Figure 1 as background data and model. We structure the following account into two subsections arising from the actions performed by the two experts. The problems they wanted to solve with Interact were akin to their daily tasks and are representative of two main challenges acoustic engineers face in their work: (1) problem solving; (2) knowledge externalization and exploratory analysis.

Problem solving

Car manufacturers, provide tire performance specifications including a list of metrics to be met during the tire design phase, for example, acoustics and mileage for a tire fitment on a specific vehicle. This is a common trait in all industries where users specify a set of desirable product characteristics. In this section, we will focus on a hypothetical yet realistic case where meeting the customer target requires, for instance, the tire’s first vertical oscillation mode (natural frequency) to be below 70 Hz. To the layperson, we highlight that all actions presented below relate to influencing tire mass or stiffness. Classical mechanics show that the natural frequency in a mass-spring set up can be expressed in the form $\sqrt{\frac{k}{m}}$ with k being the stiffness and m the mass.¹²¹ Increasing mass and decreasing the stiffness will lower the natural frequency. This is what our expert tried to achieve with Interact. To start with, the expert set the mandatory values provided by the customer for tire size, that is, tire width, aspect ratio and rim diameter, using the input sliders, hence obtaining a prediction of 77 Hz in Figure 1(b) (DST4). Next, he changed the rim width, a parameter which could be changed in a limited range. Looking at the PD line on top of the slider (fifth input feature in Figure 1(a)), reducing the value would also lower the first vertical mode frequency. Shrinking rim width from 7.5″ to 7.0″ led to a predicted 74.3 Hz, which is a small improvement. Likewise, the inflation pressure was reduced to 2.0 Bars to reach a mere 74.2 Hz. The next attempt consisted of adding a layer of sealant material, which happens to make the tire heavier¹²² (first toggle in Figure 1(a)). As shown by the red triangle next to the toggle, adding a sealant would get us closer to the target. This took us to 67.5 Hz, even better than the customer request. While this effect is known to decrease the first vertical mode frequency,¹⁰³ we might prefer a less drastic design solution to achieve the same performance. He changed the toggle back to investigate other options. One of the top features in the list was tire weight. Despite a high ranking (third), the PD line was quite flat. Looking at ICE lines for this feature, typical crossing patterns pointed to statistical interactions (DST2). He then inspected the interaction matrix (Figure 1(c)). The highest interaction was with seasonality (third row, before-last cell). Winter tires exhibited a falling trend, while summer tires were rather flat. Hence, he checked the current status. Interact predicted 74.2 Hz for summer tires. Toggling the season to winter took the prediction below 70 at 69.2 Hz. This could probably “make sense,” he said. Across the market, winter tires typically have more rubber, that is, mass in the tread area (DST5). “In addition, the ICE lines are useful to evaluate the feasible design space. We can look at the range (thickness of blue lines) of ICE lines and already estimate how challenging a target could be”, the expert said. He then checked other statistical interactions, acknowledging interesting information to be investigated more in depth.

Knowledge externalization and exploratory analysis

Engineers in tire acoustics must give recommendations to reduce tire noise. They usually make suggestions based on company guidelines or simulation tools. Often, these guidelines can be generic and high-level. Other practices include the comparison with previous projects, experience sharing and potentially back-to-back analysis.^123,124 These methods work but are time-consuming and add a data retrieval burden. Yet, drawing conclusions from few data points is more risky. Experts need a vast domain knowledge to answer such requests. Before using and trusting the model (DST3), both experts sought to verify that the model agrees with physics. They inspected and confirmed the most important features (DST1) but also audited the actual vs. predicted plot. Next, the PD lines (in green) were analyzed. The amplitude and direction of change with various features were inspected, looking for directly or inversely proportional relations. “The green line is showing a trend which is already good, this is something I was looking for!,” an expert said. The directions and amplitudes made sense and were confirmed for most features. Another observation was linked to the inspection of the ICE lines (Figure 1(d)) for tire width. One can observe various patterns in the data. For instance, a group of lines has a falling trend, reaches a minimum then goes up. Others exhibit a plateau or a rising trend, then go down. “This means that there are groups of tires that behave in some way and others differently. Based on hundreds of tires, it’s really good!.” To explain this difference, for instance in Figure 5, we can see that these two trends are reflected and are grouped by rim width.

One of the experts made another hypothesis linked to a temporal evolution of another noise metric. In essence, she thought that due to progress made in the past years, this metric increased until it reached a peak and then plateaued. This trend was easily confirmed in Interact (DST5).

Discussion and conclusion

In the past decade, many visualization-assisted ML tools have been proposed. While many techniques and visualization tools can explain the learned model, or create what-if scenarios, to our knowledge, none supports the analysis of statistical interactions. Interact was proposed to fill this gap and was validated in the context of virtual tire design. Interact has new capabilities such as real-time, interactive what-if scenarios, statistical interaction analysis, comparative analysis for input-prediction pairs and means to assess the trustworthiness of predictions. Besides having two domain experts heavily involved in the tool design (the two co-authors), Interact was evaluated by 10 independent domain experts who found it useful in their applied context and provided feedback for further improvements.

Limitations

Currently, Interact supports regression models only. So far, we developed scented widgets for numeric and binary input features. Categorical and date-type features are also supported by Interact, albeit with limited feedforward information.⁹¹ The computation of interaction scores for categorical input variables has yet to be integrated in the tool. Also, an alternative to ICE plots may be needed for multilevel categorical features, as the levels might lack a natural order. The tool can also be extended to support classification models. Then, we could reuse some of the current visual designs, for example, the input panel (Figure 2), but the other views may have to be redesigned.

Interact currently approaches the trust perspective from three key angles: examining the raw data, assessing the concrete model, and understanding user expectations (refer to TL1,4,5 in Chatzimparmpas et al.²³). In this paragraph, we will outline the support provided in Interact to build trust in the model as well as possible extensions that might be required in other contexts. For raw data analysis (TL1), we employ input feature distribution visualizations, enabling us to visualize the data used to train the model and determine whether its ranges align with the problem at hand. Going beyond what is offered in Interact, more detailed information such as point by point data inspection and data attributes descriptions could also be useful. To evaluate the model’s quality, Interact presents standard goodness-of-fit metrics. These metrics can help the user to build trust in the model across the design space (TL4). Going beyond building trust in a model, it might be desirable to be able to compare multiple versions of a model, for example when retraining the model as new data becomes available, as mentioned in TL4. Furthermore, we offer trust-building visualizations related to the model, such as the PDP, ICE and ALE plots. The latter allows users to quickly observe whether outcomes vary as expected with specific inputs – TL5 (see Input space visualization). Besides fostering trust in the model as a whole, we provide localized goodness-of-fit information, as illustrated in Figure 3. This information is generated based on the distribution of actual data in the vicinity of the current prediction. We supply a prediction interval that encompasses 95% of the actual data. However, it’s worth noting that this interval isn’t solely based on the current prediction but rather on all predictions within its neighborhood. To further enhance the accuracy of individual predictions and reduce uncertainty, techniques like Gaussian processes can be employed.¹²⁵ Addressing this specific limitation is a part of our future development plans. Similarly, more work for the development of visualization of causality¹²⁶ can be undertaken to further enhance trust.

The feature interaction metrics used by Interact can scale up to higher-order interactions, but the current version of the tool displays second-order (i.e. pairwise) interactions only. The interpretation of second-order statistical interactions is known to be hard, and things worsen in the case of higher-order interactions.^45,127 This is why we chose not to go beyond the second order. As the interpretation of interactions relies also on domain expertise, Interact puts the expert in the loop by affording her to visualize interaction effects and anticipate the direction and magnitude of their impact on predictions. The ease of use and the snappy response achieved by Interact enables the expert to examine thoroughly and build trust in the ML model eventually.

Instead of shipping a closed list of pretrained ML models, some of our study participants wanted to use the tool with improvised models. Similarly, users might prefer a different arrangement of the sliders, which are currently sorted based on model’s feature importance metrics. Users might be accustomed and/or grouped to some preferred order and this poses visual search challenges. We will improve the flexibility of Interact in these directions in the future.

Some users also wanted a more detailed instance-based view of the data. The tool currently allows to inspect individual data points only by mousing over the line plots or scatter plots.

Applicability beyond tire design

Interact is a visual what-if tool based on regression models with enhanced support for the analysis of statistical interactions. The tool was developed in collaboration with tire industry engineers. The validation of the tool was mainly based on a user study and a case study in this industry, presented earlier in this paper.

Nevertheless, it is reasonable to believe that our approach is transferable to other application areas where regression models are used to make predictions based on multiple inputs. This is justified by the fact that four out of five tasks (DST) that we support with Interact are well-documented in the XAI literature (see Tasks). This paper puts forth one new task, which is the analysis of statistical interactions (DST2). We believe this is an underexplored area as, to our knowledge, no other what-if tools propose such analysis (see Related Work). We propose a solution that fits the tire industry needs and call for more work in this direction. Although we cannot assert the relevance of statistical interactions in general for every regression problem, we strongly believe it is applicable within the broader scope of product design, where physics laws often induce such feature interactions. In this context, models can be tested against the physics of objects, requiring from experts to assess the behavior of models and hence their trustworthiness and reliability. A number of questions may guide the transfer of our approach to other fields: (1) Are higher-order interactions important to the target fields? If so, we still miss a way of visualizing them. (2) Are statistical interactions for categorical features essential to the target fields? If so, how can they be visualized? (3) Do other fields need to combine statistical interactions with optimization? Answering positively to any of these questions is grounds to not use Interact in its current form.

Research opportunities

We base our thoughts for future work on our experience while developing Interact, but also on the expert feedback received at various occasions, such as the focus group.

Single versus multi-model approaches

Interact activates and queries one model at a time currently. It allows users to understand the active ML model and use it, for example, for product design. Yet, in applied environments, experts usually need to meet many, often antagonistic constraints, which may take advantage of multiple models activated in parallel. An example of such work is VisProm.¹²⁸ In the future, we would like to test this system in our applied context, that is, virtual tire development. We think more research is needed to: (1) explore the suitable design space of visualizations for multidimensional inputs together with multi-model outputs; (2) provide useful feedforward information in such contexts; (3) scale the visualizations of statistical interactions up for multi-model approaches.

The field of Multi-Criteria Decision Making (MCDM) is very rich (see Related Work). Many MCDM techniques were devised to help domain experts. We believe that this line of work can be even more useful, for instance, if combined with state-of-the-art multi-model XAI visualization tools. MCDM is by default multi-model and multidimensional and fits many problem-solving techniques encountered in product design. We call for more visualization work integrating MCDM with multi-model explainable ML approaches in various fields, not limited to product design.

Provenance and storytelling

When using what-if tools, users typically aim to reach one or a few working solutions to their problem. Often, beyond the interaction with the system, these outcomes are discussed, in other contexts with other colleagues or with management representatives. To maintain a consistent and structured dialog, information related to the provenance¹²⁹ of the solutions is beneficial. The presentation of these solutions as a narrative storytelling¹³⁰ can further increase the cohesion of the solutions with the overarching goal. Therefore, we call for more work in integrating provenance analysis and storytelling with what-if tools.

Feature importance in ML methods is defined differently across model types, which makes it hard to understand or compare feature importance metrics.¹³¹ Researchers might invent ways to design visualizations to allow the user to construct and visualize her own feature importance metric.

Dimensionality reduction (DR)

Interact currently provides ICE lines for the analysis of individual features. Recent work has proposed to use clustering to aggregate these lines.⁴³ We wish to understand how DR techniques could be combined from a visualization perspective with supervised learning. Is DR useful in what-if analyses?

Lessons learned for what-if tools

Reflecting on 30 months of development of Interact, the following takeaways may interest visualization researchers.

Instant feedback

Systems such as Interact need to provide instant feedback. When multiple scenarios need to be assessed rapidly, the expert cannot wait long to obtain a prediction. In contrast to the “What-If Tool,”⁶¹Interact computes predictions on-the-fly as the user moves input sliders, without waiting for the user to hit a “run” button. Users get, hence, a smoother experience.

Explainability mechanism

When exploiting complex models, their outcomes might be unpredictable or conflicting with expected behavior. What-if tools need to support the end users in understanding how the application reached its output. This will further enhance their trust in the system and allow for constructive criticism, for example for improving the underlying models. In our case, besides the classical ICE, ALE and Partial Dependence visualizations, we use the concept of statistical interactions to support this goal.

Comparative analysis

The ability to create and save multiple scenarios is essential in the context of what-if tools. The users need to be able to effectively visualize and reload previous states effortlessly during a working session and beyond. In the context of product design, we relate such a need to the creation of several virtual designs before manufacturing. Iterating from a previously saved virtual design and comparing it to the current design is a standard need in such contexts. The selection of a saved design among multiple prior designs needs to be affordable. The user needs to be supported in the choice of one of the saved alternatives for instance by having some visualization that helps in assessing the related model prediction or its configuration compared to others. We could even argue that the history or provenance information of the saved virtual designs could be of interest.

Feedforward information

The simplest what-if tool could probably allow for the creation of multiple scenarios by trial-and-error. We argue that this method would be too tedious in the context of product design and could lead to a loss of user engagement. We believe that the user needs to be accompanied in their search for a solution. We use and recommend the use of scented widgets displaying feedforward information to guide the user in the design space. Visual cues need to be available and affordable to the user for reaching their final goal and anticipate on actions.

Automation and need

Another consideration that needs to be taken into account when creating what-if tools is whether there is a real need for such a tool. We think typically of situations where optimization engines could provide successfully the expected results and the need for a what-if tool is not fully justified. Like data visualization is not always the most-suited solution for data-centric problems, a what-if tool is suitable when an algorithm (e.g. optimization algorithm) would require more information than is available to solve the problem, and/or the task is not fully clear (see the information location and task clarity dimensions described by Sedlmair et al.¹³²). However, users need to benefit of as much automation as possible to reach their goals. They should focus on their decision making process without being distracted by usability issues. Nonetheless, automation must be carefully designed to not alter the exploratory nature of the what-if tool.

Realism

Often, models will provide a prediction even if the inputs do not make sense in reality. For example, a model that predicts maximum speed of a car based on a number of inputs such as car brand, engine type and, year of production, could accept an input such as a V8 diesel engine from 1992 on an electric car. While such an input is semantically void, the model will still provide a prediction. The industry is gradually adopting virtual design to reduce waste and needs to provide new and virtual inputs to models which can fairly interpolate/extrapolate. Users need an indication of the realism of such an input in light of the existing data/reality, or even the ability to compare and interactively select the most similar, yet realistic data points. Today, Interact uses a similarity metric (with respect to training data) which puts equal weights on the full set of inputs. Given the high dimensionality of the data, the resulting similarity scores lack sensitivity. Future research might focus on assigning weights to features, like in WeightLifter,¹³³ and guiding the user toward feasible design spaces, similar to the TOP-slider.¹³⁴ This was a strong feedback from the focus group that we would like to tackle in the future.

Maintainability of what-if tools

What-if tools use models behind the scene. While model-agnostic approaches provide a good separation of concerns between the visual user interface and the underlying models, an obstacle still exists in the way of long-term deployment of what-if tools. Indeed, such deployment raises classic software architecture considerations in terms of versioning and compatibility issues for both the what-if tool and the models. An even better separation of concerns and maintainability could be achieved through a microservices architecture,¹³⁵ that is use individual APIs for each of its models to avoid such issues in future.

Interoperability

Lastly, what-if tools need to be integrated in broader ecosystems where other software solutions are used by domain experts. What-if tools need to be interoperable, for example by their ability to load and export common data formats. Users could, for instance, extract data from the system and use it in their own presentations and storytelling.

General lessons learned

Visualization guidelines and preregistration

When designing visualization systems together with domain experts, the visualization researcher needs to exploit existing practical guidance. Throughout our project, we proactively recognized and confirmed most of Sedlmair et al.’s pitfalls,¹³² which helped greatly in our collaboration and developments. In addition, we found it helpful to preregister our evaluation/validation protocols. This allowed us to be better prepared and to spot potential methodological flaws early on.

Pre-existing software environment

When creating a new visualization system, the visualization researcher should analyze the software already in use by the target population, including commercial applications. This simplifies the discussion with domain experts, as they often refer to visualizations and methods available in this type of software. This will also facilitate discussions about change management with the target users. The risk of having a comment such as “but this already exists in software X” is hence mitigated. An example of this in our case was the JMP statistical software.⁸⁹ This might also help to delineate the new contribution. Knowledge about company software development practices is also important to facilitate the deployment of the new application, for example, to ensure its compatibility with existing orchestration tools.

Be or work with a liaison

The design of effective visualizations relies on a sufficient knowledge of the field for which the software is intended. When the visualization researchers have no background in the target field, it’s crucial to be immersed in the domain and work with field experts. Sitting in a few meetings to gather requirements is not enough. Such collaboration must result in a team member taking the role of the liaison,⁸⁰ that is, a person with sufficient knowledge of both the target field and the visualization field to facilitate the interdisciplinary communication required for a successful project.

Supplemental Material

sj-pdf-1-ivi-10.1177_14738716231216030 – Supplemental material for Interact: A visual what-if analysis tool for virtual product design

Supplemental material, sj-pdf-1-ivi-10.1177_14738716231216030 for Interact: A visual what-if analysis tool for virtual product design by Vasile Ciorna, Guy Melançon, Frank Petry and Mohammad Ghoniem in Information Visualization

Footnotes

Acknowledgements

We would like to thank Prof. Seppe vanden Broucke from KU Leuven for authorizing the use of his H-statistic code,¹³⁶ and Alexandre Dillon for GUI improvements. We thank Julien Vaissaud for the fruitful collaboration on integrating acoustic ML models and his great feedback. We thank Dr. Peter Kindt and Dr. Doris Maus for their precious comments as domain experts and their availability for the case study.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Luxembourg National Research Fund (FNR) grant #14221651 and the Goodyear Innovation Center Luxembourg (GICL).

ORCID iDs

Vasile Ciorna

Guy Melançon

Supplemental material

Supplemental material for this article is available online.

References

Kwon

Choi

Kim

, et al. RetainVis: visual analytics with interpretable and interactive recurrent neural networks on electronic medical records. IEEE Trans Vis Comput Graph 2019; 25(1): 299–309. DOI: 10.1109/TVCG.2018.2865027

Gotz

Borland

Data-driven healthcare: challenges and opportunities for interactive visualization. IEEE Comput Graph Appl 2016; 36(3): 90–96.

Stirnberg

Cermak

Kotthaus

, et al. Meteorology-driven variability of air pollution (PM₁) revealed with explainable machine learning. Atmos Chem Phys 2021; 21(5): 3919–3948.

Rautenhaus

Bottinger

Siemen

, et al. Visualization in meteorology-a survey of techniques and tools for data analysis tasks. IEEE Trans Vis Comput Graph 2018; 24(12): 3268–3296.

Sukhija

Tatineni

Brown

, et al. Topic modeling and visualization for big data in social sciences. In 2016 Int. IEEE conf. on ubiquitous intelligence & computing, advanced and trusted computing, scalable computing and communications, cloud and big data computing, internet of people, and smart world congress, 2016, pp.1198–1205. IEEE. DOI:10.1109/UIC-ATC-ScalCom-CBDComIoP-SmartWorld.2016.0183.

Gunning

Stefik

Choi

, et al. XAI-Explainable artificial intelligence. Sci Robot 2019; 4(37): 7120. DOI: 10.1126/scirobotics.aay7120.

Chatzimparmpas

Martins

Jusufi

, et al. A survey of surveys on the use of visualization for interpreting machine learning models. Inf Vis 2020; 19(3): 207–233.

Gent

Walter

JD.

Pneumatic tire. Mechanical Engineering Faculty Research. 2006.

Gillespie

TD.

Fundamentals of vehicle dynamics. Technical report, SAE technical paper, 1992. https://www-sae-org-s.web.bisu.edu.cn/publications/books/content/r-114/.

10.

Wang

(ed.) Automotive tire noise and vibrations: analysis, measurement and simulation. Oxford: Butterworth-Heinemann, 2020.

11.

Clark

Dodge

The rolling resistance of pneumatic tires. Final Report. Report, U.S. Department of Transportation, NHTSA, Office of Research and Development, 1979.

12.

Jazar

RN.

Vehicle dynamics. Boston, MA: Springer, 2008. Vol. 1.

13.

Pierce

Acoustics. New York, NY: Springer, 2019. Vol. 3.

14.

Wiener

FM.

Experimental study of the airborne noise generated by passenger automobile tires. Noise Control 1960; 6(4): 13–16.

15.

Lee

AE.

Structure-borne tire noise statistical energy analysis model. Tire Sci Technol 1997; 25(3): 177–186.

16.

Hastie

Tibshirani

Friedman

, et al. The elements of statistical learning: data mining, inference, and prediction. New York, NY: Springer, 2009. Vol. 2.

17.

Iwao

Yamazaki

A study on the mechanism of tire/road noise. JSAE Rev 1996; 17(2): 139–144.

18.

Guidotti

Monreale

Ruggieri

, et al. A survey of methods for explaining black box models. ACM Comput Surv 2019; 51(5): 1–42.

19.

Endert

Ribarsky

Turkay

, et al. The state of the art in integrating machine learning into visual analytics. Comput Graph Forum 2017; 36(8): 458–486.

20.

Hohman

Kahng

Pienta

, et al. Visual analytics in deep learning: an interrogative survey for the next frontiers. IEEE Trans Vis Comput Graph 2019; 25(8): 2674–2693. DOI: 10.1109/TVCG.2018.2843369

21.

Yuan

Chen

Yang

, et al. A survey of visual analytics techniques for machine learning. Comput Vis Media 2021; 7(1): 3–36.

22.

Barredo Arrieta

Díaz-Rodríguez

Del Ser

, et al. Explainable Artificial Intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf Fusion 2020; 58: 82–115.

23.

Chatzimparmpas

Martins

Jusufi

, et al. The state of the art in enhancing trust in machine learning models with the use of visualizations. Comput Graph Forum 2020; 39(3): 713–756.

24.

Adadi

Berrada

Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 2018; 6: 52138–52160.

25.

Zhao

Lee

, et al. IForest: interpreting random forests via visual analytics. IEEE Trans Vis Comput Graph 2019; 25(1): 407–416. DOI: 10.1109/TVCG.2018.2864475

26.

Kahng

Thorat

Chau

DHP

, et al. GAN Lab: understanding complex deep generative models using interactive visual experimentation. IEEE Trans Vis Comput Graph 2018; 25(1): 310–320. DOI: 10.1109/TVCG.2018.2864500

27.

Wang

Gou

Shen

, et al. DQNViz: A visual analytics approach to understand deep Q-networks. IEEE Trans Vis Comput Graph 2018; 25(1): 288–298. DOI: 10.1109/TVCG.2018.2864504

28.

Kahng

Andrews

Kalro

, et al. ActiVis: visual exploration of industry-scale deep neural network models. IEEE Trans Vis Comput Graph 2018; 24(1): 88–97.

29.

Liu

, et al. Analyzing the noise robustness of deep neural networks. In: Proceedings of the 2018 IEEE conference on visual analytics science and technology. DOI: 10.1109/VAST.2018.8802509.2001.09395.

30.

Ming

Cao

Zhang

, et al. Understanding hidden memories of recurrent neural networks. In: Proceedings of the 2017 IEEE conference on visual analytics science and technology. DOI: 10.1109/VAST.2017.8585721.1710.10777.

31.

Spinner

Schlegel

Schafer

, et al. ExplAIner: a visual analytics framework for interactive and explainable machine learning. IEEE Trans Visual Comput Graph 2020; 26: 1064–1074.

32.

Ribeiro

Singh

Guestrin

“Why should I trust you?” Explaining the predictions of any classifier. In: Proc. ACM SIGKDD Int. conf. on knowledge discovery and data mining. DOI: 10.1145/2939672.2939778.

33.

Lundberg

Lee

SI.

A unified approach to interpreting model predictions. In: Advances in neural information processing systems. 2017, p.30.

34.

Ming

Bertini

RuleMatrix: visualizing and understanding classifiers with rules. IEEE Trans Vis Comput Graph 2019; 25(1): 342–352. DOI: 10.1109/TVCG.2018.2864812

35.

Zhang

Wang

Molino

, et al. Manifold: a model-agnostic framework for interpretation and diagnosis of machine learning models. IEEE Trans Vis Comput Graph 2019; 25: 364–373.

36.

Kwon

Eysenbach

Verma

, et al. Clustervision: visual supervision of unsupervised clustering. IEEE Trans Vis Comput Graph 2018; 24: 142–151.

37.

Jaccard

Wan

Turrisi

The detection and interpretation of interaction effects between continuous variables in multiple regression. Multivariate Behav Res 1990; 25: 467–478.

38.

Coulton

Chow

Interaction effects in multiple regression. J Soc Serv Res 1992; 16: 179–199.

39.

Jaccard

Interaction effects in logistic regression. Quantitative Applications in the Social Sciences, Sage Publications, 2001. DOI: 10.4135/9781412984515

40.

VanderWeele

Knol

MJ.

A tutorial on interaction. Epidemiol Methods 2014; 3(1): 33–72. DOI: 10.1515/em-2013-0005

41.

Friedman

Popescu

BE.

Predictive learning via rule ensembles. Ann Appl Stat 2008; 2(3): 916–954. DOI: 10.1214/07-aoas148

42.

Greenwell

Boehmke

McCarthy

AJ.

A simple and effective model-based variable importance measure. arXiv preprint arXiv:180504755, 2018. DOI: 10.48550/arXiv.1805.04755.1805.04755.

43.

Britton

VINE: visualizing statistical interactions in black box models. arXiv preprint arXiv:190400561, 2019. DOI: 10.48550/arXiv.1904.00561.1904.00561.

44.

Goldstein

Kapelner

Bleich

, et al. Peeking inside the Black Box: visualizing statistical learning with plots of individual conditional expectation. J Comput Graph Stat 2015; 24(1): 44–65. DOI: 10.1080/10618600.2014.907095.1309.6392

45.

Lamina

Sturm

Kollerits

, et al. Visualizing interaction effects: a proposal for presentation and interpretation. Clin Epidemiol 2012; 65(8): 855–862.

46.

Hooker

Discovering additive structure in black box functions. In: Proc. ACM SIGKDD int. conference on knowledge discovery and data mining, pp.575–580. DOI: 10.1145/1014052.1014122.

47.

Amer

Daim

Jetter

A review of scenario planning. Futures 2013; 46: 23–40.

48.

Harries

Correspondence to what? Coherence to what? What is good scenario-based decision making?

Technol Forecast Soc Change 2003; 70: 797–817.

49.

Golfarelli

Rizzi

Proli

Designing what-if analysis: towards a methodology. In DOLAP: Proc. ACM international workshop on data warehousing and OLAP. DOI:10.1145/1183512.1183523.

50.

Barnes

Quiason

Benson

, et al. Success stories in simulation in health care. In: Winter simulation conference proceedings. DOI: 10.1145/268437.268772.

51.

Bohanec

Zupan

Rajkovič

Applications of qualitative multi-attribute decision models in health care. Int J Med Inform 2000; 58-59: 191–205.

52.

Card

Ward

Clarkson

PJ.

Beyond FMEA: the structured what-if technique (SWIFT). J Healthc Risk Manag 2012; 31(4): 23–29.

53.

Swart

Raskin

Robinson

The problem of the future: Sustainability science and scenario analysis. Glob Environ Change 2004; 14(2): 137–146.

54.

Reilly

Willenbockel

Managing uncertainty: a review of food system scenario analysis and modelling. Philos Trans Royal Soc B Biol Sci 2010; 365(1554): 3049–3063.

55.

van Sluisveld

MAE

Hof

Carrara

, et al. Aligning integrated assessment modelling with socio-technical transition insights: an application to low-carbon energy scenario analysis in Europe. Technol Forecast Soc Change 2020; 151: 119177.

56.

Salas

Priest

Wilson

, et al. Scenario-based training: improving military mission performance and adaptability. In: Adler

Castro

Britt

(eds.) Military life: the psychology of serving in peace and combat (Vol. 2): operational stress. Praeger Security International. Westport, CT: Praeger Security International, 2006, pp.32–53.

57.

Karvetski

Lambert

Linkov

Scenario and multiple criteria decision analysis for energy and environmental security of military and industrial installations. Integr Environ Assess Manag 2011; 7(2): 228–236.

58.

Huss

Honton

EJ.

Scenario planning—what style should you use?

Long Range Plann 1987; 20(4): 21–29.

59.

Huss

WR.

A move toward scenario analysis. Int J Forecast 1988; 4(3): 377–388.

60.

Krause

Perer

Interacting with predictions: visual inspection of black-box machine learning models. In: Proceedings of the CHI conference on human factors in computing systems, 2016, pp. 5686–5697. DOI: 10.1145/2858036.2858529.

61.

Wexler

Pushkarna

Bolukbasi

, et al. The what-if tool: interactive probing of machine learning models. IEEE Trans Vis Comput Graph 2020; 26: 56–65.

62.

Sohns

Garth

Leitte

Decision boundary visualization for counterfactual reasoning. Comput Graph Forum 2023; 42: 7–20.

63.

Molnar

Interpretable machine learning. 2nd ed.2022. https://christophm.github.io/interpretable-ml-book

64.

Edwards

The theory of decision making. Psychol Bull 1954; 51(4): 380–417.

65.

Slovic

Lichtenstein

Fischhoff

Decision making. In: Atkinson

Herrnstein

Lindzey

, et al. (eds.) Stevens’ handbook of experimental psychology. New York: Wiley, 1988, pp.673–738, 2nd ed.Vol. 2, https://hdl-handle-net.web.bisu.edu.cn/1794/22321

66.

Kahneman

Tversky

Prospect theory: an analysis of decision under risk. In: MacLean

Ziemba

(eds.) Handbook of the fundamentals of financial decision making: Part I. Hackensack: World Scientific, 2013, pp.99–127.

67.

Tversky

Kahneman

The framing of decisions and the psychology of choice. In: Wright

(ed.) Behavioral decision making. Boston, MA: Springer, 1985, pp.25–41.

68.

Tversky

Kahneman

Rational choice and the framing of decisions. In: Karpak

Zionts

(eds) Multiple criteria decision making and risk analysis using microcomputers. Berlin; New York: Springer, 1989, pp.81–126.

69.

Kahneman

Slovic

, et al. Judgment under uncertainty: Heuristics and biases. Cambridge: Cambridge University Press, 1982.

70.

Milkman

Chugh

Bazerman

MH.

How can decision making be improved?

Perspect Psychol Sci 2009; 4(4): 379–383. x.

71.

Wilson

Daugherty

Collaborative intelligence: Humans and AI are joining forces. Harv Bus Rev 2018; 96(4): 114–123.

72.

Bastani

Sinchaisri

WP.

Improving human decision-making with machine learning. arXiv preprint arXiv:210808454, 2021. DOI: 10.48550/arXiv.2108.08454.

73.

Khosravi

Shahabi

Pham

, et al. A comparative assessment of flood susceptibility modeling using multi-criteria decision-making analysis and machine learning methods. J Hydrol 2019; 573: 311–323.

74.

Secinaro

Calandra

Secinaro

, et al. The role of artificial intelligence in healthcare: a structured literature review. BMC Med Inform Decis Mak 2021; 21(1): 125–223.

75.

Veropoulos

Machine learning approaches to medical decision making. PhD Thesis, University of Bristol, England, 2001.

76.

Sahoo

Pradhan

Das

Performance evaluation of different machine learning methods and deep-learning based convolutional neural network for health decision making. In: Nature inspired computing for data science. Springer, 2020, pp.201–212.

77.

Thirumalai

Duba

Reddy

Decision making system using machine learning and pearson for heart attack. In: International conference on electronics, communication and aerospace technology, 2017, pp.206–210, vol. 2. New York, NY: IEEE. DOI: 10.1109/ICECA.2017.8212797.

78.

Duan

Edwards

Dwivedi

YK.

Artificial intelligence for decision making in the era of Big Data – evolution, challenges and research agenda. Int J Inf Manag 2019; 48: 63–71.

79.

Dwivedi

Hughes

Ismagilova

, et al. Artificial intelligence (ai): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. Inf Manage 2021; 57: 101994.

80.

Simon

Mittelst¨adt

Keim

, et al. Bridging the gap of domain and visualization experts with a liaison. In: Eurographics conference on visualization (EuroVis) short papers. The Eurographics Association. DOI:10.2312/eurovisshort.20151137.

81.

Munzner

A nested model for visualization design and validation. IEEE Trans Vis Comput Graph 2009; 15: 921–928.

82.

Aboutorabi

HMR

Kung

. Application of coupled structural acoustic analysis and sensitivity calculations to a tire noise problem. Tire Sci Technol 2012; 40(1): 25–41.

83.

Garcia

Hansen

, et al. The state-of-the-Art in predictive visual analytics. Comput Graph Forum 2017; 36(3): 539–562.

84.

Sacha

Kraus

Keim

, et al. VIS4ML: an ontology for visual analytics assisted machine learning. IEEE Trans Vis Comput Graph 2019; 25(1): 385–395. DOI: 10.1109/TVCG.2018.2864838

85.

Zhou

Arshad

Luo

, et al. Effects of uncertainty and cognitive load on user trust in predictive decision making. In: Human-computer interaction–INTERACT 2017: 16th IFIP TC 13 international conference, Mumbai, India, 25–29 September 2017, pp.23–39, Proceedings, Part IV 16. Springer.

86.

Kindt

Sas

Desmet

Measurement and analysis of rolling tire vibrations. Opt Lasers Eng 2009; 47(3-4): 443–453.

87.

Willett

Heer

Agrawala

Scented widgets: improving navigation cues with embedded visualizations. IEEE Trans Vis Comput Graph 2007; 13(6): 1129–1136.

88.

Alin

Minitab. Wiley Interdiscip Rev Comput Stat 2010; 2: 723–727.

89.

JMP 12 basic analysis . Sas Institute, 2015.

90.

Newman

Scholl

BJ.

Bar graphs depicting averages are perceptually misinterpreted: the within-the-bar bias. Psychon Bull Rev 2012; 19(4): 601–607.

91.

Coppers

Luyten

Vanacken

, et al. Fortunettes: feedforward about the future state of GUI widgets. Proc ACM Hum-Comput Interact 2019; 3: 1–20.

92.

Gutierrez-Gomez

Petry

Khadraoui

A comparison framework of machine learning algorithms for mixed-type variables datasets: a case study on tire-performances prediction. IEEE Access 2020; 8: 214902–214914.

93.

Gleicher

Albers

Walker

, et al. Visual comparison for information visualization. Inf Vis 2011; 10(4): 289–309.

94.

Piñeiro

Perelman

Guerschman

, et al. How to evaluate models: observed vs. Predicted or predicted vs. observed? Ecol Modell 2008; 216: 316–322.

95.

Ciesielski

Set theory for the working mathematician. Cambridge: London Mathematical Soc. Student Texts, Cambridge University Press, 1997.

96.

du Prel

Hommel

Röhrig

, et al. Confidence interval or p-value?: part 4 of a series on evaluation of scientific publications. Dtsch Arztebl Int 2009; 106: 335–339.

97.

Harter

HL.

The method of least squares and some alternatives: Part V. Int Stat Rev 1975; 43: 269.

98.

Fox

Weisberg

Fitting linear models. JMP, A Business Unit of SAS 2011.

99.

Ghoniem

Fekete

Castagliola

A comparison of the readability of graphs using node-link and matrix-based representations. In: Proceedings of the IEEE symposium on information visusalization, INFOVIS. DOI: 10.1109/INFVIS. 2004.1.

100.

Boos

Stefanski

LA.

P-value precision and reproducibility. Am Stat 2011; 65: 213–221.

101.

Huber

PJ.

Projection pursuit. Ann Stat 2007; 13(2): 435–475.

102.

Pearson

Notes on the history of correlation. Biometrika 1920; 13: 25–45.

103.

Siramdasu

Wheeler

Understanding tire dynamic characteristics for vehicle dynamics ride using simulation methods. Tire Sci Technol 2020; 48(3): 188–206.

104.

Merkel

Docker: lightweight linux containers for consistent development and deployment. Linux J 2014; 2014(239): 2.

105.

Lam

Bertini

Isenberg

, et al. Empirical studies in information visualization: seven scenarios. IEEE Trans Vis Comput Graph 2012; 18(9): 1520–1536.

106.

Saraiya

North

Duca

An insight-based methodology for evaluating bioinformatics visualizations. IEEE Trans Vis Comput Graph 2005; 11(4): 443–456.

107.

Stasko

Value-driven evaluation of visualizations. In: Proceedings of the fifth workshop on beyond time and errors: novel evaluation methods for visualization, BELIV ’14, ACM, 2014, pp.46–53. DOI: 10.1145/2669557.2669579.

108.

Isenberg

Chen

, et al. A systematic review on the practice of evaluating visualization. IEEE Trans Vis Comput Graph 2013; 19(12): 2818–2827.

109.

Elmqvist

JS.

Patterns for visualization evaluation. Inf Vis 2015; 14(3): 250–269.

110.

Carpendale

Evaluating information visualizations. In: Kerren

Stasko

Fekete

, et al. (eds) Information Visualization. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, 2008, pp.19–45, Vol. 4950.

111.

Weber

Carpendale

Ebert

, et al. Apply or die: on the role and assessment of application papers in visualization. IEEE Comput Graph Appl 2017; 37(3): 96–104.

112.

Mazza

Berre

Focus group methodology for evaluating information visualization techniques and tools. In: Proceedings of the international conference on information visualization (IV’07), 2007, pp.74–80. New York, NY: IEEE. DOI: 10.1109/IV.2007.51.

113.

Erlingsson

Brysiewicz

A hands-on guide to doing content analysis. Afr J Emerg Med 2017; 7(3): 93–99.

114.

Janesick

VJ.

Peer debriefing. In: Ritzer (ed.) The Blackwell encyclopedia of sociology. New York, NY: Blackwell Publishing, 2007.

115.

Joshi

Kale

Chandel

, et al. Likert scale: explored and explained. Br J Appl Sci Technol 2015; 7(4): 396–403.

116.

Jamieson

Likert scales: How to (ab)use them. Med Educ 2004; 38(12): 1217–1218.

117.

Harpe

SE.

How to analyze likert and other rating scale data. Curr Pharm Teach Learn 2015; 7(6): 836–850.

118.

Bishop

Herron

RL.

Use and misuse of the likert item responses and other ordinal measures. Int J Exerc Sci 2015; 8(3): 297–302.

119.

Ross

Willson

VL.

One-sample T-test. Rotterdam: SensePublishers, 2017. pp.9–12.

120.

Sprent

Sign Test. Berlin, Heidelberg: Springer, 2011. pp.1316–1317.

121.

Taylor

Classical mechanics. G-Reference, information and interdisciplinary subjects series. University Science Books, 2005.

122.

Nagaya

Ikai

Chiba

, et al. Tire with self-repairing mechanism. JSME Int J Ser C Mech Syst Mach Element Manuf 2006; 49(2): 379–384.

123.

Student. The probable error of a mean. Biometrika 1908; 6: 1–25.

124.

De Winter

. Using the student’s t-test with extremely small sample sizes. Pr Assess Res Eval 2013; 18(1): 10.

125.

Tavazza

DeCost

Choudhary

Uncertainty prediction for machine learning models of material properties. ACS Omega 2021; 6(48): 32431–32440.

126.

Guyon

, et al. Practical feature selection: from correlation to causality. In: Mining massive data sets for security: advances in data mining, search, social networks and text mining, and their applications to security. 2008, pp.27–43. DOI: 10.3233/978-1-58603-898-4-27

127.

De Gonzalez

Cox

. Interpretation of interaction: a review. Ann Appl Stat 2007; 1(2): 371–385.

128.

Chakhchoukh

Boukhelifa

Bezerianos

Understanding how in-visualization provenance can support trade-off analysis. IEEE Trans Vis Comput Graph 2023; 29: 3758–3774.

129.

Gotz

Zhou

MX.

Characterizing users’ visual analytic activity for insight provenance. In: 2008 IEEE symposium on visual analytics science and technology, 2008, pp.123–130. New York, NY: IEEE.

130.

Segel

Heer

Narrative visualization: telling stories with data. IEEE Trans Vis Comput Graph 2010; 16(6): 1139–1148.

131.

Stijven

Minnebo

Vladislavleva

Separating the wheat from the chaff: on feature selection and feature importance in regression random forests and symbolic regression. In: Proceedings of the annual conference companion genetic evolutionary computing, 2011, pp.623–630. DOI: 10.1145/2001858.2002059.

132.

Sedlmair

Meyer

Munzner

Design study methodology: reflections from the trenches and the stacks. IEEE Trans Vis Comput Graph 2012; 18(12): 2431–2440.

133.

Pajer

Streit

Torsney-Weir

, et al. Weightlifter: visual weight space exploration for multi-criteria decision making. IEEE Trans Vis Comput Graph 2017; 23(1): 611–620.

134.

Laurillau

Nguyen

Coutaz

, et al. The TOP-slider for multi-criteria decision making by non-specialists. In: ACM Int. Conf. Proceeding Series, NordiCHI ’18, 2018, pp.642–653. ACM. DOI: 10.1145/3240167.3240185.

135.

Nadareishvili

Mitra

McLarty

, et al. Microservice architecture: aligning principles, practices, and culture. O’Reilly Media, Inc, 2016.

136.

vanden Broucke

. Discovering interaction effects in ensemble models, 2019. https://blog.macuyiko.com/post/2019/discovering-interaction-effects-in-ensemble-models.html.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB