Enhancing graph drawings through edge bundling using clustering ensembles

Abstract

Edge bundling is a technique used to improve the readability of large graph drawings by grouping edges to reduce visual complexity. This paper treats this task as a clustering problem, using compatibility metrics to evaluate solutions in an optimization pipeline combined with a clustering ensemble approach. The aim is to present the Clustering Ensemble-based Edge Bundling (CEBEB) method for solving the General-based Edge Bundling (GBEB) problem and report results for some given graphs. The CEBEB method proved very promising and generated better solutions than an existing evolutionary algorithm. Additionally, the paper introduces a new ensemble algorithm, specific for the GBEB, and reviews some previous results.

Keywords

Edge bundling optimization clustering ensemble machine learning

Introduction

The research area of Information Visualization primarily focuses on developing techniques to create visual representations of abstract data, making it easier for people to understand and comprehend. According to Card et al.¹ Information Visualization involves using interactive visual representations of data through computers to enhance cognitive abilities. This field of research includes several areas, such as Graph Drawing, which studies the problems and techniques related to the visual representation of graphs, to generate useful representations.

Graphs consist of mathematical models that are commonly used to represent physical or abstract entities and their relationships. To construct a graph, it’s often necessary to create a graphical arrangement that presents the nodes (or vertices) and edges. There are numerous ways to design such an arrangement, including different ways of displaying nodes and edges, various layouts, and approaches to positioning elements in a graphic presentation. The classical graph drawing literature discusses these aspects in detail.^2–5 Some of them involve adopting specific layout patterns that may change according to the type of graphs (like hierarchical and circular layouts, general node-link layouts or the visibility representation), using straight lines, poly-lines or curves to represent the edges, adopting distinct graphical shapes (like rectangles or ellipses) for the nodes and creating a 2D or a 3D layout.

The aim of a good graph drawing is to contribute to improve the user’s cognitive process, helping to identify the global structure of the data, its relationships, and other patterns contained within.⁶ Therefore, using esthetically pleasing drawings can make it easier to understand.

Graph drawing has applications in many areas,⁷ such as in designing digital circuits in microelectronics, modeling systems in software engineering, and designing physical structures using computer-aided design (CAD). Besides that, in recent decades, among the most popular technologies are the increasing mobility of applications, cloud computing, and the Internet of Things (IoT). The use of these technologies generates a large volume of data, mostly coming from simple transactions, such as electronic purchases, e-mail, documents, messages, information disseminated on social networks, and even data generated by sensors embedded in computing devices. This type of information natively represents relationships between objects, and one of the most common ways of structuring such information is through the use of graphs.

However, for relatively large graphs with many edge crossings, legibility is sometimes significantly impaired by screen space constraints, coupled with limited human cognitive power and the overlapping of other visual elements in the drawing.⁶ This visual complexity, commonly called “visual clutter,” can compromise even relatively simple tasks, such as finding the endpoint of an edge or perceiving the flow of information in the graph drawing. In that sense, edge bundling has been widely used for reducing visual complexity and improving the level of understanding of the information represented by graph drawings. Such an approach draws the edges of a graph as interpolated curves that share the same path, thus forming bundles, as seen in Figure 1.

Figure 1.

Graph drawings with (a) straight lines and with (b) the application of the edge bundling technique. Note that some edges in (b) are drawn together, which causes them to curve.

The academic scientific community has presented several methods for edge bundling in general graphs, and specific methods for the special cases of circular graphs, hierarchical digraphs, and parallel coordinates, among others. The first approach in this area was proposed by Holten,⁸ with the concept of joining adjacent edges in hierarchical data. Since then, many other entirely different approaches have been reported, for example, the image-based method,⁹ which uses digital image processing operations to assemble edge bundles; the geometry-based method,¹⁰ which employs a mesh for controlling the formation of the bundles; and the force-directed method¹¹ which uses the concepts of classical force-based graph drawing algorithms for joining edges. Figure 2 presents drawings generated using the mentioned approaches. It is important to note that there is a significant difference between the bundles and the style of the drawings produced by each approach.

Figure 2.

Different drawings with edge bundles for the US Migration graph (6.517 nodes and 9.780 edges). The first image is the drawing without joining edges: (a) straight lines drawing, (b) image-based method,⁹ (c) geometry-based method¹⁰ and (d) force-directed method.¹¹

In 2017,¹² an attempt was made to unify the concepts within this research area. Even so, there still exist several definitions of what a bundle of edges can be, and some definitions are not mathematically precise. In response to this, Ferreira et al. ¹³ proposed a general formulation of Edge Bundling (EB) from an optimization perspective, with definitions that enable the comparison and evaluation of specific edge bundling problems and methods. The authors considered that “edge bundling involves the creation of a graph drawing in which the edges are grouped through sharing segments in the plane, improving readability and preserving the connectivity and information of the original graph.” They formulated three specific EB problems: Angular-based Edge Bundling (ABEB), Compatibility-based Edge Bundling (CBEB), and General-based Edge Bundling (GBEB), which incrementally increase in complexity.

These problems fall in the category of explicit edge bundling, in which the set of edges that comprise each bundle has to be explicitly defined before the drawing step. The authors also presented an evolutionary algorithm, called Evolutionary Edge Bundling (EEB), to solve these problems. The EEB algorithm provided useful explicit bundling solutions, but at a high computational cost, particularly for the GBEB, which made its application infeasible for large graphs.

Another evolutionary approach for edge bundling was proposed by Saga et al. ¹⁴ It consists of a genetic algorithm that segments the edges using dummy vertices (false vertices, created by the split of a real edge into a sequence of line segments. The dummy vertices connect such segments) for routing them to make the bundles. The authors include esthetic criteria in their objective (fitness) function, for evaluating the visual quality of the generated drawings.

The current paper aims to investigate an alternative solution to the GBEB problem by following the perception mentioned by Lhuillier et al.¹² when reviewing other studies,^15,16 that there is a conceptual similarity between edge bundling, clustering, and image segmentation problems, not yet fully analyzed. In that regard, the present work revisits the GBEB problem and discusses a method (abbreviated as CEBEB) for solving it that employs both clustering and clustering ensemble algorithms, which will be further detailed, embedded in an optimization pipeline. A first introduction to the CEBEB was made in a preliminary paper¹⁷ and it is further explained and evaluated here. This work also reviews the previous results, by considering a correct implementation of the visibility compatibility criterion in both the proposed method and in the Evolutionary Edge Bundling (EEB) method, used for comparison.

The main contributions of this paper are:

An enhanced description and evaluation of an optimization method for the General-based Edge Bundling (GBEB) that uses clustering and clustering ensemble algorithms guided by edge-bundling compatibility metrics;

A new consensus algorithm for the GBEB based on edge compatibility; and

Proper results that support the idea of solving the GBEB problem as an edge clustering problem.

The remainder of this paper is organized as follows: Section “General-based edge bundling” defines the GBEB problem; Section “Clustering edge bundling and ensemble learning” provides a brief overview of the usage of clustering approaches to edge bundling and conceptualizes clustering ensemble learning; Section “Clustering ensemble-based edge bundle” introduces the proposed optimization method for the GBEB problem; Section “Experiments” shows the experimental setup and the results of the evaluation of the proposed method; and, finally, Section “Conclusion” suggests steps to further develop the reported work.

General-based edge bundling

Ferreira et al.¹³ define three problems in edge bundling: ABEB, CBEB, and GBEB. Since only the last problem will be treated in this paper, only its definition is given below.

Let $D$ be a given drawing of a graph $G = (V, E)$ . Also, let $C (p, q)$ be the compatibility measure between each pair of edges $p, q \in E$ based on the visual representation of $p$ and $q$ in $D$ . The GBEB problem is to determine a partition of $E$ into disjoint subsets $E_{1}, E_{2}, . . ., E_{n}$ , with $E = \cup_{i = 1}^{n} E_{i}$ , that maximizes the total compatibility of the graph $C_{G} = \sum_{i = 1}^{n} C_{E_{i}}$ and minimizes $n$ , with $C_{E_{i}} = \sum_{p, q \in E_{i}} C (p, q)$ .

For the definition of $C (a, b)$ , the compatibility measures between two edges proposed by Holten and van Wijk¹¹ were used: angle compatibility $(C_{a})$ , scale compatibility $(C_{s})$ , position compatibility $(C_{p})$ and visibility compatibility $(C_{v})$ . In addition, a fifth measure, called the distance compatibility $(C_{d})$ , as proposed by¹³ was also used. The overall edge compatibility measure employed by the latter authors is:

C (p, q) = C_{a} (p, q) \cdot C_{s} (p, q) \cdot C_{p} (p, q) \cdot C_{v} (p, q) \cdot C_{d} (p, q)

(1)

Finally, Ferreira et al. ¹³ investigated a particular instance of the GBEB problem that aims to maximize a weighted sum of the optimization goals. The $C_{E_{i}}$ concept was redefined to consider some minimum quality thresholds. The objective function $f$ maximized by the authors is:

f = w_{1} \cdot C_{G} + w_{2} \cdot \frac{1}{n}

(2)

The numerical thresholds $T_{a}$ , $T_{s}$ , $T_{p}$ , $T_{v}$ , and $T_{d}$ were used to define the constraining measures of angle, scale, position, visibility, and distance, respectively. The total threshold was defined as $T = T_{a} \cdot T_{s} \cdot T_{p} \cdot T_{v} \cdot T_{d}$ . Furthermore, $\min_C_{i}$ was assumed to be the smallest compatibility $C (p, q)$ among all pairs $p$ and $q$ in $E_{i}$ . The redefined $C_{E_{i}}$ is:

C_{E_{i}} = {\begin{matrix} \begin{matrix} \sum_{p, q \in E_{i}} C (p, q), if \min_C_{i} \geq T; \\ P_{E_{i}}, if \min_C_{i} < T; \\ 0, if | E_{i} | = 1 . \end{matrix} \end{matrix}

(3)

such that

P_{E_{i}} = {\begin{matrix} \sum_{p, q \in E_{i}} C (p, q) \cdot (- 3), & if \sum_{p, q \in E_{i}} C (p, q) > 0; \\ - 3, & if \sum_{p, q \in E_{i}} C (p, q) = 0 . \end{matrix}

(4)

To solve the GBEB problem, the authors developed the EEB method, which consists of an evolutionary algorithm that uses equation (2) as its fitness function. An individual in EEB is represented by a vector of size $| E |$ that indicates the bundle $Id$ to which each edge is assigned. The EEB can also activate extra restrictions, such as imposing a minimum angle between the edges to be joined, which is important when solving other edge bundling problems.

Clustering edge bundling and ensemble learning

Clustering algorithms have already been used or adapted by some authors for edge bundling. These approaches, however, did not focus on solving an explicit edge bundling optimization problem.

For example, Zhou et al.¹⁸ presented a technique that performs the creation of edge bundles using triangulation and a hierarchical clustering. That approach creates a triangulation based on a drawing of the graph with straight-line edges. The edge points generated by the triangulation are then subjected to a hierarchical clustering algorithm to assemble the bundles.

Telea and Ersoy⁹ applied agglomerative hierarchical clustering as an intermediate step in the proposed Image-based Edge Bundling approach. In this case, the first step is to generate a preliminary bundled graph drawing generated with Hierarchical Edge Bundles (HEB). Subsequently, the drawing is submitted to an agglomerative hierarchical clustering algorithm to identify the bundles on the image and therefore perform rendering. However, the drawing used to perform the clustering already contained information about the bundles generated in the preliminary design.

Yamashita and Saga¹⁹ proposed a concept of edge bundling using edge cluster information by a line graph. Such an approach consists of a clustering-based method for edge bundling that uses a node clustering algorithm for community detection in the line graph. Thereafter, the edges are bundled based on the cluster information using Force-Directed Edge Bundling.¹¹

An idea that has been quite popular is the ensemble learning, which consists of selecting a collection, or set, of hypotheses and combining their predictions using an average, voting, or applying some machine learning technique. Russell and Norvig²⁰ says that there are basically two reasons for implementing this type of approach. The first of these is to reduce bias, once the hypothesis space of a model can be very restrictive, imposing a strong bias. In this sense, an ensemble can be more expressive and therefore have fewer biases than the basic models. The second reason is to reduce variance: consider, for example, a set of classifiers that combine the solutions using majority voting. For the ensemble to misclassify a new example, the majority of classifiers must misclassify it. The probability of this occurring is much lower than a single incorrect classification by a single classifier.

The concept of ensemble was first used in the combination of supervised classifiers and, due to the success obtained, the idea was applied to the combination of results from different clusters. The approach emerged as an alternative to improve the quality of results of clustering algorithms and also to help clarify the following questions: “If there are two different clustering algorithms, and they are applied to the same set of data, different results can be obtained. But which one is correct? How can the results be evaluated?”²¹

Given a set of objects, a clustering ensemble method consists of two main steps: (1) Generation, which deals with creating a set of partitions of these objects, and (2) Consensus Function, where a new separation, which is an integration of all partitions obtained in the generation step, is calculated.²²

Among the different types of consensus functions, some methods can be highlighted that transform the problem of combining clusters into the problem of graph or hypergraph partitioning. The difference between these methods is the way the graph is modeled from the set of clusters and how the cuts are defined to obtain the consensus partition. Strehl and Ghosh²³ defined the consensus partition as the partition that has the most information shared with all partitions in the cluster set. The authors proposed three heuristics based on partitioning to solve the problem. The set of clusters is represented as a hypergraph, and each partition is represented by a hyperedge. Such heuristics are detailed below:

Cluster-based Similarity Partition Algorithm (CSPA) – From the hypergraph, a $n \times n$ similarity matrix (the co-association matrix) is constructed, with $n$ corresponding to the size of the dataset $X$ . Such a matrix can be considered the adjacency matrix of a fully connected graph, where the nodes are the elements of the set $X$ and an edge between two objects has an associated weight equal to the number of times the objects are in the same cluster. Thereafter, the METIS²⁴ algorithm was used to obtain the consensus partition.²¹

Hypergraph Partition Algorithm (HGPA) – This is a straightforward approach to clustering ensemble that partitions the data using the clusters provided as indications of strong ties. The problem is formulated as partitioning the hypergraph by cutting a minimum number of hyperedges, considering that they all have the same weight. The hypergraph is partitioned into $k$ connected components of approximately the same dimension.²³ To implement the method, the hypergraph partitioning package HMETIS²⁴ is used.

Metaclustering Algorithm (MCLA) – Firstly, all similarity between pairs of clusters $C_{i}$ and $C_{j}$ is defined in terms of the number of objects grouped in both. Then, a similarity matrix between the clusters is formed, which represents the adjacency matrix of the graph constructed considering the clusters as nodes and assigns a weight to the edge between two nodes, equal to the similarity between the clusters. Thereafter, this graph is partitioned using the METIS²⁴ algorithm, and the clusters obtained are called meta clusters. Finally, to find the final partition, the number of times each object appears in a meta cluster is calculated, and each object is assigned to the meta cluster to which it has been assigned the most times.²¹

Another method, also based on graph partitioning, called Hybrid Bipartite Graph Formulation (HGBF), was proposed by Fern and Brodley,²⁵ in which clusters and objects are modeled together in the same graph. In this method, the bipartite graph is constructed in such a way that there are no edges between vertices that are instances or clusters. There is only an edge between two nodes if one of the nodes represents a cluster and the other represents an object that belongs to this cluster. The consensus partition is obtained by partitioning this graph using the METIS²⁴ algorithm or the spectral clustering²⁶ algorithm.

With a different approach, Li et al. ²⁷ proposed the clustering consensus solution using Non-Negative Matrix Factorization (NMF). The authors define this approach as a “simple and demonstrably convergent algorithm for solving the consensus problems of clustering and semi-supervised clustering.” This method defines consensus as the median partition problem, using distance as a measure of similarity between partitions. The original problem definition is relaxed consecutively to transform the problem into an optimization problem that can be solved by an iterative process. However, this process may only find local minima, rather than a global minimum of the problem.²¹

Clustering ensemble-based edge bundle

The proposed method, called Clustering Ensemble-based Edge Bundle (CEBEB), solves the GBEB problem on undirected graphs by using clustering and clustering ensemble algorithms for producing edge clusters and by combining them.

In short, the CEBEB clustering ensemble activity is divided as described below:

Level 1 Generate various solutions using some clustering algorithms with multiple parameter combinations;

Level 2 Submit the solutions generated at Level 1 to consensus algorithms;

Level 3 The best result that was obtained among the solutions generated at Levels 1 and 2 constitutes the output of the ensemble.

At each level, the generated solutions are evaluated by the objective function $f$ in equation (2). It was adopted $w_{1} = 0.4$ and $w_{2} = 0.6$ , the same values used in.¹³ Each cluster obtained by the methods represents a bundle of edges and is evaluated as such. As in the EEB, a minimum compatibility threshold was defined for the generated bundles, and the same penalty mechanism (see equations (3) and (4)) was used for bundles with total compatibility below this limit. This penalty mechanism may result in negative values.

The CEBEB method has an optimization pipeline that divides the resolution of the GBEB problem into three steps, as can be seen in Figure 3, detailed in the following subsections.

Figure 3.

The steps of CEBEB method.

Compatibility calculation

Firstly, the compatibility between the edges of the graph is calculated. The geometric compatibility metrics to be used are the same as those already used to solve the GBEB problem.¹³ The general compatibility of the graph $C_{G}$ is obtained by the general compatibility between each pair of edges of the graph. The pairwise compatibilities are represented by a square matrix of features of order $E$ , that serves as input for the clustering algorithms. In this matrix, (termed the feature matrix), each edge is represented by a vector of $| E |$ dimensions.

Bundle assembly

Next, a bundle assembly step is performed. It consists of three activities, which are structured as follows and are shown in Figure 4.

Figure 4.

Bundle assembly pipeline of CEBEB.

Estimation of the number of clusters

In addition to the feature matrix constructed in the previous step, some clustering methods require an estimate of the number of clusters (bundles) as input. To calculate the estimated value of this parameter, which is represented by $k$ , there are used three algorithms that are available in the relevant literature: the Elbow Method, the Hartigan Method, and the Silhouette Coefficient.^28–30 The highest estimated $k$ calculated by those algorithms is selected and passed on to the next activity, where it is used as a parameter for the clustering methods that require such a piece of information.

Clustering

The employed approach involved the utilization of 10 clustering algorithms that are widely cited in the literature. To select these algorithms, diversity was sought in the way the clusters were constructed so that the solutions generated were as different as possible. Various combinations of parameters were systematically tested to yield diverse results in clustering. Each set of parameters was inputted and executed across all 10 algorithms (see Table 1).

Table 1.

Number of executions and parameters of each clustering algorithm used in the CEBEB.

Algorithm	Parameters	Number of Executions
Agglomerative Hierarchical (AH)³¹	affinity = [‘euclidean’, ‘l1’, ‘l2’, ‘cosine’, ‘manhattan’], compute_full_tree = [‘auto’, True, False], linkage = [‘ward’, ‘complete’, ‘average’, ‘single’], n_clusters = $[2, \min (2 k, \| E \|))$	$48 \cdot (\min (2 k, \| E \|) - 2)$
Affinity Propagation (AP)³²	affinity = [‘euclidean’, ‘precomputed’], convergence_iter = [30, 50, 100], damping = [0.5, 0.6, 0.7, 0.8, 0.9], max_iter = [100,300]	60
Density-Based Spatial Clustering (DBSCAN)³³	algorithm = [‘auto’, ‘ball_tree’, ‘kd_tree’, ‘brute’], eps = [0.3, 0.5, 1], metric = [‘chebyshev’, ‘cityblock’, ‘euclidean’, ‘l1’, ‘l2’, ‘manhattan’], min_samples = [1,2,3,4,5], p = [0,1,2]	1.080
Gaussian Mixture Model (GMM)³⁴	covariance_type = [‘full’, ‘tied’, ‘diag’, ‘spherical’], init_params = [‘kmeans’, ‘random’], max_iter = [100,300], n_init = [1,10,100], random_state = [0], n_clusters = $[2, \min (2 k, \| E \|))$	$48 \cdot (\min (2 k, \| E \|) - 2)$
Hierarchical DBSCAN (HDBSCAN)³⁵	algorithm = [‘best’, ‘generic’, ‘prims_kdtree’, ‘prims_balltree’, ‘boruvka_kdtree’, ‘boruvka_balltree’], alpha = [0.5, 1.0, 1.3, 1.5, 2.0], eps = [0.3, 0.5, 1], method = [‘eom’, ‘leaf’], metric = [‘chebyshev’, ‘cityblock’, ‘euclidean’, ‘l1’, ‘l2’, ‘manhattan’], min_samples = [1,2,3,4,5]	5.400
Mini-Bach K-Means (MBKM)³⁶	init = [‘k-means++’, ‘random’], batch_size = [50, 100, 300], max_iter = [100, 300], max_no_improvement = [30, 50, 100], n_init = [5, 10, 100], random_state = [0], n_clusters = $[2, \min (2 k, \| E \|))$	$108 \cdot (\min (2 k, \| E \|) - 2)$
Meanshift (MS)¹⁶	bandwidth = [None, 0.1, 0.5, 1,3], max_iter = [100, 300]	10
K-Means (KM)³⁷	algorithm =[‘auto’, ‘full’, ‘elkan’], init = [‘k-means++’, ‘random’], max_iter = [100, 300], n_init = [5, 10, 100], random_state = [0], n_clusters = $[2, \min (2 k, \| E \|))$	$36 \cdot (\min (2 k, \| E \|) - 2)$
Spectral Clustering (SC)²⁶	affinity = [‘nearest_neighbors’, ‘rbf’], assign_labels = [‘kmeans’, ‘discretize’], eigen_solver = [‘arpack’], n_init = [5,10,100], n_neighbors = [3,5,10,15], random_state = [0], n_clusters = $[2, \min (2 k, \| E \|))$	$48 \cdot (\min (2 k, \| E \|) - 2)$
Spectral Co-Clustering (SCC)³⁸	init = [‘k-means++’, ‘random’], mini_batch = [False, True], svd_method = [‘arpack’, ‘randomized’], random_state = [0], n_clusters = $[2, \min (2 k, \| E \|))$	$8 \cdot (\min (2 k, \| E \|) - 2)$

Several of these selected clustering methods require an estimate of the number of clusters as part of their input data. In order to determine this value, some algorithms proposed in the literature were used. However, since the present objective is to explore the potential of clustering approaches to the problem of edge bundling, and as the results of these methods are approximate, there are used entries in the interval $[2, \min (2 k, | E |))$ , $k \in ℕ$ as estimates.

The input values for the numerical parameters were defined empirically. Those parameters based on initialization (n_init) and the maximum number of iterations (max_iter) were the same for all methods. Finally, the seed for the random-number generator (parameter random_state), when available, was set to zero to enable the reproduction of the achieved results.

In this context, the proposed clustering ensemble combines the two ways of the generation mechanisms proposed by Vega-Pons and Ruiz-Shulcloper²¹: “different clustering algorithms” and “different initialization parameters.”

The results obtained from all executions of the algorithms used were evaluated by the objective function to order them by the quality of the solution. After this evaluation, the best result found by each method was selected and submitted to the next activity: the clustering consensus.

Clustering consensus

The best solutions generated by running these diverse algorithms and their parameter combinations are submitted to standard consensus algorithms in the literature: CSPA,²³ HGBF,²⁵ HGPA,²³ NMF²⁷ and MCLA,²³ detailed in the previous section.

However, these mentioned algorithms perform consensus using only the sets of clusters already produced by the clustering algorithms as initial information. In this sense, the authors proposed a new algorithm, called the Compatibility-based Cluster Consensus Algorithm (CCCA), which was developed to include the concept of edge compatibility in the consensus process.

The proposed Compatibility-based Cluster Consensus Algorithm (CCCA) performs a qualitative analysis of the clusters of the solutions generated by the clustering algorithms applied in the previous activity. The method works as follows:

Add every cluster (bundle) in all solutions to a unique list $L$ .

Compute the $C_{E_{i}}$ value of every cluster $E_{i}$ in $L$ . That value there is referred to here just as the “evaluation value.”

Eliminate duplicates in $L$ . If a cluster $E_{i}$ has $m_{i}$ copies in $L$ , after removal, its evaluation value must be updated to $m_{i} * C_{E_{i}}$ .

Sort the clusters in $L$ in increasing order of their evaluation values.

Try to combine, two by two, the highest-rated clusters in $L$ that share at least one element: remove the duplicate elements from the lower-rated cluster, join them together, and assess whether the evaluation value of this new cluster is better (higher) than the sum of the evaluation values of the two separated clusters. If the new cluster is better, eliminate the two original clusters from $L$ and add the new one to it, sorted by its evaluation value. Otherwise, the two original clusters remain, although the lower-rated one will not keep the duplicated element. Repeat this step, while there are clusters to combine.

If, $| E_{i} | = 1$ and $| E_{j} | = 1$ , for a pair of solitary clusters $(E_{i}, E_{j})$ in $L$ , then combine them in a new joint cluster $E_{ij}$ . Identify the joint cluster with the highest evaluation value. If this value is positive, replace the original solitary clusters in $L$ with the corresponding joint cluster. Repeat this step until no change can be made to $L$ .

Output $L$ , as the new clustering solution.

The Maximum (MAX) algorithm is also used to select the solution with the best evaluation value obtained at the bundle assembly activity.

Finally, the best result obtained, that is, the one with the maximum evaluation value, is selected among the solutions generated until the moment and constitutes the output of the ensemble, as shown in Figure 5, that summarizes the clustering and clustering ensemble algorithms used in CEBEB pipeline.

Figure 5.

Summary of clustering and clustering ensemble algorithms used on the pipeline: this is the data processing flow in the CEBEB approach.

Graph drawing

The third and final step of the CEBEB approach consists of providing the bundles to a rendering module to create a graph drawing that reflects the joining of the edges. However, similar to GBEB, edge routing was not the initial focus of CEBEB and has not yet been fully addressed in the context of this proposal.

This activity has been performed via a specialized implementation of the Force-Directed Edge Bundling approach by Holten and van Wijk,¹¹ carried out by Ferreira et al. ¹³ The algorithm was modified to take the bundles as input and draw them individually. According to Ferreira et al.,¹³“the force-directed algorithm ensures that the individual axial symmetry of each bundle looks plausible.”

Experiments

To validate the CEBEB approach, some experiments were performed. The objective of these experiments was to ascertain whether the clustering algorithms, either individually or in an ensemble, are capable of assembling edge bundles for the GBEB problem with comparable or superior quality to those found by EEB. The experiments are now described for a set of five graphs from distinct domains, which were also analyzed by Ferreira et al. ¹³ in the context of the GBEB. The graphs employed are a synthetic general graph (20 nodes, 28 edges)³⁹ and four real-world graphs: Zachary Karate Club (34 nodes, 78 edges),⁴⁰ Dolphin Social Network (62 nodes, 159 edges),⁴¹ Movie Lens (160 nodes, 161 edges)⁴² and Les Misérables (77 nodes, 254 edges).⁴³

The compatibility measures used in these experiments were generated assuming an angular compatibility of $T_{a} (30^{°}) = 0.154$ . For the other thresholds, $T_{s} = 0.70$ , $T_{p} = 0.96$ , $T_{v} = 0.72$ and $T_{d} = 0.96$ were employed. All these values were defined empirically in¹³ and reused in the current work in order to allow a comparison of the results.

An important problem to note, however, is that Ferreira et al. ¹³ implemented the visibility compatibility measure differently. The authors calculated the visibility value between two edges using the order of the edges as they appear in a bundle. Thus, to facilitate comparison, both approaches had to be adjusted to consider the minimum value of $C_{v} (p, q)$ e $C_{v} (q, p)$ , as defined by Holten and van Wijk.¹¹ The experiments using EEB to solve GBEB, as performed by Ferreira et al.,¹³ had to be redone to allow for this adjustment and to make the comparison viable. The results obtained with EEB after this adjustment are shown in Table 2.

Table 2.

Results of 100 independent runs of EEB with the adjustment in visibility compatibility.

Graph	Total Time (s)	Best Solution		Median Solution		Worst Solution
Graph	Total Time (s)	Fitness	Bundles	Fitness	Bundles	Fitness	Bundles
Synthetic	1075	57.7910	15	51.4276	16	41.8775	19
Zachary Karate Club	14,484	137.1031	36	104.0863	44	85.0413	45
Dolphin Social Network	66,800	181.6772	81	146.5510	86	91.4569	113
Movie Lens	50,884	561.5985	63	438.1910	69	353.5073	88
Les Misrables	123,408	546.1595	114	428.0971	121	363.4282	130

In the CEBEB pipeline, after calculating the compatibility between the edges of the graph, and as part of the Bundle Assembly step, the estimate of $k$ (the number of clusters) was performed. Then, for each clustering algorithm used, several executions were carried out combining the different input parameters listed in Table 1. Next, the set of best solutions obtained by the clustering algorithm was submitted to the consensus methods for producing clustering ensembles.

The best results found by the algorithms in each activity of the Bundle Assembly step mentioned above are presented in Tables 3 to 7 for the various graphs used. The “Evaluation” column contains the values of the objective function (the higher the value, the better). The number of executions and the execution time for each algorithm are also shown.

Table 3.

Results of the CEBEB for the graph Synthetic.

Clustering Algorithms
Algorithm	No. Exec.	Time (s)	Evaluation	Bundles
AH	1248	3.2700	68.7083	13
KM	936	68.0961	59.3734	12
HDBSCAN	5400	34.3705	58.5294	16
MBKM	2808	316.4192	56.9816	10
SC	1248	99.1525	56.8614	11
GMM	1248	539.4711	55.4350	22
SCC	208	9.1719	50.1290	26
DBSCAN	1080	4.8721	45.3090	18
MS	10	0.2203	45.3090	18
AP	60	0.6180	28.7804	7
Clustering Consensus
Method	No. Exec.	Time (s)	Evaluation	Bundles
MAX	1	0.0000	68.7083	13
CCCA	1	0.0370	63.8656	9
HBGF	1	0.0319	37.1759	18
NMF	1	0.0122	30.7415	15
HGPA	1	0.0299	24.6644	18
CSPA	1	0.0112	24.3844	18
MCLA	1	4.0879	21.1594	16

Table 4.

Results of the CEBEB for the graph Zachary Karate Club.

Clustering Algorithms
Algorithm	No. Exec.	Time (s)	Evaluation	Bundles
AH	3648	22.0079	144.7386	34
MBKM	8208	7038.0345	110.0497	34
SC	3648	1101.7077	108.3638	33
KM	2736	679.7235	106.1520	41
GMM	3648	13,138.9911	100.2429	47
HDBSCAN	5400	64.6502	68.8067	48
SCC	608	67.5254	61.0898	55
DBSCAN	1080	9.1418	4.1434	77
MS	10	1.0252	0.0000	78
AP	60	1.0512	−70.7654	15
Clustering Consensus
Method	No. Exec.	Time (s)	Evaluation	Bundles
MAX	1	0.0000	144.7386	34
CCCA	1	0.5906	129.8735	32
MCLA	1	42.7004	55.7801	61
HBGF	1	0.1007	27.4718	59
NMF	1	0.1750	24.2822	70
CSPA	1	0.0374	12.1342	72
HGPA	1	0.1274	0.0000	78

Table 5.

Results of the CEBEB for the graph Dolphin Social Network.

Clustering Algorithms
Algorithm	No. Exec.	Time (s)	Evaluation	Bundles
AH	5088	99.4197	227.5186	89
SC	5088	31,757.0667	170.5272	85
KM	3816	2311.1646	154.2048	105
MBKM	11,448	17,150.6088	145.7646	88
MS	10	11.0166	134.4412	120
GMM	5088	51,451.8264	129.6849	97
DBSCAN	1080	45.5213	104.2349	135
HDBSCAN	5400	306.2281	69.0743	87
SCC	848	326.5419	55.7421	88
AP	60	7.6205	−256.3213	28
Clustering Consensus
Method	No. Exec.	Time (s)	Evaluation	Bundles
MAX	1	0.000	227.5186	89
CCCA	1	3.9453	216.5574	52
MCLA	1	43.4380	139.4733	115
NMF	1	0.6826	115.2808	118
HBGF	1	0.1275	45.0936	114
HGPA	1	0.2879	40.3009	135
CSPA	1	0.1287	15.8225	134

Table 6.

Results of the CEBEB for the graph Movie Lens.

Clustering Algorithms
Algorithm	No. Exec.	Time (s)	Evaluation	Bundles
AH	3840	33.1333	660.9262	57
MBKM	8640	12,793.2626	565.5901	55
KM	2880	1858.1729	551.3079	53
GMM	3840	34,103.0592	551.3079	53
SC	3840	1905.6845	537.0041	48
HDBSCAN	5400	149.7454	351.5596	87
SCC	640	142.7173	336.5863	81
MS	10	9.9257	278.7039	70
DBSCAN	1080	27.4801	191.3582	126
AP	60	4.5862	−723.8763	23
CCCA	1	1.5854	722.6440	46
Clustering Consensus
Method	No. Exec.	Time (s)	Evaluation	Bundles
MAX	1	0.0000	660.9262	57
MCLA	1	86.4034	424.6657	85
NMF	1	0.4386	211.1120	113
HBGF	1	0.1583	139.8875	105
HGPA	1	0.2730	114.5210	126
CSPA	1	0.0572	62.9266	126

Table 7.

Results of clustering algorithms used in the CEBEB for the graph Les Misrables.

Clustering Algorithms
Algorithm	No. Exec.	Time (s)	Evaluation	Bundles
AH	8448	192.4673	731.0542	108
MBKM	19,008	53,348.2374	591.7608	96
KM	6336	10,313.4852	511.9459	108
SC	8448	12,215.8971	503.5303	84
GMM	8448	186,718.6547	473.4432	128
MS	10	9.2101	440.1377	180
HDBSCAN	5400	316.4621	420.4440	121
DBSCAN	1080	49.2134	169.8969	183
SCC	1408	625.4648	130.2518	173
AP	60	7.5387	−800.6619	28
Clustering Consensus
Method	No. Exec.	Time (s)	Evaluation	Bundles
MAX	1	0.000	731.0542	108
CCCA	1	8.3533	646.7745	74
MCLA	1	92.7601	451.2268	154
NMF	1	7.2069	320.3031	170
HGPA	1	0.2869	169.0590	183
HBGF	1	0.1929	156.2996	168
CSPA	1	0.0995	49.9543	183

Finally, Figure 6 presents the comparison between the execution time of the CEBEB approach and 100 executions of the EEB, a non-deterministic evolutionary method used in Ferreira et al.¹³ to solve the GBEB problem. Since, solving the GBEB repeatedly by EEB, implies the possibility of a different result at each time of execution, the CEBEB result was compared to those 100 independent EEB runs. Figure 7 also compares the results of CEBEB with the best, median, and worst case of EEB for the GBEB problem. As CEBEB has deterministic characteristics, the data were obtained from a single execution of the complete optimization pipeline proposed.

Figure 6.

Comparison between the total execution time in seconds (s) of the CEBEB and 100 independent runs of the EEB for each of the five graphs studied.

Figure 7.

Comparison between the evaluation of the solution obtained by the CEBEB and the best, median, and worst solutions obtained by the EEB for each graph.

The tests were performed on a Dell G3 3590 laptop with a 2.6 GHz Intel Core i7 processor and 8 GB of RAM, with the support of the Scikit-learn⁴⁴ as a machine learning API.

The edge-bundling graph drawing solutions generated by the experiments are presented in Figures 8 to 12. In these figures, the edges that were joined into bundles appear in red and the bundle with the highest number of edges is highlighted in blue. If there is more than one bundle with the same highest number of edges, both are painted in blue.

Figure 8.

Comparison between the drawings generated by the EEB and CEBEB approaches for EB instances of Synthetic graph: (a) original drawing, (b) EEB (Bundles:15, Evaluation:57.79), and (c) CEBEB (Bundles:13, Evaluation:68.71).

Figure 9.

Comparison between the drawings generated by the EEB and CEBEB approaches for EB instances of Zachary Karate Club graph: (a) original drawing, (b) EEB (Bundles:36, Evaluation:137.10), and (c) CEBEB (Bundles:34, Evaluation:144.74).

Figure 10.

Comparison between the drawings generated by the EEB and CEBEB approaches for EB instances of the Dolphin Social Network graph: (a) original drawing, (b) EEB (Bundles:81, Evaluation:181.68), and (c) CEBEB (Bundles:89, Evaluation:227.52).

Figure 11.

Comparison between the drawings generated by the EEB and CEBEB approaches for EB instances of Movie Lens graph: (a) original drawing, (b) EEB (Bundles:63, Evaluation:561.60), and (c) CEBEB (Bundles:46, Evaluation:722.64).

Figure 12.

Comparison between the drawings generated by the EEB and CEBEB approaches for EB instances of Les Misrables graph: (a) original drawing, (b) EEB (Bundles:114, Evaluation:546.16), and (c) CEBEB (Bundles:108, Evaluation:731.05).

Discussion

One of the difficulties reported by Ferreira et al.¹³ when conducting the GBEB tests was related to the execution time of the evolutionary algorithm for this specific problem, which made it impossible to carry out experiments on larger graphs. However, when the adjustment on the visibility compatibility calculation was made, the execution time of the algorithm decreased significantly, and it is quite similar to the CEBEB processing time for most of the analyzed graphs, as shown in Figure 6. Despite the time required by the CEBEB to solve the problem being similar or greater in some cases, it is worth mentioning that the quality of the solutions found by this method was about 30% better than the median of the EEB. In addition, in all the five graphs studied, the solution found by the CEBEB is even better than the best solution of the EEB (Figure 7).

One aspect that deserves special attention is the fact that the Agglomerative Hierarchical Clustering (AH) approach managed to find the best CEBEB solutions at Level 1 for all the graphs tested, and that solution also was selected as the best at Level 2 in four of the studied graphs. Such solutions are better than the median of the solutions found by the EEB and were obtained in a very short time, when considering only the execution of the different combinations of the AH parameters (Level 1), and comparing with the time spent in a single execution of the evolutionary algorithm.

On the other hand, the Affinity Propagation (AP) algorithm did not present good results for any of the graphs tested, being the worst solution in all cases. It is also worth mentioning that the method with the highest time and processing costs, the Gaussian Mixture Model (GMM), does not appear among the three best solutions for any of the studied graphs. This indicates that the CEBEB approach can be improved by excluding from the clustering ensemble those algorithms that are not so efficient in the domain of the GBEB problem. Consequently, the CEBEB run-time would also be improved.

Among the consensus methods, the MAX algorithm (which takes the solution with the highest objective function value) produced the best solution in four of the five graphs. The proposed CCCA method produced the best solution only for the Movie Lens graph. Nevertheless, in all cases, the latter method resulted in solutions with a much lower number of clusters than the other ensemble approaches.

Besides achieving the best of edge compatibility values, regarding the human element in the evaluation of edge bundles, the present authors consider most of the drawings generated from the solutions given by CEBEB to be more “visually pleasing” compared to the drawings of the best solutions found by EEB. As can be seen in Figures 8 and 10 to 12 (except in Figure 9), CEBEB’s visual solutions have fewer crossings between edges and/or bundles and appear to have captured geometric compatibility better.

It should be noted that, in some of these drawings, both the EEB and CEBEB methods have assembled edge bundles that can create ambiguity. This may indicate that the penalty value used in equation (4) (see the −3 constant) should be higher for bundles with low compatibility. Future investigations may be necessary to determine adjustments for that constant and for the thresholds.

Conclusion

This paper treats the task of solving the general edge bundling problem (the GBEB) as a clustering problem using compatibility metrics to evaluate the generated solutions in an optimization pipeline, combined in a Clustering Ensemble-based Edge Bundling (CEBEB) approach. A CEBEB-based method was developed and tested on various existing and synthetic graphs. As far as we know, it is the first reported approach that uses clustering ensemble algorithms applied to edge bundling with an optimization focus.

The proposed method was capable of generating relatively good solutions compared to those produced by an existing, evolutionary algorithm for the GBEB, called EEB. It also has the potential to be accelerated and become faster than the previous algorithm, as it is possible to select only the most effective clustering algorithms to compose Level 1 or employs hyperparameter analysis to select the most effective combination of parameters. Furthermore, the clustering algorithms can be run in parallel since they do not depend on each other within Levels 1 and 2.

Based on the results obtained so far, the proposed method appears to be a very promising alternative for identifying edge bundling solutions that have few bundles while maintaining a high compatibility between the edges.

Furthermore, the study helped improve the EEB results through an efficient implementation of the visibility compatibility criterion.

As future work, the specialized Force-Directed Edge Bundling used as the post-processing algorithm for creating the graph drawings could be tuned to avoid routing edges too closely to unrelated nodes. The proposed method could also be extended to perform a multi-objective assessment, so that each compatibility measure is treated as an independent criterion, thus providing compromise solutions for a final user manual selection step. Additionally, the CEBEB method should be tested on larger and more complex graphs, for which the edge bundling technique has a greater impact on visual readability. For example, it could be applied to very large social networks, featuring community clusters varying in density of nodes and with different distances between them.

Footnotes

Acknowledgements

We thank LaMCAD/UFG for providing the computational support necessary for conducting some of the experiments mentioned in this paper.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Raissa dos Santos Vieira

References

Card

Mackinlay

Shneiderman

. Readings in information visualization: using vision to think. San Francisco, CA: Interactive Technologies Elsevier Science, 1999.

Tollis

Battista

Eades

, et al. (eds). Graph drawing: algorithms for the visualization of graphs. 1st ed. Upper Saddle River, NJ: Pearson, 1998.

Sugiyama

(ed.). Graph Drawing and applications for software and Knowledge Engineers. illustrat ed. Farrer Road, Singapore: World Scientific Publishing Company, 2002.

Jünger

Mutzel

(eds). Graph Drawing Software. 2004th ed. Berlin Heidelberg: Springer, 2003. editors.

Tamassia

(ed.). Handbook of graph drawing and visualization. 1st ed. Boca Raton London New York: Chapman and Hall/CRC, 2016. editor.

Tarawaneh

Keller

Ebert

. A General Introduction To Graph Visualization Techniques. In: Garth

Middel

Hagen

(eds.) Visualization of Large and Unstructured Data Sets: Applications in Geospatial Planning, Modeling and Engineering - Proceedings of IRTG 1131 Workshop 2011. vol. 27 of OpenAccess Series in Informatics (OASIcs); 2012. p. 151164. Dagstuhl, Germany: Schloss DagstuhlLeibniz-Zentrum fuer Informatik, http://drops.dagstuhl.de/opus/volltexte/2012/3748 (accessed 18 February 2024).

Battista

Eades

Tamassia

, et al. Algorithms for drawing graphs: an annotated bibliography. Comput Geom 1994; 4(5): 235–282.

Holten

Hierarchical edge bundles: visualization of adjacency relations in hierarchical data. IEEE Trans Vis Comput Graph 2006; 12(5): 741–748.

Telea

Ersoy

. Image-based edge bundles: simplified visualization of large graphs. Comput Graph Forum 2010; 29(3): 843–852.

10.

Cui

Zhou

, et al. Geometry-based edge clustering for graph visualization. IEEE Trans Vis Comput Graph 2008; 14(6): 1277–1284.

11.

Holten

van Wijk

. Force-Directed Edge Bundling for Graph Visualization. In: Proceedings of the 11th Eurographics / IEEE - VGTC Conference on Visualization. EuroVis09, 2009. p. 983998. Chichester, GBR: The Eurographs Association & John Wiley & Sons, Ltd.

12.

Lhuillier

Hurter

Telea

. State of the art in Edge and trail bundling techniques. Comput Graph Forum 2017; 36(3): 619–645.

13.

Ferreira

do Nascimento

Foulds

. An evolutionary algorithm for an optimization model of Edge bundling. Information 2018; 9(7): 154.

14.

Saga

Yoshikawa

Wakita

, et al. A genetic algorithm optimising control point placement for edge bundling. In: VISIGRAPP 2020 - Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol. 3; 2020. p. 217222. https://www.scitepress.org/Papers/2020/89832/89832.pdf (accessed 18 February 2024).

15.

Jain

Murty

Flynn

. Data clustering: a review. ACM Comput Surv 1999; 31(3): 264–323.

16.

Comaniciu

Meer

. Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 2002; 24(5): 603–619.

17.

Vieira

Do Nascimento

Ferreira

, et al. Clustering Ensemble-based Edge Bundling to Improve the Readability of Graph Drawings. In: 2022 26th International Conference Information Visualisation (IV), Vienna, Austria, 2022, pp. 21–26. DOI: 10.1109/IV56949.2022.00013.

18.

Zhou

Yuan

Cui

, et al. Energy-Based Hierarchical Edge Clustering of Graphs. In: 2008 IEEE Pacific Visualization Symposium; 2008. p.5561.

19.

Yamashita

Saga

. Cluster-based edge bundling based on a line graph. In: International Conference on Information Visualization Theory and Applications, vol. 4. SCITEPRESS; 2017. p.311316, https://www.scitepress.org/papers/2017/62360/62360.pdf (accessed 18 February 2024).

20.

Russell

Norvig

. Artificial Intelligence: A Modern Approach. Prentice Hall series in artificial intelligence. Hoboken, NJ: Prentice Hall, 2021.

21.

Vega-Pons

Ruiz-Shulcloper

. A survey of clustering ensemble algorithms. Int J Pattern Recognit Artif Intell 2011; 25(03): 337–372.

22.

Ghaemi

Sulaiman

Ibrahim

, et al. A survey: clustering ensembles techniques. World Acad Sci Eng Technol. 2009;50:636645.

23.

Strehl

Ghosh

. Cluster ensemblesa knowledge reuse framework for combining multiple partitions. J Mach Learn Res 2002; 3(Dec): 583617.

24.

Karypis

Kumar

. METIS: A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices. Minneapolis, MN: University of Minnesota, 1997, https://hdl.handle.net/11299/215346 (accessed 18 February 2024).

25.

Fern

Brodley

. Solving Cluster Ensemble Problems by Bipartite Graph Partitioning. In: Proceedings of the Twenty-First International Conference on Machine Learning. ICML

0'

4; 2004. p. 36. New York, NY, USA: Association for Computing Machinery, https://doi.org/10.1145/1015330.1015414 (accessed 18 February 2024).

26.

Jordan

Weiss

. On spectral clustering: Analysis and an algorithm. In: Advances in neural information processing systems; 2002. p. 849856.

27.

Ding

Jordan

. Solving consensus and semi-supervised clustering problems using nonnegative matrix factorization. In: Seventh IEEE International Conference on Data Mining (ICDM 2007); 2007. p. 577582. IEEE.

28.

Rokach

. A survey of clustering algorithms. In: Maimon

Rokach

(eds) Data mining and knowledge discovery handbook. New York, NY: Springer Science + Business Media, 2010, pp.269–298.

29.

Yuan

Yang

. Research on K-value selection method of K-means clustering algorithm. J 2019; 2(2): 226–235.

30.

Tian

. A comprehensive survey of clustering algorithms. Ann Data Sci 2015; 2(2): 165–193.

31.

Müllner

. Modern hierarchical, agglomerative clustering algorithms. arXiv. preprint arXiv:11092378. 2011.

32.

Frey

Dueck

. Clustering by passing messages between data points. Science 2007; 315(5814): 972–976.

33.

Ester

Kriegel

Sander

, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In: kdd. vol. 96; 1996. p. 226231.

34.

Rasmussen

. The Infinite Gaussian Mixture Model. In: Proceedings of the 12th International Conference on Neural Information Processing Systems. NIPS’99; 1999. p. 554560. Cambridge, MA, USA: MIT Press.

35.

Campello

RJGB

Moulavi

Sander

. Density-based clustering based on hierarchical density estimates. In: Pei

Tseng

Cao

, et al. (eds) Advances in knowledge discovery and data mining. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp.160–172.

36.

Sculley

. Web-scale k-means clustering. In: Proceedings of the 19th international conference on World wide web; 2010. p. 11771178.

37.

Béjar Alonso

. K-means vs mini batch k-means: a comparison. Universitat Politècnica de Catalunya; 2013, https://upcommons.upc.edu/handle/2117/23414 (accessed 18 February 2024).

38.

Dhillon

. Co-clustering documents and words using bipartite spectral graph partitioning. In: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining [Internet]; 2001 [cited 2024Feb 20]. p. 26974. New York, NY, USA: Association for Computing Machinery.

39.

Ferreira

Do Nascimento

Quigley

, et al. Computational complexity of edge bundling problems, Federal University of Gois; 2017, http://inf.ufg.br/biblioteca-digital (accessed 18 February 2024).

40.

Zachary

. An information flow model for conflict and fission in small groups. J Anthropol Res 1977; 33(4): 452–473.

41.

Girvan

Newman

. Community structure in social and biological networks. Proc Natl Acad Sci 2002; 99(12): 7821–7826.

42.

GroupLens. MovieLens. https://eecs.wsu.edu/∼yyao/StreamingGraphs.html (accessed 01 July 2021).

43.

Knuth

. The Stanford GraphBase: a platform for combinatorial computing, Vol. 1. New York: AcM Press, 1993

44.

Pedregosa

Varoquaux

Gramfort

, et al. Scikit-Learn: machine learning in Python. J Mach Learn Res 2011; 12: 28252830.