Cluster ensemble selection using balanced normalized mutual information

Abstract

A bad partition in an ensemble will be removed by a cluster ensemble selection framework from the final ensemble. It is the main idea in cluster ensemble selection to remove these partitions (bad partitions) from the selected ensemble. But still, it is likely that one of them contains some reliable clusters. Therefore, it may be reasonable to apply the selection phase on cluster level. To do this, a cluster evaluation metric is needed. Some of these metrics have been recently introduced; each of them has its limitations. The weak points of each method have been addressed in the paper. Subsequently, a new metric for cluster assessment has been introduced. The new measure is named Balanced Normalized Mutual Information (BNMI) criterion. It balances the deficiency of the traditional NMI-based criteria. Additionally, an innovative cluster ensemble approach has been proposed. To create the consensus partition considering the elected clusters, a set of different aggregation-functions (called also consensus-functions) have been utilized: the ones which are based upon the co-association matrix (CAM), the ones which are based on hyper graph partitioning algorithms, and the ones which are based upon intermediate space. The experimental study indicates that the state-of-the-art cluster ensemble methods are outperformed by the proposed cluster ensemble approach.

Keywords

Cluster ensembles enhanced stability extended-EAC CAM cluster evaluation

1 Introduction

Partitioning has been used in various applications such as optimization [1, 2], bioinformatics [3 –5], domain adaptation [6 –8], healthcare [9] and so on. Therefore, it is an important task; also, because of at least two reasons it is a challenging task: (1) it is unsupervised and there is no clear way to verify how good a partition is and (2) a cluster can be viewed in different perspectives (like spherical, linear, unstructured, hierarchical and so on). Therefore, it is widely considered to be more challenging than supervised classification [10 –18].

Inspired by combining classifier [9, 19] and supervised ensemble learning [20], cluster ensemble emerged being applied in various fields such as bioinformatics [21 –23], multimedia [24] and datamining [25 –33]. Clustering ensemble is a framework where it tries to use information of a number of primary partitions in producing an aggregated partition usually named consensus partition. The mentioned primary partitions, as a whole, are considered to be an ensemble of clusterings.

The cluster ensemble involves 2 steps. In the first step, it produces a number of the weak base clusterings [34 –42]. To obtain a number of weak base clusterings, a simple basic clustering algorithm, like kmeans, can be used with different initializations or different parameters [34 –42], or a simple basic clustering algorithm, like kmeans can be applied on different data perspectives [43, 44], or different data resamples [45, 46]. In the proposed method, all of these mentioned mechanisms have been used.

In the subsequent step of the cluster ensemble, a partition, called consensus partition, should be extracted from the created ensemble in the previous step such that all of them have the maximum similarity with it. The mapping which takes the ensemble as input and gives consensus partition as output is named consensus-function. Evidence accumulation clustering (EAC), as an exemplary consensus-function first illustrated by Jain and Fred, transforms the ensemble to a CAM and then applies a single linkage hierarchical clustering algorithm in order to find the consensus partition [47]. In Fig. 1, an appropriate taxonomy of various consensus-function is indicated.

Fig. 1

Different consensus-functions.

In clustering ensemble, there are two general categories of methods. The first category contains some works that concentrate on the first abovementioned step and the second category contains some other works that concentrate on the second abovementioned step, as they are two independent problems. We are among the former category. Methods of the first category are the ones that try to prepare the consensus function with a suitable ensemble of partitions. An ensemble of partitions is considered to be suitable if any of its partitions is high-quality. Therefore, some of these methods propose to produce a large number of partitions (referred to as primitive ensemble) and then, select a subset of them to be used as our final ensemble in construction of a consensus partition (this type of clustering ensemble is sometimes referred to as clustering ensemble selection). To do so, many approaches and metrics have been proposed to determine quality of a partition. It is highly likely that a good cluster that is a member of a bad partition. Therefore, it is unexpectedly removed from the ensemble by a clustering ensemble selection strategy. Therefore, some others propose to evaluate each cluster in the primitive ensemble separately. These methods are referred to as cluster ensemble selection. Some of the cluster evaluation metrics can be named as follows: (1) Normalized Mutual Information (NMI) measure [48], (2) MAX measure [49], (3) ENMI measure [38] and (4) APMM measure [50].

To evaluate a cluster, the Normalized Mutual Information measure [48] has many weaknesses that have been described in [49]. One of its different editions called MAX as well is shown that is not flawless. To improve the MAX, the APMM has been proposed [50]. All of the mentioned criteria have suffered from the complement cluster effect in some way. Therefore, this paper introduces an innovative criterion for evaluating a cluster, meanwhile it aims at solving the complement cluster effect. It estimates the averaged similarity of that cluster with the other ones through removing the effect of its complement cluster. We will deal with all these methods and their weak points and justify how our method solves their problems latter in the Section 4. We will discuss in detail what the mentioned so-called improper effect of complement cluster is and how it affects the consensus partition in the Section 4.

In the current article, an innovative cluster ensemble selection approach has been proposed. The co-association based consensus-functions are one category of the applied consensus functions. As the method of EAC, from a clusters’ subset, cannot derive the co-association matrix, therefore, via clusters subset the Extended EAC (E-EAC) is applied to form the matrix of co-association. The employed consensus-functions second category is based upon partitioning algorithms with hyper-graph. The subsequent class of employed consensus-functions looks at the selected clusters as a novel space of features and employs a simple clusterer algorithm to find the final aggregated partition.

2 Related work

The stability of a cluster has been introduced to be a method to evaluate a basic cluster quality. Since then, it has been incrementally used in the literature [51]. To validate a cluster, resampling explained in [52] and afterward generalized in [53] has been used. If a cluster has a high stability value, it demonstrates that various calls of the clusterer algorithm upon a dataset probably reproduce that cluster frequently [53].

Up to now, many metrics to evaluate a partition quality have been introduced [54]. Naldi et al. [31] have employed 6 clustering assessment measures to evaluate the clustering worth and incorporated them into a final evaluation value. Wang et al. [55] have employed rough-set concept to choose an improved sub-set of clustering members. Lu et al. [56] have introduced an innovative covariance-based diversity criterion. Akbari et al. [57] have calculated a matrix of pair-wise diversity measurement. After that, they used a clustering algorithm, which is hierarchical, to take both diversity and quality in consideration. But metrics to evaluate a cluster have recently emerged.

Some cluster assessment metrics have been introduced based on clustering stability [58 –63]. A clustering ensemble framework has been introduced in which the most stable partitions are elected [64]. The sum of NMI (S-NMI) has been widely considered to be a partition evaluator [47 , 65]. While an almost diminutive ensemble has been chosen, it has outperformed the simple ensemble methods.

Azimi introduces an algorithm that dynamically chooses a different subset of primitive partitions for the datasets with different complexity levels [66]. Extending his work, Alizadeh tried to select adaptively a subset of the primitive ensemble [67]. To handle the weakness of the traditional NMI criterion [48], they introduced MAX criterion for their cluster evaluation measure [49]. To handle a shared weakness in both MAX and NMI metrics, they introduced another metric, i.e. APMM metric [50].

Zhong et al have proposed a clustering ensemble framework that is somewhat similar to the proposed cluster ensemble framework. In [68], Zhong et al. also provide a cluster evaluating and weighting scheme, which is not based on the complement cluster. The main novelty of their work is a new cluster evaluation measure and weighting clusters correspondingly. In fact, our suggested cluster ensemble framework can be considered essentially dissimilar to their clustering ensemble framework. In contradiction of their clustering ensemble framework, our clustering ensemble framework does not require the main dataset attributes and also we do not presume any distribution in dataset. Their only disadvantage to our method is their need to original features of dataset during integration of basic partitions into the consensus partition.

Fuzzy clustering ensemble is also a new approach in clustering ensemble where the base partitions are fuzzy [34]. Although some of these works do cluster selection, they first transform fuzzy clusters into crisp clusters; therefore, indeed, they are not real fuzzy cluster ensemble [36]. While some of them make use of information of different partitions according to their quality in consensus partition generation [40], but some other works use both cluster selection and weighting [37]. As we use crisp clustering ensemble, we neglect these methods as our direct rivals.

3 Proposed cluster ensemble framework

In this part, the details of the proposed cluster-ensemble framework have been explained. The proposed cluster ensemble outline like MAX [49] and APMM [50] utilizes a cluster selection strategy. The most stable clusters are selected to be given to the consensus-function. The stability value of a cluster can be computed based on BNMI. The proposed cluster ensemble framework has been presented in Fig. 2. The proposed Cluster Ensemble Selection based on BNMI is denoted by the CESBNMI .

Fig. 2

The suggested CESBNMI cluster ensemble framework.

Based on Fig. 2, the proposed CESBNMI contains 4 steps:

Extraction of an ensemble of the size B (the i th partition in the ensemble is denoted by partition_i) is the 1st step. All of the partitions have been attained by the kmeans clustering technique with different random initializations of the seed-points. Then, the ensemble has been transformed into cluster representation. Each base partition has divided the data into a number of clusters. It means that partition_i has divided the data into k ( i ) clusters where k ( i ) is a random number in ${2, 3, 4, \dots, \sqrt{n}}$ . This step will be discussed in the subsection 4.1. It means ${partition}_{i} = {c_{1}^{i}, c_{2}^{i}, \dots, c_{k (i)}^{i}}$ where $c_{j}^{i}$ is the jth cluster of the ith partition.

A stability value has been assigned to each basic cluster obtained in previous step. The stability for the cluster $c_{j}^{i}$ (the jth cluster of the ith partition) is denoted by ${stab}_{j}^{i}$ . The stability computation for a basic cluster will be discussed in the subsection 4.2.

The 3rd step will be discussed at the subsection 4.3. The selection should be accomplished in such a way that the most fixed clusters are selected to participate in the consensus-function.

The last step, i.e. the consensus-function step, is done using a method that is among one of the following categories: (1) CAM-based consensus-functions (section 4.4), (2) HyperGraph consensus-functions, and (3) intermediate-based consensus-functions (it views the space of clusters as a novel data space and applies an algorithm to cluster the data projected in the new space). This step will be discussed at the subsection 4.4.

3.1 Production of primary clusters

A set of B partitions are first produced via using the algorithm of kmeans clustering. Every kmeans clustering algorithm partitions data in a random number of clusters. The number of clusters in every kmeans clustering algorithm is selected randomly from the set ${2, 3, \dots, \sqrt{n}}$ where n is the data points number; It is a widely accepted approach [27 , 38]. The base clustering algorithm is always the kmeans clustering algorithm. Let’s denote the j th cluster of the i th partition by $c_{j}^{i}$ . It means that the i th partition is defined as ${partition}_{i} = {c_{1}^{i}, c_{2}^{i}, \dots, c_{k (i)}^{i}}$ where $c_{j}^{i} \subset {1, 2, \dots, n}$ (n is the number of data points) and $\forall p, r \in {1, 2, \dots, k (i)} : c_{p}^{i} \cap c_{r}^{i} = \emptyset$ .

3.2 Cluster stability

Different definitions have been presented for the stability [49 , 54]. It has been defined as a cluster that almost always appears in a clustering. It means that if an algorithm is run to partition a data 10 times and a set of data points emerges as a cluster about more than 5 times, the cluster can be named as a stable cluster. These clusters are favorable to be selected in the final consensus-function [68]. At this time, suppose that the cluster stability $c_{j}^{i}$ is to be calculated. To carry out that, over the given dataset, partitions are generated first. The mentioned set of partitions is named the Reference Set (RS). The RS size (i.e. partitions’ number generated as the RS) can be small to reduce the algorithm runtime. In our symbols, D presents dataset and RS^q presents qth partition over dataset D in the RS. At this moment, we want to know the times that cluster $c_{j}^{i}$ emerged in the RS. We first need the definition of NMI. The NMI (partition¹, partition²) stands for the normalized mutual information between two partitions partition¹ and partition² can be computed as Equation. 1 [48]:

$\begin{matrix} NMI ({partition}^{1}, {partition}^{2}) \\ = \frac{- 2 \sum_{i - 1}^{k_{1}} \sum_{j - 1}^{k_{2}} | c_{i}^{1} \cap c_{j}^{1} | \times log (\frac{| c_{i}^{1} \cap c_{j}^{1} | \times n}{| c_{i}^{1} | \times | c_{j}^{1} |})}{\sum_{i = 1}^{k_{1}} | c_{i^{1}} | \times log (\frac{c_{i}^{1}}{n}) + \sum_{i - 1}^{k_{2}} | c_{i^{2}} | \times log (\frac{c_{i^{2}}}{n})} \end{matrix}$ (1)

where k ₁ and k ₂ are the clusters’ numbers in the partitions partition¹ and partition² respectively, and | . | is the size of a set (for example $| c_{i}^{1} |$ means the size of i th cluster in the partition partition¹). Note that throughout the paper 0 × log 0 is considered to be 0. As you can see, the NMI is defined between two partitions [65], while we intend to extend it. We want to redefine it in such a way that it can be applied between a cluster and a partition. Therefore, we define the $NMI (c_{j}^{i}, {RS}^{q})$ as the normalized mutual information between a cluster $c_{j}^{i}$ and a partition RS^q. It shows the amount of existence of the cluster $c_{j}^{i}$ in the partition RS^q. It is defined based on Equation 2 [69]:

$\begin{matrix} NMI (c_{j}^{i}, {RS}^{q}) = NMI ({c_{j}^{i}, {\bar{c}}_{j}^{i}}, {RS}^{q}) \\ = \frac{- 2 \sum_{r = 1}^{| {RS}^{q} |} | c_{j}^{i} \cap {RS}_{r}^{q} | \times log (\frac{| c_{j}^{i} \cap {RS}_{r}^{q} | \times n}{| c_{j}^{i} | \times | {RS}_{r}^{q} |}) - 2 \sum_{r = 1}^{| {RS}^{q} |} | c_{j}^{i} \cap {RS}_{r^{q}} | \times log (\frac{| {c_{j}}^{\bar{1}} \cap {RS}_{r^{q}} | \times n}{| {c_{j}}^{\bar{1}} | \times | {RS}_{r^{q}} |})}{| c_{j}^{i} | \times log (\frac{| c_{j}^{i} |}{n}) + | {c_{j}}^{\bar{1}} | \times log (\frac{| {c_{j}}^{\bar{1}} |}{n}) + \sum_{r = 1}^{| {RS}^{q} |} | {RS}_{r^{q}} | \times log (\frac{| {RS}_{r^{q}} |}{n})} \end{matrix}$ (2) where | . | is the size of a set (for example |RS^q| means the size of RS^q or the number of clusters in q th member of the RS ), $\bar{c_{j}^{i}}$ is the complement of cluster $c_{j}^{i}$ . But this method is not good as the |RS^q| is not equal to 2 (it may be very larger). Therefore, the NMI-Cluster evaluator has been introduced as follows [69]. The $R (c_{j}^{i}, {RS}^{q}, 1)$ is defined as a cluster as follows:

$R (c_{j}^{i}, {RS}^{q}, 1) = \underset{t \in {r \in {1, 2, \dots, | {RS}^{q} |} | \frac{| {RS}_{r}^{q} \cap c_{j}^{i} |}{| {RS}_{r}^{q} |} > 0.5}}{\cup} {RS}_{t}^{q}$ (3)

The R ( c _jⁱ, RS^q , 2) is defined as the complement cluster of the cluster R ( c_^ji , RS ^q, 1) (it is worthy to be mentioned that complement of a cluster 1 is defined as a cluster 2 containing all data points that are not in cluster 1). Now $NMI (c_{j}^{i}, {RS}^{q})$ is defined as follows [69]:

$\begin{matrix} NMI (c_{j}^{i}, {RS}^{q}) \\ = NMI ({c_{j}^{i}, \bar{c_{j}^{i}}}, {R (c_{j}^{i}, {RS}^{q}, 1), R (c_{j}^{i}, {RS}^{q}, 2)}) \end{matrix}$ (4)

The above method has been described by a toy example depicted in Fig. 3. Figure 3 depicts this method which is also called the NMI-Cluster evaluation method. The partition in the Fig. 3 (a) depicts a clustering in the ensemble. The partition in the Fig. 3 (c) depicts a clustering in the reference set. We want to evaluate the cluster $c_{1}^{i}$ of the partition depicted in the Fig. 3 (a) with regard to the reference partition depicted in the partition in the Fig. 3 (c). Based on Equation 2, we first transform the partition depicted in the Fig. 3 (a) into the partition depicted in the Fig. 3 (b). After that, based on Equation 3, we transform the partition depicted in the Fig. 3 (c) into the partition depicted in the Fig. 3 (d).

Fig. 3

The process of NMI value computation for the cluster $c_{1}^{i}$ of the partition presented in Figure 3 (a) considering reference clustering presented in Figure 3 (c), when the NMI-Cluster evaluation method is used. (a) Our target partition. (b) Defining ${c_{1}^{i}, \bar{c_{1}^{i}}}$ . (c) Our target RS^q. (d) Determining ${R (c_{1}^{i}, {RS}^{q}, 1), R (c_{1}^{i}, {RS}^{q}, 2)}$ .

Then MAX-Cluster evaluator has been introduced as follows [70]. The R ( c_j ⁱ, RS ^q, 1) is defined as a cluster as follows: $R (c_{j}^{i}, {RS}^{q}, 1) = {RS}_{f}^{q}$ (5) where f is an element of { r ∈ { 1, 2, …, | RS ^q| } | ∀ f : { 1, 2, …, | RS ^q| } - { r } • | RS _r^q| ⩾ | RS _f^q| }. The R ( c _{j

ⁱ}, RS ^q, 2) is defined as the complement cluster of the cluster $R (c_{j}^{i}, {RS}^{q}, 1)$ . Now, $NMI (c_{j}^{i}, {RS}^{q})$ is defined based Equation 4 [69]. Figure 4 depicts this method which is also called the MAX-Cluster evaluation method.

Fig. 4

The process of NMI value computation for the cluster $c_{1}^{i}$ of the partition presented in Figure 4 (a) considering reference clustering presented in Figure 4 (c), when the MAX-Cluster evaluation method is used. (a) Our target partition. (b) Defining ${c_{1}^{i}, \bar{c_{1}^{i}}}$ . (c) Our target RS^q. (d) Determining ${R (c_{1}^{i}, {RS}^{q}, 1), R (c_{1}^{i}, {RS}^{q}, 2)}$ .

After introducing the MAX-Cluster evaluator, Alizadeh et al. have introduced the APMM-Cluster evaluator to solve the weak points of the previous criteria. It has been defined as follows [70]. The $R (c_{j}^{i}, {RS}^{q}, v)$ is defined as a cluster as follows: $R (c_{j}^{i}, {RS}^{q}, v) = {RS}_{v}^{q} \cap c_{j}^{i}$ (6) where v is an element of {1, 2, … , |RS^q| }. Now $NMI (c_{j}^{i}, {RS}^{q})$ is defined based Equation 7 [69].

$\begin{matrix} NMI (c_{j}^{i}, {RS}^{q}) = NMI ({c_{j}^{i}}, {R (c_{j}^{i}, {RS}^{q}, 1), \\ R (c_{j}^{i}, {RS}^{q}, 2), \dots, R (c_{j}^{i}, {RS}^{q}, | {RS}^{q} |)}) \end{matrix}$ (7)

Figure 5 depicts this method which is also called the APMM-Cluster evaluation method.

Fig. 5

The process of NMI value computation for the cluster $c_{1}^{i}$ of the partition presented in Figure 5 (a) considering reference clustering presented in Figure 5 (c), when the APMM-Cluster evaluation method is used. (a) Our target partition. (b) Defining ${c_{1}^{i}}$ . (c) Our target RS^q. (d) Determining ${R (c_{1}^{i}, {RS}^{q}, 1), R (c_{1}^{i}, {RS}^{q}, 2), R (c_{1}^{i}, {RS}^{q}, 3), R (c_{1}^{i}, {RS}^{q}, 4)}$ .

After introducing the APMM-Cluster evaluator, this paper has introduced the BNMI-Cluster evaluator to solve the weak points shared between all of the previous criteria. The BNMI-Cluster evaluator has been defined as follows. The $R (c_{j}^{i}, {RS}^{q}, v)$ is defined as a cluster as follows: $R (c_{j}^{i}, {RS}^{q}, v) = {\begin{matrix} {RS}_{v}^{q} & {RS}_{v}^{q} \cap c_{j}^{i} \neq \emptyset \\ \emptyset & {RS}_{v}^{q} \cap c_{j}^{i} = \emptyset \end{matrix}$ (8) where v is an element of {1, 2, … , | RS ^q| }. Therefore, partition R has at most | RS ^q| clusters. The $Z (c_{j}^{i}, {RS}^{q}, v)$ is defined as a cluster according to Equation 9.

$Z (c_{j}^{i}, {RS}^{q}, ϑ) = \underset{v \in {1, 2, \dots, | {RS}^{q} |}}{\cup} R (c_{j}^{i}, {RS}^{q}, v) \cap c_{ϑ}^{i}$ (9)

where ϑ is an element of {1, 2, … , k ( i ) }. Note that partition Z also has at most min(| RS ^q|, k ( i )) clusters. Now, $NMI (c_{j}^{i}, {RS}^{q})$ can be defined based Equation 10 [69].

$\begin{matrix} NMI (c_{j}^{i}, {RS}^{q}) \\ = NMI (v {Z (c_{j}^{i}, {RS}^{q}, 1), \dots, Z (c_{j}^{i}, {RS}^{q}, k (i))}, \\ {R (c_{j}^{i}, {RS}^{q}, 1), \dots, R (c_{j}^{i}, {RS}^{q}, | {RS}^{q} |)}) \end{matrix}$ (10)

Figure 6 depicts this method which is also called the BNMI-Cluster evaluation method. After defining a NMI criterion for computing the NMI between a partition and a cluster, we define stability of a cluster $c_{j}^{i}$ as the average NMI between the cluster $c_{j}^{i}$ and all partitions of the RS. The stability of the cluster $c_{j}^{i}$ is denoted by $Stab (c_{j}^{i}, RF)$ or briefly ${Stab}_{j}^{i}$ and is defined according to Equation 11 as follows. $Stab (c_{j}^{i}, RF) = {Stab}_{j}^{i} = \frac{\sum_{q = 1}^{| RS |} NMI (c_{j}^{i}, {RS}^{q})}{| RS |}$ (11)

Fig. 6

The process of NMI value computation for the cluster $c_{1}^{i}$ of the partition presented in Figure 6 (a) considering reference clustering presented in Figure 6 (c), when the BNMI-Cluster evaluation method is used. (a) Our target partition. (b) Defining ${Z (c_{1}^{i}, {RS}^{q}, 1), Z (c_{1}^{i}, {RS}^{q}, 2), Z (c_{1}^{i}, {RS}^{q}, 3), Z (c_{1}^{i}, {RS}^{q}, 4)}$ . (c) Our target RS^q. (d) Determining ${R (c_{1}^{i}, {RS}^{q}, 1), R (c_{1}^{i}, {RS}^{q}, 2), R (c_{1}^{i}, {RS}^{q}, 3), R (c_{1}^{i}, {RS}^{q}, 4)}$ .

Consider a dataset with n = 1000 data samples. Let’s assume we have an ensemble of 100 primary partitions. In 50 primary partitions, we have a cluster {1, 2, 3} and another cluster {4, 5, 6}. In other 50 primary partitions, we have a cluster {1, 2, 4} and another cluster {3, 5, 6}. Let’s assume we use the main ensemble, i.e. 100 primary partitions, as our reference set. We want to calculate stability of the clusters {1, 2, 3}. The NMI cluster evaluator says the stability for the cluster {1, 2, 3} is 0.7597. Let’s reconsider the same example with one difference when dataset contains n = 10000 data samples. The NMI cluster evaluator says the stability for the cluster {1, 2, 3} is 0.7783! This is called the effect of complement cluster. This can be severe in real examples where the number of clusters are large. This is a little less severe in MAX cluster evaluator; but it still exists. MAX cluster evaluator also suffers from difficulty in managing outliers. The effect of complement cluster is a little less severe in ENMI cluster evaluator; but it still exists. Although the effect of complement cluster is more or less solved by APMM cluster evaluator, it has caused another problem. It totally ignores the shape of clusters overlapping with the target cluster. For example, if two of more reliable clusters divide our target cluster equally, it is worse than if two (or more) two of unreliable clusters divide it equally. While it is solved by BNMI cluster evaluator.

3.3 Cluster selection

The third step is accomplished as follows. The clusters are first sorted based on their stabilities. Then, the top 50% of the most stable clusters have been selected to participate in the final ensemble.

3.4 Consensus-function

The phase that collects the chosen clusters to the consensus or final partition is the last section of clustering ensemble. One favourable strategy is to determine the chosen clusters as inputs of the CSPA, MCLA, and HGPA [65]. The abovementioned algorithms’ output is the resultant partition of the ensemble. For clarification, consider Fig. 7. Four partitions, partition ₂, partition ₃ and partition ₄ are attained by applying on a simple imaginary dataset containing twelve instances and 2 clusters. Every partition is attained by an clusterer algorithm (for example k-means clustering), with various values for the parameter of k . The parameter k in the kmeans clustering algorithm has been chosen as three, four, two and two respectively in partitions partition ₁, partition ₂, partition ₃ and partition ₄. These partitions have been transformed into eleven clusters shown in Fig. 8. Now these clusters can be considered to be the inputs for the CSPA, HGPA and MCLA.

Fig. 7

Four partitions partition₁, partition₂, partition₃ and partition₄ derived from an asssuptive toy dataset by kmeans.

Fig. 8

The binary representation of the clusters extracted from partitions presented in Figure 7. For example $c_{3}^{1} = {5, 6}$ .

As another popular technique to produce the consensus partition using the selected clusters, the clusters can be viewed as a data new perspective, and a clusterer, like fuzzy c-means (FCM) clustering algorithm, can be used to cluster the new data. For example, again consider the example of Fig. 7. The partitions are divided to 11 clusters illustrated in the Fig. 8 as before. Then, the clusters of Fig. 8 are determined as the new data in a novel feature space and a FCM clustering algorithm is applied to generate the final ensemble partition from them.

Thus, three methods are employed to generate final clusters from the chosen clusters: (a) HyperGraph Based Models, (b) Simple Clustering Based Models, and (c) Co-Association Based Model. Firstly, it is important to build the CAM and then to apply a hierarchical clusterer. E-EAC is used to build the CAM from the chosen clusters. In the method of EAC, the B primitive partitions are accumulated in an n × n CAM. In accordance with Equation 12, each entry in that matrix is calculated. ${CAM}_{ij} = \frac{n_{ij}}{m_{ij}}$ (12) where n_ij will be number of the partitions that encompass ith and jth instances simultaneously. Also, m_ij will be number of the partitions that encompass the ith or jth instances. Due to existence of a fraction of base clusters in the proposed CESBNMI , we should use the E-EAC for computation of the CAM. In the E-EAC, CAM_ij is computed according to Equation 13. ${CAM}_{ij} = \frac{n_{ij}}{n_{i} + n_{j}}$ (13) where n_i will be the number of partitions that encompass ith instance. This paper proposes using the following EEAC-based CAM. Here CAM_ij is computed according to Equation 14.

$\begin{matrix} {CAM}_{ij} \\ = \frac{\sum_{c_{j}^{i} \in Selected Clusters} IsTrue (x_{i} \in c_{j}^{i} \land x_{j} \in c_{j}^{i})}{\sum_{c_{j}^{i} \in SelectedClusters} IsTrue (x_{i} \in c_{j}^{i} \lor x_{j} \in c_{j}^{i})} \end{matrix}$ (14)

where Selected Clusters is all clusters selected in the previous step and Is True (condition) will return one if condition is true; otherwise it returns zero. This method is named as ItoU-based CAM (ItoU stands for Intersection-to-Union). Finally, the consensus partition can be obtained by a hierarchical clusterer algorithm. In another way, we can serve these elected clusters as the input of the algorithms which are hyper-graph partitioning like MCLA, CSPA, and HGPA [65]. The third method for extraction of the consensus partition can be accomplished through applying a simple clusterer algorithm on the data with the elected clusters as their features. Indeed, the elected clusters are considered to be an intermediate space for dataset (for example, we consider Fig. 8 as a dataset and apply a simple clusterer algorithm on it).

As this is mentioned before, the last step is accomplished based on one of the techniques presented in the Fig. 9. Therefore, there are 50 different clustering ensemble approaches in this paper. The mentioned different clustering ensemble approaches are presented in Fig. 10.

Fig. 9

Different consensus-functions used in the paper.

Fig. 10

Different methods for clustering ensemble.

4 Experimental results

In the current section, the evaluation metrics to assess a clustering output have been introduced at the first sub-section. The benchmark datasets have been introduced in the next sub-section. The initialization of the algorithms’ parameters has been discussed in the third sub-section. The experimental results have been presented in the last sub-section of this section.

4.1 Assessment metrics

Having generated the consensus clustering, an important problem is its assessment approach. The clustering assessment problem is very crucial in spite of classification. The NMI can be largely considered to be one of the most well-known clustering assessment criteria. The output/final clustering is the one which is attained via using any approach of clustering upon the dataset. It is clarified that upon the dataset, after using a clusterer method, the attained partition is the output clustering/partition for the abovementioned clusterer method. After clusterer method termination, we are able to employ the dataset real labels to assess how suitable clusterer method performed the assignment of clustering. Between two clusterings, partition¹ and partition², the NMI is computed based upon Equation 1.

Accuracy metric, is another alternative to assess a partition if the clusters’ number and their actual labels are clear. To calculate the final clustering algorithm performance regarding to accuracy, first one should re-label the attained clusters in that kind of strategy that have maximum correspondence with the ground real labels and after that calculating the accurately classified samples percentage. Therefore, after solving the problem of correspondence, the rate of error can be considered between the known clusters and derived labels. To solve the matching problem with minimum weight bipartite, the Hungarian algorithm is used. It has been indicated that it is able to solve the label correspondence problem effectively [71].

The last applied measure to assess a clustering algorithm is F-measure (F-score) in this paper. The output clustering and dataset real labels are employed in computation formula of F-measure, to calculate the clustering algorithm final performance regarding to F-measure, after the clustering algorithm termination. The F-measure is calculated by applying both recall measures and the precision. As a weighted average of the recall and precision, the F-measure can be explained, where its worst score is at 0 and its best value at 1 [72]. In this study, based upon Equation 15, the F-measure formula applied.

$\begin{matrix} FM ({partition}^{1}, {partition}^{2}) \\ = \max_{ϕ} \sum_{i = 1}^{k_{1}} \frac{2 \times | c_{i}^{1} | \times (\frac{| c_{i}^{1} \cap c_{ϕ (i)}^{2} |}{| c_{i}^{1} |} \times \frac{| c_{i}^{1} \cap c_{ϕ (i)}^{2} |}{| c_{ϕ (i)}^{2} |})}{n \times (\frac{| c_{i}^{1} \cap c_{ϕ (i)}^{2} |}{| c_{i}^{1} |} + \frac{| c_{i}^{1} \cap c_{ϕ (i)}^{2} |}{| c_{ϕ (i)}^{2} |})} \end{matrix}$ (15) where a permutation of {1, … , k ₂ } is shown by ϕ for re-labeling of partition². It is worth mentioning that in all experiments, the accuracy, F-measure and the NMI, (for convention) are measured between [0–100]. Therefore, they are always recorded in the range [0–100].

4.2 Benchmarks

One of the serious problems in artificial intelligence communities is the selection of a suitable benchmark. The datasets of the UCI machine learning data repository have been widely used for clustering problems [73]. Therefore, the authors have also used some of them into this manuscript because they can be considered to be suitable benchmark datasets for clustering problems.

Experimentations have been done on 23 typical datasets. Two of the used datasets are huge ones (datasets with huge number of records, i.e. (a) MNIST dataset and (b) MNIST 1vs2 dataset where it contains 2000 samples of class one and 2000 samples of class two randomly chosen from MNIST dataset), one of them is a high dimensional spare one, i.e. Lymphoma dataset, one of them is a high-dimensional and huge one, i.e. MNIST dataset, some of them have high number of clusters, some of them have low number of clusters, some of them are challenging datasets to the clustering problem; therefore, the actual numbers of clusters, samples and features in the employed datasets are diverse enough to make fair conclusions from the experimental results of the paper. Table 1 provides some information about the employed datasets [73].

Table 1
Fundamental characteristics of the employed benchmarks.

# Dataset Name # of Clusters # of Features # of Patterns Cluster Distribution Source

1 Wine (W) 3 13 178 71-59-48 UCI [73]

2 Breast-Cancer (BC) 2 9 683 444-239 UCI [73]

3 Bupa (B) 2 6 345 200-145 UCI [73]

4 Ionosphere (Is) 2 34 351 225-126 UCI [73]

5 Iris (I) 3 4 150 50-50-50 UCI [73]

6 Glass (G) 6 9 214 76-70-29-17-13-9 UCI [73]

7 WDBC (WD) 2 30 569 357-212 UCI [73]

8 Yeast (Y) 10 8 1484 463-429-244-163-51-44-35-30-20-5 UCI [73]

9 Galaxy (Gy) 7 4 323 80-51-46-45-38-35-28 UCI [73]

10 SAHeart (SH) 2 9 462 302-160 UCI [73]

11 Image (Im) 15 16 640 80-64-64-64-48-48-48-32-32-32-32-32-32-16-16 UCI [73]

12 Lymphoma (L) 9 4026 96 46-11-10-9-6-6-4-2-2 UCI [73]

13 OQ (OQ) 2 16 1536 783-753 UCI [73]

14 Pima (P) 2 8 768 444-239 UCI [73]

15 Sonar (S) 2 60 208 111-97 UCI [73]

16 MNIST 1vs2 (1vs2) 2 784 4000 2000-2000 [74]

17 MNIST (MN) 10 784 60000 6000-6000-6000-6000-6000-6000-6000-6000-6000-6000 [74]

18 2-Spiral (2S) 2 2 250 125-125 Hand-made

19 Aggregation (A) 7 2 788 273-170-130-102-45-34-34 Hand-made

20 Flame (F) 3 2 240 148-90-2 Hand-made

21 3-Spiral (3S) 3 2 291 97-97-97 Hand-made

22 Open Flame (OF) 2 2 240 150-90 Hand-made

23 Halfring (H) 2 2 400 300-100 Hand-made

#	Dataset Name	# of Clusters	# of Features	# of Patterns	Cluster Distribution	Source
1	Wine (W)	3	13	178	71-59-48	UCI [73]
2	Breast-Cancer (BC)	2	9	683	444-239	UCI [73]
3	Bupa (B)	2	6	345	200-145	UCI [73]
4	Ionosphere (Is)	2	34	351	225-126	UCI [73]
5	Iris (I)	3	4	150	50-50-50	UCI [73]
6	Glass (G)	6	9	214	76-70-29-17-13-9	UCI [73]
7	WDBC (WD)	2	30	569	357-212	UCI [73]
8	Yeast (Y)	10	8	1484	463-429-244-163-51-44-35-30-20-5	UCI [73]
9	Galaxy (Gy)	7	4	323	80-51-46-45-38-35-28	UCI [73]
10	SAHeart (SH)	2	9	462	302-160	UCI [73]
11	Image (Im)	15	16	640	80-64-64-64-48-48-48-32-32-32-32-32-32-16-16	UCI [73]
12	Lymphoma (L)	9	4026	96	46-11-10-9-6-6-4-2-2	UCI [73]
13	OQ (OQ)	2	16	1536	783-753	UCI [73]
14	Pima (P)	2	8	768	444-239	UCI [73]
15	Sonar (S)	2	60	208	111-97	UCI [73]
16	MNIST 1vs2 (1vs2)	2	784	4000	2000-2000	[74]
17	MNIST (MN)	10	784	60000	6000-6000-6000-6000-6000-6000-6000-6000-6000-6000	[74]
18	2-Spiral (2S)	2	2	250	125-125	Hand-made
19	Aggregation (A)	7	2	788	273-170-130-102-45-34-34	Hand-made
20	Flame (F)	3	2	240	148-90-2	Hand-made
21	3-Spiral (3S)	3	2	291	97-97-97	Hand-made
22	Open Flame (OF)	2	2	240	150-90	Hand-made
23	Halfring (H)	2	2	400	300-100	Hand-made

All of the experimentations have been conducted on the standardized versions of datasets, i.e. the version whose features are linearly transformed in such a way that have the averages equal to 0 and the standard deviations equal to 1. The hand-made datasets used in the paper can be considered to be the most challenging datasets for clustering task. They have been depicted in Fig. 11.

Fig. 11

(a) Halfring dataset, (b) 2-Spiral dataset, (c) Aggregation dataset, (d) Flame dataset, (e) 3-Spiral dataset, (f) Open Flame dataset.

4.3 Experimental settings

For making the results more conclusive, the experimentations, have been reported as averaged on 100 independent performance calculations. Inside all experiments, there are 30 independent basic clusterings attained by 30 k-means clustering algorithm independent runs with various initializations (for seed-points) and various random values as the number of clusters in the range 2 to $\sqrt{n}$ where n is the data points number. It means that the size of ensemble is always 30.

After selecting a sub-set of high quality clusters, to extract the consensus clustering, the different consensus-functions employ the number of the real clusters in the data set, i.e. the third column in Table 1, as their reference for the number of the consensus clusters.

As it is known, the soft clustering algorithm (i.e. FCM here) appoints every object toward all clusters with various values of membership. To determine the consensus clustering from an FCM clustering algorithm output as consensus-function, every object is appointed toward the cluster with the most value of membership.

4.4 Experimental results

The value of NMI regarding to using various ratios, from the steadiest clusters, in final ensemble is explained in Fig. 12-a. The results are over Wine dataset. Since it is clear that from Fig. 12, the value of NMI enhances as ratio of the most stable clusters which participate in the final ensemble enhances (before 50%). But, it after 50%, does not enhance, or even in some cases, it declines.

Fig. 12

(a) The stable clusters rate that are chosen applying the method of BNMI is presented by the horizontal axis. The value of NMI for Wine dataset is presented by the vertical axis. (b) The stable clusters rate that are chosen applying the method of BNMI is presented by the horizontal axis. The averaged value of NMI over all used datasets in Table 1 is illustrated by the vertical axis. (c) The stable clusters rate that are chosen employing the method of BNMI is presented by horizontal axis. The averaged accuracy over all applied datasets in Table 1 is illustrated by the vertical axis (d) The stable clusters rate that are chosen applying the method of BNMI is indicated by the horizontal axis. The averaged F-measure of all applied datasets in Table 1, is shown by the vertical axis (e) The stable clusters rate that are chosen applying the method of NMI, is illustrated by the horizontal axis. The averaged accuracy of all applied datasets in Table 1, is presented by the vertical axis. (f) The stable clusters rate that are chosen applying the method of ENMI is presented by the horizontal axis. The averaged accuracy of all employed datasets in Table 1, is indicated by the vertical axis. (g) The stable clusters rate that are chosen applying the method of APMM, is illustrated by the horizontal axis. The averaged accuracy of all applied datasets in Table 1, is presented by the vertical axis. (h) The stable clusters rate that are selected using the method of MAX is illustrated by the horizontal axis. The averaged accuracy of all applied datasets in Table 1 is presented by the vertical axis.

The value of NMI concerning to using various ratios from the steadiest clusters in final ensemble is generated by employing further datasets. It indicates that for any of the other datasets with the same experiment like the one for wine dataset (represented in Fig. 12.a) is generated. It is resulted in approximately the same consequences for all cases (datasets); in the final ensemble, the value of NMI enhances as ratio of the steadiest clusters which participate in the final ensemble enhances up to 50%. Whereas it, after 50%, declines in all cases. But inside the Yeast dataset case, in the final ensemble, the value of NMI enhances as ratio of the steadiest clusters which participate in the final ensemble enhances up to 30% and it, after 30%, declines. By the way, in Yeast dataset, the ensemble with 50% of the most stable clusters still is better than the ensemble with all clusters.

The averaged value of NMI concerning to applying various ratios from the steadiest clusters in final ensemble of all employed datasets is illustrated in Fig. 12-b. It is obvious that, in the final ensemble, the value of NMI enhances as ratio of the steadiest clusters which participate in the final ensemble enhances up to 50% in all cases. But it after 50%, does not enhance, and even declines in a number of methods. Nevertheless, the supposed technique when 50% of the steadiest clusters are taken into final ensemble, hits its maximum via using the algorithm named Average-Linkage as consensus-function with the method of EEAC when the equation of ItoU is used to construct the matrix of co-association.

The averaged value of accuracy regarding to using various ratios from the steadiest clusters in last ensemble of all used datasets is illustrated in Fig. 12-c. The accuracy rates presented in Fig. 12-c confirm the NMI rates presented in Fig. 12-b. It is obvious that the value of accuracy enhances as ratio of the steadiest clusters which participate in the final ensemble enhances up to 50% in nearly in all cases. Therefore, the same 50% of the steadiest clusters still becomes the best value. The same algorithm of Average-Linkage as consensus-function with the method of EEAC, whenever the method of ItoU is used, construct co-association matrix which is the best option. In terms of various ratios of the steadiest clusters in final ensemble, the averaged F-measure over all employed datasets is illustrated by Fig. 12.d. The conclusion is the same as Fig. 12.c and Fig. 12.b for the F-measure results presented in Fig. 12.d.

The same experimentations of Fig. 12.c in terms of averaged accuracy are repeated by NMI [48] criterion as cluster evaluator. It clarifies that the averaged value of accuracy (averaged over all of the used datasets) regarding to using various ratios from the steadiest clusters into the decisive ensemble are presented in Fig. 12.e. In Fig. 12.e, no longer, the measure of stability is used for BNMI. It uses the measure of NMI to estimate a cluster stability. The result of Fig. 12.e again approves that the aggregator of Average-Linkage is better than all aggregators when the equation of ItoU is used for constructing co-association matrix. It is worth mentioning that the best option for ratio of the steadiest clusters which participate in the final ensemble is still 50%.

The same experimentations of Fig. 12.c and Fig. 12.e are repeated by ENMI [38], MAX [49] and APMM [50] criteria as cluster evaluator in terms of averaged accuracy. It indicates that the averaged value of accuracy (averaged over all of the used datasets) regarding to using various ratios from the steadiest clusters in last decisive ensemble are reported in Fig. 12.f, Fig. 12.g and Fig. 12.h respectively when MAX [49], ENMI [38], and APMM [50] criteria are used as cluster evaluator criterion. In Fig. 12.f, Fig. 12.g and Fig. 12.h, the stability measure employed is no longer BNMI. They use ENMI [38], MAX [49] and APMM [50] criteria respectively to estimate a cluster stability. The results presented in Fig. 12.h, Fig. 12.f and Fig. 12.g, again confirm that the aggregator of Average-Linkage is better toward all aggregators whenever the equation of ItoU is used to construct the matrix of co-association. It is worth mentioning that the best option for ratio of the steadiest clusters which participate in the final ensemble is still 50%.

While comparing the results presented by Fig. 12.h, Fig. 12.f, Fig. 12.e, Fig. 12.g and Fig, 12.c, it can be concluded that the BNMI usage as a measure to assess a cluster in our clustering ensemble framework, causes a superior final ensemble partition instead of other measures. It can be also inferred that employing the algorithm of Average-Linkage as aggregator with the method of EEAC, for consensus-function is the best option. Additionally, it is inferred that the consequences will be superior whenever the technique of ItoU is used to make co-association matrix.

Generally, for making a decisive comparison among the five used cluster evaluation criteria, i.e. NMI, MAX, ENMI, APMM, and BNMI, consider Table 2. In order to attain the consequences displayed in Table 2, we use the Average-Linkage algorithm as aggregator along with EEAC method and employ ItoU technique for making the matrix of co-association. It is obvious, the BNMI, APMM, ENMI and MAX are completely better than NMI. In addition to the BNMI based cluster ensemble almost outperforms the ENMI, APMM and MAX based clustering ensembles. Therefore, the study infers that employing BNMI as the evaluator of cluster is the best option. Friedman’s ANOVA test on Table 2 is depicted by Fig. 13. The p-value is 1.3e^∧-27 that indicates a significant difference.

Table 2

Accuracy of clusterings generated by cluster-selection employing the evaluator of NMI-based cluster, the ENMI-based cluster evaluator, the APMM-based cluster evaluator, and the MAX-based cluster evaluator verified via paired t-test [78] with 95% p-value

Evaluation Measure	Dataset Number																							t-test results win-draw-loss
	W	BC	B	Is	I	G	WD	Y	Gy	SH	H	Im	L	OQ	P	S	1vs2	MN	2S	A	F	3S	OF
Adaptive [57]
	95.43-	88.00-	54.73-	63.42-	71.51±	44.39-	88.54-	29.69-	48.52-	70.73-	74.50-	60.86-	66.66-	69.18-	65.08-	54.82-	59.33-	48.57-	50.00±	79.09-	66.42-	33.97-	82.50-	0-2-21
EAC (Full Ens.)
	95.17-	89.33±	54.49-	63.20-	70.66-	46.26±	85.63-	30.96±	44.21-	70.22-	74.50-	60.55-	66.69-	69.04-	65.14-	55.18-	58.52-	49.45-	50.00±	79.16-	65.59-	33.97-	82.92-	0-4-19
Azimi [66]
	96.91±	89.33±	54.75-	56.06-	70.74±	45.05-	83.46-	29.97-	43.40-	60.95-	67.70-	61.82-	65.37-	68.86-	65.22-	54.69-	59.38-	49.96-	50.00±	79.07-	66.77-	33.97-	82.37-	0-4-19
NMI [48]
	95.73-	76.13-	54.33-	63.36-	70.60-	47.76+	87.89-	31.27±	42.93-	69.38-	74.48-	62.37-	68.55-	69.33-	65.79-	58.23-	67.84-	53.96-	50.00±	84.71-	67.57-	34.12±	83.06-	1-3-19
MAX [49]
	96.49±	84.87-	57.42±	63.87-	57.75-	44.35-	88.12-	29.85-	51.27-	70.00-	74.55-	62.97-	68.73-	69.38-	66.11-	58.15-	68.22-	54.57-	50.00±	85.01-	68.01-	34.11±	83.01-	0-4-19
APMM [50]
	95.46-	90.00±	55.07-	63.85-	70.66-	45.79±	88.09-	30.65±	53.10+	70.23-	54.00-	62.78-	68.85-	69.81-	66.13-	58.07-	67.78-	55.08-	50.00±	85.61-	68.15-	34.15±	83.23-	1-5-17
ENMI [38]
	96.93±	89.67±	55.78-	63.20-	71.73±	45.05-	88.17-	30.65±	52.47-	71.23±	88.00±	63.03-	68.97-	69.76-	66.41-	58.27-	69.57-	54.89-	50.00±	87.07-	68.21-	34.11±	83.28-	0-8-15
NMI [48] + ItoU
	96.02-	88.67-	54.78-	63.42-	70.09-	44.86-	88.05-	29.85-	51.86-	70.22-	74.50-	63.21-	68.67-	69.72-	66.01-	58.18-	68.18-	56.22-	50.00±	86.33-	67.92-	34.17±	83.13-	0-2-21
MAX [49] + ItoU
	96.93±	90.00±	54.78-	64.50-	71.51±	44.86-	88.52-	29.41-	51.45-	71.35±	87.25-	63.28-	69.07-	69.75-	66.37-	58.21-	68.41-	55.12-	50.00±	86.92-	68.92-	34.09±	83.22-	0-6-17
APMM [50] + ItoU
	95.43-	88.00-	54.73-	63.42-	71.51±	44.39-	88.49-	30.69-	51.52-	70.73±	74.50-	63.31-	69.11-	69.97-	66.58-	58.33-	68.84-	55.24-	50.00±	87.61-	68.12-	34.14±	83.59±	0-5-18
ENMI [38] + ItoU
	96.98±	90.00±	57.07+	64.50-	71.51±	45.99-	88.72-	31.03-	52.47-	71.23±	88.00±	63.48-	69.08-	70.07±	66.78-	58.42-	70.17-	56.11-	50.11-	87.32-	68.51-	34.26±	83.68±	2-8-13
BNMI
	96.86	89.48	54.33	64.98	71.51	46.11	89.01	30.89	52.18	71.23	87.25	64.02	69.56	70.00	67.02	58.18	71.41	56.41	50.00	87.31	69.07	34.37	83.56
BNMI + ItoU
	96.91	90.00	56.39	65.17	71.51	46.31	89.09	31.41	52.88	71.35	88.17	64.84	69.58	70.37	67.15	58.78	71.55	56.57	50.00	88.10	69.58	34.33	83.75

Fig. 13

Friedman’s ANOVA test on Table 2. The p-value is 1.29773e^∧-27.

The base clustering algorithms and some of the state of the art clustering ensemble approaches has been compared in Table 3 in terms of accuracy. It is obvious that there is no dominant method. An ensemble of size 100 produced by kmeans algorithm is used for this experimentation. The seed points have been randomly reinitialized. The average linkage algorithm has been used as consensus-function to derive the consensus results. Since the last five columns are cluster selection based approaches, the ensemble sizes are 50% of the best clusters of ensemble there. The number of clusters in each partition of ensemble is a random number in $[2, \sqrt{n}]$ .

Table 3

Experimental results in terms of Accuracy

Simple Methods (%)				Ensemble Methods (%)
Dataset	Single Linkage	Average Linkage	Complete Linkage	kmeans	kmeans Ensemble	Full Ensemble	Cluster Selection by NMI Measure [48]	Cluster Selection by MAX Measure [49]	Cluster Selection by APMM Measure [50], [70]	Cluster Selection by ENMI Measure [38]	Cluster Selection by BNMI Measure
Wine*	37.64	38.76	83.71	93.63	93.63	94.08	95.75	96.27	95.31	95.73	97.13
Breast-C*	65.15	70.13	88.73	89.37	89.46	89.10	77.75	84.89	90.56	89.73	89.91
Yeast*	24.38	25.11	28.91	28.20	30.46	30.47	31.23	29.27	30.50	31.54	31.62
Glass*	36.45	37.85	40.65	45.28	47.01	47.83	50.76	49.35	50.79	51.62	51.97
Bupa*	57.68	57.10	55.94	54.64	54.49	55.83	54.33	57.42	55.07	58.38	58.64

The last five columns of Table 3 are the results of the ensemble methods which use only a sub-set of primary clusters. The 7th, 8th, 9th, 10th, and 11th columns show the results of the ensembles which use the traditional NMI, MAX, APMM, and ENMI and proposed BNMI to validate the base clusters. The results of these columns indicate although these approaches use a sub-set of the primary clusters, they usually outperform the full ensemble. Also, comparing these five columns indicates the superiority of BNMI based stability to the NMI, MAX, APMM, and ENMI based stability.

The performance of clustering ensemble that is based on the BNMI method has been presented in comparison to those of the state-of-the-art clustering ensemble methods in Table 4. In Table 4, the paired t-test [75 –78] have been employed to validate the conclusions with a 95% confidence interval. In the last row, several sets of triples have been shown in the format of A-B-C. The number A shows the number of datasets that the proposed method is superior to the given method validated by paired t-test with a p-value of 0.05 (i.e. the confidence level is 95%). The number B shows the number of datasets that the proposed method is neither significantly worse nor better than the given method validated by paired t-test with a p-value of 0.05. The number C shows the number of datasets that the given method is superior to the proposed method validated by paired t-test with a p-value of 0.05.

Table 4

Experimental results of the proposed method comparing with those of the state of the art approaches. Sign “-” indicates the method mentioned in that column is significantly worse than the proposed method (validated by paired t-test). Sign “+” indicates the method mentioned in that column is significantly better than the proposed method (validated by paired t-test). Sign “±” indicates the method mentioned in that column is neither significantly worse nor better than the proposed method (validated by paired t-test). t-test is validated by a p-value of 0.05. The results are in terms of F-measure

	Ensemble Methods (%)
Dataset	Naldi et al. [64]	Wang et al. [67]	Lu et al. [68]	Alizadeh et al. [69]	Yu et al. [70]	Akbari et al. [71]	Parvin et al. [72]	Proposed Method
Wine*	97.26-	97.71-	98.39-	98.27-	98.23-	98.93±	98.41-	98.89
Breast-C*	96.09-	95.73-	96.38±	96.49±	96.30±	96.18-	96.21-	96.52
Yeast*	36.08-	37.79-	33.55-	41.27-	38.61-	42.72±	42.94±	43.14
Glass*	49.63-	50.45-	52.38-	51.35-	51.77-	52.41-	52.74-	53.03
Bupa*	58.11-	59.45-	57.95-	60.42±	59.05-	58.69-	59.56-	60.47
Summery	5-0-0	5-0-0	4-1-0	3-2-0	4-1-0	3-2-0	4-1-0

Different rates of noisy data points have been added to wine dataset. In each noise level, the performances of the cluster selection by NMI measure [48], the cluster selection by MAX measure [49], the cluster selection by APMM measure [50], the cluster selection by ENMI measure [38], and the cluster selection by BNMI measure [48] have been computed in terms of NMI. These results have been depicted by Fig. 14.

Fig. 14

Noise Analysis. Horizontal axis depicts rate of the added noisy data samples.

4.5 Final conclusive results

According to this subsection, 10 actual benchmark datasets have been used: MNIST (with 5000 data points, 10 classes and 784 features) [74], USPS (with 11000 data points, 10 classes and 256 features) [79], Multiple Features (MF) (with 2000 data points, 10 classes and 649 features), Semeion (with 1593 data points, 10 classes and 256 features), Image-Segmentation (IS) (with 2310 data points, 7 classes and 19 features), Optical-Digit-Recognition (ODR) (with 5620 data points, 10 classes and 64 features), Forest-Cover-Type (FCT) (with 3780 data points, 7 classes and 54 features), Landsat-Satellite (LS) (with 6435 data points, 6 classes and 36 features), Letter-Recognition (LR) (with 20000 data points, 26 classes and 16 features) and ISOLET (with 7797 data points, 26 classes and 617 features) [73].

The basic partitions are generated via the algorithm of k-means clustering. The clusters number in every ensemble member has been randomly chosen from the set { k, k + 1, …, 2k }. Each experiment has been repeated 30 times and the averaged performances are reported. Here, the size of ensemble is the same as before.

The NMI values of our suggested clustering ensemble method is evaluated and compared with those of ten state-of-the-art algorithms; it proves that the suggested method is better than all of state-of-the-art algorithms verified via paired t-test [78] with 95% confidence level.

The runtime of the different state-of-the-art clusterer ensembles indicated by Table 6 at different data sizes has been analyzed and computed in Fig. 15. Upon the dataset of LR, all those experiments have been performed. Every clusterer ensemble algorithm has been employed for clustering a randomly selected subsample of the mentioned dataset and meanwhile the runtime of that algorithm is recorded. Apparently, the best (fastest) method is MCLA. But the suggested method because of its massive calculation during cluster assessment, is among the other methods. Whereas, it is not among the quickest methods, it is acceptable (it means that the suggested method is tractable). The suggested method time complexity severely depends on the size of reference set. Thus, definitely the suggested algorithm runtime will decline, if we determine a set with separate small reference, along with the ensemble. The time order of second step in CESBNMI is member of O (B²n²), i.e. $O (\sqrt{n} BBn \sqrt{n})$ . The third step in CESBNMI contains a sort of clusters for selection; it is member of $O (B \sqrt{n} log Bn)$ , i.e. $O (B \sqrt{n} log B \sqrt{n})$ . If B ⪡ n, selection is member of O (Bn^0.5 log n). This step may also contain a co-association matrix computation with the time order O (Bn^2.5), i.e. $O (nB \sqrt{n} n)$ . Therefore, the time order of third step in CESBNMI is member of O (Bn^2.5). If $B ⪡ \sqrt{n}$ , the third step time order is dominant to the time orders of the first and second steps. The time order of fourth step in CESBNMI is also less than O (n^2.5). Therefore, the time order of CESBNMI is member of O (Bn^2.5), or more accurately O ((Bn^0.5 + B²) n²).

Table 5
The suggested approach performances (i.e. the NMI values) in comparison to those of ten state-of-the-art approaches verified via paired t-test [78] with 95% confidence level

Semeion MF IS FCT MNIST ODR LS ISOLET USPS LR Average Summery

CSPA [65] 53.41- 60.33- 59.43- 19.98- 51.36- 71.96- 50.66- 65.30- 49.22- 32.99- 51.46 0:0:10

HGPA [65] 48.85- 52.18- 48.54- 15.56- 41.01- 60.08- 36.20- 61.89- 35.75- 33.84- 43.39 0:0:10

MCLA [65] 57.39- 63.45- 61.61+ 21.18- 55.76- 77.02- 53.16- 69.99- 53.55- 38.41- 55.15 1:0:9

SRS [80] 62.95- 62.93- 60.25- 21.24- 59.78- 79.41- 57.70- 73.62+ 57.18- 40.76- 57.58 1:0:9

WEAC [81] 62.39- 61.84- 59.97- 21.68- 60.24- 79.80- 59.74- 72.98± 57.93- 41.82± 57.84 0:2:8

TME [68] 58.82- 68.44+ 55.91- 20.85- 56.98- 79.72- 49.81- 69.17- 58.44- 40.43- 55.86 1:0:9

ECSEAC [82] 61.71- 65.13- 61.30± 21.74- 54.96- 79.59- 58.00- 71.72- 59.00± 40.92- 57.41 0:2:8

WCE [83] 61.90- 65.43- 61.79+ 21.44- 60.16- 79.12- 58.69- 71.88- 58.57- 40.64- 57.96 1:0:9

MAX + ItoU [49] 61.78- 63.98- 60.23- 21.53- 59.37- 78.41- 60.36± 72.71± 57.87- 40.86- 57.71 0:2:8

APMM + ItoU [50] 61.95- 61.65- 61.24± 21.81- 60.43- 79.80- 61.12+ 72.75± 57.79- 41.52- 58.01 1:2:7

ENMI + ItoU [38] 63.18± 64.85- 61.57± 21.95- 61.95- 80.25- 60.10- 73.24± 58.08- 41.32- 58.65 0:3:7

BNMI + ItoU 63.21 66.72 61.38 22.14 62.31 80.68 60.53 73.11 59.19 42.02 59.13 -

	Semeion	MF	IS	FCT	MNIST	ODR	LS	ISOLET	USPS	LR	Average	Summery
CSPA [65]	53.41-	60.33-	59.43-	19.98-	51.36-	71.96-	50.66-	65.30-	49.22-	32.99-	51.46	0:0:10
HGPA [65]	48.85-	52.18-	48.54-	15.56-	41.01-	60.08-	36.20-	61.89-	35.75-	33.84-	43.39	0:0:10
MCLA [65]	57.39-	63.45-	61.61+	21.18-	55.76-	77.02-	53.16-	69.99-	53.55-	38.41-	55.15	1:0:9
SRS [80]	62.95-	62.93-	60.25-	21.24-	59.78-	79.41-	57.70-	73.62+	57.18-	40.76-	57.58	1:0:9
WEAC [81]	62.39-	61.84-	59.97-	21.68-	60.24-	79.80-	59.74-	72.98±	57.93-	41.82±	57.84	0:2:8
TME [68]	58.82-	68.44+	55.91-	20.85-	56.98-	79.72-	49.81-	69.17-	58.44-	40.43-	55.86	1:0:9
ECSEAC [82]	61.71-	65.13-	61.30±	21.74-	54.96-	79.59-	58.00-	71.72-	59.00±	40.92-	57.41	0:2:8
WCE [83]	61.90-	65.43-	61.79+	21.44-	60.16-	79.12-	58.69-	71.88-	58.57-	40.64-	57.96	1:0:9
MAX + ItoU [49]	61.78-	63.98-	60.23-	21.53-	59.37-	78.41-	60.36±	72.71±	57.87-	40.86-	57.71	0:2:8
APMM + ItoU [50]	61.95-	61.65-	61.24±	21.81-	60.43-	79.80-	61.12+	72.75±	57.79-	41.52-	58.01	1:2:7
ENMI + ItoU [38]	63.18±	64.85-	61.57±	21.95-	61.95-	80.25-	60.10-	73.24±	58.08-	41.32-	58.65	0:3:7
BNMI + ItoU	63.21	66.72	61.38	22.14	62.31	80.68	60.53	73.11	59.19	42.02	59.13	-

Fig. 15

Runtime of different clustering ensembles at different sizes of data-set.

5 Conclusions and future work

This study recommends a measure to decide whether a cluster should participate in a cluster ensemble. In this paper, a clustering ensemble method is developed based upon cluster selection. A measure called BNMI is devised to test the stability of clusters, and it is simply used to choose a sub-set of the steadiest clusters. Then, the output clustering of the ensemble is obtained based on the selected clusters. A large number of consensus-functions have been used to find consensus partition, after choosing a sub-set of clusters. Experimentally, this study indicates that the average-linkage hierarchical clusterer is most efficient consensus-function. One of the main contributions of the article has been the new method of EAC matrix construction. It has been called E-EAC.

The empirical results in some datasets of the UCI repository in this study claim that the suggested method often outshines the state of the art ones. In addition, the experiments display the usefulness of suggested method in comparison to the state of the art cluster ensembles in terms of the robustness and the quality. Additionally, it is inferred that whenever the equation of ItoU is used to construct co-association matrix, the consequences will be superior. Finally, this study infers that the best option is to employ BNMI as the cluster assessment. Additionally, it is inferred that the best option for consensus-function is to apply the algorithm named average-linkage on E-EAC-based co-association extracted by ItoU equation. The consensus-partition performance obtains its maximum whenever ratio of the steadiest clusters participating in final ensemble reaches at 50%.

References

Moradi

, Nejatian

, Parvin

and Rezaie

, CMCABC: Clustering and memory-based chaotic artificial bee colony dynamic optimization algorithm, International Journal of Information Technology & Decision Making 17(04) (2018), 1007–1046.

Parvin

, Nejatian

and Mohamadpour

, Explicit memory based ABC with a clustering strategy for updating and retrieval of memory in dynamic environments, Applied Intelligence 48(11) (2018), 4317–4337.

Hosseinpoor

M.J.

, Parvin

, Nejatian

, Rezaie

, Bagherifard

, Dehzangi

, Beheshti

and Alinejad-Rokny

, Proposing a novel community detection approach to identify co-interacting genomic regions, Mathematical Biosciences and Engineering 17(3) (2020), 2193–2217.

Hosseinpoor

M.J.

, Parvin

, Nejatian

and Rezaie

, Gene Regulatory Elements Extraction in Breast Cancer by Hi-C Data Using a Meta-Heuristic Method, Russian Journal of Genetics 55(9) (2019), 1152–1164.

Jenghara

M.M.

, Ebrahimpour-Komleh

and Parvin

, Dynamic protein–protein interaction networks construction using firefly algorithm, Pattern Analysis and Applications 21(4) (2018), 1067–1081.

Pirbonyeh

, Rezaie

, Parvin

, Nejatian

and Mehrabi

, A linear unsupervised transfer learning by preservation of cluster-and-neighborhood data organization, Pattern Analysis and Applications 22(3) (2019), 1149–1160.

Nejatian

, Rezaie

, Parvin

, Pirbonyeh

, Bagherifard

and Yusof

S.K.S.

, An innovative linear unsupervised space adjustment by keeping low-level spatial data structure, Knowledge and Information Systems 59(2) (2019), 437–464.

, Parvin

, Mahmoudi

M.R.

, Tuan

B.A.

and Pho

K.-H.

, A linear space adjustment by mapping data into an intermediate space and keeping low level data structures, Journal of Experimental & Theoretical Artificial Intelligence, (2020), DOI: 10.1080/0952813X.2020.1764634

Nejatian

, Parvin

and Faraji

, Using sub-sampling and ensemble clustering techniques to improve performance of imbalanced classification, Neurocomputing 276 (2018), 55–66.

10.

Shabaniyan

, Parsaei

, Aminsharifi

, Movahedi

M.M.

, Jahromi

A.T.

, Pouyesh

and Parvin

, An artificial intelligence-based clinical decision support system for large kidney stone treatment, Australasian Physical & Engineering Sciences in Medicine 42(3) (2019), 771–779.

11.

Aminsharifi

, Irani

, Pooyesh

, Parvin

, Dehghani

, Yousofi

, Fazel

and Zibaie

, Artificial neural network system to predict the postoperative outcome of percutaneous nephrolithotomy, Journal of Endourology 31(5) (2017), 461–467.

12.

Tavana

, Parvin

and Rezazadeh

, Parkinson detection: an image processing approach, Journal of Medical Imaging and Health Informatics 7(2) (2017), 464–472.

13.

Szetoa

P.M.

, Parvin

, Mahmoudi

M.R.

, Tuan

B.A.

and Pho

K.H.

, Deep neural network as deep feature learner, Journal of Intelligent & Fuzzy Systems DOI:10.3233/JIFS-191292

14.

Niu

, Xu

, Akbarzadeh

, Parvin

, Beheshti

and Alinejad-Rokny

, Deep feature learnt by conventional deep neural network, Computers & Electrical Engineering 84, 106656.

15.

, Parvin

and Izadparast

, Deep Learning Neural Network for Unconventional Images Classification, Neural Process Lett (2020). https://doi.org/10.1007/s11063-020-10238-3.

16.

Yasrebi

, Rafe

, Parvin

and Nejatian

, An efficient approach to state space management in model checking of complex software systems using machine learning techniques, J Intell Fuzzy Syst 38(2) (2020), 1761–1773.

17.

Bahrani

, Minaei-Bidgoli

, Parvin

, Mirzarezaee

, Keshavarz

and Alinejad-Rokny

, User and item profile expansion for dealing with cold start problem, J Intell Fuzzy Syst 38(4) (2020), 4471–4483.

18.

Partabian

, Rafe

, Parvin

and Nejatian

, An approach based on knowledge exploration for state space management in checking reachability of complex software systems, Soft Comput 24(10) (2020), 7181–7196.

19.

Jamalinia

, Khalouei

, Rezaie

, Nejatian

, Bagheri-Fard

and Parvin

, Diverse classifier ensemble creation based on heuristic dataset modification, Journal of Applied Statistics 45(7) (2018), 1209–1226.

20.

Jenghara

M.M.

, Ebrahimpour-Komleh

, Rezaie

, Nejatian

, Parvin

and Syed-Yusof

S.K.

, Imputing missing value through ensemble concept based on statistical measures, Knowledge and Information Systems 56(1) (2018), 123–139.

21.

, Chen

, You

, Han

and Li

, Hybrid Fuzzy Cluster Ensemble Framework for Tumor Clustering from Biomolecular Data,3, IEEE/ACM Transactions on Computational Biology and Bioinformatics 10 (2013), 657–670.

22.

Hanczar

and Nadif

, Ensemble methods for biclustering tasks, Pattern Recognition 45(11) (2012), 3938–3949.

23.

, Wong

H.S.

, You

, Yang

and Liao

, Knowledge based cluster ensemble for cancer discovery from biomolecular data, IEEE Transactions on Nanobioscience 10(2) (2011), 76–85.

24.

Rafiee

, Dlay

S.S.

and Woo

W.L.

, Region-of-interest extraction in low depth of field images using ensemble clustering and difference of Gaussian approaches, Pattern Recognition 46(10) (2013), 2685–2699.

25.

Zhang

, Wong

H.S.

and Shen

, Generalized adjusted rand indices for cluster ensembles, Pattern Recognition 45(6) (2012), 2214–2226.

26.

Yousefnezhad

, Huang

S.J.

and Zhang

, WoCE: A framework for Clustering Ensemble by Exploiting the Wisdom of Crowds Theory, IEEE Transactions on Cybernetics 99 (2017), 1–14.

27.

Huang

, Wang

C.D.

and Lai

J.H.

, Locally Weighted Ensemble Clustering, IEEE Transactions on Cybernetics 99 (2017), 1–14. doi: 10.1109/TCYB.2017.2702343

28.

, Zhu

, Wong

H.S.

, You

, Zhang

and Han

, Distribution- Based Cluster Structure Selection, IEEE Transactions on Cybernetics 99 (2016), 1–14. doi: 10.1109/TCYB.2016.2569529

29.

Jiang

, Chung

F.L.

, Wang

, Deng

, Wang

and Qian

, Collaborative Fuzzy Clustering From Multiple Weighted Views, IEEE Transactions on Cybernetics 45(4) (2015), 688–701.

30.

, You

, Wong

H.S.

and Han

, From cluster ensemble to structure ensemble, Information Sciences 198 (2012), 81–99.

31.

Naldi

M.C.

, De Carvalho

A.C.M.

and Campello

R.J.

, Cluster ensemble selection based on relative validity indexes, Data Mining and Knowledge Discovery 27(2) (2013), 259–289.

32.

Franek

and Jiang

, Ensemble clustering by means of clustering embedding in vector spaces, Pattern Recognition 47(2) (2014), 833–842.

33.

Hong

, Kwong

, Chang

and Ren

, Unsupervised feature selection using clustering ensembles and population based incremental learning algorithm, Pattern Recognition 41(9) (2008), 2742–2756.

34.

Mojarad

, Nejatian

, Parvin

and Mohammadpoor

, A fuzzy clustering ensemble based on cluster clustering and iterative Fusion of base clusters, Applied Intelligence 49(7) (2019), 2567–2581.

35.

Mojarad

, Parvin

, Nejatian

and Rezaie

, Consensus Function Based on Clusters Clustering and Iterative Fusion of Base Clusters, International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 27(1) (2019), 97–120.

36.

Nazari

, Dehghan

, Nejatian

, Rezaie

and Parvin

, A comprehensive study of clustering ensemble weighting based on cluster quality and diversity, Pattern Analysis and Applications 22(1) (2019), 133–145.

37.

Bagherinia

, Minaei-Bidgoli

, Hossinzadeh

and Parvin

, Elite fuzzy clustering ensemble based on clustering diversity and quality measures, Applied Intelligence 49(5) (2019), 1724–1747.

38.

Abbasi

, Nejatian

, Parvin

, Rezaie

and Bagherifard

, Clustering ensemble selection considering quality and diversity, Artificial Intelligence Review 52(2) (2019), 1311–1340.

39.

Najafi

, Parvin

, Mirzaie

, Nejatian

and Rezaie

, Dependability-based cluster weighting in clustering ensemble, Statistical Analysis and Data Mining 13(2) (2020), 151–164.

40.

Bagherinia

, Minaei-Bidgoli

, Hosseinzadeh

and Parvin

, Reliability-based fuzzy clustering ensemble, Fuzzy Sets and Systems DOI: 10.1016/j.fss.2020.03.008

41.

Niu

, Khozouie

, Parvin

, Alinejad-Rokny

, Beheshti

and Mahmoudi

M.R.

, An Ensemble of Locally Reliable Cluster Solutions, Appl Sci 10 (2020), 1891.

42.

Rashidi

, Nejatian

, Parvin

and Rezaie

, Diversity based cluster weighting in cluster ensemble: an information theory approach, Artificial Intelligence Review (2019), 1–28.

43.

Ayad

H.G.

and Kamel

M.S.

, Cumulative Voting Consensus Method for Partitions with a Variable Number of Clusters, IEEE Trans. on Pattern Analysis and Machine Intelligence 30(1) (2008), 160–173.

44.

Parvin

, Beigi

and Mozayani

, A clustering ensemble learning method based on the ant colony clustering algorithm, Int J Appl Comput Math 11(2) (2012), 286–302.

45.

Minaei-Bidgoli

, Parvin

, Alinejad-Rokny

and Alizadeh

, WF Punch, Effects of resampling method and adaptation on clustering ensemble efficacy, Artificial Intelligence Review 41(1), 27–48.

46.

Parvin

, Minaei-Bidgoli

and Alinejad-Rokny

, WF Punch, Data weighing mechanisms for clustering ensembles, Computers & Electrical Engineering 39(5), 1433–1450.

47.

Fred

and Jain

A.K.

, Data Clustering Using Evidence Accumulation, Intl. Conf. on Pattern Recognition, ICPR02, Quebec City, (2002), 276–280.

48.

Fred

and Jain

A.K.

, Combining Multiple Clusterings Using Evidence Accumulation, IEEE Trans. on Pattern Analysis and Machine Intelligence 27(6) (2005), 835–850.

49.

Alizadeh

, Minaei-Bidgoli

and Parvin

, A New Criterion for Clusters Validation, Artificial Intelligence Applications and Innovations (AIAI 2011), IFIP, Springer, Heidelberg, Part I, (2011), 240–246.

50.

Alizadeh

, Minaei-Bidgoli

, Parvin

and Moshki

, An Asymmetric Criterion for Cluster Validation, Developing Concepts in Applied Intelligence, Studies in Computational Intelligence 363 (2011), 1–14.

51.

Fred

and Jain

A.K.

, Learning Pairwise Similarity for Data Clustering, Int Conf on Pattern Recognition (2006).

52.

Breckenridge

, Replicating cluster analysis: Method, consistency and validity, Multivariate Behavioral Research, (1989).

53.

Rakhlin

and Caponnetto

, Stability of k-means clustering, In Advances in Neural Information Processing Systems 19 MIT Press, Cambridge, (2007).

54.

Lange

, Roth

, Braun

M.L.

and Buhmann

J.M.

, Stability-based validation of clustering solutions, Neural Computation 16(6) (2004), 1299–1323.

55.

Wang

, Han

and Han

, Rough set based cluster ensemble selection, In Proceedings of the 16th International Conference on Information Fusion (2013), 438–444.

56.

, Yang

and Wang

, Selective clustering ensemble based on covariance, Multiple Classifier Systems (2013), 179–189.

57.

Akbari

, Dahlan

H.M.

, Ibrahim

and Alizadeh

, Hierarchical cluster ensemble selection, Engineering Applications of Artificial Intelligence 39 (2015), 146–156.

58.

Brandsma

and Buishand

T.A.

, Simulation of extreme precipitation in the Rhine basin by nearest-neighbour resampling, Hydrology and Earth System Sciences 2 (1998), 195–209.

59.

Das

A.K.

and Sil

, Cluster Validation using Splitting and Merging Technique, Int. Conf. on Computational Intelligence and Multimedia Applications, ICCIMA, (2007).

60.

Davison

A.C.

, Hinkley

D.V.

and Young

G.A.

, Recent developments in bootstrap methodology, Statistical Science 18 (2003), 141–157.

61.

Inokuchi

, Nakamura

and Miyamoto

, Kernelized Cluster ValidityMeasures and Application to Evaluation of Different Clustering Algorithms, IEEE Int. Conf. on Fuzzy Systems, Canada, July (2006), 16–21.

62.

Möller

and Radke

, Performance of data resampling methods based on clustering, Intelligent Data Analysis 10(2) (2006).

63.

Xie

X.L.

and Beni

, A Validity measure for Fuzzy Clustering, IEEE Trans. on Pattern Analysis and Machine Intelligence 13(4) (1991), 841–846.

64.

Fern

X.Z.

and Lin

, Cluster Ensemble Selection, SIAM International Conference on Data Mining, (2008).

65.

Strehl

and Ghosh

, Cluster ensembles-a knowledge reuse framework for combining multiple partitions, Journal of Machine Learning Research 3 (2002), 583–617.

66.

Azimi

, An Informed Clustering Ensemble, Iran University of Science & Technology, M.Sc. Dissertation, (2008). (in Persian)

67.

Alizadeh

, Clustering Ensemble Based on a Sub-set of Primary Clusters, Iran University of Science & Technology, M.Sc. Dissertation, (2008). (in Persian)

68.

Zhong

, et al., A clustering ensemble: Two-level-refined co-association matrix with path-based transformation, Pattern Recognition (2015).

69.

Law

M.H.C.

, Topchy

A.P.

and Jain

A.K.

, Multi-objective data clustering, IEEE Conference on Computer Vision and Pattern Recognition 2 (2004), 424–430.

70.

Alizadeh

, Minaeibidgoli

and Parvin

, Cluster ensemble selection based on a new cluster stability measure, Intelligent Data Analysis 18(3) (2014), 389–408.

71.

Munkres

, Algorithms for the Assignment and Transportation Problems, Journal of the Society for Industrial and Applied Mathematics 5(1) (1957), 32–38.

72.

Marxer

, Holonowicz

, Purwins

and Hazan

, Dynamical hierarchical self-organization of harmonic motivic, and pitch categories. In Music, Brain and Cognition, Part 2: Models of Sound and Cognition, held at NIPS, (2007).

73.

Newman

C.B.D.J.

, Hettich

and Merz

, UCI repository of machine learning databases, http://www.ics.uci.edu/˜mlearn/MLSummary.html, (1998).

74.

http://yann.lecun.com/exdb/mnist/.

75.

Nejatian

, Omidvar

, Mohamadi

, Baghbani

A.E.

, Rezaie

and Parvin

, An optimization algorithm based on behavior of see-see partridge chicks, Journal of Intelligent & Fuzzy Systems 33(6) (2017), 3227–3240.

76.

Omidvar

M.N.

, Nejatian

, Parvin

and Rezaie

, A new natural-inspired continuous optimization approach, Journal of Intelligent & Fuzzy Systems (2018), 1–17.

77.

Yasrebi

, Eskandar-Baghban

, Parvin

and Mohammadpour

, Optimisation inspiring from behaviour of raining in nature: droplet optimisation algorithm, International Journal of Bio-Inspired Computation 12(3) (2018), 152–163.

78.

Dietterich

T.G.

, Approximate statistical tests for comparing supervised classification learning algorithms, Neural Computation 7(10) (1998), 1895–1924.

79.

Dueck

, Affinity propagation: Clustering data by passing messages, PhD dissertation, University of Toronto, (2009).

80.

Iam-On

, Boongoen

and Garrett

, Refining pairwise similarity matrix for cluster ensemble problem with cluster relations, in Proc. of International Conference on Discovery Science (ICDS), (2008), 222–233.

81.

Huang

, Lai

J.H.

and Wang

C.D.

, Combining multiple clusterings via crowd agreement estimation and multi-granularity link analysis, Neurocomputing 170 (2015), 240–250.

82.

Parvin

and Minaei-Bidgoli

, A clustering ensemble framework based on selection of fuzzy weighted clusters in a locally adaptive clustering algorithm, Pattern Anal Appl 18(1) (2015), 87–112.

83.

Alizadeh

, Yousefnezhad

and Minaei-Bidgoli

, Wisdom of Crowds cluster ensemble, Intell Data Anal 19(3) (2015), 485–503.