Spread binary artificial fish swarm algorithm combined with double-fault measure for ensemble pruning

Abstract

Ensemble pruning is usually used to improve classification ability of an ensemble using less number of classifiers, and it is an NP-hard problem. Existing ensemble pruning approaches always find the optimal sub-ensemble using diversity of classifiers or running heuristic search algorithms separately. Diversity and accuracy of classifiers are widely recognized as two important properties of an ensemble. The increase of the diversity of classifiers must lead to the decrease of the average accuracy of the whole classifiers, and vice versa, so there is a tradeoff between diversity and accuracy of classifiers. Finding the tradeoff is the key to a successful ensemble. Heuristic algorithms have good results when it comes to finding the tradeoff, but it is unfeasible to do an exhaustive search. Hence, we propose a Spread Binary Artificial Fish swarm algorithm combined with a Double-fault measure for Ensemble Pruning (SBAFDEP) using a combination of diversity measures and heuristic algorithms. First, the classifiers in an initial pool are pre-pruned using a double-fault measure, which significantly alleviates the computational complexity of ensemble pruning. Second, the final ensemble is efficiently assembled from the retaining classifiers after pre-pruning using the proposed Spread Binary Artificial Fish Swarm Algorithm (SBAFSA). Simulation and experiment results on 25 UCI datasets show that SBAFDEP performs better than other state-of-the-art pruning approaches. It provides a novel research idea for ensemble pruning.

Keywords

Artificial fish swarm algorithm spread behavior double-fault measure diversity ensemble pruning

1 Introduction

Ensemble learning is a challenging task in pattern recognition [1], machine learning [2], and data mining [3]. There are mainly two steps with respect to constructing a classification prediction model. In the first step, multiple classifiers with a large diversity are generated; in the second step, the final results are achieved by merging the predictions of all the classifiers. Remarkable improvement of an ensemble composed of multiple classifiers in predictive ability in comparison with a single classifier [4, 5], which has been applied in many fields, e. g., face recognition [6], age prediction [7], image processing [8], medical diagnoses [9], bioactive molecule prediction [10], intrusion detection [11], and the like. The increase of rapid growth in the size of data, which brings heavy storage requirements and computational overheads for ensemble learning [4]. Therefore it motivates the appearance of ensemble pruning [4]. Ensemble \nobreak pruning aims at achieving a better predictive ability using an sub-ensemble composed of a part of classifiers extracted from an initial pool of classifiers, which requires less resource. Typically, the final results are attained by aggregating a fraction of classifiers in an initial pool [4, 5].

It is well-accepted that diversity of classifiers is the key to the success of an ensemble [12]. The increase of the diversity among classifiers must lead to the decrease of the average accuracy of the whole classifiers, and vice versa. Hence there is a tradeoff between diversity and the average accuracy of classifiers [13]. The remarkable improvements of an ensemble can be attained when finding the tradeoff [14]. However, finding the tradeoff will bring on large computational requirements, including the training costs, pruning costs, and prediction costs [5].

For an ensemble with M classifiers, there are 2^M - 1 nonempty subsets, and it is an NP-hard problem [4]. Thus, it is unfeasible for most of the ensemble pruning approaches to find the exact solution; we just need to find the near-optimal sub-ensemble [5]. To find the near-optimal sub-ensemble, the literature in the Section 1.1 proposed various ensemble pruning approaches.

Based on the above analysis, it is easy to see that most ensemble pruning approaches search for the final ensemble using diversity measures or heuristic searching algorithms separately. Those pruning approaches based on diversity measures, using different strategies, cannot exactly find the optimal sub-ensemble; Those pruning methods based on heuristic algorithms cannot also exhaustively search for it. To address the said issue, we attempt to search for the final ensemble using a combination of heuristic algorithms and diversity measures. It is the reasons that a part of classifiers with bad performance are pre-pruned using diversity measures [4], which dramatically alleviates the computational complexity of ensemble pruning, and that the sub-ensemble very close to being optimal is efficiently achieved from the retaining classifiers after pre-pruning using heuristic algorithms [15].

In addition, Double-Fault measure (DF) do well on measuring the diversity of classifiers [14, 16], and pre-pruning the classifiers who perform badly. Artificial Fish Swarm Algorithm (AFSA) has some advantages with respect to searching for the optimal solution [17 –22], and AFSA can be used as a searching strategy to find the final ensemble extracted from the retaining classifiers after pre-pruning. Therefore, Spread Binary Artificial Fish swarm algorithms combined with Double-fault measure for Ensemble Pruning (SBAFDEP) is proposed. The mission is achieved in two steps: (1) The classifiers in an initial pool of classifiers are pre-pruned using the double-fault measure, which markedly reduces the computational overheads of ensemble pruning. (2) The final ensemble is efficiently attained from the retaining classifiers after pre-pruning using the proposed SBAFSA. The combination of DF and SBAFSA provides a new research technique for ensemble pruning.

1.1 Related work

Plenty of researchers studied the pruning techniques, so a lot of effective ensemble pruning methods have been proposed. In general, these algorithms are classified into four categories [5, 23], which are listed as follows.

Ordering-based pruning refers to those methods which first assign a rank to each classifier in an initial pool of classifiers based on a certain diversity measure. Then the diversity measures of the classifiers with over a predefined value are selected to construct the final ensemble. Martınez-Muñoz et al. [4 , 25] proposed ordering-based approaches, including Reduce-Error (RE), Complementarity measure (COM), Margin Distance Minimization (MDM), and found that the classifiers are complementary. Experimental results on UCI datasets demonstrate that an ordered ensemble outperforms the original ensemble. Margineantu et al. [26] proposed Kappa pruning, which sorts the classifiers (smallest to largest) according to their Kappa measure, and the classifiers with the smaller Kappa measures are integrated. Lu et al. [27] proposed an ordering-based approach, which achieves the final ensemble by ordering the ensemble members according to their contribution of each classifier in a descending order. Margin-based Ordered AGgregation for ensemble pruning (MOAG) is proposed by Guo et al. [28]. It selects the classifiers with larger margin-based criterion to constitute the final ensemble. Guo et al. [29] proposed a novel metric named margin and diversity-based measure, and the final ensemble is composed of members in a decreasing order based on that measure. Galar et al. [3] proposed a new ordering-based pruning method, which improves the ensemble performance of ensembles composed of a fraction of classifiers for imbalanced datasets, and achieves a good result.

Optimization-based approaches convert the ensemble pruning problem into a combinatorial optimization problem which attempts to find the optimal sub-ensemble that performs the best [5]. Zhou et al. [15] proposed a well-known approach called a Genetic Algorithm based Selective ENsemble (GASEN), random weights are assigned to the classifiers in an initial pool of classifiers, and GA is used to evolve the weights. The classifiers whose weights are above a predefined threshold can be selected to constitute the final ensemble. Zhang et al. [30] proposed a Selective ENsemble algorithm based on the Glowworm Swarm Optimization algorithm (GSOSEN). It uses the GSO as a searching strategy instead of the GA of GASEN.

Rokach et al. [31] proposed Collective Agreement-based ensemble Pruning (CAP), which considers the diversity and the accuracy of classifiers simultaneously. CAP needs to calculate the agreement measure between the member predictions and the true class labels, as well as the agreement measure between any two members. The member which highly agrees with the true class labels and has low inter-agreement with others can be selected to constitute the final ensemble.

Clustering-based approaches employ clustering techniques to find some representative classifiers to construct the final ensemble. It involves two steps: In the first step, the classifiers in an initial pool of classifiers are partitioned into different clusters. The classifiers extracted from the same cluster make similar results, while the classifiers created from different clusters perform in a more diverse manner. There are several clustering techniques which can be used in ensemble pruning, for example: k-means [32], deterministic annealing [33], hierarchical agglomerative clustering [34], etc. In the second step is to make the classifiers in the ensemble perform diversely. To do this we should extract the classifiers from the different clusters. The classifiers at the centroid of each cluster are selected to constitute the pruned ensemble, which was proposed by Bakker and Heskes [33].

This section discusses other pruning approaches, which are the algorithms that do not belong to any one of the aforementioned three categories. Martınez-Muñoz et al. [26] proposed an ensemble pruning approach, which prunes an ensemble using AdaBoost. An et al. [16] proposed the Double-Fault measure based on voting-based ELM (DF-D), which achieves the final ensemble using an one-side confidence interval according to the double-fault measures of the classifiers. An effective Ensemble Pruning algorithm based on Frequent Pattern (EP-FP) was proposed by Zhou et al. [35], which finds the frequent classifiers by calculating a Boolean matrix, then integrating the classifiers. Cavalcanti et al. [36] proposed combining Diversity measures for ensemble Pruning (DivP). On one hand, the different diversity measures are combined using Genetic Algorithm (GA) to constitute the combining diversity measure. On the other hand, the pruned ensemble is achieved using a graph coloring method. Ykhlef et al. [23] proposed ensemble Pruning based on Simple Coalitional Games (SCG-P), which prunes an ensemble using a minimal winning coalition achieved by calculating the Banzeff power factor of each classifier. Dai et al. [37] proposed three new diversity measures for ensemble pruning by considering diversity and accuracy simultaneously, with the final ensemble achieved using a Greedy Ensemble Pruning (GEP) algorithm.

1.2 Contributions and outline

The contributions of this work are described as follows:

A novel approach for ensemble pruning is proposed using the combination of SBAFSA and DF.

The proposed SBAFSA performs well when it comes to the convergence speed and precision.

DF can be used to pre-prune the classifiers with bad performance, which significantly alleviates the computational complexity of ensemble pruning.

Experimental results on 25 UCI datasets demonstrate the effectiveness and efficiency of the proposed method.

It provides a new research idea for ensemble pruning.

This paper’s basic structure is as follows. In Section 2, the basic concept of diversity measures is presented. In Section 3, SBAFSA is proposed. We introduce SBAFDEP in Section 4, and how to use it. Simulation and experiments are expressed in Section 5. Finally, conclusions and future works are given in Section 6.

2 Diversity measures

Diversity of classifiers is very important for a successful ensemble, but what standard should be used to measure the diversity of the classifiers? How can we make use of the existing diversity measures to design a classifier with a good amount of diversity and a strong generalization performance? The above problems are unresolved issues [16]. Diversity in ensemble systems determines the performance of ensemble learning, so we need an effective diversity measure. Although many diversity measures are used in the literature, there is not a widely accepted diversity measure. No matter which diversity measure we take, solid empirical and theoretical validation should be presented. Five pair-wise diversity measures are listed as follows [36].

The main notations used in this work are summarized as follows:

N: the number of samples;

X: the sample dataset, X = {x₁, x₂, ⋯ , x_N};

Y: the class labels of sample set X, Y = {y₁, y₂, ⋯ , y_N};

M: the size of the classifiers;

F: the classifier set, F = {f₁, f₂, ⋯ , f_M};

f_i (x_k): classification result of the sample x_k classified by the classifier f_i, i = 1, 2, ⋯ , M, k = 1, 2, ⋯ , N;

Table 1
Contingency table for two classifiers

f_i (x_k) = y_k f_i (x_k) ≠ y_k

f_j (x_k) = y_k a b

f_j (x_k) ≠ y_k c d

	f_i (x_k) = y_k	f_i (x_k) ≠ y_k
f_j (x_k) = y_k	a	b
f_j (x_k) ≠ y_k	c	d

To calculate diversity measures between two classifiers, the contingency table [36] between f_i and f_j is presented in Table 1.

In Table 1, a is the number of examples in X correctly classified by both f_i and f_j; b is the number of examples correctly classified by f_i and incorrectly classified by f_j; c is for the number of examples incorrectly classified by f_i and correctly classified by f_j; d is the number of examples incorrectly classified by both classifiers. Thus, a + b + c + d = N.

The correlation coefficient measure (ρ) is formulated using Equation (1). $ρ_{ij} = \frac{ad - bc}{\sqrt{(a + b) (c + d) (a + c) (b + d)}}$ (1)

The Q-statistic measure (Q) is similar to a correlation coefficient measure. It rewritten as Equation (2). $Q_{ij} = \frac{ad - bc}{ad + bc}$ (2)

The pair-wise Kappa measure (Kp) was usually employed to analyze the diversity of classifiers. It is presented as Equation (3). ${Kp}_{ij} = \frac{2 (ad - bc)}{(a + b) (b + d) + (a + c) (c + d)}$ (3)

The disagreement measure is expressed as Equation (4). ${Dis}_{ij} = \frac{b + c}{N}$ (4)

The double-fault measure is calculated as Equation (5).

${DF}_{ij} = \frac{d}{N}$ (5)

3 Spread binary artificial fish swarm algorithm

Li et al. [17] first proposed Artificial Fish Swarm Algorithm (AFSA) in 2002, which is a novel swarm-intelligence technique, inspired by the natural fish behaviors, including swarming, following, and preying behaviors. It has been applied in many optimization problems, such as combinatorial optimization, function optimization et al. [18, 19]. An initial population of Artificial Fishes (AFs) is produced using a random function, AFSA has the abilities of self-organized, and it does well when it comes to converging on an optimal solution. The global optimal solution recorded in the bulletin board is updated using swarming, following, and preying behaviors at every iteration process. The parameters of AFSA include N (the size of the AF’s population), Visual (a visual scope), Step (the maximal moving step length), and δ (a crowd factor). The swarming, following, and preying behaviors are introduced in detail in Ref. [17, 20]. To solve binary discrete problems, the proposed SBAFSA is presented.

3.1 Mapping operation

To make a discrete AFSA simple yet efficient, we can update the Afs’ position by mapping operation in a simple way, and the position is mapped into 0 or 1. Assuming that the current state of AF is X_i = (x_i1, ⋯ , x_ik, ⋯ , x_in) (n is the number of the space’s dimension). The mapping operation is modified as follows [38]. $x_{ik} = {\begin{matrix} 1, if (rand ⩽ S (x_{ik})) \\ 0, otherwise \end{matrix}$ (6)

Where rand is a random number in (0, 1), and S (x) = 1/ - (1 + exp(- x)) is a sigmoid function.

3.2 Spread behavior

In the basic AFSA, AFs will rapidly gather together using swarming and following behavior, which may lead to low population diversity and prematurity convergence. To avoid prematurity convergence, spread behavior is introduced into AFSA. As we know, during biological evolution, organisms can move from one habitat to another one for the environmental stress, population density and change of living environment [39]. It cannot only balance and keep the stabilization of eco-environment, but prevent further extinction of the organism. We are inspired by that, SBAFSA is presented.

Assign nutrition value ${Nutrition}_{i}^{t = 0} = N_{0}, 1 ⩽ i ⩽ N$ to AF X_i. Nutrition value of X_i can be updated as Equation (7). ${Nutrition}_{i}^{t + 1} = {\begin{matrix} {Nutrition}_{i}^{t} + 1, if (Y_{i}^{t} < Y_{i}^{t + 1}) \\ {Nutrition}_{i}^{t} - 1, if (Y_{i}^{t} ⩾ Y_{i}^{t + 1}) \end{matrix}$ (7)

Where t is the current iteration, N₀ is a constant, $Y_{i}^{t}$ is the fitness function of X_i at the tth iteration, $Y_{i}^{t + 1}$ is the fitness function of X_i at the (t+1)th iteration.

Then set a upper limit N_up and a lower limit N_down. If ${Nutrition}_{i}^{t} > N_{up}$ , then it executes breeding operation and spreads next generation. The number of the offspring generated using breeding operation is in direct proportion to difference between $Y_{i}^{t + 1}$ and $Y_{i}^{t}$ , which can be expressed as Equation (8). $N_{b} = ⌊ B_{min} + (B_{max} - B_{min}) \times \frac{Y_{i} - Y_{min}}{Y_{max} - Y_{min}} ⌋$ (8)

Where Y_i is the fitness function of the current AF, ⌊⌋ is a rounded down, Y_max, Y_min, B_max and B_min are the optimal and the worst value of the current population, the number of the maximum and the minimum seeds, respectively.

Offspring generated using breeding operation are distributed into the binary solution space, who follow Gaussian distributions. We use the parent AF, its visual as the axis, the variance of Gaussian distributions [40], respectively. The positions of the generated offspring are shown as Equations (9) and (10). $λ = λ_{0} \times tan (0.875 \times \frac{t_{max} - t}{t_{max}}) \times Visual$ (9) $X_{ij} = X_{i} + λ (j = 1, 2, \dots, N_{b})$ (10)

Where λ₀ is a constant, t_max is the number of the maximal iteration, X_ij is a new generated offspring.

The value of the tangent function decreases with the decrease of the independent variable. It is easy to see that, the greater the value of λ is, the more far away from their parent the generated offspring is. At the later evolution stage of AFSA, value of λ decreases with the decrease of the iteration t, and the generated offspring is getting closer to their parent. All of the above can broaden the searching range of the AFs.

3.3 Selection behavior

After the spread behavior, there will have a great increase with respect to the population size. To adequately decrease those AFs who perform badly, and a selection behavior is used. In SBAFSA, the generated offspring will compete with their parents. If ${Nutrition}_{i}^{t} < N_{down}$ , and then it eliminates the individual AF, and releases its space. At the end of the current iteration, the first N AFs who perform better are retained, when the population size of AFs is above N.

4 SBAFDEP

The proposed approach is presented in this section, which uses the combination of SBAFSA and DF. It involves two steps: the classifiers in an initial pool are pre-pruned using the double-fault measure, and the retaining classifiers after pre-pruning are further pruned using SBAFSA. The proposed SBAFDEP is presented as follows.

4.1 Initial pool of classifiers

An initial pool with M classifiers is generated using the bootstrap sampling method in Bagging. As we know, the base classifiers in ensemble pruning must be unstable [36]. The Extreme Learning Machine (ELM) is unstable [16], and it has fast learning speed. Thus, the ELM is suitable as the base classifier in this work.

4.2 Pre-pruning based on double-fault measure

For an initial pool of classifiers which contains M classifiers, it has 2^M - 1 nonempty subsets, which is an NP-hard problem. Heuristic algorithms do well when it comes to finding the tradeoff between the average accuracy and the diversity of classifiers. While it is hard for heuristic algorithms to exhaustively search for the optimal solution. We need to pre-prune the classifiers in the pool, which can significantly reduce the computational burdens of ensemble pruning.

Generally speaking, five pair-wise diversity measures work well with respect to measuring the diversity among classifiers, so which one can be used as diversity measure for pre-pruning? We find that the double-fault measure performs better than others via past experiments. We attempt to employ the double-fault measure to pre-prune the classifiers with bad performance. We will discuss the above question in detail in the Section 5.1.

4.3 Pruning based on SBAFSA

The fitness function of the selective problem can be formulated as follows [15, 30]. $Fitness = A$ (11)

Where A indicates the classification accuracy between the predictive results and the class labels $A = \frac{1}{m} \sum_{j = 1}^{m} Acc ({\hat{y}}_{j}, y_{j})$ , $Acc ({\hat{y}}_{j}, y_{j}) = {\begin{matrix} 1, if ({\hat{y}}_{j} = y_{j}) \\ 0, if ({\hat{y}}_{j} \neq y_{j}) \end{matrix}$ , m is the number of testing samples. ${\hat{y}}_{j}$ is the predictive results on the jth samples, and y_j is the class label on the jth samples. Let $H^{'} = {h_{1}, h_{2}, \dots, h_{M}^{'}}$ be the members after pre-pruning, M′ be the number of the retaining classifiers. The classifier is expressed as binary strings using a binary encoding method, which is composed of 0 or 1. The combinatorial optimization problem can be converted into the data which SBAFSA can tackle.

4.4 The outline of SBAFDEP

The outline of SBAFDEP is shown in Fig. 1. First, the diverse classifiers are generated using the bootstrap sampling method, which are used to constitute an initial pool of classifiers. Second, the classifiers are sorted (smallest to largest) according to their double-fault measures, and the first M′ classifiers with smaller double-fault measures are retained. Third, SBAFSA is employed to search for the optimal sub-ensemble from the retaining M′ classifiers. The classification accuracy of sub-ensemble is taken as the objective function, and the bulletin board is used to record the optimal solution by calculating the fitness value of each AF in the initialization population. The optimal solution in the bulletin board can be updated using leaping, swarming, following, preying, spread and selection behaviors in SBAFSA, the sub-ensemble extracted from the retaining classifiers can be efficiently achieved.

Fig. 1

The outline of SBAFDEP.

5 Experiments

To evaluate the performance of SBAFDEP, we have carried out experiments on 25 UCI classification tasks. The experiments were implemented in Matlab 2017a. The experiments were repeated 30 times independently for a goal of reducing random effects of experiments, and the average results were attained. The parameters of SBAFSA were set as follows: the visual scope is half of the number of the retaining classifiers after pre-pruning, the crowd factor δ = 0.9, Step = 1, B_max = 5, B_min = 1, λ₀ = 1.5, t_max = 500, N_up = 5, N_down = 1.

The performance of SBAFDEP was assessed based on 25 UCI datasets, which are shown in Table 2. Experimental datasets were divided into five folds using k-fold cross validation with k = 5, and three folds for training, one for validation, and one for testing.

Table 2
Description of UCI datasets used in this work

Datasets Instances Attributes Classes

Heart 270 13 2

Cleveland 303 13 5

Column 310 6 2

Ecoli 336 8 8

Bupa 345 6 2

Ionosphere 351 34 2

Wholesale 440 7 2

ILP 583 10 2

Balance 625 4 3

Australian 690 14 2

BCW 699 10 2

Diabetes 768 8 2

Tic-tac-toe 958 9 2

German 1000 20 2

QSAR 1055 41 2

Diabetic-r 1151 19 2

CMC 1473 9 3

Yeast 1484 9 10

Wineq-r 1599 11 6

Car 1728 6 4

Segment 2310 19 7

Abalone 4177 8 3

Spambase 4601 57 2

Wineq-w 4898 11 7

Landsat 6435 36 6

Datasets	Instances	Attributes	Classes
Heart	270	13	2
Cleveland	303	13	5
Column	310	6	2
Ecoli	336	8	8
Bupa	345	6	2
Ionosphere	351	34	2
Wholesale	440	7	2
ILP	583	10	2
Balance	625	4	3
Australian	690	14	2
BCW	699	10	2
Diabetes	768	8	2
Tic-tac-toe	958	9	2
German	1000	20	2
QSAR	1055	41	2
Diabetic-r	1151	19	2
CMC	1473	9	3
Yeast	1484	9	10
Wineq-r	1599	11	6
Car	1728	6	4
Segment	2310	19	7
Abalone	4177	8	3
Spambase	4601	57	2
Wineq-w	4898	11	7
Landsat	6435	36	6

5.1 Experimental results

The classifiers in an initial pool were pre-pruned using the double-fault measure, which markedly reduced the number of the classifiers with worse performance and less diversity. However, how many classifiers should be retained using the double-fault measure to constitute the final ensemble? We will discuss the above question as follows.

The average ensemble accuracies decease as the increase of five pair-wise diversity measures. The ensemble accuracy can then be improved by pre-pruning a collection of classifiers with the larger pair-wise diversity measures, which has been proven in literature [12, 14]. Figure 2 shows that the average classification accuracies of random bagging and ordered bagging based on the pair-wise diversity measures with initial pools composed of 100, 200 and 300 classifiers on Column and Spambase datasets. We can find from Fig. 2 that, the curves of ordered ensemble accuracy based on the five pair-wise diversity measures go up first and then go down, and that the double-fault measure outperforms the other measures. In the beginning, there are a few classifiers in the ensemble, so the ensemble accuracy goes up. As the increase of the size of the initial pool, there are many redundant classifiers in the ensemble, which results in the decline of the ensemble accuracy. We can also see from Fig. 2 that, the ensemble accuracy can achieve the maximum before the number of classifiers reaches 25. In other words, the first 25 classifiers with the smaller double-fault measures should be remained. Therefore, we advise that the number of the retaining classifiers is 25, namely, M′ = 25. The retaining classifiers are used for further pruning based on SBAFSA.

Tables 3 and 4 indicate the results of SBAFDEP on 25 datasets, and in comparison to the original ensemble (Bagging). SBAFDEP does well when it comes to pruning the classifiers in the initial pool, and more than 80% of the classifiers in the pool has been pruned using SBAFDEP. The number of classifiers has been significantly reduced after pre-pruning, so that SBAFSA can achieve the final ensemble with a good efficiency. It is easy to see from Tables 3 and 4 that, SBAFDEP performs well when the number of the initial pool of classifiers is set at 200. When the number of classifiers is over 200, the ensemble accuracy can be slightly improved. So it is unnecessary to continue increasing the size of the initial pool. We advise that the size of the initial pool of classifiers is set at 200.

Fig. 2

Average ensemble performance of random bagging and ordered bagging based on pair-wise diversity measures on Column and Spambase datasets.

Table 3

Comparison with Bagging on different pool sizes (50, 100, 150) on the test datasets. SB: SBAFDEP, Bag: Bagging

Datasets	50				100				150
	Bag	n	SB	n	Bag	n	SB	n	Bag	n	SB	n
Heart	67.41	50	78.54	10	67.88	100	81.29	10	68.21	150	82.10	11
Cleveland	53.66	50	57.23	9	53.47	100	58.52	9	52.29	150	57.59	11
Column	78.53	50	89.72	11	78.72	100	91.49	11	79.95	150	93.28	12
Ecoli	85.54	50	88.38	10	86.06	100	89.34	11	86.20	150	89.35	10
Bupa	67.44	50	78.83	12	68.53	100	80.73	11	67.08	150	80.44	12
Ionosphere	90.45	50	96.12	9	89.52	100	96.09	8	90.49	150	97.24	9
Wholesale	86.14	50	92.12	12	85.46	100	92.18	12	86.28	150	93.32	10
ILP	69.38	50	71.16	7	72.53	100	74.00	9	70.97	150	72.43	8
Balance	91.68	50	93.28	10	91.60	100	93.44	9	90.96	150	93.20	9
Australian	67.29	50	69.46	8	67.44	100	69.67	7	69.75	150	72.86	10
BSB	65.68	50	65.68	12	63.84	100	65.84	11	65.35	150	65.35	12
Diabetes	66.21	50	73.28	10	65.28	100	74.52	10	66.17	150	76.19	10
Tic-tac-toe	90.78	50	95.68	11	90.27	100	96.81	13	90.54	150	96.87	12
German	70.70	50	77.90	9	70.90	100	81.60	10	71.90	150	82.40	11
QSAR	82.83	50	90.64	11	80.99	100	90.45	10	81.77	150	91.13	12
Diabetic-r	68.94	50	79.44	11	69.03	100	81.38	11	70.67	150	83.73	12
CMC	54.72	50	62.94	11	52.84	100	61.57	10	54.65	150	65.04	11
Yeast	58.02	50	62.56	10	56.51	100	61.30	10	58.98	150	64.12	11
Wineq-r	57.59	50	65.61	10	53.67	100	63.25	9	54.45	150	65.13	9
Car	90.87	50	94.34	11	90.10	100	94.79	11	91.07	150	95.42	9
Segment	84.43	50	89.25	11	83.74	100	89.62	10	83.76	150	89.52	11
Abalone	55.07	50	58.53	10	57.21	100	61.30	9	54.59	150	59.01	9
Spambase	73.17	50	82.69	9	73.40	100	82.96	12	74.13	150	83.85	10
Wineq-w	45.44	50	50.48	11	47.34	100	52.59	9	46.70	150	51.81	8
Landsat	77.79	50	81.29	13	79.06	100	83.16	13	80.16	150	83.86	11

To further evaluate the performance of SBAFDEP, we compare it with the following approaches: Bagging [41], Kappa [26], AGOB [24], POBE [25], DREP [42], DF-D [16], GASEN [15], GSOSEN [30], MOAG [28], EP-FP [35], RRE [5], DivP [36], SCG-P [23] and SDAcc [37]. Bagging can make the initial pool of classifiers perform diversely by selecting each sample with the same probability. Kappa, AGOB, POBE and MOAG show that the order in the classifiers is of importance. DREP starts the ensemble with just one classifier, and grows the ensemble by adding new classifiers, which makes the pruned ensemble perform in a diverse number of ways, and achieves good results. DF-D selects the classifiers whose double-fault measures belong to the one-side confidence interval. GASEN and GSOSEN employ GA and GSO to search for the optimal sub-ensemble, respectively. RRE achieves the final ensemble composed of the selected classifiers using RE and the discarded classifiers. DivP combines different pair-wise diversity matrices, and achieves the final ensemble using a graph coloring method. SCG-P assesses the diversity contribution of each classifier using the Banzhaf power index, and the final ensemble with the minimal winning coalition extracted from the classifiers in an initial pool can be achieved. SDAcc considers the diversity and the accuracy of the classifiers simultaneously, which makes the pruned ensemble attain a good result.

Table 4

Comparison with Bagging on different pool sizes (200, 250, 300) on the test datasets. SB: SBAFDEP, Bag: Bagging

Datasets	200				250				300
	Bag	n	SB	n	Bag	n	SB	n	Bag	n	SB	n
Heart	67.44	200	82.75	11	66.51	250	82.51	10	67.22	300	83.31	9
Cleveland	53.71	200	59.65	10	55.03	250	60.94	10	54.51	300	60.50	11
Column	79.28	200	92.47	11	79.78	250	93.10	11	79.46	300	93.39	11
Ecoli	87.03	200	90.43	10	87.51	250	90.61	12	87.14	300	90.56	11
Bupa	68.79	200	84.81	11	67.01	250	82.53	11	69.30	300	85.15	13
Ionosphere	91.54	200	97.35	11	90.80	250	97.82	8	90.21	300	97.20	10
Wholesale	86.96	200	94.61	13	86.81	250	94.07	11	84.58	300	93.18	11
ILP	71.43	200	73.57	11	71.49	250	74.07	9	74.63	300	76.79	9
Balance	90.96	200	92.96	9	91.36	250	93.84	10	90.88	300	92.80	10
Australian	68.08	200	71.19	9	68.52	250	71.42	8	67.42	300	70.83	7
BCW	65.90	200	64.90	12	66.74	250	66.74	12	63.76	300	63.76	13
Diabetes	64.34	200	75.59	11	67.56	250	77.61	11	63.77	300	74.65	11
Tic-tac-toe	90.50	200	97.13	12	90.34	250	97.44	13	90.29	300	97.81	13
German	72.10	200	83.70	11	71.20	250	82.50	11	73.30	300	85.60	11
QSAR	83.17	200	93.00	12	81.95	250	92.54	10	81.99	300	92.61	11
Diabetic-r	68.30	200	83.20	12	69.02	250	85.03	14	69.96	300	85.20	12
CMC	52.92	200	64.20	12	53.88	250	65.92	11	54.65	300	65.48	11
Yeast	58.80	200	63.89	11	58.07	250	63.97	10	59.07	300	65.36	11
Wineq-r	53.61	200	64.51	10	55.19	250	65.59	10	54.21	300	65.42	10
Car	91.36	200	95.65	12	90.10	250	95.25	10	91.46	300	95.88	11
Segment	84.40	200	89.93	11	83.82	250	90.22	11	84.38	300	90.53	12
Abalone	56.89	200	61.37	11	55.82	250	60.74	10	56.70	300	61.94	10
Spambase	73.63	200	84.92	10	73.28	250	85.34	11	72.61	300	85.07	11
Wineq-w	47.80	200	53.12	10	48.10	250	53.47	10	46.46	300	53.49	8
Landsat	77.68	200	82.32	13	79.17	250	83.62	12	79.20	300	83.44	14

Table 5

Classification accuracy and size of the pruned ensembles achieved by comparative approaches on pool size of 200

Datasets	SBAFDEP	n	Bagging	n	Kappa	n	AGOB	n	POBE	n	DREP	n	DF-D	n	GASEN	n
Heart	82.75	11	67.44	200	68.66	40	74.65	29	74.06	42	71.69	100	71.03	104	74.09	95
Cleveland	59.65	10	53.71	200	54.82	40	57.05	21	55.23	40	54.82	100	54.96	83	53.14	100
Column	92.47	11	79.28	200	82.51	40	86.85	24	85.71	41	83.65	100	82.95	104	85.15	92
Ecoli	90.43	10	87.03	200	87.69	40	88.48	23	88.72	38	87.35	100	88.01	97	87.22	101
Bupa	84.81	11	68.79	200	70.23	40	76.70	28	75.94	43	74.60	100	73.67	107	72.42	97
Ionosphere	97.35	11	91.54	200	94.39	40	95.35	21	94.34	41	92.55	100	92.82	106	92.99	100
Wholesale	94.61	13	86.96	200	88.15	40	89.87	28	90.06	43	88.61	100	88.63	106	90.05	97
ILP	73.57	11	71.43	200	70.57	40	70.85	20	71.50	26	70.74	100	71.09	67	72.27	103
Balance	92.96	9	90.96	200	90.99	40	91.79	25	92.21	34	91.25	100	91.57	106	91.92	100
Australian	71.19	9	68.08	200	66.71	40	68.06	29	67.58	41	67.29	100	67.15	95	65.31	101
BCW	65.90	12	64.90	200	64.21	40	64.20	25	65.12	43	64.91	100	65.39	96	67.57	100
Diabetes	75.59	11	64.34	200	69.29	40	71.34	23	71.33	39	69.29	100	69.24	101	69.90	97
Tic-tac-toe	97.13	12	90.50	200	93.70	40	93.18	29	94.05	42	92.10	100	92.62	109	92.16	97
German	83.70	11	72.10	200	76.07	40	77.73	28	79.00	41	76.80	100	76.47	105	76.30	96
QSAR	93.00	12	83.17	200	85.49	40	86.88	31	87.00	41	85.43	100	84.86	107	86.02	100
Diabetic-r	83.20	12	68.30	200	70.68	40	74.68	29	75.09	42	71.67	100	70.40	107	73.76	95
CMC	64.20	12	52.92	200	54.36	40	60.42	25	59.84	46	58.39	100	58.12	108	57.58	96
Yeast	63.89	11	58.80	200	58.45	40	60.90	24	61.12	41	59.91	100	60.03	104	59.74	96
Wineq-r	64.51	10	53.61	200	54.46	40	57.71	23	56.96	41	55.81	100	54.93	103	57.57	95
Car	95.65	12	91.36	200	92.26	40	92.73	27	93.03	38	91.53	100	91.88	102	92.19	95
Segment	89.93	11	84.40	200	85.48	40	86.71	27	86.02	42	84.63	100	84.91	106	87.11	95
Abalone	61.37	11	56.89	200	57.64	40	59.17	27	59.10	42	58.57	100	58.26	106	57.05	94
Spambase	84.92	10	73.63	200	78.77	40	80.84	29	79.71	39	78.01	100	77.17	97	78.90	94
Wineq-w	53.12	10	47.80	200	46.94	40	48.35	32	47.87	43	47.55	100	46.61	93	46.36	99
Landsat	82.32	13	77.68	200	78.37	40	82.62	35	81.09	47	81.11	100	80.37	114	81.04	96
Win/Tie/Loss	20/2/3		0/0/25		0/0/25		0/0/25		0/0/25		0/0/25		0/0/25		0/0/25

To evaluate the performance of SBAFDEP, we compare it with other ensemble pruning approaches with the initial pool size of 200. The results achieved by all the pruning approaches using the initial pool sizes of 200 are reported in Tables 5 and 6. “Win”/“Tie”/ “Loss” indicate that the number of times in which SBAFDEP scores were better/neutral/inferior than the other pruning approaches. We can see from Tables 5 and 6 that, SBAFDEP can achieve better results than the other pruning approaches with less number of classifiers on most validation datasets. The members of the pruned ensembles achieved by SBAFDEP are more than that of DivP, but SBAFDEP performs better than DivP.

Table 6

Classification accuracy and size of the pruned ensembles achieved by comparative approaches on pool size of 200

Datasets	GSOSEN	n	MOAG	n	EP-FP	n	RRE	n	DivP	n	SCG-P	n	SDAcc	n
Heart	73.87	60	70.13	33	67.18	93	80.11	21	78.98	8	77.10	17	79.06	19
Cleveland	55.56	42	54.95	24	53.95	85	57.86	21	57.93	2	57.88	4	58.59	6
Column	84.38	97	82.38	58	80.32	96	91.18	21	88.20	7	84.88	13	86.60	11
Ecoli	88.21	32	87.46	13	86.21	84	89.43	21	89.37	3	87.81	19	89.49	6
Bupa	71.30	58	72.26	29	69.76	95	81.95	21	77.45	12	76.04	44	77.36	35
Ionosphere	93.36	87	92.12	29	90.61	91	96.35	21	95.77	3	92.19	6	95.23	8
Wholesale	89.84	89	88.02	50	86.40	98	92.34	22	91.37	11	89.73	24	89.94	15
ILP	72.96	32	70.85	17	70.74	84	71.50	21	73.93	1	73.74	1	71.95	3
Balance	92.24	68	91.31	11	90.67	97	92.48	21	92.85	2	90.83	10	92.85	3
Australian	65.89	60	67.10	17	66.86	96	69.03	21	69.61	2	68.95	2	69.03	6
BCW	67.64	88	65.25	16	63.29	91	66.90	23	65.91	3	65.36	3	66.90	11
Diabetes	70.42	40	68.59	43	66.16	87	74.82	22	74.13	6	70.90	23	71.73	17
Tic-tac-toe	92.11	81	91.72	52	90.85	97	95.62	24	94.57	10	91.79	33	93.67	31
German	76.60	62	75.93	41	73.67	95	82.00	22	81.07	5	78.93	45	79.07	8
QSAR	85.93	75	84.23	57	82.09	96	90.91	22	89.40	7	87.98	27	86.69	7
Diabetic-r	73.32	72	70.28	61	67.03	99	80.35	25	78.50	5	78.41	15	81.75	8
CMC	57.31	63	57.53	32	55.40	86	64.95	22	63.09	7	61.81	41	62.97	19
Yeast	60.01	35	59.81	45	58.67	95	63.04	21	63.22	5	59.27	44	61.44	18
Wineq-r	58.18	58	55.76	37	52.95	97	62.21	22	61.85	7	60.09	8	60.67	8
Car	93.00	45	91.92	37	90.49	87	94.30	21	94.27	5	91.03	23	93.46	7
Segment	86.72	79	84.26	41	83.44	94	88.64	22	87.43	12	84.20	56	86.01	23
Abalone	57.55	23	58.45	25	57.01	92	61.37	22	61.37	4	59.07	12	60.08	3
Spambase	78.40	53	74.89	57	73.16	94	84.13	21	82.18	6	80.40	59	81.96	10
Wineq-w	47.85	42	47.57	40	46.35	93	51.79	21	51.63	6	49.73	37	50.49	3
Landsat	80.57	93	79.86	50	79.19	95	83.10	38	81.45	11	79.95	65	83.81	26
Win/Tie/Loss	0/0/25		0/0/25		0/0/25		1/2/22		1/1/23		0/0/25		1/1/23

Table 7

Comparison results using t-test method

Comparison	p-value	Results
SBAFDEP vs Bagging	0.0009*1.0e-5	H=1
SBAFDEP vs Kappa	0.0009*1.0e-5	H=1
SBAFDEP vs AGOB	0.0005*1.0e-5	H=1
SBAFDEP vs POBE	0.0004*1.0e-5	H=1
SBAFDEP vs DREP	0.0001*1.0e-5	H=1
SBAFDEP vs DF-D	0.0006*1.0e-5	H=1
SBAFDEP vs GASEN	0.0007*1.0e-5	H=1
SBAFDEP vs GSOSEN	0.0060*1.0e-5	H=1
SBAFDEP vs MOAG	0.0004*1.0e-5	H=1
SBAFDEP vs EP-FP	0.0002*1.0e-5	H=1
SBAFDEP vs RRE	0.4441*1.0e-5	H=1
SBAFDEP vs DivP	0.3175*1.0e-5	H=1
SBAFDEP vs SCG-P	0.0001*1.0e-5	H=1
SBAFDEP vs SDAcc	0.1711*1.0e-5	H=1

Remark: H = 1 shows that hypothesis H0 is rejected, i.e., SBAFDEP significantly improves the classification performance of other state-of-the-art pruning approaches at 5% significance level, respectively.

To test whether the difference between the proposed approach and other approaches is significant or not, a t-test method at a significance level of 0.05 was used in this work. If the p-value was less than 0.05, then the hypothesis H0 was rejected, and the difference between the two approaches was significant, and vice versa. The p-values produced in the test are presented in Table 7, which indicates that the proposed method attains a better result with significant differences in all fourteen cases. In addition, the trade-off between the average sizes of the pruned ensemble and the average accuracy is shown in Fig. 3. It is easy to see from Fig. 3 that, SBAFDEP outperforms other pruning approaches.

Fig. 3

Relationship between ensemble size and average performance of different approaches.

5.2 Parameter analysis

In the proposed SBAFDEP, SBAFSA is used to search for the optimal sub-ensemble from the retaining classifiers after pre-pruning. Thus, the performance of the proposed SBAFSA needs to be evaluated, and it is compared with the following heuristic algorithms: SbAFSA (Simplified binary Artificial Fish Swarm Algorithm) [19], IBAFSA (Improved Binary Artificial Fish Swarm Algorithm) [21], BAFSA (Binary Artificial Fish Swarm Algorithm) [18], IAFSA (Improved Artificial Fish Swarm Algorithm) [22], AFSA [20], IDGSO [30], and GA [15]. The following experiments using an initial pool of 200 classifiers were implemented on Column and Spambase datasets, which are displayed in Fig. 4. After pre-pruning according to the double-fault measure, the above eight heuristic algorithms were employed to search for the optimal sub-ensemble. These heuristic algorithms were implemented as described in their respective papers.

From Fig. 4, it is easy to see that SBAFSA achieves a faster convergence speed than the other seven approaches, and we can also find that the performance achieved by SBAFSA, first rise up and stay stable. When the number of iterations is over 500, the performance of SBAFSA cannot be significantly improved. We advise that the iterations is set at 500. We can see from Fig. 5.1 that, when the visual scope is 25, SBAFSA performs the best. Thus, we advise that the suitable visual scope is set at half its size of classifiers after pre-pruning. As the increase of the population size, the classification accuracy attained by SBAFSA mounts and levels off, which is shown in Fig. 5.2. We advise that the population size is set at 25.

Fig. 4

Relationship between the performance of heuristic algorithms and iterations on the Column and Spambasedatasets.

Fig. 5

Relationship between the performance of heuristic algorithms and parameters on the Column dataset.

6 Conclusions

In this work, we proposed a novel ensemble pruning approach, named spread binary artificial fish swarm algorithm combined with a double-fault measure for ensemble pruning (SBAFDEP). We then search for the final ensemble using SBAFSA after pre-pruning based on a double-fault measure. Heuristic algorithms tend to do well when it comes to finding the tradeoff between the accuracy and the diversity of classifiers. Ensemble pruning is an NP complete problem, but it is hard for heuristic algorithms to do an exhaustive search. On one hand, a double-fault measure can be used to pre-prune the classifiers in a generated pool, which can downsize the pool, and dramatically reduce the computational burdens of ensemble pruning. On the other hand, the final ensemble extracted from the retaining classifiers after pre-pruning is efficiently achieved using the proposed SBAFSA. Therefore, the combination SBAFSA and double-fault measure can tackle the ensemble pruning problem. Experimental comparisons on 25 UCI datasets demonstrate that SBAFDEP outperforms other state-of-the-art pruning approaches, and that the proposed SBAFSA performs better than other binary heuristic algorithms with respect to the convergence speed and precision.

In future work, we will try to employ other diversity measures for pre-pruning, and then search for the final ensemble using heuristic algorithms. We believe that these combinations of diversity measures and heuristic algorithms can generate promising results, which can provide new research ideas for ensemble pruning.

Footnotes

Acknowledgement

This work was supported by the National Nature Science Foundation of China under Grant No. 91546108, No. 71490725, No. 71271071, No. 71301041, and No. 61806068, the National Key Research and Development Plan under Grant No. 2016YFF0202604, the Natural science foundation of Huzhou under Grant No. 2018YZ11, and the Open Research Fund Program of Key Laboratory of Process Optimization and Intelligent Decision-making (Hefei University of Technology), Ministry of Education. The authors would like to thank the reviewers for their comments and suggestions.

References

Han

and Liu

, Ensemble of extreme learning machine for remote sensing image classification, Neurocomputing149 (2015), 65–70.

, Hu

and Yu

, A framework of multiple kernel ensemble learning for classification using two-stage feature selection method, Journal of Intelligent & Fuzzy Systems33(5) (2017), 2737–2747.

Galar

, Fernandez

, Barrenechea

et al., Ordering-based pruning for improving the performance of ensembles of classifiers in the framework of imbalanced datasets, Information Sciences354 (2016), 178–196.

Martınez-Muñoz

, Hernández-Lobato

and Suárez

, An analysis of ensemble pruning techniques based on ordered aggregation, IEEE Transactions on Pattern Analysis and Machine Intelligence31(2) (2009), 245–259.

Dai

, Zhang

and Liu

, A new reverse reduce-error ensemble pruning algorithm, Applied Soft Computing28 (2015), 237–249.

Bashbaghi

, Granger

, Sabourin

et al., Dynamic ensembles of exemplar-SVMs for still-to-video face recognition, Pattern Recognition69 (2017), 61–81.

Antipov

, Baccouche

, Berrani

S.A.

et al., Effective training of convolutional neural networks for face-based gender and age prediction, Pattern Recognition72 (2017), 15–15.

Han

, Yang

and Jiang

, An Extreme Learning Machine based on Cellular Automata of edge detection for remote sensing images, Neurocomputing198 (2016), 27–27.

Zhou

Z.H.

, Jiang

, Yang

Y.B.

et al., Lung cancer cell identification based on artificial neural network ensembles, Artificial Intelligence in Medicine24(1) (2002), 25–36.

10.

Petinrin

O.O.

and Saeed

, Bioactive molecule prediction using majority voting-based ensemble method, Journal of Intelligent & Fuzzy Systems35(1) (2018), 383–392.

11.

Kavitha

, Karthikeyan

and Maybell

P.S.

, An ensemble design of intrusion detection system for handling uncertainty using Neutrosophic Logic Classifier, Knowledge-Based Systems28 (2012), 88–96.

12.

Tang

E.K.

, Suganthan

P.N.

and Yao

, An analysis of diversity measures, Machine Learning65 (2006), 247–271.

13.

Krogh

and Vedelsby

, Neural network ensembles, cross validation, and active learning, Advances In Neural Information Processing Systems1995, pp. 231–238.

14.

Yang

, Yin

X.C.

, Hao

H.W.

et al., Classifier ensemble with diversity: effectiveness analysis and ensemble optimization, Acta Automatica Sinica40(4) (2014), 660–674.

15.

Zhou

Z.H.

, Wu

and Tang

, Enbling neural networks: many could be better than all, Artificial Intelligence137(1) (2002), 239–263.

16.

H.J.

, An

C.L.

, Zheng

E.H.

and Lu

, Dissimilarity based ensemble of extreme learning machine for gene expression data classification, Neurocomputing128 (2014), 22–30.

17.

, Shao

and Qian

, An optimizing method based on autonomous animates: fish swarm algorithm, System Engineering Theory and Practice22(11) (2002), 32–38.

18.

Singhal

P.K.

, Naresh

and Sharma

, Binary fish swarm algorithm for profit-based unit commitment problem in competitive electricity market with ramp rate constraints, IET Generation, Transmission & Distribution9(13) (2015), 1697–1707.

19.

Azad

M.A.K.

, Rocha

A.M.A.C.

and Fernandes

E.M.G.P.

, A simplified binary artificial fish swarm algorithm for 0–1 quadratic knapsack problems, Journal of Computational and Applied Mathematics259 (2014), 897–904.

20.

Chen

, Zhu

and Xu

, Finding rough set reducts with fish swarm algorithm, Knowledge-Based Systems81 (2015), 22–29.

21.

Azad

M.A.K.

, Rocha

A.M.A.C.

and Fernandes.

E.M.G.P.

, Improved binary artificial fish swarm algorithm for the 0–1 multidimensional knapsack problems, Swarm and Evolutionary Computation14 (2014), 66–75.

22.

Luan

X.Y.

, Li

Z.P.

and Liu

T.Z.

, A novel attribute reduction algorithm based on rough set and improved artificial fish swarm algorithm, Neurocomputing174 (2016), 522–529.

23.

Ykhlef

and Bouchaffra

, An efficient ensemble pruning approach based on simple coalitional games, Information Fusion34 (2017), 28–42.

24.

Martınez-Muñoz

, Suárez

Aggregation ordering in bagging, in: Proceedings of the IASTED International Conference on Artificial Intelligence and Applications, 2004, pp. 258–263.

25.

Martınez-Muñoz

, Suárez

Pruning in ordered bagging ensembles, in: Proceedings of the Twenty-third International Conference on Machine learning, 2006, pp. 609–609.

26.

Margineantu

D.D.

and Dietterich.

T.G.

, Pruning adaptive boosting, In Proceedings of the fourteenth International Conference on Machine Learning97 (1997), 211–218.

27.

, Wu

, Zhu

et al., Ensemble pruning via individual contribution ordering, in: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2010, pp. 871–880.

28.

Guo

and Boukir

, Margin-based ordered aggregation for ensemble pruning, Pattern Recognition Letters34 (2013), 609.

29.

Guo

, Liu

, Li

et al., Margin & diversity based ordering ensemble pruning, Neurocomputing275 (2018), 237–246.

30.

Z.W.

, Zhang

and Ni

L.P.

, Haze forecast method of selective ensemble based on glowworm swarm optimization algorithm, Pattern Recognition and Artificial Intelligence29(2) (2016), 143–153.

31.

Rokach

, Collective-agreement-based pruning of ensembles, Computational Statistics and Data Analysis53(4) (2009), 1015–1026.

32.

Lazarevic

and Obradovic

, Effective pruning of neural network classifier ensembles, International Joint Conference on Neural Networks2001, pp. 796–801.

33.

Bakker

and Heskes

, Clustering ensembles of neural network models, Neural Network16 (2), (2003), 261–269.

34.

Giacinto

, Roli

and Fumera

, Design of effective multiple classifier systems by clustering of classifiers, International Conference on Pattern Recognition2000, pp. 160–163.

35.

Zhou

, Zhao

and Wang

, An effective ensemble pruning algorithm based on frequent patterns, Knowledge-Based Systems56 (2014), 79–85.

36.

Cavalcanti

G.D.C.

, Oliveira

L.S.

, Moura

T.J.M.

et al., Combining diversity measures for ensemble pruning, Pattern Recognition Letters74 (2016), 38–45.

37.

Dai

, Ye

and Liu

, Considering diversity and accuracy simultaneously for ensemble pruning, Applied Soft Computing58 (2017), 75–91.

38.

Feng

, Qiu

M.H.

, Wang

Y.X.

et al., A fast divisive clustering algorithm using an improved discrete particle swarm optimizer, Pattern Recognition Letters31(11) (2010), 1216–1225.

39.

Cheng

, Ni

and Zhu

, Attribute reduction method combined with spread binary glowworm swarm optimization and rough set, Systems Engineering and Electronics38(10) (2016), 2450–2456.

40.

Sang

and Pan

, A discrete invasive weed optimization algorithm for the integrated lot-streaming flow scheduling problem, Control Theory and Application32(2) (2015), 246–250.

41.

Breiman

, Bagging predictors, Machine Learning24 (1996), 123–140.

42.

, Yu

, Zhou

Z.H.

Diversity regularized ensemble pruning, Machine Learning and Knowledge Discovery in Databases2002, pp. 330–345.

Spread binary artificial fish swarm algorithm combined with double-fault measure for ensemble pruning

Abstract

Keywords

1 Introduction

1.1 Related work

1.2 Contributions and outline

2 Diversity measures

Table 1 Contingency table for two classifiers f i (x k ) = y k f i (x k ) ≠ y k f j (x k ) = y k a b f j (x k ) ≠ y k c d

3.1 Mapping operation

4 SBAFDEP

4.1 Initial pool of classifiers

4.2 Pre-pruning based on double-fault measure

4.3 Pruning based on SBAFSA

Footnotes

Acknowledgement

References

Table 1
Contingency table for two classifiers

f_i (x_k) = y_k f_i (x_k) ≠ y_k

f_j (x_k) = y_k a b

f_j (x_k) ≠ y_k c d