Ensemble based on feature projection and under-sampling for imbalanced learning

Abstract

Imbalanced problem is concerned with the performance of classifiers on the data set with severe class imbalance distribution. For two-class, the examples can be categorized into majority class or minority class, and the cost of misclassifying minority class examples is often much higher than the contrary cases. However, traditional classifiers do not work well on the imbalanced problem due to the assumption that the number of each class examples is similar to each other. To handle these problems, this paper proposes a simple but effective ensemble learning method based on feature projection and under-sampling (EFPUS). EFPUS learns an ensemble through the following two steps: (1) under-sampling several subset from majority class and learning a novel projection matrix from each subset, and (2) constructing new training sets by projecting the original training set to different spaces defined by the matrixes and learning a class-imbalance oriented classifier from each new training set. For the first step, feature projection and under-sampling mainly aim to improve the diversity between ensemble members. With respect to the second step, the base models can be learned by any traditional class-imbalance oriented learning method such as USBagging, SMOTEBoost and BalanceCascade. Experimental results show that, compared with other state-of-the-art methods, EFPUS shows significantly better performance on measures of g-mean, f-measure, AUC, recall and accuracy.

Keywords

Ensemble learning feature projection under-sampling imbalanced learning

1. Introduction

Class imbalance problems are often encountered in real world, rising from medical diagnosis, risk management, fraud detection and other domain applications [1]. For two-class, the class imbalance data is characterized as having much less examples of one class (minority class) than the other (majority class), and the cost of misclassifying minority class examples is much expensive than the contrary cases [2, 3, 4]. However, most conventional classification methods try to achieve high accuracy with the assumption of balanced class distribution, i.e., the number of examples in any class is similar to each other, which leads to the fact that the minority class examples are often ignored and misclassified to majority class.

Many approaches have been proposed to handle the class imbalance problem, and sampling technique and ensemble learning are two candidates of the most often used methods. Sampling techniques including under-sampling and over-sampling are often used to pre-process data before learning models and the objective of sampling techniques is to balance the class distribution to eliminate the harms of imbalanced problem by sampling the data space. Under-sampling techniques try to eliminate the harms of imbalanced distribution by removing the intrinsic examples in the majority class. Random under-sampling (RUS) is the simplest yet very effective method, which randomly eliminating majority class examples [5, 6]. On the contrary to under-sampling, over-sampling techniques including randomly over-sampling (ROS) and SMOTE aim to create new minority class examples, where ROS randomly duplicates the minority class examples and SMOTE generates new synthetic examples along the line between the minority examples and their selected nearest neighbors [7, 8].

Ensemble learning, which has often used to solute challenging issues such as image recognition [9, 10, 11, 12], is an effective technique for dealing with imbalanced problem [13, 14, 15, 16, 17]. Classifier ensembles for imbalanced data can be categorized into two levels: iterative based ensembles and parallel based ensembles. Boosting is the most well-known iterative based ensemble and often used to solve class-imbalance problem by embedding techniques for data preprocessing into its learning process. In such a manner, these methods alter and bias the weight distribution to train the next classifier toward the minority class, examples including SMOTEBoost [15] and RUSBoost [16] algorithms. Parallel based ensembles refer to ensemble models in which each base classifier can be trained in parallel. As a member of parallel based ensembles, bagging is often combined with other class-imbalance oriented methods to handle imbalanced problem. Examples includes OverBagging, UnderBagging and UnderOverBagging [14], where OverBagging learns each base classifier on a data set obtained by over-sampling (or generating from) minority class. On the contrary to OverBagging, UnderBagging learns each member on under-sampled subset from majority class. UnderOverBagging makes use of both over-sampling and under-sampling techniques to improve ensemble diversity.

In this paper, we propose a novel ensemble method based on feature projection and under-sampling (EFPUS) to improve the performance of conventional class-imbalance oriented methods by pre-processing the training set. EFPUS is a parallel based ensemble, which constructs several new training sets and trains a base class-imbalance oriented classifier that can be a single model or an ensemble using each training set. These training sets are obtained though the following three steps: under-sampling several subsets from majority class, learning a feature projection matrix from each subset, and projecting the original training set to new spaces defined by the projection matrixes to obtain new training sets. Therefore, these new training sets are also imbalanced data sets. EFPUS mainly aims to obtain larger diversity between ensemble members, and thus improves the ensemble performance on imbalanced data. In order to further improve the diversity between ensemble members, following rotation forests [18], the original feature set is randomly split into $K$ subsets, Principal Component Analysis (PCA) is applied to each subset and a full feature set is reconstructed as a projection matrix. The main contributions of this paper are as follows:

•
Employing both feature projection and under-sampling technique to deal with imbalanced data to generate new training sets with diversities;
•
Proposing a novel ensemble method based on feature projection and under-sampling (EFPUS) to improve ensemble performance on imbalanced data sets;
•
Extensive experiments are conducted and the results show that, compared with other state-of-the-art classification methods, EFPUS shows much better performance on measures of g-mean, f-measure, AUC and recall.

The rest of this paper is organized as follows: after presenting the strategies of handling imbalanced problem in Section 2, Section 3 describes the proposed ensemble method, Section 4 presents the experimental results and, finally, Section 5 concludes this work.
2. Strategies to handle imbalanced problem

The imbalanced problems rise from the scare representation of the most important examples, which leads to the fact that learned models tend to focus more on normal examples, overlooking the minority class examples. Numerous techniques have been developed to deal with the problem of class-imbalance data sets, which can mainly categorized into the following three groups: data level, algorithm level and ensemble learning.

2.1 Data level

These techniques in data level try to rebalance the given imbalanced data through resampling data space such that conventional learning methods pay more attention on minority class. Reported studies of data level can be further subdivided two types: resampling and weighting the data space. Resampling techniques aim to alleviate the effect of class imbalance distribution through sampling data space to rebalance the corresponding imbalanced data set. Commonly used sampling techniques are falling to the following three categories: over-sampling methods, under-sampling methods and hybrid method. Over-sampling techniques try to create new minority class examples to eliminate the harms of imbalanced problem. Randomly duplicating the minority samples and SMOTE [6, 7] are the two most popular examples of over-sampling techniques. Under-sampling techniques such as Random Under-sampling (RUS), the simplest yet most effective method, tries to eliminate the harms of class-imbalance distribution through removing the examples of the majority class. The hybrid method is a combination of over-sampling and under-sampling. Japkowicz [19] discussed the sampling technique and observed that the technique was very effective and furthermore, she noted that using sophisticated sampling techniques does not provide any clear advantage in solving the class imbalance problem. Moreover, the researches carried out by Mani and Zhang [20] showed that the random under-sampling approaches often outperform complicated under-sampling methods. Batista et al. [21] and Xie and Qiu [22] both showed that over-sampling usually perform better than under-sampling. Estabrooks et al. [23] suggested that a combination of over-sampling and under-sampling might be more effective to solve the class imbalance problem. The strategies of weighting the data space aim to modify the training set distribution using information concerning the misclassification costs [24], cost-sensitive methods [25] and an ensemble of SVM with asymmetric misclassification costs [26].

2.2 Algorithm based level

Algorithm level approaches (also called internal) aims to improve the learning ability of existing classification algorithms to improve the classification performance for imbalanced data [26, 27, 28]. These methods require special knowledge of both the corresponding classifier and the application domain, comprehending why the classifier fails when the class distribution is uneven. Examples include enhancing the discriminatory power of the classifiers using kernel transformation [29, 30], converting the training goal to a well-deigned objective function that more severely penalize errors on minority class [28, 31, 32] and clustering algorithms [33, 34].

2.3 Ensemble learning

Classifier ensembles, also named multiple classifier systems, are known to enhance the performance of a single classifier by combining several base classifiers [35]. According to Galar et al. [36], classifier ensembles for class imbalance problem can be grouped into three categories: 1) bagging-, 2) boosting-, and 3) hybrid-based approaches.

•
Many bagging-based ensembles such as OverBagging, UnderBagging and UnderOverBagging [14] have been proposed to deal with data imbalance due to their simplicity and good generalization ability, where OverBagging learns each base classifier on a data set obtained by over-sampling (or generating from) minority class. On the contrary to OverBagging, UnderBagging learns each member on under-sampled subset from majority class. UnderOverBagging makes use of both over-sampling and under-sampling techniques to improve ensemble diversity.
•
Boosting-based ensembles embed techniques for data preprocessing into boosting algorithms. In such a manner, these methods alter and bias the weight distribution used to train the next classifier toward the minority class, examples including SMOTEBoost [15, 37] and RUSBoost [16] algorithms. SMOTEBoost introduces synthetic examples using the SMOTE data preprocessing algorithms. The weights of the new examples are proportional to the total number of examples in the new data set. RUSBoost performs similarly to SMOTEBoost, but it removes examples from the majority class by randomly under-sampling the data set.
•
A hybrid ensemble is an ensemble of ensembles, and EasyEnsemble and BalanceCascade are the two specific examples of this category [13]. Both EasyEnsemble and BalanceCascade use bagging as the main ensemble learning method, but in spite of training a classifier for each new bag, they train each bag using Adaboost. The difference between the two methods is the way in which they treat the examples. EasyEnsemble simply under-samples several subsets and learns an Adaboost on each subset. Hence, all the Adaboosts can be trained in parallel. BalanceCascade trains each Adaboost sequentially, where in each step the majority class examples which are correctly classified by the current trained learners are removed from further consideration.

3. Ensemble based on feature projection and under-sampling

3.1 Leaning process

This section proposes a novel ensemble method based on feature projection and under-sampling (EFPUS) for class imbalance problem. EFPUS learns several projection matrixes and using each matrix, obtains new training sets by projecting the training set to the space defined by the matrixes and trains a model on each new training set using conventional class-imbalance oriented learning methods. Therefore, the key of EFPUS is how to construct new training sets for learning base classifiers.

Let $\bm{x}=[x_{1},x_{2},...,x_{n}]^{T}$ be a sample described by $n$ features, and let $\bm{X}_{\textit{min}}$ and $\bm{X}_{\textit{maj}}$ be the minority class set and majority class set respectively. Denote $H_{i}$ as the base ensemble learned from the projected data set $D_{i}$ and denote $F$ as the original feature set. We carry out the following steps to construct the training set $D_{i}$ for learning $H_{i}$ .

1)
Randomly sample a subset $\bm{X}_{i,\textit{maj}}$ from $\bm{X}_{\textit{maj}}$ without replacement, $|\bm{X}_{i,\textit{maj}}|=|\bm{X}_{\textit{min}}|$ and $\bm{X}_{i}=\bm{X}_{i,\textit{maj}}\cup\bm{X}_{\textit{min}}$ .
2)
Split $F$ randomly into $K=n/L$ subsets, namely ${\{}F_{i,j}|j=1,...,K{\}}$ . The subsets may be disjoint or intersecting. To maximize the chance for high diversity, we chose disjoint subsets.
3)
For every such subset $F_{i,j}$ , draw a bootstrap sample $\bm{X}_{i,j}$ with size 50 percent of the data points in $\bm{X}_{i}$ . Run PCA using $F_{i,j}$ and $\bm{X}_{i,j}$ , and store the coefficients of the principal components, $a_{i,j}^{1}$ , $a_{i,j}^{2}$ , …, $a_{i,j}^{L}$ .
4)
Organize the obtained vectors with coefficients in a sparse projection matrix $\bm{R}_{i}$ , then

$\displaystyle\bm{R}_{i}=\left[{{\begin{array}[]{*{20}c}{a_{i,1}^{1},a_{i,1}^{2% },\ldots,a_{i,1}^{L}}&{\left[0\right]}&\ldots&{\left[0\right]}\\ {\left[0\right]}&{a_{i,2}^{1},a_{i,2}^{2},\ldots,a_{i,2}^{L}}&\ldots&{\left[0% \right]}\\ \vdots&\vdots&\ddots&\vdots\\ {\left[0\right]}&{\left[0\right]}&\ldots&{a_{i,n/L}^{1},a_{i,n/L}^{2},...,a_{i% ,n/L}^{L}}\\ \end{array}}}\right]$ (1)
5)
Obtain $D_{i}=D_{i,\textit{maj}}\cup D_{i,\textit{min}}$ , where $D_{i,\textit{maj}}=\bm{X}_{\textit{maj}}\bm{R}_{i}$ and $D_{i,\textit{min}}=\bm{X}_{\textit{min}}\bm{R}_{i}$ .

The method of constructing new training sets is similar to rotation forest [18], with difference in class manipulation and the sampling techniques, i.e., HE learns projection matrixes focusing more on minority class. The process of EFPUS is as shown in Algorithm 1. The

Algorithm 1 EFPUS

Training Phase

Input:

$\bm{X}_{\textit{maj}}$ – the majority class set

$\bm{X}_{\textit{min}}$ – the negative class set

$M$ – the number of base classifiers

$K$ – the number of feature subsets

Output: a hybrid-based ensemble $\bm{H}$ for the two class imbalance problem

Begin:

1. for $i=$ 1 to $M$ do

2. Randomly sample a subset $X_{i,\textit{maj}}$ from $\bm{X}_{\textit{maj}}$ without replacement, $|\bm{X}_{i,\textit{maj}}|=|\bm{X}_{\textit{min}}|$ ;

3. $\bm{X}_{i}=\bm{X}_{i,\textit{maj}}\cup\bm{X}_{\textit{min}}$

4. Split feature set $F$ into subsets $F_{i,j}$ , $j=$ 1, 2, …, $K$ ;

5. for $j=$ 1 to $n/K$ do

6. Randomly draw a subset $\bm{X}_{i,j}$ of $\bm{X}_{i}$ without replacement, with size 50% of $\bm{X}_{i}$ ;

7. Apply PCA on $\bm{X}_{i,j}$ and $F_{i,j}$ to obtain projection matrix $\bm{R}_{i,j}$ ;

8. end for

9. Arrange the $\bm{R}_{i,j}$ in a projection matrix $\bm{R}_{i}$ as in Eq. (1);

10. $D_{i,\textit{maj}}=\bm{X}_{\textit{maj}}\bm{R}_{i}$ ,

11. $D_{i,\textit{min}}=\bm{X}_{\textit{min}}\bm{R}_{i}$ ,

12. Learn $H_{i}$ using $D_{i,\textit{maj}}\cup D_{i,\textit{min}}$

13. end for

3.2 Discussion

Algorithm 1 EFPUS
	$\bm{X}_{\textit{maj}}$ – the majority class set
	$\bm{X}_{\textit{min}}$ – the negative class set
	$M$ – the number of base classifiers
	$K$ – the number of feature subsets
Output: a hybrid-based ensemble $\bm{H}$ for the two class imbalance problem
	Begin:
	1. for $i=$ 1 to $M$ do
	2. Randomly sample a subset $X_{i,\textit{maj}}$ from $\bm{X}_{\textit{maj}}$ without replacement, $\|\bm{X}_{i,\textit{maj}}\|=\|\bm{X}_{\textit{min}}\|$ ;
	3. $\bm{X}_{i}=\bm{X}_{i,\textit{maj}}\cup\bm{X}_{\textit{min}}$
	4. Split feature set $F$ into subsets $F_{i,j}$ , $j=$ 1, 2, …, $K$ ;
	5. for $j=$ 1 to $n/K$ do
	6. Randomly draw a subset $\bm{X}_{i,j}$ of $\bm{X}_{i}$ without replacement, with size 50% of $\bm{X}_{i}$ ;
	7. Apply PCA on $\bm{X}_{i,j}$ and $F_{i,j}$ to obtain projection matrix $\bm{R}_{i,j}$ ;
	8. end for
	9. Arrange the $\bm{R}_{i,j}$ in a projection matrix $\bm{R}_{i}$ as in Eq. (1);
	10. $D_{i,\textit{maj}}=\bm{X}_{\textit{maj}}\bm{R}_{i}$ ,
	11. $D_{i,\textit{min}}=\bm{X}_{\textit{min}}\bm{R}_{i}$ ,
	12. Learn $H_{i}$ using $D_{i,\textit{maj}}\cup D_{i,\textit{min}}$
	13. end for

The running time of EFPUS(the proposed method) is mainly dominated by learning the feature projection matrix (lines 5 $\sim$ 8, Algorithm 1) and base classifiers (line 12). Using PCA to learn a sub-projection matrix (line 7), which is complexity of $O(L^{3})$ , and therefore learning a projection matrix consumes $O(L^{3}n/K)=O(nK^{2})$ , where $n$ is the feature number of the data set. We set $L=n/K=3$ equal to 3 in experiments and then learning each projection matrix consumes $O(n)$ , and therefore the time complexity of EFPUS is same to other traditional ensemble learning methods.

Dietterich stated that a necessary and sufficient condition for an ensemble of classifiers to be more accurate than any of its individual members is if the classifiers are accurate and diverse [38]. Here, EFPUS is mainly used to improve ensemble diversity through feature projection and under-sampling.

Using the projection heuristic, the number of different partitions of the feature set with $n$ features into $K$ subsets of size $L$ can be calculated by

$\displaystyle T=\frac{n!}{\left(K\right)!\left({L!}\right)^{K}}=\frac{n!}{% \left(K\right)!\left(L\right)^{K}}$ (2)

where $L=n/K$ . If the ensemble consists of $M$ classifiers, assuming that each partition of the feature set is equally probable, the probability that all classifiers will be different is

$\displaystyle P\left({\textit{different classifiers}}\right)=\frac{T!}{\left({% T-M}\right)!T^{M}}$ (3)

For instance, the chance of all different classifiers in an ensemble of $M=$ 50 classifiers for $n=$ 9 is less than 0.01, and thus an extra randomization of the ensemble is meaningful, Therefore, there is a need for. This can be done by applying feature projection learning method to a bootstrap sample (refer to line 6 in Algorithm 1). Following [18], we use Principal Component Analysis (PCA) to determine its feasibility and find out whether it does contribute to increased precision and diversity.

Under-sampling is a class-imbalance learning method which uses only a subset of majority class and thus is very efficient for imbalanced learning problem [26, 38]. Under-sampling technique contributes to the diversity of ensemble members in the processes of learning projection matrixes (line 2, Algorithm 1) and the larger the ratio between the size of negative set and positive set, the larger the diversity of ensemble members. When the number of majority class examples is equal to that of minority class, the diversity of ensemble members is determined by the process of learning projection heuristic. Under-sampling can also make base learner bias towards minority class through under-sampling majority class such that the learned projection matrixes capture the minority class characterization.

4. Experiments

4.1 Experimental setup

In order to facilitate the experiment, we design a platform named Loyang Spoon (LySpoon) based on python (Please contact hpguo_cm@163.com for the related source code).

Twenty-three very imbalanced data sets were selected from the KEEL imbalanced data sets [39]. Many data sets have some missing attribute values, such as breast-cancer, hepatitis and sick. C4.5, the base classifier adopted in this paper, handles missing attribute values automatically. More details about data sets see Table 1, where #Attrs, #Exps, and #IR are the attribute number, example number, and imbalance ratio defined as the ratio between the size of minority class set and that of majority class set respectively.

Table 1
The description of the data sets used in this paper

ID	Name	#Attrs	#Exps	#IR
id1	wisconsin	9	683	0.0186
id2	iris0	4	150	0.0200
id3	glass0	9	214	0.0206
id4	yeast1	8	1484	0.0246
id5	vehicle2	18	846	0.0288
id6	vehicle1	18	846	0.0290
id7	vehicle3	18	846	0.0299
id8	glass-0-1-2-3_vs_4-5-6	9	214	0.0320
id9	vehicle0	18	846	0.0325
id10	ecoli1	7	336	0.0336
id11	new-thyroid1	5	215	0.0514
id12	ecoli2	7	336	0.0546
id13	segment0	19	2308	0.0602
id14	glass6	9	214	0.0638
id15	ecoli3	7	336	0.0860
id16	ecoli-0-2-3-4_vs_5	7	202	0.0910
id17	yeast-0-2-5-7-9_vs_3-6-8	8	1004	0.9140
id18	yeast-0-2-5-6_vs_3-7-8-9	8	1004	0.9140
id19	ecoli-0-1_vs_2-3-5	7	244	0.0917
id20	ecoli-0-2-6-7_vs_3-5	7	224	0.0918
id21	ecoli-0-1-4-7_vs_2-3-5-6	7	336	0.1059
id22	ecoli-0-1_vs_5	6	240	0.1100
id23	ecoli-0-1-4-6_vs_5	6	280	0.1300

To evaluate the performance of the proposed method EFPUS, 10-fold cross-validation strategy was conducted: each data set was divided into ten folds with similar number of examples, and for each fold, the other nine folds are used to train a model and the current fold is used to test the model [41]. We repeated ten times of the 10-fold cross-validation, and therefore, 100 learners were trained by a learning algorithm on each data set.

We selected RUSC45, SMOTEC45, RUSBagging, SMOTEBagging, RUSBoost, SMOTEBoost, EasyEnsemble and BalanceCascade as comparing methods to test the performance of EFPUS:

•

RUSC45 (RC) [38] pre-processed training sets using random under-sampling technique to obtain relatively balanced data set and then learned a model using C4.5 on the balanced set.

•

EFPUS-RUSC45 (EFPUS-RC). RUSC45 (RC) was used as base learners of EFPUS and $M$ was set to be 40 (refer to Algorithm 1).

•

SMOTEC45 (SC) [38] pre-processed training set using SMOTE [7] to obtain relatively balanced data set and then learned a model using C4.5 on the balanced set.

•

EFPUS-SMOTEC45 (EFPUS-SC). SMOTEC45 (SC) was used as the base learner and $M$ was set to be 40 (refer to Algorithm 1).

•

RUSBagging (RB) [14] learned each member on under-sampled subset from majority class. C4.5 was selected as the base learner. The number of iterations was set to be 40.

•

EFPUS-RUSBagging (EFPUS-RB). RUSBagging (RB) was treated as the base learner of EFPUS and C4.5 is selected as the base learner for RUSBagging. We set $M=$ 8 (refer to Algorithm 1) and set the number of iterations of RUSBagging to be 5. Therefore, 40 models were constructed by C4.5.

•

SMOTEBagging (SB) [14] learned each member on the rebalanced data by over-sampling training set using SMOTE and C4.5 was selected as the base classifier learner. The number of iterations was set to be 40.

•

EFPUS-SMOTEBagging (EFPUS-SB) is similar to EFPUS-RB except that SMOTEBagging (SB) was used as the base learner.

•

RUSBoost (RBT) [16] combined under-sample technique into the boost learning process. The number of iterations was set to be 40, and for each iteration, a subset with size equal to minority class was sampled (without replacement) from majority class. Then, C4.5 was used to train a classifier using the subset and minority class set.

•

EFPUS-RUSBoost (EFPUS-RBT) treated RUSBoost (RBT) to lean base classifier and C4.5 as the base learner for RUSBoost. Similar to EFPUS-SB, we set $M=$ 8 and the number of iterations of RUSBoost to be 5.

•

SMOTEBoost (SBT) [15] used SMOTE to get new minority class examples. C4.5 was used to train base classifiers. The number of iterations was 40. The $k$ nearest neighbor parameter of SMOTE was 5. The amount of new data generated using SMOTE was the number of minority class examples.

•

EFPUS-SMOTEBoost (EFPUS-SBT) selects SMOTEBoost (SBT) as base classifier. C4.5 was used to train base classifiers of SBT. Similar to EFPUS_RBT, we set $M=$ 8 and the number of iterations of RUSBoost to be 5.

•

EasyEnsemble (EE) [13] samples $T$ subsets from the majority class, trains an Adaboost with $J$ weak learner using each of them. We selected C4.5 as weak classifiers and set $T=$ 4 and $J=$ 10.

•

EFPUS-EasyEnsemble (EFPUS-EE) used EasyEnsemble (EE) to learn $M$ classifiers. We set $M=$ 4, $T=$ 2 and $J=$ 5, and set C4.5 as the base learner of EE. Therefore, 40 models were conducted by C4.5.

•

BalanceCascade (BC) [13] is similar to EasyEnsemble except that it removes correctly classified major class examples of trained Adaboosts from further consideration. C4.5 was selected to train weak classifiers, and we set $T=$ 4 and $J=$ 10.

•

EFPUS-BalanceCascade (EFPUS-BC) was similar to EFPUS-EE except that it used BC as base classifier. We set $M=$ 4, $T=$ 2 and $J=$ 5. Therefore, 40 decision trees were constructed.

4.2 Experimental results

To evaluate the performance of EFPUS (the proposed method), RUSC45 (RC), SMOTEC45 (SC), RUSBagging (RB), SMOTEBagging (SB), RUSBoost (RBT), SMOTEBoost (SBT), EasyEnsemble (EE) and BalanceCascade (BC) were selected as comparing methods (more details refer to 4.1). The corresponding results were reported in nine tables, where Tables 2, 4, 6, 8 and 10 report the summary results of the comparing methods on the measures of g-mean, f-measure, AUC, recall, and accuracy, and Tables 3, 5, 7 and 9 show the ranks of the compared methods on g-mean, g-measure, AUC and recall respectively, where the last column of the nine tables shows the average performance of algorithms on each data set. In these tables with even numbers, EFPUS is compared to the corresponding base classifier and the value in parentheses next to a result is the $p$ -value of pair-wise $t$ -test. For example, on the first dataset (the second column), EFPUS-RC (the second row) is compared with RC (the third row) and the $p$ -value is zero ( $p=$ 0.00). If we use the pair-wise $t$ -test at $\alpha=$ 0.05 significance levels to test the performance difference between classifiers, then EFPUS-RC statistically outperforms RC on the first dataset since $p<\alpha$ . For simplicity, on the tables with even numbers, EFPUS comparing with RC, SC, RB, SB, RBT, SBT, EE and BC indicates that EFPUS-RC vs. RC, EFPUS-SC vs. SC, EFPUS-RB vs. RB, EFPUS-SB vs. SB, EFPUS-RBT vs. RBT, EFPUS-SBT vs. SBT, EFPUS-EE vs. EE and EFPUS-BC vs. BC. The ranks of these methods on measures of g-mean, f-measure, AUC and recall shown in the these tables with odd numbers are calculated as follow [41, 42]: on a data set, the best performing algorithm gets the rank of 1.0, the second best rank 2.0, and so on. In case of ties, average ranks are assigned.

Tables 2 and 3 report the summary results and the ranks of the 16 algorithms on the measure of g-mean respectively. We statistically compared the performance of algorithms using pair-wise $t$ -test at 0.05 significance levels. From Table 2, EFPUS statistically outperforms RC, SC, RB, SB, RBT, SBT, EE and BC on most of the 23 datasets. Specifically, EFPUS statistically outperforms the corresponding comparing algorithms on 23, 21, 16, 17, 22, 19, 16 and 17 out of the 23 datasets respectively. The ranks of EFPUS on most of the datasets are much smaller than other methods, and EFPUS-RC ranks first with the average rank of 2.52, followed by EFPUS-RB with average rank of 2.83, EFPUS-RBT with 4.96, EFPUS-BC (5.48), EFPUS-EE (5.61), EFPUS-SC (6.43), EFPUS-SB (7.11), RB (7.15), EFPUS-SBT (7.26), EE (10.28), BC (10.54), SB (11.89), RBT (13.04), RC (13.35), SBT (13.43) and SC (14.11). Besides, we also observe from Tables 2 and 3 that, the methods with random under-sampling technique outperform the ones with SMOTE.

Table 2
The performance of compared algorithms on g-mean

	d1	d2	d3	d4	d5	d6	d7	d8	d9	d10	d11	d12	d13	d14	d15	d16	d17	d18	d19	d20	d21	d22	d23	AVE
EFPUS-RC	0.976	0.906	1.000	0.819	0.727	0.984	0.819	0.814	0.937	0.972	0.903	0.984	0.996	0.922	0.885	0.902	0.911	0.806	0.880	0.828	0.840	0.875	0.881	0.894
RC	0.953 (0.00)	0.851 (0.00)	0.987 (0.00)	0.770 (0.00)	0.696 (0.00)	0.938 (0.00)	0.723 (0.00)	0.720 (0.00)	0.903 (0.00)	0.926 (0.00)	0.884 (0.00)	0.931 (0.00)	0.983 (0.00)	0.880 (0.00)	0.838 (0.00)	0.789 (0.00)	0.894 (0.00)	0.786 (0.00)	0.837 (0.00)	0.790 (0.00)	0.776 (0.00)	0.820 (0.02)	0.817 (0.00)	0.847
EFPUS-SC	0.977	0.896	1.000	0.832	0.724	0.984	0.773	0.773	0.928	0.974	0.881	0.990	0.993	0.922	0.808	0.845	0.884	0.739	0.828	0.827	0.830	0.871	0.859	0.876
SC	0.954 (0.00)	0.864 (0.00)	0.987 (0.00)	0.798 (0.00)	0.695 (0.00)	0.958 (0.00)	0.699 (0.00)	0.722 (0.00)	0.911 (0.06)	0.914 (0.00)	0.890 (0.17)	0.913 (0.00)	0.989 (0.00)	0.887 (0.00)	0.758 (0.00)	0.800 (0.02)	0.860 (0.00)	0.682 (0.00)	0.777 (0.01)	0.774 (0.00)	0.792 (0.01)	0.788 (0.00)	0.786 (0.00)	0.835
EFPUS-RB	0.977	0.904	1.000	0.814	0.729	0.981	0.814	0.812	0.939	0.970	0.902	0.970	0.994	0.929	0.885	0.897	0.910	0.803	0.891	0.825	0.840	0.904	0.892	0.895
RB	0.962 (0.00)	0.890 (0.04)	0.987 (0.00)	0.816 (0.87)	0.722 (0.01)	0.969 (0.00)	0.791 (0.00)	0.793 (0.00)	0.932 (0.12)	0.948 (0.00)	0.892 (0.00)	0.965 (0.38)	0.988 (0.00)	0.906 (0.00)	0.880 (0.04)	0.853 (0.02)	0.913 (0.12)	0.800 (0.28)	0.857 (0.00)	0.825 (0.96)	0.795 (0.00)	0.851 (0.02)	0.887 (0.66)	0.879
EFPUS-SB	0.977	0.893	1.000	0.829	0.727	0.981	0.777	0.773	0.928	0.971	0.881	0.985	0.992	0.915	0.783	0.861	0.881	0.744	0.833	0.827	0.823	0.850	0.849	0.873
SB	0.961 (0.00)	0.869 (0.00)	0.987 (0.00)	0.820 (0.20)	0.707 (0.00)	0.964 (0.00)	0.706 (0.00)	0.733 (0.00)	0.929 (0.97)	0.928 (0.00)	0.901 (0.00)	0.911 (0.00)	0.988 (0.00)	0.884 (0.00)	0.777 (0.67)	0.804 (0.00)	0.879 (0.67)	0.676 (0.00)	0.836 (0.84)	0.788 (0.01)	0.810 (0.35)	0.793 (0.00)	0.776 (0.00)	0.845
EFPUS-RB	0.975	0.901	1.000	0.802	0.713	0.973	0.790	0.781	0.927	0.959	0.897	0.975	0.994	0.919	0.880	0.872	0.909	0.801	0.886	0.850	0.838	0.887	0.873	0.887
RB	0.952 (0.00)	0.864 (0.00)	0.987 (0.00)	0.773 (0.00)	0.697 (0.00)	0.942 (0.00)	0.742 (0.00)	0.731 (0.00)	0.886 (0.00)	0.929 (0.00)	0.880 (0.00)	0.930 (0.00)	0.984 (0.00)	0.898 (0.00)	0.837 (0.00)	0.830 (0.03)	0.904 (0.11)	0.777 (0.00)	0.814 (0.00)	0.801 (0.00)	0.769 (0.00)	0.809 (0.00)	0.819 (0.00)	0.850
EFPUS-RBT	0.976	0.887	1.000	0.825	0.725	0.981	0.793	0.798	0.938	0.969	0.880	0.982	0.992	0.910	0.807	0.860	0.883	0.744	0.826	0.813	0.819	0.850	0.859	0.875
RBT	0.956 (0.00)	0.876 (0.10)	0.987 (0.00)	0.803 (0.01)	0.701 (0.00)	0.960 (0.00)	0.710 (0.00)	0.727 (0.00)	0.912 (0.00)	0.921 (0.00)	0.886 (0.39)	0.916 (0.00)	0.991 (0.26)	0.879 (0.00)	0.750 (0.00)	0.811 (0.01)	0.867 (0.00)	0.677 (0.00)	0.782 (0.03)	0.786 (0.12)	0.789 (0.07)	0.797 (0.02)	0.764 (0.00)	0.837
EFPUS-EE	0.976	0.903	1.000	0.802	0.709	0.970	0.798	0.780	0.935	0.957	0.892	0.973	0.992	0.918	0.874	0.884	0.906	0.795	0.862	0.837	0.843	0.880	0.869	0.885
EE	0.958 (0.00)	0.878 (0.00)	0.987 (0.00)	0.802 (1.00)	0.710 (0.78)	0.955 (0.00)	0.759 (0.00)	0.755 (0.00)	0.917 (0.00)	0.940 (0.00)	0.893 (0.84)	0.941 (0.00)	0.985 (0.00)	0.897 (0.00)	0.870 (0.54)	0.833 (0.00)	0.906 (0.98)	0.786 (0.07)	0.851 (0.40)	0.811 (0.03)	0.798 (0.01)	0.844 (0.04)	0.816 (0.00)	0.865
EFPUS-BC	0.975	0.906	1.000	0.797	0.710	0.971	0.796	0.784	0.937	0.958	0.897	0.970	0.992	0.916	0.880	0.894	0.906	0.795	0.875	0.843	0.832	0.886	0.867	0.883
BC	0.959 (0.00)	0.881 (0.00)	0.987 (0.00)	0.795 (0.81)	0.711 (0.60)	0.956 (0.00)	0.757 (0.00)	0.759 (0.00)	0.915 (0.00)	0.939 (0.00)	0.892 (0.30)	0.935 (0.00)	0.985 (0.00)	0.898 (0.00)	0.872 (0.23)	0.822 (0.00)	0.905 (0.75)	0.783 (0.01)	0.842 (0.02)	0.797 (0.00)	0.800 (0.01)	0.837 (0.03)	0.839 (0.13)	0.864

Table 3

The ranks of compared algorithms on g-mean

	d1	d2	d3	d4	d5	d6	d7	d8	d9	d10	d11	d12	d13	d14	d15	d16	d17	d18	d19	d20	d21	d22	d23	AVE
EFPUS-RC	5.0	4.5	5.0	2.5	1.5	1.0	1.0	3.5	2.0	1.0	3.0	1.5	1.0	2.5	1.5	1.0	2.0	1.0	3.0	4.0	2.5	5.0	3.0	2.52
RC	15.0	12.5	16.0	15.0	16.0	13.0	16.0	15.0	14.0	12.0	12.0	16.0	16.0	15.0	9.0	16.0	10.0	7.5	9.0	13.0	15.0	12.0	12.0	13.35
EFPUS-SC	2.0	4.5	1.0	5.0	1.5	9.0	8.5	8.5	1.0	13.5	1.0	6.0	4.0	2.5	11.0	9.0	10.0	13.0	12.0	5.5	6.0	6.0	7.5	6.43
SC	14.0	12.5	12.0	16.0	12.0	16.0	15.0	14.0	16.0	10.0	15.0	14.5	10.0	12.5	15.0	15.0	16.0	14.0	16.0	16.0	13.0	16.0	14.0	14.11
EFPUS-RB	2.0	4.5	7.0	1.0	4.0	2.0	2.0	1.0	4.0	2.0	7.5	3.0	2.5	1.0	1.5	2.0	3.0	2.0	1.0	7.5	2.5	1.0	1.0	2.83
RB	9.0	12.5	6.0	6.0	9.0	6.0	4.0	6.0	9.0	8.0	9.0	8.0	11.5	9.0	4.0	8.0	1.0	4.0	6.0	7.5	12.0	7.0	2.0	7.15
EFPUS-SB	2.0	4.5	2.0	2.5	4.0	8.0	8.5	8.5	3.0	13.5	2.0	7.0	6.5	7.0	13.0	6.0	13.0	11.5	11.0	5.5	7.0	8.5	9.0	7.11
SB	10.0	12.5	4.0	12.0	10.0	15.0	12.0	7.0	13.0	3.0	16.0	13.0	11.5	14.0	14.0	14.0	14.0	15.5	10.0	14.0	9.0	15.0	15.0	11.89
EFPUS-RBT	7.5	4.5	10.0	7.0	6.0	7.0	6.0	10.0	6.0	4.5	5.0	5.0	2.5	4.0	4.0	5.0	4.0	3.0	2.0	1.0	4.0	2.0	4.0	4.96
RBT	16.0	12.5	15.0	14.0	15.0	12.0	13.0	16.0	12.0	15.5	13.0	14.5	15.0	10.5	10.0	11.0	10.0	10.0	14.0	11.0	16.0	13.0	11.0	13.04
EFPUS-SBT	5.0	4.5	3.0	4.0	4.0	5.0	3.0	2.0	5.0	15.5	4.0	9.0	6.5	8.0	12.0	7.0	12.0	11.5	13.0	9.0	8.0	8.5	7.5	7.26
SBT	13.0	12.5	8.0	13.0	11.0	14.0	14.0	13.0	15.0	11.0	14.0	12.0	9.0	16.0	16.0	13.0	15.0	15.5	15.0	15.0	14.0	14.0	16.0	13.43
EFPUS-EE	5.0	4.5	9.0	11.0	7.5	3.0	7.0	5.0	8.0	8.0	6.0	4.0	6.5	5.0	6.0	4.0	6.0	5.5	5.0	3.0	1.0	4.0	5.0	5.61
EE	12.0	12.5	11.0	9.5	14.0	10.0	11.0	11.0	10.0	6.0	10.0	11.0	13.5	12.5	8.0	10.0	6.0	7.5	7.0	10.0	11.0	10.0	13.0	10.28
EFPUS-BC	7.5	4.5	13.0	9.5	7.5	4.0	5.0	3.5	7.0	4.5	7.5	1.5	6.5	6.0	4.0	3.0	6.0	5.5	4.0	2.0	5.0	3.0	6.0	5.48
BC	11.0	12.5	14.0	8.0	13.0	11.0	10.0	12.0	11.0	8.0	11.0	10.0	13.5	10.5	7.0	12.0	8.0	9.0	8.0	12.0	10.0	11.0	10.0	10.54

Table 4

The performance of compared algorithms on f-measure

	d1	d2	d3	d4	d5	d6	d7	d8	d9	d10	d11	d12	d13	d14	d15	d16	d17	d18	d19	d20	d21	d22	d23	AVE
EFPUS-RC	0.960	1.000	0.751	0.608	0.969	0.685	0.671	0.876	0.921	0.785	0.951	0.806	0.992	0.826	0.600	0.750	0.766	0.598	0.664	0.654	0.582	0.691	0.640	0.771
RC	0.933 (0.00)	0.986 (0.00)	0.697 (0.00)	0.574 (0.00)	0.883 (0.00)	0.574 (0.00)	0.564 (0.00)	0.829 (0.00)	0.849 (0.00)	0.765 (0.00)	0.848 (0.00)	0.674 (0.00)	0.941 (0.00)	0.697 (0.00)	0.508 (0.00)	0.495 (0.00)	0.716 (0.00)	0.530 (0.00)	0.578 (0.00)	0.524 (0.00)	0.457 (0.00)	0.498 (0.00)	0.455 (0.00)	0.677
EFPUS-SC	0.963	1.000	0.771	0.616	0.977	0.656	0.653	0.894	0.954	0.803	0.975	0.847	0.992	0.872	0.665	0.798	0.814	0.638	0.760	0.783	0.768	0.826	0.817	0.819
SC	0.937 (0.00)	0.986 (0.00)	0.728 (0.00)	0.577 (0.00)	0.932 (0.00)	0.559 (0.00)	0.578 (0.00)	0.869 (0.02)	0.865 (0.00)	0.800 (0.64)	0.875 (0.00)	0.790 (0.00)	0.982 (0.00)	0.811 (0.00)	0.610 (0.00)	0.683 (0.00)	0.772 (0.00)	0.542 (0.00)	0.661 (0.00)	0.693 (0.00)	0.701 (0.00)	0.678 (0.00)	0.672 (0.00)	0.752
EFPUS-RB	0.962	1.000	0.747	0.611	0.962	0.680	0.670	0.878	0.921	0.783	0.937	0.806	0.990	0.826	0.592	0.727	0.765	0.594	0.667	0.641	0.583	0.712	0.627	0.769
RB	0.945 (0.00)	0.986 (0.00)	0.747 (0.95)	0.603 (0.01)	0.938 (0.00)	0.653 (0.00)	0.647 (0.00)	0.879 (0.91)	0.879 (0.00)	0.767 (0.00)	0.918 (0.02)	0.767 (0.00)	0.965 (0.00)	0.764 (0.00)	0.579 (0.00)	0.612 (0.00)	0.752 (0.00)	0.564 (0.00)	0.570 (0.00)	0.626 (0.28)	0.487 (0.00)	0.574 (0.00)	0.576 (0.01)	0.731
EFPUS-SB	0.962	1.000	0.769	0.619	0.974	0.661	0.650	0.889	0.949	0.801	0.968	0.844	0.990	0.868	0.639	0.813	0.807	0.641	0.766	0.775	0.761	0.806	0.806	0.816
SB	0.945 (0.00)	0.986 (0.00)	0.754 (0.10)	0.594 (0.00)	0.941 (0.00)	0.573 (0.00)	0.600 (0.00)	0.895 (0.52)	0.881 (0.00)	0.821 (0.00)	0.868 (0.00)	0.808 (0.00)	0.981 (0.00)	0.816 (0.00)	0.658 (0.28)	0.710 (0.00)	0.812 (0.47)	0.566 (0.00)	0.743 (0.13)	0.722 (0.00)	0.730 (0.04)	0.701 (0.00)	0.672 (0.00)	0.773
EFPUS-RBT	0.959	1.000	0.733	0.592	0.941	0.648	0.632	0.853	0.894	0.773	0.923	0.781	0.988	0.778	0.567	0.644	0.752	0.581	0.641	0.594	0.540	0.629	0.546	0.739
RBT	0.932 (0.00)	0.986 (0.00)	0.700 (0.00)	0.576 (0.00)	0.889 (0.00)	0.597 (0.00)	0.577 (0.00)	0.804 (0.00)	0.855 (0.00)	0.756 (0.01)	0.841 (0.00)	0.714 (0.00)	0.942 (0.00)	0.728 (0.00)	0.514 (0.00)	0.547 (0.00)	0.721 (0.00)	0.494 (0.00)	0.544 (0.00)	0.557 (0.10)	0.455 (0.00)	0.491 (0.00)	0.443 (0.00)	0.681
EFPUS-SBT	0.962	1.000	0.763	0.615	0.972	0.672	0.668	0.903	0.947	0.800	0.964	0.837	0.991	0.862	0.650	0.807	0.811	0.641	0.757	0.767	0.753	0.794	0.806	0.815
SBT	0.939 (0.00)	0.986 (0.00)	0.737 (0.01)	0.585 (0.00)	0.934 (0.00)	0.576 (0.00)	0.593 (0.00)	0.871 (0.00)	0.871 (0.00)	0.799 (0.96)	0.878 (0.00)	0.802 (0.00)	0.984 (0.00)	0.797 (0.00)	0.625 (0.20)	0.712 (0.00)	0.777 (0.00)	0.536 (0.00)	0.686 (0.00)	0.696 (0.00)	0.688 (0.00)	0.714 (0.00)	0.648 (0.00)	0.758
EFPUS-EE	0.961	1.000	0.733	0.588	0.940	0.659	0.631	0.873	0.894	0.773	0.912	0.785	0.983	0.794	0.580	0.669	0.750	0.582	0.615	0.581	0.557	0.645	0.543	0.741
EE	0.940 (0.00)	0.986 (0.00)	0.733 (0.99)	0.590 (0.54)	0.915 (0.00)	0.615 (0.00)	0.605 (0.00)	0.852 (0.03)	0.872 (0.00)	0.772 (0.92)	0.851 (0.00)	0.739 (0.00)	0.948 (0.00)	0.737 (0.00)	0.567 (0.26)	0.561 (0.00)	0.744 (0.44)	0.517 (0.00)	0.585 (0.06)	0.574 (0.69)	0.493 (0.00)	0.493 (0.00)	0.454 (0.00)	0.702
EFPUS-BC	0.959	1.000	0.727	0.589	0.939	0.656	0.635	0.877	0.897	0.779	0.908	0.798	0.983	0.790	0.586	0.702	0.744	0.581	0.631	0.604	0.556	0.652	0.547	0.745
BC	0.940 (0.00)	0.986 (0.00)	0.723 (0.70)	0.591 (0.55)	0.918 (0.00)	0.613 (0.00)	0.609 (0.00)	0.847 (0.00)	0.871 (0.00)	0.772 (0.30)	0.844 (0.00)	0.750 (0.00)	0.949 (0.00)	0.738 (0.00)	0.563 (0.02)	0.549 (0.00)	0.742 (0.79)	0.516 (0.00)	0.556 (0.00)	0.539 (0.00)	0.486 (0.00)	0.512 (0.00)	0.476 (0.00)	0.700

Table 5

The ranks of compared algorithms on f-measure

	d1	d2	d3	d4	d5	d6	d7	d8	d9	d10	d11	d12	d13	d14	d15	d16	d17	d18	d19	d20	d21	d22	d23	AVE
EFPUS-RC	6.0	4.5	5.0	5.0	4.0	1.0	1.0	8.0	4.5	7.0	4.0	5.5	1.5	4.0	7.0	4.0	7.0	4.0	7.0	7.0	8.0	7.0	7.0	5.17
RC	16.0	12.5	16.0	16.0	16.0	14.0	16.0	15.0	16.0	15.0	14.0	0.7	16.0	16.0	16.0	16.0	16.0	13.0	13.0	16.0	15.0	14.0	14.0	14.44
EFPUS-SC	1.0	4.5	1.0	2.0	1.0	6.5	4.0	3.0	1.0	2.0	1.0	1.0	1.5	1.0	1.0	3.0	1.0	3.0	2.0	1.0	1.0	1.0	1.0	1.93
SC	14.0	12.5	12.0	14.0	12.0	16.0	14.0	11.0	14.0	4.5	11.0	9.0	10.0	7.0	6.0	9.0	6.0	11.0	8.0	6.0	5.0	8.0	4.5	9.76
EFPUS-RB	4.0	4.5	6.5	4.0	5.0	2.0	2.0	6.0	4.5	8.0	5.0	5.5	4.5	5.0	8.0	5.0	8.0	5.0	6.0	8.0	7.0	5.0	8.0	5.50
RB	9.5	12.5	6.5	6.0	10.0	8.0	6.0	5.0	9.0	14.0	7.0	12.0	12.0	12.0	11.0	12.0	10.0	10.0	14.0	9.0	13.0	12.0	9.0	9.98
EFPUS-SB	2.0	4.5	2.0	1.0	2.0	4.0	5.0	4.0	2.0	3.0	2.0	2.0	4.5	2.0	4.0	1.0	4.0	1.5	1.0	2.0	2.0	2.0	2.5	2.61
SB	9.5	12.5	4.0	7.0	6.5	15.0	12.0	2.0	8.0	1.0	12.0	4.0	11.0	6.0	2.0	7.0	2.0	9.0	4.0	4.0	4.0	6.0	4.5	6.65
EFPUS-RBT	7.5	4.5	10.0	8.0	6.5	9.0	8.0	12.0	7.5	10.5	6.0	11.0	6.0	11.0	12.0	11.0	9.0	7.5	9.0	11.0	11.0	11.0	11.0	9.13
RBT	16.0	12.5	15.0	15.0	15.0	12.0	15.0	16.0	15.0	16.0	16.0	15.0	15.0	15.0	15.0	15.0	15.0	16.0	16.0	14.0	16.0	16.0	16.0	15.11
EFPUS-SBT	3.0	4.5	3.0	3.0	3.0	3.0	3.0	1.0	3.0	4.5	3.0	3.0	3.0	3.0	3.0	2.0	3.0	1.5	3.0	3.0	3.0	3.0	2.5	2.91
SBT	13.0	12.5	8.0	13.0	11.0	13.0	13.0	10.0	11.5	6.0	10.0	7.0	7.0	8.0	5.0	6.0	5.0	12.0	5.0	5.0	6.0	4.0	6.0	8.57
EFPUS-EE	5.0	4.5	10.0	12.0	8.0	5.0	9.0	9.0	7.5	10.5	8.0	10.0	8.5	9.0	10.0	10.0	11.0	6.0	11.0	12.0	9.0	10.0	12.0	9.00
EE	11.5	12.5	10.0	10.0	14.0	10.0	11.0	13.0	10.0	12.5	13.0	14.0	14.0	14.0	13.0	13.0	13.0	14.0	12.0	13.0	12.0	15.0	15.0	12.59
EFPUS-BC	7.5	4.5	13.0	11.0	9.0	6.5	7.0	7.0	6.0	9.0	9.0	8.0	8.5	10.0	9.0	8.0	12.0	7.5	10.0	10.0	10.0	9.0	10.0	8.76
BC	11.5	12.5	14.0	9.0	13.0	11.0	10.0	14.0	11.5	12.5	15.0	13.0	13.0	13.0	14.0	14.0	14.0	15.0	15.0	15.0	14.0	13.0	13.0	13.04

Table 6

The performance of compared algorithms on AUC

	d1	d2	d3	d4	d5	d6	d7	d8	d9	d10	d11	d12	d13	d14	d15	d16	d17	d18	d19	d20	d21	d22	d23	AVE
EFPUS-RC	0.976	1.000	0.827	0.728	0.984	0.824	0.821	0.940	0.972	0.906	0.984	0.911	0.996	0.927	0.889	0.915	0.913	0.818	0.890	0.861	0.862	0.895	0.892	0.901
RC	0.953 (0.00)	0.988 (0.00)	0.781 (0.00)	0.700 (0.00)	0.939 (0.00)	0.727 (0.00)	0.724 (0.00)	0.908 (0.00)	0.927 (0.00)	0.888 (0.00)	0.936 (0.00)	0.857 (0.00)	0.983 (0.00)	0.888 (0.00)	0.847 (0.00)	0.827 (0.00)	0.898 (0.00)	0.796 (0.00)	0.851 (0.00)	0.820 (0.00)	0.799 (0.00)	0.850 (0.00)	0.832 (0.00)	0.857
EFPUS-SC	0.977	1.000	0.837	0.731	0.984	0.778	0.779	0.932	0.975	0.885	0.990	0.904	0.993	0.931	0.833	0.884	0.892	0.775	0.859	0.872	0.863	0.903	0.882	0.890
SC	0.954 (0.00)	0.988 (0.00)	0.805 (0.00)	0.703 (0.00)	0.958 (0.00)	0.708 (0.00)	0.727 (0.00)	0.917 (0.05)	0.916 (0.00)	0.894 (0.13)	0.922 (0.00)	0.875 (0.00)	0.989 (0.00)	0.902 (0.00)	0.798 (0.00)	0.849 (0.00)	0.871 (0.00)	0.732 (0.00)	0.819 (0.00)	0.832 (0.00)	0.837 (0.00)	0.846 (0.00)	0.827 (0.00)	0.855
EFPUS-RB	0.977	1.000	0.822	0.730	0.982	0.819	0.819	0.941	0.971	0.905	0.972	0.909	0.994	0.933	0.889	0.911	0.912	0.816	0.899	0.857	0.860	0.917	0.901	0.902
RB	0.962 (0.00)	0.988 (0.00)	0.822 (0.92)	0.724 (0.01)	0.969 (0.00)	0.795 (0.00)	0.798 (0.00)	0.935 (0.13)	0.949 (0.00)	0.895 (0.00)	0.968 (0.39)	0.895 (0.02)	0.988 (0.00)	0.912 (0.00)	0.885 (0.06)	0.883 (0.02)	0.915 (0.15)	0.812 (0.09)	0.866 (0.00)	0.856 (0.82)	0.823 (0.00)	0.8773 (0.00)	0.8935 (0.50)	0.887
EFPUS-SB	0.977	1.000	0.836	0.733	0.981	0.782	0.778	0.933	0.971	0.885	0.986	0.902	0.992	0.927	0.814	0.894	0.889	0.778	0.859	0.868	0.861	0.893	0.877	0.888
SB	0.962 (0.00)	0.988 (0.00)	0.825 (0.15)	0.715 (0.00)	0.964 (0.00)	0.717 (0.00)	0.741 (0.00)	0.933 (0.90)	0.929 (0.00)	0.904 (0.00)	0.919 (0.00)	0.881 (0.00)	0.988 (0.00)	0.901 (0.00)	0.807 (0.46)	0.854 (0.00)	0.887 (0.73)	0.730 (0.00)	0.864 (0.64)	0.842 (0.00)	0.846 (0.12)	0.852 (0.00)	0.825 (0.00)	0.864
EFPUS-RBT	0.976	1.000	0.813	0.716	0.974	0.802	0.797	0.930	0.959	0.901	0.976	0.905	0.994	0.924	0.885	0.894	0.911	0.813	0.894	0.868	0.854	0.900	0.882	0.894
RBT	0.953 (0.00)	0.988 (0.00)	0.783 (0.00)	0.701 (0.00)	0.942 (0.00)	0.746 (0.00)	0.736 (0.00)	0.892 (0.00)	0.930 (0.00)	0.884 (0.00)	0.934 (0.00)	0.871 (0.00)	0.984 (0.00)	0.904 (0.00)	0.845 (0.00)	0.854 (0.00)	0.906 (0.08)	0.788 (0.00)	0.836 (0.00)	0.832 (0.00)	0.802 (0.00)	0.839 (0.00)	0.831 (0.00)	0.860
EFPUS-SBT	0.977	1.000	0.832	0.730	0.981	0.795	0.800	0.941	0.969	0.884	0.983	0.896	0.993	0.923	0.828	0.893	0.890	0.778	0.857	0.863	0.857	0.889	0.886	0.889
SBT	0.957 (0.00)	0.988 (0.00)	0.809 (0.01)	0.709 (0.00)	0.960 (0.00)	0.720 (0.00)	0.735 (0.00)	0.918 (0.00)	0.922 (0.00)	0.889 (0.34)	0.923 (0.00)	0.886 (0.05)	0.991 (0.25)	0.895 (0.00)	0.796 (0.00)	0.856 (0.00)	0.876 (0.00)	0.728 (0.00)	0.825 (0.01)	0.839 (0.01)	0.830 (0.02)	0.856 (0.01)	0.819 (0.00)	0.858
EFPUS-EE	0.976	1.000	0.811	0.711	0.971	0.805	0.791	0.937	0.957	0.895	0.974	0.907	0.992	0.924	0.880	0.898	0.909	0.809	0.871	0.856	0.857	0.897	0.880	0.892
EE	0.958 (0.00)	0.988 (0.00)	0.810 (0.96)	0.713 (0.67)	0.956 (0.00)	0.762 (0.00)	0.759 (0.00)	0.921 (0.00)	0.941 (0.00)	0.896 (0.83)	0.944 (0.00)	0.884 (0.00)	0.985 (0.00)	0.903 (0.00)	0.875 (0.50)	0.857 (0.00)	0.908 (0.94)	0.797 (0.00)	0.865 (0.50)	0.840 (0.09)	0.824 (0.01)	0.858 (0.00)	0.831 (0.00)	0.873
EFPUS-BC	0.975	1.000	0.807	0.712	0.971	0.802	0.794	0.939	0.959	0.900	0.971	0.911	0.992	0.921	0.884	0.908	0.908	0.809	0.882	0.858	0.855	0.903	0.877	0.893
BC	0.959 (0.00)	0.988 (0.00)	0.803 (0.69)	0.714 (0.58)	0.956 (0.00)	0.761 (0.00)	0.763 (0.00)	0.918 (0.00)	0.940 (0.00)	0.896 (0.35)	0.939 (0.00)	0.887 (0.00)	0.985 (0.00)	0.904 (0.00)	0.877 (0.24)	0.852 (0.00)	0.908 (0.81)	0.794 (0.00)	0.857 (0.03)	0.827 (0.00)	0.821 (0.00)	0.863 (0.01)	0.854 (0.12)	0.872

Table 7

The ranks of compared algorithms on AUC

	d1	d2	d3	d4	d5	d6	d7	d8	d9	d10	d11	d12	d13	d14	d15	d16	d17	d18	d19	d20	d21	d22	d23	AVE
EFPUS-RC	6.0	4.5	4.0	5.0	1.5	1.0	1.0	3.0	2.0	1.0	3.0	1.5	1.0	3.5	1.5	1.0	2.0	1.0	3.0	5.0	2.0	6.0	3.0	2.72
RC	15.5	12.5	16.0	16.0	16.0	13.0	16.0	15.0	14.0	12.0	12.0	16.0	16.0	16.0	9.0	16.0	10.0	8.0	13.0	16.0	16.0	14.0	11.0	13.87
EFPUS-SC	2.5	4.5	1.0	2.0	1.5	9.0	8.0	9.0	1.0	13.5	1.0	6.0	4.5	2.0	11.0	8.0	11.0	13.0	9.5	1.0	1.0	2.5	5.5	5.57
SC	14.0	12.5	13.0	14.0	12.0	16.0	15.0	14.0	16.0	10.0	15.0	14.0	10.0	13.0	15.0	15.0	16.0	14.0	16.0	13.5	10.0	15.0	14.0	13.78
EFPUS-RB	2.5	4.5	6.5	3.5	3.0	2.0	2.0	1.5	3.5	2.0	7.0	3.0	2.5	1.0	1.5	2.0	3.0	2.0	1.0	7.0	4.0	1.0	1.0	2.91
RB	9.5	12.5	6.5	6.0	9.0	6.5	4.0	6.0	9.0	8.5	9.0	9.0	11.5	9.0	3.5	9.0	1.0	4.0	6.0	8.5	13.0	9.0	2.0	7.48
EFPUS-SB	2.5	4.5	2.0	1.0	5.0	8.0	9.0	7.5	3.5	13.5	2.0	7.0	7.0	3.5	13.0	5.5	13.0	11.5	9.5	2.5	3.0	7.0	8.5	6.50
SB	9.5	12.5	5.0	8.0	10.0	15.0	12.0	7.5	13.0	3.0	16.0	13.0	11.5	14.0	14.0	12.5	14.0	15.0	8.0	10.0	9.0	13.0	15.0	11.33
EFPUS-RBT	6.0	4.5	8.0	7.0	6.0	4.5	5.0	10.0	6.5	4.0	5.0	5.0	2.5	5.5	3.5	5.5	4.0	3.0	2.0	2.5	8.0	4.0	5.5	5.11
RBT	15.5	12.5	15.0	15.0	15.0	12.0	13.0	16.0	12.0	16.0	13.0	15.0	15.0	10.5	10.0	12.5	9.0	10.0	14.0	13.5	15.0	16.0	12.5	13.39
EFPUS-SBT	2.5	4.5	3.0	3.5	4.0	6.5	3.0	1.5	5.0	15.0	4.0	8.0	4.5	7.0	12.0	7.0	12.0	11.5	11.5	4.0	5.5	8.0	4.0	6.41
SBT	13.0	12.5	11.0	13.0	11.0	14.0	14.0	12.5	15.0	11.0	14.0	11.0	9.0	15.0	16.0	11.0	15.0	16.0	15.0	12.0	11.0	12.0	16.0	13.04
EFPUS-EE	6.0	4.5	9.0	12.0	7.5	3.0	7.0	5.0	8.0	8.5	6.0	4.0	7.0	5.5	6.0	4.0	5.0	5.5	5.0	8.5	5.5	5.0	7.0	6.28
EE	12.0	12.5	10.0	10.0	13.5	10.0	11.0	11.0	10.0	6.5	10.0	12.0	14.0	12.0	8.0	10.0	7.0	7.0	7.0	11.0	12.0	11.0	12.5	10.43
EFPUS-BC	8.0	4.5	12.0	11.0	7.5	4.5	6.0	4.0	6.5	5.0	8.0	1.5	7.0	8.0	5.0	3.0	7.0	5.5	4.0	6.0	7.0	2.5	8.5	6.17
BC	11.0	12.5	14.0	9.0	13.5	11.0	10.0	12.5	11.0	6.5	11.0	10.0	13.0	10.5	7.0	14.0	7.0	9.0	11.5	15.0	14.0	10.0	10.0	11.00

Table 8

The performance of compared algorithms on Recall

	d1	d2	d3	d4	d5	d6	d7	d8	d9	d10	d11	d12	d13	d14	d15	d16	d17	d18	d19	d20	d21	d22	d23	AVE
EFPUS-RC	0.994	1.000	0.894	0.724	0.986	0.906	0.917	0.957	0.996	0.962	0.988	0.878	0.990	0.900	0.920	0.890	0.870	0.710	0.870	0.800	0.820	0.850	0.860	0.900
RC	0.961 (0.00)	0.976 (0.00)	0.833 (0.00)	0.684 (0.00)	0.947 (0.00)	0.782 (0.00)	0.771 (0.00)	0.907 (0.00)	0.939 (0.00)	0.931 (0.00)	0.927 (0.00)	0.838 (0.01)	0.985 (0.00)	0.890 (0.312)	0.890 (0.16)	0.830 (0.02)	0.859 (0.04)	0.709 (0.95)	0.832 (0.10)	0.807 (0.55)	0.762 (0.00)	0.860 (0.74)	0.835 (0.40)	0.860
EFPUS-SC	0.993	1.000	0.844	0.640	0.976	0.719	0.715	0.900	0.969	0.850	0.988	0.829	0.990	0.880	0.720	0.780	0.800	0.570	0.730	0.760	0.740	0.820	0.770	0.830
SC	0.952 (0.00)	0.976 (0.00)	0.801 (0.01)	0.619 (0.01)	0.946 (0.00)	0.627 (0.00)	0.673 (0.00)	0.878 (0.15)	0.879 (0.00)	0.887 (0.00)	0.862 (0.00)	0.786 (0.00)	0.982 (0.00)	0.837 (0.00)	0.645 (0.00)	0.740 (0.09)	0.764 (0.00)	0.497 (0.00)	0.673 (0.01)	0.687 (0.00)	0.697 (0.02)	0.725 (0.00)	0.680 (0.00)	0.780
EFPUS-RB	0.993	1.000	0.877	0.721	0.986	0.891	0.912	0.957	0.994	0.961	0.963	0.874	0.990	0.920	0.920	0.890	0.870	0.710	0.890	0.800	0.820	0.900	0.890	0.900
RB	0.968 (0.00)	0.976 (0.00)	0.871 (0.61)	0.703 (0.00)	0.975 (0.00)	0.857 (0.00)	0.878 (0.00)	0.934 (0.00)	0.973 (0.00)	0.952 (0.07)	0.962 (0.97)	0.864 (0.38)	0.986 (0.01)	0.900 (0.02)	0.921 (0.57)	0.885 (0.82)	0.883 (0.00)	0.713 (0.05)	0.870 (0.25)	0.793 (0.77)	0.782 (0.01)	0.865 (0.27)	0.895 (0.64)	0.890
EFPUS-SB	0.991	1.000	0.830	0.643	0.971	0.723	0.720	0.906	0.963	0.852	0.981	0.826	0.990	0.870	0.680	0.800	0.800	0.580	0.740	0.750	0.740	0.800	0.760	0.820
SB	0.964 (0.00)	0.976 (0.00)	0.829 (0.92)	0.618 (0.00)	0.954 (0.00)	0.615 (0.00)	0.661 (0.00)	0.901 (0.62)	0.903 (0.00)	0.895 (0.00)	0.858 (0.00)	0.789 (0.00)	0.980 (0.02)	0.828 (0.00)	0.649 (0.16)	0.740 (0.00)	0.791 (0.56)	0.481 (0.00)	0.753 (0.31)	0.702 (0.01)	0.710 (0.19)	0.730 (0.00)	0.675 (0.00)	0.780
EFPUS-RBT	0.994	1.000	0.904	0.729	0.986	0.935	0.952	0.955	0.989	0.963	0.983	0.883	0.990	0.920	0.940	0.900	0.870	0.710	0.890	0.870	0.830	0.900	0.880	0.910
RBT	0.960 (0.00)	0.976 (0.00)	0.839 (0.00)	0.706 (0.02	0.951 (0.00)	0.798 (0.00)	0.784 (0.00)	0.889 (0.00)	0.941 (0.00)	0.933 (0.00)	0.925 (0.00)	0.840 (0.00)	0.986 (0.00)	0.903 (0.01)	0.886 (0.00)	0.875 (0.40)	0.879 (0.23)	0.713 (0.41)	0.807 (0.00)	0.798 (0.00)	0.765 (0.00)	0.840 (0.03)	0.835 (0.06)	0.860
EFPUS-SBT	0.990	1.000	0.833	0.649	0.971	0.772	0.797	0.920	0.959	0.853	0.975	0.814	0.990	0.860	0.710	0.800	0.800	0.580	0.730	0.740	0.730	0.790	0.780	0.830
SBT	0.956 (0.00)	0.976 (0.00)	0.786 (0.00)	0.616 (0.00)	0.950 (0.00)	0.624 (0.00)	0.652 (0.00)	0.876 (0.00)	0.891 (0.00)	0.871 (0.10)	0.864 (0.00)	0.808 (0.53)	0.985 (0.81)	0.827 (0.01)	0.633 (0.00)	0.745 (0.02)	0.775 (0.02)	0.490 (0.00)	0.677 (0.04)	0.705 (0.10)	0.685 (0.05)	0.735 (0.03)	0.665 (0.00)	0.770
EFPUS-EE	0.992	1.000	0.864	0.700	0.978	0.898	0.908	0.951	0.981	0.941	0.988	0.883	0.990	0.910	0.910	0.900	0.870	0.690	0.850	0.840	0.830	0.880	0.890	0.900
EE	0.966 (0.00)	0.976 (0.00)	0.863 (0.90)	0.698 (0.76)	0.960 (0.00)	0.806 (0.00)	0.806 (0.00)	0.926 (0.01	0.955 (0.00)	0.946 (0.53)	0.949 (0.00)	0.854 (0.04)	0.985 (0.24)	0.897 (0.02)	0.909 (0.71)	0.865 (0.16)	0.871 (0.50)	0.711 (0.02)	0.860 (0.76)	0.796 (0.01)	0.795 (0.12)	0.895 (0.55)	0.820 (0.03)	0.870
EFPUS-BC	0.990	1.000	0.876	0.705	0.980	0.894	0.914	0.953	0.982	0.950	0.980	0.886	0.990	0.910	0.920	0.900	0.870	0.700	0.870	0.840	0.820	0.890	0.880	0.900
BC	0.967 (0.00)	0.976 (0.00)	0.859 (0.22)	0.701 (0.53)	0.958 (0.00)	0.804 (0.00)	0.816 (0.00)	0.924 (0.01)	0.954 (0.00)	0.946 (0.58)	0.939 (0.00)	0.855 (0.01)	0.985 (0.11)	0.900 (0.16)	0.918 (0.78)	0.860 (0.07)	0.869 (0.97)	0.706 (0.10)	0.860 (0.57)	0.788 (0.01)	0.788 (0.06)	0.885 (0.86)	0.860 (0.63)	0.880

Table 9

The ranks of compared algorithms on recall

	d1	d2	d3	d4	d5	d6	d7	d8	d9	d10	d11	d12	d13	d14	d15	d16	d17	d18	d19	d20	d21	d22	d23	AVE
EFPUS-RC	1.5	4.5	2.0	2.0	2.0	2.0	2.0	1.5	1.0	2.0	2.0	4.0	1.0	7.0	5.0	4.5	4.0	4.0	5.0	5.5	3.5	9.0	7.0	3.57
RC	12.0	12.5	11.5	10.0	15.0	10.0	11.0	10.0	13.0	10.0	12.0	10.0	12.0	10.0	9.0	10.0	10.0	5.0	9.0	4.0	10.0	8.0	8.5	10.11
EFPUS-SC	2.5	4.5	9.0	13.0	6.0	13.0	13.0	13.0	7.0	16.0	2.0	11.0	6.0	11.0	11.0	13.0	11.0	13.0	13.0	11.0	11.0	11.0	12.0	10.13
SC	15.0	12.5	15.0	14.0	16.0	14.0	14.0	15.0	16.0	12.0	15.0	16.0	15.0	14.0	15.0	15.5	16.0	14.0	16.0	16.0	15.0	16.0	14.0	14.83
EFPUS-RB	2.5	4.5	3.0	3.0	2.0	5.0	4.0	1.5	2.0	3.0	8.0	5.0	3.0	2.0	2.0	4.5	6.0	7.5	2.0	7.0	5.0	2.5	2.5	3.80
RB	8.0	12.5	5.0	6.0	7.0	6.0	6.0	6.0	6.0	4.0	9.0	6.0	8.0	7.0	3.0	6.0	1.0	1.5	4.0	9.0	8.0	7.0	1.0	5.96
EFPUS-SB	5.0	4.5	13.0	12.0	8.5	12.0	12.0	11.0	8.0	15.0	5.0	12.0	12.0	12.0	13.0	11.5	13.0	11.0	12.0	12.0	12.0	12.0	13.0	10.93
SB	11.0	12.5	14.0	15.0	12.0	16.0	15.0	12.0	14.0	11.0	16.0	15.0	16.0	15.0	14.0	15.5	14.0	16.0	11.0	15.0	14.0	15.0	15.0	14.09
EFPUS-RBT	1.5	4.5	1.0	1.0	2.0	1.0	1.0	3.0	3.0	1.0	4.0	2.5	2.0	1.0	1.0	2.0	3.0	6.0	1.0	1.0	1.5	1.0	4.0	2.13
RBT	13.0	12.5	10.0	4.0	13.0	9.0	10.0	14.0	12.0	9.0	13.0	9.0	8.0	5.0	10.0	7.0	2.0	1.5	10.0	5.5	9.0	10.0	8.5	8.91
EFPUS-SBT	6.5	4.5	11.5	11.0	8.5	11.0	9.0	9.0	9.0	14.0	7.0	13.0	8.0	13.0	12.0	11.5	12.0	12.0	14.0	13.0	13.0	13.0	11.0	10.72
SBT	14.0	12.5	16.0	16.0	14.0	15.0	16.0	16.0	15.0	13.0	14.0	14.0	12.0	16.0	16.0	14.0	15.0	15.0	15.0	14.0	16.0	14.0	16.0	14.72
EFPUS-EE	4.0	4.5	6.0	8.0	5.0	3.0	5.0	5.0	5.0	8.0	2.0	2.5	5.0	3.0	7.0	2.0	9.0	10.0	8.0	2.0	1.5	6.0	2.5	4.96
EE	10.0	12.5	7.0	9.0	10.0	7.0	8.0	7.0	10.0	6.5	10.0	8.0	12.0	9.0	8.0	8.0	5.0	3.0	6.5	8.0	6.0	2.5	10.0	7.96
EFPUS-BC	6.5	4.5	4.0	5.0	4.0	4.0	3.0	4.0	4.0	5.0	6.0	1.0	4.0	4.0	6.0	2.0	7.5	9.0	3.0	3.0	3.5	4.0	5.0	4.43
BC	9.0	12.5	8.0	7.0	11.0	8.0	7.0	8.0	11.0	6.5	11.0	7.0	12.0	7.0	4.0	9.0	7.5	7.5	6.5	10.0	7.0	5.0	6.0	8.15

Table 10

The performance of compared algorithms on Accuracy

	d1	d2	d3	d4	d5	d6	d7	d8	d9	d10	d11	d12	d13	d14	d15	d16	d17	d18	d19	d20	d21	d22	d23	AVE
EFPUS-RC	97.12	100.00	80.39	72.96	98.33	78.49	77.27	93.09	95.91	87.58	98.21	93.36	99.78	94.68	86.79	93.56	94.63	90.53	90.68	91.08	89.46	93.17	92.36	90.84
RC	95.12 (0.00)	99.20 (0.00)	76.31 (0.00)	70.62 (0.00)	93.41 (0.00)	70.06 (0.00)	70.03 (0.00)	90.81 (0.00)	92.05 (0.00)	86.48 (0.00)	94.22 (0.00)	87.00 (0.00)	98.19 (0.00)	88.51 (0.00)	81.27 (0.00)	82.54 (0.00)	92.70 (0.00)	86.56 (0.00)	86.71 (0.00)	83.01 (0.00)	82.89 (0.00)	84.12 (0.00)	82.89 (0.00)	85.86
EFPUS-SC	97.29	100.00	83.50	77.02	98.83	80.66	81.07	94.97	97.77	90.42	99.16	95.54	99.77	96.69	92.57	96.73	96.44	93.83	95.88	96.58	96.52	97.58	97.82	93.77
SC	95.50 (0.00)	99.20 (0.00)	80.61 (0.00)	73.80 (0.00)	96.45 (0.00)	74.70 (0.00)	75.36 (0.00)	93.79 (0.01)	93.56 (0.00)	89.75 (0.06)	96.22 (0.00)	93.66 (0.00)	99.48 (0.00)	94.95 (0.00)	91.82 (0.03)	93.66 (0.00)	95.64 (0.00)	91.98 (0.00)	93.55 (0.00)	94.83 (0.00)	95.27 (0.00)	94.71 (0.00)	95.36 (0.00)	91.47
EFPUS-RB	97.19	100.00	80.25	73.38	97.93	78.37	77.26	93.32	95.86	87.49	97.82	93.34	99.73	94.49	86.22	92.71	94.63	90.46	90.56	90.55	89.46	93.46	91.50	90.69
RB	96.06 (0.00)	99.20 (0.00)	80.52 (0.70)	73.23 (0.54)	96.65 (0.00)	76.52 (0.00)	75.86 (0.00)	93.52 (0.59)	93.57 (0.00)	86.45 (0.00)	97.14 (0.01)	91.64 (0.00)	98.96 (0.00)	91.96 (0.00)	85.56 (0.01)	88.09 (0.00)	94.10 (0.00)	89.06 (0.00)	86.08 (0.00)	90.54 (0.98)	85.59 (0.00)	88.75 (0.00)	89.21 (0.00)	89.06
EFPUS-SB	97.29	100.00	83.77	77.10	98.64	81.02	80.67	94.64	97.52	90.24	98.93	95.45	99.72	96.69	92.27	96.93	96.29	93.82	95.89	96.30	96.46	97.37	97.75	93.69
SB	96.09 (0.00)	99.20 (0.00)	82.40 (0.03)	75.63 (0.00)	96.89 (0.00)	76.74 (0.00)	78.02 (0.00)	95.02 (0.30)	94.26 (0.00)	90.98 (0.01)	95.98 (0.00)	94.49 (0.00)	99.45 (0.00)	95.28 (0.00)	93.07 (0.03)	94.45 (0.00)	96.45 (0.17)	93.05 (0.00)	95.14 (0.01)	95.49 (0.01)	95.87 (0.01)	95.42 (0.00)	95.43 (0.00)	92.38
EFPUS-RBT	97.01	100.00	78.13	71.01	96.76	73.74	71.98	91.74	94.34	86.72	97.14	92.03	99.65	92.32	84.34	89.34	94.18	89.83	89.35	86.85	87.29	89.92	88.43	88.79
RBT	95.06 (0.00)	99.20 (0.00)	76.36 (0.06)	69.85 (0.01)	93.78 (0.00)	72.12 (0.00)	71.20 (0.20)	89.36 (0.00)	92.38 (0.00)	85.74 (0.04)	94.05 (0.00)	89.20 (0.00)	98.24 (0.00)	90.38 (0.00)	81.16 (0.00)	83.66 (0.00)	92.80 (0.00)	84.83 (0.00)	85.82 (0.00)	85.79 (0.36)	83.20 (0.00)	83.79 (0.00)	82.79 (0.00)	86.12
EFPUS-SBT	97.23	100.00	83.16	76.45	98.57	80.64	80.07	95.20	97.45	90.09	98.79	95.28	99.74	96.55	92.06	96.78	96.36	93.79	95.80	96.39	96.28	97.12	97.64	93.54
SBT	95.68 (0.00)	99.20 (0.00)	81.76 (0.04)	74.83 (0.00)	96.54 (0.00)	76.71 (0.00)	77.61 (0.00)	93.94 (0.00)	93.78 (0.00)	89.94 (0.66)	96.31 (0.00)	93.96 (0.00)	99.54 (0.00)	94.52 (0.00)	92.39 (0.46)	94.45 (0.00)	95.66 (0.00)	91.94 (0.00)	94.32 (0.00)	94.56 (0.00)	94.95 (0.00)	95.67 (0.00)	95.11 (0.00)	91.88
EFPUS-EE	97.17	100.00	79.22	71.62	96.70	75.98	73.19	93.01	94.43	87.02	96.55	92.36	99.52	93.04	85.28	89.98	94.14	90.13	88.53	87.01	87.70	91.12	87.57	89.19
EE	95.62 (0.00)	99.20 (0.00)	79.21 (0.99)	71.90 (0.48)	95.34 (0.00)	74.08 (0.00)	73.54 (0.50)	91.82 (0.04)	93.29 (0.00)	86.90 (0.79)	94.03 (0.00)	90.48 (0.00)	98.45 (0.00)	90.65 (0.00)	84.76 (0.41)	85.11 (0.00)	93.81 (0.18)	86.61 (0.00)	86.69 (0.02)	87.52 (0.54)	84.83 (0.00)	82.67 (0.00)	84.04 (0.00)	87.42
EFPUS-BC	97.01	100.00	78.28	71.49	96.65	75.79	73.44	93.19	94.60	87.28	96.52	92.89	99.51	93.04	86.00	91.75	93.96	90.05	88.90	87.59	88.19	91.42	87.93	89.37
BC	95.68 (0.00)	99.20 (0.00)	78.38 (0.91)	71.95 (0.33)	95.51 (0.00)	73.91 (0.00)	73.72 (0.58)	91.50 (0.00)	93.24 (0.00)	86.87 (0.33)	93.84 (0.00)	90.93 (0.00)	98.48 (0.00)	90.59 (0.00)	84.42 (0.00)	84.47 (0.00)	93.85 (0.59)	86.47 (0.00)	85.29 (0.00)	85.77 (0.07)	84.79 (0.00)	84.50 (0.00)	84.89 (0.00)	87.32

Tables 4 and 5 report the summary results and the ranks of the sixteen algorithms on the measure of f-measure respectively. The pair-wise $t$ -test at 0.05 significant levels was used to statistically compare the performance of algorithms. Similarly to the results of algorithms on g-mean, Table 4 shows that EFPUS statistically outperforms RC, SC, RB, SB, RBT, SBT, EE and BC on most of the datasets. Specifically, EFPUS statistically outperforms the corresponding comparing algorithms on 23, 22, 21, 19, 23, 21, 17 and 20 out of the 23 datasets respectively. The results indicate that EFPUS can also improve the performance of traditional methods on measure of f-measure. Besides, Table 5 shows that the ranks of EFPUS on the most of the datasets are much smaller than other methods, and EFPUS-SC ranks first with the average rank of 1.93, followed by EFPUS-SB with average rank of 2.61, EFPUS-SBT with 2.91, EFPUS-RC (5.17), EFPUS-RB (5.5), SB (6.65), SBT (8.57), EFPUS-BC (8.76), EFPUS-EE (9.00), EFPUS-RBT (9.13), SC (9.76), RB (9.98), EE (12.59), BC (13.04), RC (14.44) and RBT (15.11). We also observe from Tables 4 and 5 that, the methods with SMOTE outperform the ones with random under-sampling technique.

Tables 6 and 7 show the experimental results of compared algorithms on measure of AUC, where Table 6 summarizes the results of different algorithms and Table 7 reports the ranks of the algorithms. From Table 6, we can observe that EFPUS statistically outperforms RC, SC, RB, SB, RBT, SBT, EE and BC on 23, 21, 16, 17, 22, 22, 17 and 17 out of the 23 datasets where the pair-wise $t$ -test at 0.05 significant levels is used to statistically compare the performance of algorithms. Table 7 shows that the ranks of EFPUS on the most of the datasets are much smaller than other methods, and EFPUS-RC ranks first with the average rank of 2.72, then EFPUS-RB (2.91), EFPUS-RBT (5.11), EFPUS-SC (5.57), EFPUS-BC (6.17), EFPUS-EE (6.28), EFFPUS-SBT (6.41), EFPUS-SB (6.50), RB (7.48), EE (10.43), BC (11.00), SB (11.33), SBT (13.04), RBT (13.39), SC (13.78) and RC (13.87).

Tables 8 and 9 depict the recalls and ranks of the five comparing methods respectively. Similar to the results above, EFPUS significantly wins on most of the 23 sets comparing to other methods. The average recalls (ranks) of EFPUS-RC, RC, EFPUS-BC, BC, EFPUS-RBT, RBT, EFPUS-SBT, SBT, EFPUS-EE, EE, EFPUS-BC and BC are 0.900 (3.57), 0.860 (10.11), 0.830 (10.13), 0.780 (14.48), 0.900 (3.80), 0.890 (5.96), 0.820 (10.93), 0.780 (14.09), 0.910 (2.13), 0.860 (8.91), 0.830 (10.72), 0.770 (14.72), 0.900 (4.96), 0.870 (7.96), 0.900 (4.43) and 0.880 (8.15) respectively.

Though accuracy is inadequate to evaluate the classification performance, poor accuracy means a bad classifier. An efficient classifier may improve g-mean, f-measure, AUC or recall without decreasing accuracy. Table 10 presents the accuracy of the comparing algorithms. As shown in Table 9, EFPUS statistically outperforms RC, SC, RB, SB, RBT, SBT, EE and BC on 23, 23, 21, 23, 20, 21, 16 and 17 out of the 23 datasets with the pairwise $t$ -test of 0.05 significant levels.

All the experimental results we provided above indicate that EFPUS is an efficient method to improve the generalization performance of conventional class-imbalance oriented methods, no matter the method is a single model, an ensemble or a hybrid ensemble. Moreover, the results show that constructing ensemble through feature projection and under-sampling can improve the performance of imbalanced learning and is worth of further research.

Figure 1.

The effects of missing value on the performance of EFPUS.

Figure 2.

The effects of noise on the performance of EFPUS.

4.3 Robustness to missing values and noise

We investigate in this section the effects of missing values and noise on the performance of the EFPUS (the proposed method). In these experiments, classifiers were built using corrupted versions of the original data. The data set “yeast-0-2-5-6_vs_3-7-8-9” was selected as the candidate and EFPUS-RC was selected as the candidate of EFPUS. For testing the robust of EFPUS on missing value, the attribute values of each instance of the training set are set to be missing value with $p$ percentage, and $p$ gradually grows from 2 to 20 with step length of 2. Figure 1 presents the corresponding results, where horizontal axis is $p$ ’s value and vertical axis is the average performance of EFPUS with 10-fold cross-validation. EFPUS is robust to the missing value on minority class, since the recall of EFPUS does not drop with the $p$ growing (Fig. 1d). From Fig. 1b, the f-measure of EFPUS drops very much as $p$ greater than 6. Combining the results of Fig. 1d, we have that the impact of missing value is significantly high on majority class.

For testing the effects of noise for EFPUS, class labels of a fixed percentage $p$ of examples selected at random are modified. Figure 2 shows the corresponding results, where horizontal axis is $p$ ’s value and vertical axis is the average performance of EFPUS with 10-fold cross-validation. From Fig. 2, on the whole, the performance of EFPUS gradually drops with $p$ growing on the four measures, namely g-mean, f-measure, AUC and recall, and EFPUS drops much slower on recall than on the other measures, which indicate that the impact of noise is significantly less on minority class than on majority class.

5. Conclusion

In this paper, we propose a simple but effective ensemble method through feature projection and under-sampling (EFPUS) to tackle the imbalanced problem. The main heuristic in EFPUS consists of sampling subsets from majority class, learning a projection matrix from each subset, obtaining training sets by projecting the original training set to new spaces defined by the matrixes, and constructing a conventional class-imbalance oriented method on each new training set. Therefore, EFPUS mainly obtains the diversity between ensemble members through feature projection and under-sampling. Experimental results show that EFPUS can significantly improve traditional class-imbalanced oriented method for imbalanced data sets on the measure of g-mean, f-measure, AUC, recall and accuracy, no matter the method is a single model, an ensemble or a hybrid ensemble.

The main issue in this paper is to use sampling technique and feature projection to create new training sets, and therefore one of the future work is to carry out more experiments to study the effects of differ sampling technique and feature projection. Besides, it is very important to gain deep insight into behavior of the algorithm clearly, and thus another future work is to carry out simulation for studying the behavior of the proposed algorithm.

Footnotes

Acknowledgments

This work is supported in part by the National Natural Science Foundation of China (No. 61602422, No. 61501393), part by Project of Science and Technology Department of Henan Province (No. 182102210132) and in part by Nanhu Scholars Program for Young Scholars of XYNU.

References

Park

and Ghosh

, Ensembles of α-Trees for Imbalanced Classification Problems, IEEE Transactions on Knowledge and Data Engineering 26(1) (2014), 131–143.

and Garcia

E.A.

, Learning from imbalanced Data, IEEE Transactions on Knowledge and Data Engineering 21 (2009), 1263–1284.

and Ma

, Eds, Imbalanced Learning: Foundations, Algorithms, and Applications, Wiley-IEEE, 2013.

Zhi

Guo

Fan

and Ye

, Instance-based ensemble pruning for imbalanced learning, Intelligent Data Analysis 19(4) (2015), 779–794.

Barandela

Valdovinos

and Sánchez

, New applications of ensembles of classifiers, Pattern Analysis and Applications 6(3) (2003), 245–256.

Tahir

M.A.

Kittler

Mikolajczyk

and Yan

, A multiple expert approach to the class imbalance problem using inverse random under sampling, Proceedings of Multiple Classifier Systems, Lecture Notes in Computer Science 5519 (2009), 82–91.

Chawla

N.V.

Bowyer

K.W.

Hall

L.O.

and Kegelmeyer

W.P.

, SMOTE: Synthetic Minority Over-sampling Technique, Journal of Artificial Intelligence Research 16 (2002), 321–357.

Han

Wang

W.Y.

and Mao

B.H.

, Borderline-smote: a new over-sampling method in imbalanced datasets learning, in: Proceedings of International Conference on Intelligent Computing, 2005, pp. 878–887.

Zhou

Wang

Q.M.J.

Yang

C.-N.

and Sun

, Effective and Efficient Global Context Verification for Image Copy Detection, IEEE Transactions on Information Forensics and Security 12(1) (2017), 48–63.

10.

Xia

Wang

Zhang

Qin

Sun

and Ren

, A Privacy-preserving and Copy-deterrence Content-based Image Retrieval Scheme in Cloud Computing, IEEE Transactions on Information Forensics and Security 11(11) (2016), 2594–2608.

11.

Yang

and Sun

, Segmentation-based Image Copy-move Forgery Detection Scheme, IEEE Transactions on Information Forensics and Security 10(3) (2015), 507–518.

12.

Zhou

Yang

C.-N.

Chen

Sun

Liu

and Wu

Q.M.J.

, Effective and Efficient Image Copy Detection with Resistance to Arbitrary Rotation, IEICE Transactions on Information and Systems E99-D(6) (2016), 1531–1540.

13.

Liu

X.Y.

and Zhou

Z.H.

, Exploratory Under-Sampling for Class-Imbalance Learning, IEEE Transactions on Systems, Man, and Cybernetics, Part B 39(2) (2009), 965–969.

14.

Wang

and Yao

, Diversity analysis on imbalanced data sets by using ensemble models, in: Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining, 2009, pp. 324–331.

15.

Chawla

Lazarevic

Hall

and Bowyer

, SMOTEboost: improving prediction of the minority class in boosting, in: Proceeding of the 7th European Conference on Principles and Practice of Knowledge Discovery in Databases, 2003, pp. 107–119.

16.

Seiffert

Khoshgoftaar

Van Hulse

and Napolitano

, RUSboost: a hybrid approach to alleviating class imbalance, IEEE Transactions on Systems, Man, and Cybernetics, Part A 40(1) (2010), 185–197.

17.

Díez-Pastor

Diez

J.J.R.

García-Osorio

C.I.

and Kuncheva

L.I.

, Random Balance: Ensembles of variable priors classifiers for imbalanced data, Knowledge Based Systems 85 (2015), 96–111.

18.

Rodriguez

J.J.

Kuncheva

L.I.

and Alonso

C.J.

, Rotation Forest: A New Classifier Ensemble Method, IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (2006), 1619–1630.

19.

Japkowicz

, The class imbalance problem: Significance and strategies, in: Proceedings of the International Conference on Artificial Intelligence, 2000, pp. 111–117.

20.

Mani

and Zhang

, KNN approach to unbalanced data distributions: a case study involving information extraction, in: Proceedings of the ICML’2003 Workshop on Learning from Imbalanced Datasets II, 2003.

21.

Batista

Prati

and Monard

, A study of the behavior of several methods for balancing machine learning training data, SIGKDD Explorations 6 (2004), 20–29.

22.

Xie

and Qiu

, The effect of imbalanced datasets on lda: a theoretical and empirical analysis, Pattern Recognition 40 (2007), 557–562.

23.

Estabrooks

and Japkowicz

, A multiple resampling method for learning from imbalanced data sets, Computational Intelligence 20 (2004), 18–36.

24.

Ling

Sheng

and Yang

, Test strategies for cost-sensitive decision trees, IEEE Transactions on Knowledge and Data Engineering 18 (2006), 1055–1067.

25.

Wang

B.X.

and Japkowicz

, Boosting support vector machines for imbalanced data sets, Knowledge and Information Systems 25 (2010), 1–20.

26.

Drummond

and Holte

R.C.

, C4.5, class Imbalance, and cost sensitivity: Why under-sampling beats over-sampling, in: Proceedings of the ICML’ 03 Workshop on learning from Imbalanced Data Sets, 2003.

27.

Lin

Lee

and Wahba

, Support vector machines for classification in nonstandard situations, Machine Learning 46 (2002), 1–3.

28.

Guo

Liu

Zhi

Xiao

and She

, Logistic discrimination based on G-mean and F-measure for imbalanced problem, Journal of Intelligent and Fuzzy Systems 31 (2016), 1155–1166.

29.

Zhang

Liu

and Chen

, Imbalanced data classification based on scaling kernel-based support vector machine, Neural Computing and Applications 25(3–4) (2014), 927–935.

30.

Gao

Chen

Tang

Zhang

and Li

, Adaptive weighted imbalance learning with application to abnormal activity recognition, Neurocomputing 173 (2016), 1927–1935.

31.

Kim

and Namkoong

, Ordinal Classification of Imbalanced Data with Application in Emergency and Disaster Information Services, IEEE Intelligent Systems 31(5) (2016), 50–56.

32.

Duan

Xie

Bai

and Wang

, A new support vector data description method for machinery fault diagnosis with unbalanced datasets, Expert Systems with Applications 64 (2016), 239–246.

33.

Vigneron

and Chen

, A multi-scale seriation algorithm for clustering sparse imbalanced data: application to spike sortin, Pattern Analysis and Applications 19(4) (2015), 1–19.

34.

Zhang

, Cost-sensitive spectral clustering for photo-thermal infrared imaging data, in: Proceedings of International Conference on Information Science and Technology, 2016.

35.

López

Fernández

García

Palade

and Herrera

, An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics, Information Sciences 250 (2013), 113–141.

36.

Galar

Fernandez

Barrenechea

Bustince

and Herrera

, A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid based approaches, IEEE Transactions on Systems, Man, and Cybernetics, Part C 42 (2012), 463–484.

37.

Ren

Cao

Zhao

and Zaiane

, Ensemble based adaptive over-sampling method for imbalanced data learning in computer aided detection of microaneurysm, Computerized Medical Imaging and Graphics 55 (2017), 54–67.

38.

Chawla

N.V.

, C4.5 and imbalanced data sets: investigating the effective of sampling method, probabilistic estimate, and decision tree structure, in: Proceeding of the ICML’ 03 Workshop on learning from Imbalanced Data Sets, 2003.

39.

Alcalá-Fdez

Fernandez

Luengo

Derrac

García

Sánchez

and Herrera

, KEEL Data-Mining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Framework, Journal of Multiple-Valued Logic and Soft Computing 17(2–3) (2011), 255–287.

40.

Yuan

Sun

and LV

, Fingerprint Liveness Detection Based on Multi-Scale LPQ and PCA, China Communications 13(7) (2016), 60–65.

41.

Demsar

, Statistical comparisons of classifiers over multiple data sets, Journal of Machine Learning Research 6 (2006), 1–30.

42.

García

Fernández

Luengo

and Herrera

, Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power, Information Sciences 180 (2010), 2044–2064.

Algorithm 1 EFPUS
Training Phase
Input:
	$\bm{X}_{\textit{maj}}$ – the majority class set
	$\bm{X}_{\textit{min}}$ – the negative class set
	$M$ – the number of base classifiers
	$K$ – the number of feature subsets
Output: a hybrid-based ensemble $\bm{H}$ for the two class imbalance problem
	Begin:
	1. for $i=$ 1 to $M$ do
	2. Randomly sample a subset $X_{i,\textit{maj}}$ from $\bm{X}_{\textit{maj}}$ without replacement, $\|\bm{X}_{i,\textit{maj}}\|=\|\bm{X}_{\textit{min}}\|$ ;
	3. $\bm{X}_{i}=\bm{X}_{i,\textit{maj}}\cup\bm{X}_{\textit{min}}$
	4. Split feature set $F$ into subsets $F_{i,j}$ , $j=$ 1, 2, …, $K$ ;
	5. for $j=$ 1 to $n/K$ do
	6. Randomly draw a subset $\bm{X}_{i,j}$ of $\bm{X}_{i}$ without replacement, with size 50% of $\bm{X}_{i}$ ;
	7. Apply PCA on $\bm{X}_{i,j}$ and $F_{i,j}$ to obtain projection matrix $\bm{R}_{i,j}$ ;
	8. end for
	9. Arrange the $\bm{R}_{i,j}$ in a projection matrix $\bm{R}_{i}$ as in Eq. (1);
	10. $D_{i,\textit{maj}}=\bm{X}_{\textit{maj}}\bm{R}_{i}$ ,
	11. $D_{i,\textit{min}}=\bm{X}_{\textit{min}}\bm{R}_{i}$ ,
	12. Learn $H_{i}$ using $D_{i,\textit{maj}}\cup D_{i,\textit{min}}$
	13. end for

Ensemble based on feature projection and under-sampling for imbalanced learning

Abstract

Keywords

1. Introduction

2.1 Data level

2.2 Algorithm based level

2.3 Ensemble learning

3.1 Leaning process

4.1 Experimental setup

Table 1 The description of the data sets used in this paper

Table 2 The performance of compared algorithms on g-mean

5. Conclusion

Footnotes

Acknowledgments

References

Table 1
The description of the data sets used in this paper

Table 2
The performance of compared algorithms on g-mean