Optimization of selective ensemble for cost-sensitive classification: An empirical study

Abstract

Cost-sensitive classification is broadly investigated in many real-life decision-making applications where the different misclassification errors may cause asymmetric costs. However, the cost values are usually hardly specified in practice for decision makers especially when they are faced with the complex multi-class decision problems. In this paper we attempt to take advantage of the pairwise comparisons originally proposed in Analytic Hierarchy Process (AHP) and offer decision makers a flexible, qualitative manner for cost specification rather than the definite, quantitative cost assignment. The cost ratios associated with the classes are then derived from the comparison matrix and used as the parameter of the subsequent cost-sensitive classifiers. To promote the performance of single classifier we construct the ensembles of multiple cost-sensitive Learning Vector Quantization Neural Networks (LVQ-NNs) trained independently and then combined together by various weighted voting approaches. Empirical study on some real-world databases reveal that the well optimized cost-sensitive ensembles based on evolutionary computing approaches perform significantly better than the best single classifier within the ensemble in terms of generalization ability and stability.

Keywords

Cost-sensitive classification selective ensemble learning evolutionary computing pairwise comparison matrix learning vector quantization neural network

1. Introduction

The misclassification cost has an important role in many real-life applications which favor some decision errors over the others under the consideration of the potential cost. Cost-sensitive learning has therefore received increasing attention in machine learning research with the emphasis on dealing with asymmetric costs of different misclassification errors. Cost-sensitive learning is also regarded as one approach to solve the imbalanced problems that the class distribution in data is highly skewed (i.e., some class samples are overwhelmed by the others) leading to the much lower classification accuracy on the minority class than the majority class. The majority of the existing works in the area of cost-sensitive classification include resampling, instance-weighting, instance-relabeling, post hoc threshold adjusting, and algorithm-level approach [1, 2, 3]. In the literature, the reported research mostly focused on developing advanced learning algorithms, with little attention on cost assignment problem. Most variants for cost-sensitive classification are based on a cost matrix that specifies definitely the cost values of various misclassification errors. In some problems the cost values could be estimated statistically and represented quantitatively. For example, in credit risk assessment the cost of predicting a real “bad” credit incorrectly can be determined by the loss of loan; Alternatively, the cost of predicting a real “good” credit incorrectly can be estimated by the potential profit. However, in reality, it is unrealistic for many decision problems especially when the decision makers are faced with complex multi-class decision problems where a quantity of cost values are involved. It therefore becomes a problem of practical interest to specify the misclassification cost through a qualitative way.

Ensemble learning characterized by a consensus or committee of decision makers refers to the process of constructing a set of learners and aggregating the individual outputs appropriately to attain the final decision. The ensemble designed for classification problems is also called Multiple Classifier System (MCS) and the members within the ensemble are called base (component, or elementary) classifier. Selective ensemble (SEN) is gaining a lot of interest in both academic and industry with a view of reducing the size of ensembles whilst receiving remarkable generalization ability through selecting an optimal subsets of learners. A lot of reported research have found that selective ensemble of appropriate classifiers can improve the generalization error in a variety of real-world decision applications including financial credit assessment [4], image classification [5], chemical process prediction [6], sea-ice thickness estimation [7], rotary machinery fault diagnosis [8], and so forth. The selection (or weighting) of the component classifiers becomes a challenging issue in recent selective ensemble learning studies.

In this paper we propose a selective ensemble framework adopting Learning Lector Quantization Neural Networks (LVQ-NNs) as the base classifiers for cost-sensitive classification. Firstly, motivated by the pairwise comparison matrix originally proposed in Analytic Hierarchy Process (AHP), we divide the cost assignment problem into a series of simple pairwise comparisons which indicate the relative importance of two classes with respect to the impact on the misclassification cost. The cost ratios associated with the classes are then derived from the comparison matrix and applied as parameters of the subsequent cost-sensitive classifiers (i.e., weighted LVQ-NN in this paper). This provides a flexible and qualitative way for decision makers to determine the cost ratios instead of the definite, quantitative cost values. Secondly, a generalized selective ensemble is designed to promote the overall generalization ability via weighted voting combination. We construct an ensemble of multiple weighted LVQ-NN classifiers that are trained independently and then combined appropriately via weight assignment. Various heuristic approaches including simple voting, direct weighted voting, Particle Swarm Optimization (PSO), Genetic Algorithm (GA), and Simulated Annealing (SA) are employed to specify (and optimize) the weights that denote the contribution of learners to the ensemble. Experiments on some popular datasets from UCI database and a real-world financial database reveal that the well optimized cost-sensitive ensembles based on evolutionary computing paradigms can achieve considerably stable performance significantly better than the best single classifier within the ensemble.

The main contribution of this approach is twofold. Firstly, it offers a qualitative and intuitive way to determine the misclassification cost when the definite cost values are difficultly assigned. Secondly, it proposes a generalized selective ensemble able to boost the generalization ability based on evolutionary computing paradigms. The rest of this paper is organized as follows. The related research background is introduced briefly regarding cost-sensitive classification and ensemble learning in Section 2. Section 3 describes the methodology of a generalized cost-sensitive selective ensemble framework focusing attention on the cost determination by means of pairwise comparisons, weighed LVQ-NN algorithm used in the cost-sensitive context, and selective ensemble schemes based on weight optimization. Section 4 includes the description of experimental databases and the comparative study of different selective ensemble schemes. Finally, we conclude the paper along with some future research lines in Section 5.

Figure 1.

General framework of the cost-sensitive selective ensemble and experimental design.

2. Research background

It is well known that all learning methods have their own strength and limitations as stated by the famous “No Free Lunch” (NFL) Theorem in real-world machine learning applications. Therefore, ensemble learning emerged as a promising computing paradigm where several learners are jointly together used for the same decision tasks [9]. To ensure the effective fusion, the learners comprised in the ensemble should have high performance (individual requirement) and low intercorrelation (committee requirement). Many empirical and theoretical studies in a variety of real-world applications have demonstrated that ensemble learning is actually of benefit to improve the stand-alone classifiers although it was not always reported superior to the single best classifier [10, 11].

To make up an ensemble of high overall performance, two ingredients should be considered: one is increasing the diversity of learners, and the other is selecting appropriate classifiers to constitute the ensemble. A variety of approaches have been proposed to construct diverse component classifiers in either homogeneous or heterogenous structure. The homogeneous ensemble is built on multiple classifiers of the same machine learning family by varying the training data (e.g., bagging and boosting), selecting the features of data (e.g., random subspace and rainforest), adjusting the parameters of learning algorithms (e.g., the neuron topology of ANN), or injecting the randomness to the algorithms. Alternatively the heterogenous ensemble is constructed upon multiple classifiers of different learning families. These strategies, no matter what, were developed with the motivation to increase the diversity of learners that make independent misclassifications and potentially lead to better overall decision after combination. The recent studies paid much attention to hybridizing the aforementioned strategies for the purpose of high diversity of elementary classifiers. For example, SVM-based ensembles developed by bagging and random subspace simultaneously [12], or by different kernel functions and feature subsets [13] were shown to outperform the individual SVMs.

Table 1
Some research work on selective ensemble learning

Year	Component classifiers	Fusion approaches	Work
2002	Bootstrap of MLP	GA	[14]
2003	Bootstrap of decision tree	GA	[17]
2005	Bootstrap of SVM	Immune clonal algorithm	[18]
2008	Embedded hidden markov model	Weighted voting	[19]
2009	Decision tree, MLP, SVM	Expected probability	[20]
2009	Random subspace of KNN	Vote, sum, SVM, correspondence analysis, GA	[21]
2012	Bivariate regression	Convex-hull selective-voting algorithm	[22]
2013	KNN, MLP, SVM, NB, C45, ADTree, RBF, LR, BLR, decision table	Greedy search based on individual performance and pairwise diversity	[4]
2013	Bootstrap of MLP	GA, SA	[23]
2013	SVMs of selected features	SA-based multiobjective optimization	[24]
2015	Bootstrap of LVQ-NN	PSO	[25]
2015	Neural networks	PSO	[16]
2015	Random subspace of SVM	Matching pursuit optimization ensemble classifier (MPSEC)	[26]
2017	LR, random forest, KNN, SVM, NB, decision tree	Parallel optimization and hierarchical selection	[27]

Notations: NB: Naive bayes, LR: Logistic regression, BLR: Bayesian logistic regression, LVQ-NN: Learning vector quantization neural networks, GA: Genetic algorithm, PSO: Particle swarm optimization, SA: Simulated annealing.

The prevailing fusion approaches of multiple learners include linear approach (e.g., averaging), nonlinear approach (e.g., majority voting and weighted voting), probabilistic approach (e.g., Bayesian), correspondence analysis, entropy modelling, and stacking methods. Recent studies put emphasis on selective ensembles (SEN) to enhance the generalization ability with pruned ensembles. A pioneer work is a selective ensemble method (GASEN) using GA [14], which specifies initial weights to a set of neural networks and then optimizes the weights using genetic operators with respect to the overall accuracy as the fitness value. The evolved weights determine which learners are retained to make up the ensemble. They demonstrated by selecting the appropriate learners GASEN can produce smaller-sized ensemble of better generalization accuracy compared with Bagging and Boosting. In another work [15] GA was used to select the near optimal subsets of base classifiers with respect to the prediction accuracy and variance influence factor (VIF). Liu et al. designed a general framework of selective ensemble based on particle swarm optimization [16]. Table 1 summarizes some related research work in the discipline of selective ensemble learning, including the component classifiers and fusion approach.

The prior research work demonstrate that the ensemble learning, particular selective ensemble, is a well-recognized technique to improve the prediction performance, and evolutionary computing approaches are commonly employed in building selective ensembles. In this article we employed three evolutionary computing paradigms, namely, PSO, GA, and SA, in cost-sensitive selective ensembles aiming to pursue the optimal weights of ensemble members with respect to overall performance in cost-sensitive context.

3. Methodology of cost-sensitive selective ensemble

Figure 1 outlines the framework of the proposed cost-sensitive selective ensemble and the experimental design in Section 4. At the beginning the relative importance of one class against another is assigned by decision makers to construct a pairwise comparison matrix as was done in Analytic Hierarchy Process (AHP). The matrix is then processed mathematically to calculate the weights (or priorities) of the compared classes. Also the consistency of the comparisons is evaluated under a criterion for accepting or rejecting the pairwise comparisons. If the criterion is not accepted the comparison matrix has to be revised. Once the consistency is verified, the class weights are transformed to the cost ratios of misclassifications so that higher ratio indicates the more importance (or higher cost potentially caused by misclassification) of the corresponding class. In the following phase, the training dataset ( $D_{t}$ ) as well as the derived cost ratios are used to feed a series of cost-sensitive classifiers forming an ensemble through weighted voting strategies. To specify the optimal weights of component classifiers within the ensemble, some evolutionary optimization approaches like Particle Swarm Optimization (PSO), Genetic Algorithm (GA), Simulated Annealing (SA) are used respectively to minimize a defined cost-dependent performance measure using the validation dataset ( $D_{v}$ ). Finally the performance of well-trained ensembles is evaluated using a separated test dataset ( $D_{e}$ ).

3.1 Pairwise comparison matrix and cost ratio calculation

The analytic hierarchy process (AHP) is a mathematical technique able to combine qualitative and quantitative factors for solving the complex decision problems. Since originally developed by Thomas L. Saaty in the 1970s [28], AHP has been extensively studied and applied with success in a wide variety of fields. The process of AHP is characterized by (1) decomposing the decision problem into a hierarchy of comprehended sub-problems where the root is the decision goal, the leaves are the compared alternatives, and the other nodes are the related factors, (2) comparing each element with another with respect to the impact on the above element and forming the pairwise comparison matrices, (3) deriving the weights from the pairwise comparison matrices respectively, (4) evaluating the consistency of comparison matrices and the whole hierarchy, (5) calculating the priorities of the alternatives over the decision goal.

In pairwise comparison matrix, each value represents the relative importance of one element against another with respect to the impact over the element above (probably a factor or the decision goal) within the hierarchy. The traditional method is to specify the comparison judgement as a positive integer from $[1,9]$ for the upper triangular matrix, and fill the lower triangular matrix with the corresponding reciprocal values. Particularly the diagonal values of matrices are 1. By definition the judgement values imply the relative importance scale of two elements shown in Table 2.

Table 2
Paired comparison judgements

Values	Implication
1	The same importance of the two elements
3	The former is slightly more important than the latter
5	The former is significantly more important than the latter
7	The former is strongly more important than the latter
9	The former is extremely more important than the latter
2, 4, 6, 8	The middle value of the above judgments
Reciprocal value	If the relative importance of element $i$ to element $j$ is $a_{ij}$ , then the relative importance of element $j$ to element $i$ is $a_{ji}=1/a_{ij}$

Table 3

An example of pairwise comparison matrix with 6 classes

	C1	C2	C3	C4	C5	C6	Weights	Cost ratios ( $S_{i}$ )
C1	1	2	1	1/7	1/2	1	0.0918	1.8192
C2	1/2	1	2	1/3	1/8	3	0.0879	1.7438
C3	1	1/2	1	1/7	1/4	1	0.0588	1.1655
C4	7	3	7	1	1	9	0.3822	7.5789
C5	2	8	4	1	1	6	0.3289	6.5206
C6	1	1/3	1	1/9	1/6	1	0.0504	1

Consistency evaluation: CI $=$ 0.0990, RI $=$ 1.26, CR $=$ 0.0798.

Given a $k\times k$ pairwise comparison matrix $A$ satisfying that $a_{ij}>0$ , $a_{ji}=1/a_{ij}$ , and $a_{ii}=1$ , we can derive the eigenvector $V$ associated with the biggest eigenvalue $\lambda_{\max}$ of $A$ . The normalized $V$ summing up to one denotes the weights or priorities of the compared elements. It is necessary to evaluate the consistency of comparison matrices in terms of consistency ratio (CR), calculated by CI (consistency index) and RI (random index).

$\displaystyle\textit{CI}=\frac{\lambda_{\max}-k}{k+1}$ $\displaystyle\textit{CR}=\frac{\textit{CI}}{\textit{RI}}$

where RI is the pre-calculated average CI of a large number of randomly generated comparison matrices. The smaller the CR, the better the consistency of the comparison matrix. Normally the consistency of A is acceptable if $\textit{CR}<0.1$ (particularly $\textit{CR}=0$ implies the complete consistency). Otherwise, the comparison matrix does not meet the consistency requirement.

In this paper we take advantage of the comparison matrix to specify the misclassification cost by means of simple and intuitive paired comparisons. In the comparison matrix, the classes are the compared alternatives and the goal is the misclassification cost. Let $k$ denote the number of classes. A $k\times k$ comparison matrix $A$ is established, where the value $A(i,j)$ denotes the relative importance of two classes $C i$ and $C j$ with respect to the potential misclassification cost. Afterwards the weights are calculated from matrix $A$ and the consistency is evaluated. If the consistency does not meet the requirement, the comparison matrix will be reconstructed. Finally the weights are normalized to get the cost ratio of classes $S_{i}(i=1,\ldots,k)$ so that the smallest cost ratio is one. Table 3 shows an example of pairwise comparison matrix composed of 6 classes. The derived weights imply the relative priority of 6 classes with respect to the misclassification cost. The cost ratios after normalization are showed in the last column where the smallest cost ratio (corresponding to C6 in the table) is one. In this example C4 has the highest cost ratio followed by C5, and the other three classes (C1, C2, and C3) have the similar cost ratios. The comparison matrix is accepted as the consistency ratio $\textit{CR}<0.1$ .

Although represented in the similar format, the comparison matrix is different from the traditional cost matrix. (1) In the former the values indicate the qualitative judgement of relative importance of two classes with respect to the potential misclassification cost, whilst in the latter the values indicate the definite cost of misclassification errors. (2) In a $k\times k$ cost matrix the decision maker should specify $k\times(k-1)$ values totally, whilst only half should be specified in the corresponding comparison matrix. Therefore this offers the decision makers a simple and intuitive approach to determine the cost ratios by comparing the classes in a pairwise manner. In this paper a simple hierarchy of two levels is used for cost determination, however it is possible to set up a complicated hierarchy with goal, criteria, and alternatives for complex cost decision problems.

3.2 Weighted learning vector quantization neural network for cost-sensitive classification

Learning vector quantization (LVQ) is a neural network (NN) defined for supervised learning as a variant of self-organized feature map. The neurons are arranged regularly and each is associated with a prototype which represents the class region. LVQ-NN is trained competitively in the manner that for each input the neuron closest (called best-matching neuron) is activated by updating the prototype and adjusting the class boundary accordingly. LVQ is widely applied in a variety of domains with success compared with support vector machines, multivariate statistical methods, and other learning methods [25]. In recent studies some variants of LVQ-NN were developed for learning in the cost-sensitive context [29, 1]. In this paper we used the weighted LVQ-NN algorithm (wLVQ-NN) proposed in [29]. This learning algorithm is applicable for multi-class cost-sensitive classification problems by treating the cost ratios as the weight of training instances during the learning and labeling of the network.

Let $N$ be the number of training instances, $N_{j}$ the number of instances in class $j$ , and $S_{j}$ the cost ratio of class $j$ derived from the comparison matrix. The cost ratios are then distributed to the whole training data so that each instance $x$ is assigned a weight $w(x)$ satisfying that (1) all instance weights sum up to $N$ ; (2) the instances of a class share the same weight; (3) the instances belonging to a costly class are assigned higher weights.

$\displaystyle w(x)=S_{i}\frac{N}{\sum_{j}S_{j}N_{j}},\textit{Class}(x)=i$

Table 4
Weight specification strategies of learners within an ensemble

Strategies	Methods
S-Vote	Assign the same weight to all learners within the ensemble, i.e., $\lambda_{i}=1/m,i=1,\ldots,m$ .
DW-Vote	(1)	Feed the validation data to all learners and calculate the TCR respectively;
	(2)	Compute the weight of learners as the reciprocal value of TCR;
	(3)	Normalize the weights to sum to one.
PSO-WV	(1)	Feed the validation data to all learners and make the prediction independently;
	(2)	Initialize the population of particles characterized by the location (the weights $\lambda_{i},i=1,\ldots,m$ ) and velocity (the learning grade) in the multidimensional space;
	(3)	Initialize the best location of individuals (local best) and swarm (global best);
	(4)	For each individual $x$ in the population do (4.1) Normalize the weights to sum to one; (4.2) Combine the decisions across all learners by the weighted voting principle; (4.3) Evaluate the ensemble in terms of TCR and use the reciprocal as the fitness value of the individual; (4.4) Update the location and velocity of $x$ with respect to the local best and global best;
	(5)	Update the local best and global best for the current population;
	(6)	If the stopping condition is satisfied, output the optimal weights, otherwise go to (4).
GA-WV	(1)	Feed the validation data to all learners and make the prediction independently;
	(2)	Initialize the population by encoding the weights $\lambda_{i}=1/m,i=1,\ldots,m$ in the chromosomes;
	(3)	For each individual x in the population do (3.1) Decode the weights from $x$ and normalize the weights to sum to one; (3.2) Combine the decisions across all learners by the weighted voting principle; (3.3) Evaluate the ensemble in terms of TCR and use the reciprocal as the fitness value of the individual;
	(4)	Select the individuals with high fitness as parents;
	(5)	Perform crossover and mutation on parents;
	(6)	Replace the parents with offsprings if they have higher fitness;
	(7)	If the stopping condition is satisfied, output the optimal weights, otherwise go to (3).
SA-WV	(1)	Feed the validation data to all learners and make the prediction independently;
	(2)	Initialize the weights randomly as the current solution;
	(3)	Generate a new solution from the neighborhood of the current one;
	(4)	Normalize the weights to sum to one;
	(5)	Combine the decisions across all learners by the weighted voting principle;
	(6)	Evaluate the ensemble in terms of TCR used as the objective function value of the individual;
	(7)	Accept the new solution if it is better than the current one, otherwise accept it with a probability determined by Metropolis criteria;
	(8)	Decrease the temperature gradually;
	(9)	Go to (3) until the temperature reaches the predefined minimum.
B-One	(1)	Feed the validation data to all learners and calculate the TCR respectively;
	(2)	Set one as the weight to the learners that has the lowest TCR, and 0 to the others.

The wLVQ-NN algorithm is trained online by incorporating the instance weights into the updating of prototypes. The rational is that the instances of higher weights impact more on the best-matching neuron so that they are harder to be misclassified. The wLVQ-NN algorithm is described in the following.

Algorithm 1: Weighted learning vector quantization Neural Network (wLVQ-NN)

(1) (1)

For $i=1\ldots M$ Initialize $m_{i}$ and $\textit{Class}(m_{i})$ (2)

$t=0$

(3)

For $i=1\ldots N$

(3.1) (3.1)

Input $x_{i}$ to the neural network

(3.2)

For $j=1\ldots M$ Calculate $d(x_{i},m_{j})$

(3.3)

$m_{p}=\mathop{\rm{argmin}}\nolimits_{j}$ $d(x_{i},m_{j})$ // find the best-matching neuron of $x_{i}$

(3.4)

if $\textit{Class}(m_{p})=Class(x_{i})$ then

(3.5)

$m_{p}=m_{p}+\alpha w(x_{i})(x_{i}-m_{p})$

(3.6)

else $m_{p}=m_{p}-\alpha w(x_{i})(x_{i}-m_{p})$ // update the prototype of $m_{p}$

(4)

Assign $m_{p}$ by the weighted voting principle

(5)

$t=t+1$

(6)

Goto (3) until $t$ $=$ max_iterations

Notations: $N$ : number of training instances, $M$ : number of neurons, $x_{i}(i=1,\ldots,N$ ): training instances, $m_{i}(i=1,\ldots,M$ ): prototype of neurons, $w(x_{i})$ : instance weight of $x_{i}$ , $\textit{Class}(x_{i})$ : class label of $x_{i}$ , $\textit{Class}(m_{i})$ : class label of $m_{i}$ , $\alpha$ : learning rate, max_iterations: maximal number of iterations.

After each iteration the neurons are re-labeled by weighted voting principle. The training instances are projected to the best-matching neurons via the nearest principle forming a Voronoi set for each neuron. The Voronoi set is normally composed of instances from different classes. Let $w_{j}$ be the instance weight assigned to class $j$ , $N_{j}$ the number of class $j$ instances in the Voronoi set of the neuron $i$ . The vote of a class $j$ is calculated as the occurrence $N_{j}$ multiplied by the corresponding weights $w_{j}$ . Consequently, the label with the highest vote is assigned to the neuron $i$ .

$\displaystyle\textit{Class}(m_{i})=\underset{j}{\operatorname{argmax}}\,N_{j}w% _{j}$

The number of neurons is the factor that impacts mostly on the performance of LVQ. To generate a series of diverse learners, we adopt wLVQ-NN as the base learning algorithm with Bootstrap (a sampling technique with replacement) and randomly selected number of neurons. Each solution provides a different cost-sensitive classifier to make up the ensembles.

3.3 Performance evaluation

In the cost-sensitive classification context, the performance of classifiers is usually evaluated by the total test cost [30] or expected misclassification cost [31] due to the misclassification errors. In this problem using cost ratios as parameter of cost-sensitive learning, we define the total cost ratio as an extension of expected misclassification cost for multi-class classification. Let $\textit{Err}_{i}$ be the number of misclassified instances in class $i$ , $S_{i}$ the cost ratio of class $i$ , and $k$ the number of classes. The total cost ratio (TCR) is defined as:

$\displaystyle\textit{TCR}=\sum_{i=1}^{k}S_{i}\textit{Err}_{i}/N$

3.4 Generalized selective ensemble schemes

It is known that an ensemble of multiple classifiers is a feasible approach to promote the performance of a single classifier as reported in many studies. Within the ensemble the component classifiers that make the decision independently are combined in some manners. One of the mostly used combination approaches is weighted voting, which directly quantifies the weight of a vote (denoting the degree of confidence to the final decision) to all available learners. Particularly simple ensemble uses majority voting that assigns the same weight to all learners, i.e., the learners are equally important. Ensemble based on weighted voting combination can be seen as a generalized selective ensemble in the sense that in selective ensembles the discarded learners are equivalently specified with a weight of $0$ . Intuitively a higher weight should be specified to the learner of better generalization ability that can be estimated on a validation dataset. Consequently the weight specification becomes a crucial problem and also a key issue of ensemble learning.

In the paper we attempt to apply six strategies, i.e., S-Vote (Simple majority Voting), DW-Vote (Direct Weighted Voting), PSO-WV (Particle Swarm Optimization Weighted Voting), GA-WV (Genetic Algorithm Weighted Voting), SA-WV (Simulated Annealing Weighted Voting), and B-One (Best One), to specify the weights $\lambda_{i}$ of $m$ component classifiers within the ensemble satisfying that $0\leqslant\lambda_{i}\leqslant 1,\sum_{i=1}^{m}\lambda_{i}=1$ . The objective function of the three evolutionary computing approaches, namely PSO, GA, and SA, is the minimization of the total cost ratio on the validation dataset. Table 4 depicts the methodology of these strategies. Accordingly, we can make up different SENs via the weighted voting combination.

SV-SEN:
Combine the component classifiers with the same voting weights (S-Vote);
DWV-SEN:
Combine the component classifiers with the voting weights proportional to the individual performance (DW-Vote);
PSOWV-SEN:
Combine the component classifierswith the voting weights optimized by particle swarm optimization (PSO-WV);
GAWV-SEN:
Combine the component classifiers with the voting weights optimized by genetic algorithm (GA-WV);
SAWV-SEN:
Combine the component classifiers with the voting weights optimized by simulated annealing (SA-WV);
BOne-SEN:
Select the single component classifier of the best performance (B-One).

Table 5
Parameter setting in the experiments

Parameters Values

wLVQ-NN

Number of neurons [10,100]

Learning rate 0.05

Particle swarm optimization

Swarm size 20

Maximal iterations 500

Value of velocity weight at the beginning 0.95

Value of velocity weight at the end 0.4

Maximum velocity step 100

Genetic algorithm

Number of subpopulation 4

Population size 200

Maximal iterations 100

Simulated annealing

Initial temperature 1

Stop temperature 0.01

Maximum number of consecutive rejections 1000

Maximum number of tries within one temperature 300

Maximum number of successes within one temperature 20

Other parameters

No. of learners [10, 50], 10 by default

Number of trails per dataset 50 (10 times of 5-CV)

Given an instance $x$ , the decision of all learners can be represented by a $m\times k$ decision matrix ( $m$ is the number of available learners, and $k$ is the number of classes) where each element $c(i,j)=1$ if $x$ is classified as class $j$ by learner $i$ , and $c(i,l)=0,l\neq j$ . The final decision of the ensemble can be determined by the weighted voting principle, i.e., the class that has the maximal weighted vote over all learners.

$\displaystyle\textit{Class}(x)=\underset{j}{\operatorname{argmax}}\,\,\sum_{i=% 1}^{m}\lambda_{i}c(i,j)$
4. Experiments and results

Parameters	Values
wLVQ-NN
Number of neurons	[10,100]
Learning rate	0.05
Particle swarm optimization
Swarm size	20
Maximal iterations	500
Value of velocity weight at the beginning	0.95
Value of velocity weight at the end	0.4
Maximum velocity step	100
Genetic algorithm
Number of subpopulation	4
Population size	200
Maximal iterations	100
Simulated annealing
Initial temperature	1
Stop temperature	0.01
Maximum number of consecutive rejections	1000
Maximum number of tries within one temperature	300
Maximum number of successes within one temperature	20
Other parameters
No. of learners	[10, 50], 10 by default
Number of trails per dataset	50 (10 times of 5-CV)

In the experiments, we carry out an empirical comparison of cost-sensitive selective ensembles built on various combination strategies. The experiments were conducted in the MATLAB environment as follows:

Step 1: Step 1:
The pairwise comparison matrix is generated randomly and the consistency is verified.
Step 2:
Given an experimental data set $D$ , it is divided into three subsets through 5-fold cross-validation (5-CV) followed by holdout (2/3 parts for training and 1/3 part for validation): $D_{t}$ (training data to train the learners), $D_{v}$ (validation data to specify the weight of learners used in all strategies except S-Vote), and $D_{e}$ (test data to evaluate the performance).
Step 3:
For each generated training data set, the ensembles are constructed based on a series of weighted LVQ-NNs by using Bootstrap and varying the number of neurons randomly selected from 10 to 100. The classifiers are then combined by different weighted voting approaches respectively.
Step 4:
The test data is input to the constructed ensembles respectively, and the performance is evaluated in terms of TCR.
Step 5:
The process is repeated 10 times with randomly generated pairwise comparison matrix, and the average results are calculated.

The selective ensembles are implemented in Matlab environment. Table 5 lists the parameters and values (most parameters use the default setting) used in the experiments.

Table 6
Financial ratios of french database

Variable description

$x_{1}$ - Number of employees previous year $x_{16}$ - Cashflow/turnover

$x_{2}$ - Capital employed/fixed assets $x_{17}$ - Working capital/turnover days

$x_{3}$ - Financial debt/capital employed $x_{18}$ - Net current assets/turnover days

$x_{4}$ - Depreciation of tangible assets $x_{19}$ - Working capital needs/turnover

$x_{5}$ - Working capital/current assets $x_{20}$ - Export

$x_{6}$ - Current ratio $x_{21}$ - Added value per employee in $k$ EUR

$x_{7}$ - Liquidity ratio $x_{22}$ - Total assets turnover

$x_{8}$ - Stock turnover days $x_{23}$ - Operating profit margin

$x_{9}$ - Collection period days $x_{24}$ - Net profit margin

$x_{10}$ - Credit period days $x_{25}$ - Added value margin

$x_{11}$ - Turnover per employee $k$ EUR $x_{26}$ - Part of employees

$x_{12}$ - Interest/turnover $x_{27}$ - Return on capital employed

$x_{13}$ - Debt period days $x_{28}$ - Return on total assets

$x_{14}$ - Financial debt/equity $x_{29}$ - EBIT margin

$x_{15}$ - Financial debt/cashflow $x_{30}$ - EBITDA margin

Table 7
Performance comparison of various ensemble schemes in terms of average TCR. Each is associated with $x/y/z$ which denotes the number of approaches inferior to ( $x$ ), comparable with ( $y$ ) or superior over ( $z$ ) the underlining strategy respectively at 5% significance level. The best results for each dataset are bolded

Datasets Size No. No. SV-SEN DWV-SEN PSOWV-SEN GAWV-SEN SAWV-SEN BOne-SEN W-One

attr class

French 1200 30 2 0.4212 (1/0/5) 0.4038 (2/1/3) 0.3714 (3/3/0) 0.3697 (4/2/0) 0.3682 (4/2/0) 0.3912 (2/2/2) 0.6891 (0/0/6)

Horse 368 22 2 0.6285 (1/5/0) 0.5726 (2/4/0) 0.5778 (2/4/0) 0.5981 (1/5/0) 0.5958 (1/5/0) 0.6318 (1/3/2) 1.4093 (0/0/6)

German 1000 24 2 0.5937 (1/1/4) 0.5862 (1/1/4) 0.4708 (4/2/0) 0.4774 (3/3/0) 0.4887 (3/2/1) 0.4768 (3/3/0) 0.7029 (0/0/6)

Lung 33 56 2 0.5133 (1/4/1) 0.4500 (1/5/0) 0.4334 (1/5/0) 0.4134 (1/5/0) 0.4034 (2/4/0) 0.4700 (1/5/0) 1.0367 (0/0/6)

Credit 690 15 2 0.3623 (1/1/4) 0.3599 (1/1/4) 0.3331 (3/3/0) 0.3271 (3/3/0) 0.3283 (3/3/0) 0.3239 (3/3/0) 0.5740 (0/0/6)

Abalone 4177 8 3 1.5490 (1/0/5) 1.5300 (2/0/4) 1.3220 (5/1/0) 1.3404 (4/2/0) 1.3693 (3/1/2) 1.3567 (3/2/1) 1.7957 (0/0/6)

Iris 150 4 3 0.1438 (1/5/0) 0.1412 (1/5/0) 0.1641 (1/4/1) 0.1493 (1/5/0) 0.1476 (1/5/0) 0.1381 (2/4/0) 0.3994 (0/0/6)

Glass 241 10 6 1.8756 (1/4/1) 1.8603 (1/4/1) 1.7860 (4/2/0) 1.8256 (1/5/0) 1.8152 (1/5/0) 1.8871 (1/4/1) 2.8811 (0/0/6)

Wine 178 13 3 0.2124 (1/3/2) 0.1683 (4/2/0) 0.2000 (1/3/2) 0.1740 (3/3/0) 0.1920 (1/5/0) 0.2183 (1/4/1) 0.5227 (0/0/6)

Zoo 101 7 7 0.3530 (1/2/3) 0.3425 (1/2/3) 0.2821 (4/2/0) 0.2737 (4/2/0) 0.2987 (4/2/0) 0.3896 (1/2/3) 1.1054 (0/0/6)

Ave. 0.6102 0.5853 0.5424 0.5431 0.5482 0.5745 1.1402

4.1 Data sets

Variable description
$x_{1}$ -	Number of employees previous year	$x_{16}$ -	Cashflow/turnover
$x_{2}$ -	Capital employed/fixed assets	$x_{17}$ -	Working capital/turnover days
$x_{3}$ -	Financial debt/capital employed	$x_{18}$ -	Net current assets/turnover days
$x_{4}$ -	Depreciation of tangible assets	$x_{19}$ -	Working capital needs/turnover
$x_{5}$ -	Working capital/current assets	$x_{20}$ -	Export
$x_{6}$ -	Current ratio	$x_{21}$ -	Added value per employee in $k$ EUR
$x_{7}$ -	Liquidity ratio	$x_{22}$ -	Total assets turnover
$x_{8}$ -	Stock turnover days	$x_{23}$ -	Operating profit margin
$x_{9}$ -	Collection period days	$x_{24}$ -	Net profit margin
$x_{10}$ -	Credit period days	$x_{25}$ -	Added value margin
$x_{11}$ -	Turnover per employee $k$ EUR	$x_{26}$ -	Part of employees
$x_{12}$ -	Interest/turnover	$x_{27}$ -	Return on capital employed
$x_{13}$ -	Debt period days	$x_{28}$ -	Return on total assets
$x_{14}$ -	Financial debt/equity	$x_{29}$ -	EBIT margin
$x_{15}$ -	Financial debt/cashflow	$x_{30}$ -	EBITDA margin

Datasets	Size	No.	No.	SV-SEN	DWV-SEN	PSOWV-SEN	GAWV-SEN	SAWV-SEN	BOne-SEN	W-One
		attr	class
French	1200	30	2	0.4212 (1/0/5)	0.4038 (2/1/3)	0.3714 (3/3/0)	0.3697 (4/2/0)	0.3682 (4/2/0)	0.3912 (2/2/2)	0.6891 (0/0/6)
Horse	368	22	2	0.6285 (1/5/0)	0.5726 (2/4/0)	0.5778 (2/4/0)	0.5981 (1/5/0)	0.5958 (1/5/0)	0.6318 (1/3/2)	1.4093 (0/0/6)
German	1000	24	2	0.5937 (1/1/4)	0.5862 (1/1/4)	0.4708 (4/2/0)	0.4774 (3/3/0)	0.4887 (3/2/1)	0.4768 (3/3/0)	0.7029 (0/0/6)
Lung	33	56	2	0.5133 (1/4/1)	0.4500 (1/5/0)	0.4334 (1/5/0)	0.4134 (1/5/0)	0.4034 (2/4/0)	0.4700 (1/5/0)	1.0367 (0/0/6)
Credit	690	15	2	0.3623 (1/1/4)	0.3599 (1/1/4)	0.3331 (3/3/0)	0.3271 (3/3/0)	0.3283 (3/3/0)	0.3239 (3/3/0)	0.5740 (0/0/6)
Abalone	4177	8	3	1.5490 (1/0/5)	1.5300 (2/0/4)	1.3220 (5/1/0)	1.3404 (4/2/0)	1.3693 (3/1/2)	1.3567 (3/2/1)	1.7957 (0/0/6)
Iris	150	4	3	0.1438 (1/5/0)	0.1412 (1/5/0)	0.1641 (1/4/1)	0.1493 (1/5/0)	0.1476 (1/5/0)	0.1381 (2/4/0)	0.3994 (0/0/6)
Glass	241	10	6	1.8756 (1/4/1)	1.8603 (1/4/1)	1.7860 (4/2/0)	1.8256 (1/5/0)	1.8152 (1/5/0)	1.8871 (1/4/1)	2.8811 (0/0/6)
Wine	178	13	3	0.2124 (1/3/2)	0.1683 (4/2/0)	0.2000 (1/3/2)	0.1740 (3/3/0)	0.1920 (1/5/0)	0.2183 (1/4/1)	0.5227 (0/0/6)
Zoo	101	7	7	0.3530 (1/2/3)	0.3425 (1/2/3)	0.2821 (4/2/0)	0.2737 (4/2/0)	0.2987 (4/2/0)	0.3896 (1/2/3)	1.1054 (0/0/6)
Ave.				0.6102	0.5853	0.5424	0.5431	0.5482	0.5745	1.1402

In the experiments, 9 datasets, namely Horse, German, Lung, Credit, Iris, Wine, Abalone, Glass, and Zoo, obtained from UCI database [32] are used for performance comparison both in binary class and multi-class. Another benchmark dataset studied in the experiments is French, a financial data set of small or middle scaled business companies described in Table 6. It was used to predict the status (healthy or distress) of a company over a period of years. The data is preprocessed to generate a balanced subset composed of 600 distressed companies and 600 healthy ones. The properties of the experimental data sets are characterized in Table 7. The categorical features contained in horse and credit are converted to several binary ones by means of one categorical value corresponding to one binary feature in the new data set.

Table 8
Pairwise significance test of various ensemble schemes in terms of average TCR over 10 datasets

	DWV-SEN	PSOWV-SEN	GAWV-SEN	SAWV-SEN	BOne-SEN
SV-SEN	0.0002*	0.0000*	0.0000*	0.0000*	0.0013*
DWV-SEN		0.0000*	0.0000*	0.0000*	0.2440
PSOWV-SEN			0.8798	0.2878	0.0000*
GAWV-SEN				0.2116	0.0001*
SAWV-SEN					0.0011*
BOne-SEN

*denote the statistical significance at 5% level.

Figure 2.

Stability comparison of different SEN schemes.

Figure 3.

Performance comparison by varying the size of ensembles on French dataset.

4.2 Results and discussion

In Table 7, the performance results of various ensemble schemes with a fixed number (10 by default) of learners on the test dataset are presented in terms of the TCR value. The results on each dataset have been averaged over 50 trials. The last column gives the performance resulted from the worst single learner (W-One). Each ensemble is associated with a 3-tuple $(x/y/z)$ where $x(z)$ denotes the number of alternatives inferior to (superior over) the underlining scheme at 5% significance level, and y denotes the number of alternatives showing no statistical significance with the competing strategy. It is showed that the five ensembles always outperform significantly the worst single classifier regardless of the experimental databases. Particularly GAWV-SEN achieves the best performance on 2 datasets (namely, Credit and Zoo) and comparable performance with the winner on the rest. By contrast, the other four ensembles perform significantly worse than at least one competitor in some datasets. It is worth to note that the B-One scheme receives the best performance only on Iris dataset without significant benefit compared with GAWV-SEN and SAWV-SEN, however it performs significantly inferior to the winner on 6 datasets. It demonstrates the effectiveness of evolutionary computing on boosting the generalization error of ensembles. The average value over all experiments (50 trails for 10 databases) are shown in the last row. On the average across ten databases, PSO-ensemble yields the best performance in terms of TCR, followed by GA-ensemble and SA-ensemble. The three ensembles achieve the comparable results without statistical significance as shown in Table 8. It should be noted that the three selective ensembles based on evolutionary computing achieve the significantly better performance than the other schemes (including the best single classifier). Particularly the commonly used simple majority voting combination does not contribute to the generalization ability of the ensemble compared with the best single classifier.

The stability property denotes the robustness of an algorithm on different data sets. Let $D_{1},\ldots,D_{n}$ be the experimental datasets, $\textit{SEN}_{1},\ldots,\textit{SEN}_{m}$ the compared SEN schemes, $v_{i,j}$ denote the TCR value of scheme $\textit{SEN}_{i}$ on data $D_{j}$ . For a particular data set, the maximal value among all schemes described in Table 8 corresponds to the worst performance, then the relative performance $b_{i,j}$ is the absolute value divided by the maximum. The stability $r_{i}$ of a particular scheme is evaluated by summing up the relative performance over all datasets. A smaller value of stability indicates the more robust of a particular scheme.

$\displaystyle b_{i,j}=\frac{v_{i,j}}{\max_{i=1}^{m}v_{i,j}}$ $\displaystyle r_{i}=\sum\nolimits_{j=1}^{n}b_{i,j}$

The stability of different SEN schemes in terms of the sum of relative importance w.r.t. TCR is shown in Fig. 2. The distribution of relative performance on each data set is plotted in stack. It is found that referring to the stability, the ensemble schemes based on evolutionary algorithms (namely, PSOWV-SEN, GAWV-SEN, and SAWV-SEN) perform better than the ensembles upon simple voting and direct weighted voting combination, as well as the best single classifier.

In the following experiment we investigate the performance of different ensemble schemes by varying the size of ensemble from 10 to 50 at the same pairwise comparison matrix. In Fig. 3, French dataset is taken as an example. It was found that increasing the size of ensembles does not always contribute to the overall generalization ability. The ensembles based on evolutionary computing paradigms (particulary PSO and GA) outperform those using simple and direct weighted voting combination.

5. Conclusions and future work

Cost-sensitive learning and ensemble computing have drawn a lot of research interests in the machine learning field. This paper focuses on the cost-sensitive selective ensemble framework and provides two insights to the investigated field. Firstly different from the commonly used cost matrix, a novel approach is presented that specifies the relative importance of two classes (alternatives) regarding the misclassification cost (decision goal) through a series of pairwise comparisons. The element weights derived from the comparison matrix denote the cost ratios of compared classes. This approach assists decision makers to determine the misclassification cost through a qualitative and intuitive way when the definite cost values in traditional cost matrix are difficultly assigned and hence limit the application in real-life decision problems. Secondly, a selective ensemble framework is constructed to promote the overall generalization ability by adopting the weighted LVQ-NNs as the base classifiers and weighted voting combination through some heuristic approaches in order to achieve better performance. Experiments on some real-life datasets from diverse domains reveal the impact of combination strategies on the generalization ability of ensembles. The selective cost-sensitive ensembles based on evolutionary computing paradigm for ensemble weight adjustments demonstrate promising performance compared with the best single classifier in terms of discriminatory power and stability.

In the future work, some limitations of this empirical study will be addressed as the research directions. Firstly, the selective ensembles were established in homogeneous structure in the sense that all learners were designed by weighted LVQ-NN as the learning method by varying the training instances and parameters. As the framework is applicable to any learning method that can take the cost ratios as parameters, we will constitute various ensembles in heterogenous structure to enhance the diversity of members. Further comparative experiments with competing ensembles such as Bagging and Random Subspace will be done in future research. Secondly, in the proposed approaches the weights of learners are optimized to best fit the validation data. It is of worth to investigate the overfitting problem of evolutionary optimization that probably degrades the generalization ability of ensembles. Thirdly, the proposed ensembles are based on weighted voting combination so that all learners join the ensembles with specified voting weights. A threshold will be used to discard some learners completely and make up an ensembles of small size in future study.

Footnotes

Acknowledgments

This work was supported by national funds through the Beijing National Science Foundation (9182017), President Youth Fund of Institutes of Sciences and Development, CAS (Y7X1151Q01), National Natural Science Foundation of China (11601129), and Introduction of Talent Research Fund of Henan Polytechnic University (Y2017-1).

References

Chen

Ribeiro

Vieira

Duarte

Neves

JAC

. Incorporate cost matrix into learning vector Quantization modeling: A comparative study of genetic algorithm, simulated annealing and particle swarm optimization. International Journal of Computer Theory and Engineering (IJCTE). 2011; 122-129.

Tapkan

Ozbakir

Kulluk

Baykasoglu

. A cost-sensitive classification algorithm: BEE-Miner. Knowledge-based Systems. 2016; 95(C): 99-113.

Bernard

Chatelain

Adam

Sabourin

. The multiclass ROC front method for cost-sensitive classification. Pattern Recognition. 2016; 52(C): 46-60.

Chen

Ribeiro

. A consensus approach for combining multiple classifiers in cost-sensitive bankruptcy prediction. in: 11th International Conference on Adaptive and Natural Computing Algorithms (ICANNGA’13). Lausanne, Switzerland. 2013; 266-276.

Han

Zhang

Wang

. Polarimetric SAR image classification based on selective ensemble learning of sparse representation. in: IGARSS-IEEE International Geoscience and Remote Sensing Symposium. 2016; 4964-4967.

Shao

Tian

. Adaptive soft sensor for quality prediction of chemical processes based on selective ensemble of local partial least squares models. Chemical Engineering Research & Design. 2015; 95: 113-132.

Mozaffari

Scott

Azad

Chenouri

. A hierarchical selective ensemble randomized neural network hybridized with heuristic feature selection for estimation of sea-ice thickness. Applied Intelligence. 2016; 1-18.

Wang

Zhou

. Fault diagnosis for rotary machinery with selective ensemble neural networks. Mechanical Systems & Signal Processing. 2017; Available from: http//dx-doi-org.web.bisu.edu.cn/10.1016/j.ymssp.2017.03.051.

Rokach

. Ensemble-based classifiers. Kluwer Academic Publishers. 2010.

10.

Lorena

Carvalho

ACPLF

, A review on the combination of binary classifiers in multiclass problems. Artificial Intelligence Review. 2008; 30(1-4): 19-37.

11.

Tulyakov

Jaeger

Govindaraju

Doermann

. Review of classifier combination methods. in: Machine Learning in Document Analysis and Recognition. vol. 90 of Studies in Computational Intelligence. Marinai S, Fujisawa H, editors. Springer Berlin Heidelberg. 2008; 361–386.

12.

Wang

. A hybrid ensemble approach for enterprise credit risk assessment based on support vector machine. Expert Systems with Applications. 2012; 39(5): 5325-5331.

13.

Sun

. Financial distress prediction using support vector machines: Ensemble vs. individual. Applied Soft Computing. 2012; 12(8): 2254-2265.

14.

Zhou

Tang

. Ensembling neural networks: Many could be better than all. Artificial Intelligence. 2002; 137(1): 239-263.

15.

Kim

Kang

. Classifiers selection in ensembles using genetic algorithms for bankruptcy prediction. Expert Systems with Applications. 2012; 39(10): 9308-9314.

16.

Liu

Dong

Shen

Yan

Rui

, et al. Particle swarm optimization based selective ensemble of online sequential extreme learning machine. Mathematical Problems in Engineering. 2015; 2015: 1-10.

17.

Zhou

Tang

. Selective ensemble of decision trees. Lecture Notes in Computer Science. 2003; 2639: 476-483.

18.

Zhang

Wang

Shan

Jiao

. Selective SVMs ensemble driven by immune clonal algorithm. Lecture Notes in Computer Science. 2005; 3449: 325-333.

19.

Gao

Zhong

Tian

. Face sketch synthesis algorithm based on E-HMM and selective ensemble. IEEE Transactions on Circuits & Systems for Video Technology. 2008; 18(4): 487-496.

20.

Hung

Chen

. A selective ensemble based on expected probabilities for bankruptcy prediction. Expert Systems with Applications. 2009; 36(3, Part 1): 5297-5303.

21.

Nanni

Lumini

. A genetic encoding approach for learning methods for combining classifiers. Expert Systems with Applications. 2009; 36(4): 7510-7514.

22.

Kodell

Zhang

Siegel

Nagarajan

. Selective voting in convex-hull ensembles improves classification accuracy. Artificial Intelligence in Medicine. 2012; 54(3): 171-179.

23.

Soares

Antunes

Araujo

. Comparison of a genetic algorithm and simulated annealing for automatic neural network ensemble development. Neurocomputing. 2013; 121: 498-511.

24.

Ekbal

Saha

. Combining feature selection and classifier ensemble using a multiobjective simulated annealing approach: application to named entity recognition. Soft Computing. 2013; 17(1): 1-16.

25.

Yang

. Monitoring and diagnosing of mean shifts in multivariate manufacturing processes using two-level selective ensemble of learning vector quantization neural networks. Journal of Intelligent Manufacturing. 2015; 26(4): 1-15.

26.

Liu

. A selective multiclass support vector machine ensemble classifier for engineering surface classification using high definition metrology. Journal of Manufacturing Science & Engineering. 2015; 137(1): 011003-1-15.

27.

Wei

Wan

Guo

Wong

. A novel hierarchical selective ensemble classifier with bioinformatics application. Artificial Intelligence in Medicine. 2017; Available from: DOI: 10.1016/j.artmed.2017.02.005.

28.

Saaty

. Modeling unstructured decision problems? the theory of analytical hierarchies. Mathematics & Computers in Simulation. 1978; 20(3): 147-158.

29.

Chen

Ribeiro

Vieira

Duarte

Neves

. Weighted learning vector quantization to cost-sensitive learning. in: International Conference on Artificial Neural Networks. 2010; 277-281.

30.

Qiu

Jiang

. Randomly selected decision tree for test-cost sensitive learning. Applied Soft Computing. 2017; 53: 27-33.

31.

Chen

Ribeiro

Chen

. Comparative study of classifier ensembles for cost-sensitive credit risk assessment. Intelligent Data Analysis. 2015; 19(1): 127-144.

32.

Dheeru

Karra Taniskidou

. UCI Machine Learning Repository. http://archiveics.uci.edu/ml. 2017; Available from: http//archive.ics.uci.edu/ml.

Optimization of selective ensemble for cost-sensitive classification: An empirical study

Abstract

Keywords

1. Introduction

Table 1 Some research work on selective ensemble learning

3.1 Pairwise comparison matrix and cost ratio calculation

Table 2 Paired comparison judgements

Table 4 Weight specification strategies of learners within an ensemble

3.4 Generalized selective ensemble schemes

Table 8 Pairwise significance test of various ensemble schemes in terms of average TCR over 10 datasets

5. Conclusions and future work

Footnotes

Acknowledgments

References

Table 1
Some research work on selective ensemble learning

Table 2
Paired comparison judgements

Table 4
Weight specification strategies of learners within an ensemble

Table 8
Pairwise significance test of various ensemble schemes in terms of average TCR over 10 datasets