Deterministic oversampling methods based on SMOTE

Abstract

In supervised classification if one of the classes has fewer objects than the other, we have a class imbalance problem. One of the most common solutions to address class imbalance problems is oversampling, and SMOTE is the most referenced and well-known oversampling method. However, SMOTE creates synthetic objects in a random way, therefore it produces a different result each time it is applied, and in practice the user has to apply SMOTE several times for choosing the best of all the generated balanced datasets. For this reason, in this paper, we present SMOTE-D, a deterministic version of SMOTE, and propose new deterministic SMOTE-D-based versions of some of the most recent and successful SMOTE-based methods. In our experiments, we show that all proposed deterministic methods produce as good results as random methods but our proposals need to be applied just once. This is very important from a practical point of view since our proposals save time by avoiding multiple applications of them as SMOTE does and they provide one unique result.

Keywords

Imbalanced datasets oversampling supervised classification

1 Introduction

In supervised classification when one of the classes has fewer objects compared to the other class, it means we have a class imbalance problem [9 , 36]. This difference in the number of objects between classes produces a bias in the classifiers towards the class with the largest number of objects (majority class), which results in poor classification performance for the class with fewer objects (minority class).

In the literature, some authors [10 , 33] identify two types of solutions for the class imbalance problem: algorithmic-level solutions and data-level solutions. Algorithmic-level solutions [8 , 38] change the strategies of the classifiers to impose a bias in order to improve the classification performance for the minority class. Meanwhile, data-level solutions add or remove objects to/from the classes in order to balance the dataset. Data-level solutions are divided into three types: subsampling [13 , 35], oversampling [3 , 29] and hybrid resampling [2 , 25]. In addition, some authors [20] recognize another type of solutions denominated cost-sensitive solutions [19 , 37], which take into account different cost of misclassification for each class. It is important to underline that some authors [27] argue that cost-sensitive solutions are included into algorithmic-level solutions.

In this paper, we will focus on SMOTE [4] (Synthetic Minority Over-sampling TEchnique), which is one of the most successful and referenced data-level solutions for class imbalance problems. SMOTE, in order to balance the classes, randomly creates synthetic objects between objects of the minority class. Due to its simplicity and good results, SMOTE has been modified and/or combined with other techniques; resulting in other oversampling methods (SMOTE-based methods) which perform better than SMOTE. However, since SMOTE is based on randomly creating synthetic objects, it produces a different balanced sample each time it is applied. This makes that, in the practice, SMOTE has to be applied several times in order to choose among all the balanced samples the best one. For this reason, in this paper, we present a deterministic oversampling method named SMOTE-D, which, unlike SMOTE, creates synthetic objects for the minority class in a deterministic way, producing the same balanced sample each time that it is applied. Preliminary ideas of SMOTE-D were reported in [32], but in this new paper, we further explain each one of its steps as well as we show a more complete experimental evaluation, including a comparison of SMOTE-D against the original SMOTE. Additionally, we introduce and evaluate new deterministic oversampling methods by incorporating SMOTE-D into some of the most recent and successful oversampling SMOTE-based methods. Similarly, we perform an extensive experimental comparison of SMOTE-based and SMOTE-D-based methods.

The rest of this paper is organized as follows: in section 2, we review the related work; in section 3, the proposed methods are introduced; in section 4, our experimental results are reported and discussed; Finally, in section 5, our conclusions and some directions for future work are provided.

2 Related work

In this section, we review SMOTE and some of the most successful and recent SMOTE-based methods proposed in the literature which can be divided into two types: SMOTE-combinations and SMOTE-modifications. SMOTE-combinations [2 , 25– 27] combine SMOTE with other methods like subsampling methods, classifiers, noise filters or dimensionality reduction methods. On the other hand, SMOTE-modifications [3 , 24] modify some of the steps of SMOTE, for example, how the objects used as basis for creating new synthetic objects are chosen; allowing the use of objects from the majority class for creating synthetic objects; creating synthetic objects closer or farther away from the minority class objects; among others.

In this work, we focus on SMOTE modifications, since for SMOTE combinations do not become new over-sampling methods.

2.1 SMOTE

SMOTE [4], see Algorithm 2.1, is an oversampling method in which depending upon the required amount of oversampling (it is a parameter), some synthetic objects are created and added to the minority class to balance the classes. For each object in the minority class, SMOTE randomly selects one of its nearest neighbors, also from the minority class, to create new synthetic objects between them. The new synthetic objects are created as follows: the difference between the attribute vector of an object of the minority class and the attribute vector of the randomly selected nearest neighbor is computed. After, this difference is multiplied by a random number between 0 and 1 and added to the attribute vector of the object of the minority class.

Algorithm 2.1: SMOTE algorithm for randomly creating synthetics objects:

1. em Compute the amount of oversampling (n) for the objects of the minority class.

2. Whilen ≠ 0

3. ForEach object in the minority class

4. Randomly choose one of its k-nearest neighbors.

5. ForEach attribute l of the object

6. Compute the attribute difference between the neighbor and the object. (dif [l] = neighbor [l] - object [l])

7. Synthetic_new [l] = obecj [l] + dif [l] * random (0, 1).

8. End ForEach

9. Add the new synthetic object to the synthetic subset

10. End ForEach

11. n = n - 1

12. End While

13. Add the synthetic subset to the training set.

2.2 SMOTE modifications

In this subsection, we will briefly describe some of the most successful and recent modifications of SMOTE. These methods oversample the minority class by randomly creating synthetic objects following different strategies.

Borderline-SMOTE1 [12], for balancing a dataset, first computes for each object in the minority class its k-nearest neighbors in the whole training set. Then, each object of the minority class is labeled as border if the number of its k-nearest neighbors belonging to the majority class is greater than or equal to k/2 but smaller than k. Finally, Borderline-SMOTE1 creates synthetic objects as SMOTE does, but only between objects labeled as border and their k-nearest neighbors in the minority class.

Borderline-SMOTE2 [12] labels the objects of the minority class as Bordeline-SMOTE1 does. Then, for each border object in the minority class, Borderline-SMOTE2 computes its k-nearest neighbors in the minority class and its nearest neighbor in the majority class, and applies SMOTE over the border objects using these k+1 neighbors. But, when a neighbor is randomly chosen for creating a synthetic object between it and a border object, if the chosen nearest neighbor is the one belonging to the majority class, the difference between the attribute vector of the border object under consideration and the chosen neighbor is multiplied by a random number between 0 and 0.5. Otherwise the attribute difference is multiplied by a random number between 0 and 1 as in SMOTE.

Safe Level SMOTE [3], for each object of the minority class, computes a safe level (sl) value depending on the amount of objects of the minority class among its k-nearest neighbors in the whole training set. The higher the number of neighbors of the minority class, the higher the safe level value for an object. Then, Safe Level SMOTE creates synthetic objects as SMOTE does, but if the safe level of an object is 0, Safe Level SMOTE will not create synthetic objects for this object. For creating synthetic objects, when a nearest neighbor is randomly chosen, Safe Level SMOTE works as follows: if the safe level of the chosen neighbor is 0, the object of the minority class is duplicated. Otherwise, Safe Level SMOTE computes a safe level ratio (slr) by dividing the safe level of the object of the minority class under consideration and the safe level of the chosen neighbor. Later, if slr = 1, the difference between the attribute vector of the object under consideration and its neighbor (attribute difference) is multiplied by a random number between 0 and 1; if slr > 1, the attribute difference is multiplied by a random number between 0 and 1/slr, this will create a synthetic object closer to the object than to the neighbor; if slr < 1, the attribute difference is multiplied by a random number between 1 - slr and 1, this will create a synthetic object closer to the neighbor than to the object.

M-SMOTE [14] labels each object of the minority class as latent noise if all its k-nearest neighbors computed in the whole training set belong to the majority class; as security if anyone of its k-nearest neighbors belongs to the majority class; or as border if the object is neither security nor latent noise. For latent noise objects, M-SMOTE does not create synthetic objects; for security objects, M-SMOTE creates synthetic objects between them and their k-nearest neighbors in the minority class; and for each border object, M-SMOTE only creates synthetic objects between it and its nearest neighbor. For both, security and border objects, the synthetic objects are created in the same way as in SMOTE.

SMOTE-Out [15] works as SMOTE, but when creating a synthetic object for an object in the minority class, SMOTE-Out first creates a synthetic object between the object of the minority class and one of its k-nearest neighbors in the majority class (randomly chosen), but multiplying the attribute difference by a random number between -0.3 and 0, in this way this synthetic object will be closer to the object of the minority class than to its neighbor in the majority class, but in the opposite side. After, the created synthetic object is used for creating another synthetic object between it and one of the k-nearest neighbors in the minority class (also randomly chosen) of the object from which the first synthetic object was created, but multiplying the attribute difference by a random number between 0 and 0.5, by doing this the synthetic object will be closer to the first synthetic object created than to the chosen nearest neighbor of the minority class.

SMOTE-Cosine [15] creates synthetic objects in the same way as SMOTE, but combining the Euclidean and Cosine distances to get the k-nearest neighbors. First, separately for each distance SMOTE-Cosine ranks the neighbors of an object of the minority class based on its distance value. This produces two rankings. After, the neighbors are ranked according to the sum of their rank position in each ranking. Finally, based on this new ranking SMOTE-Cosine computes the k-nearest neighbors.

Selected SMOTE [15] works as SMOTE but, for creating synthetic objects, it only uses some attributes chosen by a feature selection method (any feature selection method can be used). In the created synthetic object the non-selected attributes will have the same values as the object from which they were created.

It is important to point out that SMOTE and all SMOTE-based methods described above have random steps either at choosing neighbors or at creating synthetic objects. Therefore, they produce a different result each time they are applied, and in the practice they have to be applied several times in order to chose the best one balanced sample.

3 Proposed Method

In this section, we first describe SMOTE-D, which is a deterministic version of SMOTE. Later, we introduce deterministic versions of some SMOTE-modifications. We only describe how to use SMOTE-D for SMOTE-modifications because in SMOTE-combinations replacing SMOTE by SMOTE-D is a trivial task.

3.1 SMOTE-D

As we mentioned in section 2, since SMOTE and SMOTE-based methods randomly create synthetic objects, these methods produce a different result each time they are applied. Therefore, these methods usually have to be applied several times for choosing the best result from all balanced datasets. In order to eliminate this randomness from SMOTE and SMOTE-based methods, we propose a deterministic version of SMOTE which we name SMOTE-D. SMOTE-D is based on three ideas: 1) deterministically computing the number of synthetic objects that will be created around each object of the minority class. 2) From the number of synthetic objects computed in (1) for each object of the minority class, deterministically computing how many synthetic objects will be created between each object of the minority class and each one of its k-nearest neighbors. 3) uniformly creating synthetic objects between each object of the minority class and each one of its k-nearest neighbors.

SMOTE-D, as SMOTE, first determines the total amount of over-sampling (n) for the minority class, but according to the difference between the number of objects of the minority class (|m|) and the majority class(|M|); unlike SMOTE which only takes into account the size of the minority class. SMOTE-D computes n as in Expression 1:

$n = round ((| M | - | m |) * R)$ (1)

Where |M| is the number of objects in the majority class, |m| is the number of objects in the minority class, R ∈ I R is a parameter that allows computing the amount of oversampling as a portion of the difference between |M| and |m|. For example, if |m|=10, |M|=100 and R = 1 then n = round ((100 - 10) *1) =90. If we want an oversampling of 80% of the difference between the classes, we need R = 0.8, consequently, the value of n will be n = round ((100 - 10) *0.8) =72.

Given the amount of oversampling n, in order to compute the portion of objects from n that will be created around each object of the minority class, SMOTE-D takes into account the standard deviation of the distances between each object in the minority class and its k-nearest neighbors. Notice that a high standard deviation represents that there is a high difference among the distances from an object and its k-nearest neighbors, we assume this situation as a high opportunity for creating synthetic objects. Therefore, the greater the standard deviation of the distances between an object and its k-nearest neighbors, the greater the number of synthetic objects to be created around this object.

Then, for computing the portion of synthetic objects p_i from n that will be created around each object O_i (i = 1, . . . , |m|) in the minority class, SMOTE-D computes the standard deviation σ_i of the distances between O_i and its k-nearest neighbors $O_{i}^{j}$ (j = 1, . . . , k) and divides this value by the sum of the standard deviations for all the objects in the minority class; see Expression 2.

$p_{i} = σ_{i} / \sum_{s = 1}^{| m |} σ_{s}$ (2)

According to Expression 2, more synthetic objects will be created around those objects of the minority class having higher standard deviation of distances between them and their k-nearest neighbors.

For example, let O₁, O₂ and O₃ be three objects in the minority class and suppose that the standard deviations of the distances between them and their k-nearest neighbors are σ₁ = 2, σ₂ = 3 and σ₃ = 5, see Fig. 1. According to Expression 2 we have $\sum_{s = 1}^{| m |} σ_{s} = 2 + 3 + 5 = 10$ , then the portions of synthetic objects from the total amount of oversampling, that will be created around each object are p₁ = 2/10 = 0.2, p₂ = 3/10 = 0.3 and p₃ = 5/10 = 0.5. This means, since p₃ = 0.5, that 50% from the total amount of oversampling will be created around O₃, 30% around O₂ and 20% around O₁.

Fig.1

Three objects of the minority class (O₁, O₂ and O₃) with 3-nearest neighbors each one and their standard deviations of the distances (dashed circles).

Once determined the portion p_i of synthetic objects from the total amount of oversampling, that will be created around an object O_i of the minority class, these synthetic objects will be created between it and its k-nearest neighbors depending on their distance. The farther a nearest neighbor is, the more room for creating synthetic objects. Then, the portion $p_{i}^{j}$ , from p_i, of synthetic objects to be created between O_i and $O_{i}^{j}$ computed by dividing their distance, denoted as $d (O_{i}, O_{j}^{i})$ , by the sum of the distances between O_i and each one of its k-nearest neighbors; see Expression 3:

$p_{i}^{j} = d (O_{i}, O_{j}^{i}) / \sum_{r = 1}^{k} d (O_{i}, O_{j}^{r})$ (3) where d () denotes a distance function. In this way, more synthetic objects will be created between O_i and its farthest neighbors.

Considering the object O₃ in our previous example, let suppose the distances between O₃ and its neighbors $O_{3}^{1}$ , $O_{3}^{2}$ and $O_{3}^{3}$ are 0.1, 0.2 and 0.4 respectively, see Fig. 2. According to Expression 3, we have $\sum_{r = 1}^{k} d (O_{3}; O_{3}^{r}) = 0.1 + 0.2 + 0.4 = 0.7$ , then the portions $p_{i}^{j}$ , from p_i, of synthetic objects that will be created between O₃ and each one of its k-nearest neighbors are $p_{3}^{1} = 0.1 / 0.7 = 0.14$ , $p_{3}^{2} = 0.2 / 0.7 = 0.28$ and $p_{3}^{3} = 0.4 / 0.7 = 0.57$ . This means that 57%, from p₃, of the synthetics objects will be created between O₃ and its neighbor $O_{3}^{3}$ , since $p_{3}^{3} = 0.57$ , while the 28% of synthetic objects from p₃ will be created between O₃ and $O_{3}^{2}$ and the remaining 14% of synthetic objects from p₃ will be created between O₃ and $O_{3}^{1}$ .

Fig.2

An object (O₃) of the minority class with 3-nearest neighbors ( $O_{3}^{j} j = 1, . . ., 3$ ) and their distances.

Summarizing, up to this point, given the amount of oversampling (n), we have explained how to compute the portion p_i, from n, of synthetic objects to create around each object O_i in the minority class; i = 1, . . . , |m|. Also, we have explained how to compute the portion $p_{i}^{j}$ , from p_i, of synthetic objects to create between the object O_i and each one of its neighbors $O_{i}^{j}$ ; j = 1, . . . , k. Now, SMOTE-D computes the number of objects $s_{i}^{j}$ to create between an O_i and each one of its k-nearest neighbors ( $O_{i}^{j}$ ) as in Expression 4:

$s_{i}^{j} = round (n * p_{i} * p_{i}^{j})$ (4)

In order to avoid the random creation of synthetic objects, which could produce undesired situations such as synthetic objects too close (or too far) from O_i and/or $O_{i}^{j}$ ; or the creation of stacked objects in the same region. SMOTE-D creates the synthetic objects between O_i and $O_{i}^{j}$ in a uniform way. For doing this, the difference between the feature vector of O_i and its nearest neighbor $O_{i}^{j}$ is computed and then this difference is divided by the number of synthetic objects to create $s_{i}^{j}$ plus one, as in Expression 5. Finally, SMOTE-D adds this divided difference to O_i as many times as $s_{i}^{j}$ indicates; obtaining after each addition a new synthetic object.

${diff}_{i}^{j} = (O_{i}^{j} - O_{i}) / (s_{i}^{j} + 1)$ (5)

In order to illustrate the uniform creation of synthetic objects, let (1.0, 0.5, - 1.0) be the feature vector of O_i and let (5.0, 1.0, 1.0) be the feature vector of its nearest neighbor $O_{i}^{j}$ . Also suppose, for our example, that n = 10, p_i = 0.5 and $p_{i}^{j} = 0.57$ . Evaluating Expression 4 $s_{i}^{j} = round (10 * 0.5 * 0.57) = 3$ , and according to Expression 5 we get the vector (1.0, 0.12, 0.5). Then, three synthetics objects are created by adding $s_{i}^{j} = 3$ times the divided difference vector to O_i, as it is shown in Table 1. In Fig.3, the three synthetic objects created by SMOTE-D are shown.

Table 1

Three synthetic objects created for O_i = (1.0, 0.5, - 1.0) and $O_{i}^{j} = (5.0, 1.0, 1.0)$ .

	Attributes
O _i	1.0	0.5	-1.0
$O_{i}^{j}$	5.0	1.0	1.0
${diff}_{i}^{j}$	1.0	0.125	0.5
Synthetic	2.0	0.625	-0.5
objects	3.0	0.750	0.0
	4.0	0.875	0.5

Fig.3

3 synthetic objects uniformly created between the object O₃ of the minority class and one of its k-nearest neighbors $O_{3}^{3}$ .

Algorithm 3.1: SMOTE-D algorithm for deterministically and uniformly creating synthetics objects.

1. em Compute the amount of oversampling n for the minority class (using Expression 1).

2. Compute the portions p_i from n for each object O_i in the minority class (using Expression 2).

3. Compute the portions $p_{i}^{j}$ from p_i for each one of the k-nearest neighbors $O_{i}^{j}$ of each object O_i in the minority class (using Expression 3).

4. Compute the number $s_{i}^{j}$ of synthetic objects to be created between each object O_i and each one of its neighbors $O_{i}^{j}$ (using Expression 4).

5. ForEach object O_i in the minority class

6. ForEach one of the k-nearest neighbors $O_{i}^{j}$ of O_i

7. Divide the attribute difference between the object and the nearest neighbor by the number of synthetic objects to be created between them plus one (using Expression 5).

8. Generate new synthetic objects by adding ${diff}_{i}^{j}$ to O_i as many times as $s_{i}^{j}$ indicates.

9. Add the new synthetic objects to the synthetic subset.

10. End ForEach

11. End ForEach

12. Add the synthetic subset to the training set.

3.2 Deterministic SMOTE-D-based methods

In this subsection, we introduce deterministic versions of the most recent and referenced SMOTE-based methods by using SMOTE-D.

In Borderline SMOTE-D1 border objects of the minority class are labeled in the same way as in Borderline SMOTE1 [12]. After, SMOTE-D is applied over the border objects in order to deterministically create synthetic objects between each one of them and its k-nearest neighbors in the minority class.

In Borderline SMOTE-D2 border objects are labeled in the same way as in Borderline SMOTE1 [12]. Then, in the same way as Borderline SMOTE2 does, Borderline SMOTE-D2 will work over each border object, its k-nearest neighbors, and its nearest neighbor in the majority class, but applying SMOTE-D. For this, first the number of synthetic objects that SMOTE-D will create around each border object and between it and each one of its k-nearest neighbors is computed, but for computing this number, the distance between a border object and its nearest neighbor in the majority class is multiplied by 0.5, in order to create more synthetic objects around neighbors from the minority class than around neighbors from the majority class. Additionally, when SMOTE-D creates a synthetic object between a border object and its nearest neighbor in the majority class, the difference of attributes is multiplied by 0.5; this preserves the idea of Borderline SMOTE2 about creating the synthetic objects closer to the border object than to the nearest neighbor in the majority class.

Safe Level SMOTE-D computes for each object in the minority class a safe level value (sl), as it is done in Safe Level SMOTE [3]. Then, the number of synthetic objects to be created around each object of the minority class and between it and each one of its k-nearest neighbors is computed in the same way as in SMOTE-D. But, if the safe level of an object of the minority class is 0, Safe Level SMOTE-D will not create synthetic objects between this object and its k-nearest neighbors. When creating synthetic objects, if the safe level of a neighbor is 0, the object of the minority class is duplicated as many times as the computed number of synthetic objects to be created between them. Otherwise, Safe Level SMOTE-D computes a safe level ratio (slr) between the object of the minority class and its neighbors; as it is done in Safe Level SMOTE. Then, before creating the synthetic objects (in a uniform way, as in SMOTE-D) between an object of the minority class and one of its k-nearest neighbors, the attribute difference is multiplied by a different value depending on the slr value as follows: if slr = 1, the attribute difference is multiplied by 1, this means, this difference is used as in SMOTE-D; if slr > 1, the attribute difference is divided by the safe level ratio, in order to create synthetic objects closer to the object of the minority class than to its nearest neighbor, as in Safe Level SMOTE; and if slr < 1, the attribute difference is multiplied by slr, besides, the synthetic objects will be created between the object of the minority class plus the attribute difference (multiplied by the safe level ratio) and its nearest neighbor, this allows creating synthetic objects closer to the nearest neighbor than to the object, just as in Safe Level SMOTE.

In M-SMOTE-D the objects of the minority class are labeled as latent noise, border or security as in M-SMOTE [14]. Then, M-SMOTE-D computes the number of synthetic objects to be created around each object of the minority class in the same way as SMOTE-D. Then, following the idea of M-SMOTE for latent noise objects, M-SMOTE-D does not create synthetic objects; for security objects, M-SMOTE-D creates synthetic objects as SMOTE-D; and for each border object, M-SMOTE-D only creates synthetic objects between the border object and its nearest neighbor in the minority class, in a uniformly way as SMOTE-D.

SMOTE-D Out computes the number of synthetic objects to be created around each object of the minority class and each one of its k-nearest neighbors as in SMOTE-D. Then, SMOTE-D Out creates a synthetic object between each object of the minority class and its nearest neighbor of the majority class (always the nearest to avoid the randomness). This synthetic object is created by multiplying the attribute difference between them by -0.3, this allows creating this synthetic object close to the object of the minority class and in the opposite side from its nearest neighbor in the majority class. Then, the created synthetic objects are used, instead of the objects of the minority class for creating the synthetic objects for the minority class, but multiplying the attribute difference by 0.5. This will create the synthetic object closer to the first created synthetic objects than to the nearest neighbors.

SMOTE-D Cosine works in the same way as SMOTE-D, but for computing the k-nearest neighbors, SMOTE-D Cosine uses as distance the sum of the Euclidean and Cosine distances. This distance is also used for computing the number of synthetic objects to be created around each object of the minority class as well as the number of synthetic objects to be created between the object of the minority class and each one of its k-nearest neighbors.

Selected SMOTE-D creates synthetic objects as in SMOTE-D, but using only the selected attributes, as it is done in Selected SMOTE [15]; leaving the remaining attributes as they are in the object of the minority class.

It is important to highlight that, in all SMOTE-D-based methods, all the random steps are eliminated; resulting in deterministic versions which produce the same outcome each time they are applied.

4 Experimental results

In this section, we compare the results of SMOTE-D and the SMOTE-D-based methods proposed in this paper against the results of SMOTE and SMOTE-based methods. We perform the assessment through several supervised classifiers. The experimental setup is shown in subsection 4.1. Subsection 4.2 shows a comparison between SMOTE-D and SMOTE, and subsection 4.3 shows a comparison of the proposed SMOTE-D-based methods against SMOTE-based methods.

4.1 Experimental Setup

In our experiments we used the datasets shown in Table 2 taken from the KEEL repository [1], all these datasets contain two-class problems and have a 5-fold cross validation partition. Table 2 shows the number of attributes (# attributes), the number of objects (# objects) and the IR of each dataset. The IR is computed as the ratio between the size of the majority class and the size of the minority class. In Table 2, the datasets appear in ascending order regarding their IR. In our comparisons, we use the following supervised classifiers: Decision Trees (DT), Support Vector Machines (SVM) and K-Nearest Neighbors (KNN) with K = 5, which have been commonly used in the literature. For assessing the results of oversampling methods we employ Area Under ROC Curve (AUC) and F-measure (F-M). For validating our results we apply a t-test with a confidence level of 95%.

Table 2
Datasets from the KEEL repository used in our experiments

Dataset # Attributes # Objects IR Dataset # Attributes # Objects IR

glass1 9 214 1.82 glass-0-4_vs_5 9 92 9.22

ecoli-0_vs_1 7 220 1.86 ecoli-0-3-4-6_vs_5 7 205 9.25

wisconsin 9 683 1.86 ecoli-0-3-4-7_vs_5-6 7 257 9.28

pima 8 768 1.87 yeast-0-5-6-7-9_vs_4 8 528 9.35

iris0 4 150 2.00 vowel0 13 988 9.98

glass0 9 214 2.06 ecoli-0-6-7_vs_5 6 220 10.00

yeast1 8 1484 2.46 glass-0-1-6_vs_2 9 192 10.29

haberman 3 306 2.78 ecoli-0-1-4-7_vs_2-3-5-6 7 336 10.59

vehicle2 18 846 2.88 led7digit-0-2-4-5-6-7-8-9_vs_1 7 443 10.97

vehicle1 18 846 2.90 glass-0-6_vs_5 9 108 11.00

vehicle3 18 846 2.99 ecoli-0-1_vs_5 6 240 11.00

glass-0-1-2-3_vs_4-5-6 9 214 3.20 glass-0-1-4-6_vs_2 9 205 11.06

vehicle0 18 846 3.25 glass2 9 214 11.59

ecoli1 7 336 3.36 ecoli-0-1-4-7_vs_5-6 6 332 12.28

new-thyroid1 5 215 5.14 cleveland-0_vs_4 13 177 12.62

new-thyroid2 5 215 5.14 ecoli-0-1-4-6_vs_5 6 280 13.00

ecoli2 7 336 5.46 shuttle-c0-vs-c4 9 1829 13.87

segment0 19 2308 6.02 yeast-1_vs_7 7 459 14.30

glass6 9 214 6.38 glass4 9 214 15.47

yeast3 8 1484 8.10 ecoli4 7 336 15.80

ecoli3 7 336 8.60 page-blocks-1-3_vs_4 10 472 15.86

page-blocks0 10 5472 8.79 abalone9-18 8 731 16.40

ecoli-0-3-4_vs_5 7 200 9.00 glass-0-1-6_vs_5 9 184 19.44

yeast-2_vs_4 8 514 9.08 shuttle-c2-vs-c4 9 129 20.50

ecoli-0-6-7_vs_3-5 7 222 9.09 yeast-1-4-5-8_vs_7 8 693 22.10

ecoli-0-2-3-4_vs_5 7 202 9.10 glass5 9 214 22.78

glass-0-1-5_vs_2 9 172 9.12 yeast-2_vs_8 8 482 23.10

yeast-0-3-5-9_vs_7-8 8 506 9.12 yeast4 8 1484 28.10

yeast-0-2-5-7-9_vs_3-6-8 8 1004 9.14 yeast-1-2-8-9_vs_7 8 947 30.57

yeast-0-2-5-6_vs_3-7-8-9 8 1004 9.14 yeast5 8 1484 32.73

ecoli-0-4-6_vs_5 6 203 9.15 ecoli-0-1-3-7_vs_2-6 7 281 39.14

ecoli-0-1_vs_2-3-5 7 244 9.17 yeast6 8 1484 41.40

ecoli-0-2-6-7_vs_3-5 7 224 9.18 abalone19 8 4174 129.44

Dataset	# Attributes	# Objects	IR	Dataset	# Attributes	# Objects	IR
glass1	9	214	1.82	glass-0-4_vs_5	9	92	9.22
ecoli-0_vs_1	7	220	1.86	ecoli-0-3-4-6_vs_5	7	205	9.25
wisconsin	9	683	1.86	ecoli-0-3-4-7_vs_5-6	7	257	9.28
pima	8	768	1.87	yeast-0-5-6-7-9_vs_4	8	528	9.35
iris0	4	150	2.00	vowel0	13	988	9.98
glass0	9	214	2.06	ecoli-0-6-7_vs_5	6	220	10.00
yeast1	8	1484	2.46	glass-0-1-6_vs_2	9	192	10.29
haberman	3	306	2.78	ecoli-0-1-4-7_vs_2-3-5-6	7	336	10.59
vehicle2	18	846	2.88	led7digit-0-2-4-5-6-7-8-9_vs_1	7	443	10.97
vehicle1	18	846	2.90	glass-0-6_vs_5	9	108	11.00
vehicle3	18	846	2.99	ecoli-0-1_vs_5	6	240	11.00
glass-0-1-2-3_vs_4-5-6	9	214	3.20	glass-0-1-4-6_vs_2	9	205	11.06
vehicle0	18	846	3.25	glass2	9	214	11.59
ecoli1	7	336	3.36	ecoli-0-1-4-7_vs_5-6	6	332	12.28
new-thyroid1	5	215	5.14	cleveland-0_vs_4	13	177	12.62
new-thyroid2	5	215	5.14	ecoli-0-1-4-6_vs_5	6	280	13.00
ecoli2	7	336	5.46	shuttle-c0-vs-c4	9	1829	13.87
segment0	19	2308	6.02	yeast-1_vs_7	7	459	14.30
glass6	9	214	6.38	glass4	9	214	15.47
yeast3	8	1484	8.10	ecoli4	7	336	15.80
ecoli3	7	336	8.60	page-blocks-1-3_vs_4	10	472	15.86
page-blocks0	10	5472	8.79	abalone9-18	8	731	16.40
ecoli-0-3-4_vs_5	7	200	9.00	glass-0-1-6_vs_5	9	184	19.44
yeast-2_vs_4	8	514	9.08	shuttle-c2-vs-c4	9	129	20.50
ecoli-0-6-7_vs_3-5	7	222	9.09	yeast-1-4-5-8_vs_7	8	693	22.10
ecoli-0-2-3-4_vs_5	7	202	9.10	glass5	9	214	22.78
glass-0-1-5_vs_2	9	172	9.12	yeast-2_vs_8	8	482	23.10
yeast-0-3-5-9_vs_7-8	8	506	9.12	yeast4	8	1484	28.10
yeast-0-2-5-7-9_vs_3-6-8	8	1004	9.14	yeast-1-2-8-9_vs_7	8	947	30.57
yeast-0-2-5-6_vs_3-7-8-9	8	1004	9.14	yeast5	8	1484	32.73
ecoli-0-4-6_vs_5	6	203	9.15	ecoli-0-1-3-7_vs_2-6	7	281	39.14
ecoli-0-1_vs_2-3-5	7	244	9.17	yeast6	8	1484	41.40
ecoli-0-2-6-7_vs_3-5	7	224	9.18	abalone19	8	4174	129.44

All of the oversampling methods described in this paper were implemented in MATLAB, in a PC with an AMD FX-8320 Eight-Core processor running at 3.5 GHz, 8GB of RAM and 64-bit Windows7.

For Selected SMOTE-D and Selected SMOTE we use Stepwise Regression [7] as feature selection method. This is a commonly used method for dimensionality reduction.

4.2 SMOTE-D vs SMOTE results

In this subsection, we show the results of comparing of SMOTE-D against SMOTE in Table 3.

Table 3
Results of SMOTE-D vs SMOTE

Distance k Classifier AUC F-M

SMOTE-D SMOTE SMOTE-D SMOTE

Euc DT 0.820 0.814±0.027 0.648 0.642±0.052

3 SVM 0.837 0.844±0.007 0.633 0.617±0.010

KNN 0.822 0.822±0.007 0.651 0.654±0.014

DT 0.822 0.820±0.034 0.644 0.644±0.056

5 SVM 0.832 0.844±0.007 0.616 0.617±0.016

KNN 0.828 0.827±0.009 0.649 0.654±0.017

DT 0.820 0.821±0.036 0.640 0.643±0.061

7 SVM 0.833 0.844±0.007 0.613 0.618±0.024

KNN 0.828 0.829±0.010 0.645 0.652±0.020

HVDM DT 0.820 0.819±0.030 0.648 0.644±0.056

3 SVM 0.836 0.845±0.007 0.635 0.617±0.015

KNN 0.822 0.826±0.007 0.654 0.653±0.014

DT 0.817 0.821±0.031 0.633 0.643±0.057

5 SVM 0.835 0.845±0.009 0.614 0.619±0.019

KNN 0.828 0.831±0.008 0.654 0.655±0.017

DT 0.824 0.823±0.037 0.645 0.643±0.064

7 SVM 0.835 0.845±0.011 0.608 0.618±0.023

KNN 0.828 0.832±0.009 0.648 0.655±0.018

Distance	k	Classifier	AUC	F-M
Euc		DT	0.820	0.814±0.027	0.648	0.642±0.052
	3	SVM	0.837	0.844±0.007	0.633	0.617±0.010
		KNN	0.822	0.822±0.007	0.651	0.654±0.014
		DT	0.822	0.820±0.034	0.644	0.644±0.056
	5	SVM	0.832	0.844±0.007	0.616	0.617±0.016
		KNN	0.828	0.827±0.009	0.649	0.654±0.017
		DT	0.820	0.821±0.036	0.640	0.643±0.061
	7	SVM	0.833	0.844±0.007	0.613	0.618±0.024
		KNN	0.828	0.829±0.010	0.645	0.652±0.020
HVDM		DT	0.820	0.819±0.030	0.648	0.644±0.056
	3	SVM	0.836	0.845±0.007	0.635	0.617±0.015
		KNN	0.822	0.826±0.007	0.654	0.653±0.014
		DT	0.817	0.821±0.031	0.633	0.643±0.057
	5	SVM	0.835	0.845±0.009	0.614	0.619±0.019
		KNN	0.828	0.831±0.008	0.654	0.655±0.017
		DT	0.824	0.823±0.037	0.645	0.643±0.064
	7	SVM	0.835	0.845±0.011	0.608	0.618±0.023
		KNN	0.828	0.832±0.009	0.648	0.655±0.018

The first column (Distance) of table 3 shows the distance/metric used for finding the k-nearest neighbors, the second column (k) shows the number of k-nearest neighbors used for oversampling, the third column (Classifier) indicates the used supervised classifier, the following double columns (AUC and FM) show the results of SMOTE-D and SMOTE in terms of the metric on the top. To get the results under these columns, SMOTE-D and SMOTE were applied over each one of the 5-folds cross validation partitions of the 66 datasets shown in Table 2 and the average is reported. For SMOTE we report the average of 10 runs of SMOTE and the standard deviation. In this table we can see that both methods obtain similar classification results. Moreover, by applying a t-test with a confidence level of 95%, we did not find statistical significant differences in the results produced by both methods.

4.3 SMOTE-D-based methods vs SMOTE-based methods

In this subsection, in Tables 4 and 5, we show the results of the comparison of the proposed SMOTE-D-based methods against SMOTE-based methods using the Euclidean and HVD metrics respectively.

Table 4
Results of SMOTE-D-based methods vs SMOTE-based methods using the Euclidean distance

Method Classifier AUC F-M

SMOTE-D SMOTE SMOTE-D SMOTE

Cos DT 0.822 0.822±0.032 0.642 0.647±0.060

SVM 0.842 0.850±0.007 0.639 0.633±0.016

KNN 0.829 0.828±0.009 0.649 0.654±0.016

B1 DT 0.806 0.811±0.029 0.635 0.642±0.045

SVM 0.825 0.839±0.007 0.618 0.621±0.015

KNN 0.819 0.824±0.007 0.657 0.657±0.013

B2 DT 0.804 0.809±0.029 0.632 0.633±0.054

SVM 0.825 0.839±0.010 0.603 0.620±0.021

KNN 0.821 0.825±0.009 0.656 0.655±0.014

SL DT 0.816 0.804±0.018 0.648 0.648±0.034

SVM 0.828 0.811±0.005 0.628 0.609±0.012

KNN 0.826 0.816±0.002 0.660 0.666±0.005

Out DT 0.826 0.818±0.029 0.649 0.646±0.047

SVM 0.838 0.851±0.006 0.613 0.634±0.012

KNN 0.831 0.824±0.013 0.657 0.656±0.022

M DT 0.812 0.812±0.024 0.645 0.641±0.044

SVM 0.835 0.849±0.012 0.643 0.639±0.018

KNN 0.821 0.822±0.004 0.661 0.662±0.007

Sel DT 0.819 0.815±0.026 0.642 0.643±0.047

SVM 0.835 0.843±0.008 0.620 0.614±0.017

KNN 0.826 0.829±0.013 0.645 0.658±0.025

Method	Classifier	AUC	F-M
Cos	DT	0.822	0.822±0.032	0.642	0.647±0.060
	SVM	0.842	0.850±0.007	0.639	0.633±0.016
	KNN	0.829	0.828±0.009	0.649	0.654±0.016
B1	DT	0.806	0.811±0.029	0.635	0.642±0.045
	SVM	0.825	0.839±0.007	0.618	0.621±0.015
	KNN	0.819	0.824±0.007	0.657	0.657±0.013
B2	DT	0.804	0.809±0.029	0.632	0.633±0.054
	SVM	0.825	0.839±0.010	0.603	0.620±0.021
	KNN	0.821	0.825±0.009	0.656	0.655±0.014
SL	DT	0.816	0.804±0.018	0.648	0.648±0.034
	SVM	0.828	0.811±0.005	0.628	0.609±0.012
	KNN	0.826	0.816±0.002	0.660	0.666±0.005
Out	DT	0.826	0.818±0.029	0.649	0.646±0.047
	SVM	0.838	0.851±0.006	0.613	0.634±0.012
	KNN	0.831	0.824±0.013	0.657	0.656±0.022
M	DT	0.812	0.812±0.024	0.645	0.641±0.044
	SVM	0.835	0.849±0.012	0.643	0.639±0.018
	KNN	0.821	0.822±0.004	0.661	0.662±0.007
Sel	DT	0.819	0.815±0.026	0.642	0.643±0.047
	SVM	0.835	0.843±0.008	0.620	0.614±0.017
	KNN	0.826	0.829±0.013	0.645	0.658±0.025

Table 5

Results of SMOTE-D-based methods vs SMOTE-based methods using HVDM

Method	Classifier	AUC		F-M
		SMOTE-D	SMOTE	SMOTE-D	SMOTE
Cos	DT	0.822	0.820 ± 0.033	0.642	0.643 ± 0.063
	SVM	0.842	0.851 ± 0.004	0.636	0.634 ± 0.011
	KNN	0.829	0.828 ± 0.009	0.649	0.653 ± 0.017
B1	DT	0.808	0.810 ± 0.031	0.637	0.634 ± 0.053
	SVM	0.830	0.839 ± 0.006	0.639	0.625 ± 0.016
	KNN	0.818	0.826 ± 0.007	0.658	0.659 ± 0.016
B2	DT	0.815	0.812 ± 0.035	0.641	0.625 ± 0.066
	SVM	0.833	0.838 ± 0.011	0.617	0.617 ± 0.019
	KNN	0.820	0.826 ± 0.008	0.651	0.650 ± 0.016
SL	DT	0.813	0.799 ± 0.022	0.644	0.637 ± 0.036
	SVM	0.817	0.772 ± 0.004	0.624	0.569 ± 0.014
	KNN	0.822	0.810 ± 0.003	0.667	0.659 ± 0.006
Out	DT	0.827	0.823 ± 0.035	0.641	0.640 ± 0.062
	SVM	0.839	0.848 ± 0.007	0.618	0.634 ± 0.011
	KNN	0.838	0.830 ± 0.013	0.660	0.657 ± 0.025
M	DT	0816	0.805 ± 0.029	0.648	0.638 ± 0.054
	SVM	0.822	0.836 ± 0.018	0.629	0.627 ± 0.032
	KNN	0.817	0.819 ± 0.003	0.660	0.658 ± 0.007
Sel	DT	0.814	0.819 ± 0.026	0.628	0.640 ± 0.047
	SVM	0.833	0.845 ± 0.009	0.612	0.616 ± 0.022
	KNN	0.833	0.833 ± 0.014	0.654	0.658 ± 0.025

In these tables, the first column shows the used oversampling methods: Cosine (COS), Borderline (B1), Borderline2 (B2), Safe Level (SL), Out (Out), M (M) and Selected (Sel). The second column shows the employed supervised classifier. After, two double columns show the results, in terms of AUC and F-M, of the SMOTE-D-based methods and the SMOTE-based methods, respectively. To get the results under these columns, the respective oversampling method was applied over each one of the 5-folds cross validation partitions of the 66 datasets shown in Table 2, and the average is reported. In the column SMOTE, we report the average of 10 runs of the respective SMOTE-based method and the standard deviation.

After applying a t-test with a confidence level of 95%, over the results shown in Tables 4 and 5, we did not find statistical significant differences, which means that both kinds of methods produce similar results.

Additionally, we measured the time spent by SMOTE and SMOTE-D when they are applied once over the 66 datasets. After applying a t-test with a confidence level of 95%, we did not find a statistically significant difference between their runtimes. However, since SMOTE produces a different outcome each time it is applied, SMOTE must be applied several times to select from all the results the best oversampled dataset. This selection involves training a classifier each time SMOTE is applied. In practice, the time required for selecting the best oversampled partition highly increases the total time required for applying SMOTE.

5 Conclusions

This paper introduces SMOTE-D, a deterministic version of SMOTE, as well as some deterministic SMOTE-D-based oversampling methods. The comparison of SMOTE-D vs SMOTE and SMOTE-D-based methods vs SMOTE-based methods, in terms of F-Measure and AUC using different classifiers show that SMOTE-D and SMOTE-D-based methods obtain as good results as SMOTE and SMOTE-based methods, but working in a deterministic way, i.e. producing the same good result every time they are applied. Therefore, SMOTE-D and SMOTE-D-based methods do not need to be applied several times to obtain a good result. This is important from a practical point of view since our proposals save time by avoiding multiple applications of them, unlike SMOTE-based methods do, and they provide the same result each time they are executed.

In this paper, we have modified some SMOTE-based methods for getting deterministic versions of these oversampling methods by including the SMOTE-D approach. As future work, we will develop deterministic oversampling methods specifically designed for taking advantage of the SMOTE-D characteristics. Additionally, we are going to extend the proposed method for working on mixed data.

Footnotes

Acknowledgments

This work was partly supported by National Council of Science and Technology of Mexico under the scholarship grant 627301.

References

Alcalá

, Fernández

, Luengo

, Derrac

, García

, Sánchez

and Herrera

, Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework, Journal of Multiple-Valued Logic and Soft Computing17(2-3) (2010), 255–287.

Batista

G.E.A.P.A.

, Prati

R.C.

and Monard

M.C.

, A study of the behavior of several methods for balancing machine learning training data, ACM Sigkdd Explorations Newsletter6(1) (2004), 20–29.

Bunkhumpornpat

, Sinapiromsaran

and Lursinsap

, Safe-level-smote: Safe-level-synthetic minority over-sampling technique for handling the class imbalanced problem, In Advances in Knowledge Discovery and Data Mining, Springer, 2009, pp. 475–482.

Chawla

N.V.

, Bowyer

K.W.

, Hall

L.O.

and Kegelmeyer

W.P.

, Smote: Synthetic minority oversampling technique, Journal of Artificial Intelligence Research (2002), 321–357.

Deepa

and Punithavalli

, An e-smote technique for feature selection in high-dimensional imbalanced dataset, In Electronics Computer Technology (ICECT), 2011 3rd International Conference on, volume 2, 2011, pp. 322–324. IEEE.

Dong

and Wang

, A new over-sampling approach: Random-smote for learning from imbalanced data sets, In Knowledge Science, Engineering and Management, Springer, 2011, pp. 343–352.

Draper

N.R.

, Applied regression analysis bibliography update 1994-97, Communications in Statistics-Theory and Methods27(10) (1998), 2581–2623.

Ducange

, Lazzerini

and Marcelloni

, Multi-objective genetic fuzzy classifiers for imbalanced and cost-sensitive datasets, Soft Computing14(7) (2010), 713–728.

Fernández

, del Río

, Chawla

N.V.

and Herrera

, An insight into imbalanced big data classification: Outcomes and challenges, Complex & Intelligent Systems (2017), 1–16.

10.

Fernández

, Garcia

, Herrera

and Chawla

N.V.

, Smote for learning from imbalanced data: Progress and challenges, marking the 15-year anniversary, Journal of Artificial Intelligence Research61 (2018), 863–905.

11.

Guerrero-Enamorado

and Ceballos-Gastell

, An experimental study of evolutionary product-unit neural network algorithm, Computación y Sistemas20(2) (2016).

12.

Han

, Wang

W.-Y.

and Mao

B.-H.

, Borderlinesmote: A new over-sampling method in imbalanced data sets learning, In Advances in Intelligent Computing, 2005, pp. 878–887. Springer.

13.

Hart

, The condensed nearest neighbor rule (corresp.), IEEE Transactions on Information Theory14(3) (1968), 515–516.

14.

, Liang

, Ma

and He

, Msmote: Improving classification performance when training data is imbalanced, In 2009 Second International Workshop on Computer Science and Engineering, 2009, pp. 13–17. IEEE.

15.

Koto

, Smote-out, smote-cosine, and selected-smote: An enhancement strategy to handle imbalance in data level, In Advanced Computer Science and Information Systems (ICACSIS), 2014 International Conference on, 2014, pp. 280–284. IEEE.

16.

Kubat

, Matwin

, et al., Addressing the curse of imbalanced training sets: One-sided selection, In ICML, volume 97, Nashville, USA, 1997, pp. 179–186.

17.

Laurikkala

, Improving identification of difficult small classes by balancing class distribution, Springer, 2001.

18.

Lin

W.-J.

and Chen

J.J.

, Class-imbalanced classifiers for high-dimensional data, Briefings in Bioinformatics14(1) (2013), 13–26.

19.

Liu

X.-Y.

and Zhou

Z.-H.

, The influence of class imbalance on cost-sensitive learning: An empirical study, In Sixth International Conference on Data Mining (ICDM’06), 2006, pp. 970–974. IEEE.

20.

López

, Fernández

, García

, Palade

and Herrera

, An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics, Information Sciences250 (2013), 113–141.

21.

Luengo

, Fernández

, García

and Herrera

, Addressing data complexity for imbalanced data sets: Analysis of smote-based oversampling and evolutionary undersampling, Soft Computing15(10) (2011), 1909–1936.

22.

Lunardon

, Menardi

and Torelli

, Rose: A package for binary imbalanced learning, A Peer-Reviewed, Open-Access Publication of the R Foundation for Statistical Computing, 2014, p. 79.

23.

Maciejewski

and Stefanowski

, Local neighbourhood extension of smote for mining imbalanced data, In Computational Intelligence and Data Mining (CIDM), 2011 IEEE Symposium on, 2011, pp. 104–111. IEEE.

24.

Patil

S.S.

and Sonavane

S.P.

, Enhanced over_sampling techniques for imbalanced big data set classification, In Data Science and Big Data: An Environment of Computational Intelligence, Springer, 2017, pp. 49–81.

25.

Ramentol

, Caballero

, Bello

and Herrera

, Smote-rsb*: A hybrid preprocessing approach based on oversampling and undersampling for high imbalanced data-sets using smote and rough sets theory, Knowledge and Information Systems33(2) (2012), 245–265.

26.

Rastogi

A.K.

, Narang

and Siddiqui

Z.A.

, Imbalanced big data classification: A distributed implementation of smote, In Proceedings of the Workshop Program of the 19th International Conference on Distributed Computing and Networking, ACM, 2018, p. 14.

27.

Sáez

J.A.

, Luengo

, Stefanowski

and Herrera

, Smote–ipf: Addressing the noisy and borderline examples problem in imbalanced classification by a re-sampling method with filtering, Information Sciences291 (2015), 184–203.

28.

Sharifirad

, Nazari

and Ghatee

, Modified smote using mutual information and different sorts of entropies, arXiv preprint arXiv:1803.11002 (2018).

29.

Stefanowski

and Wilk

, Selective pre-processing of imbalanced data for improving classification performance, In International Conference on Data Warehousing and Knowledge Discovery, Springer, 2008, pp. 283–292.

30.

Sun

, Kamel

M.S.

, Wong

A.K.C.

and Wang

, Cost-sensitive boosting for classification of imbalanced data, Pattern Recognition40(12) (2007), 3358–3378.

31.

Tomek

, Two modifications of cnn, IEEE Trans Syst Man Cybern6 (1976), 769–772.

32.

Torres

F.R.

, Carrasco-Ochoa

J.A.

and Martínez-Trinidad

J.F.

, Smote-d a deterministic version of smote, In Springer, pp, Mexican Conference on Pattern Recognition (2016), 177–188.

33.

Verbiest

, Ramentol

, Cornelis

and Herrera

F.C.

, Improving smote with fuzzy rough prototype selection to detect noise in imbalanced classification data, In Advances in Artificial Intelligence–IBERAMIA, Springer, 2012, pp. 169–178.

34.

Wang

, You

, Li

and Xu

, Extract minimum positive and maximum negative features for imbalanced binary classification, Pattern Recognition45(3) (2012), 1136–1145.

35.

Wilson

D.L.

, Asymptotic properties of nearest neighbor rules using edited data, Systems, Man and Cybernetics, IEEE Transactions on (3) (1972), 408–421.

36.

Zhang

, Li

, Kotagiri

, Wu

, Tari

and Cheriet

, Krnn: K rare-class nearest neighbour classification, Pattern Recognition62 (2017), 33–44.

37.

Zhou

Z.-H.

and Liu

X.-Y.

, Training cost-sensitive neural networks with methods addressing the class imbalance problem, IEEE Transactions on Knowledge and Data Engineering18(1) (2006), 63–77.

38.

Zong

, Huang

G.-B.

and Chen

, Weighted extreme learning machine for imbalance learning, Neurocomputing101 (2013), 229–242.