A straightforward feature selection method based on mean ratio for classifiers

Abstract

Feature Selection (FS) is currently a very important and prominent research area. The focus of FS is to identify and to remove irrelevant and redundant features from large data sets in order to reduced processing time and to improve the predictive ability of the algorithms. Thus, this work presents a straightforward and efficient FS method based on the mean ratio of the attributes (features) associated with each class. The proposed filtering method, here called MRFS (Mean Ratio Feature Selection), has only equations with low computational cost and with basic mathematical operations such as addition, division, and comparison. Initially, in the MRFS method, the average from the data sets associated with the different outputs is computed for each attribute. Then, the calculation of the ratio between the averages extracted from each attribute is performed. Finally, the attributes are ordered based on the mean ratio, from the smallest to the largest value. The attributes that have the lowest values are more relevant to the classification algorithms. The proposed method is evaluated and compared with three state-of-the-art methods in classification using four classifiers and ten data sets. Computational experiments and their comparisons against other feature selection methods show that MRFS is accurate and that it is a promising alternative in classification tasks.

Keywords

Mean ratio feature selection filter method classification computational intelligence

1. Introduction

Classification tasks require an algorithm to process different data types and learn how to identify each different class of output, label different patterns in the data, and correctly classify the largest number of data. However, some factors that can make hard the classification are found in the data sets themselves. Among these factors, one can cite: a large number and unrepresentative, highly correlated, and noisy attributes [1, 2]. Unrepresentative data and a large volume of attributes without relevant information can make the processing difficult, increase the response time, and decrease the classifiers’ predictive capability. Thus, one solution to minimize the probability of error in the classification and improve its predictive capacity is the use of Feature Selection (FS) methods. The FS methods aim to provide smaller subsets of attributes, but with greater representativeness, to make the algorithms faster and more assertive in data processing [3, 4, 5].

The FS methods use statistical analysis or heuristics to identify the most relevant attributes. The statistical methods are based on equations such as the chi-squared test, $T$ -test, data correlation analysis, sample distance, and information gain [6, 7, 8]. Statistic definitions [9, 10] state that some methods exhibit better results for non-parametric data sets, which are sample sets that do not present the characteristics of randomness and normal distribution [11, 12]. Other statistic methods produce better results for parametric data, for which the sample sets respect the normality requirements [13, 14]. In Heuristic FS methods, the goal is to minimize the execution time and to select a subset with good, but not necessarily the best, results. Heuristic methods are commonly based on Evolutionary Computing [15].

FS algorithms are usually classified into three main classes: filter, wrapper, and embedded approaches. The filter method pre-processes the data, then identifies the most relevant attributes and rank them from the best to the worst. The wrapper method selects and tests different subsets of attributes with a learning algorithm, which chooses the best subset of them all. In the embedded method, the selection process is an internal and natural part of the classifier. The selection of subsets of attributes is carried out simultaneously with the training step classification process. Each method is developed specifically for its classifying algorithm. It should be noted that the classifier and the attribute selection process cannot be separated.

As mentioned, filter methods are used to obtain and rank the most relevant attributes from the data set. However, they do not identify how many attributes should be used. Consequently, the number of most relevant attributes processed in an algorithm must be selected manually or selected with the help of another method. The filter method can be found in several works, and in [16] the interaction dominance-based feature selection (IDFS) is proposed. In IDFS, the mutual information and three-way interaction information techniques are used to rank the attributes. The mutual information is used to measure the relevance of the attributes associated with the output classes, while the three-way interaction information is applied to calculate the attributes redundancy. Only then does the method use a greedy search to rank the attributes. In [17] the multi-filter feature selection approach (MFFS) is proposed, where initially the Logistic Regression, Support Vector Machine (SVM) and Random Forest (RF) algorithms are used to filter the most relevant attributes. Then, linear correlation (MFFS-lr) and Information Gain (MFFS-IG) are applied to the remaining attributes to measure the level of association and, therefore, to select the set of the best input variables. The stratified feature ranking (SFR) is a method proposed in [18]. To allow the ranking of the attributes, initially a method called subspace feature clustering (SFC) is performed, in which the attributes are grouped by their importance according to each class of output. After that, the SRF analyzes the groups of the different attributes and separately classifies them according to the weight produced by the SFC. In [19] the unsupervised feature selection method using feature clustering (EUFSFC) is proposed. This method performs an analysis of the redundancies and the significance of the attributes. The redundancy is calculated using the generated groups, and the significance is measured by grouping the attributes. In [20] another FS method is presented, in which the ranking is accomplished by a fuzzy model, where all the processing is started generating the fuzzy sets and membership function for each attribute; the process is performed using the clustering algorithm Fuzzy C-Means. Subsequently, in a second step, the information value of each attribute in the set is measured, selecting the most relevant ones.

Several state-of-the-art investigations present algorithms derived from different methods and with great complexity in their development and equations. A straightforward filter-type feature selection (FS) method applied for classification problems is proposed in the current paper. Known as MRFS (Mean Ratio for Feature Selection), the proposed method is based on the mean ratio of the attributes of different classes. The choice of the mean ratio is due to its simplicity, transparency, and low computational cost, even for a large number of attributes. In the proposed method, the averages of the data associated with the different classes of outputs are calculated for each attribute. Afterwards, the ratio of the averages is computed. Finally, the attributes are ordered by the results of the ratio, from the lowest to the highest value. The ratio results allow to verify which of the lowest values attributes have the most different averages. In the method, the attributes with different averages are more relevant for classification.

After this brief introduction, the rest of the paper is organized as follows. Section 2 introduces the proposed method. Section 3 shows the computational experiments and analyses the results obtained. Finally, Section 4 presents the final considerations.

2. Proposed approach

The Mean Ratio for Feature Selection uses the difference among sample averages associated with the different output classes to select the most relevant attributes (input variables). The proposed method aims to explore a more direct, efficient, and easily applied solution in environments that require rapid processing. The focus is on a feature selection method with low computational cost equations and basic mathematical operations such as addition, division, and comparison.

Before developing the MRFS method, experiments were performed with statistical tests such as analysis of variance (ANOVA) and Confidence Interval [9, 10] to prove that excluding statistically equal sets when associated with different output classes improve the accuracy of the classification algorithms. Figure 1 shows the steps of the statistical analysis performed.

In the first step, a descriptive analysis of the data is carried out. The goal is to extract information to define the data type (categorical or non-categorical) and to determine the best statistical method for its processing (ANOVA or Confidence Interval). The hypothesis test is applied to check for statistical equality (or not) among the different samples of an evaluated population, given a specific significance $\alpha$ . The significance is associated with the percentage of test acceptance, verifying whether there is an equality in the sample means examined; the hypotheses of equality can be confirmed in Eq. (1).

$\displaystyle\left\{\begin{array}[]{l}H_{0}:\mu_{0}-\mu_{1}=0\\ H_{1}:\mu_{0}-\mu_{1}\neq 0\\ \end{array}\right.$ (1)

in which $H_{0}$ assumes a risk significance so that the averages do not present equality between them, i.e., the result of this test has a chance of not being in an acceptance region and of being in the critical region of the test. The alternative hypothesis, or $H_{1}$ , is the rejection or critical region of the equality hypothesis. $\mu_{0}$ and $\mu_{1}$ are the evaluated averages. The experiments were based on the null hypothesis $H_{0}:\mu_{0}-\mu_{1}=0$ , evaluating whether the different subsets had statistically equal averages. The attributes that showed subsets with statistically equal average shall be removed.

Figure 1.

Steps of the data analysis before the development of MRFS.

In the second step, categorical data are processed using the statistical confidence interval model, whereas non-categorical data are processed using ANOVA. The measure evaluated in the results of these experiments is the $p$ -value, which is a descriptive measure for statistical tests. The adopted level of the confidence value is 95%, which defines the acceptance region with a significance level $\alpha$ of 5% for equal averages. Thus, based on the $p$ -value it is possible to determine whether the sample sets of each attribute have a statistically different or equal mean of a population. The association with the output classes characterizes each sample set of a population.

In the third step, the attributes that presented statistical equality are deleted, reducing the number of attributes for the classifier. In the fourth step – the test step –, the performances of the classification algorithms are compared using the data set with all inputs and the same data set without attributes that showed statistical equality (deleted in the third step). The results suggest that attributes with statistically similar sample mean decrease the predictive capacity of the classifiers.

Note that, independently of the value of the significance level $\alpha$ , it is observed that the higher the $p$ -value, the more it belongs to the equality hypothesis for the sample average associated with the desired output classes. Thereby, it is verified that the ratio of the means is similar to the p-value, regardless of the data distribution and its variance. The ratio of the means follows a pattern “the higher the value of the ratio, the more equal the means are”. Another concept adopted that motivated the use of the ratio of the means, ignoring measures such as variance and data distribution, is the central limit theorem (CLT) [9, 10]. The CLT states that large sample sets tend to be equal to the population average, further supporting the use of the ratio of the means.

Based on the results, MRFS is developed to rank the attributes from the best to the worst, given the ratio of the averages of the sample sets associated with the different output classes. It uses only a central idea of the statistical methods, which are the sample means, and ignores other mathematical equations that would require further processing such as variance, standard deviation, residual means, Torricelli, Bhaskara, among others. The proposed MRFS is detailed in the next section.

2.1 Mean ratio for feature selection

The filter method proposed here is based on the sample averages of each population and aims to be straightforward, effective, and fast. In the proposal, each population is the attribute or input variable of a data set, and the sample averages are the sets associated with the desired output classes. Hence, given a data set with two output classes and four input attributes, each one of the attributes is split into two different sample averages according to the output class. This statement can be represented by Eq. (2):

$\displaystyle S_{ij}=\sum_{j=1}^{n}(\bar{X_{ji}}\in k_{j})$ (2)

in which $k$ are the classes indexed by $j$ , $n$ is the total of samples from the training set, and $\bar{X}$ is the average of the $i$ attributes associated with the different $j$ classes.

Despite the well-defined sample levels and separate populations, it is still necessary to measure which sets of samples from the same population have more dispersed averages. To calculate the dispersion between the sample sets, the ratio between the two values is applied. The ratio is a measure that expresses a fraction or a percentage between two values; it is a mathematical formula used to compare two quantities. Thus, the proposed final method can be represented by Eq. (3):

$\displaystyle R_{i}=\sum_{j=2}^{k}\left(\min\left(\frac{{S}_{ij}}{{S}_{i(j-1)}% },\frac{{S}_{i(j-1)}}{{S}_{ij}}\right)\right)$ (3)

in which $R$ is a direct consequence of the mean value theorem [21, 22]. This theorem is given by the density of a continuous probability function for the mean intervals for each class $j$ of each attribute $i$ , based on the intervals between the minimum value of the ratio of the mean $j$ to $j-1$ or $j-1$ to $j$ .

MRFS stands out for its simple equations, which can be associated with calculating the ratio between two values, in which sample averages can be easily computed from the sum of each new sample presented to its proper set, therefore eliminating the recursive calculations of past samples to recalculate information values when a new sample is presented. The method also features an equation with a low computational cost, unlike other methods in the literature that require more complex calculations such as the variance or standard deviation assigned in formulas with more operations. A pseudocode of the MRFS filter method is presented in Algorithm 2.1.

[!h] Data set Input: X, Y Output: rankCols extract different classes in Y $k=$ unique classes in Y m $=$ number of attributes i $=$ 1 to m n $=$ number of samples j $=$ 1 to n $S_{ij}=$ Calculates Sample Average – Eq. (3) $R_{i}=$ Calculates Mean Ratio – Eq. (4) ranking outputs rankCols $=$ sort columns by $R_{i}$ Pseudocode of the MRFS filter method for Feature Selection.

3. Computational experiments and results

MRFS was evaluated using ten data sets and four classifiers. The results were compared with three alternative feature selection methods: Kruskal Wallis, ReliefF, and Chi-Square. The classifiers used in the experiments were: Artificial Neural Network (ANN) MultiLayer Perceptron, K-Nearest Neighbor (KNN), Naive Bayes (NB), and Support Vector Machine (SVM). The algorithms were developed in Python, using the sklearn library [23].

First of all, it is necessary to find the best setting for the classifiers. For each data set, different settings of the classifiers with all attributes were tested. The setting that obtained the best performance was used in the experiments to compare the FS methods. The structure and parameters tested for the classifiers are as follows:

•
ANN: hidden layers $=$ 2; neurons per hidden layer $=$ 10; activation function $=$ hyperbolic tangent, linear, logistic sigmoid, rectified linear; learning algorithm $=$ quasi-newton, stochastic gradient descent, stochastic gradient-based optimizer.
•
KNN: distance $=$ Euclidean, Manhattan; number of neighbors $=$ 2 to 25 – step 1.
•
NB: Gaussian, Bernoulli, Categorical, Multinomial.
•
SVM: kernel $=$ linear; $C=$ 0.1 to 1 – step 0.1.

Table 1
Description of data sets

Name # Samples # Attributes # Classes

Heart Disease [24] 303 13 2

Cervical Cancer [24] 668 32 2

Mobile Price [25] 2000 20 4

COIL 2000 [25] 9821 85 5

Page Blocks [25] 5471 10 2

QSAR Biodegradation [25] 1054 41 2

South African Health [26] 461 09 2

Spambase [25] 4600 57 2

Glass Identification [25] 214 10 6

Wine [25] 178 13 3

Table 2
Best parameter with all inputs for the ANN

Data set Activation function Learning

Heart Disease Linear Quasi-Newton

Cervical Cancer Logistic Sigmoid S. Gradient Optimizer

Mobile Price Linear S. Gradient Optimizer

COIL 2000 Linear S. Gradient Optimizer

Page Blocks Hyperbolic Tangent S. Gradient Optimizer

QSAR Biodegradation Logistic Sigmoid Quasi-Newton

South African Health Rectified Linear S. Gradient Optimizer

Spambase Hyperbolic Tangent S. Gradient Optimizer

Glass Identification Rectified Linear Quasi-Newton

Wine Logistic Sigmoid Quasi-Newton

Table 3
Experimental results with ANN

Instance MRFS Kruskal-Wallis ReliefF Chi-Square

Acc/Attribute Acc/Attribute Acc/Attribute Acc/Attribute

H. Disease 94.39/09 89.77/07 87.46/05 80.53/07

C. Cancer 97.75/26 97.16/20 97.46/08 97.90/14

Mobile Price 68.25/16 62.40/16 70.50/16 72.90/16

COIL 94.18/68 94.07/68 94.25/58 94.35/68

Pg. Blocks 97.08/04 95.91/06 95.83/06 95.59/08

QSAR 89.18/24 88.99/32 88.80/32 89.18/32

S.A. Health 73.97/04 73.10/08 69.63/07 70.07/03

Spambase 95.24/46 96.00/46 93.80/46 95.20/46

Glass Iden. 87.85/06 68.22/08 68.22/06 68.22/08

Wine 40.11/09 88.70/07 40.11/09 40.11/09

Average 88.76 87.18 87.22 86.97

The filter FS methods rank the variables by their relevance, but they do not identify which ones should be used by the algorithms. For the experiments, the number of attributes was reduced with 20% intervals in the number of all inputs for each data set used. The number of attributes that achieve the best performance is presented in the next sections. The ordered rank of the attributes by the FS methods and the number of attributes used in the experiments are shown in Appendix A. Attributes ranked by FS methods. To allow replicating the results, the classifiers and the data sets employed in the experiments are available at the URL: shorturl.at/hlD67.

The data sets used in the experiments are described in Table 1, which presents the name, the source, the number of samples, the number of attributes, and the number of output classes. The FS methods performance was evaluated by the Accuracy (Eq. (4)):

$\displaystyle\texttt{Acc}(y,\hat{y})=\frac{1}{n}\sum_{i=0}^{n-1}1(\hat{y}_{i}=% y_{i})$ (4)

in which $\hat{y}_{i}$ is the class predicted by the classifier of $i{-th}$ sample, $y_{i}$ is the correct class, and $n$ is the total number of samples.

The following sections presents the experiments and results with ANN (Section 3.1), KNN (Section 3.2), Naive Bayes (Section 3.3) and SVM (Section 3.4).

Table 4
Best parameter with all inputs for the KNN

Data set Distance Neighbors

Heart Disease Manhattan 2

Cervical Cancer Manhattan 3

Mobile Price Manhattan 3

COIL 2000 Euclidean 3

Page Blocks Manhattan 2

QSAR Biodegradation Manhattan 2

South African Health Manhattan 3

Spambase Manhattan 2

Glass Identification Euclidean 4

Wine Manhattan 3

Table 5
Experimental results with KNN

Instance MRFS Kruskal-Wallis ReliefF Chi-Square

Acc/Attribute Acc/Attribute Acc/Attribute Acc/Attribute

Heart Disease 89.11/07 89.11/09 88.45/05 87.79/05

Cervical Cancer 96.56/26 96.26/08 95.81/08 96.41/08

Mobile Price 96.70/16 96.55/16 96.60/16 96.60/12

COIL 2000 94.96/68 94.85/51 94.89/34 94.95/51

Pg. Blocks 97.42/08 97.61/06 97.50/06 97.42/08

QSAR 92.13/24 91.84/24 91.27/16 91.37/24

S.A. Health 82.00/07 84.38/06 80.48/07 82.21/04

Spambase 94.52/35 93.72/46 92.54/13 92.65/46

Glass Iden. 100.00/06 99.53/06 99.53/08 99.53/06

Wine 90.96/07 90.40/11 90.40/11 90.40/07

Average 93.44 93.43 92.75 92.93

Table 6
Best parameter with all inputs for the NB

Data set Methods

Heart Disease Gaussian

Cervical Cancer Bernoulli

Mobile Price Gaussian

COIL 2000 Bernoulli

Page Blocks Bernoulli

QSAR Biodegradation Bernoulli

South African Health Bernoulli

Spambase Bernoulli

Glass Identification Gaussian

Wine Gaussian

Figure 2.
Results between FS methods with ANN.

3.1 ANN experiments

Name	# Samples	# Attributes	# Classes
Heart Disease [24]	303	13	2
Cervical Cancer [24]	668	32	2
Mobile Price [25]	2000	20	4
COIL 2000 [25]	9821	85	5
Page Blocks [25]	5471	10	2
QSAR Biodegradation [25]	1054	41	2
South African Health [26]	461	09	2
Spambase [25]	4600	57	2
Glass Identification [25]	214	10	6
Wine [25]	178	13	3

Data set	Activation function	Learning
Heart Disease	Linear	Quasi-Newton
Cervical Cancer	Logistic Sigmoid	S. Gradient Optimizer
Mobile Price	Linear	S. Gradient Optimizer
COIL 2000	Linear	S. Gradient Optimizer
Page Blocks	Hyperbolic Tangent	S. Gradient Optimizer
QSAR Biodegradation	Logistic Sigmoid	Quasi-Newton
South African Health	Rectified Linear	S. Gradient Optimizer
Spambase	Hyperbolic Tangent	S. Gradient Optimizer
Glass Identification	Rectified Linear	Quasi-Newton
Wine	Logistic Sigmoid	Quasi-Newton

Instance	MRFS	Kruskal-Wallis	ReliefF	Chi-Square
H. Disease	94.39/09	89.77/07	87.46/05	80.53/07
C. Cancer	97.75/26	97.16/20	97.46/08	97.90/14
Mobile Price	68.25/16	62.40/16	70.50/16	72.90/16
COIL	94.18/68	94.07/68	94.25/58	94.35/68
Pg. Blocks	97.08/04	95.91/06	95.83/06	95.59/08
QSAR	89.18/24	88.99/32	88.80/32	89.18/32
S.A. Health	73.97/04	73.10/08	69.63/07	70.07/03
Spambase	95.24/46	96.00/46	93.80/46	95.20/46
Glass Iden.	87.85/06	68.22/08	68.22/06	68.22/08
Wine	40.11/09	88.70/07	40.11/09	40.11/09
Average	88.76	87.18	87.22	86.97

Data set	Distance	Neighbors
Heart Disease	Manhattan	2
Cervical Cancer	Manhattan	3
Mobile Price	Manhattan	3
COIL 2000	Euclidean	3
Page Blocks	Manhattan	2
QSAR Biodegradation	Manhattan	2
South African Health	Manhattan	3
Spambase	Manhattan	2
Glass Identification	Euclidean	4
Wine	Manhattan	3

Instance	MRFS	Kruskal-Wallis	ReliefF	Chi-Square
Heart Disease	89.11/07	89.11/09	88.45/05	87.79/05
Cervical Cancer	96.56/26	96.26/08	95.81/08	96.41/08
Mobile Price	96.70/16	96.55/16	96.60/16	96.60/12
COIL 2000	94.96/68	94.85/51	94.89/34	94.95/51
Pg. Blocks	97.42/08	97.61/06	97.50/06	97.42/08
QSAR	92.13/24	91.84/24	91.27/16	91.37/24
S.A. Health	82.00/07	84.38/06	80.48/07	82.21/04
Spambase	94.52/35	93.72/46	92.54/13	92.65/46
Glass Iden.	100.00/06	99.53/06	99.53/08	99.53/06
Wine	90.96/07	90.40/11	90.40/11	90.40/07
Average	93.44	93.43	92.75	92.93

Data set	Methods
Heart Disease	Gaussian
Cervical Cancer	Bernoulli
Mobile Price	Gaussian
COIL 2000	Bernoulli
Page Blocks	Bernoulli
QSAR Biodegradation	Bernoulli
South African Health	Bernoulli
Spambase	Bernoulli
Glass Identification	Gaussian
Wine	Gaussian

In this section, the FS methods are evaluated using Artificial Neural Network. Initially, the ANN algorithms were executed with all attributes to find the best setting for each data set. Then, the best setting was used to evaluate the FS methods. Table 2 shows the activation function and learning algorithm that obtained better performance for each data set.

Table 3 shows the accuracy and the number of attributes for the experiments with ANN, and Fig. 2 illustrates the results graphically. The MRFS method obtained the best accuracy in four of the ten experiments, Chi-Square in three and Kruskal-Wallis in two. In the experiment with the QSAR data set, MRFS achieved the same accuracy as Chi-Square, but with fewer attributes. In the average of the experimental results with the ANN, the best overall performance was obtained by MRFS, followed by ReliefF, Kruskal-Wallis, and Chi-Square.

3.2 KNN experiments

This section evaluates the FS methods with the KNN classifier. Initially, the best parameters of the KNN with all inputs were found. Table 4 shows the parameters that achieved the best result for each data set.

The performance of the FS methods is summarized in Table 5 and Fig. 3. In the experiments with Cervical Cancer, Mobile Price, COIL 2000, QSAR, Spambase, Glass Identification, and Wine, MRFS outperformed the other methods. Kruskal-Wallis achieved the best accuracy in the Page Blocks and South African Health experiments. MRFS and Kruskal-Wallis obtained the best accuracy in the Heart Disease experiment; however, MRFS with fewer attributes. MRFS achieved the best average performance, followed by Kruskal-Wallis, Chi-Square, and ReliefF.

Table 7
Results with NB

Instance	MRFS	Kruskal-Wallis	ReliefF	Chi-Square
	Acc/Attribute	Acc/Attribute	Acc/Attribute	Acc/Attribute
Heart Disease	83.83/11	81.52/11	83.50/11	82.84/11
Cervical Cancer	96.11/14	95.21/08	96.11/08	95.06/14
Mobile Price	81.95/16	80.40/16	81.75/16	82.25/16
COIL 2000	93.81/17	93.37/34	92.01/17	92.75/17
Page Blocks	89.78/03	89.78/03	89.78/03	89.78/03
QSAR	80.36/16	79.32/32	70.87/24	77.61/32
S.A. Health	72.67/04	72.23/04	70.72/08	71.37/08
Spambase	91.85/35	85.72/46	85.59/46	90.35/35
Glass Iden.	88.79/06	91.59/04	92.52/08	89.25/06
Wine	97.74/11	97.18/11	97.74/07	97.74/07
Average	87.69	86.63	86.06	86.90

Table 8

Best parameter with all inputs for the SVM

Data set	C parameter
Heart Disease	0.2
Cervical Cancer	0.5
Mobile Price	0.7
COIL 2000	0.4
Page Blocks	1.0
QSAR Biodegradation	0.8
South African Health	0.1
Spambase	0.6
Glass Identification	0.2
Wine	0.5

Figure 3.

Results between FS methods with KNN.

3.3 Naive Bayes experiments

The Naive Bayes classifier that achieved the best performance with the complete data sets is presented in Table 6. These classifiers were used to evaluate the performance of the FS methods.

Table 7 and Fig. 4 presents the experimental results. The MRFS method obtained the best performance in six experiments. In the experiment with the Cervical Cancer data set, the best results were obtained by MRFS and by ReliefF. With Page Blocks, all FS methods achieved the same accuracy. With the Wine data set, MRFS, ReliefF, and Chi-Square methods achieved the same accuracy. On average, the best performance is achieved by MRFS, followed by Chi-Square, Kruskal-Wallis, and ReliefF.

3.4 SVM experiments

In the experiments carried out with SVM, firstly, the best settings were established. Then, these settings were used to evaluate FS methods. Table 8 presents the best parameter setting for each data set.

Table 9
Results with SVM

Instance	MRFS	Kruskal-Wallis	ReliefF	Chi-Square
	Acc/Attribute	Acc/Attribute	Acc/Attribute	Acc/Attribute
Heart Disease	85.81/12	84.16/09	85.81/12	84.72/12
Cervical Cancer	96.11/14	95.81/26	96.11/20	96.11/26
Mobile Price	98.55/16	96.45/16	98.75/16	98.80/16
COIL 2000	94.03/17	94.03/17	94.03/17	94.03/17
Page Blocks	96.56/08	96.62/08	97.00/08	96.36/08
QSAR	87.19/32	88.43/32	87.76/32	88.43/32
S.A. Health	75.05/08	73.97/08	72.02/08	75.05/08
Spambase	93.50/46	91.72/46	92.13/46	93.13/46
Glass Iden.	100.00/06	100.00/06	100.00/04	100.00/ 04
Wine	98.31/07	97.74/11	97.74/11	98.31/11
Average	92.51	91.89	92.14	92.49

Figure 4.

Results between FS methods with NB.

Figure 5.

Results between FS methods with SVM.

Table 9 and Fig. 5 summarize the results with SVM. MRFS obtained the best performance in the experiment with the Spambase data set, while ReliefF was the best with Page Block. In the experiments with Heart Disease, Cervical Cancer, COIL 2000, South African Health, Glass Identification, and Wine, MRFS achieved higher accuracy together with other FS methods. In the average of the experiments, MRFS outperforms the alternative FS methods.

4. Final considerations

This paper introduced a new and straightforward filter method for feature selection (FS). The proposed method, called MRFS, is based on mean ratio and uses only basic mathematical equations.

Extensive computational experiments in classification tasks suggest that MRFS is promising. The MRFS method was evaluated and compared with three alternative FS methods using four classifiers and ten data sets. The MRFS results achieved higher accuracy in 72.5% of the experiments. To be more specific, in 50%, MRFS outperformed alternative methods, and in 22.5% it achieved the best performance together with other methods. In summary, MRFS gave the best results in 9 out of 10 data sets with Naive Bayes, it was also better in 8 out of 10 data sets when used together with K-Nearest Neighbor, it showed the best results in 7 out of 10 data sets with Support Vector Machine, and in 5 out of 10 when applied in conjunction with ANN.

Future work shall address the development of a new wrapper FS based on MRFS.

Footnotes

Appendix

A. Attributes ranked by FS methods

In this appendix, Table 10 introduces the number of attributes used in the experiments; this value was a one-step reduction of 20% of the total amount. This table shows the data set name, the number of attributes present in the data set, and the number of attribute selected for the experiments. Tables 11–20 shows the attributes ranked by the FS methods. These tables present the FS method and the ordered attributes by their respective FS method.

Table 10

Number of attributes used in each experiment

Data set	All attributes	Selected attributes
Heart Disease	13	11, 9, 7, 5
Cervical Cancer	33	26, 20, 14, 8
Mobile Price	20	16, 12, 8, 4
COIL 2000	85	68, 51, 34, 17
Page Blocks	10	8, 6, 4, 3
QSAR Biodegradation	41	32, 24, 16, 8
South African Health	9	8, 7, 6, 5
Spambase	57	46, 35, 24, 13
Glass Identification	10	8, 6, 4, 3
Wine	13	11, 9, 7, 5

Table 11

Ranking of attributes – Heart Disease Data set

Method	Order of attributes
MRFS	9, 12, 3, 10, 2, 11, 7, 13, 6, 8, 1, 4, 5
Kruskal-Wallis	3, 7, 11, 6, 2, 10, 12, 9, 1, 4, 5, 8, 13
ReliefF	12, 13, 3, 10, 11, 8, 7, 1, 4, 2, 9, 6, 5
Chi-Square	8, 10, 12, 3, 9, 5, 1, 4, 11, 2, 13, 7, 6

Table 12

Ranking of attributes – Cervical Cancer Data set

Method	Order of attributes
MRFS	15, 18, 19, 20, 25, 24, 16, 22, 21, 31, 32, 33, 29, 27, 28, 30, 23, 17, 14, 13, 12, 26, 10, 11, 9, 6, 5, 7, 4, 1, 8, 3, 2
Kruskal-Wallis	32, 31, 33, 29, 27, 30, 17, 14, 12, 13, 26, 10, 11, 23, 7, 20, 5, 28, 6, 15, 22, 24, 21, 19, 25, 16, 18, 9, 8, 2, 4, 3, 1
ReliefF	32, 33, 31, 8, 9, 4, 5, 1, 29, 20, 30, 3, 23, 27, 18, 28, 10, 26, 6, 12, 13, 11, 2, 19, 14, 16, 7, 17, 15, 22, 25, 24, 21
Chi-Square	32, 31, 33, 9, 29, 27, 13, 30, 20, 6, 11, 17, 14, 12, 26, 1, 23, 28, 10, 4, 18, 7, 5, 16, 3, 25, 19, 21, 24, 8, 2, 15, 22

Table 13

Ranking of attributes – Mobile Price Data set

Method	Order of attributes
MRFS	14, 12, 1, 13, 6, 7, 19, 16, 10, 4, 2, 5, 8, 9, 15, 20, 17, 11, 18, 3
Kruskal-Wallis	3, 5, 18, 8, 6, 20, 4, 2, 19, 14, 17, 16, 15, 1, 13, 12, 10, 9, 7, 11
ReliefF	14, 1, 13, 12, 17, 5, 7, 10, 9, 15, 11, 3, 16, 18, 8, 2, 6, 4, 19, 20
Chi-Square	14, 12, 1, 13, 9, 7, 16, 17, 5, 15, 11, 10, 19, 6, 8, 2, 3, 4, 20, 18

Table 14

Ranking of attributes – Coil Data set

Method	Order of attributes
MRFS	53, 50, 71, 74, 82, 61, 81, 60, 46, 67, 57, 64, 78, 85, 56, 73, 54, 75, 69, 76, 21, 58, 48, 52, 55, 84, 79, 51, 83, 68, 47, 72, 44, 65, 62, 59, 16, 77, 37, 29, 34, 30, 25, 40, 45, 20, 13, 80, 63, 19, 12, 18, 31, 43, 24, 39, 66, 36, 11, 1, 5, 23, 42, 28, 22, 26, 6, 35, 10, 15, 17, 9, 32, 70, 3, 7, 8, 27, 33, 38, 41, 4, 14, 2, 49
Kruskal-Wallis	76, 55, 64, 85, 61, 82, 83, 62, 57, 78, 49, 70, 51, 72, 63, 84, 45, 66, 58, 79, 60, 81, 41, 52, 73, 48, 56, 69, 77, 50, 71, 53, 74, 46, 67, 75, 54, 20, 65, 80, 59, 47, 68, 1, 44, 22, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 21, 23, 2, 24, 42, 40, 39, 38, 37, 36, 35, 34, 33, 32, 31, 30, 29, 28, 27, 26, 25, 43
ReliefF	44, 65, 55, 64, 85, 78, 82, 47, 61, 57, 76, 62, 68, 1, 5, 59, 25, 83, 84, 40, 16, 81, 20, 58, 39, 60, 51, 63, 72, 49, 17, 26, 42, 79, 14, 70, 41, 33, 3, 43, 19, 27, 69, 15, 71, 74, 77, 2, 18, 66, 56, 50, 28, 48, 30, 36, 31, 35, 53, 8, 4, 45, 7, 32, 6, 22, 10, 80, 38, 12, 73, 23, 13, 9, 11, 24, 29, 21, 34, 37, 52, 75, 46, 67, 54
Chi-Square	47, 1, 61, 59, 30, 16, 44, 31, 68, 37, 64, 43, 82, 18, 25, 21, 34, 19, 39, 36, 57, 55, 65, 5, 13, 12, 76, 54, 29, 40, 24, 42, 46, 10, 85, 35, 28, 80, 32, 22, 60, 23, 20, 78, 15, 75, 81, 17, 52, 11, 9, 58, 26, 67, 53, 48, 73, 83, 50, 3, 51, 7, 6, 74, 62, 56, 69, 84, 72, 71, 45, 79, 27, 63, 8, 33, 66, 77, 38, 70, 4, 14, 2, 41, 49

Table 15

Ranking of attributes – Page Blocks Data set

Method	Order of attributes
MRFS	9, 4, 7, 10, 2, 8, 3, 1, 5, 6
Kruskal-Wallis	1, 7, 5, 10, 4, 6, 2, 3, 9, 8
ReliefF	1, 6, 2, 8, 5, 9, 3, 10, 4, 7
Chi-Square	3, 8, 9, 10, 4, 7, 1, 2, 5, 6

Table 16

Ranking of attributes – QSAR Biodegradation Data set

Method	Order of attributes
MRFS	28, 21, 19, 41, 20, 6, 4, 3, 25, 26, 40, 24, 29, 5, 33, 32, 11, 34, 7, 38, 23, 10, 14, 31, 9, 36, 30, 13, 39, 8, 1, 27, 35, 15, 22,
	37, 17, 18, 16, 2, 12
Kruskal-Wallis	30, 9, 40, 35, 10, 28, 16, 32, 19, 29, 26, 24, 21, 4, 20, 14, 6, 34, 25, 11, 38, 5, 41, 23, 22, 3, 39, 7, 37, 36, 8, 12, 33, 13, 31,
	15, 17, 18, 27, 2, 1
ReliefF	38, 19, 22, 1, 37, 28, 35, 40, 39, 8, 36, 4, 2, 27, 24, 14, 16, 12, 15, 18, 29, 32, 10, 31, 23, 34, 13, 30, 11, 9, 20, 5, 26, 17, 6,
	25, 41, 21, 3, 7, 33
Chi-Square	8, 11, 7, 34, 5, 23, 41, 33, 30, 3, 31, 39, 10, 38, 36, 15, 14, 6, 1, 13, 9, 12, 25, 27, 32, 37, 20, 40, 22, 35, 4, 24, 21, 26, 29,
	16, 19, 28, 17, 18, 2

Table 17

Ranking of attributes – South African Health Data set

Method	Order of attributes
MRFS	2, 9, 3, 5, 4, 8, 1, 6, 7
Kruskal-Wallis	2, 8, 1, 3, 4, 5, 6, 7, 9
ReliefF	2, 1, 8, 6, 9, 4, 3, 7, 5
Chi-Square	9, 2, 4, 1, 8, 3, 6, 5, 7

Table 18

Ranking of attributes – Spambase Data set

Method	Order of attributes
MRFS	41, 27, 29, 4, 32, 42, 31, 25, 26, 34, 23, 7, 30, 20, 35, 48, 44, 46, 53, 15, 24, 33, 28, 39, 43, 16, 47, 17, 56, 11, 8, 22, 36, 52, 37, 9, 55, 38, 6, 54, 45, 18, 21, 57, 5, 51, 49, 13, 40, 10, 1, 3, 14, 19, 2, 50, 12
Kruskal-Wallis	52, 53, 21, 16, 5, 7, 3, 24, 23, 17, 10, 18, 8, 6, 11, 2, 9, 1, 12, 54, 20, 19, 13, 15, 50, 14, 45, 40, 22, 49, 56, 55, 48, 47, 46, 41, 44, 43, 42, 51, 29, 39, 38, 37, 36, 35, 34, 33, 32, 31, 30, 28, 27, 26, 25, 4, 57
ReliefF	19, 16, 7, 52, 21, 8, 18, 24, 53, 5, 11, 17, 9, 10, 57, 6, 50, 23, 1, 56, 20, 2, 13, 55, 4, 22, 46, 14, 38, 49, 54, 45, 40, 12, 47, 28, 51, 3, 36, 34, 33, 32, 27, 31, 48, 35, 44, 15, 41, 39, 29, 42, 30, 37, 43, 26, 25
Chi-Square	57, 56, 55, 27, 25, 21, 16, 26, 7, 52, 19, 23, 20, 46, 4, 24, 17, 5, 53, 42, 22, 45, 8, 18, 29, 35, 30, 28, 15, 9, 33, 44, 6, 37, 31, 11, 39, 3, 10, 36, 32, 34, 41, 43, 48, 54, 13, 1, 40, 2, 14, 49, 50, 38, 51, 47, 12

Table 19

Ranking of attributes – Glass Identification Data set

Method	Order of attributes
MRFS	9, 7, 10, 4, 5, 1, 8, 3, 6, 2
Kruskal-Wallis	5, 2, 1, 9, 3, 4, 6, 7, 8, 10
ReliefF	1, 7, 2, 8, 5, 6, 9, 3, 4, 10
Chi-Square	1, 9, 4, 7, 5, 3, 8, 10, 6, 2

Table 20

Ranking of attributes – Wine Data set

Method	Order of attributes
MRFS	10, 7, 13, 12, 6, 2, 9, 8, 11, 4, 1, 3, 5
Kruskal-Wallis	8, 2, 11, 9, 4, 1, 3, 5, 6, 7, 10, 12
ReliefF	13, 1, 4, 5, 7, 9, 3, 10, 11, 12, 2, 6, 8
Chi-Square	13, 10, 7, 5, 4, 2, 12, 6, 9, 1, 11, 8, 3

References

Naik

Kuppili

Edla

. Efficient feature selection using one-pass generalized classifier neural network and binary bat algorithm with a novel fitness function. Soft Computing. 2019 jul; 1-13. Available from: doi: 10.1007/s00500-019-04218-6.

Gauthama Raman

Nivethitha

Kannan

Shankar Sriram

. A hybrid approach using rough set theory and hypergraph for feature selection on high-dimensional medical datasets. Soft Computing. 2019 dec; 23(23): 12655-12672. Available from: doi: 10.1007/s00500-019-03818-6.

Guyon

Elisseeff

. An introduction to variable and feature selection. The Journal of Machine Learning Research. 2003 mar; 3: 1157-1182.

Vora

Yang

. A comprehensive study of eleven feature selection algorithms and their impact on text classification. In: 2017 Computing Conference. vol. 2018-Janua. IEEE; 2017; pp. 440-449. Available from: http://ieeexplore.ieee.org/document/8252136/.

Blum

Langley

. Selection of relevant features and examples in machine learning. Artificial Intelligence. 1997 dec; 97(1-2): 245-271. Available from: https://www-sciencedirect-com-443.web.bisu.edu.cn/science/article/pii/S0004370297000635.

Kang

Tian

. Machine Learning: Data Pre-processing. In: Prognostics and Health Management of Electronics. Chichester, UK: John Wiley and Sons Ltd; 2018; pp. 111-130. Available from: doi: 10.1002/9781119515326.ch5.

Jimenez

Martinez

Marzano

Palma

Sanchez

Sciavicco

. Multiobjective evolutionary feature selection for fuzzy classification. IEEE Transactions on Fuzzy Systems. 2019 may; 27(5): 1085-1099. Available from: https://ieeexplore.ieee.org/document/8607010/.

Alippi

Boracchi

Roveri

. Ensembles of change-point methods to estimate the change point in residual sequences. Soft Computing. 2013 nov; 17(11): 1971-1981. Available from: doi: 10.1007/s00500-013-1130-7.

Runger

DCMGC

. Applied Statistics and Probability for Engineers. Wiley; 2003.

10.

Montgomery

. Design and Analysis of Experiments, 8th Edition. John Wiley & Sons, Incorporated; 2013.

11.

Mirshab

Anneberg

. Fuzzy logic extension of nonparametric approach to feature selection and binary decision tree design. In: Proceedings of 36th Midwest Symposium on Circuits and Systems. IEEE; pp. 828-831. Available from: http://ieeexplore.ieee.org/document/343197/.

12.

Bennett

. A non-parametric test for randomness in a sequence of multinomial trials. Biometrics. 1964 mar; 20(1): 182.

13.

Borah

Chutia

Baruah

Raju

PLN

. Dimension reduction of hyperion data for improving classification performance – An assessment. In: 2017 2nd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT). IEEE; 2017; pp. 1359-1363. Available from: http://ieeexplore.ieee.org/document/8256820/.

14.

Influence of Data Geometry in Random Subset Feature Selection. International Journal of Data Mining & Knowledge Management Process. 2017 jul; 7(4): 33-41.

15.

Suchetha

Nikhil

Hrudya

. Comparing the wrapper feature selection evaluators on twitter sentiment classification. In: ICCIDS 2019 – 2nd International Conference on Computational Intelligence in Data Science, Proceedings. Institute of Electrical and Electronics Engineers Inc.; 2019.

16.

Zeng

Heng

. Feature selection and visualization based on interaction dominance. In: Proceedings – 2019 IEEE 4th International Conference on Data Science in Cyberspace, DSC 2019. vol. 2019-Janua. Institute of Electrical and Electronics Engineers Inc.; 2019; pp. 668-673.

17.

Haq

Zhang

Peng

Rahman

. Combining multiple feature-ranking techniques and clustering of variables for feature selection. IEEE Access. 2019; 7: 151482-151492.

18.

Chen

Sun

Chen

Yang

. Supervised feature selection with a stratified feature weighting method. IEEE Access. 2018 mar; 6: 15087-15098.

19.

Yan

Nazmi

Erol

Homaifar

Gebru

Tunstel

. An efficient unsupervised feature selection procedure through feature clustering. Pattern Recognition Letters. 2020 mar; 131: 277-284.

20.

Azhagusundari

. An integrated method for feature selection using fuzzy information measure. In: 2017 International Conference on Information Communication and Embedded Systems (ICICES). IEEE; 2017; pp. 1-6. Available from: http://ieeexplore.ieee.org/document/8070711/.

21.

Lang

. In: The Mean Value Theorem. New York, NY: Springer New York; 1986; pp. 159-180.

22.

Sahoo

Riedel

. Mean Value Theorems and Functional Equations. World Scientific; 1998. Available from: https://books.google.com.br/books?id=My0wvbZgoF4C.

23.

scikit-learn: machine learning in Python; 2020. Available from: https://scikit-learn.org/.

24.

Kaggle. The Home of Data Science & Machine Learning. https://www.kaggle.com/.

25.

UCI. UC Irvine Machine Learning Repository. https://archive.ics.uci.edu/ml/index.php.

26.

KEEL. Knowledge Extraction based on Evolutionary Learning. https://sci2s.ugr.es/keel/category.php?cat=clas.