An effective feature subset selection approach based on Jeffries-Matusita distance for multiclass problems

Abstract

Jeffries-Matusita (JM) distance, a transformation of the Bhattacharyya distance, is a widely used measure of the spectral separability distance between the two class density functions and is generally used as a class separability measure. It can be considered to have good potential to be used for evaluation of the effectiveness of a feature in discriminating two classes. The capability of JM distance as a ranking based feature selection technique for binary classification problems has been verified in some research works as well as in our earlier work. It was found by our simulation experiments with benchmark data sets that JM distance works equally well compared to other popular feature ranking methods based on mutual information, information gain or Relief. Extension of JM distance measure for feature ranking in multiclass problems has also been reported in the literature. But all of them are basically rank based approaches which deliver the ranking of the features and do not automatically produce the final optimal feature subset. In this work, a novel heuristic approach for finding out the optimum feature subset from JM distance based ranked feature lists for multiclass problems have been developed without explicitly using any specific search technique. The proposed approach integrates the extension of JM measure for multiclass problems and the selection of the final optimal feature subset in a unified process. The performance of the proposed algorithm has been evaluated by simulation experiments with benchmark data sets in comparison with two other previously developed multiclass JM distance measures (weighted average JM distance and another multiclass extension equivalent to Bhattacharyya bound) and some other popular filter based feature ranking algorithms. It is found that the proposed algorithm performs better in terms of classification accuracy, F-measure, AUC with a reduced set of features and computational cost.

Keywords

Feature selection JM distance multiclass extension feature ranking and subset selection

1 Introduction

With the rapid development of science and technology, the amount of generated data in every sphere of life has been increased tremendously, which causes the classification or mining of data increasingly difficult. These data are often high dimensional, making their analysis more complicated and computationally costly. The data need to be preprocessed to get rid of redundant and irrelevant information. Feature selection is the most important processing step prior to classification or clustering for any pattern recognition or data mining problem [1, 2]. This is a process of dimensionality reduction in which discriminatory and relevant information is retained by discarding redundant and irrelevant information leading to better performance of the classification model in terms of classification accuracy as well as computational cost [3, 4]. Feature subset selection produces the best or the optimum feature subset of d features from an original set of n features where d < n.

The optimum feature subset selection process comprises two main tasks, defining a feature evaluation criterion for evaluation of an individual feature or a feature subset and a selection process to find out the best feature subset from the full feature set according to the evaluation criterion. Based on the nature of the feature evaluation criterion, there are two main categories of feature selection algorithms, filter and wrapper. Filter algorithms [5, 6] use feature evaluation criteria based on intrinsic characteristics of data set and are independent of the classifier model while wrapper algorithms [7] use classifier performance as the feature evaluation criterion. Filter methods are generally faster than wrapper methods in producing optimal feature subset, but classification performance of the selected feature subset might be poor compared to the wrapper selected feature subset, which is tuned to the particular classifier. There are some embedded algorithms that combine the merits of both filter and wrapper approaches [8].

Based on the selection process of the optimal feature subset, two main approaches exist, rank based and search based. Rank based approaches evaluate each feature independently, rank them according to their merit/goodness and then select an appropriate portion of the top ranked features to form the final feature subset. Though simple and computationally light, rank based approaches ignore the interaction between features and cannot guarantee the optimality of the selected subset. Moreover, some strategies need to be adopted to fix the optimum percentage of top ranking features to be selected. According to [9], the best two individual features do not produce the best feature subset of two features. An exhaustive evaluation of all possible feature subsets can only guarantee the optimality of the selected feature subset. But for high dimensional data, it leads to an explosion of computational time with increasing dimension of data. To solve this combinatorial optimization problem, a lot of search algorithms have been developed so far for the selection of optimum feature subset, which include mainly statistical or mathematical and soft computing based techniques. The feature evaluation measures for filter approaches are generally classified into four categories such as distance based, dependency or relevance based, information theoretic and consistency measure [10]. Distance based evaluation measures include class separability measures such as divergence or Kullback-Liebler distance, Bhattacharyya distance, Jeffries-Matushita distance, Mahalanobis distance. Dependency based measures consider correlation or similarity of a feature to a class. Information theoretic measures determine the information gain of a feature by its inclusion. Consistency based measure penalizes inconsistent features where inconsistency is defined as two instances having the same feature values but different class labels. Some efficient popular filter evaluation measures used for feature ranking are Mutual Information (MI), Information Gain (IG), Gain Ratio (GR), Symmetrical uncertainty (SU), Chi-squared (CS), One-R, Relief, Jeffries-Matusita (JM) distance and Correlation [11].

As noted above, Jeffries-Matsushita (JM) distance, a modification of Bhattacharyya distance, can be used to rank a feature by assessing the degree of separability between two classes using the feature. Unlike other class separability measures like Bhattacharyya distance or Divergence, JM distance has an upper bound which makes it a more realistic measure for feature evaluation [12]. It is also noted in a very early study [13] that JM distance provides a much more reliable criterion for feature selection compared to Divergence. An extended definition of JM distance for the multiclass problem, the weighted average J_ave, has been reported in [14]. In [15], the authors proposed J_min, the best JM distance between the least separable pair of classes as the evaluation criterion for multiclass problems. The other two approaches for multiclass extension of JM distance reported in [16] and [17] respectively are found to be effective measures for feature selection. The authors of this work studied JM distance as a tool for feature selection in [18] and found that JM distance performs equally well compared to other popular filter based feature selection measures. In our other previous work [19], it was also found that the stability of JM distance based feature selection algorithm is better compared to other popular filter based ranking algorithms for feature subset selection.

In this work, an efficient feature subset selection algorithm for multiclass problems based on JM distance has been proposed. Until recently, JM based approaches that have been proposed for feature selection in multiclass problems are all univariate feature ranking strategy in which average JM distance for all the class pairs are used for final feature ranking whereas we propose here a heuristic approach to select the final optimal feature subset. The proposed approach consists of two steps. In the first step, similar to existing JM distance based feature selection approaches, features are ranked according to the JM distances for all class pairs and multiple ranked feature lists are created corresponding to each class pair. The second step is the novel contribution of this work in which a heuristic approach is developed to select the final optimum feature subset from the multiple ranked feature lists (corresponding to each class pair) based on the average JM distance values of the top ranking features of each class pair. Unlike traditional approaches, the proposed algorithm does not use any explicit search mechanism to find out the optimum feature subset. The proposed algorithm has been evaluated by comparing with other multiclass JM distance based feature selection as well as with some other popular filter based ranking methods by simulation experiment with benchmark data sets. The rest of the paper is organized as follows. Section 2, the next section presents the background and related works. In section 3, our proposed feature selection approach using JM distance for multiclass problems has been described. The following section describes simulation experiments, results and analysis followed by the final section of conclusion and future works.

2 Rank based filter approach for feature selection

Feature selection with filter approach has been extensively used in machine learning for many years due to its simplicity, low computational complexity and non reliance on any particular classifiers. Feature ranking based filter methods also perform very well in many real life problems of pattern recognition, machine learning and data mining [11 , 20–22] compared to search based methods. Some of the well-known filter type evaluation measures for feature ranking are Mutual Information(MI), Information Gain (IG), Gain Ratio (GR), Symmetrical uncertainty (SU), Chi-squared (CS), One-R and Relief-F [23]. In this work, JM distance, a modification of Bhattacharyya distance, is used as the evaluation measure for ranking based feature selection in multiclass problems.

2.1 Feature evaluation measures

Some of the filter type feature evaluation measures including JM distance are briefly described in this section.

2.1.1 Mutual information (MI)

Mutual Information (MI), one of the most used feature evaluation measure, is an information theoretic measure which expresses the mutual dependency between two random variables. Mutual information between a feature and the class can be used to assess relevance of a feature for representing the class. MI of two random variables x and y are defined in terms of their probabilistic density functions p (x),p (y), p (x, y): $MI (x, y) = \int \int p (x, y) \log \frac{p (x, y)}{p (x) p (y)} dx dy$ (1)

There are several modifications of MI that have been proposed in the literature over the years. Normalized MI, which maps the value of MI between 0 and 1, is an important improvement of MI, popularly used for feature selection [24].

2.1.2 Information gain (IG)

Information Gain (IG), also an information theoretic measure, is the amount of information gained about a random variable from observing another variable. It is measured as the reduction in entropy of the class C when the feature A is observed and is expressed by the following equation [25]. $IG (C, A) = H (C) - H (C / A)$ (2) where H (C) and H (C/A) represent the entropy of class C and the conditional entropy of class C given the value of feature A respectively. The expected value of IG is the mutual information MI (C, A).

2.1.3 Gain ratio (GR)

Gain Ratio (GR) of a feature A is defined as in [26]. $GR (A) = \frac{IG (S, A)}{split (S)}$ (3) where IG (S, A) is information gain of the feature A for the training data set S, and the denominator split (S) indicates splitting information that is defined as follows: $split (S) = - \sum_{j = 1}^{k} \frac{| S_{j} |}{| S |} {log}_{2} \frac{| S_{j} |}{| S |}$ (4) where S_j (j = 1, 2, . . . , k) is the jth partition and k is the total number of partitions.

2.1.4 Symmetrical uncertainty (SU)

Symmetrical Uncertainty (SU) is obtained by normalizing Mutual Information (MI) to the entropies of the two variables and is defined as in [27]. $SU (C, A) = 2 \frac{MI (C, A)}{H (C) + H (A)}$ (5)

2.1.5 Chi-squared (CS)

Chi-squared algorithm for feature selection, based on χ² statistics to test the independence of two variables (here class and a feature), discretizes numeric attributes repeatedly until some inconsistencies are found in the data and achieve feature selection via discretization. It is measured for two adjacent intervals by comparing the expected and actual values as in the following equation [28].

$CS = \sum_{i = 1}^{2} \sum_{j = 1}^{C} \frac{(O_{ij} - E_{ij})}{E_{ij}}$ (6)

C is the number of classes, O_ij is the number of instances (feature values) of the jth class in the ith interval and E_ij is the expected frequency of O_ij, where E_ij = R_iC_j/N, R_i is the number of instances in the ith interval and C_j and N are the number of instances of the jth class and the total number of instances (counting both the intervals), respectively. The higher value of CS expresses the higher dependency between the class and the feature and the feature is selected.

2.1.6 One-R

One-R ranks the feature using a rule based classification algorithm proposed by Holte [29]. For each feature, the algorithm generates simple rules and selects the rule which has the lowest error rate for the majority of the classes. The features are ordered according to their quality of the corresponding rules. One-R treats all numerically valued attributes as continuous and uses a straightforward method to divide the range of values into several disjoint intervals.

2.1.7 Relief-F

Relief-F is an extension of the original Relief algorithm that can be used for multiclass problems [30]. The original Relief algorithm is formulated iteratively from an instance based learning approach that evaluates a feature by assigning a weight to the feature. This feature weight actually represents the feature’s relevance with respect to the target concept and is calculated by locating for each instance the nearest neighbor from the same class (nearest Hit) and the nearest neighbor from the opposite class (nearest Miss). Weight W_i is initialized to 0, and then in each step, it is updated according to the following rule [31]. $W_{i} = W_{i} - (X_{i} - {nearHit}_{i})^{2} + (X_{i} - {nearMiss}_{i})^{2}$ (7)

Finally, an average of W_i is taken over the iterations. Relief-F is also more robust and has the ability to deal with incomplete and noisy data.

2.1.8 Bhattacharyya distance

Bhattacharya Distance measures the similarity between two probability distributions. It is used to measure the separability or overlap of two classes for each feature in a binary classification problem and rank them to select the best performing features. If the two distributions for two classes i and j are considered to be Gaussian then Bhattacharya distance B_ij can be defined as [32]:

$\begin{matrix} B_{ij} = & \frac{1}{8} (μ_{i} - μ_{j})^{T} (\frac{Σ_{i} + Σ_{j}}{2})^{- 1} (μ_{i} - μ_{j}) \\ + \frac{1}{2} ln \frac{| \frac{Σ_{i} + Σ_{j}}{2} |}{\sqrt{| Σ_{i} | | Σ_{j} |}} \end{matrix}$ (8) where μ_i, μ_j and Σ_i, Σ_j represent the mean vectors and the covariance matrices, respectively, for the classes w_i and w_j.

2.1.9 Jeffries-matushita (JM) distance

Jeffries-Matushita (JM) distance is also a measure of statistical separability for two classes. For feature x, it is defined for two classes w_i and w_j as in [15]. ${JM}_{ij} = {\int_{x} [\sqrt{p (x / w_{i})} - \sqrt{p (x / w_{j})}]^{2} dx}^{1 / 2}$ (9)

JM distance is bounded by a range of values from 0 to 2. It is related to Bhattacharyya distance as ${JM}_{ij} = \sqrt{2 (1 - e^{- B_{ij}})}$ (10)

The JM distance yields a general measure of separability between the two classes based on the average distance between their class density functions. A larger JM distance value indicates a clearer separation between two classes. It is the more reliable criterion for correct classification compared to divergence or Bhattacharyya distance.

The original JM distance measure, used for binary classes, has been extended for multiclass problems as Weighted average JM distance (JM_ave). For m number of classes, it is defined according to [14] as: ${JM}_{ave} = \frac{1}{m (m - 1)} \sum_{i = 1}^{m - 1} \sum_{j = i + 1}^{m} {JM}_{ij}$ (11)

In [16], the authors proposed another multiclass extension of JM distance, JM_Bh, which is an equivalent of Bhattacharyya bound to Bayes error, and justified its efficiency over JM_ave in feature selection for multiclass problems by simulation experiments. For m class problem, JM_Bh is defined as: ${JM}_{Bh} = \sum_{i = 1}^{m} \sum_{j \geq i}^{m} {JM}_{ij}^{2}$ (12)

2.2 Related work on feature selection algorithms

Some of the popular filter based feature ranking algorithms are described here in brief. In [20], authors have used the feature ranking methods such as information gain, gain ratio, correlation, symmetrical uncertainty and chi-squared for recognizing the handwritten digits. The effect of feature selection on classification accuracy is analyzed in [33]. This paper uses six feature ranking based filter methods of Information Gain, Gain Ratio, Symmetrical Uncertainty, One-R, Chi-square and Relief-F for the feature selection process and uses three classification models for comparative study. In [34], authors investigated Cancer Classification using feature selection with filter methods of Signal-to-noise statistic, CFS, Chi-square and Relief-F for Probabilistic Neural Networks. Generally, for high dimensional data such as gene expression data, the filter method is extensively used. In [35], the authors proposed an unsupervised feature selection algorithm with feature ranking method of Chi-squared for maximizing the classifier performance. This algorithm achieved better prediction accuracy and also reduced the number of features compared to other methods. For handwriting recognition, Cilia et al. in [11], used five univariate feature ranking based methods for ranking the features while feature subset was chosen by a greedy search approach. They used Chi-square, Relief, Gain ratio, Information gain and Symmetrical uncertainty as the feature ranking method and the Best First (BF) search strategy combined with consistency criterion and correlation based feature selection criterion as for searching feature subsets. Chen et al. in [36], used filter based ranking feature selection (FRFS) methods for Security vulnerability prediction (SVP) and showed that FRFS can improve the performance of SVP compared to others. They also performed the diversity analysis on identified vulnerable modules by using different FRFS methods. In [37], Ghazy et al. used the different ranking and subset-based feature selection techniques for finding the optimum number of features to find an appropriate classifier. They mainly used these feature selection techniques to verify the performance of the intrusion detection system (IDS). In [38], authors proposed a task of feature ranking for multi-target regression (MTR). They studied two types of feature ranking scores for MTR, one was ensemble based, and the other was an extension of the RReliefF method. Lee et al. in [39], proposed an efficient multivariate feature ranking method for gene selection and for improving the accuracy of microarray data classification. In their work, they created a new feature ranking method using the Markov blanket (MB), which embedded with relevance. They showed that the proposed feature ranking method possesses high classification accuracy as well as good efficiency.

2.2.1 JM distance based feature selection algorithms

In [40], author proposes a novel spectral matching technique by combining the JM distance and the Spectral Angle Mapper (SAM) algorithm in hyperspectral image data. Their proposed JM-SAM approach performs very well than the individual JM distance measure and SAM algorithm with the least average entropy in spectral matching. Dalponte et al. in [41], used the Jeffries-Matusita distance (JM) combined with sequential forward floating selection (SFFS) search strategy for fast and reliable feature selection. In this case, JM distance also has been used for hyperspectral data. In [42], the authors presented an analysis of the linear attenuation coefficients, which were used as a useful feature of mono-spectral and multispectral images using statistical pattern classification tools. In this paper, feature extraction was performed by JM distance and Karhunen-Loeve transformation. Daamouche et al. [43] proposed a particle swarm optimization (PSO) based approach for very high resolution (VHR) image classification, in which JM distance, support vector machine (SVM), cross-validation (CV) accuracy and normal Bhattacharyya distance were used as the fitness function. In [44], authors developed a new technique for crop identification by combining the wavelet variance and the JM distance (CIWJ). The proposed CIWJ approach outperforms other approaches for efficient crop mapping, such as agricultural crop identification with high spatial resolution images and classifications for more general or specific land use. In this paper [45], JM distance is applied as an evaluator of image segmentation in the area of remote sensing images. Here authors proposed an unsupervised evaluation method for evaluating the performance of segmentation using the JM distance and the area-weighted variance (WV). Authors in the paper of [16] proposed an extension of the JM distance measure for multiclass problems of feature selection. They formulated an equation for JM distance measure and used optical remote-sensing data for the experiment. They also compared their results with the most familiar weighted average JM distance. Sen et al. [18] studied JM distance as an efficient tool for feature selection in binary classification problems compared to their other feature ranking methods.

3 Proposed feature selection approach with JM distance for multiclass problems (JM_mc)

In this work, a novel optimum feature subset selection approach JM_mc, for a multiclass problem based on the feature evaluation by JM measure is proposed, which is described in detail in this section. Traditionally optimum feature subset selection approaches require a measure for evaluating feature or feature subset and a search strategy for finding out the best feature subset from possible feature subsets. The proposed approach is composed of two steps. In the first step (algorithm 1), similar to existing JM distance based feature selection approaches, features are ranked according to the JM distances for all class pairs and multiple ranked feature lists are created corresponding to each class pair. Other multiclass JM distance based approaches for feature selection use some kind of average JM measure, averaged over all class pairs, which is described in the previous section by Equation 11 and Equation 12, for final feature ranking. In our work, a novel heuristic approach for selecting an optimum feature subset from the multiple lists of ranked features corresponding to each class pair, obtained after the first step, is proposed in the second step (algorithm 2). Moreover, our approach does not include traditional search based methods for final feature subset selection. The heuristics proposed in our work is based on the following core concept. If the average JM values of the top ranking features for a class pair is high, the classes are considered to be well separated and few number of top ranking features are needed for good classification accuracy. The case is opposite for low average JM values of the top ranking features. The classes are not well separated, and we need to consider features in the final feature subset to provide better classification accuracy. The detailed algorithm is presented below.

Algorithm 1: For any multiclass data set, let the number of features be n and the number of classes be m where m > 2. The number of class pairs (NC) will be $\frac{m (m - 1)}{2}$ . JM distance is calculated for each feature and for each pair of classes according to Equation (9). Each set of calculations are repeated for K times with K different seed values and average value of K trials is taken as the JM distance JMmean_pq of qth class pair of pth feature where p = 1, ⋯ , n and $q = 1, \dots, NC = \frac{m (m - 1)}{2}$ . The procedure is shown in Algorithm 1. Figure 1 illustrate an example of class pair- feature table after the execution of Algorithm 1. In this table, the value of a particular cell indicates a JMmean_pq for K trials.

Algorithm 2: In Algorithm 2, for qth class pair, features are ranked in descending order according to the values of JMmean_pq distance of all the features (JMmean_pq, p = 1, ⋯ , n). This is done for all class pairs. So we have now feature lists (FL_q, where q = 1, ⋯ , NC) of ranked features corresponding to each class pair. The number of features in all the feature lists are the same as the total number of features n. Now our proposed approach selects the most important feature subset from the (FL_q, where q = 1, ⋯ , NC) ordered feature lists for the multiclass problem. The underlying concept of the selection of features from the feature lists is presented below.

Algorithm 1 Calculation of average JM distance for all class-pairs and features

1: Procedure JMClassPairclassPair, featureList, K

2: JFC_k : JM distances for all class-pairs of all features of the k^th iteration

3: JMmean : Average JM distance for all class- pairs of all features for K trials

4: for each q of classPair do

5: for each p of featureList do

6: sum← 0

7: for each k of K do

8: sum← sum+JFC_k [p, q]

9: end for

10: JMmean [p, q] ← sum/K

11: end for

12: end for

13: return JMmean

14: end procedure

JMmean_pq distance is a separability measure, which determines the average class separability of a feature p for a class pair q. The larger JMmean_pq distance value of a feature p indicates that the feature can separate the class pair q very well, which implies an increase in classification accuracy. For multiclass problems, JMmean_pq distance is calculated for each class pair q and different class pair shows a different value of JMmean_pq distance for a single feature p. If for a particular class pair q, the JMmean_pq distances of the top ranking features are near to 2 (the upper limit of standard JM distance is theoretically 2), then the features are very strong to easily separate classes. In this case, a lesser number of features can provide good classification accuracy. On the other hand, if the JMmean_pq distance values of the top ranking features are low, the features have not good separability, then comparatively more features are required to provide moderate classification accuracy. Based on this concept, different percentages of features are selected from different class pairs for improvement of classification accuracy.

The final feature subset selection from the ranked feature lists of all class pairs need to be done according to the following rules.

If the top JMmean_pq distance value ( $\underset{q}{FLmax}$ ) of the feature list of the qth class pair (FL_q) is greater than 1, then top α% of features are selected from the feature list of that class pair FL_q. (α has to be predefined).

If the top JMmean_pq distance value of the feature list ( $\underset{q}{FLmax}$ ) of any class pair is greater than 0.5 but less than or equal to 1, then top 2α% of features are selected from the feature list of that class pair.

If the top JMmean_pq distance value of the feature list ( $\underset{q}{FLmax}$ ) of the class pair is less than or equal to 0.5, then top 3α% of features are selected from the feature list of that class pair.

The selected features from all the class pairs are put in a list (selected feature list SFL). As there is a possibility that a particular feature might be selected more than once, the frequency of occurrence of each feature p in the selected feature list SFL is counted and let it be |SFL_p|, p = 1, ⋯ , n. Let the median value be SFL_med. Now for finding out the final feature subset, depending on the relative number of features and the number of class pairs, two cases are considered.

If the total number of features n in the data set is less than or equal to the total number of class pairs (n ≤ NC), then the final feature subset will constitute the features (from the selected feature list SFL) whose occurrence frequency is more than or equal to SFL_med

If the total number of features is greater than the total number of class pairs (n > NC), then all the features in SFL will be selected as the final feature subset.

The proposed heuristics selection process is presented clearly in Algorithm 2.

Fig. 1

Class pair-feature table.

Algorithm 2 Selection of final feature subset

1: procedure JMMultiClassclassPair, featureList, JM

2: n : len(featureList)

3: NC : len(classPair)

4: SFL : candidate features list

5: SFL ← empty list

6: for each q of classPair do

7: FLmax [q] ← find the feature with the highest JMmean [p, q] , p = 1, ⋯ , n

8: SortedfeatureList ← sort the fea-tures descending order of JMmean [p, q] , p = 1, ⋯ , n

9: F_s= empty list

10: if FLmax [q] > 1 then

11: F_s ← Take α% of top features in SortedfeatureList

12: else if FLmax [q] > 0.5 AND FLmax [q] < = 1 then

13: F_s ← Take 2α% of top features in SortedfeatureList

14: else if FLmax[q] < = 0.5 then

15: F_s ← Take 3α% of top features in SortedfeatureList

16: end if

17: for each Feature of F_s do

18: SFL.insert(Feature)

19: end for

20: end for

21: Freq ← dictionary for count the frequency of features in SFL

22: for each feature of SFL do

23: Freq[feature] ← SFL.count(feature)

24: end for

25: if n< = NC then

26: SFL_med ← find the median value of frequency in Freq

27: selectedFeature ← all features in Freq > = SFL_med

28: else if n > NC then

29: selectedFeature ← features in Freq

30: end if

31: return selectedFeature

32: end procedure

4 Simulation experiments and results

The implementation of the proposed algorithm has been done with benchmark data set for simulation experiments. The data set description is presented in the next subsection.

4.1 Dataset description

In this work, 37 datasets are used for performing a simulation experiment to validate the proposed approach. Among them, 25 datasets are collected from UCI repository [46] and rest are collected from OpenML [47]. Some datasets need to be preprocessed, which have missing values or are categorical in nature. Here, categorical type missing values in the datasets are replaced with the most frequently used value, and after that, the whole dataset is converted into numeric type. Numeric type missing values are replaced with the average value. Categorical type datasets without missing values are directly converted to numeric type. Table 1 represents the summary of datasets which includes the number of features, the number of instances, the number of classes and a short description of each data set.

Table 1
Summary of Datasets

Dataset Feature Instance Class Description

Analcat-authorship [47] 70 841 4 Analcatdata: Authorship was used in the book “Analyzing Categorical Data”

Analcat-marketing [47] 32 364 5 Analcatdata: Marketing was used in the book “Analyzing Categorical Data”

Breast-tissue [46] 9 106 6 Originally used for classification of 6 or 4 classes

Bridges [46] 11 105 6 Pittsburgh Bridges database. This database has original and numeric-discretized datasets.

Cars [47] 7 406 3 It was a version of the CRCARS data set of Donoho; David and Ramos; Ernesto (1982).

Cmc [46] 9 1473 3 Contraceptive Method Choice dataset. It was a subset of the 1987 National Indonesia Contraceptive Prevalence Survey.

Dermatology [46] 34 366 6 This dataset was used for determining the type of Eryhemato-Squamous Disease.

Dna [47] 180 3186 3 Primate Splice-Junction Gene Sequences (DNA) from Genbank 64.1 which originally collected from the StatLog project.

Eye-movements [47] 27 10936 3 This dataset was used for straight-forward classification task.

Gas-drift [46] 128 13910 6 Gas Sensor Array Drift Dataset.

Har [46] 561 10299 6 Human Activity Recognition (HAR) database. The data was recorded by using Smartphones.

Indian-pines [47] 220 9144 8 The image data was collected on 12 June 1992; contains the information of 9144 samples form 220 spectral bands.

Iris [46] 4 150 3 Iris Plants Database. The data set contains 3 classes of 50 instances each; where each class refers to a type of iris plant.

Mfeat-factors [46] 216 2000 10 Multiple Features Dataset: Factors

Mfeat-fourier [46] 76 2000 10 Multiple Features Dataset: Fourier

Mfeat-karhunen [46] 64 2000 10 Multiple Features Dataset: Karhunen

Mfeat-morph [46] 6 2000 10 Multiple Features Dataset: Morphological

Mfeat-pixel [46] 240 2000 10 Multiple Features Dataset: Pixel

Mfeat-zernike [46] 47 2000 10 Multiple Features Dataset: Zernike

Mice-protein [46] 77 1080 8 Mice Protein Expression Data Set.

Page-blocks [46] 10 5473 5 Page Blocks Classification Data Set.

Pasture [47] 21 36 3 Pasture Production dataset.

Pendigits [46] 16 10992 10 Pen-Based Recognition of Handwritten Digits dataset.

Satimage [46] 36 6430 6 Statlog (Landsat Satellite) Data Set.

Seeds [46] 7 210 3 This dataset represents the geometrical properties of kernels belonging to three different varieties of wheat.

Segment [46] 18 2310 7 Image Segmentation Data Set.

Splice [46] 60 3190 3 Molecular Biology (Splice-junction Gene Sequences) Data Set.

Squash-stored [47] 24 52 3 Squash Harvest Stored dataset.

Squash-unstored[47] 23 52 3 Squash Harvest Unstored dataset.

Synthetic-control [46] 60 600 6 Synthetic Control Chart Time Series Data Set.

Teaching-assistant[47] 5 151 3 Teaching Assistant dataset.

Vehicle [46] 18 846 4 Statlog (Vehicle Silhouettes) Data Set.

Vertebra-column [46] 6 310 3 Vertebral Column Data Set.

Vowel [46] 12 990 11 Vowel Recognition (Deterding data) dataset.

Waveform-5000 [46] 40 5000 3 Waveform Database Generator (Version 2) Data Set.

Wine [47] 13 178 3 It represents the output of a chemical analysis of wines grown in the same region in Italy but derived from three different cultivars.

Wine-quality [47] 11 4898 7 Wine Quality dataset.

Dataset	Feature	Instance	Class	Description
Analcat-authorship [47]	70	841	4	Analcatdata: Authorship was used in the book “Analyzing Categorical Data”
Analcat-marketing [47]	32	364	5	Analcatdata: Marketing was used in the book “Analyzing Categorical Data”
Breast-tissue [46]	9	106	6	Originally used for classification of 6 or 4 classes
Bridges [46]	11	105	6	Pittsburgh Bridges database. This database has original and numeric-discretized datasets.
Cars [47]	7	406	3	It was a version of the CRCARS data set of Donoho; David and Ramos; Ernesto (1982).
Cmc [46]	9	1473	3	Contraceptive Method Choice dataset. It was a subset of the 1987 National Indonesia Contraceptive Prevalence Survey.
Dermatology [46]	34	366	6	This dataset was used for determining the type of Eryhemato-Squamous Disease.
Dna [47]	180	3186	3	Primate Splice-Junction Gene Sequences (DNA) from Genbank 64.1 which originally collected from the StatLog project.
Eye-movements [47]	27	10936	3	This dataset was used for straight-forward classification task.
Gas-drift [46]	128	13910	6	Gas Sensor Array Drift Dataset.
Har [46]	561	10299	6	Human Activity Recognition (HAR) database. The data was recorded by using Smartphones.
Indian-pines [47]	220	9144	8	The image data was collected on 12 June 1992; contains the information of 9144 samples form 220 spectral bands.
Iris [46]	4	150	3	Iris Plants Database. The data set contains 3 classes of 50 instances each; where each class refers to a type of iris plant.
Mfeat-factors [46]	216	2000	10	Multiple Features Dataset: Factors
Mfeat-fourier [46]	76	2000	10	Multiple Features Dataset: Fourier
Mfeat-karhunen [46]	64	2000	10	Multiple Features Dataset: Karhunen
Mfeat-morph [46]	6	2000	10	Multiple Features Dataset: Morphological
Mfeat-pixel [46]	240	2000	10	Multiple Features Dataset: Pixel
Mfeat-zernike [46]	47	2000	10	Multiple Features Dataset: Zernike
Mice-protein [46]	77	1080	8	Mice Protein Expression Data Set.
Page-blocks [46]	10	5473	5	Page Blocks Classification Data Set.
Pasture [47]	21	36	3	Pasture Production dataset.
Pendigits [46]	16	10992	10	Pen-Based Recognition of Handwritten Digits dataset.
Satimage [46]	36	6430	6	Statlog (Landsat Satellite) Data Set.
Seeds [46]	7	210	3	This dataset represents the geometrical properties of kernels belonging to three different varieties of wheat.
Segment [46]	18	2310	7	Image Segmentation Data Set.
Splice [46]	60	3190	3	Molecular Biology (Splice-junction Gene Sequences) Data Set.
Squash-stored [47]	24	52	3	Squash Harvest Stored dataset.
Squash-unstored[47]	23	52	3	Squash Harvest Unstored dataset.
Synthetic-control [46]	60	600	6	Synthetic Control Chart Time Series Data Set.
Teaching-assistant[47]	5	151	3	Teaching Assistant dataset.
Vehicle [46]	18	846	4	Statlog (Vehicle Silhouettes) Data Set.
Vertebra-column [46]	6	310	3	Vertebral Column Data Set.
Vowel [46]	12	990	11	Vowel Recognition (Deterding data) dataset.
Waveform-5000 [46]	40	5000	3	Waveform Database Generator (Version 2) Data Set.
Wine [47]	13	178	3	It represents the output of a chemical analysis of wines grown in the same region in Italy but derived from three different cultivars.
Wine-quality [47]	11	4898	7	Wine Quality dataset.

4.2 Simulation experiment

For the simulation experiment, a 10-fold cross validation method is used. The training set samples are used for feature selection. JM distance for each feature and for each class-pair is calculated according to Equations (9) and (8). The proposed approach in Algorithm 2 is used to select the subset of features based on average JM values for each class pair and each feature. The selected feature subset is evaluated by its performance for supervised classification using the Naive Bayes classifier. The same training samples are used for training the classifier, and the test samples are used for measuring classification accuracy, F-measeure and AUC of the classifier of the feature subset. The classification experiment is also repeated 10 times, and average classification accuracy, F-measeure and AUC are taken as the performance measure of the selected feature subset.

In order to set the value of parameter α, a preliminary experiment is conducted with 10 datasets. The value of α is changed from 1 to 20, and in each case, the classification performance of the proposed approach on these datasets is observed. Results show that classification accuracy for most of the datasets is the highest when the α value is near 10. Therefore, the value of α is here fixed as 10.

For comparative evaluation of the proposed algorithm, feature subset selection has been done with two other available multiclass extensions of JM distance, weighted average JM distance (JM_ave) and (JM_Bh) another one equivalent to Bhattacharyya bound according to Equatios (11) and (12) respectively and the results have been compared. The performance of the proposed algorithm also has been compared with other filter based feature ranking algorithms such as IG, GR, SU, CS, One-R and Relief-F by simulation experiment. The classification accuracy, F-measure, and AUC of Naive Bayes Classifier with the selected features by those algorithms, the number of features being same as the number of features selected by the proposed multiclass JM distance based algorithm, are used for performance comparison.

For all simulation experiments, Intel(R) Core(TM) i5-4590 CPU @3.30GHz Processor and 8GB RAM with a 64 bit operating system of Windows 8.1 Pro is used. R (version 3.5.3) is used with several key standard packages such as SpatialEco, FSelector, varSel, MLmetrics, caret, and e1071 for implementation of the algorithms.

4.3 Performance, measures

4.3.1 Percentage of selected features

Since our proposed method selects a specific number of features from the full set of feature, the percentage of feature selection is important. The percentage of feature selection is calculated by $\begin{matrix} Selection Rate (%) \\ = \frac{Num . of Selected Features}{Total Feature} \times 100 \end{matrix}$ (13)

4.3.2 Classification accuracy

The proposed feature selection approach with multiclass JM distance is evaluated by the classification accuracy as a performance measure which is defined as [48]. $Accuracy (%) = \frac{TP + TN}{TP + TN + FP + FN} \times 100$ (14)

Where TP, true positive, TN, true negative, FP, false positive, and FN, false negative, represent the number of positive cases correctly detected, the number of negative cases correctly detected, the number of negative cases detected as positive and the number of positive cases detected as negative respectively.

4.3.3 Precision

Precision (also called positive predictive value) is a measure of the correctness of a positive prediction. For any classification task, the precision of a class (target value) is defined as the number of true positives divided by the total number of elements labelled as belonging to the positive class. It is calculated by using the following formula: $Precision (p) = \frac{TP}{TP + FP}$ (15)

4.3.4 Recall

Recall is the measure of how many true positives get predicted out of all the positive class elements. It is sometimes also called sensitivity. The measure is collected by the following formula: $Recall (r) = \frac{TP}{TP + FN}$ (16)

4.3.5 F-measure

F-measure combines precision and recall. It can be defined as the (weighted) harmonic mean of precision and recall by the following equation [49]: $F = 2 * \frac{Precision * Recall}{Precision + Recall}$ (17)

4.3.6 Area under the ROC curve (AUC)

The area under a receiver operating characteristic (ROC) curve, or AUC, is a single scalar value that calculates the general performance of a binary classifier [50]. The range of AUC is [0.5, 1], where the minimum value indicates that the performance of the classifier is random, and the maximum value indicates that the classifier is perfect with a zero error rate. The AUC is an important measure to evaluate the overall performance of a classifier because its calculation relies on the complete ROC curve, which involves all possible classification thresholds.

4.4 Simulation results

Table 2 represents the selected feature subset with our proposed approach (JM_mc) for 37 multiclass datasets. For different datasets, the percentage of selected features is different. For the ‘Segment’ dataset, the percentage of feature selection is about 11.11%, and it is the lowest among the 37 datasets. For the ‘Pendigits’ dataset, the percentage is the second lowest and is about 12.50%. Among the 37 datasets, 15 datasets have feature selection rate lower than 50%, and for 13 datasets, the rate is between 50-70% and rest of the 9 datasets, the feature selection rate is more than 70%. Table 2 also highlights the feature selection time in seconds for 37 datasets. For all these datasets, time is very short, and the range is about 0.050 to 0.110 seconds.

Table 2
Feature Selection with Proposed Approach (JM_mc)

Dataset Total Feature (TF) Selected Feature (SF) Selection rate (%) Selection time (second)

Analcat-authorship 70 24 34.29 0.0625

Analcat-marketing 32 25 78.13 0.0625

Breast-tissue 9 5 55.56 0.0625

Bridges 11 4 36.36 0.0781

Cars 7 3 42.86 0.0547

Cmc 9 6 66.67 0.0621

Dermatology 34 24 70.59 0.0625

Dna 180 32 17.78 0.0703

Eye-movements 27 13 48.15 0.0737

Gas-drift 128 85 66.41 0.1017

Har 561 318 56.68 0.1094

Indian-pines 220 174 79.1 0.1016

Iris 4 2 50 0.0516

Mfeat-factors 216 183 84.72 0.0607

Mfeat-fourier 76 51 67.11 0.0704

Mfeat-karhunen 64 45 70.31 0.0625

Mfeat-morph. 6 4 66.67 0.0522

Mfeat-pixel 240 185 77.08 0.1074

Mfeat-zernike 47 43 91.49 0.0703

Mice-protein 77 56 72.72 0.0781

Mage-blocks 10 7 70 0.0625

Pasture 21 6 28.57 0.0611

Pendigits 16 2 12.5 0.0748

Satimage 36 20 55.56 0.0703

Seeds 7 3 42.86 0.0586

Segment 18 2 11.11 0.0547

Splice 60 11 18.33 0.0703

Squash-stored 24 6 25 0.0625

Squash-unstored 23 7 30.43 0.0647

Synthetic-control 60 33 55 0.0765

Teaching-assistant 5 4 80 0.0671

Vehicle 18 12 66.67 0.0681

Vertebra-column 6 3 50 0.0659

Vowel 12 7 58.33 0.0566

Waveform-5000 40 15 37.5 0.073

Wine 13 3 23.08 0.0703

Wine-quality 11 4 36.37 0.0622

Dataset	Total Feature (TF)	Selected Feature (SF)	Selection rate (%)	Selection time (second)
Analcat-authorship	70	24	34.29	0.0625
Analcat-marketing	32	25	78.13	0.0625
Breast-tissue	9	5	55.56	0.0625
Bridges	11	4	36.36	0.0781
Cars	7	3	42.86	0.0547
Cmc	9	6	66.67	0.0621
Dermatology	34	24	70.59	0.0625
Dna	180	32	17.78	0.0703
Eye-movements	27	13	48.15	0.0737
Gas-drift	128	85	66.41	0.1017
Har	561	318	56.68	0.1094
Indian-pines	220	174	79.1	0.1016
Iris	4	2	50	0.0516
Mfeat-factors	216	183	84.72	0.0607
Mfeat-fourier	76	51	67.11	0.0704
Mfeat-karhunen	64	45	70.31	0.0625
Mfeat-morph.	6	4	66.67	0.0522
Mfeat-pixel	240	185	77.08	0.1074
Mfeat-zernike	47	43	91.49	0.0703
Mice-protein	77	56	72.72	0.0781
Mage-blocks	10	7	70	0.0625
Pasture	21	6	28.57	0.0611
Pendigits	16	2	12.5	0.0748
Satimage	36	20	55.56	0.0703
Seeds	7	3	42.86	0.0586
Segment	18	2	11.11	0.0547
Splice	60	11	18.33	0.0703
Squash-stored	24	6	25	0.0625
Squash-unstored	23	7	30.43	0.0647
Synthetic-control	60	33	55	0.0765
Teaching-assistant	5	4	80	0.0671
Vehicle	18	12	66.67	0.0681
Vertebra-column	6	3	50	0.0659
Vowel	12	7	58.33	0.0566
Waveform-5000	40	15	37.5	0.073
Wine	13	3	23.08	0.0703
Wine-quality	11	4	36.37	0.0622

Figure 2 shows the average classification accuracy (Avg) of 37 multiclass datasets using various JM distance measures. From the figure, it is clearly expressed that the classification accuracy of JM distance with our proposed approach (JM_mc) is very much comparable to the other two multiclass JM distance measures. For some datasets, our approach performs much better than the other two measures such as JM_ave and JM_Bh. For almost 22 datasets over 37, classification accuracy of JM_mc is higher than JM_ave and JM_Bh.

Fig. 2

Classification accuracy using various JM measures for all datasets.

Figure 3 represents the comparison of classification accuracy on the average of all datasets with nine different measures of ranking based filter approaches. This figure depicts that JM distance with our proposed approach (JM_mc) produced the average highest classification accuracy of 73.02% for all the data sets compared to other methods. The weighted average JM distance JM_ave and the JM distance equivalent to Bhattacharyya bound JM_Bh have classification accuracy 70.53% and 70.55% respectively. Among other feature ranking methods excluding our approach, Relief-F produced the highest value of 71.59%.

Fig. 3

Classification performance over all datasets with different methods.

Table 3 shows the detailed comparison of classification accuracy with the selected feature subset using the proposed feature selection technique with other methods for multiclass datasets. Here, average classification accuracy (Avg) and standard deviation (SD) are calculated for ten iterations. The highest average value and the lowest SD value are represented in boldface in the table. From this table, it is highlighted that proposed JM_mc can achieve the highest classification accuracy for 15 data sets; on the other hand, JM_ave and JM_Bh have the highest accuracy for four datasets and three datasets respectively. Other feature ranking measures such as IG, GR, SU and CS show the highest accuracy for three datasets, five datasets, four datasets and three datasets respectively. One-R has the highest accuracy for four datasets, and Relief-F shows the highest accuracy for five datasets. In the case of four datasets such as Iris, Pendigits, teaching-assistant and waveform-5000, we got the same average classification accuracy as well as standard deviation (SD) values using all the methods. In the case of SD values, JM_mc provides the lowest SD for 12 datasets compared to other approaches. CS possesses very poor SD of classification accuracy, and for only two data sets, the SD value is minimum. The rest of the measures, such as JM_ave, JM_Bh, IG, GR, SU, One-R and Relief-F have the lowest SD values for three datasets, three datasets, seven datasets, five datasets, ten datasets, four datasets and four datasets respectively. From this result, we can infer that our proposed approach JM_mc produces a stable output compared to others.

Table 3

Classification accuracy of proposed approach and other methods

Dataset	JM _mc	JM _ave	JM _Bh	IG	GR	SU	CS	One-R	Relief-F
Analcat-authorship	Avg	0.981	0.9786	0.9798	0.9738	0.9786	0.9786	0.9774	0.9822	0.9798
	SD	0.0143	0.0185	0.0149	0.0193	0.0185	0.0193	0.0173	0.0151	0.0186
Analcat-marketing	Avg	0.5194	0.5224	0.5168	0.5193	0.5193	0.5193	0.5193	0.5193	0.5167
	SD	0.0679	0.076	0.0778	0.0851	0.0851	0.0851	0.0851	0.0851	0.0776
Breast-tissue	Avg	0.4382	0.4382	0.4382	0.4336	0.4336	0.4336	0.4336	0.4336	0.3945
	SD	0.1666	0.1666	0.1666	0.1742	0.1742	0.1742	0.1742	0.1742	0.132
Bridges	Avg	0.3873	0.3691	0.3873	0.3518	0.3518	0.3518	0.3518	0.3518	0.3864
	SD	0.1462	0.1482	0.1462	0.1291	0.1291	0.1291	0.1291	0.1291	0.1773
Cars	Avg	0.7113	0.6473	0.6473	0.6473	0.6473	0.6473	0.645	0.6573	0.6473
	SD	0.0676	0.0623	0.0623	0.0623	0.0623	0.0623	0.067	0.0615	0.0623
Cmc	Avg	0.4867	0.4582	0.4562	0.4806	0.4806	0.4806	0.4806	0.486	0.4806
	SD	0.0448	0.0425	0.0333	0.0447	0.0447	0.0447	0.0447	0.0451	0.0447
Dermatology	Avg	0.8265	0.8139	0.8165	0.8358	0.8332	0.8358	0.7812	0.8219	0.8357
	SD	0.0576	0.0553	0.0576	0.0439	0.0496	0.0439	0.0661	0.0584	0.0511
Dna	Avg	0.9284	0.9316	0.9316	0.9269	0.9284	0.9269	0.9272	0.8544	0.915
	SD	0.0194	0.0182	0.0182	0.0166	0.0178	0.0166	0.0172	0.023	0.0223
Eye-movements	Avg	0.4359	0.4252	0.4252	0.4344	0.433	0.4344	0.4356	0.4397	0.4487
	SD	0.0119	0.012	0.012	0.0126	0.0112	0.0126	0.0128	0.0103	0.0084
Gas-drift	Avg	0.5835	0.6006	0.5967	0.5892	0.5996	0.5947	0.5905	0.5861	0.5608
	SD	0.0338	0.0287	0.0319	0.0293	0.0273	0.028	0.0315	0.0352	0.0299
Har	Avg	0.7739	0.5091	0.5199	0.5286	0.5004	0.504	0.6899	0.7077	0.8461
	SD	0.0112	0.0087	0.0067	0.0076	0.0096	0.0097	0.0112	0.0073	0.0097
Indian-pines	Avg	0.6553	0.6425	0.6428	0.6422	0.6424	0.6426	0.6421	0.6422	0.647
	SD	0.009	0.0096	0.0103	0.01	0.0096	0.0096	0.0095	0.01	0.0123
Iris	Avg	0.96	0.96	0.96	0.96	0.96	0.96	0.96	0.96	0.96
	SD	0.0466	0.0466	0.0466	0.0466	0.0466	0.0466	0.0466	0.0466	0.0466
Mfeat-factors	Avg	0.9285	0.924	0.9235	0.9225	0.9195	0.9215	0.922	0.9285	0.9265
	SD	0.018	0.0223	0.0222	0.0216	0.0205	0.021	0.0232	0.0204	0.0197
Mfeat-fourier	Avg	0.7765	0.776	0.7785	0.7785	0.773	0.7795	0.779	0.7805	0.7745
	SD	0.0183	0.0213	0.0221	0.0211	0.0187	0.0186	0.0211	0.0222	0.0234
Mfeat-karhunen	Avg	0.9385	0.9415	0.9415	0.9405	0.941	0.9405	0.943	0.9385	0.939
	SD	0.0187	0.0176	0.0176	0.0188	0.0166	0.0188	0.016	0.0194	0.022
Mfeat-morph.	Avg	0.5955	0.542	0.542	0.6075	0.542	0.6075	0.542	0.589	0.542
	SD	0.0194	0.0346	0.0346	0.0237	0.0346	0.0237	0.0346	0.0208	0.0346
Mfeat-pixel	Avg	0.928	0.923	0.9215	0.9255	0.926	0.9255	0.923	0.9275	0.9245
	SD	0.0218	0.0242	0.0257	0.024	0.0218	0.0249	0.0242	0.0187	0.0183
Mfeat -zernike	Avg	0.7395	0.7355	0.7365	0.7395	0.7365	0.7395	0.7365	0.739	0.7405
	SD	0.0296	0.0272	0.0242	0.0237	0.0242	0.0237	0.0242	0.0244	0.0273
Mice-protein	Avg	0.7917	0.788	0.7843	0.7917	0.7852	0.7917	0.787	0.7806	0.813
	SD	0.0236	0.0192	0.0227	0.0192	0.0234	0.0192	0.0195	0.0192	0.0194
Page-blocks	Avg	0.9305	0.9326	0.9326	0.9432	0.9355	0.9355	0.8692	0.9305	0.9003
	SD	0.0069	0.008	0.008	0.0075	0.0048	0.0048	0.0292	0.0069	0.0145
Pasture	Avg	0.7919	0.8333	0.8333	0.75	0.75	0.75	0.7917	0.8333	0.875
	SD	0.1968	0.1972	0.1972	0.2838	0.2838	0.2838	0.2486	0.1972	0.1291
Pendigits	Avg	0.4196	0.4196	0.4196	0.4196	0.4196	0.4196	0.4196	0.4196	0.4196
	SD	0.0123	0.0123	0.0123	0.0123	0.0123	0.0123	0.0123	0.0123	0.0123
Satimage	Avg	0.7946	0.7862	0.7801	0.7862	0.7801	0.7801	0.7872	0.7995	0.777
	SD	0.0109	0.0096	0.0094	0.0096	0.0094	0.0094	0.011	0.0099	0.0098
Seeds	Avg	0.9048	0.8667	0.8667	0.8667	0.9048	0.9048	0.9048	0.8667	0.8667
	SD	0.0502	0.0398	0.0666	0.0666	0.0502	0.0502	0.0502	0.0666	0.0666
Segment	Avg	0.7338	0.5965	0.5658	0.5658	0.7307	0.7307	0.7307	0.5658	0.5489
	SD	0.029	0.0571	0.051	0.051	0.034	0.034	0.034	0.051	0.0479
Splice	Avg	0.9063	0.9053	0.9053	0.9053	0.906	0.9053	0.9053	0.9044	0.9044
	SD	0.0214	0.022	0.022	0.0221	0.0214	0.0221	0.0221	0.0223	0.0237
Squash-stored	Avg	0.66	0.64	0.6367	0.54	0.54	0.54	0.54	0.54	0.6533
	SD	0.1999	0.1776	0.1889	0.1999	0.1999	0.1999	0.1999	0.1999	0.1989
Squash-unstored	Avg	0.8467	0.7167	0.7583	0.8083	0.9083	0.9083	0.8083	0.8083	0.8083
	SD	0.0869	0.1906	0.1722	0.1426	0.0983	0.0983	0.1426	0.1426	0.1426
Synthetic-control	Avg	0.945	0.8667	0.8533	0.8617	0.84	0.8533	0.8317	0.865	0.8667
	SD	0.0305	0.0437	0.0443	0.0445	0.0573	0.052	0.0474	0.0481	0.0437
Teaching-assistant	Avg	0.5158	0.5158	0.5158	0.5158	0.5158	0.5158	0.5158	0.5158	0.5158
	SD	0.1241	0.1241	0.1241	0.1241	0.1241	0.1241	0.1241	0.1241	0.1241
Vehicle	Avg	0.461	0.449	0.449	0.4136	0.4419	0.4065	0.4065	0.4136	0.397
	SD	0.0312	0.0607	0.0607	0.0527	0.0512	0.0403	0.0603	0.0527	0.0547
Vertebra-column	Avg	0.8387	0.7903	0.7903	0.7645	0.7903	0.7903	0.7903	0.7645	0.8032
	SD	0.034	0.0613	0.0613	0.0664	0.0613	0.0613	0.0613	0.0664	0.0355
Vowel	Avg	0.6768	0.6768	0.6798	0.6626	0.6929	0.6626	0.6626	0.6626	0.6727
	SD	0.0347	0.0347	0.0369	0.0327	0.0354	0.0327	0.0393	0.0327	0.0416
Waveform-5000	Avg	0.8004	0.8004	0.8004	0.8004	0.8004	0.8004	0.8004	0.8004	0.8004
	SD	0.021	0.021	0.021	0.021	0.021	0.021	0.021	0.021	0.021
Wine	Avg	0.9212	0.9212	0.9212	0.9444	0.8869	0.899	0.9663	0.9444	0.9212
	SD	0.054	0.054	0.054	0.0586	0.0711	0.035	0.0391	0.0586	0.054
Wine-quality	Avg	0.4849	0.451	0.4504	0.4461	0.4504	0.4461	0.4504	0.4806	0.4806
	SD	0.0255	0.0205	0.0263	0.0203	0.0263	0.0203	0.0263	0.0255	0.0211

Table 4 shows the comparison of F-measure among all the nine methods for multiclass datasets. Results show that, for 15 out of 37 datasets, JM_mc provides the highest F-measure. For other two JM measures JM_ave and JM_Bh, each approach provides the highest F-measure value only for two datasets. The rest of the measures, such as IG, GR, SU, CS, One-R and Relief-F, have the highest F-measure values for five datasets, eight datasets, five datasets, seven datasets, nine datasets and eight datasets respectively.

Table 4

F-measure of proposed approach and other methods

Dataset	JM _mc	JM _ave	JM _Bh	IG	GR	SU	CS	One-R	Relief-F
Analcat-authorship	0.9862	0.971	0.9854	0.9704	0.9785	0.9784	0.9832	0.9822	0.9806
Analcat-marketing	0.3882	0.3716	0.3434	0.3434	0.3434	0.3434	0.3434	0.3434	0.3477
Breast-tissue	0.4137	0.4137	0.4137	0.4254	0.4254	0.4254	0.4254	0.4254	0.3724
Bridges	0.4819	0.4687	0.4819	0.4427	0.4427	0.4427	0.4427	0.4427	0.4888
Cars	0.6219	0.5525	0.5525	0.5525	0.5525	0.5525	0.5675	0.5464	0.5525
Cmc	0.4861	0.4592	0.4577	0.4781	0.4781	0.4781	0.4781	0.4805	0.4781
Dermatology	0.8416	0.84	0.8416	0.8628	0.8641	0.8628	0.8179	0.8118	0.8362
Dna	0.9213	0.925	0.925	0.9192	0.9213	0.9192	0.9198	0.8415	0.9072
Eye-movements	0.4137	0.4001	0.4001	0.4103	0.4094	0.4103	0.4103	0.418	0.4407
Gas-drift	0.5846	0.6004	0.5965	0.5846	0.5961	0.5904	0.5866	0.5814	0.554
Har	0.7666	0.483	0.4966	0.5088	0.473	0.4775	0.6731	0.6919	0.8439
Indian-pines	0.5864	0.5502	0.5538	0.5504	0.5506	0.5506	0.5502	0.5504	0.5597
Iris	0.9622	0.9622	0.9622	0.9622	0.9622	0.9622	0.9622	0.9622	0.9622
Mfeat-factors	0.9278	0.9254	0.9248	0.9239	0.921	0.923	0.9233	0.9298	0.928
Mfeat-fourier	0.7785	0.779	0.7812	0.7818	0.7761	0.783	0.7822	0.7835	0.7776
Mfeat-karhunen	0.9387	0.9429	0.9429	0.942	0.9425	0.942	0.9446	0.9397	0.9404
Mfeat-morph.	0.57	0.5406	0.5406	0.5774	0.5406	0.5774	0.5406	0.5948	0.5406
Mfeat-pixel	0.9274	0.9253	0.924	0.9282	0.9285	0.9281	0.9257	0.9298	0.927
Mfeat-zernike	0.741	0.7373	0.7384	0.7407	0.7384	0.7407	0.7384	0.7402	0.7389
Mice-protein	0.7929	0.7862	0.7826	0.7919	0.7836	0.7919	0.7853	0.7779	0.817
Page-blocks	0.6627	0.6529	0.6529	0.6909	0.6789	0.6789	0.5826	0.6627	0.6254
Pasture	0.7726	0.7438	0.7475	0.7924	0.7924	0.7924	0.7489	0.7823	0.8331
Pendigits	0.339	0.339	0.339	0.339	0.339	0.339	0.339	0.339	0.339
Satimage	0.7766	0.7667	0.7606	0.7667	0.7606	0.7606	0.7686	0.7808	0.7577
Seeds	0.9076	0.8701	0.8701	0.8701	0.9076	0.9076	0.9076	0.8701	0.8701
Segment	0.7463	0.5751	0.5695	0.5695	0.7048	0.7048	0.7048	0.5695	0.542
Splice	0.8981	0.897	0.897	0.8964	0.8976	0.8964	0.8964	0.8955	0.8959
Squash-stored	0.7287	0.6923	0.7033	0.5203	0.5203	0.5203	0.5203	0.5203	0.7105
Squash-unstored	0.8015	0.6307	0.5646	0.7724	0.8251	0.8035	0.758	0.7724	0.7123
Synthetic-control	0.946	0.8692	0.8555	0.8633	0.8406	0.8555	0.8344	0.8678	0.8692
Teaching-assistant	0.5098	0.5083	0.5098	0.5083	0.5083	0.5083	0.5083	0.5083	0.5083
Vehicle	0.4436	0.4159	0.4159	0.368	0.4018	0.3548	0.356	0.368	0.3469
Vertebra-column	0.7971	0.7424	0.7424	0.7087	0.7424	0.7424	0.7424	0.7087	0.7379
Vowel	0.6777	0.6777	0.6803	0.6621	0.6911	0.6621	0.6629	0.6621	0.6753
Waveform-5000	0.7893	0.7893	0.7893	0.7893	0.7893	0.7893	0.7893	0.7893	0.7893
Wine	0.9266	0.9266	0.9266	0.9494	0.8985	0.9118	0.9694	0.9494	0.9266
Wine-quality	0.2676	0.2076	0.2261	0.2193	0.2261	0.2193	0.2261	0.2675	0.2353

Table 5 represents the comparison among the nine methods in terms of AUC. Again, it is observed that JM_mc produces the highest AUC value for 15 in 37 datasets. For seven datasets, both JM_ave and JM_Bh outperform other approaches regarding AUC value. In addition, each of SU, CS, and One-R exhibits the highest AUC results for eight datasets, whereas IG, GR, and Relief-F show the highest AUC values for five datasets, 12 datasets, and six datasets respectively.

Table 5

AUC of proposed approach and other methods

Dataset	JM _mc	JM _ave	JM _Bh	IG	GR	SU	CS	One-R	Relief-F
Analcat-authorship	0.9892	0.9848	0.989	0.9822	0.9887	0.9869	0.9857	0.9866	0.987
Analcat-marketing	0.7303	0.7424	0.7318	0.7154	0.7154	0.7154	0.7154	0.7154	0.7136
Breast-tissue	0.8261	0.8261	0.8261	0.8525	0.8525	0.8525	0.8525	0.8525	0.8425
Bridges	0.8422	0.8155	0.8422	0.8162	0.8162	0.8162	0.8162	0.8162	0.8022
Cars	0.7621	0.7105	0.7105	0.7105	0.7105	0.7105	0.7179	0.7063	0.7105
Cmc	0.6463	0.6347	0.6325	0.6443	0.6443	0.6443	0.6443	0.6425	0.6443
Dermatology	0.9185	0.9366	0.9185	0.9359	0.9411	0.9359	0.925	0.8936	0.9153
Dna	0.9381	0.9418	0.9418	0.9341	0.9377	0.9341	0.9354	0.8651	0.9276
Eye-movements	0.6324	0.6301	0.6301	0.6279	0.6303	0.6279	0.6316	0.6316	0.5736
Gas-drift	0.813	0.8206	0.8184	0.8138	0.82	0.8165	0.8154	0.8152	0.8009
Har	0.9143	0.8426	0.8445	0.8506	0.8391	0.8408	0.8683	0.8784	0.9388
Indian-pines	0.8592	0.8569	0.8573	0.8569	0.8569	0.8569	0.8571	0.8569	0.8546
Iris	0.98	0.98	0.98	0.98	0.98	0.98	0.98	0.98	0.98
Mfeat-factors	0.9462	0.9466	0.9466	0.9457	0.9422	0.945	0.9432	0.9493	0.9457
Mfeat-fourier	0.8465	0.8439	0.8465	0.8481	0.8463	0.8507	0.8494	0.8514	0.8449
Mfeat-karhunen	0.9452	0.9489	0.9489	0.9475	0.9474	0.9475	0.9505	0.946	0.9452
Mfeat-morph.	0.8228	0.8595	0.8595	0.8387	0.8595	0.8387	0.8595	0.8454	0.8595
Mfeat-pixel	0.9358	0.9333	0.9325	0.9358	0.935	0.9359	0.9326	0.938	0.9365
Mfeat-zernike	0.8719	0.872	0.8689	0.8716	0.8689	0.8716	0.8689	0.871	0.8734
Mice-protein	0.8347	0.8336	0.8341	0.8391	0.835	0.8391	0.8356	0.8321	0.8495
Page-blocks	0.7808	0.7829	0.7829	0.8166	0.8244	0.8244	0.7835	0.7808	0.8021
Pasture	0.8889	0.9028	0.9167	0.9306	0.9306	0.9306	0.9028	0.8889	0.8472
Pendigits	0.6978	0.6978	0.6978	0.6978	0.6978	0.6978	0.6978	0.6978	0.6978
Satimage	0.8432	0.8292	0.8226	0.8292	0.8226	0.8226	0.8302	0.8458	0.8224
Seeds	0.9194	0.8701	0.8701	0.8701	0.9194	0.9194	0.9194	0.8701	0.8701
Segment	0.8265	0.7216	0.7093	0.7093	0.8231	0.8231	0.8231	0.7093	0.6742
Splice	0.9405	0.9395	0.9395	0.9399	0.9405	0.9399	0.9399	0.9398	0.9388
Squash-stored	0.8083	0.8028	0.7778	0.8347	0.8347	0.8347	0.8347	0.8347	0.7903
Squash-unstored	0.8854	0.6771	0.8438	0.8438	0.9271	0.9271	0.8438	0.8438	0.8021
Synthetic-control	0.979	0.9448	0.9414	0.9452	0.94	0.9421	0.935	0.9452	0.9448
Teaching-assistant	0.6429	0.6612	0.6429	0.6612	0.6612	0.6612	0.6612	0.6612	0.6612
Vehicle	0.6667	0.6991	0.6991	0.6988	0.6986	0.6929	0.6896	0.6988	0.6774
Vertebra-column	0.7397	0.6728	0.6728	0.6492	0.6728	0.6728	0.6728	0.6492	0.6934
Vowel	0.8257	0.8257	0.8231	0.8322	0.8447	0.8322	0.8284	0.8322	0.8175
Waveform-5000	0.8209	0.8209	0.8209	0.8209	0.8209	0.8209	0.8209	0.8209	0.8209
Wine	0.9302	0.9302	0.9302	0.9308	0.8672	0.8813	0.9583	0.9308	0.9302
Wine-quality	0.7392	0.6988	0.7015	0.7006	0.7015	0.7006	0.7015	0.6909	0.6708

Table 6 illustrates a comparison among nine methods based on the average execution time (in seconds) for all the datasets. The execution times of three different JM distance measures are comparable and slightly higher than other methods except for Relief-F. In this table, the lowest execution time is highlighted in boldface. For 17 datasets, CS method shows the lowest execution time. JM_mc, have the lowest execution time for three datasets, and the other two JM distance measures, JM_ave and JM_Bh have the lowest execution time for nine datasets and six datasets respectively. One-R method takes the lowest time for 3 data sets, and Relief-F needs the highest time, which is much more than others. Relief-F’s execution time is approximately 212 times greater or more than other methods.

Table 6

Average execution time (seconds)

Dataset	JM _mc	JM _ave	JM _Bh	IG	GR	SU	CS	One-R	Relief-F
Analcat-authorship	0.3848	0.3855	0.3886	0.3768	0.3671	0.3737	0.3308	0.3512	78.5809
Analcat-marketing	0.3032	0.2948	0.291	0.1695	0.1711	0.1661	0.1585	0.1851	19.4746
Breast-tissue	0.1340	0.1340	0.1558	0.0684	0.0781	0.0669	0.0629	0.0706	2.5687
Bridges	0.1593	0.1608	0.1593	0.1048	0.0678	0.071	0.0551	0.0616	3.0848
Cars	0.0421	0.0405	0.0436	0.0557	0.05	0.063	0.0501	0.0493	4.2216
Cmc	0.0763	0.0798	0.0783	0.1083	0.1081	0.103	0.0964	0.0990	16.4370
Dermatology	0.4671	0.4643	0.4648	0.4966	0.2019	0.1952	0.1674	0.226	22.2093
Dna	0.7068	0.720	0.7296	1.6520	1.6399	1.6526	1.4597	1.4746	657.5220
Eye-movements	0.7304	0.7237	0.7450	1.3826	1.3465	1.2486	1.2457	1.2543	317.4830
Gas-drift	6.2794	5.6701	5.6232	17.1375	17.0589	16.8604	15.8028	18.0575	1820.112078
Har	20.2938	20.1048	19.9776	33.9376	33.9989	32.0043	30.3931	29.0529	8216.780
Indian-pines	10.6209	10.3521	10.3141	10.8417	10.1151	10.1829	9.8049	10.4443	2476.826
Iris	0.0276	0.0213	0.026	0.0374	0.0337	0.0328	0.0291	0.0285	1.3149
Mfeat-factors	8.8999	9.1327	9.0809	2.992	2.9463	2.9463	2.8187	2.9289	960.5638
Mfeat-fourier	3.0015	3.0041	3.0515	1.1353	1.0719	1.0836	1.0022	1.0648	212.2076
Mfeat-karhunen	2.528	2.5284	2.5638	0.9327	0.9773	0.9361	0.8939	0.8893	173.6740
Mfeat-morph.	0.3698	0.3784	0.3651	0.2244	0.2174	0.2206	0.2303	0.2287	18.8695
Mfeat-pixel	10.2853	10.3834	10.2991	3.6476	3.5441	3.5571	3.3424	3.2469	1151.7150
Mfeat-zernike	2.0013	1.9880	1.9813	0.7956	0.7804	0.812	0.7811	0.7346	124.1485
Mice-protein	1.836	1.8371	1.8224	0.6889	0.6754	0.6904	0.6469	0.6493	139.1284
Page-blocks	0.2932	0.2799	0.2897	0.3569	0.3494	0.3691	0.3502	0.3491	61.2673
Pasture	0.0604	0.0604	0.0604	0.0834	0.0728	0.0815	0.0556	0.0765	3.101
Pendigits	1.1599	1.1685	1.2143	0.9925	0.9629	0.9476	0.8977	0.9076	197.3686
Satimage	1.2023	1.1983	1.1312	1.2697	1.1986	1.1876	1.1122	1.1756	278.5883
Seeds	0.0359	0.0312	0.0343	0.0484	0.0474	0.0390	0.0469	0.0462	2.4187
Segment	0.3856	0.3966	0.3838	0.2463	0.2724	0.2481	0.2316	0.2549	47.5422
Splice	0.3068	0.3123	0.3282	0.6517	0.6343	0.6256	0.573	0.5771	202.4043
Squash-stored	0.0689	0.0627	0.0674	0.0824	0.0868	0.0849	0.0684	0.0746	4.1449
Squash-unstored	0.0651	0.0612	0.0651	0.0804	0.0800	0.0808	0.0732	0.0629	3.5465
Synthetic-control	0.7500	0.7471	0.7436	0.3824	0.3669	0.3424	0.3297	0.3453	61.7121
Teaching-assistant	0.0514	0.0436	0.0436	0.0967	0.0446	0.0393	0.0342	0.0349	1.5811
Vehicle	0.1383	0.1378	0.1360	0.1769	0.1504	0.1591	0.1390	0.1361	18.5617
Vertebra-column	0.0428	0.0366	0.0366	0.0476	0.0533	0.0482	0.0417	0.0456	3.0889
Vowel	0.6452	0.6192	0.6202	0.2151	0.2157	0.2106	0.2077	0.2104	17.4169
Waveform-5000	0.3672	0.3697	0.3737	0.7156	0.7099	0.7201	0.6417	0.67	201.8161
Wine	0.0486	0.0455	0.0455	0.0800	0.0672	0.0547	0.0530	0.0474	3.6554
Wine-quality	0.3869	0.3886	0.3909	0.3275	0.3311	0.3428	0.3097	0.3464	63.9681

Table 7 represents a summary of results over all the datasets for all the methods regarding classification accuracy, F-measure, AUC and computational time. In this table, the computed average rank of the different approaches is shown. The nine methods are ranked (from the best to the worst as 1 to 9) based on the value of the evaluation metric individually for all the datasets. If multiple methods show the same effectiveness, they are given the same ranking value. This ranking process is performed for all datasets, and finally, the average rank value over all the data sets is calculated for all the methods. It is found that JM_mc achieves the highest rank among all the approaches in terms of the classification accuracy, F-measure and AUC, which are shown in boldface in the table. For execution time, the average ranking over the data set is not suitable as computational time depends on the size of the data set. It seems that the average computational time for JM_mc is the third lowest, losing to other JM measures.

Table 7

Overall comparison between the methods

Method	Classification Accuracy	F-measure	AUC	Execution Time
	Avg (%)	Rank	Avg (%)	Rank	Avg (%)	Rank	Avg (Sec)	Rank
JM _mc	73.04	2.324	70.60	2.568	83.76	3.405	2.031	4.719
JM _ave	70.53	3.973	67.40	4.703	82.26	4.405	2.010	4.378
JM _Bh	70.55	4.027	67.28	4.297	82.57	4.135	2.003	4.324
IG	70.33	4.108	67.52	4.054	82.86	3.730	2.234	5.865
GR	70.88	4.135	67.98	3.946	83.5	3.486	2.191	5.027
SU	70.98	3.838	67.91	3.946	83.43	3.514	2.131	4.784
CS	70.94	4.514	67.88	4.324	83.32	3.459	2.015	2.784
One-R	70.92	3.946	67.80	4.297	82.47	4.432	2.066	3.622
Relief-F	71.59	4.243	68.56	4.622	82.18	5.595	475.381	9.000

Table 8 represents the results of pair wise t-test with the proposed approach, JM_mc and each of the other approaches regarding classification accuracy, F-measure, AUC and execution time over all the data sets. In this t-test, p-value less than 0.05 indicates JM_mc is significantly better than other approaches. Otherwise, there is no significant difference between the approaches. The results on classification accuracy and F-measure indicate that the proposed approach performs significantly better than other approaches. Regarding AUC, proposed JM_mc shows significantly better performance for four methods, including JM_ave, JM_Bh, One-R, and Relief-F. On the other hand, there is no significant variation among the approaches regarding execution time because JM_mc shows significantly better results only for Relief-F.

Table 8

Result of Pair wise t-tests

Methods	Classification Accuracy		F-measure		AUC		Execution Time
	t-value	p-value	t-value	p-value	t-value	p-value	t-value	p-value
JM _ave	2.8064	0.008	3.2725	0.0024	2.0846	0.0443	1.0765	0.2889
JM _Bh	2.8106	0.0079	3.161	0.0032	2.4	0.0217	1.2666	0.2134
IG	3.0426	0.0044	2.9622	0.0054	1.6373	0.1103	– 0.3726	0.7116
GR	2.4326	0.0201	2.5773	0.0142	0.5347	0.5961	– 0.2946	0.77
SU	2.3565	0.024	2.6682	0.0114	0.7167	0.4782	– 0.1974	0.8447
CS	3.6508	0.0008	3.6987	0.0007	1.171	0.2493	0.0358	0.9716
One-R	3.1191	0.0036	3.345	0.0019	2.407	0.0213	– 0.0727	0.9424
Relief-F	2.0441	0.0483	2.6324	0.0124	2.6943	0.0107	– 2.0455	0.0482

5 Discussion and conclusion

Optimum feature selection from a set of large number features in real world applications with noisy data is a tricky problem. One needs to balance classification accuracy with the number of features and computational cost in addition to model independence and good stability. Ranking based filter approaches produce model independent feature selection with low computational cost compared to search based filter approaches for optimum feature subset selection, but for final subset selection, the wrapper method is used to select an appropriate number of features from top ranked features to have a practical solution. In this work, a two step feature subset selection approach for multiclass problems is proposed in which feature ranking with JM distance is used in the first step for individual class pairs while a set of heuristic rules are framed for finding out the final optimal feature subset from the ranked feature lists of all class pairs without using any explicit search technique or a wrapper classifier. The proposed algorithm also shows a new approach to use JM distance for feature selection in multiclass scenario which has been shown effective compared to other previous extensions of JM distance for multiclass problems. In fact, the elegance of the proposed approach lies in the fact that it integrates the selection of final feature subset from the ranked feature lists and the extension of the JM distance measure for multiclass problems in a unified process. This also eliminates the need of using any wrapper classifier later to find out appropriate feature subset from the top ranked features of the feature list for a particular problem.

In our previous works, we have examined the efficiency of JM distance as an effective feature selection measure compared to some other well known filter measures for binary problems in terms of classification accuracy, feature reduction and stability with simulation experiments using benchmark data sets. Here we have evaluated our proposed approach for multiclass problems with 37 benchmark data sets with regard to classifier accuracy, F-measure, AUC, execution time and percentage of feature selection compared to two different extensions of JM distance measures for multiclass problems and six different popular feature evaluation measures for rank based feature selection. It is found that the average classification accuracy, F-measure, AUC (over all the data sets) of the proposed approach JM_mc, is the highest compared to all other methods. The computational cost of all the methods except Relief-F is very comparable. Relief-F’s computational cost is almost 234 times higher than our proposed approach JM_mc. The analysis of the standard deviation of the results over 10 independent trials reveals that our proposed approach is quite stable. In our work, feature -feature interaction is not considered to restrict the computational time. The features are ranked according to their individual goodness, and all the methods selected for comparison also work in the same way. We are now working on extending this work in specific application areas such as gene expression data, where the number of features is large and computationally efficient algorithm is important. In our proposed approach, after some trial and error, we fixed the value of α at 10 for all the data sets. In the future, we would also like to investigate the role of α over the individual data set.

References

Duda

R.O.

, Hart

P.E.

and Stork

D.G.

, Pattern Classification, 2nd Ed. Wiley- Interscience (2000).

Liu

and Motoda

, Feature Selection for Knowledge Discovery and Data Mining, Springer (1998).

Wang

, Huang

, Shao

, Hu

and Chen

, Feature Selection Based on Neighborhood Self-Information, IEEE Trans Cybernetics 50(9) (2020), 4031–4042.

Gao

, Hu

, Zhang

and He

, Feature selection considering the composition of feature relevancy, Pattern Recognit Lett 112 (2018), 70–74.

Goswami

, Das

A.K.

, Chakrabarti

and Chakraborty

, A feature cluster taxonomy based feature selection technique, Expert Syst Appl 79 (2017), 76–89.

, Cheng

, Wang

, Morstatter

, Trevino

R.P.

, Tang

and Liu

, Feature Selection: A Data Perspective, ACM Comput Surv 50(6) (2018), 1–45.

Uysal

A.K.

, On Two-Stage Feature Selection Methods for Text Classification, IEEE Access 6 (2018), 43233–43251.

Zhang

, Nie

, Li

and Wei

, Feature selection with multiview data: A survey, Information Fusion 50 (2019), 158–167.

Cover

T.M.

, The Best Two Independent Measurements are Not the Two Best, IEEE Trans Syst Man Cybernet 4(1) (1974), 116–117.

10.

Liu

and Yu

, Toward Integrating Feature Selection Algorithms for Classification and Clustering, IEEE Trans Knowl Data Eng 17(4) (2005), 491–502.

11.

Cilia

N.D.

, De Stefano

, Fontanella

and Scotto di Freca

, A ranking-based feature selection approach for handwritten character recognition, Pattern Recognit Lett 121 (2019), 77–86.

12.

Serpico

S.B.

, Inca

M.D.

, Melgani

and Moser

, A comparison of feature reduction techniques for classification of hyperspectral remote-sensing data, in Proceedings of SPIE – The International Society for Optical Engineering (2003).

13.

Swain

P.H.

, Robertson

T.V.

and Wacker

A.G.

, Comparison of the Divergence and B-Distance in Feature Selection, Information Note 020871, Laboratory for Applications of Remote Sensing, Purdue University, West Lafayette, Indiana 47907, February (1971).

14.

Swain

P.H.

and King

R.C.

, Two Effective Feature Selection Criteria for Multispectral Remote Sensing, LARS Technical Reports (1973).

15.

Swain

P.H.

and Davis

S.M.

, Remote Sensing: The quantative approach, New York McGraw-Hill (1978).

16.

Bruzzone

, Roli

and Serpico

S.B.

, An Extension of the Jeffreys-Matusita Distance to Multiclass Cases for Feature Selection, IEEE Trans Geosci Remote Sens 33(6) (1995).

17.

Bruzzone

and Serpico

S.B.

, A technique for feature selection in multiclass problems, Int J Remote Sens 21(3) (2000), 549–563.

18.

Sen

, Goswami

and Chakraborty

, Jeffries-Matusita distance as a tool for feature selection, in International Conference on Data Science and Engineering (ICDSE) (2019).

19.

Sen

, Mandal

A.K.

, Goswami

and Chakraborty

, A Comparative Study of the Stability of Filter based Feature Selection Algorithms, IEEE 10th International Conference on Awareness Science and Technology (iCAST), Morioka, Japan (2019), 1–6.

20.

Roy

, Das

, Saha

, Sarkar

and Basu

, A Comparative Study of Feature Ranking Methods in Recognition of Handwritten Numerals, in, book: Artificial Intelligence and Evolutionary Algorithms in Engineering Systems 324 (2015), 473–479.

21.

Sánchez-maroño

V.B.N.

, A review of feature selection methods on synthetic data, Knowl Inf Syst (2013), 483–519.

22.

Guru

D.S.

, Suhil

, Raju

L.N.

and Kumar

N.V.

, An alternative framework for univariate filter based feature selection for text categorization, Pattern Recognit Lett 103 (2018), 23–31.

23.

Strbac

J.N.Ć.P.

and Ć

D.B.

, Toward Optimal Feature Selection using Ranking Methods and Classification Algorithms, Yugosl J Oper Res 21(1) (2011), 119–135.

24.

Estévez

P.A.

, Tesmer

, Perez

C.A.

and Zurada

J.M.

, Normalized mutual information feature selection, IEEE Trans Neural Networks 20(2) (2009), 189–201.

25.

Hall

M.A.

, Correlation-based Feature Selection for Machine Learning, Ph.D. thesis, University of Waikato, (1999).

26.

Karegowda

A.G.

, Manjunath

A.S.

and Jayaram

M.A.

, Comparative study of attribute selection using gain ratio and correlation based feature selection, Int J Infor Tech Knowl Managem 2(2) (2010), 271–277.

27.

Bommert

, Sun

, Bischl

, Rahnenführer

and Lang

, Benchmark for filter methods for feature selection in highdimensional classification data, Comput Stat Data Anal 143 (2020), 106839.

28.

Liu

, Setiono

, Science

and Ridge

, Chi2: Feature Selection and Discretization of Numeric Attributes, in Seventh International Conference on Tools with Artificial Intelligence (ICTAI), IEEE Computer Society, Washington, DC, USA (1995), 388–391.

29.

Holt

R.C.

, Very simple classification rules perform well on most commonly used datasets, Mach Learn 11 (1993), 63–91.

30.

Kenji

and Rendell

A.L.

, A practical approach to feature selection, in 9th international workshop on machine learning, San Francisco, CA, USA, (1992), 249–256.

31.

Urbanowicz

R.J.

, Meeker

, Lacava

, Olson

R.S.

and Jason

, Relief-Based Feature Selection: Introduction and Review, J Biomed Inform 85 (2018), 189–203.

32.

Thomas

I.L.

, Ching

N.P.

, Benning

V.M.

and Aguanno

J.A.D.

, A review of multi-channel indices of class separability, Int J Remote Sens 8(3) (1987), 331–350.

33.

Mahsereci

, Ay

and Turgay

İ.

, A comparative study on the effect of feature selection on classification accuracy, Procedia Technol 1 (2012), 323–327.

34.

Huang

and Liao

W.-C.

, A Comparative Study of Feature Selection Methods for Probabilistic Neural Networks in Cancer Classification, in IEEE International Conference on Tools with Artificial Intelligence (ICTAI’03) (2003), 1–7.

35.

Asir

and Gnana

, An Unsupervised Feature Selection Algorithm with Feature Ranking for Maximizing Performance of the Classifiers, Int J Autom Comput 12 (2015), 511–517.

36.

Chen

, Yuan

, Cui

, Zhang

and Ju

, Empirical studies on the impact of filter-based ranking feature selection on security vulnerability prediction, IET Software 15 (2021), 75–89.

37.

Ghazy

R.A.

, El-Rabaie

El-S.M.

, Dessouky

M.I.

, El-Fishawy

N.A.

and Abd El-Samie

F.E.

, Feature Selection Ranking and Subset-Based Techniques with Diferent Classifers for Intrusion Detection, Wireless Pers Commun 111 (2020), 375–393.

38.

Petkovi

, Kocev

and Džeroski

, Feature ranking for multi-target regression, Mach Learn 109 (2019), 1179–1204.

39.

Lee

, Choi

I.Y.

and Jun

C.-H.

, An efficient multivariate feature ranking method for gene selection in high-dimensional microarray data, Expert Syst Appl 116 (2021), 0957–4174.

40.

Padma

and Sanjeevi

, Jeffries Matusita based mixedmeasure for improved spectral matching in hyperspectral image analysis, Int J Appl Earth Obs Geoinf 32 (2014), 138–151.

41.

Dalponte

, ∅rka

H.O.

, Gobakken

, Gianelle

and Næsset

, Tree Species Classification in Boreal Forests With Hyperspectral Data, IEEE Trans Geosci Remote Sens 51(5) (2013), 2632–2645.

42.

Homem

M.R.P.

, Mascarenhas

N.D.A.

and Cruvinel

P.E.

, The linear attenuation coefficients as features of multiple energy CT image classification, Nucl Instruments Methods Phys Res 452 (2000), 351–360.

43.

Daamouche

, Melgani

, Member

and Alajlan

, Swarm Optimization of Structuring Elements for VHR Image Classification, IEEE Trans Geosci Remote Sens Lett 10 (2013), 1334–1338.

44.

Qiu

, Fan

, Zhong

, Tang

and Chen

, A new approach for crop identification with wavelet variance and JM distance, Environ Monit Assess 186(11) (2014), 7929–7940.

45.

Wang

, Qi

and Liu

, Unsupervised Segmentation Evaluation Using Area-Weighted Variance and Jeffries-Matusita Distance for Remote Sensing Images, Remote Sens 10 (2018), 1193.

46.

Dua

and Graff

, UCI Machine Learning Repository, http://archive.ics.uci.edu/ml. (2019).

47.

Vanschoren

, van Rijn

J.N.

, Bischl

and Torgo

, OpenML: Networked Science in Machine Learning, SIGKDD Explorations 15(2) (2013), 49–60. https://www.openml.org/search?type=data.

48.

Loey

, Jasim

M.W.

and El-bakry

H.M.

, Machine Learning in Gene Expression Profile for Central Nervous System Tumor Classification, J Converg Inf Technol 14(1) (2019), 49–60.

49.

Bolón-Canedo

and Alonso-Betanzos

, Ensembles for feature selection: A review and future trends, Information Fusion 52 (2019), 1–12.

50.

Mcneil

B.J.

and Hanley

J.A.

, The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve, Radiology 143 (1982), 29–36.

An effective feature subset selection approach based on Jeffries-Matusita distance for multiclass problems

Abstract

Keywords

1 Introduction

2 Rank based filter approach for feature selection

2.1 Feature evaluation measures

2.1.1 Mutual information (MI)

2.1.7 Relief-F

2.2.1 JM distance based feature selection algorithms

3 Proposed feature selection approach with JM distance for multiclass problems (JM mc )

4.1 Dataset description

4.3 Performance, measures

4.3.1 Percentage of selected features

4.4 Simulation results

References

3 Proposed feature selection approach with JM distance for multiclass problems (JM_mc)