Feature selection is an important machine learning field which can provide a key role for the challenging problem of classifying high-dimensional data. This problem is finding effective features among the set of all features in such that the final feature set can improve accuracy and reduce complexity. Since feature selection is an NP-Hard problem, many heuristic algorithms have been studied so far to solve this problem.
In this paper, we propose a novel method based on hyper-heuristic approach to find an efficient proper feature subset which is named Hyper-Heuristic Feature Selection (HHFS). In the proposed method, Low level heuristics are categorized into two groups: the first group contains exploiters which cause to exploit the search space efficiently by improving the quality of the candidate solution at hand; the second one includes explorer heuristics which explore the solution space by dwelling on random perturbations. Since each region of the solution space can have its own characteristics, an appropriate low level heuristic should be selected and applied to the current solution. We propose Genetic Algorithm to select among the set of low level heuristic and balance between exploitation and exploration. It chooses the low level heuristic based on the existing functional history of low level heuristic. We aim to investigate the role of cooperation between low level heuristics within a hyper-heuristic framework to find the best feature subset. Since different low level heuristics have different strengths and weaknesses, we believe that cooperation can allow the strengths of one low level heuristic to compensate for the weaknesses of another. In this study, we also propose Adaptive Hyper-Heuristic Feature Selection (AHHFS) which is an extension of HHFS. Empirical study of the proposed method on several commonly used data sets from UCI repository indicates that it outperforms recent methods in the literature for feature selection.
BahrampourA. and MontazeriM., Prediction of breast cancer mortality by hidden markov model, School of Health, Kerman University of Medical Sciences, Kerman, Iran, 2015.
2.
MontazeriM. and MontazeriM., Machine learning models for predicting the diagnosis of liver disease, Koomesh16(1) (2014), 53-59.
YuL. and LiuH., Efficient feature selection via analysis of relevance and redundancy, Journal of Machine Learning Research (2004), 1205-1224.
5.
ReifM. and ShafaitF., Efficient feature size reduction via predictive forward selection, Pattern Recognition47(4) (2014), 1664-1673.
6.
LiuH., MotodaH. and YuL., Feature selection with selective sampling, in: Proceedings of the Nineteenth International Conference on Machine Learning, (2002).
7.
Robnik-SikonjaM. and KononenkoI., Theoretical and empirical analysis of relief and reliefF, Machine Learning53 (2003).
8.
KimY., StreetW. and MenczerF., Feature selection for unsupervised learning via evolutionary search, in: Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2000.
9.
DashM. et al., Feature selection for clustering - A filter solution, in: Proceedings of the Second International Conference on Data Mining, 2002.
10.
HastieT., TibshiraniR. and FriedmanJ., The elements of statistical learning, Springer, 2001.
JefferyI.B., HigginsD.G. and CulhaneA.C., Comparison and evaluation of methods for generating differentially expressed gene lists from microarray data, BMC Bioinformatics7(1) (2006), 359.
13.
MontazeriM., BaghshahM.S. and EnhesariA., Hyper-heuristic algorithm for finding efficient features in diagnose of lung cancer disease, J Basic Appl Sci Res3(10) (2013), 134-140.
14.
MontazeriM. et al., Identifying efficient clinical parameters in diagnose of liver disease, HealthMED (2014), 1115.
15.
TusherV.G., TibshiraniR. and ChuG., Significance analysis of microarrays applied to the ionizing radiation response, Proceedings of the National Academy of Sciences98(9) (2001), 5116-5121.
16.
BreitlingR. et al., Rank products: A simple, yet powerful, new method to detect differentially regulated genes in replicated microarray experiments, FEBS Letters573(1) (2004), 83-92.
17.
MoustakidisS.P. and TheocharisJ.B., SVM-FuzCoC: A novel SVM-based feature selection method using a fuzzy complementary criterion, Pattern Recognition43 (2010), 3712-3729.
18.
GuyonI. et al., Feature extraction foundations and applications, Springer, 2006.
19.
MaoQ. and TsangI.W.-H., A feature selection method for multivariate performance measures, Pattern Analysis and Machine Intelligence, IEEE Transactions on35(9) (2013), 2051-2063.
20.
BellmanR. et al., Adaptive control processes: A guided tour, Princeton University Press Princeton, 4 (1961).
21.
PechenizkiyM., PuuronenS. and TsymbalA., Feature extraction for classification in knowledge discovery systems, in: Knowledge-Based Intelligent Information and Engineering Systems, Springer, 2003.
22.
TheodoridisS. and KoutroumbasK., Chapter 5 - Feature selection, in: Pattern Recognition (Third Edition), Academic Press: San Diego, (2006), 213-262.
23.
DashM. and LiuH., Feature selection for classification, Intelligent Data Analysis1(1) (1997), 131-156.
24.
LiuH. et al., Feature selection with dynamic mutual information, Pattern Recognition42(7) (2009), 1330-1339.
25.
DoakJ., An evaluation of feature selection methods and their application to computer security, University of California, Computer Science, 1992.
26.
LiuH. and MotodaH., Less is more, in: Feature Extraction, Construction and Selection, Springer, 1998, 3-12.
27.
VinhN.X. and BaileyJ., Comments on supervised feature selection by clustering using conditional mutual information-based distances, Pattern Recognition46(4) (2013), 1220-1225.
28.
BenabdeslemK. and HindawiM., Efficient semi-supervised feature selection: Constraint, relevance and redundancy, Knowledge and Data Engineering, IEEE Transactions on26(5) (2014), 1131-1143.
29.
PramokchonP. and Piamsa-ngaP., An unsupervised, fast correlation-based filter for feature selection for data clustering, in: Proceedings of the First International Conference on Advanced Data and Information Engineering (DaEng-2013), Springer, 2014.
30.
BermejoP., GámezJ.A. and PuertaJ.M., Speeding up incremental wrapper feature subset selection with naive bayes classifier, Knowledge-Based Systems55 (2014), 140-147.
31.
RodriguesD. et al., A wrapper approach for feature selection based on bat algorithm and optimum-path forest, Expert Systems with Applications41(5) (2014), 2250-2258.
32.
ZhuZ., OngY.-S. and ZuradaJ.M., Identification of full and partial class relevant genes, Computational Biology and Bioinformatics, IEEE/ACM Transactions on7(2) (2010), 263-277.
33.
MontazeriM., NajiH.R. and MontazeriM., Memetic feature selection algorithm based on efficient filter local search, Journal of Basic and Applied Scientific Research3(10) (2013), 126-133.
34.
O'BrienR., Ant algorithm hyperheuristic approaches for scheduling problems, University of Nottingham, 2008.
35.
WolpertD.H. and MacreadyW.G., No free lunch theorems for optimization, Evolutionary Computation, IEEE Transactions on1(1) (1997), 67-82.
36.
DuanQ., LiaoT.W. and YiH., A comparative study of different local search application strategies in hybrid metaheuristics, Applied Soft Computing13(3) (2013), 1464-1477.
37.
XinB. et al., Hybridizing differential evolution and particle swarm optimization to design powerful optimizers: A review and taxonomy, Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on42(5) (2012), 744-767.
38.
MoscatoP., On evolution, search, optimization, genetic algorithms and martial arts: Towards memetic algorithms, Caltech Concurrent Computation Program, C3P Report826 (1989), 1989.
39.
MontazeriM., NajiH.R. and MontazeriM., eds, A novel memetic feature selection algorithm, IKT 2013-2013 5th Conference on Information and Knowledge Technology, 2013.
40.
OngY.S. and KeaneA.J., Meta-Lamarckian learning in memetic algorithms, Evolutionary Computation, IEEE Transactions on8(2) (2004), 99-110.
41.
CowlingP., KendallG. and SoubeigaE., A hyperheuristic approach to scheduling a sales summit, in: Practice and Theory of Automated Timetabling III, Springer, 2001, 176-190.
42.
MitraM. et al., Cooperating of local searches based hyperheuristic approach for solving traveling salesman problem, in: IJCCI (ECTA-FCTA), 2011.
43.
MontazeriM., Nezamabadi-pourH. and BahrololoumA., Exploring and exploiting effectively based hyper-heuristic approach for solving travelling salesman problem, in: The Fifth Iran Data Mining Conference (IDMC), 2011 5th Conference on, Amirkabir University of Technology, Tehran, Iran, 2011.
44.
MontazeiM., BaghshahM.S. and NiknafsA., Selecting efficient features via a hyper-heuristic approach, in: The Fifth Iran Data Mining Conference (IDMC), 2011 5th Conference on, Amirkabir University of Technology, Tehran, Iran, 2011.
45.
NewmanD.J. et al., UCI Repository of machine learning databases, 1998.
46.
GuyonI. et al., Feature Extraction: Foundations and Applications, Springer, 207 (2008).
47.
PressW.H. et al., Numerical Recipes in C: The Art of Scientific Computing, Cambridge University Press, Cambridge, MA, 131 (1992), 243-262.
48.
GhiselliE.E., Theory of psychological measurement, McGraw-Hill New York, 13 (1964).
49.
HogarthR.M., Methods for aggregating opinions, Springer, 1977.
50.
HamiltonN.Z., Correlation-based feature subset selection for machine learning, 1998.
51.
AmaldiE. and KannV., On the approximability of minimizing nonzero variables or unsatisfied relations in linear systems, Theoretical Computer Science209(1) (1998), 237-260.
52.
FlachP., The many faces of ROC analysis in machine learning, ICML Tutorial, 2004.
53.
WangC.-H. et al., Automatically integrating multiple rule sets in a distributed-knowledge environment, Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on28(3) (1998), 471-476.
54.
DavisL., Handbook of genetic algorithms, 1991.
55.
MitchellM. and ForrestS., Fitness landscapes: Royal road functions, Handbook of Evolutionary Computation, 1997.
56.
DavisL., Bit Climbing, Representational Bias, and Test Suite Design, in: 4th International Conference on Genetic Algorithms, 1991.
57.
MitchellM. and ForrestS., B.2.7.5: Fitness landscapes: Royal road functions, Handbook of Evolutionary Computation, 1997.
58.
McLachlanG., DoK.-A. and AmbroiseC., Analyzing microarray gene expression data, John Wiley and Sons, 422 (2005).
59.
TahirM.A. and SmithJ., Creating diverse nearest-neighbour ensembles using simultaneous metaheuristic feature selection, Pattern Recognition Letters31(11) (2010), 1470-1480.
60.
SunD. and ZhangD., Bagging constraint score for feature selection with pairwise constraints, Pattern Recognition43(6) (2010), 2106-2118.
61.
SunX. et al., Feature evaluation and selection with cooperative game theory, Pattern Recognition45(8) (2012), 2992-3002.
62.
MadadizadehF., BahrampourA., MousaviS.M. and MontazeriM., Using Advanced Statistical Models to Predict the Non-Communica ble Diseases, Iranian Journal of Public Health44(12) (2015), 1714-1715.
63.
MontazeriM., MontazeriM., MontazeriM. and BeigzadehA., Machine learning models in breast cancer survival prediction, Technology and Health Care24(1) (2016), 31-42. doi:10.3233/THC-151071.
64.
MontazeriM., MontazeriM. and SaryazdiS., Eye detection in digital images: challenges and solutions. 2th National Conference of Electrical Engineering (NEEC2011); Esfehan, Iran2011.
65.
MontazeriM. and Nezamabadi-pourH., editors, Automatic extraction of eye field from a gray intensity image using intensity filtering and hybrid projection function. Communications, Computing and Control Applications (CCCA), 2011 International Conference on; 2011: IEEE.
66.
MontazeriM., Nezamabadi-pourH. and MontazeriM., Automatically Eye Detection with Different Gray Intensity Image Conditions, Computer Technology and Application3(8) (2012).
