A new variable selection method applied to credit scoring

Abstract

Credit scoring (CS) is an important process in both banking and finance. Lenders or creditors have to use CS to predict the probability that a borrower will default or become delinquent. CS is usually based on variables related to the applicant such as: his age, his historical payments, his behavior, etc. This paper first proposes a new method for variable selection. The proposed method (VS-VNS) is based on the variable neighborhood search meta-heuristic. VS-VNS allows us to select a set of significant variables for the data classification task. The VS-VNS is combined then with a Bayesian network (BN) to build models for CS and select counterparties. Further, six search methods are studied for BN on different sets of variables. The different techniques and combinations are evaluated on some well-known financial datasets. The numerical results are promising and show the benefits of the new proposed approach (VS-VNS) for data classification and credit scoring.

Keywords

Credit scoring variable selection variable neighborhood search search technique Bayesian network Hill climbing tabu search simulated annealing TAN classification

1 Introduction

Variable selection called also attribute selection or feature selection is the operation that permits to select a subset of relevant or significant variables to be used in the data classification task. Variable selection is a pre-processing step before launching the classification task. In this work, we are interested in variable selection and classification for credit scoring (CS).

As shown by Mester (1997), CS is a crucial problem in financial institutions and banks. In order to select counterparties, the financial institutions have to use good techniques to distinguish between "bad" and "good" counterparties and decide if the credit will be granted or not. For example, in banks, lenders or creditors have to use CS to predict the probability that a borrower will default or become delinquent. CS is based generally on some variables related to applicants to evaluate their creditworthiness. These variables can be: the age of applicant, his historic, payments, guarantees, default rates and so on.

Various studies in finance and banking have shown the importance of CS. For instance, the credit scores are one of the most powerful predictors of risk as shown by Miller (2003). It helps in providing an objective analysis of the applicant’s creditworthiness which reduces discrimination and credit risk. CS can be used in the decision making whether grant credit to applicant or not. Further, CS can be used in corporate and collection scorecards.

To handle CS, researchers have used several techniques. Among them, we give the following ones: Abdou (2009) proposed a genetic programming technique for CS. Desay et al. (1996) gave a comparison of neural networks and linear scoring models in the credit union environment. Henley and Hand (1996) used a k-Nearest Neighbor (k-NN) classifier for assessing consumer credit risk. An interesting support vector machines (SVM) was proposed by Bellotti and Crook (2009) for credit scoring and discovery of significant features. Also, Sousaa et al. (2015) proposed a dynamic modeling framework for credit risk assessment.

Statistical methods are also studied for CS. We give as examples: the linear regression proposed by Hand and Henley (1997), the decision trees proposed by Quinlan (1987) and the classification and regression trees (CART) studied by Breiman, et al. (1984). Wiginton (1980) proposed the discriminant analysis and logistic regression based methods which are one of the most broadly established statistical techniques used to classify clients as “good” or “bad”. Also, Friedman et al. (1997) proposed Bayesian networks (BN) that may be used to build models for CS.

In this work, we study the impact of variable selection and search method on CS when combined with BN. This paper makes two main contributions:

We propose a new variable selection called VS-VNS. The latter is based on the variable neighborhood search meta-heuristic to select a significant set of variables for the data classification task. This new technique is compared to two filtering methods to show its performance.

We study the impact of different search methods on BN when combined with variable selection methods for CS.

An extensive experiment is conducted on four credit datasets to evaluate the performance of the different proposed combinations and techniques for CS. We perform credit scoring task on Australian, German, Japanese and the huge "Give me some credit" datasets. We discuss the performance of the different combinations by using various metrics.

The rest of this paper is organized as follows: Section 2 gives a background on BN and the search methods considered in this research. Section 3 details the proposed new technique for variable selection and the different combinations studied in this work. Section 4 presents the empirical studies on the four credit datasets. Finally Section 5 concludes and gives some perspectives.

2 Background

The aim of this section is to give an overview of some important concepts used in this study.

2.1 Bayes Networks (BN)

Bayes network (BN) is a well-known machine learning technique called also Bayesian network or belief network. BN is a statistical model based on combination of directed acyclic graph of nodes and link, and a set of conditional probability table. As shown by Friedman et al. (1997), John and Langley (1995), BN is a probabilistic graphical model that represents a set of random variables and their conditional dependencies via a directed acyclic graph. The default BN uses "K2" search method which is a greedy algorithm. The latter is run several times with random ordering of variables.

2.2 Search method and meta-heuristics

Meta-heuristics are computational search techniques that have been used successfully for solving several optimization problems in several areas. Meta-heuristics can be divided into two main categories: the population-based methods and the single solution-oriented methods (called also trajectory methods). The population based methods maintain and evolve a population of solutions. The trajectory methods or the single solution oriented methods work on a current single solution. Among the population based methods for optimization problems, we cite genetic algorithms used by Gen (2006) and evolutionary computation used by Li et al. (2011). Among the trajectory methods, we find local search proposed by Hansen (1986);, Hoos and Boutilier (2000); Boughaci et al. (2010), simulated annealing developed by Kirkpatrick et al. (1983) and tabu search proposed by Glover (1989).

In this work, we are interested in the trajectory methods, especially in hill climbing, tabu search and simulated annealing.

Local search based method (LS): is a simple hill-climbing technique proposed by Hansen (1986). LS starts with a randomly generated solution and tries to find better solutions in the current neighborhood. Neighborhood solutions are obtained by modifying one position from the solution vector. LS is an iterative process that should be repeated until a certain number of iterations or a criterion is reached.

Simulated annealing (SA): is a local search based method. SA starts with a randomly generated solution. Then it generates neighboring configurations by using a move operator. The new configuration is accepted when it has a lower energy than the preview one. Otherwise the new configuration can be accepted with a certain probability p which helps the system to escape from a local minimum. SA process is repeated for a certain time fixed empirically as presented by Kirkpatrick et al. (1983).

Tabu search (TS): is a local search meta-heuristic. The method was proposed initially by Glover (1989). TS starts with an initial random solution. Then, it tries to locate good solutions by applying iterative modifications on the current solutions. In order to avoid the local optima effectively, TS uses a list to store solutions already visited. This list permits the storage of solution trajectory which avoids local optima.

Tree augmented Naive Bayes (TAN): is a Bayes Network learning algorithm augmented with a tree. The tree is formed by computing the maximum weight spanning tree by using Chow and Liu algorithm. This algorithm permits to learn a BN with a tree structure that maximizes the likelihood of the training data.

3 Proposed method and combinations

Variable selection is a pre-processing that can be launched before any classification task. It is the process that selects variables for the data classification task. It removes the redundant variables that are deemed irrelevant to the data classification task. Several methods have been studied for variable selection. These methods can be divided in two main methods: the wrapper methods proposed by Kohavi and John (1996) and the filtering methods studied by Caruana and Freitag (1994).

The filtering methods are based usually on heuristics. The filtering methods eliminate and filter out the undesirable variables before launching the classification task.

The wrapper methods are based generally on machine learning algorithm for searching the best subset of variables. The aim of the machine learning algorithm is to select the best variables with high classification accuracy. However, the wrapper methods are time consuming compared to filtering methods because the machine learning algorithm is run iteratively while selecting variables.

For the filtering methods, we choose the best-first search proposed by mich21 and the ranking filter information gain methods proposed by Caruana and Freitag (1994) from WEKA package which is available at Waikato (2017). In the rest of this section, we detail the new proposed variable selection VS-VNS for data classification. We give the variable vector representation used in VS-VNS. Then we detail the main components of the proposed VS-VNS method.

3.1 The variable vector representation

The aim of the variable selection is to search for a significant set of variables to be used with the classifier in the classification task. The variable vector can be represented as a binary vector which denote the variables present in the dataset, with the length of the vector equal to n, where n is the number of variables. To represent such vector we use the following assignment: if a variable is selected, the value 1 is assigned to it, a value 0 is assigned to it otherwise. For example, Fig.1 represents an assignment. We have a dataset of seven variables where the second, the third and the sixth variables are selected.

Fig.1

The variable vector representation.

3.2 Proposed VS-VNS for variable selection

The proposed VS-VNS is a variable selection method based on the variable neighborhood search (VNS). VNS is a local search meta-heuristic working on a set of different neighborhood. The basic idea is a systematic change of k neighborhood combined with a local search as shown by Mladenovic and Hansen (1997).

Neighbors solutions are generated by randomly adding or deleting a variable from the variable vector of size n. We use three structures of neighborhood (k=3) which are:

N ₁: where the neighbor solution x′ of the solution x is obtained by modifying only one bit as done with local search method. For example, if n the number of variables equals to 8 and x = 11111111 is a current solution vector, then the possible neighbor solutions in N ₁ can be represented as: 01111111, 10111111, 11011111, 11101111, 11110111, 11111011, 11111101, 11111110.

N ₂: where the neighbor solution x′ of the solution x is obtained by modifying two bits simultaneously. For example, if x = 11111111 is a current solution vector, then the possible neighbor solutions in N ₂ can be: 00111111, 10011111, 11001111, 11100111, 11110011, 11111001, 11111100.

N ₃: where neighboring solution x′ of the solution x is obtained by modifying three bits simultaneously. For example, if we take the same x = 111111111111 as a current solution vector, then the possible neighbor solutions in N ₃ can be: 00011111, 10001111, 11000111, 11100011, 11110001, 11111000.

To avoid exploiting the same region, the proposed VS-VNS performs a certain number of local steps that combines intensification and diversification strategies to locate a good solution. The intensification step is applied with a fixed probability wp > 0 and the diversification step with a probability (1 - wp). The wp is a probability fixed empirically.

Step 1: the intensification that consists in selecting the best neighbor solution having the best objective function value.

Step 2: the diversification that consists in selecting a random neighbor solution.

The proposed VS-VNS is combined with BN. The overall method starts with an initial solution considering all the variables and then tries to find a good solution in the whole neighborhood in an iterative manner. The BN classifier is built for each candidate solution constructed by VS-VNS method. The solution is evaluated according to both accuracy and ROC (the area under the ROC Curve) values. This means that the solution quality is measured by using an objective function given as: objective function (f) = (Accuracy + ROC)/2. The objective is to find an optimal subset of variables by finding optimal combinations of variables from the dataset. The VS-VNS process is repeated for a certain number of iterations max _ iterations fixed empirically. The overall VS-VNS algorithm for variable selection is sketched in Algorithm 1.

3.3 Combination techniques

The second contribution of this paper is the study of the impact of search method on BN when combined with variable selection methods. We study six variants of BN for CS: BN with K2, BN with hill climbing, BN with repeated hill climbing, BN with TS, BN with SA and BN with TAN. For each variant, we consider the three variable selection methods: the best-first search given by Kohavi and John (1996), the ranking filter information gain method given by Caruana and Freitag (1994) and our new VS-VNS method. The numerical results are detailed in the next section.

4 Empirical study for credit scoring

All experiments were run on an Intel Core(TM) i5-2217U CPU@1.70 GHz with 6 GB of RAM under Windows 8, 64 bits, processor x64. The source code is written in Java under NetBeans IDE 8.2 and using the WEKA machine learning package.

4.1 Datasets description

We perform credit scoring task on four financial datasets: German, Australian and Japanese datasets available on UCI (University of California at Irvine) Machine Learning Repository 1 . We consider also the “Give me some Credit” dataset from Kaggle 2 . The description of the four datasets is given in Table 1.

Table 1
Description of the datasets used in the study

Dataset #Loans #Good Loans #Bad Loans #variables

Australian 690 307 383 14

German 1000 700 300 20

Japanese 690 307 383 15

Give me

Some Credit 150000 139974 10026 10

Dataset	#Loans	#Good Loans	#Bad Loans	#variables
Australian	690	307	383	14
German	1000	700	300	20
Japanese	690	307	383	15
Give me
Some Credit	150000	139974	10026	10

Table 2 to 5 give the summary statistics of quantitative variables of Australian, German, Japanese and “Give me some credit” datasets respectively. The column Min gives the minimum value, the column Max is the maximum value, the column Mean is the average value and the column stdDev is the standard deviation.

Table 2

Summary statistics of quantitative variables of the Australian dataset\label A1

Variables	Min	Max	Mean	stdDev
A2	13.75	80.25	31.568	11.853
A3	0	28	4.759	4.978
A7	0	28.5	2.223	3.347
A10	0	67	2.4	4.863
A13	0	2000	184.014	172.159
A14	1	100001	1018.396	5210.103

Table 3

Summary statistics of quantitative variables of the German dataset\label A2

Variables	Min	Max	Mean	stdDev
A2 (duration)	4	72	20.903	12.059
A5 (credit_amount)	250	18424	3271.258	2822.737
A8 (installment_commitment)	1	4	2.973	1.119
A11 (residence_since)	1	4	2.845	1.104
A13 (age)	19	75	35.546	11.375
A16 (existing_credits)	1	4	1.407	0.578
A18 (num_dependents)	1	2	1.155	0.362

Table 4

Summary statistics of quantitative variables of the Japanese Credit dataset\label A33

Variables	Min	Max	Mean	stdDev
A2	13.5	80.25	31.568	11.958
A3	0	28	4.759	4.978
A8	0	28.5	2.223	3.347
A11	0	67	2.4	4.863
A14	0	2000	184.015	173.807
A15	0	100000	1017.386	5210.103

Table 5

Summary statistics of quantitative variables of the Give me some Credit dataset\label A4

Variables	Min	Max	Mean	stdDev
revolving utilization of unsecured lines	0	50708	6.048	249.755
age	0	109	52.295	14.772
number of time 30-59 days past due not worse	0	98	0.421	4.193
debt ratio	0	392664	353.005	2037.819
monthly income	0	3008750	5348.139	13152.057
number of open credit lines and loans	0	58	8.453	5.146
number of times 90 days late	0	98	0.266	4.169
number real estate loans or lines	0	54	1.018	1.13
number of time 60-89 days past due not worse	0	98	0.24	4.155
number of dependents	0	20	0.737	1.107

4.2 Evaluation measures

We used both split training/test partition and a 10 fold cross-validation to evaluate models. The experiment (not reported here) showed that the splitting partition is more effective than cross-validation in our case. In consequence, the evaluation technique considered in this study is to run the BN classifier on the training data to get a model. Then, we apply this model on the test data to find the appropriate class. The example data are partitioned into training and test examples, approximately in the proportion of 66.6% to 33.4% , respectively.

We use several metrics to evaluate the performance of credit scoring models. Table 6 gives the confusion matrix where True Positives (TP) indicates the number of positive examples, labeled as such. False Positives (FP): is the number of negative examples, labeled as positive. True Negatives (TN): is the number of negative examples, labeled as such. False Negatives (FN): is the number of positive examples, labeled as negative. The diagonal elements the confusion matrix given in Table 6 (TP and TN) represents the data properly classified by the classifier while the diagonal elements (FN and FP) represents the misclassified data.

Table 6
Confusion Matrix

Predicted

Positive Negative

Real class Positive TP FP

Negative FN TN

Total P = TP + FN N = FP + TN

		Predicted
Real class	Positive	TP	FP
	Negative	FN	TN
	Total	P = TP + FN	N = FP + TN

We consider the following metrics presented by Powers (2011):

Recall, Sensitivity or true positive rate (TPR): Recall =TPR= $\frac{TP}{P} = \frac{TP}{TP + FN}$

Specificity or true negative rate (TNR): TNR = $\frac{TN}{N} = \frac{TN}{TN + FP}$

Precision or positive predictive value = $\frac{TP}{TP + FP}$

False positive rate (FPR) = $\frac{FP}{N} = \frac{FP}{FP + TN} = 1 - TNR$

The harmonic mean of precision and sensitivity (F-measure) = $\frac{2 TP}{2 TP + FP + FN}$

Matthews correlation coefficient (MCC): MCC = $\frac{TP \cdot TN - FP \cdot FN}{\sqrt{(TP + FP) (TP + FN}) (TN + FP) (TN + FN)}$

Accuracy (ACC) or PRC Area where PRC curves plot precision versus recall. ACC= $\frac{TP + TN}{P + N} = \frac{TP + TN}{TP + TN + FP + FN}$

The area under the ROC curve (AUC). The ROC Area is a common evaluation metric for binary classification problems. ROC plots the value of the Recall against that of the FP Rate at each FP Rate considered.

We note that both ROC and PRC are important performance parameter. However, ROC is more robust than PRC in imbalanced class case because ROC is independent of the fraction of the test population which is class 0 or class 1.

4.3 Impact of variable selection

In this section, we study the impact of the three variable selection methods on CS: the two filtering methods and our VS-VNS. We evaluate their performance when they are combined with search methods for BN. The same variable selection methods are used within the four datasets. The corresponding empirical studies are given in following.

Table 7 gives the number of selected variables with the three variable selection methods on the four datasets.

Table 7
Number of selected variables with the three variable selection methods\label t10

#Selected variables Australian German Japanese Give me Some Credit

ALL variables 14 20 15 10

with CFS 7 3 7 4

with Ranking filter 14 16 15 10

with VS-VNS 12 19 15 10

#Selected variables	Australian	German	Japanese	Give me Some Credit
ALL variables	14	20	15	10
with CFS	7	3	7	4
with Ranking filter	14	16	15	10
with VS-VNS	12	19	15	10

As already mentioned, we consider two filtering methods for variable selection. CFS is a Correlation Based Feature Selection presented by Hall (1999) that selects the best set of variables where variables are assumed to be conditionally independent. However CFS is not able to select all relevant variables when there are strong dependencies. For this reason, we use Information Gain based Ranking filter and VS-VNS. The Ranking filter finds the best subset of variables from the original dataset by using score. The variables are weighted by using the proxy measure rather than error rate as shown by Hall (1999); Caruana and Freitag (1994). VS-VNS applied the process already given in section 3.2 to select the relevant variables. The variables selected with CFS are as follows:

For Australian dataset, 7 variables are selected: A4, A5, A7, A8, A10, A13 and A14.

For German dataset, only 3 variables are chosen: A1, A2 and A3.

For Japanese dataset, 7 variables are selected: A4, A6, A8, A9, A11, A14 and A15.

For Give me some credit, there are 4 variables which are: A2, A3, A7and A9.

The variables selected with the ranking filter are as follows:

For the Australian dataset, we remark that all the variables are ranked and selected: A8, A10, A9, A14, A7, A5, A6, A3, A13, A4, A2, A12, A11 and A1. The corresponding rank values are respectively: 0.425709, 0.213511, 0.156286, 0.110235, 0.110022, 0.10916, 0.050189, 0.041099, 0.036708, 0.029603, 0.022884, 0.010036, 0.000721 and 0.000139.

For the German dataset, there 16 selected variables among 20. The selected variables: A1, A3, A2, A6, A4, A5, A12, A7, A15, A13, A14, A9, A20, A10, A17 and A19 with the following rank value respectively 0.094739, 0.043618, 0.0329, 0.028115, 0.024894, 0.018709, 0.016985, 0.013102, 0.012753, 0.011278, 0.008875, 0.006811, 0.005823, 0.004797, 0.001337 and 0.000964.

For the Japanese dataset, the selected variables are: A9, A11, A10, A15, A8, A6, A14, A7, A3, A5, A4, A2, A13, A12 and A1 sorted by rank. The corresponding rank values are respectively: 0.425709, 0.213511, 0.156286, 0.110235, 0.110022, 0.107525, 0.05371, 0.049456, 0.041099, 0.02875, 0.02875, 0.021239, 0.010036, 0.000721 and 0.000423.

For Give me some credit dataset, the ranking filter select all the variables sorted by rank as follows: A1, A7, A3, A9, A2, A6, A4, A5, A8 andA10. The ranks are: 0.0529, 0.04621, 0.03813, 0.03171, 0.01085, 0.00556, 0.00473, 0.00405, 0.00299 and 0.00155.

The set of variables selected with the proposed VS-VNS when SA is applied as a search method are as follows:

For Australian dataset, VS-VNS selects 12 variables. The unselected variables are A1 and A6.

For German dataset there are 19 selected variables. The variable A5 is removed.

For Japanese dataset there are 15 selected variables where the variable A12 is removed.

For Give me some credit dataset all the variables are selected.

4.4 Variable selection with search method for BN

In this section we present the different results when considering the six search methods combined with the three variable selection techniques in BN (CFS, Ranking and VS-VNS). The max _ iterations in VS-VNS is fixed empirically to 10 and the wp value is equal to 0.6. The numerical results are reported in Tables 8 to 11.

Table 8
BN with variable selection and search methods on Australian Credit dataset\label A7

Search Method Variable Selection #Selected variables TPR% FPR% Precision% Recall% F-Measure% MCC% ROC% PRC%

K2 CFS 7 88.5 11.4 88.6 88.5 88.5 77.0 93.7 93.4

Ranking 14 85.5 14.4 85.6 85.5 85.5 71.1 92.9 92.7

VS-VNS 18 88.0 12.6 88.0 88.0 88.0 75.6 93.6 93.5

CFS 7 88.5 11.4 88.6 88.5 88.5 77.0 93.7 93.4

Hill climbing Ranking 14 86.0 14.0 86.0 86.0 86.0 71.9 92.8 92.5

VS-VNS 12 87.8 12.7 87.8 87.8 87.8 75.3 93.5 93.4

CFS 7 88.5 11.4 88.6 88.5 88.5 77.0 93.7 93.4

Repeated Hill Ranking 14 86.0 14.0 86.0 86.0 86.0 71.9 92.8 92.5

VS-VNS 12 87.8 12.7 87.8 87.8 87.8 75.3 93.5 93.4

CFS 7 88.5 11.4 88.6 88.5 88.5 77.0 93.7 93.4

TS Ranking 14 86.0 14.0 86.0 86.0 86.0 71.9 92.8 92.5

VS-VNS 13 87.2 13.3 87.2 87.2 87.2 74.1 93.5 93.3

CFS 7 87.2 12.5 87.5 87.2 87.2 74.7 94.1 93.9

SA Ranking 14 86.0 13.7 86.2 86.0 86.0 72.2 94.8 94.7

VS-VNS 12 89.0 11.8 89.0 89.0 89.0 77.7 95.9 95.9

CFS 7 88.1 11.6 88.3 88.1 88.1 76.4 94.0 93.8

TAN Ranking 14 87.2 12.8 87.2 87.2 87.2 74.4 93.1 92.6

VS-VNS 13 88.1 13.0 88.2 88.1 88.1 75.9 94.3 94.0

Search Method	Variable Selection	#Selected variables	TPR%	FPR%	Precision%	Recall%	F-Measure%	MCC%	ROC%	PRC%
K2	CFS	7	88.5	11.4	88.6	88.5	88.5	77.0	93.7	93.4
	Ranking	14	85.5	14.4	85.6	85.5	85.5	71.1	92.9	92.7
	VS-VNS	18	88.0	12.6	88.0	88.0	88.0	75.6	93.6	93.5
	CFS	7	88.5	11.4	88.6	88.5	88.5	77.0	93.7	93.4
Hill climbing	Ranking	14	86.0	14.0	86.0	86.0	86.0	71.9	92.8	92.5
	VS-VNS	12	87.8	12.7	87.8	87.8	87.8	75.3	93.5	93.4
	CFS	7	88.5	11.4	88.6	88.5	88.5	77.0	93.7	93.4
Repeated Hill	Ranking	14	86.0	14.0	86.0	86.0	86.0	71.9	92.8	92.5
	VS-VNS	12	87.8	12.7	87.8	87.8	87.8	75.3	93.5	93.4
	CFS	7	88.5	11.4	88.6	88.5	88.5	77.0	93.7	93.4
TS	Ranking	14	86.0	14.0	86.0	86.0	86.0	71.9	92.8	92.5
	VS-VNS	13	87.2	13.3	87.2	87.2	87.2	74.1	93.5	93.3
	CFS	7	87.2	12.5	87.5	87.2	87.2	74.7	94.1	93.9
SA	Ranking	14	86.0	13.7	86.2	86.0	86.0	72.2	94.8	94.7
	VS-VNS	12	89.0	11.8	89.0	89.0	89.0	77.7	95.9	95.9
	CFS	7	88.1	11.6	88.3	88.1	88.1	76.4	94.0	93.8
TAN	Ranking	14	87.2	12.8	87.2	87.2	87.2	74.4	93.1	92.6
	VS-VNS	13	88.1	13.0	88.2	88.1	88.1	75.9	94.3	94.0

Table 9

BN with variable selection and search methods on German Credit dataset

Search Method	Variable Selection	#Selected variables	TPR%	FPR%	Precision%	Recall%	F-Measure%	MCC%	ROC%	PRC%
	CFS	3	73.8	50.0	71.7	73.8	72.3	26.8	75.5	77.7
K2	Ranking	16	75.3	35.9	76.0	75.3	75.6	38.4	80.1	82.4
	VS-VNS	17	76.6	35.2	76.0	76.6	76.2	42.8	81.2	81.8
	CFS	3	73.8	50.0	71.7	73.8	72.3	26.8	75.5	77.7
Hill Climbing	Ranking	16	74.7	45.4	73.4	74.7	73.9	31.4	76.8	80.4
	VS-VNS	19	74.6	35.3	74.6	74.6	74.6	39.4	78.3	79.5
	CFS	3	73.8	50.0	71.7	73.8	72.3	26.8	75.5	77.7
Repeated Hill	Ranking	16	74.7	45.4	73.4	74.7	73.9	31.4	76.8	80.4
	VS-VNS	19	74.6	35.3	74.6	74.6	74.6	39.4	78.3	79.5
	CFS	3	73.8	50.0	71.7	73.8	72.3	26.8	75.5	77.7
TS	Ranking	16	75.9	37.8	75.9	75.9	75.9	38.0	77.8	80.5
	VS-VNS	19	76.2	36.7	75.4	76.2	75.7	41.3	79.5	79.9
	CFS	3	73.8	50.0	71.7	73.8	72.3	26.8	75.5	77.7
SA	Ranking	16	73.2	46.6	72.1	73.2	72.5	28.1	74.8	79.6
	VS-VNS	19	81.9	31.9	81.4	81.9	81.1	54.6	85.1	86.5
	CFS	3	73.8	50.0	71.7	73.8	72.3	26.8	74.1	76.9
TAN	Ranking	16	74.7	40.4	74.5	74.7	74.6	34.6	78.3	80.9
	VS-VNS	19	78.7	32.6	78.1	78.7	78.3	47.8	83.1	84.4

Table 10

BN with variable selection and search methods on Japanese Credit dataset\label A9

Search Method	Variable Selection	#Selected variables	TPR%	FPR%	Precision%	Recall%	F-Measure%	MCC%	ROC%	PRC%
	CFS	7	85.5	15.4	85.8	85.5	85.4	71.1	91.4	90.5
K2	Ranking	14	86.0	15.2	86.6	86.0	85.8	72.3	90.3	89.5
	VS-VNS	13	87.1	13.4	87.1	87.1	87.1	73.9	93.3	93.0
	CFS	7	85.5	15.4	85.8	85.5	85.4	71.1	91.4	90.5
Hill Climbing	Ranking	14	86.0	15.2	86.6	86.0	85.8	72.3	90.5	89.7
	VS-VNS	14	87.1	13.2	87.1	87.1	87.1	73.9	93.3	93.0
	CFS	7	85.5	15.4	85.8	85.5	85.4	71.1	91.4	90.5
Repeated Hill	Ranking	14	86.0	15.2	86.6	86.0	85.8	72.3	90.5	89.7
	VS-VNS	14	87.1	13.2	87.1	87.1	87.1	73.9	93.3	93.0
	CFS	7	85.5	15.4	85.8	85.5	85.4	71.1	91.4	90.5
TS	Ranking	14	86.0	15.2	86.6	86.0	85.8	72.3	90.5	89.7
	VS-VNS	13	87.7	12.7	87.7	87.7	87.7	75.0	93.3	93.0
	CFS	7	85.5	15.6	86.1	85.5	85.4	71.3	91.8	90.9
SA	Ranking	14	83.8	17.6	84.7	83.8	83.6	68.1	91.0	90.0
	VS-VNS	14	89.7	10.4	89.7	89.7	89.7	79.2	95.8	95.7
	CFS	7	85.5	15.6	86.1	85.5	85.4	71.3	90.5	89.4
TAN	Ranking	14	85.5	15.6	86.1	85.5	85.4	71.3	91.3	90.4
	VS-VNS	13	89.0	11.7	89.0	89.0	89.0	77.7	94.8	94.5

Table 11

BN with variable selection and search methods on Give me some Credit dataset

Search Method	Variable Selection	#Selected variables	TPR%	FPR%	Precision%	Recall %	F-Measure%	MCC %	ROC%	PRC %
K2	CFS	4	93.2	64.4	92.1	93.2	92.5	35.0	81.1	93.5
	Ranking	10	92.6	54.9	92.4	92.6	92.5	38.6	85.5	94.5
	VS-VNS	8	92.7	54.9	92.4	92.7	92.6	39.4	85.9	94.5
	CFS	4	93.2	64.4	92.1	93.2	92.5	35.0	81.1	93.5
Hill Climbing	Ranking	10	92.6	54.9	92.4	92.6	92.5	38.6	85.5	94.5
	VS-VNS	8	92.7	54.9	92.4	92.7	92.6	39.4	85.9	94.5
Repeated Hill	CFS	4	93.2	64.4	92.1	93.2	92.5	35.0	81.1	93.5
	Ranking	10	92.6	54.9	92.4	92.6	92.5	38.6	85.5	94.5
	VS-VNS	8	92.7	54.9	92.4	92.7	92.6	39.4	85.9	94
TS	CFS	4	93.6	79.9	91.7	93.6	91.8	26.2	73.1	92.0
	Ranking	10	92.9	57.4	92.4	92.9	92.6	38.1	85.3	94.5
	VS-VNS	10	92.7	56.0	92.4	92.7	92.5	38.7	85.5	94.5
	CFS	4	93.7	79.1	91.9	93.7	91.9	27.8	81.2	93.5
SA	Ranking	10	93.5	70.4	92.0	93.5	92.4	32.6	85.0	94.4
	VS-VNS	10	93.6	69.2	92.2	93.6	92.5	34.7	86.2	94.7
TAN	CFS	4	93.5	71.6	92.0	93.5	92.3	32.0	81.2	93.4
	Ranking	10	93.3	64.5	92.2	93.3	92.6	35.9	85.5	94.5
	VS-VNS	9	93.3	64.7	92.2	93.3	92.6	36.1	86.0	94.6

According to the numerical results, we can say that in general BN with K2, BN with hill climbing, BN with Repeated hill climbing and BN with TS are comparable on the four considered datasets. The numerical results show a slight performance in favor of BN with TAN when compared it to BN with K2, BN with BN with hill climbing, BN with Repeated hill and BN with TS. Moreover BN succeeds in finding good results when combined with SA search method for all the considered datasets compared to the others BN variants.

For example for Australian dataset, BN with SA gives a ROC% value equal to 94.1 when the CFS is used as variable selection method while BN with K2 gives a ROC% equal to 93.7. The same behavior can be seen when we use Ranking, and VS-VNS variable selection methods. Further, the results are much better when we use SA with VA-VNS. The resulting method gives the best ROC% value which is equal to 95.9 for the Australian dataset. The performance of BN with SA is confirmed on German, Japanese and Give me some credit datasets. BN with SA provides good results compared to the other BN variants considered in this study. According to the numerical results, we can say that the Bayesian network with SA search method is able to find high quality results in term of both ROC and PRC. The other measures TPR, FPR, Precision, Recall, F-Measure, MCC confirm this result as shown in Tables 8 to 11.

When we compare the variable selection methods, we can see that VS-VNS provides good results compared to both CFS and Ranking methods. For example for Australian dataset, BN with SA gives a ROC% value equal to 94.1 when combined with CFS, a ROC% value equal to 94.8 when we use Ranking as selection variable method. The best result is found when we use VS-VNS where the ROC% value is equal to 95.9.

In conclusion, promising results are obtained when combining BN with SA search method with our new VS-VNS technique. This improvement can be shown for all the considered datasets. The proposed method gives a ROC% value equal to 95.9 and PRC % equal to 95.9 for Australian dataset, a ROC% value equal to 85.1 and PRC % equal to 86.5 for German dataset, a ROC% value equal to 95.8 and PRC % equal to 95.7 for Japanese dataset and a ROC% value equal to 86.2 and PRC % equal to 94.7 for the Give me some credit. To clearly show this performance, we draw the curves in Figs 2 to 5 for the four datasets.

Fig.2

ROC% and PRC% found with variable selection with SA for Australian dataset.

Fig.3

ROC% and PRC% found with variable selection with SA for German dataset.

Fig.4

ROC% and PRC% found with Variable selection with SA for Japanese dataset.

Fig.5

ROC% and PRC% found with variable selection with SA for Give me some credit dataset.

4.5 ANOVA statistical analysis

To show statistically the significance of our results, we use here ANOVA (Analysis of variance) statistical tool. In our case, we use the ROC% to compare the effect of the VS-VNS variable selection when combined with SA search method in BN for CS. We compare between all the variable selection (CFS, Ranking and VS-VNS) when used with SA in BN. We compared also with BN when we consider all the variables (ALL). Table 12 describes the four ANOVA tests where the column df represents the degree of freedom. The column SS is the Sum of squares. The column MS is the mean square. The F-value is the F-statistic. The p-value in bold font makes interpretation and results analysis. The p-value is <0.05. This indicates that the values of ROC% produced by VS-VNS are high significantly different from those produced with the other BN variants. This means that the quality of the results using our proposed technique combined with SA in BN is statistically better than those of the other methods.

Table 12
ANOVA Test for BN with SA search method

BN with SA search method df SS MS F-value P-value

VS-VNS. vs. CFS 1 120.91 120.91 73.12 0.00336

VS-VNS vs. Ranking 1 105.14 105.14 15.22 0.0299

VS-VNS vs. ALL 1 107.02 107.02 17.03 0.0258

BN with SA search method	df	SS	MS	F-value	P-value
VS-VNS. vs. CFS	1	120.91	120.91	73.12	0.00336
VS-VNS vs. Ranking	1	105.14	105.14	15.22	0.0299
VS-VNS vs. ALL	1	107.02	107.02	17.03	0.0258

5 Conclusion

Credit scoring is an important process in financial institutions and banks. It helps in decision making and permits to distinguish between ’bad’ and ’good’ counterparties. CS uses variables related to applicants to evaluate their creditworthiness. In this work, we studied the impact of variables and classification on credit scoring. We proposed a new variable selection called VS-VNS for CS. The proposed method is compared to two filtering methods. Further, we explored different combinations of variable selection and various search method for BN. We considered six search methods which are K2, Hill climbing, Repeated Hill climbing, TS, SA and TAN. The different techniques are combined with BN to build models for CS. The different combinations are evaluated on German, Australian, Japanese and the huge kaggle dataset. The numerical results are promising and show the benefit of the proposed combinations. Further, the proposed VS-VNS succeeds in finding promising results compared to the other variable selection techniques in particular when combined with SA in BN. We plan to combine several variable selection strategies together to further enhance the performance. It would be nice to evaluate the considered combinations with other classifiers on other datasets.

Footnotes

1

2

References

Abdou, H.A.., 2009. Genetic programming for credit scoring: The case of Egyptian public sector banks, Expert Systems with Applications 36, 11402–11417.

Bellotti, T., Crook, J.., 2009. Support vector machines for credit scoring and discovery of significant features, Expert Systems with Applications 36, 3302–3308.

Bonilla Huerta, E.B., Duval, B., Hao, J.K.., 2006. A Hybrid GA/SVM Approach for Gene Selection and Classification of Microarray Data. In Rothlanf , et al. (Eds.) EvoWorkshops 2006, LNCS 3907, pp. 34–44.

Boughaci, D., Benhamou, B., Drias, H.., 2010. Local search methods for the optimalwinner determination problem in combinatorial auctions, In Journal of Mathematical Modelling and Algorithms 9(2), 165–180.

Breiman, L., Friedman, J., Olshen, R., Stone, C.., 1984. Classification and Regression Trees, Belmont, CA: Wadsworth.

Caruana, R., Freitag, D.., 1994. Greedy attribute selection, In Proceedings of the eleventh international conference on machine learning, (ICML 1994, New Brunswick, New Jersey), Morgan Kauffmann, pp. 28–36.

Desay, V., Crook, J.N., Overstreet, G.A.., 1996. A comparison of neural networks and linear scoring models in the credit union environment, European Journal of Operational Research 95, 24–37.

Friedman, N., Geiger, D., Goldszmidt, M.., 1997. Bayesian Network Classifiers, Machine Learning 29, 131–163.

Gen, M.., 2006. Genetic Algorithms and Their Applications. In Pham

(Ed.), Springer handbook of engineering statistics, pp: 749–773. London: Springer.

10.

Glover, F.., 1989. Tabu search, part I, ORSA Journal on Computing 1, 190–206.

11.

Hall

., 1999. Correlation-based Feature Selection for Machine Learning, Methodology 21i195-i20(April), 1–5.

12.

Hand, D.J., Henley, W.E.., 1997. Statistical classification methods in consumer credit scoring, it, Journal of the Royal Statistical Society, Series A (Statistics in Society) 160, 523–541.

13.

Henley, W.E., Hand, D.J.., 1996. A k-nearest neighbour classifier for assessing consumer credit risk, Statistician 45, 77–95.

14.

Hansen, P.., 1986. The Steepest Ascent Mildest Descent Heuristic for Combinatorial Programming, Presented at the Congresson Numerical Methods in Combinatorial Optimization, Capri, Italy.

15.

Hoos, H.H., Boutilier, C.., 2000. Solving combinatorial auctions using stochastic local search, In Proceedings of the th national conference on artificial intelligence, pp: 22–29 .

16.

John, G.H., Langley, P.., 1995. Estimating continuous distributions in Bayesian classifiers, In Proceedings of the eleventh conference on uncertainty in artificial intelligence. San Mateo: Morgan Kaufman,pp. 338–345.

17.

Jolliffe, I.T.., 2002. Principal Component Analysis, Series: Springer Series in Statistics, 2nd ed., Springer, NY, 2002, XXIX, 487 p. 28 illus. ISBN 978-0-387-95442-4.

18.

Kohavi, R., John, G.., 1996. Wrappers for feature subset selection, Artificial intelligence, Special issue on relevance, pp. 273–324.

19.

Kirkpatrick, S., Gelatt

C.D.

Jr , Vecchi, M.P.., 1983. Optimization by simulated annealing, Science 220(4598), 671–680.

20.

Li, J., Wei, L., Li, G., Xu, W.., 2011. An evolution strategy-based multiple kernels multi-criteria programming approach: The case of credit decision making, Decision Support Systems 51, 292–298.

21.

Mladenovic, N., Hansen

., 1997. Variable neighbourhood decomposition search, in the Computer and Operations Research 24, 1097–1110.

22.

Mester, L.J.., 1997. What’s the point of credit scoring? Business Review (September) 3–16.

23.

Miller, M.., 2003. Research confirms value of credit scoring, National Underwriter 107(42), 30.

24.

Powers, D.M W.., 2011. Evaluation: From precision, recall and fmeasure to ROC, informedness, markedness and correlation, Journal of Machine Learning Technologies 2(1), 37–63.

25.

Quinlan

J.R.

,. Simplifying decision trees, Int J Man-Machine Studies, 21, 221–234.

26.

Sousaa, M.R., Gamaa, J., Brandao, E.., 2015. A new dynamic modeling framework for credit risk assessment, Expert Systems With Applications October 16, 2015.

27.

Waikato Environment for Knowledge Analysis (WEKA), Version 3.9. The University ofWaikato, Hmilton, New Zealand, Software available at http://www.cs.waikato.ac.nz/_ml/weka/: Accessed: On November 2017.

28.

Wiginton, J.C.., 1980. A note on the comparison of logit and discriminant models of consumer credit behavior, Journal of Financial and Quantitative Analysis 15, 757–770.