Appropriate machine learning techniques for credit scoring and bankruptcy prediction in banking and finance: A comparative study

Abstract

Machine learning techniques have been used successfully in several areas such as banking and finance. These techniques are used mainly for prediction, classification and partitioning data into different groups according to a certain common characteristic. In this work, we are interested in machine learning techniques for credit scoring and bankruptcy prediction in finance and banking. We evaluate and compare a range of machine learning techniques on several datasets issued from banks and financial institutions where the aim is to select the most appropriate methods suitable for each dataset. We use several metrics to evaluate the performance of the obtained models. The empirical studies are conducted on German, Australian, Japanese, Polish, Indian Qualitative Bankruptcy and Taiwan datasets. Also, we consider the huge “Give Me Some Credit dataset”.

The machine learning methods produce scores for applicants and companies and help a lot in the decision making. In other words, these methods permit us to distinguish between bad and good applicants or companies. The numerical study shows that there is no method able to consistently outperform the others on all the datasets. Also, there are significant differences between the studied methods on some datasets. For German and Give Me Some Credit datasets, the Bayes net method is able to produce good scores compared to the others studied methods. The LogitBoost method is competitive on both Polish and Australian datasets, while AdaBoost method is most appropriate for Japanese dataset. For Taiwan dataset, Random Forest method gives the best results compared to the other considered techniques. However, on Indian Qualitative Bankruptcy dataset, almost the methods are comparable due to the nature of this dataset.

Keywords

Banking finance credit scoring machine learning classification bankruptcy prediction

1. Introduction

When a borrower or a loan applicant (also called “a counterparty”) applies for credit, lenders or creditors will check its credit score to determine whether it can get credit or not. A credit score is an evaluation that helps lenders and creditors to see how good the borrower is and indicates how likely the borrower is to pay back the owed debt, based on its past borrowing behavior.

Credit scoring (CS) is a process used to predict the probability that a borrower will default or become delinquent (Mester [20]). CS is the process of evaluating the creditworthiness of applicants in order to decide if the credit will be granted or not. The evaluation process is usually based on some variables related to applicants such as historical payments, guarantees, default rates, ect.

CS has several advantages that accrue to both lenders and borrowers. Several studies in finance and banking have shown that credit scores are one of the most powerful predictors of risk (Miller [21]). CS enables banks and financial institutions to produce scores for applicants and helps them to decide whether to grant credit to the applicant or not. It can be used for avoiding risk as well as supporting retail, corporate and collection scorecards. CS also provides an objective analysis of the applicant’s creditworthiness which reduces discrimination and credit risk. Further, credit scoring allows the automation of the lending process which leads to increased speed and consistency of the loan application.

Over the last few years, several models and techniques for credit scoring have been proposed and discussed. Almost all of them are based on statistical techniques, machine learning or data mining. Among these techniques we give the following ones: the linear regressions which are statistical methods (Hand et al. [15]) that permit us to analyze data and verify if the credit can be granted to a given applicant or not; the discriminant analysis and logistic regression based methods which are one of the most broadly established statistical techniques used to classify applicants as good or bad (Wiginton [29]); the decision trees (Vapnik [27], Quinlan [23]), Classification and Regression Trees (CART) (Breiman et al. [5]) and the Bayesian networks based models (Friedman [12]).

Computational intelligence based techniques are also investigated for developing credit scoring models. Among them, we cite: the neural networks (Desay et al. [10], Quinlan [24]), the k-Nearest Neighbor (k-NN) classifier (Henley et al. [16]), the support vector machines (SVM) (Bellotti and Crook [4]), the ensemble classifiers (Abellan and Mantas [3]), the genetic programming method (Abdou [2]), the evolution strategies (Li et al. [19]), the cooperative agents based system (Boughaci and Alkawaldeh [7]) and recently the new variable selection for credit scoring proposed in Boughaci and Alkawaldeh [8].

In this paper, we evaluate and compare the performance of various machine learning (ML) classifiers for CS and bankruptcy prediction. The empirical studies are conducted on seven well-known financial datasets which are: Australian, German, Japanese, Taiwan, Polish, Indian and the huge “Give Me Some Credit” from Kaggle. The Polish and Indian are dedicated to bankruptcy prediction while the other datasets are focused on CS.

The rest of this paper is organized as follows: Section 2 gives the problem formulation and then describes the set of the machine learning algorithms used in this study. Section 3 gives the considered datasets and the evaluation measures. Section 4 presents the empirical studies. Finally, Section 5 concludes and provides future work.

2. Classification and machine learning techniques

Classification is a supervised learning process that permits to find the best described computer model from a dataset with the correct class variable. This process is called a supervised learning because we have a set of input variables and an output variable (the label or class). We use an algorithm to learn the mapping function from the input to the output. The supervised learning technique creates a model based on training dataset and uses this model to classify new data. When we have only input data and no corresponding output variables, we deal with unsupervised learning technique also called clustering or segmentation. Unlike classification, clustering is an unsupervised learning algorithm that may be used to split a large dataset into clusters or groups according to a certain common characteristic.

In this paper, we discuss the credit scoring as a classification problem. The classification problem is to predict whether the test data belongs to one of the considered classes. Let us consider a test sample or a novel data to be classified. The problem is to predict whether the test data belongs to one of the considered classes. The training data is a set of examples of the form $\{x_{i},y_{i}\}∼(i=∼1,\ldots ,l)$ , where the x _i are called input vector. Each input vector has a number of variables also called features. y _i are the response variables also called labels. These input vectors are paired with corresponding labels to find the correct class variable. $y_{i}\in \{1,0\}$ in binary classification.

In the following, we give the problem formulation and the set of the machine learning algorithms used in this study.

2.1. Problem formulation

The CS problem can be stated as follows (Miline et al. [14]):

We consider a set of features that can be used to characterize a financial dataset of applicants. Each feature may be: a numeric or a category. The applicant age and the interest rates are examples of numerical features that can be represented as a numeric. The categorical features are qualitative features such as the credit history or a geographic region code.

In a CS problem, the financial dataset consists of a set of applicants where each applicant is represented by a set of features. The problem is to classify the applicants into two classes: “bad” (Y = 0, who defaulted on their loans) or “good” (where the label class is Y = 1).

Mathematically, the CS problem can be formulated as follows (Miline et al. [14]):

–
The credit data can be organized as a matrix D of m rows and n columns where n is the number of features and m is the number of past applicants. $\begin{eqnarray}\mathbf{D}=\left[\begin{array}{@{}cccc@{}}a_{11} & a_{12} & \cdots ∼ & a_{1n}\\ a_{21} & a_{22} & \cdots ∼ & a_{2n}\\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots ∼ & a_{mn}\end{array}\right]\end{eqnarray}$ For example, the first row a ₁₁, a ₁₂, ⋯a _{1 n} represents the specific data values of the first applicant where a _{1 i} is the value of feature i of the applicant number 1.
–
The creditworthiness of new applicant can be determined by using the data on past applicants recorded in the matrix D.
–
The decision is then represented as a vector Y with m elements where each element y _i has two possible values 0 or 1. An element y _i receives the value 1 when the applicant i is accepted, 0 otherwise. $\begin{eqnarray}\mathbf{Y }=\left[\begin{array}{@{}c@{}}y_{1}\\ y_{2}\\ y_{3}\\ \vdots \\ y_{n}\end{array}\right]\end{eqnarray}$
–
The classification is then the problem of determining the decision vector Y that indicates the accepted applicants (y _i = 1) or the rejected ones (y _i = 0).

2.2. Machine learning techniques

In this study, we evaluate a range of machine learning classifiers on some credit scoring and bankruptcy datasets. The set of the considered machine learning classifiers is given as follows:

–
Random Forest (RF): a learning method proposed by Tin Kam Ho [18] for both classification and regression tasks. Random Forest is a forest decision tree classifier based on a multitude of decision trees where each tree gives a classification. Then the forest chooses the classification having the most votes over all the trees in the forest.
–
k-Nearest Neighbor classifier (k-NN): this method classifies instances by comparing feature vectors of the different points and finding the k training examples that are closest to the test example. The predicted class is the most frequent one (Henley et al. [16]). In other word, the classifier predicts the class by the majority class of the k most similar training examples stored in the model based on a distance metric. We note that the value of k is fixed by empirical study. In this study, we set k to 3 and we use the Euclidian distance as a standard in the field.
–
Naive Bayes classifier (NB): a simple probabilistic classifiers based on the Bayesian theorem with independence assumptions between the features or variables. In order to derive a conditional probability for the relationships between the feature values and the class, NB analyses the relationship between each feature and the class for each instance (Rennie et al. [25]).
–
Bayes Network classifier (BN): also known as Bayesian network or belief network is a statistical model based on a combination of a directed acyclic graph of nodes and links, and a set of conditional probabilities held in a table. It is a probabilistic graphical model that represents a set of random variables and their conditional dependencies via a directed acyclic graph (Friedman et al. [12]).
–
Multi-layer Perceptron (MLP): a feed-forward artificial neural network that uses a supervised learning back-propagation technique for training the network. An MLP is a set of layers of nodes in a directed graph where each node is a neuron with a nonlinear activation function. MLP model permits the mapping sets of input data onto a set of appropriate outputs (Frank et al. [26]).
–
Logistic classifier: a regression model also known as logit regression. It was developed by the statistician David Cox in 1958 (Cox [9]). It is used to estimate the probability of a qualitative response based on one or more predictor or independent variables.
–
Support Vector Machine (SVM): a machine learning technique that was proposed by Vladimir Vapnik for classification and regression (Friedman et al. [12]). The SVM classification method learns from a training data set and attempts to generalize and make correct predictions on novel data. SVM is a Kernel based method where the kernel represents similarity measures of examples of the training data. We find the following kernel functions: Linear, Polynomial, Laplacian, sigmoid and Gaussian also called radial Basis Function (RBF) (Friedman et al. [12]).
–
Bagging: a Bootstrap aggregating machine learning. It is an ensemble algorithm used in classification and regression designed to improve the stability and accuracy of machine learning algorithms. The method was proposed by Leo Breiman in 1994 to improve the classification by combining classifications of randomly generated training sets (Breiman [6]).
–
AdaBoost: Boosting is a process of increasing the performance of weak learning algorithms. It is a combination of a set of classifiers produced by the learning algorithm over a number of distributions of the training data. AdaBoost is an adaptive boosting machine learning algorithm. The classifier may be added to many other types of learning algorithms to improve their performance. The output of the other learning algorithms is combined into a weighted sum that represents the final output of the boosted classifier (Freund et al. [11]).
–
OneR: a simple classification based on rules. The method learns a one-level decision tree for generating a set of rules that test one particular attribute (Holte, [17]).
–
Logit Boost: a boosting algorithm that combines an AdaBoost machine learning algorithm with a logistic regression. We obtain the Logit boost algorithm by considering AdaBoost as a generalized additive model and applying the cost functional of logistic regression (Driedman [13]).

3. The considered datasets

This study considers seven financial datasets which are: German, Australian, Japanese, Polish, Indian Qualitative Bankruptcy and Taiwan default of credit card clients datasets available on UCI (University of California at Irvine) Machine Learning Repository¹. Further, we consider also the Give Me Some Credit data set from the Kaggle web site². The descriptions of the considered datasets are given in Tables 1 and 2.

Table 1
Description of the datasets (case of customers) used in the study

Dataset #Loans #Good loans #Bad loans #variables

Australian 690 307 383 15

German 1000 700 300 21

Japanese 690 307 383 16

Taiwan 30000 6636 23364 24

Give Me Some Credit 150000 139974 10026 11

Dataset	#Loans	#Good loans	#Bad loans	#variables
Australian	690	307	383	15
German	1000	700	300	21
Japanese	690	307	383	16
Taiwan	30000	6636	23364	24
Give Me Some Credit	150000	139974	10026	11

Table 2

Description of the datasets (case of companies) used in the study

Dataset	#Companies	Non-Bankruptcy	Bankruptcy	#variables
Polish	5910	410	5500	65
Indian	250	143	107	7

Table 3

Classifiers performances on German credit dataset

	ML performance metrics		Other performance metrics
Classifier	ROC%	PRC%	TPR%	FPR%	Precision%	Recall%	F-Measure%	MCC%
Random Forest	77.3	81.8	77.3	47.1	75.5	77.3	75.6	35.4
K-NN	71.4	73.8	74.3	44.1	73.4	74.3	73.8	31.
Bayes Net	80.6	82.0	77.3	34.9	77.5	77.3	77.4	42.1
Naive Bayes	78.8	81.9	75.3	40.5	74.9	75.3	75.1	35.4
Logistic	78.5	79.8	75.2	39.8	74.1	75.2	74.4	37.9
MLP	74.4	77.2	74.7	36.7	75.5	74.7	75.0	36.8
SVM	66.4	69.9	75.7	42.9	74.7	75.7	75.1	34.5
OneR	54.3	63.2	71.7	63.0	66.5	71.7	67.3	12.0
Bagging	77.1	80.8	74.3	45.8	73.1	74.3	73.5	30.4
AdaBoost	75.9	78.3	75.0	52.9	72.3	75.0	72.6	27.1
LogitBoost	76.1	79.1	75.7	46.1	74.0	75.7	74.5	32.5

3.1. Evaluation measures

The example data are partitioned into training and testing examples, approximately in the proportion of 70% to 30%, respectively. Credit scoring models are constructed on the training example data by using the considered machine learning algorithms and validated on the testing data. We evaluate each classifier on the training data to get a model. Then, we apply this model on the test data to find the appropriate class.

We use several metrics to evaluate the performance of credit scoring models where True Positives (TP) indicates the number of positive examples, labeled as such. False Positives (FP): is the number of negative examples, labeled as positive. True Negatives (TN): is the number of negative examples, labeled as such. False Negatives (FN): is the number of positive examples, labeled as negative. We consider the following metrics (Powers [22]):

–
Recall, Sensitivity or true positive rate (TPR): $\text{Recall}=\text{TPR}=\frac{TP}{TP+FN}$
–
Specificity or true negative rate (TNR): $\text{TNR}=\frac{TN}{TN+FP}$
–
Precision or positive predictive value $=∼\frac{TP}{TP+FP}$
–
False positive rate (FPR) $=∼\frac{FP}{FP+TN}=1-TNR$
–
The harmonic mean of precision and sensitivity (F-measure) $=∼\frac{2TP}{2TP+FP+FN}$
–
Matthews correlation coefficient (MCC): $\text{MCC}=\frac{TP\cdot TN-FP\cdot FN}{\sqrt{(TP+FP)(TP+FN)}(TN+FP)(TN+FN)}$
–
Accuracy (ACC) or PRC Area where PRC curves plot precision versus recall. $\text{ACC}=\frac{TP+TN}{TP+TN+FP+FN}$
–
The area under the ROC curve (AUC). The ROC Area is a common evaluation metric for binary classification problems. ROC plots the value of the Recall against that of the FP Rate at each FP Rate considered.

We note that the ROC measure is more robust than PRC in imbalanced class case because ROC is independent of the fraction of the test population which is class 0 or class 1.

Table 4
Classifiers performances on Australian dataset

ML performance metrics Other performance metrics

Classifier ROC% PRC% TPR% FPR% Precision% Recall% F-Measure% MCC%

RandomForest 93.1 92.8 85.5 13.8 86.0 85.5 85.5 71.4

K-NN 89.5 86.0 86.0 13.4 86.4 86.0 86.0 72.2

Bayes Net 92.9 92.8 87.9 12.2 87.9 87.9 87.9 75.7

Naive Bayes 91.2 90.9 81.6 21.2 83.8 81.6 81.0 64.6

Logistic 90.6 87.8 85.4 14.4 85.6 85.4 85.4 70.7

MLP 89.1 87.6 85.5 14.7 85.5 85.5 85.5 70.8

SVM 85.7 80.7 85.0 13.7 86.2 85.0 85.0 71.2

OneR 86.2 81.3 85.5 13.1 86.8 85.5 85.5 72.3

Bagging 92.3 92.0 85.5 13.3 86.6 85.5 85.5 72.0

AdaBoost 94.5 93.7 85.5 13.1 86.8 85.5 85.5 72.3

LogitBoost 95.4 95.3 85.5 14.2 85.7 85.5 85.5 71.0

Table 5
Classifiers performances on Give Some Credit dataset

ML performance metrics Other performance metrics

Classifier ROC% PRC% TPR% FPR% Precision% Recall% F-Measure% MCC%

Random Forest 83.6 94 93.6 74.6 91.9 93.6 92.1 30.6

K-NN 63.6 89.7 92.9 81.3 90.5 92.9 91.2 20

Bayes Net 85.5 94.5 92.6 54.9 92.4 92.6 92.5 38.6

Naive Bayes 67.9 90.7 93.3 90.8 90.2 93.3 90.4 9.8

Logistic 70.0 90.7 93.5 89.8 91.3 93.5 90.7 13.8

MLP 81.4 93.7 93.6 81.2 91.9 93.6 91.7 26.2

SVM 50.8 87.8 93.4 91.9 91.4 93.4 90.4 9.2

OneR 52.5 88 93.2 88.3 90.2 93.2 90.7 13.3

Bagging 85.4 94.5 93.6 75.8 91.8 93.6 92.1 29.7

AdaBoost 83 93.6 93.5 85.4 91.7 93.5 91.2 21.3

LogitBoost 85.1 94.3 93.7 76.6 92.1 93.7 92.1 30.3

4. Empirical study

	ML performance metrics	Other performance metrics
RandomForest	93.1	92.8	85.5	13.8	86.0	85.5	85.5	71.4
K-NN	89.5	86.0	86.0	13.4	86.4	86.0	86.0	72.2
Bayes Net	92.9	92.8	87.9	12.2	87.9	87.9	87.9	75.7
Naive Bayes	91.2	90.9	81.6	21.2	83.8	81.6	81.0	64.6
Logistic	90.6	87.8	85.4	14.4	85.6	85.4	85.4	70.7
MLP	89.1	87.6	85.5	14.7	85.5	85.5	85.5	70.8
SVM	85.7	80.7	85.0	13.7	86.2	85.0	85.0	71.2
OneR	86.2	81.3	85.5	13.1	86.8	85.5	85.5	72.3
Bagging	92.3	92.0	85.5	13.3	86.6	85.5	85.5	72.0
AdaBoost	94.5	93.7	85.5	13.1	86.8	85.5	85.5	72.3
LogitBoost	95.4	95.3	85.5	14.2	85.7	85.5	85.5	71.0

	ML performance metrics	Other performance metrics
Random Forest	83.6	94	93.6	74.6	91.9	93.6	92.1	30.6
K-NN	63.6	89.7	92.9	81.3	90.5	92.9	91.2	20
Bayes Net	85.5	94.5	92.6	54.9	92.4	92.6	92.5	38.6
Naive Bayes	67.9	90.7	93.3	90.8	90.2	93.3	90.4	9.8
Logistic	70.0	90.7	93.5	89.8	91.3	93.5	90.7	13.8
MLP	81.4	93.7	93.6	81.2	91.9	93.6	91.7	26.2
SVM	50.8	87.8	93.4	91.9	91.4	93.4	90.4	9.2
OneR	52.5	88	93.2	88.3	90.2	93.2	90.7	13.3
Bagging	85.4	94.5	93.6	75.8	91.8	93.6	92.1	29.7
AdaBoost	83	93.6	93.5	85.4	91.7	93.5	91.2	21.3
LogitBoost	85.1	94.3	93.7	76.6	92.1	93.7	92.1	30.3

All experiments were run on an Intel Core(TM) i5-2217U CPU@1.70 GHz with 6 GB of RAM under Windows 8 64 bits, processor x64.

We conduct intensive experiments using the Waikato environment for knowledge Analysis (Weka) (Waikato [28]) version 3.9. Weka is a machine learning software developed by the University of Waikato. It allows preprocessing, analyzing and data classification. Weka is available under the GNU license (General Public License). In the following, we give the numerical results found when applying machine learning algorithms on the seven credit datasets. We compare the classifiers by using the ML performance metrics already given in Section 3.1. To choose the best method, we first look at the ML performance metrics: ROC and PRC rates. Then we look at the other performance metrics as a tie-breaker to confirm the decision based on ROC and PRC rates. The best results are in bold font.

4.1. Comparison of ML methods on CS data

Tables 3 through 7 give the numerical results found when applying machine learning algorithms on the CS datasets. As we can see from Table 3, the Bayes Net classifier gives a good classification with a PRC rate equals to 82% and ROC rate equals to 80.6% for the German dataset. As given in Table 3, the TPR is 77.3%, the FPR is 34.9%, the Precision rate is 77.5%, the Recall rate is 77.3%, the F-Measure rate is 77.4% and the MCC rate is 42.1%. This confirms the effectiveness of Bayes Net as classifier for German dataset.

According to Table 4, we can see that Bayes Net (BN) gives the best results on Australian dataset when we considered TPR%, FPR%, Precision%, Recall%, F-Measure% and MCC% as measures of performances. With BN, we obtain TPR rate equals to 87.9%, FPR rate is 12.2%, Precision rate is 87.9%, Recall rate is 87.9%, F-measure rate is 87.9% and MCC rate equals to 75.7%. However, BN gives a ROC rate equals to 92.9% and PRC rate is 92.8% as shown in Table 4.

When we compare classifiers according to ROC area, we can conclude that LogitBoost method is the best one since it provides competitive results on Australian dataset. LogitBoost method gives a ROC rate equals to 95.4% and PRC rate equals to 95.3%. The TPR rate is 85.5% and the FPR rate is 14.2%. This proves the performance of LogitBoost classifier on Australian dataset compared to the other classifiers.

For Give Me Some Credit dataset, from Table 5, we remark that Bayes Net method gives a classification with a ROC rate equals to 85.5% and a PRC rate equals to 94.5%. Bagging method succeeds in finding good results with a ROC rate equals to 85.4% and the PRC rate is 94.5%. With AdaBoost method, the ROC rate is 83% and the PRC rate is 93.6%. With LogitBoost method the ROC rate is 85.1% and the PRC rate is 94.3%. When, we take ROC rate as a measure to compare the different classifiers, we can conclude that Bayes Net classifier is the best one on the Give Me Some Credit dataset with a ROC rate equals to 85.5% compared to the other classifiers. The others metrics shown in Table 5 confirms the performance of Bayes Net method on the Give Me Some credit dataset.

From Table 6, we can see that AdaBoost method performs well on Japanese dataset. The ROC rate is equal to 91.8%, the PRC rate is equal 90.6%, the TPR rate is equal to 84.7%, the FPR rate is equal to 16.1%, the Precision rate is equal to 84.8%, the Recall rate is equal to 84.7%, the F-Measure rate is equal to 84.6% and the MCC rate is equal to 69.3%.

Table 6
Classifiers performances on Japanese dataset

ML performance metrics Other performance metrics

Classifier ROC% PRC% TPR% FPR% Precision% Recall% F-Measure% MCC%

RandomForest 91.3 90.2 84.3 16.7 84.6 84.3 84.1 68.5

K-NN 77.8 72.0 78.3 22.6 78.4 78.3 78.2 56.3

Bayes Net 90.3 89.5 86.0 15.2 86.6 86.0 85.8 72.3

Naive Bayes 87.0 86.7 75.3 27.3 77.8 75.3 74.3 52.2

Logistic 90.6 88.3 83.4 17.3 83.5 83.4 83.3 66.6

MLP 86.2 82.7 80.9 20.5 81.5 80.9 80.6 61.9

SVM 83.9 78.5 83.8 16.1 83.9 83.8 83.8 67.6

OneR 85.9 80.9 85.5 13.8 86.2 85.5 85.5 71.7

Bagging 89.4 86.0 85.5 14.4 85.6 85.5 85.5 71.0

AdaBoost 91.8 90.6 84.7 16.1 84.8 84.7 84.6 69.3

LogitBoost 91.4 90.1 85.5 14.6 85.5 85.5 85.5 70.9

	ML performance metrics	Other performance metrics
RandomForest	91.3	90.2	84.3	16.7	84.6	84.3	84.1	68.5
K-NN	77.8	72.0	78.3	22.6	78.4	78.3	78.2	56.3
Bayes Net	90.3	89.5	86.0	15.2	86.6	86.0	85.8	72.3
Naive Bayes	87.0	86.7	75.3	27.3	77.8	75.3	74.3	52.2
Logistic	90.6	88.3	83.4	17.3	83.5	83.4	83.3	66.6
MLP	86.2	82.7	80.9	20.5	81.5	80.9	80.6	61.9
SVM	83.9	78.5	83.8	16.1	83.9	83.8	83.8	67.6
OneR	85.9	80.9	85.5	13.8	86.2	85.5	85.5	71.7
Bagging	89.4	86.0	85.5	14.4	85.6	85.5	85.5	71.0
AdaBoost	91.8	90.6	84.7	16.1	84.8	84.7	84.6	69.3
LogitBoost	91.4	90.1	85.5	14.6	85.5	85.5	85.5	70.9

For Taiwan dataset, Table 7 shows that Random Forest method succeeds in finding good results compared to the other classifiers where the ROC rate is equal to 76.6% and the PRC rate is equal to 82.2%. The same behavior can be seen when considering the other metrics as shown in Table 7. More precisely, the TPR rate is equal to 81.3%, the FPR rate is equal to 49.3%, the Precision rate is equal to 79.4%, the Recall rate is equal to 81.3%, the F-Measure is equal to 79.5% and the MCC rate is equal to 38.8%.

Table 7

Classifiers performances on Taiwan dataset

	ML performance metrics		Other performance metrics
Classifier	ROC%	PRC%	TPR%	FPR%	Precision%	Recall%	F-Measure%	MCC%
RandomForest	76.6	82.2	81.3	49.3	79.4	81.3	79.5	38.8
K-NN	60.9	70.1	73.2	.514	73.0	.732	73.1	22.0
Bayes Net	75.3	80.9	78.5	42.3	78.2	78.5	78.4	36.9
Naive Bayes	72.8	78.9	63.5	30.1	76.6	63.5	66.6	27.9
Logistic	72.3	79.1	80.8	60.0	79.1	80.8	76.9	32.9
MLP	74.3	80.4	80.4	45.4	79.0	80.4	79.4	38.7
SVM	60.5	70.9	80.8	59.9	79.1	80.8	76.9	32.8
OneR	64.4	73.0	81.9	53.1	80.2	81.9	79.3	38.9
Bagging	75.5	81.6	81.3	50.3	79.4	81.3	79.4	38.4
AdaBoost	75.0	80.1	81.9	52.4	80.2	81.9	79.4	39.3
LogitBoost	75.8	81.1	81.6	54.0	79.8	81.6	78.9	37.7

As shown in Tables 3 to 7, Bayes Net, Random Forest, AdaBoost and LogitBoost methods are comparable on the CS considered datasets. These four methods give better results than the other classifiers in term of ROC, PRC, TPR, FPR, Precision, Recall, F-Measure and MCC points of view.

4.2. Comparison of ML methods on Bankruptcy data

Tables 8 and 9 give the numerical results found when applying machine learning algorithms on the two bankruptcy datasets. From Table 8, we can see the performance of LogitBoost method on Polish dataset. It gives a ROC rate equals to 92.5% and a PRC rate equals to 96.9%. The TPR rate is equal to 95.2%, the FPR rate is equal to 47.8%, the Precision rate is equal to 94.7%, the recall rate is equal to 95.2%, the F-measure rate is 94.7% and the MCC rate is equal to 58.5%.

Table 8
ROC% versus PRC% on Polish dataset

ML performance metrics Other performance metrics

Classifier ROC% PRC% TPR% FPR% Precision% Recall% F-Measure% MCC%

RandomForest 87.9 95.1 93.7 67.8 92.4 93.7 92.5 38.9

K-NN 51.3 86.9 92.3 90.3 87.9 92.3 89.5 05.5

Bayes Net 82.2 92.9 83.3 35.9 91.5 83.3 86.5 31.5

Naive Bayes 58.2 87.7 91.8 85.2 88.5 91.8 89.7 11.4

Logistic 83.4 93.5 92.9 72.3 91.1 92.9 91.6 30.4

MLP 86.9 95.5 95.2 55.5 94.8 95.2 94.4 56.7

SVM 50.3 86.8 92.8 92.2 89.8 92.8 89.5 5.2

OneR 62.9 89.6 94.1 68.4 93.3 94.1 92.8 42.6

Bagging 87.2 95.2 94.2 67.1 93.4 94.2 92.9 44.1

AdaBoost 91.8 96.6 95.0 45.2 94.5 95.0 94.6 57.6

LogitBoost 92.5 96.9 95.2 47.8 94.7 95.2 94.7 58.5

	ML performance metrics	Other performance metrics
RandomForest	87.9	95.1	93.7	67.8	92.4	93.7	92.5	38.9
K-NN	51.3	86.9	92.3	90.3	87.9	92.3	89.5	05.5
Bayes Net	82.2	92.9	83.3	35.9	91.5	83.3	86.5	31.5
Naive Bayes	58.2	87.7	91.8	85.2	88.5	91.8	89.7	11.4
Logistic	83.4	93.5	92.9	72.3	91.1	92.9	91.6	30.4
MLP	86.9	95.5	95.2	55.5	94.8	95.2	94.4	56.7
SVM	50.3	86.8	92.8	92.2	89.8	92.8	89.5	5.2
OneR	62.9	89.6	94.1	68.4	93.3	94.1	92.8	42.6
Bagging	87.2	95.2	94.2	67.1	93.4	94.2	92.9	44.1
AdaBoost	91.8	96.6	95.0	45.2	94.5	95.0	94.6	57.6
LogitBoost	92.5	96.9	95.2	47.8	94.7	95.2	94.7	58.5

Table 9

Classifiers performances on Indian Credit dataset

	ML performance metrics		Other performance metrics
Classifier	ROC%	PRC%	TPR%	FPR%	Precision%	Recall%	F-Measure%	MCC%
RandomForest	100	100	100	00	100	100	100	100
K-NN	100	100	100	00	100	100	100	100
Bayes Net	100	100	100	00	100	100	100	100
Naive Bayes	100	100	100	00	100	100	100	100
Logistic	100	100	100	00	100	100	100	100
MLP	100	100	100	00	100	100	100	100
SVM	100	100	100	00	100	100	100	100
OneR	96.7	96.2	97.6	4.3	97.7	97.6	97.6	94.9
Bagging	99.9	99.9	97.6	4.3	97.7	97.6	97.6	94.9
AdaBoost	100	100	100	00	100	100	100	100
LogitBoost	100	100	100	00	100	100	100	100

From Table 9 considering the Indian dataset, we remark that almost the considered classifiers succeed in finding good results with a PRC and ROC rates equal to 100%. This is due to the nature of the dataset which is simple to classify. The number of the variables is small and equals to 7. The real Indian dataset is well balanced with 250 companies where 143 companies are non-bankruptcy and 107 companies as bankruptcy.

5. Conclusion

Credit scoring is a crucial problem for banks and financial institutions especially after the 2006 financial crisis. The financial institutions have to find good techniques to select counterparties. German, Australian, Polish, Taiwan, Japanese, Indian and Give Me Some Credit datasets are used in this research. The paper evaluates eleven techniques to distinguish between bad and good counterparties. We evaluated the k-Nearest Neighbor (k-NN), Random Forest, Bayes Network Classifier, Naive Bayes, Multi-layer Perceptron (MLP), Logistic classifier, Support Vector Machine (SVM), OneR, Bagging, AdaBoost and LogitBoost classifiers on seven datasets to measure their performance. Various measures are used in this study. We considered both ROC and PRC area measures. Also, we considered TPR, FPR, Precision%, Recall%, F-Measure% and MCC% measures to compare efficiently the machine learning algorithms for credit scoring.

The numerical study showed that Bayes Net, Random Forest, AdaBoost and LogitBoost machine learning classifiers produce efficient models for credit scoring.

In order to better visualize the behavior of the four best machine learning (Bayes Net, Random Forest, AdaBoost and LogitBoost), we draw the Figs 2 to 6. We compared the ROC and PRC as performance measures of the four classifiers on German, Australian, Japanese, Polish, Taiwan and the Give Me Some Credit datasets. From Figs 2 to 6, we visualized that the four classifiers are comparable on the considered datasets. They produce consistent results in term of both ROC and PRC which confirms the effectiveness of these techniques in credit scoring.

Fig. 1.

ROC% versus PRC% on German.

Fig. 2.

ROC% versus PRC% Australian datasets.

Fig. 3.

ROC% versus PRC% on Japanese.

Fig. 4.

ROC% versus PRC% Taiwan datasets.

Fig. 5.

ROC% versus PRC% on “Give Me Some Credit”.

Fig. 6.

ROC% versus PRC% on Polish datasets.

We conclude that the Bayes Net and Boosting classifiers in general are effective models for credit scoring. Their numerical results are competitive and demonstrate the benefit of the considered models for credit scoring and bankruptcy prediction.

As future work, it would be interesting to develop deep learning techniques to handle very huge or big datasets and improve performance. Like machine learning, deep learning is a technique for data analysis. It permits to learn from data, identify patterns and make intelligent decisions. However, deep learning is a combination of computing power and a special kind of neural networks such as recurrent neural networks that may be used to solve the problem end to end. Deep learning is used mainly to learn complicated patterns in large and voluminous amounts of data. When data is very huge, complex and diverse, the traditional machine learning techniques are unable to produce good results, hence, the need to use deep learning instead of machine learning. This is due to the supremacy of deep learning in terms of accuracy when trained with huge datasets. We note that “big data” is a very huge dataset collected from a variety of sources. It describes terabytes or petabytes of complex and diverse data captured over time and where the traditional machine learning techniques are inadequate to deal with them. In such situation, it is recommended to use deep learning techniques to handle such kind of data.

Footnotes

1

2

References

Abdou

and Pointon

, Credit scoring, statistical techniques and evaluation criteria: A review of the literature, Intelligent Systems in Accounting, Finance and Management 18(2–3) (2011), 59–88.

Abdou

H.A.

, Genetic programming for credit scoring: The case of Egyptian public sector banks, Expert Systems with Applications 36 (2009), 11402–11417.

Abellan

and Mantas

C.J.

, Improving experimental studies about ensembles of classifiers for bankruptcy prediction and credit scoring, Expert Systems with Applications 41 (2014), 3825–3830.

Bellotti

and Crook

, Support vector machines for credit scoring and discovery of significant features, Expert Systems with Applications 36 (2009), 3302–3308.

Breiman

Friedman

Olshen

and Stone

, Classification and Regression Trees, Wadsworth, Belmont, CA, 1984.

Breiman

, Bagging predictors, Machine Learning 24(2) (1996), 123–140.

Boughaci

and Alkhawaldeh

A.A.K.

, A cooperative classification system for credit scoring, in: Proceedings of AUEIRC 2017, Springer, 2017 , to appear.

Boughaci

and Alkhawaldeh

A.A.K.

, A new variable selection method applied to credit scoringn, Algorithmic Finance 7(1–2) (2018), 43–52 , IOS Press.

Cox

D.R.

, The regression analysis of binary sequences (with discussion), J. Roy Stat. Soc. B 20 (1958), 215–242 , JSTOR.

10.

Desay

Crook

J.N.

and Overstreet

G.A.

, A comparison of neural networks and linear scoring models in the credit union environment, European Journal of Operational Research 95 (1996), 24–37.

11.

Freund

and Schapire

R.E.

, A decision-theoretic generalization of on-line learning and an application to boosting, Journal of Computer and System Sciences 55 (1997), 119.

12.

Friedman

Geiger

and Goldszmidt

, Bayesian network classifiers, Machine Learning 29 (1997), 131–163.

13.

Friedman

Hastie

and Tibshirani

, Additive logistic regression: A statistical view of boosting, Annals of Statistics 28(2) (2000), 337–407.

14.

Milne

Rounds

and Goddard

, Optimal feature selection in credit scoring and classification using a quantum annealer (2017) https://1qbit.com/whitepaper/optimal-feature-selection-in-credit-scoring-classification-using-quantum-annealer/.

15.

Hand

D.J.

and Henley

W.E.

, Statistical classification methods in consumer credit scoring, Journal of the Royal Statistical Society, Series A (Statistics in Society) 160 (1997), 523–541.

16.

Henley

W.E.

and Hand

D.J.

, A k-nearest neighbour classifier for assessing consumer credit risk, Statistician 45 (1996), 77–95.

17.

Holte

R.C.

, Very simple classification rules perform well on most commonly used datasets, Machine Learning 11 (1993), 63–91.

18.

T.K.

, Random decision forests, in: Proceedings of the 3rd International Conference on Document Analysis and Recognition, Montreal, QC, 14–16 August 1995 1995, pp. 278–282.

19.

Wei

and Xu

, An evolution strategy-based multiple kernels multi-criteria programming approach: The case of credit decision making, Decision Support Systems 51 (2011), 292–298.

20.

Mester

L.J.

, What’s the point of credit scoring?, Business Review (September) (1997), 3–16.

21.

Miller

, Research confirms value of credit scoring, National Underwriter 107(42) (2003), 30.

22.

Powers

D.M.W.

, Evaluation: From precision, recall and F-Measure to ROC, informedness, markedness and correlation, Journal of Machine Learning Technologies 2(1) (2011), 37–63.

23.

Quinlan

J.R.

, Simplifying decision trees, Int. J. Man-Machine Studies (1987), 221–234.

24.

Quinlan

J.R.

, C4.5: Programs for Machine Learning, Morgan Kaufmann, San Mateo, CA, 1992.

25.

Rennie

Shih

Teevan

and Karger

, Tackling the poor assumptions of Naive Bayes classifiers, in: ICML 2003.

26.

Rosenblatt

Frank X.

, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms, Spartan Books, Washington DC, 1961.

27.

Vapnik

, Statistical Learning Theory, John Wiley and Sons, New York, USA, 1998.

28.

Waikato Environment for Knowledge Analysis (WEKA), Version 3.9. The University of Waikato, Hmilton, New Zealand, Software available at http://www.cs.waikato.ac.nz/∼ml/weka/: accessed: on November 2017.

29.

Wiginton

J.C.

, A note on the comparison of logit and discriminant models of consumer credit behavior, Journal of Financial and Quantitative Analysis 15 (1980), 757–770.

Appropriate machine learning techniques for credit scoring and bankruptcy prediction in banking and finance: A comparative study

Abstract

Keywords

1. Introduction

2. Classification and machine learning techniques

2.1. Problem formulation

Table 1 Description of the datasets (case of customers) used in the study Dataset #Loans #Good loans #Bad loans #variables Australian 690 307 383 15 German 1000 700 300 21 Japanese 690 307 383 16 Taiwan 30000 6636 23364 24 Give Me Some Credit 150000 139974 10026 11

4.1. Comparison of ML methods on CS data

Footnotes

1

2

References

Table 1
Description of the datasets (case of customers) used in the study

Dataset #Loans #Good loans #Bad loans #variables

Australian 690 307 383 15

German 1000 700 300 21

Japanese 690 307 383 16

Taiwan 30000 6636 23364 24

Give Me Some Credit 150000 139974 10026 11