Hybrid credit scoring model using neighborhood rough set and multi-layer ensemble classification

Abstract

Credit scoring is a procedure to estimate the risk related with credit products which is calculated using applicants’ credentials and applicants’ historical data. However, the data may have some redundant and irrelevant information and features, which lead to lower accuracy on the credit scoring model. So, by eliminating the redundant features can resolve the problem of credit scoring dataset. In this work, we have proposed a hybrid credit scoring model based on dimensionality reduction by Neighborhood Rough Set (NRS) algorithm and layered ensemble classification with weighted voting approach to improve the classification performance. For classifiers’ raking, we have proposed a novel classifier ranking algorithm as an underlying model for representing ranks of the classifiers based on classifier accuracy. It is used on seven heterogeneous classifiers for finding the ranks of those classifiers. Further five best ranking classifiers are used as base classifier in layered ensemble framework. Results of the ensemble frameworks (Majority Voting (MV), Weighted Voting (WV), Layered Majority Voting (LMV), Layered Weighted Voting (LWV)) with all features and after feature reduction by various existing feature selection algorithms are compared in terms of accuracy, sensitivity, specificity and G-measure. Further, results of ensemble frameworks with NRS are also compared in terms of ROC curve analysis. The experimental outcomes reveal the success of proposed methods in two benchmarked credit scoring (Australian credit scoring and German loan approval) datasets obtained from UCI repository.

Keywords

Weighted voting classification feature selection ensemble learning credit scoring

1 Introduction

Credit scoring is a way of calculate the risk associated with credit products [1] using applicants’ credentials (such as annual income, job status, residential status and etc.), historical data and statistical techniques. It tries to separate the effects of various applicant characteristics on criminal behavior and defaults. The main focus of credit score model is to determine whether credit consumer belongs to either legitimate or suspicious customer group. Credit scoring is not a single step process, it is done by financial institutions in many phases such as application scoring, behavioral scoring, collection scoring and fraud detection [2]. When a new application arrives for new credit, application scoring is done for the evaluation of the legitimateness or suspiciousness of new applicant. That evaluation is done on the basis of social, financial, and other data collected at the time of the application. Behavioral scoring is similar as application scoring, but it is for the existing customers to analyses the existing consumer’s behavior patterns to support dynamic portfolio management processes. Collection scoring is about to separate the customers into different groups (early, middle, late recovery), to put more, moderate or less level of attention on these groups. Fraud scoring models rank the applicants according to the relative probability that an applicant may be dishonest. In this paper, our focus is on the application scoring problem.

Many researchers such as [3 –5] have proposed bio-inspired algorithms for feature selection to improve the performance of the classifiers. Ping et al. (2011) [5], in this study a hybrid Neighborhood Rough Set and the Support Vector Machine (SVM) based classifier is proposed for credit scoring. Hu et al. [6, 7] have presented a neighborhood rough set model to resolve the problem of heterogeneous feature subset selection. Neighborhood relations are a kind of similarity relations, which satisfy the properties of reflexivity and symmetry and draw the objects together for similarity or indistinguishability in terms of distances and the samples in the same neighborhood granule are close to each other.

Conventional credit scoring models are based on individual classifiers or a simple combination of these classifiers which tend to show moderate performance. Many classifiers such as Naive Bayes (NB), SVM, Decision Tree (DT), Neural Network based classifiers and more have been proposed to learn problems thus far. However, all of them have their own positive and negative aspects. So they are good only for specific problems. But there is no specific way to recognize which classifier is a better or good classifier for a specific problem. Thus ensemble classifier is a strong approach to produce a near to optimal classifier for any problem [8]. This method reinforces the ensemble in error-prone subspaces, and hence it can lead to better performance for the classification. Generally the result of combination of diverse classifiers, is better classification [9, 10]. Basically there are two types of ensemble frameworks as homogeneous and heterogeneous ensemble frameworks [10, 11]. In homogeneous ensemble framework, the base classifiers of same type are used whereas heterogeneous ensemble framework is composed of base classifiers that belong to different types. Most popular ways to combine the base classifiers are as majority voting and weighted voting [9, 12]. A multi-layer classifiers ensemble framework [10, 11] is used based on the optimal combination of heterogeneous classifiers. The multi-layer model overcomes the limitations of conventional performance bottlenecks by utilizing an ensemble of five heterogeneous classifiers.

Rest of the paper is organized as follows: Section 2, describes our proposed work flow for credit scoring data classification. Section 3, presents experimental results obtained from the proposed model and relative performance of different feature selection techniques followed with ensemble classification performance. Further, it is followed by concluding remarks and references.

2 Proposed methodology

Here we have proposed a hybrid approach which combines the NRS approach for feature selection and layered ensemble classification as in Fig. 1. Our proposed approach for credit scoring works on three phases, first phase for finding the rank of the classifiers and weights assigning to these classifiers, second is associated with feature selection using NRS algorithm and third phase is and construction of the ensemble framework. Preprocessed data which includes the data cleaning, data transformation and data normalization is used for rank assignment to classifier and feature selection phase. Detailed descriptions about proposed hybrid approach is given in following subsections.

Fig.1

Proposed work flow for credit scoring.

2.1 Phase-1: Assignment of rank and weights to classifiers

There is no specific way to recognize which classifier is a better or good classifier for a specific problem. Thus ensemble learning is a strong approach to produce a near to optimal classifier for any problem. In this phase seven classifiers as Naive Bayes (NB), Multilayer Feed Forward Neural Network (MLFFNN) and Decision Tree (DT), Quadratic Discriminant Analysis (QDA), Time Delay Neural Network (TDNN), Deep Tenser Neural Network (DTNN), Decision Tree (DT) and Probabilistic Neural Network (PNN) are initially utilized to find the rank of each classifier. Equation (1) is used as an underlying model for representing ranking of the classifiers. Further the five classifiers with best ranking are arranged as in Fig. 1. In case of multi-layer ensemble framework C₁ and C₂ with highest ranking are at the second layer and rest of three classifiers are in first layer.

In this work we have used weighted voting approach with heterogeneous multi-layer ensemble framework. In case of weighted voting a weight is assigned to each base classifiers. For weight assignment, classification accuracy is used as parameter to calculate the weights for the classifiers. A classifier with the highest accuracy is assigned the highest weight and the lowest weight is assigned to a classifier with the lowest accuracy. These weights are calculated by the Equation (1). Initially equal weights are assigned to each base classifier, then dataset is applied for classification to calculate the accuracy. Further the weights are updated (Equation (1)) and this procedure is repeated for n iteration and mean of the updated weights of n iterations are assigned to the respective classifier. $W_{U}^{i, j} = W_{o}^{i, j} + \frac{1}{2} \log (\frac{{Acc}_{j}}{1 - {Acc}_{j}})$ (1)

Where, $W_{U}^{i, j}$ and $W_{o}^{i, j}$ represent the weight updated and old weight at ith iteration for jth classifier and Acc_j represents the accuracy obtained by jth classifier.

2.2 Phase-2: Feature selection

Feature selection is a standout amongst the most basic issues in the field of machine learning. The main aim of feature selection is to determine a minimal subset of features from a set of features. In other words, feature selection is a process of finding a subset of features that ideally is necessary and sufficient to describe the target concept from the original set of features in a given dataset. Here we have used a NRS feature selection approach to select the best features from a feature set. Complete algorithm is described in algorithm (1) [6 , 13].

Algorithm 1 NRS algorithm for feature selection.

Input: Hybrid decision table <U, A^c∪ Aⁿ ∪ D > where U is the sample set, called the universe, A^c and Aⁿ are categorical and numerical attributes, respectively, β is the threshold for computing variable precision lower approximations and k is size of k-nearest neighborhood;

Output: One reduct red.

1: ∀a ∈ A^c: compute equivalence relation R_a;

2: ∀a ∈ Aⁿ: compute neighborhood relation N_a or k_a;

3: φ → red:red is the pool to contain the selected attributes;

4: For each a_i ∈ A - red;

Compute SIG(a_i, red, D) = $γ_{red \cup a_{i}}^{β} (D) - γ_{red}^{β} (D)$

:where $γ_{φ}^{β} (D) = 0$

end

5: select the attribute a_k which satisfies:

SIG(a_k, red, D)= maxi (SIG(a_i, red, D)

6: if (SIG(a_k, red, D) >0)

red ∪ a_k → red

Goto step 4;

else

return (red);

7: end

Table 1
Descriptions about datasets used

Dataset Samples Classes Features Class-1/Class-2 Categorical/Numerical

Australian 690 2 14 383/317 6/8

German 1000 2 20 700/300 13/7

Dataset	Samples	Classes	Features	Class-1/Class-2	Categorical/Numerical
Australian	690	2	14	383/317	6/8
German	1000	2	20	700/300	13/7

2.3 Phase-3: Ensemble framework

Proposed ensemble framework is as in Fig. 1, where C₁, C₂, C₃, C₄ and C₅ are the classifiers, which are chosen as best classifiers among seven heterogeneous classifiers in phase-1. Data with selected features is fed with weights assigned to respective classifier for evaluation of the final results against the input samples. Further the five classifiers with best ranking are arranged as in Fig. 1. In this framework C₁ and C₂ with highest ranking are at the second layer and rest of three classifiers are in first layer. Combiner-1 aggregates the results obtained by three classifiers associated with it and combiner-2 aggregates the results obtained by two classifiers associated with it and result obtained by combiner-1.

Combiner-1 and combiner-2 aggregates the output predicted by associated classifiers using the Equation (2). $O = W_{1} * X_{1} + W_{2} * X_{2} + . . . + W_{n} * X_{n}$ (2)

Where, W_i and X_i are the weight and predicted output of the ith classifier respectively.

3 Experimental results

This section have four sub section which covers about datasets used in this work, performance measures used to show the proof of proposed work, feature selection results by various feature selection approaches along with results obtained by various ensemble framework and ROC curve analysis.

3.1 Datasets used in experiment

Australian dataset and German dataset (categorical) are used in this work. These datasets are acquired from the UCI Machine Learning Repository [14, 15]. All datasets have combination of attribute types continuous and nominal. Australian dataset is related to credit approval and German dataset is related to loan application. To protect confidentiality of the data, the values of some attributes are replaced by random meaningless symbols. Detailed description about datasets are illustrated in Table 1.

3.2 Performance measures

There are several measures to evaluate the classification measures commonly available in the literature. Accuracy (Equation 3) is not sufficient as a performance measure, if there is significant class imbalance towards a class in the dataset. Dataset used in this experimental work is binary class dataset having the positive (credit approved) and negative (credit not approved) classes. Sensitivity (Equation 4) represents the accuracy of only positive samples prediction and specificity (Equation 5) is about the negative samples prediction accuracy. G-measure (Equation 6) is a measure of a test’s accuracy. It considers both the positive and negative accuracies of the test to compute the score and can be interpreted as geometric mean of sensitivity and specificity. It would be 1 at best case and 0 at worst case. Four performance measures defined with respect to the confusion matrix are as True Positive (TP), False Positive (FP), True Negative (TN) and False Negative (FN), where positive corresponds to credit approved and negative corresponds to credit not approved cases. Observed positive and actual positive is TP, observed negative and actual negative is TN, observed negative and actual positive is FP, observed positive and actual negative is FN. $Accuracy = \frac{TP + TN}{TP + TN + FP + FN}$ (3) $Sensitivity = \frac{TP}{TP + FN}$ (4) $Specificity = \frac{TN}{TN + FP}$ (5) $G - measure = \sqrt{sensitivity * specificity}$ (6)

3.3 Results and analysis

The experimental results described in this section are performed on a DELL PC with a 3.60 GHz Intel Core I7 vPRO CPU, 8 GB RAM and 64 bit Windows 7 operating system. Implementation is done using Matlab R2012a. As per the proposed model, preprocessed dataset is applied in layered ensemble framework for the classification. In this work, preprocessing consists of four main steps as treatment for the missing values, transformation, normalization and feature selection. Results of feature selection on Australian and German dataset by various approaches such as Stepwise Regression (STEP), Classification and Regression Tree (CART), Correlations (CORR), Multivariate Adaptive Regression Splines (MARS), T-Test and NRS are presented in Tables 2 and 3 respectively.

Table 2
Selected Features in Australian dataset

Method Features No of Features

STEP 7, 8, 9, 12, 14 5

CART 5, 7, 8, 9, 10, 14 6

CORR 2, 3, 4, 7, 8, 9, 10, 14 8

MARS 3, 5, 7, 8, 10, 14 6

T-Test 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14 12

NRS 8, 9, 14 3

Method	Features	No of Features
STEP	7, 8, 9, 12, 14	5
CART	5, 7, 8, 9, 10, 14	6
CORR	2, 3, 4, 7, 8, 9, 10, 14	8
MARS	3, 5, 7, 8, 10, 14	6
T-Test	2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14	12
NRS	8, 9, 14	3

Table 3

Selected Features in German dataset

Method	Features	No of Features
STEP	1, 2, 3, 7, 8, 15, 20	7
CART	1, 2, 3, 4, 5, 9, 10, 18	8
CORR	1, 2, 3, 5, 6, 7, 12, 13, 14, 15, 20	11
MARS	1, 2, 3, 4, 5, 6, 9, 15, 16, 17, 20	11
T-Test	1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 20	15
NRS	1, 2, 3, 4, 6, 7, 8, 9, 11, 12, 13, 14	12

In this work, we have used weighted voting approach. So, weights are assigned to classifiers are calculated as described earlier section in the both case (five classifiers are combined with weighted voting in single and multi-layered approach). Table 4 represents the weights assigned to classifier on single layer approach where W₁, W₂, W₃, W₄ and W₅ are the weights assigned to classifiers C₁, C₂, C₃, C₄ and C₅ respectively with respective dataset. Table 5 represents the weights assigned to classifier on multi-layer approach where W₁₁, W₁₂, and W₁₃ are the weights assigned to C₃, C₄ and C₅ (classifiers at layer 1) respectively, further W₂₁, W₂₂, and W₂₃ are the weights assigned results obtained by combiner-1, C₁ and C₂ respectively with respective dataset.

Table 4

Weights assigned to classifiers in single layer ensemble framework

Dataset	W ₁	W ₂	W ₃	W ₄	W ₅
Australian	0.1644	0.1381	0.2589	0.2365	0.2020
German	0.2185	0.1358	0.2254	0.2290	0.1912

Table 5

Weights assigned to classifiers in layered ensemble framework

Dataset	W ₁₁	W ₁₂	W ₁₃	W ₂₁	W ₂₂	W ₂₃
Australian	0.4084	0.2298	0.3618	0.2253	0.3861	0.3887
German	0.5837	0.1084	0.3079	0.1822	0.3728	0.4450

To make balanced 10-Fold Cross Validation (10-FCV), in each fold we arranged similar number of samples towards each class because German dataset is imbalance dataset. The 10-FCV, first the whole dataset is divided into two parts data-1 and data-2 which contains the samples of class 1 and class 2 respectively. Further data-1 is divided into 10 parts and data-2 is also divided into 10 parts. Data-1 part-1 and data-2 part-1 is considered as fold-1 similarly fold-2 and so on. Because classification algorithms suffer when the class distribution is imbalanced towards one of the classes [16]. In this section, we compare the average of 10-FCV on two credit scoring data sets results obtained from our proposed work with existing feature selection in terms of classification accuracy, sensitivity, specificity and G-measure. These results are presented in Tables 6 and 7 for Australian dataset and German dataset respectively.

Table 6

Performance comparisons on Australian dataset

Method	Ensemble Approach	Accuracy	Sensitivity	Specificity	G-measure
ALL	MV	90.92	95.70	85.79	90.61
	WV	90.92	95.70	85.79	90.61
	LMV	90.82	93.57	88.19	90.84
	LWV	93.02	98.80	85.86	92.10
STEP	MV	91.70	92.58	90.73	91.65
	WV	91.70	92.58	90.73	91.65
	LMV	93.59	97.18	89.79	93.41
	LWV	93.92	99.38	88.13	93.59
CART	MV	91.38	95.37	87.13	91.16
	WV	91.38	95.37	87.13	91.16
	LMV	93.04	97.89	87.86	92.74
	LWV	93.80	99.51	87.52	93.32
CORR	MV	92.81	95.96	89.46	92.65
	WV	92.81	95.96	89.46	92.65
	LMV	93.20	95.73	90.46	93.06
	LWV	94.74	98.78	90.46	94.53
MARS	MV	91.52	95.34	87.46	91.32
	WV	91.52	95.34	87.46	91.32
	LMV	92.83	96.90	88.52	92.62
	LWV	93.45	99.06	87.52	93.11
T-Test	MV	91.22	94.12	88.13	91.08
	WV	91.22	94.12	88.13	91.08
	LMV	92.65	94.06	91.19	92.61
	LWV	93.29	98.44	87.86	93.00
NRS	MV	92.37	94.45	90.13	92.26
	WV	92.37	94.46	90.13	92.27
	LMV	94.93	97.84	91.86	94.80
	LWV	95.39	99.69	90.86	95.17

Table 7

Performance comparisons on German dataset

Method	Ensemble Approach	Accuracy	Sensitivity	Specificity	G-measure
ALL	MV	79.36	92.34	51.67	69.07
	WV	79.37	73.97	69.24	71.56
	LMV	80.53	80.47	80.67	80.56
	LWV	84.26	93.62	71.00	81.53
STEP	MV	80.10	92.97	52.67	69.98
	WV	80.11	74.88	69.75	72.27
	LMV	82.98	83.91	81.00	82.44
	LWV	85.68	94.38	74.00	83.57
CART	MV	78.08	95.31	41.33	62.76
	WV	78.08	95.31	41.33	62.76
	LMV	85.85	86.72	84.00	85.35
	LWV	83.83	96.72	56.33	73.81
CORR	MV	78.62	91.88	50.33	68.00
	WV	78.62	91.88	50.33	68.00
	LMV	76.81	77.34	75.67	76.50
	LWV	82.55	94.69	56.67	73.25
MARS	MV	79.89	94.37	49.00	68.00
	WV	79.89	94.38	49.00	68.00
	LMV	82.77	83.28	81.67	82.47
	LWV	84.04	96.09	58.33	74.87
T-Test	MV	80.53	93.28	53.33	70.53
	WV	80.53	93.28	53.33	70.53
	LMV	83.94	83.75	84.33	84.04
	LWV	84.64	97.50	60.33	76.70
NRS	MV	77.98	92.66	46.67	65.76
	WV	77.98	92.66	46.67	65.76
	LMV	82.98	83.28	82.33	82.80
	LWV	86.47	98.78	72.33	84.53

Starting with the Australian dataset regarding accuracy, (sensitivity, specificity) and G-measure, the proposed approach achieves 95.39%, 99.69%, 90.86% and 95.17% respectively. In German dataset regarding accuracy, sensitivity specificity and G-measure, the proposed approach achieves 86.47%, 98.78%, 72.33% and 84.53% respectively. As per results in Tables 6 and 7, results obtained by proposed approach are the best in terms of accuracy, sensitivity and G-measure on both datasets.

3.4 ROC curve analysis

To validate the separation and discrimination ability of the proposed models and to measure their performance from a different perspective as well as measuring their sensitivity and specificity over various thresholds, ROC curves are depicted for the MV, WV, LMV, and LWV approaches. Figures 2 and 3 display the ROC curves for all the above-mentioned ensemble approaches with feature selected by NRS on datasets as aforementioned on fold-1 (first 90% as training and rest for the test dataset).

Fig.2

Fold-1 ROC on Australian dataset.

Fig.3

Fold-1 ROC on German dataset.

In case of Australian dataset, all other ensemble classifiers’ curve lies below LWV ROC curve for all threshold values. So it shows that for Australian dataset the LWV method is the best for all the required values of sensitivity and specificity. In case of German dataset, conclusions are same as for the Australian dataset.

4 Conclusion

In this paper, a hybrid approach based on NRS for feature selection and multi-layer ensemble classification which are aggregated with weighted voting approach is proposed for credit scoring. Further the results of proposed approach is compared with MV, WV and LMV with all features and with selected features by STEP, CART, CORR, MARS are compared in terms of accuracy, sensitivity, specificity and G-measure on two benchmark dataset Australian and German dataset. According to results obtained by the proposed approach outperformed all other approaches in terms of accuracy, sensitivity and G-measure on Australian and German dataset. Results obtained by the proposed approach outperformed by comparing to other approaches such as MV, WV, LMV in terms of ROC curve analysis on Australian and German dataset.

References

Mester

L.J.

, et al., WhatâĂŹs the point of credit scoring?Business Review3(Sep/Oct) (1997), 3–16.

Paleologo

, Elisseeff

and Antonini

, Subagging for credit scoring models, European Journal of Operational Research201(2) (2010), 490–499.

Chi

B.-W.

and Hsu

C.-C.

, A hybrid approach to integrate genetic algorithm into dual scoring model in enhancing the performance of credit scoring model, Expert Systems with Applications39(3) (2012), 2650–2661.

Huang

C.-L.

and Dun

J.-F.

, A distributed pso– svm hybrid system with feature selection and parameter optimization, Applied Soft Computing8(4) (2008), 1381–1391.

Oreski

and Oreski

, Genetic algorithm-based heuristic for feature selection in credit risk assessment, Expert systems with Applications41(4) (2014), 2052–2064.

, Yu

, Liu

and Wu

, Neighborhood rough set based heterogeneous feature subset selection, Information Sciences178(18) (2008), 3577–3594.

, Liu

and Yu

, Mixed feature selection based on granulation and approximation, Knowledge-Based Systems21(4) (2008), 294–304.

Parvin

, MirnabiBaboli

and Alinejad-Rokny

, Proposing a classifier ensemble framework based on classifier selection and decision tree, Engineering Applications of Artificial Intelligence37 (2015), 34–42.

Ala’raj

and Abbod

M.F.

, Classifiers consensus system approach for credit scoring, Knowledge-Based Systems104 (2016), 89–105.

10.

Bashir

, Qamar

and Khan

F.H.

, Intellihealth: A medical decision support application using a novel weighted multilayer classifier ensemble framework, Journal of Biomedical Informatics59 (2016), 185–200 .

11.

Bashir

, Qamar

, Khan

F.H.

and Naseem

, Hmv: A medical decision support framework using multi-layer classifiers for disease prediction, Journal of Computational Science13 (2016), 10–25.

12.

Ala’raj

and Abbod

M.F.

, A new hybrid ensemble credit scoring model based on classifiers consensus system approach, Expert Systems with Applications64 (2016), 36–55 .

13.

Ping

and Yongheng

, Neighborhood rough set and svm based hybrid credit scoring classifier, Expert Systems with Applications38(9) (2011), 11300–11304.

14.

“Australian dataset.” [Online]. Available: https://archive.ics.uci.edu/ml/machine-learningdatabases/statlog/australian/

15.

“Statlog (german credit data) data set.” [Online]. Available: https://archive.ics.uci.edu/ml/datasets/Statlog+(German+Credit+Data)

16.

Van Hulse

, Khoshgoftaar

T.M.

and Napolitano

, Experimental perspectives on learning from imbalanced data, in Proceedings of the 24th international conference on Machine Learning ACM, 2007, pp.935–942.