Credit scoring for microfinance using behavioral data in emerging markets

1.1 Motivation

This work is based on a real business problem from a microlender based in an emerging market in the Sub-Saharan Africa region. This company believes that technology is the best way to deal with daily problems usually present in emerging markets. Embracing this philosophy, the MFI has built a novel approach to micro-credit. This approach, alongside the challenge the company needs to face, create the perfect scenario for the use of data mining technique and Big Data.

The MFI is entirely digital, interacting with the customers by a mobile application. Through the application, customers do all the process from the loan request to the payment of it, generating useful information on each step. All starts when the customer register as a customer of the MFI, on this stage the customer needs to fill a form on which the demographics and personal information is captured. Added to the information the customer provides, the MFI also obtain access to the information of the Mobile Network Operator (MNO). The data from the MNO contains information about services used by the customer (calls and messages). This data is also rich for the use of data mining since it allows to create and study the social network of the customers of the MFI. After the customer provides the MFI with the information requested and apply for microcredit, the MFI faces another challenge, which is to estimate the risk of the applicant in order to grant or deny the loan request. This challenge will be the main problem to tackle during this study.

1.2 Problem definition

As presented before, the case of study is based on a branchless microlender operating in an emerging market that is built upon an android application. As seen on Fig. 1, the highlighted node it’s the one on which the approval or denial of the loan based on customer data is made. Currently, this process is done by humans operators taking up to 10 minutes per application. At the moment, the daily request of loan applications surpasses the capacity of evaluation of the operations team. This situation creates a queue for the application that can translate into weeks from the time of the application to the evaluation of the application. For the time that the loan is disbursed, the customer already borrows the money from another lender. Another issue in this process is the high default rate. The default rate is calculated based on the customers that have not paid at the end of the recovery process.

OPEN IN VIEWER

In this paper, we try to solve these problems by implementing a decision support system “score” to evaluate the loans on less than 1 minute and able to process multiple applications at the same time. With the credit scoring implementation, not only the time of evaluation will decrease substantially but also the company can focus on reaching more customers increasing the volume of the business. Also, the decision support system is expected to perform better than human evaluation in terms of default rate, a successful model should reduce the current default rate by at least 50%.

3.1 Structure of datasets

This section describes the datasets used to understand the behavior of our customers. Several datasets are used for experimentation. Even though all datasets contain the same features, the transformation process, selection process and time horizon considered are different. In Fig. 5 we present the process of Extraction, Transformation and Loading (ETL) of data used to create the base for each dataset.

OPEN IN VIEWER

With this process flow, we created the first dataset, denominated ETL dataset. This dataset is used for the model selection that will be presented in Section 3.4.

The second dataset, from now on called dataset A, covers all the first loans granted to the customers. Only the completed loans were considered, meaning that only paid loans and loans which have passed their respective due date were used. The third dataset, called dataset B, consists of all loans granted and not only the first loan of each customer. As in dataset A, only completed loans were considered when generating this dataset.

Following the structure of dataset B we created two more datasets, dataset C, and dataset D. As seen in Fig. 6, dataset C and dataset D consider different time windows. Dataset C considers the application of the previous three months while dataset D expand this time horizon up to five months. Aforementioned will help us to test the hypothesis that models will require more fresh data to predict more accurately in this fast growing market. Notice that only dataset A considers only the first application of each customer. All others datasets contain all type of loans. This approach was initially used because the scoring models will only be used in the first request of each customer. However, the conditions of the loans are the same regardless the number of previously granted loans. This means that a new customer has the same interest rate, same maximum amount to request and same maximum length to request as a customer who had paid five loans already. Based on this, we decided to consider all applications in order to increase the sample size for datasets B, datasets C, and datasets D.

OPEN IN VIEWER

All datasets combine demographics of customers, personal information, loan details, mobile network usage and mobile device features. We also included the performance of the loan (overdue or not) which is the target of prediction. In our scenario, a Non-Performing Loan (NPL) is a loan that has passed more than three days after due date and has not been totally repaid. Therefore this loan is considered as an overdue in the dataset and marked with 1 in the target variable.

The dataset A consists of 930 cases with 38.5% overdue loans and 61.5% paid on time loans. The datasets B, C and D contain 1,140 cases each with 43.5% overdue loans and 56.5% paid on time loans. The summarized variables on each dataset use a 30 days window, using the previous 30 days from the moment of the loan application.

The variables can be grouped by source. First, we focus on personal information of the customer. This personal information, including demographics, is collected when the customer opens the mobile app for the first time. In this first interaction with our service, the user also needs to fill a profile. Some of the variables in the profile can be changed later on, e.g. employment status. This information can provide better insights about the acquisition level of the customers. Some variables refer to the goods the customers have (e.g., house, car) and details about their employment status (e.g., time employed, industry, role). It also collects information financial load of the customer as it collects the marital status and number of children. This type of data has been considered since the building of scorecards as in Schreiner [25] to the use of advanced classification techniques as shown in Van Gool et al. [30], San Pedro et al. [21], Blanco et al. [5], among others.

We also collected mobile phone network usage variables. These variables capture whenever the customer uses one of the services provided by the MNO, e.g. incoming or outgoing calls, SMS, Multimedia Messaging Service (MMS), etc. It has been proven that being a customer of a given MNO can have an impact when building a scoring model [3]. The hypothesis is that high tier customers will relate with high tier MNO. From the mobile phone, we also collect system information, e.g. mobile applications installed. With the categories of the applications installed in the device, we can create a complete profile of the customers. The applications installed work as the “likes” and “shares” presented in [12]. Providing an idea of the real interests of the customers.

Finally, we add loan characteristics and conditions: length, amount, and purpose of the loan.

Table 1 shows in detail the list of the features used. This list presents only the core variables, we did feature engineering to create the following variables:

•

debt_ratio = amount requested/monthly income

The notation 0–30 days refers to the number of calls made in the last 30 days prior to applying for a loans, the same logic applies to 31–60 days which refers to the number of calls made from 31 to 60 days prior to applying. Note that for building the income_split we have used number of dependants which is calculated as 1 + number of children as it considers that the applicant depends on itself. Some previous works also have a similar selection of variables. Such is the case of the MobiScore [21], which considers number of SMS and number of calls as relevant features. Also, Bjorkegren and Grissen [4] conclude that the use of mobile data usage patterns can be a valuable input to build a scoring model even when lacking formal financial history. Moreover, these variables were analyzed with independence tests in order to identify any relationship between variables. Chi-Square independence test was conducted for categorical variables which reduced as seen on 3. Variables like the level of education did not have enough impact on the target variable, therefore was not taken into account.

In this section we present the description of some variables alongside some of the analysis done previous to modelling. Note that due to business matters, we do not present absolute values, we present relative values of each variable.

Our first approach included a partial description of the population based on demographics. The population met some of the expectation from the business experts based in the emerging markets. The customers are mostly males between 26 to 35 years that are settled in the evaluated region since only 10.3% live at their current address for less than 2 years. The customers also present low level of dependants, since 53.46% are single and 56.98% do not have children.

The population reflected some of the realities of the emerging markets. One of these realities is that most of the customers own a car but do not own a house. Cars in emerging markets are priority over residence, mostly because people need to cover long distances to get the goods and services they need. Furthermore, 38.2% of the customers are self employed and they might use their vehicle for their respective business. However, in terms of employment customers do not seem very steady, we observe a high proportion of customers have less than one year at their current employment.

We found high levels of education in our population with 87.5% of the customers at university level. However, this does not seem to match regarding salaries, as we focus in microcredit and the microloan represents between 10%–30% of the monthly salary for 69.19% of the customers.

In our case, the missing values were replaced by zeros. This can be easily explained with the variable Date of Employment. The variable is only available to customers who have chosen “Employed” in the Employment status variable in the previous step, as is mandatory to fill all the fields before applying we know that the missing means that it does not apply to that customer. Afterwards, we transformed date-type variables into numeric variables, e.g. Date_of_Birth → Age. And finally, we filtered the dataset (by columns) in order to remove the variables that are useless for our purpose, e.g. Customer_ID. Find in Table 2 a more detailed descriptions of the numerical features.

After doing a general overview of the population we went into further exploration, we analyze relations between the categorical variables and the target variable. We applied discretization to age and debt_ratio (age_intervals and debt_ratio_intervals) in order to analyze any kind of existing relation.

For the evaluation of the categorical variables we used a balanced sample and evaluated the sample by Pearson Chi-square statistic and Cramer’s V. These test are based contingency tables an will help us identify if there is any relation between variables and the strength of that relationship. We did the tests considering alpha equal to 0.05.

Nonetheless, the relationship strength obtained by Crammer’s V is relatively low. We did not find significant relation between the debt ratio and the target variable due to the constrains in the amount requested by customers. The customers cannot request pass a certain amount of monetary unit and most of them request the maximum amount allowed. Since most of the customers request the same amount then the variable turns into a generic variable such as gender and education level. Notice that to these categorical variables we added other categorical variables. We show the analysis of the personal information only, same type of test were used for the MNO related features and the loans related features. With a defined set of variables, selected through statistical tests, we formed a dataset that was use for the exploration of the algorithms.

As in formal banking, we wanted to predict the likelihood that a customer will repay or not the loan requested. We had a mix of numerical and categorical variables as seen in the Section 3. However, we did not have a defined algorithm to deal with this problem. Over the last few years, different classification techniques have shown success in a diversity of scenarios related to credit scoring.

This section reviews the different implementations used to deal scenarios related to risk prediction. First, the MobiScore [21] shows that Decision Tree (DT), SVM and LR algorithms can create reliable scores when lacking of financial history. The main goal of the MobiScore is to reduce the default rate in credit cards for customers in emerging markets. For this task, MobiScore train classification algorithms mostly with mobile phone data gathered from the customers. The MobiScore concludes that mobile phone data can indicate the financial risk of an individual without the needing bureau information which is the core component in formal banking and developed economies.

In other case, Blanco et al. [5] show that Artificial Neural Networks (ANN) outperformed the traditional techniques for scoring. The ANN model was more accurate when predicting defaulters. The ANN was trained with both financial and non-financial from a Peruvian MFI. As in our case, the primary goal was to reduce the type II error which in this context is to classify a customer with bad credit as a customer with good credit, therefore granting financial products. The borrowers considered in Blanco et al. [5] are micro-entrepreneurs. Financial data from their micro-enterprise is used as input to train the scoring models.

Bjorkegren and Grissen [4] highlight how useful can mobile usage data be when used for credit scoring. For this case, the use of Random Forest (RF) proved to be accurate when differentiating between good customers and bad customers. The dataset used to train the RF contains borrowers from an emerging market in the Caribbean area. This dataset also lack traditional financial data used to train scoring models in developed countries.

Bayesian Networks (BN) have also been used for credit scoring and outperformed traditional techniques [3]. Although the implementation of the BN algorithm was used in formal banking in Turkey, the variables associated with the building of the bayesian net are not entirely financial. Variables that relate the customer with a given MNO were used to identify good borrowers, presenting once again that non-traditional data makes an impact while creating a reliable credit score.

With this evidence, there is not an algorithm that outperforms the others. Each scoring technique has been able to succeed when dealing with non-traditional data to predict financial risk in emerging markets. This is an indicator that the financial risk of an individual can be estimated using pure behavioral data when the financial data is weak or existent. Then, in order to select a model, we decided to train several algorithms with the same dataset and focus their performance in the test set.

To compare the performance of the models, we focused on two metrics: the Area under Receiver Operating Characteristic (AUROC) curve as a measure of the accuracy and the true positive rate, where overdue is the positive class. The true positive rate can be measure later on when applied to real cases as will be presented in the presentation of results. To compute the metrics we turned the outcome into a probability and compare to a threshold T = 0.5. We used 10-fold cross validation to test the performance and grid-search for setting the hyper-parameters. For the SVM, the kernel had domain { ‘linear’ , ‘rbf’ }, the parameter C had domain { 0.1 , 0.2 , … , 1.0 } and parameter gamma was from 10 - 6 to 10 - 1 . The algorithms were built using Weka Experimenter 3.8.0.

As seen in Table 4, the top performers were LR, SVM and ANN. To build the ANN we tested from 1 hidden layer to N-hidden layers, being N the number of features in the input. However, the best performing ANN was built with 2 hidden layers. From these algorithms, we selected LR, SVM. This selection is the result of optimizing the cutoff of the algorithms, SVM prediction improved more than ANN with a sightly movement of the cutoff. In our case, we would prefer to have a model with better prediction of the bad customers even if this means to lose some prediction power in the good customers as the cost of default overweight the cost of opportunity.

5.1 First experiment: Implementing dummy variables

For the first experiment, we used dataset A. Our first step was to extend the dataset A by transforming the categorical variables into several (binary/dummy) variables. For variables with N categories we generated N - 1 new variables, e.g. “Gender”:{‘Male’, ‘Female’} was turned into “Male”:{‘1’, ‘0’}. Date variables were transformed into time unit elapsed since the date referenced, e.g., date of birth turned into age.

In the second step, we applied an information gain ranker in order to select the variables that contribute to the classification task and drop those that create noise. We performed this selection using Weka Explorer 3.8.0, using both InfoGain and GainRatio as attribute evaluator with the Ranker method. The final step was to build the model itself, for this task we applied supervised learning algorithms to the transformed dataset to create the models.

Following the results presented in Table 4, the two primary classification methods that we focus on our experiments are LR and SVM. Not only LR and SVM performed better regarding AUROC but also these models make no assumption on the distribution of the input data, meaning they are quite tolerant to the input received [27]. We used 10-fold cross validation to test the performance of the algorithms using a given dataset. This validation technique helps us to maximize the use of the data as it uses each case both for training and validating. Through cross validation, we can also avoid overfitting issues. Our main metrics of evaluation were accuracy and approval rate. The recall is linked to the performance of the overdue class in the target variable. We choose these metrics so we can compare it with the baseline defined in the previous evaluation process. This comparison is key to know if we solved the main issue stated in Section 1. In our use case, we preferred to reduce the accuracy of the prediction of good borrowers in order to have better predictions in the overdue class; this approach is also presented in Blanco et al. [5]. For each algorithm, we used grid-search for setting the hyperparameters in order to optimize the evaluation measures. The output for each loan O i was then transformed into a probability 𝑃 i following the expression:

P i = e O i e O i + 1(3)

The models were applied to the new incoming loan applications on which the probability P for each new application was computed by the model and compared with a given threshold. The models evaluated only applications from customers who applied for the first time and customers with only one paid loan before.

The second experiment was based on dataset B. In order to obtain comparable results, we tested LR and SVM algorithms again. For this dataset we used the WoE coding of variables. The WoE focuses on the odds ratio. Assume a variable X with domain { X 1 , … , X i } The WoE of X i is computed as:

𝑊𝑜𝐸 ⁢ ( X i ) = 𝑙𝑛 ⁢ ( 𝑇𝑜𝑡𝑎𝑙𝐺𝑜𝑜𝑑𝑠 ⁢ ( X i ) 𝑇𝑜𝑡𝑎𝑙𝐵𝑎𝑑𝑠 ⁢ ( X i ) )(4)

In Eq. (4), TotalGoods( X i ) refers to the number of borrowers that paid on time for category i in variable X . The same concept applies to TotalBads( X i ) but considering the borrowers that went overdue. Note that the value of the WoE is 0 when both the distribution of the good class and the distribution of the bad class are equal. It indicates that the category evaluated does not allow to differentiate between classes. Using the recoded dataset, we calculated the Information Value (IV) of each variable. The IV for a category i of a variable X with n number of categories is as follow:

𝐼𝑉 ⁢ ( X i ) = 𝑊𝑜𝐸 X i ∑ i = 1 𝑛 𝑊𝑜𝐸 X i(5)

The IV for variable X is the sum of the IV of each category i of X . It can be calculated as IV(X):

IV(X) = ∑ i = 1 i [ ( 𝑇𝑜𝑡𝑎𝑙𝐺𝑜𝑜𝑑𝑠 i - 𝑇𝑜𝑡𝑎𝑙𝐵𝑎𝑑𝑠 i ) * 𝑙𝑛 ⁢ ( Total Goods ⁢ ( X i ) Total Bads ⁢ ( X i ) ) ](6)

After obtaining the IV for each variable, we follow the criteria presented in Siddiqi [26] to select the relevant variables. Siddiqi [26] proposes intervals to relate the IV to the strength of the relationship with the target variable. These intervals can relate the variable under analysis in order to determine a weak, medium, strong or non-existent relationship between the analyzed variable and the target variable. The relation can be set as shown in the conditions below.

When IV:

•

Less than 0.02, the variable does not differentiate the Goods/Bads odds ratio.

In Table 6 we present an example of the calculation of IV and the corresponding WoE for each category of X. In the example, the values of each customer on the feature X will be replaced by the corresponding WoE for that category. A customer with X 1 on feature X will now have - 0.3137 on feature X. The IV for variable X is 0.1565 meaning it has a medium strength relationship with the Goods/Bads odds ratio.

For the experiment, we selected the variables with weak, medium and strong relationship to the Goods/Bads odds ratio and trained the models. In Table 7 we present the IV of the selected features. In this experiment, we trained both algorithms using dataset B with the WoE transformation. As before, we used grid-search for setting the hyperparameters to optimize the evaluation measures and 10-fold cross validation to measure the performance. The output for each loan O i was transformed using Eq. (3) and then probability Pi of each application was compare to a given threshold. The models were applied to applications under the same criteria as the first experiment.

The purpose of this experiment is to have an idea of how long should a model be valid. We intend to analyze two events that we believe could affect the performance of a credit scoring model in the long run. As Blanco et al. [5] succeeded using macroeconomic variables in a microfinance problem. We decided to have a brief analysis and try to differentiate emerging markets from developed markets concerning financial indicators. Emerging markets present a more volatile economy. These constant movements into the economy can completely change the conditions of the target population in a shorter period than expected. We analyzed the yearly inflation in these markets, presented in Fig. 8. Inflation can be defined as the variation of the prices of goods and services over a period. It is evident that emerging markets suffer highers rates of inflation. This can easily translate into a lower acquisition level or payment capacity of an individual from an emerging market. Since, if the individual does not increase the corresponding income will be unable to keep up with financial obligations.

On the other hand, we have the constant growing of the number of customers with an average of 18.7% new users per month. This means that in less than 6 months we have double the amount of users. This has direct impact in the description of the population. The demographics, proportion of goods and bad borrowers, etc., all change in a fast pace. This growth is not limited only to the number of customers but also to different regions in the targeted country. These regions mix with the already existing customers creating a whole new description of the population.

Therefore, we decide to train credit scoring models using different time windows. Based on the erratic movement of the GDP and the contently high inflation rate, our hypothesis states that the models should perform better when trained with a relatively short time window. The use of different time windows can be a determining factor as seen in [21].

OPEN IN VIEWER

The third experiment is based on dataset C and dataset D. As seen in Fig. 6, we created both datasets at the same time. However, we considered different time windows. Using three months for dataset C and five months for dataset D. This means that if we generate both datasets today, an application made four months ago will not be considered for dataset C but will be considered for dataset D. We used transformed each dataset recoding with WoE. Afterwards, we selected the useful variables based in their respective IV. With the transformed variables we then trained a LR model and a SVM model for each dataset. For the training procedure, we used grid-search for setting the hyperparameters to optimize the evaluation measures and 10-fold cross validation to measure the performance as in previous experiments.

Concerning results and deployment, the first and second experiments were tested with real applications received after model building and deployment. As for the third experiment, the results will be obtained through cross validation, comparing the AUROC and overall accuracy.

6.1 Differences between dummy variables and weight of evidence

In the first experiment, we implemented the two selected algorithms referred in Section 3.4 and used A | B testing to select which model will evaluate the corresponding loan application. The number of the application was used to divide the population into two parts; this number is a sequence of natural numbers. Therefore, assigning the applications with odd application number to the LR algorithm and the remaining (even application number) to the SVM algorithm. Even though we used A|B testing, we still calculated and stored probability of default for each application using both algorithms. This means that if the loan application in evaluation met the threshold for both models, then the result will be counted for both models as well. But if the loan application met the threshold for either LR or SVM, then the result will only be stored for that particular algorithm.

Therefore, the overdue rate represented in the first and second experiment shows the performance of each application that met the approval criteria of the corresponding algorithm. The results presented in Table 8 compare the scoring models with the manual process. The manual process refers to the all applications received before the beginning of the first experiment. This process was based on expert criteria pipeline. From the manual process, we considered only applications from customers who applied for the first time and customers with only one paid loan before. We selected these applications to match the same population targeted by the models.

OPEN IN VIEWER

The two leading indicators to measure are the overdue rate and the approval rate. Our experiments were focused on these two indicators since they are the core of a stable and sustainable business model. A high overdue rate will make the business model unprofitable, while a low approval rate will not grant enough loans even to cover the operational costs.

First, we notice that both models of the first experiment failed to classify correctly the overdues, this translate into direct costs for the MFI. The LR from the first experiment improved the approval rate. However, the overall accuracy of the LR did not perform as expected.

OPEN IN VIEWER

In Fig. 9 we compare the performance of the models in terms of classification error with real cases. The horizontal axis is the number of days after the deployment relative to the model under evaluation. Before the 30th day of deployment, both models trained with dataset A do not show overdue loans. This is due to the fact that the approval rate of LR-A and SVM-A was relatively low and just a few loans with less than 30 days of length were granted. However, after the loans granted by LR-A and SVM-A reached their respective due date, an increase on the overdue rate can be observed. Even if LR-A and SVM-A achieved a good overdue rate, these models would not be suitable for the business due to lack of approval. The models approved less than the baseline with - 4.33% and - 91.91% decrease of the approval rate respectively. Models trained with dataset B show a better behavior overall. Loans evaluated by LR-B and SVM-B presented fewer overdues and more evaluations were approved thought the evaluation by these models. As seen in Fig. 9, LR-B and SVM-B models are more stable on the overdue rate after the 30th day of deployment. These models are more constant and stable when compared to models created with dataset A. The LR-B and SVM-B models not only performed better in terms of overdue but also achieved higher approval rate than the manual process. As seen in Fig. 10, the SVM-B model was the best model in terms of approval rate. SVM-B was able to double the approval rate achieved by LR-B obtaining at the same time a slightly lower overdue rate. One of the main differences between the models, apart from datasets used for training, is the allocation of weights of the variables. Models trained with dataset A, gave more weights to variables related to the customer profile and loans conditions. The profile data of the customer is filled by himself and cannot be verified, thus making this data not really reliable. Furthermore, loans amount conditions are capped to a specific value, meaning they can only make variations within a small interval.

OPEN IN VIEWER

On the other hand, models trained with dataset B allocated more weights in variables related to historical data of the mobile phone of the customer. As the customer is not aware of its collection and/or cannot easily forged, these variables present a better profile of the customer and are more reliable, Both models trained with dataset B already surpassed the performance of the manual loan application selection process. More loans were granted when evaluated by the LR-B and SVM-B while the loans granted using the credit scoring evaluation presented fewer overdues.

While analyzing the performance of the models, we noticed some trends related to the nonpayment of the loans. As seen in Fig. 11, in general, the overdue rate for loans granted towards the end of the week is higher. This trend is also present in SVM-B which was the best model so far. LR-A also shows higher overdue rate in the last days of the week but concentrated on Friday and Saturday. LR-B shows less variation among the week days.

For Fig. 11 we did not consider SVM-A due to the low number of approved loans with this model. The low number is directly connected to the low approval rate.

Note that an overdue is not a total loss for the company since the overdue loans will enter into the recovery pipeline.

As for the third experiment, we compared the accuracy over a dataset composed of cases not present on the training set for any of the algorithms. For this we analyzed the status of the loan 30 days after due date and calculated the proportion of paid loans. In Table 9, we present the comparison of the accuracy by time window considered. With these results we cannot identify a clear winner as the difference is not significant.

This result goes against the expectation when looking at the complete overview of the business. For the current trimester, the total number of customers increased by more than 50%. Moreover, the amount of granted loans for the same period of time resulted in more than 100% increase when compared to total granted loans prior to this period. However, we remain with the problem that the models get be quickly outdated. Based on the results of the third experiment we can conclude that the time window considered for training is not as relevant as the time window considered for updating the algorithms.