Unfold income myth: Revolution in income models with advanced machine learning techniques for better accuracy 1

Abstract

Consumer IncomeView is the Equifax next-generation income model that estimates a consumer’s individual annual salary/wage income. This model provides an optimal income solution to our clients by incorporating premier multi-source data assets and advanced machine learning modeling techniques. As a result, the new model significantly increases the scorable population rate compared with the predecessor models and significantly improves the prediction accuracy. The results of Consumer IncomeView have been successfully validated by an array of new proprietary accuracy metrics for model performance measurement, both on internal out-of-time and clients’ data. This paper presents the design, development, methodology and main results obtained from the development of the Equifax Consumer IncomeView model.

Keywords

Individual salary income machine learning multi-source data trended data accuracy metric

1. Introduction

Understanding a consumer’s income level strengthens customer relationships across the entire account lifecycle (Arslan & Karan, 2010). The primary sources of income on industry are verified income from real estate-based loan application, unverified income from other loan application, or employer direct deposit into a checking account. However, many existing products use unverified income data that has been self-reported through consumer surveys or in government censuses, which generally did not yield a satisfactory result (Moore et al., 2000). Accurate modeled income estimation is urgently demanded to meet market needs. In 2008, Equifax launched the first generation of Personal Income Model (PIM) to the U.S. market, which provided in-depth income insights to identify the best places to deploy key resources (https://assets.equifax.com/assets/usis/efx_pim.pdf). However, the model performance deteriorates over time and a new model that incorporates novel data assets and state-of-the-art machine learning techniques is required.

More recently, Equifax developed and implemented next-generation income model for U.S. market, the Equifax Consumer IncomeView (https://www.equifax.com/business/consumer-incomeview/). This new product was developed to estimate a consumer’s annual salary/wage income at individual level. It enables our clients to better target their product offerings to consumers who more closely match credit requirements of the offer. This product can be used in a wide variety of industries and sectors with an aim of helping our clients to make stronger, more confident decisions by more accurately estimating consumer income. It can be used with risk-based and targeted marketing tools to achieve a more targeted population, for improved segmentation, for prescreening efforts and when conducting customer portfolio review. However, Equifax Consumer IncomeView cannot be used for adverse action or risk-based pricing decisions.

2. Methodology

2.1 Modeling data: Sources and validation

Equifax Workforce Solutions (EWS) is a subsidiary of Equifax that provides employment and income verifications on over 6,000 U.S. employers, including 75% of the Fortune 500 and Fortune 1000 companies maintained in The Work Number ${}^{\@setsize{\scriptsize}{8pt}{\viipt}{\@viipt}\textregistered}$ (TWN) database (http://equifax.com/workforce). Employment and income information is provided directly by participating employers and is updated every time their employees are paid. The individual annualized salary/wage incomes were derived directly from this database, which was used as the dependent variable. The input data sources are from Equifax proprietary consumer credit database (https://www.equifax.com/business/credit-attributes), Equifax Automated Credit Reporting Online (ACRO), including the Equifax Advanced Decisioning Attributes consumer credit file attributes (ADA), Equifax Regulation Z ability to pay attributes (RegZ), Equifax Mortgage Consumer Modeling Attributes (MCMA), and Equifax Dimensions trended credit attributes (for more information on the Equifax ACRO attributes, please refer to the Appendix section). The overlapped population between The TWN and ACRO were used as the model population. Standard exclusion criteria were then applied to this data to further remove the invalid records, such as duplicates, deceased, fraud etc. Approximately 20 MM observations were used for the model development, and the out-of-time samples of similar size were used for model validation. Model development data was further randomly split into train and in-time validation.

All the selected Equifax attributes suites in the model contain attributes for use in model and criteria development. Specifically, all are available for use in FCRA permitted prescreen or account management modeling programs, criteria development and portfolio analysis. They are developed using the Equifax consumer credit file.

The Advanced Decisioning Attributes provide extensive coverage of many credit analysis areas – depth of file to account activity, balance to high credit, recent to worst account status. Type-specific attributes are available for the general types: installment, retail, and revolving. Further industry-specific attributes are available for the following: auto, bankcard, credit union, department store, mortgage, personal finance and student loan.

The Mortgage CMA attributes provide extensive coverage of many mortgage credit analysis areas – depth of file to account activity, balance to high credit, recent to worst account status. Type-specific attributes are available for: first mortgage, home equity loan, home equity line, and non-HELOC revolving.

The Equifax Dimensions Attributes provide extensive coverage of many credit analysis areas. Equifax raw data fields are aggregated across accounts and trended for 24 months to provide a complete and historical view of a consumer’s credit behavior, such as transactor and revolver behavior types, balance direction, payment behavior, periodic spending, and activation. Type-specific attributes are available for the general types: installment, retail, and revolving. Further industry-specific attributes are available for the following: auto, bankcard, credit union and mortgage

2.2 Dependent variable

TWN verified annualized income in dollar amount, i.e. annualized individual salary/wage before tax, was used as the dependent variable. Notable data exclusion filters were used: individuals with outlier incomes that could not be validated were excluded, as well as consumers that were retired, restricted, deceased, surviving spouse and records that had recently been added to the TWN database without payroll history. The consumers with no credit activity within 24 months, or outdated were excluded. Multiple transformation schemes, such as box-cox power transformation, have been explored for the dependent variable and an internal research shows that log transformation provided the best in-sample fit. Due to the nature of income distribution, the assumption that earnings are log-normally distributed is widely accepted (Drăgulescu & Yakovenko, 2001).

2.3 Independent variables

Prior to model parametrization, the following variable treatment and selection steps were performed:

•
Apply standard data cleansing procedures to the sourced data;
•
Missing value imputation and capping/flooring; and
•
Perform exploratory data analysis to understand the stability and predictive power of each attributes (not shown).

In addition, care was taken to avoid high correlation among the independent variables, which can affect model stability. Measures such as coefficient correlation ( $\gamma$ ) and variance inflation factor (VIF) were used to assess the relationship between the independent variables. In some cases a variable could have a strong association with the dependent variable, but the relationship may be non-linear. To address this situation, analytical team(s) used standard proprietary methodologies to accomplish the following:

•
Determine the necessity of transformation for each of the independent variables;
•
Determine the optimal method for the variable transformation; and
•
Enable transformed variables to be included as independent variables in model development.

Other proprietary variable treatment includes creating additional derived variables and the use of interaction terms.

As mentioned in previous sections, all independent variables and their missing indicators were initially considered as independent variables. Together, these variables were run through computer-aided variable selection or reduction procedures in order to narrow down the candidate set of variables into a smaller, more manageable list. The variable list was then further refined through several more iterations to ensure that the model worked from both a statistical and business standpoint. In addition, from a statistical point of view, highly correlated variables were eliminated from subsequent regressions. Finally, variables were tested one at a time to determine the best possible combination of predictive variables.
2.4 Segmentation

The purpose of segmentation analysis is to determine the possibility, as well as the necessity, of defining homogeneous segments or subgroups in the population that require separate models. If such groupings can be identified, it may be deemed necessary to build separate models for these groups to enhance the overall performance.

Decision Tree was used to select the optimal segmentation scheme and splits. The size, significance, complexity and interpretation of various segmentation themes were evaluated to finalize the final segmentation. More than ten different scenarios of segmentation were studied, the goal was to separate the most accurate to least accurate group to provide different confidence level for income estimation on each segment. The best scheme is to use two layers of decision trees, by using different target variables. And research shows that four segments optimize the performance, derived by three Equifax ACRO attributes: age of trades, consumer credit capacity, and available credit on revolving accounts.

2.5 Modeling methods

Over 120 different machine learning modeling techniques were explored to optimize the best modeling approaches in order to achieve two goals: optimizing both the model estimation accuracy and interpretability. The selection of the techniques is based off several criteria: computational time, method complexity, additional accuracy lift, and feasibility of method implementation. The selected machine learning method has least computational complexity yet achieve maximum performance accuracy lift on the studied segments. The final income estimation is a combination of the following three modeling approaches (different algorithms used for each segment): linear regression with regularization (Ordinary Least Square baseline model), Multivariate Adaptive Regression Splines introduced by Friedman (Friedman, 1991) and multi-layer Neural Networks (Svozil et al., 1997) . The performance of each model was evaluated and compared by using various proprietary accuracy metrics innovated internally.

2.5.1 Multiple Linear regression

Multiple linear regression is a proven successful modeling technique designed to model the relationship between a continuous dependent variable $y$ and one or more explanatory variables (or independent variables) denoted $X$ , i.e.

$\displaystyle y_{i}=\beta_{0}1+\beta_{1}x_{i1}+\beta_{2}x_{i2}+\ldots+\beta_{p% }x_{ip}+\varepsilon_{i}\quad i=1,\ldots n$ (1)

where $y_{i}$ is the Log-transformed annualized individual income in dollar amount, and $x_{i1},x_{i2}\ldots x_{ip}$ are the Equifax proprietary consumer credit attributes as independent variables. Statistical estimation in linear regression focuses on the coefficients $\beta$ s. The elements of this parameter vector are interpreted as the partial derivatives of the dependent variable with respect to the various independent variables. The linear regression model uses the explanatory variables to estimate the transformed dependent variable takes on a given value, and then transform it back to dollar value. Model performance is evaluated on R-square and WPE20 (Section Performance Metrics) Multicollinearity issues among the independent variables were also properly handled. To select the significant independent variables and prevent model overfitting, LASSO (least absolute shrinkage and selection operator) technique (Cohen, 2006) is used, which arises from a constrained form of ordinary least squares regression where the sum of the absolute values of the regression coefficients is constrained to be smaller than a specified parameter. More precisely for a given parameter $t$ , the LASSO regression coefficients $\beta=\left(\beta_{1},\beta_{2}\ldots\beta_{m}\right)$ are the solution to the constrained optimization problem.

$\displaystyle\textit{minimize}\left|{\left|{y-X\beta}\right|}\right|^{2}\quad% \textit{subject to}\sum_{j=1}^{m}\left|{\beta_{j}}\right|\leqslant t$ (2)

Provided that the LASSO parameter $t$ is small enough, some of the regression coefficients will be exactly zero. Hence, the LASSO can be used as selecting a subset of the regression coefficients for each LASSO parameter. By increasing the LASSO parameter in discrete steps, a sequence of regression coefficients are obtained, where the nonzero coefficients at each step correspond to selected parameters. The algorithm that core modeling team implemented in SAS is a stepwise like procedure with a single addition to, or deletion, from the set of nonzero regression coefficients at any step (Cohen, 2006).

2.5.2 Multi-layer neural networks

A neural network is a series of algorithms, which assemble many “neurons”, and output the prediction neuron, as shown in Fig. 1 below (Matignon, 2005). The leftmost layer of the network is called the input layer, and the rightmost layer is called the output layer, the middle layers of nodes are the hidden layers. In this model, we chose to use two hidden layers to optimize the model prediction accuracy preventing the overfitting issue.

Figure 1.

Neural network model configuration.

The input neurons take the input $x_{1}$ , $x_{2}$ , $x_{3}$ , $x_{4}$ , and an intercept “( $+$ 1)” term or bias units, then output

$\displaystyle h_{w,b}(x)=f(W^{T}x)=f\left(\sum^{3}_{i=1}W_{i}x_{i}+b\right),$ (3)

Where $f$ (.) is the activation function, parameters $W$ is weight function. In the model, the hidden layers used hyperbolic tangent activation function. Model performance is evaluated on errors, such as percentage of average training error, percentage of average error, and percentage of maximum error etc. To prevent the stability of the optimization process, which means, prevent being trapped at a local max, the model was fit by using a fixed number of random starting points (assigning weight) with maximum iteration $=$ 1000. The optimal number of the starting points has been explored to optimize the model performance.

2.5.3 Multivariate adaptive regression splines

Multivariate adaptive regression splines constructs nested “hockey-stick” spline basis functions in an adaptive way by automatically selecting appropriate knot values for different variables, and it obtains reduced models by applying model selection techniques (Kuhfeld & Cai, 2013). The method does not assume parametric model forms and does not require specification of knot values. The bases are constructed by using truncated power functions (hockey stick function) as follow:

$\displaystyle\left({x-s}\right)_{+}=\max\left({x-s,0}\right),s\ \textit{is one% of the knots for}\ x$ (4)

The final income prediction $\hat{y}$ is a piecewise linear combination all bases:

$\displaystyle\hat{y}=\sum^{k}_{i=1}\beta_{i}\ \textit{Basis}_{i}$ (5)

The spline knot is a key concept, which connects the end of one portion of data and the beginning of another. Figure 2 demonstrates a spline with three knots, and different basis $s=$ 10 to 80, from left to right

Figure 2.

Spline example with variation of basis.

Similar to the forward selection in linear regression model, pairs of corresponding basis functions were selected and added to the model. The pair that resulted in the largest reduction in the residual sum of square was added. The next phase was backward elimination of a single basis function whose elimination minimizes the generalized cross validation criterion (GCV), a function of the residual sum of squares. Backward elimination iterates until all terms except the intercept are eliminated and then the model with the minimum GCV was chosen. Adaptivereg procedure is used to fit the final model. Like other nonparametric non-linear regression procedures, the Adaptivereg algorithm can yield complicated models that involve high-order interactions in which many knot values or subsets are considered. Besides the basis functions, both the forward selection and backward selection processes are also highly nonlinear. Because of the trade-off between bias and variance, the complicated models that contain many parameters tend to have low bias but high variance. To select models that achieve good prediction performance, GCV was used:

$\displaystyle\text{GCV}=\frac{1}{n}\sum^{n}_{i=1}\left(\frac{y_{i}-\hat{f}_{i}% }{1-\text{traces(S)}/n}\right)^{2}=\frac{\text{RSS}}{n(1-\text{trace(S)}/n)^{2}}$ (6)

where $y$ is the response, $\hat{f}$ is an estimate of the underlying smooth function, and $S$ is the smoothing matrix. The effective degrees of freedom for the smoothing spline can be defined as the trace of $S$ . Friedman uses a similar quantity as the lack-of-fit criterion,

$\displaystyle\text{LOF}=\frac{\text{RSS}}{n(1-(M+d(M-1)/2)/n)^{2}}$ (7)

where $d$ is the degrees-of-freedom cost for each nonlinear basis function and $M$ is total number of linearly independent bases in the model.

3. Results

3.1 Performance metrics

To assess the performance of Consumer IncomeView, Equifax examined the accuracy of the predicted income by the traditional accuracy metrics: Windowed Percent Error (WPE) and Concordance, and following new accuracy metrics: One-tail Accuracy, Capture Rate and Classification metric. These metrics were designed and implemented primarily for various business applications.

1.
WPE: measures percentage of predicted income falls into $\pm$ x% of actual income values. This accuracy is pinpoint accuracy measurement. WPE20 (i.e. $\pm$ 20% of actual income) is commonly used in the industry.
2.
One-tail accuracy: One tail (cut-off threshold measure) accuracy is innovated for business application:

–
It measures how accurate the model estimates a consumer’s income higher than $x.
–
For example, if the model estimate a consumer’s income is $>$ $60 K (say point estimate $=$ 62 K), it is 79% accurate that his/her true income is also $>$ $60 K, which is the worst case scenario

3.
Capture rate: is used to support one-tail accuracy to provide more comprehensive view in the following:

–
If consumers true income is higher than $x, what percent of the predicted income is higher than $x
–
Evaluate what percent of the true income the model can correctly capture directionally

One-tail accuracy and capture rate are combined accuracy measurement, they should be considered together as one measurement criteria.
4.
Classification metrics: measures the effectiveness of correctly predicted income for both upward and downward one tail accuracy. It provides a comprehensive view of how accurate the model can predict correctly on the directional for both tails.
5.
Concordance: Concordance measures “rank-ordering” properties of a statistical model. Concordance computation logic is as below:

If income1 $<$ income2 and predicted1 $<$ predicted2, then concordant;

If income1 $>$ income2 and predicted1 $>$ predicted2, then concordant;

Otherwise, discordant.

The final concordance measure is expressed as a percentage of correctly ranking pairs of income records, e.g. if the concordance statistic is 70.3%, then 70.3% of the pairs of predictions were rank-ordered correctly.

3.2 Model results

3.2.1 Income distribution

Consumer IncomeView outputs individual income scores in the range from 20–300 in the unit of one thousand dollars. Based on the out-of-time validation samples, Figure 3 compares the distributions of predicted income vs. the actual income in vingtiles. The median incomes estimated by Consumer IncomeView correspond very closely with the median of actual incomes.

Figure 3.

Predicted vs. Actual income distribution comparison.

3.2.2 Accuracy performance

When compared with the older Equifax income model PIM3, Consumer IncomeView has significant accuracy lift on overall and segmental level. WPE20 accuracy is 67% for segment #1, 38.7% for segment #2, 29% for segment #3, and 29.8% for segment #4. Model tends to achieve better accuracy on lower income, as there is less variation on income.

Figure 4 shows the overall accuracy and scorable lift that Consumer IncomeView provides. WPE20 accuracy lift is 31% incremental, Equifax proprietary data and advanced machine learning modeling technique on big data platform are the main drivers for the lift. Scorable rate is 10% incremental, which enables about 26MM additional consumers to be scored, and generate incremental revenue for end product users.

Figure 4.

Overall model results.

Figure 5 shows the One-tail and Classification Accuracy of the Consumer IncomeView. Compared with the general U.S. population distribution, the new solution significantly improves the One-tail (upwards) and classification accuracy. For instance, the One-tail has a $\sim$ 2.5 times higher accurate rate on the threshold of $60 K. It means that if a consumer has a $60 K+ modeled income, there is $\sim$ 79% of the chance that that person’s actual income is indeed $60 K+, which is much higher than the general population rate of $\sim$ 33%. Consumer IncomeView also achieves a minimum 75% classification accuracy, for example, for income $=$ 60 K, the Consumer IncomeView can predict correctly 80% for consumers’ income directionally, either $>=$ 60 K or $<$ 60 K, and significantly improved from general population without model.

Figure 5.

One-tail and classification accuracy.

Furthermore, we can calculate the area under the two classification accuracy curves (random population vs. Consumer IncomeView) and obtain the overall classification accuracy on all income range from $20 K–$300 K. Compared with the benchmark random U.S. population (without a predictive model), the Consumer IncomeView has $\sim$ 27% incremental lift on the overall classification accuracy over all predicted income ranges, which is demonstrated in third graph in Fig. 5.

Finally, concordance statistics of the Consumer IncomeView is also evaluated. When or concern focuses on the overall rank-ordering rather than individual income estimate, the nonparametric concordance metrics can assess the overall model performance. Compared with the PIM3 model, Consumer IncomeView significantly improved the concordance statistics, from 67.7% to 71.2%, generating a 5.2% incremental lift.

4. Discussion

In this paper we describe the development of the Equifax Consumer IncomeView model. This enhanced solution was built on the enriched Equifax proprietary consumer credit attributes, including the powerful newly developed trended credit attributes, featuring premier monthly consumer credit data up to 24 months of extended financial account history. Compared with the predecessor PIM3 model, the Consumer IncomeView significantly improves the overall WPE20 and expands the scorable population. When measured by the innovative One-tail and classification metrics, this new model also outperforms PIM3 model by a significant margin. The additional multi-source data assets and machine learning modeling techniques largely contributed to the huge performance improvement.

Consumer IncomeView has also been validated by both in-time validation and out-of-time validation. Segmentation distribution is almost the same as the model development sample, i.e. 2-layers segmentation scheme validation holds well. WPE20 has solid validation on out of time data, both segmentally and overall, and one-tail and classification accuracy on out of time validation hold very well (not shown).

References

Arslan,

Ö.

, & Karan,

M. B.

(2010). Consumer Credit Risk Characteristics: Understanding Income and Expense Differentials. Emerging Markets Finance & Trade, 46(2), 20-37.

Cohen,

R. A.

(2006). 207-31: Introducing the GLMSELECT Procedure for Model Selection, 18.

Drăgulescu,

, & Yakovenko,

V. M.

(2001). Exponential and power-law probability distributions of wealth and income in the United Kingdom and the United States. Physica A: Statistical Mechanics and Its Applications, 299(1), 213-221. https://doi.org/10.1016/S0378-4371(01)00298-9aaa(000)Friedman, J. H. (1991). Multivariate Adaptive Regression Splines. The Annals of Statistics, 19(1), 1-67. aaa(000)Kuhfeld, W. F., & Cai, W. (2013). Introducing the New ADAPTIVEREG Procedure for Adaptive Regression. SAS Global Forum 2013 Statistics and Data Analysis Paper. aaa(000)Matignon, R. (2005). Neural Network Modeling Using SAS Enterprise Miner. AuthorHouse.

Moore,

J. C.

Stinson,

L. L.

, & Welniak,

E. J.

(2000). Income Measurement Error in Surveys: A Review, 16(4). Retrieved from https://www.census.gov/srd/papers/pdf/sm97-05.pdf

Svozil,

Kvasnicka,

, & Pospichal,

(1997). Introduction to multi-layer feed-forward neural networks. Chemometrics and Intelligent Laboratory Systems, 39(1), 43-62. https://doi.org/10.1016/S0169-7439(97)00061-0