On coding effects in regularized categorical regression

Abstract

This discussion is a continuation of Tutz and Gertheiss (2016)’s paper, where we focus on the importance of the coding of effects in regularized categorical and ordinal regression. We show that, though that an appropriate regularization is profitable for any coding, the choice of a relevant coding can prevail over the one of the regularization term for revealing structures. We focus on predictors though the issues raised also apply to responses. We illustrate our point on a classic data set.

Keywords

categorical predictor Ordinal regression regularization coding system sparsity

1 Introduction

Regularized approaches have been at the forefront of statistical research in the last two decades. In this context, regression with metric features has been particularly studied (Bühlmann and Van De Geer, 2011; Giraud, 2014). As a result, a number of methods are now available to efficiently cope with the curse of dimensionality and high-dimensional data (the $n ≪ p$ set-up). The notion of sparsity has emerged as a natural way for structuring the estimators for metric variables (Hastie et al., 2015), but the processing of categorical variables has received much less attention. The paper of (Tutz and Gertheiss (2016)) fills this gap by providing a comprehensive high-level view of the methodological developments that have been proposed so far to apply or to construct specific penalties for categorical data.

As (Tutz and Gertheiss (2016)) state, ‘with the right penalty being chosen, [...] structures in the data can be revealed’. In this discussion, we would like to stress that the choice of a specific coding can also be useful to reveal structures in regularized methods. We show that using an appropriate combination of effect coding and penalty, it is possible to achieve good generalization prediction and interpretable models. Conversely, an ill-suited coupling of coding scheme and penalty has a detrimental effect on prediction and may lead to misinterpret the type and the importance of the role of categorical variables in the model. This completely jeopardizes the original goal of sparsity-inducing regularization, which is generally introduced to bring more interpretability in the model. Note that although we focus on predictors, the issues raised here also apply to categorical responses.

2 Coding systems and regularization

In the context of standard (non-regularized) linear models, a categorical predictor is usually represented through several auxiliary variables (e.g., dummy variables). Several choices are possible, arguably the most popular being the dummy coding. Importantly, this choice does not affect the prediction, since the vector space spanned by the design matrix does not depend on the coding. However, in regularized regression, this is no longer the case: applying the same penalty to different codings leads to different predictions, as illustrated in the following section.

2.1 A simple motivating example

Consider the simple example of one-way ANOVA (Analysis Of Variance) regularized by a ridge penalty: we aim at adjusting a linear model for predicting a vector of observations $y = (y_{1}, \dots, y_{n})^{⊤} \in ℝ^{n}$ by a single categorical predictor exhibiting two levels (with $n_{1}$ , respectively $n_{2}$ examples in each level, such that $n = n_{1} + n_{2}$ ). The associated linear model is $b = X β + ε$ , where $ε \sim N (0_{n}, σ^{2} I_{n})$ are Gaussian residuals and X is the n × 2 design matrix. The ridge estimator of $β$ is

\hat{β} λ = {(X^{Τ} X + λ I_{p})}^{- 1} X^{Τ} y .

(2.1)

Now, consider the two following classic coding schemes for the design of the ANOVA:

X^{(1)} = (\begin{matrix} 1 & 0 \\ ⋮ & ⋮ \\ 1 & 0 \\ 0 & 1 \\ ⋮ & ⋮ \\ 0 & 1 \end{matrix}), X^{(2)} = (\begin{matrix} 1 & 0 \\ ⋮ & ⋮ \\ 1 & 0 \\ 1 & 1 \\ ⋮ & ⋮ \\ 1 & 1 \end{matrix})

The first coding $X^{(1)}$ is usually referred to as the ‘factorial design’ (or ‘no reference’), while $X^{(2)}$ considers the first level as the reference (‘dummy coding’). With simple algebra, one can derive the explicit forms of the ridge estimator (2.1) corresponding to these two different choices. Letting $y_{1}^{+}$ and $y_{2}^{+}$ be the sums of the observations in the respective levels, one has

{\hat{β}}_{λ}^{(1)} = (_{\frac{y_{2}^{+}}{n_{2 + λ}}}^{\frac{y_{1}^{+}}{n_{1} + λ}}), {\hat{β}}_{λ}^{(2)} = (_{\frac{(n_{1} + λ) y_{2}^{+} - n_{2} y_{1}^{+}}{(n + λ) (n_{2} + λ) - n_{2}^{2}}}^{\frac{(n_{2} + λ) y_{1}^{+} + λ y_{2}^{+}}{(n + λ) (n_{2} + λ) - n_{2}^{2}}}) .

(2.2)

Clearly, the estimation and the predictions ${\hat{y}}_{λ}^{(1)}$ and ${\hat{y}}_{λ}^{(2)}$ will usually differ when $λ \neq 0$ . That being said, there is no universal best option: one or the other coding may be preferred according to the available information on levels, so that the bias introduced by the ridge penalty will favour solutions in accordance with prior knowledge or assumptions. The following subsection elaborates on this point when fitting ordinal and categorical variables.

2.2 Coding systems for categorical predictors

An adequate coding for categorical variables—including the choice of a reference among the levels—should reflect a knowledge about the levels and their relationships. We split the problems into three classes: (a) ordinal variables with quantifiable and known differences between levels, (b) categorical variables without any ordering between levels and (c) ordinal variables without quantified differences between levels.

Besides coding, the prior knowledge should also drive the choice of a particular penalty for regularizing the problem. To be more specific, we stick to the framework of generalized linear model (GLM), in which case the general regularized regression problem is

{\hat{β}}_{λ} =_{β ε ℝ^{p}}^{\arg \min} {- ℓ (β; X^{(c)}, y) + λ p e n (β)},

(2.3)

where

ℓ

is the log-likelihood associated with a given GLM. The

n \times p

design matrix

X^{(c)}

is defined from a specific coding scheme labelled by

(c)

. The regularization is achieved by the penalty function

pen (\cdot)

weighted by the tuning parameter

λ

. Since

pen (\cdot)

acts on

β

that applies to

X^{(c)}

, it is clear that coding and penalization interact. Depending on the situations (a), (b) and (c), we should specify a coding and a penalty function

pen (\cdot)

adapted to the structure of the problem, as proposed below.

Ordinal variables with quantifiable and known differences between levels. In this ideal situation, the categorical predictors can be treated as metric—or numeric—variables without loss of information. In practice, we replace the levels with prescribed quantitative values to define the design matrix. This handling has the advantage of not increasing the dimension of the original problem since there is only one column associated with each predictor. When the initial number of predictors remains moderate, the GLM may be adjusted by standard maximum-likelihood estimation (MLE) and significance of each predictor can be evaluated by likelihood ratio tests. If, however, the number of predictors is too large or if variable selection is desired, a natural penalty function is the simple $ℓ_{1}$ -penalty—aka the Lasso (Tibshirani, 1996)—that induces sparsity and performs selection at the coordinate level.

Table 1:

Contrasts and codings for a predictor with four levels: Usual contrast (dummy coding) and contrast for adjacent levels, with the corresponding backward difference coding

	Usual contrast			Contrast for			Backward difference
Level	(dummy coding)			adjacent levels			coding
0	0	0	0	$-$ 1	0	0	$-$ 3/4	$-$ 1/2	$-$ 1/4
1	1	0	0	1	$-$ 1	0	1/4	$-$ 1/2	$-$ 1/4
2	0	1	0	0	1	$-$ 1	1/4	1/2	$-$ 1/4
3	0	0	1	0	0	1	1/4	1/2	3/4

Categorical variables without any ordering between levels. In this case, there is little structure to be encoded. The standard solution is to introduce auxiliary variables with ‘dummy coding’ (see Table 1). The main choice consists of selecting the level used for the reference: it can be the more or the less frequent level, arbitrary picked, say the first one, and so on. It is also possible to consider a design without any reference, even when many predictors are at play, provided that regularization allows to solve the resulting underdetermined system. In any case, the dimension of the problem may be large, even for a moderate number of predictors, since there is now one column per level for each predictor in the design matrix $X^{(c)}$ . Regarding the structure of the regularization, it seems advisable to use a penalty that performs the selection at the factor level, selecting all the coefficients associated with a unique predictor at once. The most widespread groupwise penalty that meets this prerequisite is the group-Lasso (Yuan and Lin, 2006).

Ordinal variables without quantified differences between levels. In this case, a transformation of predictors in metric variables would be arbitrary and thus inadequate. Dummy coding is legitimate but fails to take order into consideration. Instead, we propose to use contrasts that compare two adjacent levels. An example of these contrasts is displayed in Table 1 with the corresponding coding, known as backward difference coding. They are simply obtained by solving a linear system (Serlin and Levin, 1985). The backward difference coding has its ‘forward’ counterpart known as forward difference coding. These difference codings enable to fuse some levels by setting to zero some increments between adjacent levels, as explained by (Tutz and Gertheiss, 2016). Regarding regularization, it is natural to perform selection at the predictor level just as in the previous case. When the relationship between the predictor and the response is assumed to be monotonic, the coop-Lasso can be used to bias the prediction in this sense. Indeed, the coop-Lasso is a groupwise penalty encouraging the selection of sign-coherent groups of variables (Chiquet et al., 2012). With difference codings, when all increments are driven towards sign-coherence, the monotonicity with respect to levels is favoured.

3 Illustration on a classic data set

We illustrate here the importance of the coding scheme on a classic data set, which is particularly well-suited to illustrate our message. Indeed, we can confidently postulate that the nine ordinal predictors at play have 10 equally spaced levels each. First, we show that using an appropriate coding with its accompanying penalty, it is possible to do about as well as if an oracle had given the order and the correct differences between levels. Second, we show that using a standard but inappropriate coding and/or an inappropriate penalty can severely deteriorate prediction.

Breast cancer data set. The data set, referenced as the ‘Breast cancer Wisconsin original data set’ in the UCI Machine Learning Repository (Lichman, 2013), was built incrementally from the measurements obtained on different cohorts spanning 1989–91, resulting in a total of 699 examples. The goal is to predict whether epithelial cells are benign or malignant, based on nine cytological features assessed on a scale of 1 to 10. Each predictor was produced by a semi-automatic labelling system relying on computer vision to extract cell nuclei boundaries in sample tissues. Nine numerical features were computed based on these boundaries, then graded on a scale of 1 to 10, such that larger values typically indicate a higher likelihood of malignancy. The grading process leading to the ordinal predictors actually used for diagnosis has not been described, but from the facts that the ordinal predictors are defined from numerical features and that these predictors were used as quantitative variables in the original diagnosis method (based on piecewise-linear classifiers), the predictors of this data set can reasonably be considered as ordinal variables with 10 equally spaced levels. A more detailed exposure of the data is not necessary for the purpose of this article. For a more comprehensive description, the reader is referred to Wolberg and Mangasarian, 1990, who present an analysis of the first cohort.

Numerical settings. Assuming that the ordinal predictors have 10 equally spaced levels each, the best codings encode levels by numeric values, say with integers from 1 to 10. We use a binomial model for the binary (Malign/Benign) response, and as the sample size is large enough to perform standard MLE, our reference model is a non-regularized logistic regression model with $p = 9$ predictors and a global intercept term. We compare this ideal model with regularized approaches for logistic regression accounting for various types of knowledge or assumption on categorical predictors. The different variations associate a more or less appropriate pair of coding and penalty choice, as follows:

Dummy coding with first level as reference and Lasso penalty.

Dummy coding with first level as reference and group-Lasso penalty.

Dummy coding with no reference and Lasso penalty.

Dummy coding with no reference and group-Lasso penalty.

Backward difference coding with coop-Lasso penalty.

Note that we never penalize the (global) intercept term.

3.1 Predictive performance

In the framework of logistic regression, the statistical performances are classically measured with the classification error and the binomial deviance of the fit. To estimate these two quantities, we randomly split the original data set ( $n = 699$ samples) into three parts: one for the estimation of the regression coefficients (learning set), one for the choice of the tuning parameter $λ$ (validation set) and one for the estimation of the predictive performance (test set). The average proportions are 1/4,1/4 and 1/2 for learning, validation and test. For the non-regularized logistic regression, the MLE is fitted by estimating the regression coefficients after merging the learning and the validation sets, since there is no tuning parameter for this method. We restart the whole process 1 000 times and display the results in Figure 1.

Figure 1:

Estimated performance for the breast cancer data set in terms of binomial deviance (left panel) and classification error (right panel) for numeric coding with maximum-likelihood (the ideal case), backward difference coding with coop-Lasso, dummy coding (without reference and with first level as reference) with Lasso and group-Lasso.

The combination of difference coding with the coop-Lasso penalty does almost as well as the reference MLE model in terms of deviance and classification error, and better than all other regularization alternatives that do not take into account the order between levels. Note that in a number of resamples, the MLE suffers from a lack of regularization which explains the large dispersion of the observed MLE deviance. In contrast, thanks to regularization, the dispersion of the coop-Lasso deviance is smaller. We, thus, observe here that an appropriate combination of the coding system and the regularization penalty is fruitful.

Dummy codings with the first level as an arbitrary reference lead to solutions that are clearly dominated by all the other ones. Interestingly, dummy codings without a reference level, which still do not account for the ordering of levels, lead to a striking 30 to 40 per cent improvement of the classification error and deviance compared to the previous dummy coding option. Note that the use of a reference in dummy coding is somewhat a heritage from the non-regularized set-up, where it numerically stabilizes the optimization process. In the regularized set-up, we suggest to avoid the use of such a reference: not only it may strongly deteriorate predictions, but it also jeopardizes coefficient interpretation, as seen in the following section.

Finally, we observe that group penalties, which introduce a coupling between the levels of a predictor, compare favourably to the Lasso penalty for the two dummy coding schemes. However, this beneficial effect is rather marginal compared to the large differences observed between coding alternatives: within the combination explored here, choosing the right coding is more instrumental than fine-tuning the penalty to introduce an appropriate bias.

3.2 Interpretation of the models and regularization paths

We have shown that a relevant combination of coding and penalty pays off in terms of predictive performances; we pursue by showing that it also benefits to the interpretability of the model, which is a key asset of sparsity-inducing regularization. To this purpose, we display the regularization paths, describing the solutions as the penalty parameter $λ$ varies. We compare these paths for the best regularized method (difference coding with a coop-Lasso penalty), appropriate for ordinal variables, with the worse ones (dummy coding with first level as reference with a Lasso or a group-Lasso penalty), which choose an arbitrary reference and reflect little information about the structure of the categorical predictors.

We adjust all methods on the whole data set. Here also, the reference model is logistic regression with numeric coding. The regression parameters for the logistic regression with numeric codings are reported in Table 2, together with the significance of predictors, as assessed by a $Z$ -test.

Table 2:
Estimation of the coefficients for the logistic regression with numeric codings

Estimate Std. Error z value Pr( $>$ $|$ z $|$ ) significance

(Intercept) -10.09 1.17 -8.59 0.0000 $★ ★ ★$

Bare.Nuclei 0.38 0.09 4.08 0.0000 $★ ★ ★$

Clump.Thickness 0.53 0.14 3.77 0.0002 $★ ★ ★$

Marginal.Adhesion 0.33 0.12 2.67 0.0075 $★ ★$

Bland.Chromatin 0.45 0.17 2.61 0.0090 $★ ★$

Normal.Nucleoli 0.21 0.11 1.88 0.0595 .

Mitoses 0.53 0.33 1.63 0.1041

Uniformity.Cell.Shape 0.32 0.23 1.39 0.1638

Single.Epithelial.Cell.Size 0.10 0.16 0.62 0.5383

Uniformity.Cell.Size -0.01 0.21 -0.03 0.9793

	Estimate	Std. Error	z value	Pr( $>$ $\|$ z $\|$ )	significance
(Intercept)	-10.09	1.17	-8.59	0.0000	$★ ★ ★$
Bare.Nuclei	0.38	0.09	4.08	0.0000	$★ ★ ★$
Clump.Thickness	0.53	0.14	3.77	0.0002	$★ ★ ★$
Marginal.Adhesion	0.33	0.12	2.67	0.0075	$★ ★$
Bland.Chromatin	0.45	0.17	2.61	0.0090	$★ ★$
Normal.Nucleoli	0.21	0.11	1.88	0.0595	.
Mitoses	0.53	0.33	1.63	0.1041
Uniformity.Cell.Shape	0.32	0.23	1.39	0.1638
Single.Epithelial.Cell.Size	0.10	0.16	0.62	0.5383
Uniformity.Cell.Size	-0.01	0.21	-0.03	0.9793

Figure 2 summarizes the overall regularization paths by predictor for the three regularized methods. For groupwise penalties applied to numerical variables, groupwise regularization paths are usually displayed by plotting the $ℓ_{2}$ norm of the groups of coefficients versus (a function of) the penalty parameter. Here, as all predictors are ordinal, we measure the predictor importance as the maximum absolute value of differences between adjacent levels. In a GLM, this measure represents the maximal absolute increment of the linear predictor for an increment in the level of the predictor. To compare the different variants of coding and penalty on the same grounds, this prediction importance is plotted versus a scaled norm of the overall vector of parameters: the $ℓ_{1}$ norm of ${\hat{β}}_{λ}$ is scaled by the norm of this vector for $λ_{cv}$ , the penalty parameter selected by cross-validation.

Figure 2:

Cancer dataset: Regularization paths of predictor importance versus shrinkage factor.

Figure 2 compares the predictor importances estimated by the models fitted using: dummy coding using the first level as reference with a Lasso penalty (left), same coding with a group-Lasso penalty (middle) and backward difference coding with a coop-Lasso penalty (right). We note that the picture is clearer, much more consistent between the different values of the penalty parameter for the last option. In particular, for large values of the penalty parameter, dummy coding with Lasso and group-Lasso gauge a large importance to the ‘Single.Epithelial.Cell.Size’ predictor, which then declines for less penalized models. This predictor is not detected as significant by the MLE (Table 2), and a closer analysis (not shown) reveals that one level of the Single.Epithelial.Cell.Size is considered as highly relevant when the order information between levels is not taken into account. The model fitted with backward difference coding combined with the coop-Lasso, which takes into account the order of levels never estimates a large importance to the Single.Epithelial.Cell.Size predictor.

We now show that even predictors that seem to be similarly handled by all methods when looking at the summary provided by Figure 2 may, in fact, be handled quite differently. For this purpose, we display in Figure 3 the parameters attached to each level for the ‘Clump.Thickness’ predictor, which is found significant by MLE, as shown in Table 2. These parameters are the regression coefficients for dummy coding and a reparametrization of the regression coefficients for backward difference coding.

These regularization paths reveal that for the two variants using dummy coding (left and middle), the parameters are not coherent with the numeric levels of the ‘Clump.Thickness’ predictor and not self-consistent across $λ$ values for the variant based on the group-Lasso penalty. On the contrary, the parameters computed from the coefficients estimated from the model fitted on backward difference coding with the cooperative-Lasso (right) are consistent across $λ$ values and coherent with a numeric representation of the levels, as the spacings between levels are similar. We thus argue that interpretation of these effects, which is typically carried out for the selected model, is safer with the later approach, as this interpretation is much less sensitive to model selection, and recalling the fact that better prediction performances were obtained with backward difference coding and the coop-Lasso penalty in the previous section.

Figure 3 :

Cancer dataset: Regularization paths of the coefficients pertaining to the levels of the ‘Clump. Thickness’ predictor versus shrinkage factor.

4 Conclusion

To summarize, we have shown that choosing an appropriate coding is at least as important as choosing the right penalty. We advocate that one should refrain from using arbitrary reference levels in codings, since most penalties will introduce a bias towards the resemblance of the other levels to the reference which may have important detrimental effects regarding predictive performances and interpretation. In the experiments reported here, backward difference coding with the coop-Lasso penalty appears to be the best choice. This combination turned out to be appropriate for the underlying structure of the categorical predictors at hand. Note that we do not state in any way that this combination should usually be preferred to other ones. Our message is rather that for regularized methods, exploring codings can be at least as sensible as exploring penalty terms.

References

Bühlmann

Van De Geer

(2011) Statistics for High-dimensional Data: Methods, Theory and Applications . New York: Springer Science & Business Media.

Chiquet

Grandvalet

Charbonnier

(2012) Sparsity in sign-coherent groups of variables via the cooperative-lasso. The Annals of Applied Statistics , 6, 795–830.

Giraud

(2014) Introduction to High-dim- ensional Statistics , Vol. 138. Boca Raton, USA: CRC Press.

Hastie

Tibshirani

Wainwright

(2015) Statistical Learning with Sparsity: The Lasso and Generalizations . Boca Raton, USA: CRC Press.

Lichman

(2013) UCI machine learning repository. Retrieved November 2015, from http://archive.ics.uci.edu/ml

Serlin

Levin

(1985) Teaching how to derive directly interpretable coding schemes for multiple regression analysis. Journal of Educational Statistics , 10, 223–38.

Tibshirani

(1996) Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 58, 267–88.

Tutz

Gertheiss

(2016) Regularized regression for categorical data. Statistical Modelling.

Wolberg

Mangasarian

(1990) Multisurface method of pattern separation for medical diagnosis applied to breast cytology. Proceedings of the National Academy of Sciences of the United States of America , 87, 9193–96.

10.

Yuan

Lin

(2006) Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 68, 49–67.