GD-RDA: A New Regularized Discriminant Analysis for High-Dimensional Data

2.1. Regularized discriminant analysis

Let K be the number of classes, and the samples be randomly drawn from a p-dimensional multivariate normal distribution with mean vector \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\bm{\mu} {_k} = ( { \mu _{k1}} , \ldots , { \mu _{kp}}{ ) ^T}$$ \end{document} and covariance matrix \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \Sigma _k}$$ \end{document} for each class, where \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$k = 1 , \ldots , K$$ \end{document} . Specifically, there are n_k independent and identically distributed (i.i.d.) samples in the kth class, \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} \textbf{\textit{x}}{_{k , 1}} , \ldots , \;\textbf{\textit{x}}{_{k , {n_k}}} \mathop \sim \limits^{i.i.d{ \rm{.}}} { \rm{MVN}} ( \bm{\mu} {_k} , { \Sigma _k} ). \tag{2.1} \end{align*} \end{document}

Let \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$n = \sum \nolimits_{k = 1}^K {n_k}$$ \end{document} be the total number of samples for all classes. Given a new observation, y , the goal of classification is to predict which class label the observed y belongs to. Let also \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \pi _k}$$ \end{document} be the proportion of observing a sample from the kth class. Throughout the article, we set \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \pi _k} = {n_k} / n$$ \end{document} so that \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\sum \nolimits_{k = 1}^K { \pi _k} = 1$$ \end{document} .

The QDA decision rule is to assign y to the class with label \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\arg { \min \nolimits_k}d_k^Q ( \textbf{\textit{y}} )$$ \end{document} , where \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$d_k^Q ( \textbf{\textit{y}} )$$ \end{document} is the discriminant score defined as \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} d_k^Q ( \textbf{\textit{y}} ) = ( \textbf{\textit{y}} - \bm{\mu} {_k}{ ) ^T} \Sigma _k^{ - 1} ( \textbf{\textit{y}} - \bm{\mu} {_k} ) + \ln \vert { \Sigma _k} \vert - 2 \ln { \pi _k}. \end{align*} \end{document}

Note that the population parameters in the score just provided are unknown and need to be estimated from the sample data. Let \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\bar{\bm{x}}{_k} = \sum \nolimits_{i = 1}^{{n_k}} \;\textbf{\textit{x}}{_{k , i}} / {n_k}$$ \end{document} be the sample means, and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${S_k} = \sum \nolimits_{i = 1}^{{n_k}} ( \textbf{\textit{x}}{_{k , i}} - \bar{\bm{x}}{_k} ) ( \textbf{\textit{x}}{_{k , i}} - \bar{\bm{x}}{_k}{ ) ^T} / ( {n_k} - 1 )$$ \end{document} be the sample covariance matrices. By replacing the unknown parameters \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$( \mu {_k} , { \Sigma _k} )$$ \end{document} with their sample estimates \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$( { \bar x_k} , {S_k} )$$ \end{document} , we have the sample version of \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$d_k^Q ( y )$$ \end{document} as \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} \hat d_k^Q ( \textbf{\textit{y}} ) = ( \textbf{\textit{y}} - \bar{\bm{x}}{_k}{ ) ^T}S_k^{ - 1} ( \textbf{\textit{y}} - \bar{\bm{x}}{_k} ) + \ln \vert {S_k} \vert - 2 \ln { \pi _k}. \tag{2.2} \end{align*} \end{document}

If we further assume that the covariance matrices are all the same, that is, \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \Sigma _k} = \Sigma$$ \end{document} for all k, and estimate the common \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\Sigma$$ \end{document} by the pooled sample covariance matrix \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${S_{pool}} = \sum \nolimits_{k = 1}^K ( {n_k} - 1 ) {S_k} / ( n - K )$$ \end{document} , then it leads to the sample version of LDA with the discriminant score defined as \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} \hat d_k^L ( \textbf{\textit{y}} ) = ( \textbf{\textit{y}} - \bar{\bm{x}}{_k}{ ) ^T}S_{pool}^{ - 1} ( \textbf{\textit{y}} - \bar{\bm{x}}{_k} ) - 2 \ln { \pi _k}. \tag{2.3} \end{align*} \end{document}

As shown in Friedman (1989), QDA and LDA usually perform well when the data follow a multivariate normal distribution and when the sample size is large compared with the dimension. In particular, QDA requires \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${n_k} \ge p$$ \end{document} to ensure S_k are nonsingular, and LDA requires \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$n \ge p$$ \end{document} to ensure \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${S_{pool}}$$ \end{document} is nonsingular. These requirements have largely limited the application of QDA and LDA for classifying high-dimensional small sample size data.

To improve the existing literature, Friedman (1989) considered a regularization method for the discriminant analysis. Specifically, he proposed to estimate \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \Sigma _k}$$ \end{document} by \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} { \hat \Sigma _k} ( \lambda ) = ( 1 - \lambda ) {S_k} + \lambda {S_{pool}} , \tag{2.4} \end{align*} \end{document}

where \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda$$ \end{document} is a regularization parameter with \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$0 \le \lambda \le 1$$ \end{document} . It controls the degree of shrinkage of the class-specific sample covariance matrix toward the pooled sample covariance matrix. When \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda = 0$$ \end{document} , it provides no shrinkage and yields QDA. When \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda = 1$$ \end{document} , it shrinks fully to the pooled sample covariance matrix and yields LDA. Noting that the regularized estimator (in 2.4) may not provide enough regularization, Friedman (1989) had also considered a second-step regularization that shrinks \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \hat \Sigma _k} ( \lambda )$$ \end{document} (in 2.4) toward a multiple of the identity matrix: \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} { \hat \Sigma _k} ( \lambda , \gamma ) = ( 1 - \gamma ) { \hat \Sigma _k} ( \lambda ) + \gamma [ { \rm{tr}} ( { \hat \Sigma _k} ( \lambda ) ) / p ] I , \tag{2.5} \end{align*} \end{document}

where \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \rm{tr}} ( \cdot )$$ \end{document} is the trace of the matrix, I is the identity matrix, and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\gamma$$ \end{document} is an additional regularization parameter with \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$0 \le \gamma \le 1$$ \end{document} .

Given the value of \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda$$ \end{document} , the additional parameter \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\gamma$$ \end{document} controls the degree of shrinkage toward the multiple of the identity matrix. Recall that the multiplier \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \rm{tr}} ( { \hat \Sigma _k} ( \lambda ) ) / p$$ \end{document} is the same as the average eigenvalue of \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \hat \Sigma _k} ( \lambda )$$ \end{document} . The second-step regularization, therefore, has the effect of decreasing the possibly over-estimated large eigenvalues and increasing the possibly under-estimated small eigenvalues. The regularized estimator \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \hat \Sigma _k} ( \lambda , \gamma )$$ \end{document} provides a two-parameter family of estimators for the class-specific covariance matrix. With this estimator, the regularized discriminant score is \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} \hat d_k^R ( \textbf{\textit{y}} ) = ( \textbf{\textit{y}} - \bar {\bm{x}}{_k}{ ) ^T}{ [ { \hat \Sigma _k} ( \lambda , \gamma ) ] ^{ - 1}} ( \textbf{\textit{y}} - \bar{\bm{x}}{_k} ) + \ln \vert { \hat \Sigma _k} ( \lambda , \gamma ) \vert - 2 \ln { \pi _k}. \tag{2.6} \end{align*} \end{document}

Accordingly, we refer to this classification method as RDA. Owing to the two parameters \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$( \lambda , \gamma ) ,$$ \end{document} RDA provides a fairly rich class of regularization alternative rules. For instance, the lower left corner with \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$( \lambda = 0 , \gamma = 0 )$$ \end{document} represents QDA, the lower right corner with \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$( \lambda = 1 , \gamma = 0 )$$ \end{document} represents LDA, and the upper right corner with \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$( \lambda = 1 , \gamma = 1 )$$ \end{document} represents the nearest neighbor classifier. In addition, fixing \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\gamma$$ \end{document} at 0 and varying \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda$$ \end{document} produces a discriminant rule between QDA and LDA; fixing \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda$$ \end{document} at 0 and varying \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\gamma$$ \end{document} yields a ridge-type analog of QDA; and fixing \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda$$ \end{document} at 1 and varying \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\gamma$$ \end{document} yields a ridge-type analog of LDA.

2.2. A diagonalization method for RDA

For high-dimensional data such as microarrays, the dimension can be much larger than the sample size. In such situations, QDA and LDA are no longer applicable since the sample covariance matrices are singular. In contrast, the regularized discriminant methods, including RDA and its variants, may still work if the parameters \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda$$ \end{document} and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\gamma$$ \end{document} are taken appropriately. For instance, Guo et al. (2007) proposed a regularized LDA for high-dimensional small sample size data, which is essentially a special case of RDA with \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda$$ \end{document} fixed at 1. Nevertheless, these approaches are usually unstable when n is relatively small or the ratio of \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$p / n$$ \end{document} is relatively large.

To overcome the singularity problem in discriminant analysis, Dudoit et al. (2002) introduced a diagonalization method for LDA and QDA. Let \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${D_k} = { \rm{diag}} ( s_{k1}^2 , \ldots , s_{kp}^2 )$$ \end{document} be the diagonal matrix of the sample covariance matrix S_k, and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${D_{pool}} = { \rm{diag}} ( s_1^2 , \ldots , s_p^2 )$$ \end{document} be the diagonal matrix of the pooled covariance matrix \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${S_{pool}}$$ \end{document} . Given that a diagonal matrix is always invertible, Dudoit et al. (2002) replaced S_k by D_k (in 2.2) and formed the DQDA, with the discriminant score defined as \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} \hat d_k^Q ( \textbf{\textit{ y}} ) = \mathop \sum \limits_{i = 1}^p { ( {y_i} - { \bar x_{ki}} ) ^2} / s_{ki}^2 + \mathop \sum \limits_{i = 1}^p \ln s_{ki}^2 - 2 \ln { \pi _k}. \tag{2.7} \end{align*} \end{document}

Similarly, DLDA was formed by replacing \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${S_{pool}}$$ \end{document} by \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${D_{pool}}$$ \end{document} (in 2.3), with the discriminant score given as \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} \hat d_k^L ( \textbf{\textit{ y}} ) = \mathop \sum \limits_{i = 1}^p { ( {y_i} - { \bar x_{ki}} ) ^2} / s_i^2 - 2 \ln { \pi _k}. \tag{2.8} \end{align*} \end{document}

To propose a diagonalization method for RDA, following the same spirit as in DLDA and DQDA, one direct approach is to estimate the covariance matrices by the respective diagonal matrix (of 3), that is, \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${D_k} ( \lambda ) = ( 1 - \lambda ) {D_k} + \lambda {D_{pool}}$$ \end{document} . Or equivalently, we estimate the sample variances by \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\hat \sigma _{ki}^2 = ( 1 - \lambda ) s_{ki}^2 + \lambda s_i^2$$ \end{document} . This results in the diagonalized discriminant score as \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} \hat d_k^R ( \textbf { \textit { y } } ) = \mathop \sum \limits_ { i = 1 } ^p { \frac { { { ( { y_i } - { { \bar x } _ { ki } } ) } ^2 } } { ( 1 - \lambda ) s_ { ki } ^2 + \lambda s_i^2 } } + \mathop \sum \limits_ { i = 1 } ^p \ln [ ( 1 - \lambda ) s_ { ki } ^2 + \lambda s_i^2 ] - 2 \ln { \pi _k } . \tag { 2.9 } \end{align*} \end{document}

Nevertheless, a form of \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$( 1 - \lambda ) s_{ki}^2 + \lambda s_i^2$$ \end{document} does not follow a chi-square distribution or a scaled chi-square distribution. It is also not easy in mathematics to deal with the term \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\ln [ ( 1 - \lambda ) s_{ki}^2 + \lambda s_i^2 ]$$ \end{document} , that is, the log-transform of an arithmetic mean. In addition, it does not sound feasible to perform the bias correction for the arithmetic version of the diagonalization method cited earlier. Note also that, if the shrinkage estimator (2.5) is employed for diagonalization instead of (3), then the resulting discriminant score will be even more complicated so that the proposed method may not be applicable in practice.

To avoid the problems mentioned earlier that were associated with the arithmetic mean, in Section 2.2, we propose a geometric method for diagonalization, form a new rule of RDA, and perform a bias correction to the proposed method for further improvement in Section 3.

2.3. Geometric diagonalization

Let \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$s_{pool , i}^2 = { \left( { \prod \nolimits_{k = 1}^K {s_{ki}^2} } \right) ^{1 / K}}$$ \end{document} be the geometric mean of the sample variances across all K classes. We now propose to estimate the individual variances \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\sigma _{ki}^2$$ \end{document} by \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} \tilde \sigma _{ki}^2 = ( s_{ki}^2{ ) ^{1 - \lambda }}{ ( s_{pool , i}^2 ) ^ \lambda } , i = 1 , \cdots , p , k = 1 , \cdots , K , \tag{2.10} \end{align*} \end{document}

where \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda$$ \end{document} is a shrinkage parameter with \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$0 \le \lambda \le 1$$ \end{document} . In essence, the estimator (2.10) is a geometric mean between the class-specific sample variance \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$s_{ki}^2$$ \end{document} and the pooled sample variance \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$s_{pool , i}^2$$ \end{document} . When \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda = 0$$ \end{document} , there is no shrinkage and we estimate each individual variance by their respective sample variance. When \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda = 1$$ \end{document} , we assume that \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\sigma _{ki}^2 = \sigma _i^2$$ \end{document} for all k and estimate all of them by the pooled geometric mean \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$s_{pool , i}^2$$ \end{document} . The shrinkage estimation by the geometric mean structure has been considered in, for example, Cohen et al. (2005) and Tong and Wang (2007). It is also noteworthy that the estimator (2.10) is different from the shrinkage estimator in Pang et al. (2009), in which our estimator borrows information across the K classes whereas their estimator borrows information across the genes within the individual class.

By (2.10), we define the new regularized discriminant score as \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} \tilde d_k^R ( \textbf { \textit { y } } ) = \mathop \sum \limits_ { i = 1 } ^p { \frac { { { ( { y_i } - { { \bar x } _ { ki } } ) } ^2 } } { { { ( s_ { ki } ^2 ) } ^ { 1 - \lambda } } { { ( s_ { pool , i } ^2 ) } ^ \lambda } } } + \mathop \sum \limits_ { i = 1 } ^p \ln \left[ { { { ( s_ { ki } ^2 ) } ^ { 1 - \lambda } } { { ( s_ { pool , i } ^2 ) } ^ \lambda } } \right] - 2 \ln { \pi _k } . \tag { 2.11 } \end{align*} \end{document}

The new rule of RDA is then defined as follows: Assign y to class k that minimizes the discriminant score \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\tilde d_k^R ( \textbf{\textit{ x}} )$$ \end{document} among all K classes. When \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda = 0$$ \end{document} , the new RDA reduces to DQDA (in 2.7). When \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda = 1 ,$$ \end{document} the new RDA reduces to DLDA (in 2.8). Noting that \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\ln [ ( s_{ki}^2{ ) ^{1 - \lambda }}{ ( s_{pool , i}^2 ) ^ \lambda } ] = ( 1 - \lambda ) \ln s_{ki}^2 + \lambda \ln s_{pool , i}^2$$ \end{document} , the proposed new variance estimator can also be regarded as an arithmetic mean between the logarithmic transformed variances of \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$s_{ki}^2$$ \end{document} and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$s_{pool , i}^2$$ \end{document} . In the next section, we will show that \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\ln [ ( s_{ki}^2{ ) ^{1 - \lambda }}{ ( s_{pool , i}^2 ) ^ \lambda } ]$$ \end{document} is a form that is much easier to work with compared with the form of \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\ln [ ( 1 - \lambda ) s_{ki}^2 + \lambda s_i^2 ]$$ \end{document} . In addition, for the purpose of bias correction for the proposed discriminant score (2.11), the expected value of \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ [ ( s_{ki}^2{ ) ^{1 - \lambda }}{ ( s_{pool , i}^2 ) ^ \lambda } ] ^{ - 1}}$$ \end{document} can be computed readily whereas there is no closed form for the expected value of \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ [ ( 1 - \lambda ) s_{ki}^2 + \lambda s_i^2 ] ^{ - 1}}$$ \end{document} . A discriminant rule based on the geometric mean can also be more robust to outliers than that with the arithmetic mean, especially when the sample size of a particular individual class (i.e., n_k) is extremely small so that the sample variance estimates \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$s_{ki}^2$$ \end{document} are very unstable.

2.4. Bias correction for GD-RDA

3.1. Simulation Studies

3.1.2. Simulation results

For each simulated dataset, we use the training samples to perform a gene selection procedure and then use the selected genes to build the classifier. Specifically, we select the top 50 differently expressed genes according to the ratio of between-group to within-group sums of squares for gene selection. For more details, see Section 5.1. Finally, to apply the CWA criterion for evaluation, we compute the CWAs by repeating the simulation 1000 times and taking an average over all the simulations. We report the CWAs along with various parameters in Figure 1 for the first two studies, and in Figure 2 for the last two studies, respectively.

OPEN IN VIEWER

FIG. 1.

The CWAs of all methods with different sample sizes for two classes. \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\delta$$ \end{document} of all plots equal to 0.5. The left panels have the same correlation matrix (Study 1), and the right panels have unequal correlation matrices (Study 2). There are three different correlation coefficients from top to bottom of the panels, specifically \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\rho = $$ \end{document} 0, 0.4, 0.8. CWA, class-weighted accuracy.

OPEN IN VIEWER

FIG. 2.

The CWAs of all methods with different sample sizes for three classes. The left panels have the same correlation matrix (Study 3), and the right panels have unequal correlation matrices (Study 4). There are three different correlation coefficients from top to bottom of the panels, specifically, \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\rho = $$ \end{document} 0, 0.4, 0.8.

Figures 1 and 2 show that GD-RDA performs significantly better than the other methods in all settings, especially for small sample sizes. We also note that, among the remaining methods, BQDA usually performs the best except for the results in Study 1, and SVM and kNN perform the worst in most studies. The CWAs of all methods increase with increasing the sample size and are related to the different choices of the correlation coefficient. The CWAs of all methods for \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\rho = 0.8$$ \end{document} are obviously lower than those for \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\rho = 0$$ \end{document} . In Study 1 (the left panels of Fig. 1), it is evident that BLDA performs a little better than BQDA and is much better than SVM and k NN. However, in Study 2 (the right panels of Fig. 1), we note that BQDA is much better than BLDA and is even comparable to GD-RDA when the sample size is large. Relative to Study 2 and Study 4, the CWAs of GD-RDA are much higher than the second performance method in Study 1 and Study 3.

3.2. Application to real data

We apply the proposed method to two real datasets, including a microarray dataset and an RNA-seq dataset, and compare it with some existing methods. We first give a brief introduction to the two datasets.

Dataset 1. The microarray dataset is about the breast cancer gene expression, which is available on the Broad institute http://portals.broadinstitute.org/cgi-bin/cancer/datasets.cgi. As described in Hoshida et al. (2007), the dataset consists of 1213 genes for 98 final expression values from three classes, including 11, 51, and 36 samples, respectively. We further standardize the dataset so that each array has mean zero and variance one across genes.

Dataset 2. We downloaded a new RNA-seq dataset as the second dataset from the UCSC cancer browser at https://genome-cancer.ucsc.edu. This dataset is about the lung squamous cell carcinoma (LUSC) miRNA expression. There are four classes of LUSC and 12,043 tags for 132 final expression values, with 28, 30, 37, and 37 samples for the four classes, respectively. After the log₂ transformation and the mean normalization across all samples (Sun and Zhao, 2015), we treat the RNA-seq data as a dataset with continuous expression values.

3.2.2. Results for real data

To compare the performance of all classification methods, we randomly divide the samples of each class into the training set and the test set. We set the training sample size from 4 to \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${t_1} + {t_2} - 1$$ \end{document} for the smallest class, where \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${t_1} + {t_2}$$ \end{document} equals to the total sample size of the smallest class. We use t₂ samples to test and the rest to train for other classes. We repeat this procedure 1000 times and report the average CWA for each method. The results for the two datasets are summarized in Figure 3. Here, we set top \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$T = 100$$ \end{document} and 500 for the two datasets, respectively.

OPEN IN VIEWER

FIG. 3.

The CWAs as a function of t₁ for the microarray (left) and RNA-seq (right) datasets.

From Figure 3, it is clear that the performance of the proposed method is consistently better than, or at least as well as, that of the other methods. We also note that, among the remaining methods, BLDA usually performs the best. SVM performs the worst in the first dataset, and the CWA of k NN is much lower than the other methods for large sample sizes in the second dataset.

We also show the results with a fixed training sample size but at a number of selected top T genes in Table 1 for the microarray dataset. As shown in Table 1, GD-RDA outperforms the other methods for all T values (the top ranked CWA highlighted in bold text). SVM shows the worst CWA for all T values. From the real data analysis, we conclude that GD-RDA is a robust method and performs better than other methods.

OPEN IN VIEWER

Table 1.

The Class-Weighted Accuracies for Microarray Datasets

T	30	50	100	200	300
DQDA	0.7491	0.7392	0.7224	0.7120	0.7096
BQDA	0.8316	0.8488	0.8569	0.8628	0.8663
DLDA	0.8403	0.8530	0.8598	0.8557	0.8584
BLDA	0.8372	0.8656	0.8839	0.8804	0.8787
SVM	0.6732	0.6806	0.6836	0.6772	0.6650
k NN	0.7324	0.7469	0.7644	0.7740	0.7762
GD-RDA	0.8594	0.8725	0.8863	0.8861	0.8892

DLDA, diagonal linear discriminant analysis; DQDA, diagonal quadratic discriminant analysis; GD-RDA, geometric diagonalization method for regularized discriminant analysis; SVM, support vector machine.

Proof of Theorem 1

To correct the bias for the discriminant score \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\tilde d_k^R ( \textbf{\textit{ y}} )$$ \end{document} (2.11), the plug-in estimates \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \tilde L_{k1}}$$ \end{document} and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \tilde L_{k2}}$$ \end{document} are used for estimating \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${L_{k1}}$$ \end{document} and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${L_{k2}}$$ \end{document} , respectively. In what follows, we show that both \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \tilde L_{k1}}$$ \end{document} and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \tilde L_{k2}}$$ \end{document} are biased estimators for any fixed \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\lambda$$ \end{document} . We then purpose two unbiased estimators for these two quantities and use them to form a bias-corrected rule for regularized discriminant analysis.

By (2.1), we have \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \bar x_{ki}} \sim N ( { \mu _{ki}} , \sigma _{ki}^2 / {n_k} )$$ \end{document} and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$s_{ki}^2 \sim \sigma _{ki}^2 \chi _{{n_k} - 1}^2 / ( {n_k} - 1 )$$ \end{document} for any \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$i = 1 , \ldots , p$$ \end{document} and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$k = 1 , \ldots , K$$ \end{document} , where \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\chi _ \nu ^2$$ \end{document} is the chi-square distribution with \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\nu$$ \end{document} degrees of freedom. Then, by \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$s_{pool , i}^2 = { \left( { \prod \nolimits_{k = 1}^K {s_{ki}^2} } \right) ^{1 / K}}$$ \end{document} and the fact that \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$s_{1i}^2 , \ldots , s_{ki}^2$$ \end{document} and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \bar x_{ki}}$$ \end{document} are mutually independent, for the plug-in estimator \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \tilde L_{k1}}$$ \end{document} we have \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} \begin{split} & E ( { \tilde L_ { k1 } } ) = \mathop \sum \limits_ { i = 1 } ^p E { ( { y_i } - { \bar x_ { ki } } ) ^2 } E \left[ { { { ( s_ { ki } ^2 ) } ^ { \lambda - 1 } } { { ( s_ { pool , i } ^2 ) } ^ { - \lambda } } } \right] \\ & = \mathop \sum \limits_ { i = 1 } ^p E { ( { y_i } - { \bar x_ { ki } } ) ^2 } E { ( s_ { 1i } ^2 ) ^ { - { \textstyle \frac { \lambda } { K } } } } \cdots \\ & E { ( s_ { ki } ^2 ) ^ { \lambda - 1 - { \textstyle \frac { \lambda } { K } } } } \cdots E { ( s_ { Ki } ^2 ) ^ { - { \textstyle \frac { \lambda } { K } } } } \\ & = \mathop \sum \limits_ { i = 1 } ^p \left[ { { { ( { y_i } - { \mu _ { ki } } ) } ^2 } + { \frac { \sigma _ { ki } ^2 } { { n_k } } } } \right] { \frac { { { ( \sigma _ { 1i } ^2 ) } ^ { - { \textstyle \frac { \lambda } { K } } } } } { h ( { n_1 } , - { \textstyle \frac { \lambda } { K } } ) } } \\ & \cdots { \frac { { { ( \sigma _ { ki } ^2 ) } ^ { \lambda - 1 - { \textstyle \frac { \lambda } { K } } } } } { h ( { n_k } , \lambda - 1 - { \textstyle \frac { \lambda } { K } } ) } } \cdots { \frac { { { ( \sigma _ { Ki } ^2 ) } ^ { - { \textstyle \frac { \lambda } { K } } } } } { h ( { n_K } , - { \textstyle \frac { \lambda } { K } } ) } } \\ & = \frac { 1 } { { { B_k } } } \mathop \sum \limits_ { i = 1 } ^p \left[ { { { ( { y_i } - { \mu _ { ki } } ) } ^2 } + { \frac { \sigma _ { ki } ^2 } { { n_k } } } } \right] { ( \sigma _ { ki } ^2 ) ^ { \lambda - 1 } } { ( \sigma _ { pool , i } ^2 ) ^ { - \lambda } } \\ & = \frac { 1 } { { { B_k } } } { L_ { k1 } } + { \frac { p { C_k } } { { n_k } { B_k } } } , \end{split} \tag { 5.13 } \end{align*} \end{document}

where \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$ { B_k } = ( h ( { n_k } , \lambda - 1 - { \textstyle \frac { \lambda } { K } } ) / h ( { n_k } , - { \textstyle \frac { \lambda } { K } } ) ) \prod \nolimits_ { j = 1 } ^K h ( { n_j } , - { \textstyle \frac { \lambda } { K } } )$$ \end{document} and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} { C_k } = \frac { 1 } { p } \mathop \sum \limits_ { i = 1 } ^p { { { \left( { { \frac { \sigma _ { ki } ^2 } { \sigma _ { pool , i } ^2 } } } \right) } ^ \lambda } } . \end{align*} \end{document}

Note that C_k is unknown in practice since it involves the unknown variances. We propose to estimate C_k by \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} { \hat C_k } = \frac { { { D_k } } } { p } \mathop \sum \limits_ { i = 1 } ^p { { { \left( { { \frac { s_ { ki } ^2 } { s_ { pool , i } ^2 } } } \right) } ^ \lambda } } , \tag { 5.14 } \end{align*} \end{document}

where \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$ { D_k } = h ( { n_k } , \lambda - { \textstyle \frac { \lambda } { K } } ) h ( { n_k } , { \textstyle \frac { \lambda } { K } } ) / \prod \nolimits_ { j = 1 } ^K h ( { n_j } , { \textstyle \frac { \lambda } { K } } )$$ \end{document} . To investigate the asymptotic properties of \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \hat C_k} ,$$ \end{document} we consider the ratios \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\sigma _{ki}^2 / \sigma _{pool , i}^2$$ \end{document} as a random sample of size p from a common distribution F with a finite second moment. Then, by the strong law of large numbers, it is easy to verify that \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \hat C_k} - {C_k} \mathop \to \limits^{a.s.} 0$$ \end{document} as \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$p \to \infty$$ \end{document} , where \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\mathop \to \limits^{a.s.}$$ \end{document} represents almost sure convergence. Finally, by (5.13) and (5.14), we propose the following asymptotically unbiased estimator for \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${L_{k1}}$$ \end{document} : \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} { \mathord {\breve{L}} _ { k1 } } = { B_k } \mathop \sum \limits_ { i = 1 } ^p { \frac { { { ( { y_i } - { { \bar x } _ { ki } } ) } ^2 } } { { { ( s_ { ki } ^2 ) } ^ { 1 - \lambda } } { { ( s_ { pool , i } ^2 ) } ^ \lambda } } } - { \frac { { D_k } } { { n_k } } } \mathop \sum \limits_ { i = 1 } ^p { \left( { { \frac { s_ { ki } ^2 } { s_ { pool , i } ^2 } } } \right) ^ \lambda } . \tag { 5.15 } \end{align*} \end{document}

Now for the plug-in estimator \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \tilde L_{k2}}$$ \end{document} , to calculate its expectation, we need to apply the formula that \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$E ( \ln \chi _ \nu ^2 ) = \Psi ( \nu / 2 ) + \ln 2$$ \end{document} , where \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$\Psi ( \cdot )$$ \end{document} is the so-called digamma function proposed by Abramowitz and Stegun (1972). Then by the facts that \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$s_{pool , i}^2 = { \left( { \prod \nolimits_{k = 1}^K {s_{ki}^2} } \right) ^{1 / K}}$$ \end{document} and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$s_{ki}^2 \sim \sigma _{ki}^2 \chi _{{n_k} - 1}^2 / ( {n_k} - 1 )$$ \end{document} , we have \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} \begin{split} E ( { \tilde L_ { k2 } } ) &= ( 1 - \lambda ) \mathop \sum \limits_ { i = 1 } ^p E ( \ln s_ { ki } ^2 ) + \frac { \lambda } { K } \mathop \sum \limits_ { i = 1 } ^p \mathop \sum \limits_ { j = 1 } ^K E ( \ln s_ { ji } ^2 ) \\ & = ( 1 - \lambda ) \mathop \sum \limits_ { i = 1 } ^p \left[ { \Psi ( \frac { { { n_k } - 1 } } { 2 } ) + \ln 2 + \ln \left( { { \frac { \sigma _ { ki } ^2 } { { n_k } - 1 } } } \right) } \right] \\ &\quad + \frac { \lambda } { K } \mathop \sum \limits_ { i = 1 } ^p \mathop \sum \limits_ { j = 1 } ^K \left[ { \Psi ( \frac { { { n_j } - 1 } } { 2 } ) + \ln 2 + \ln \left( { { \frac { \sigma _ { ji } ^2 } { { n_j } - 1 } } } \right) } \right] \\ & = { L_ { k2 } } + { E_k } , \end{split} \tag { 5.16 } \end{align*} \end{document}

where \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $$ { E_k } = ( 1 - \lambda ) p ( \Psi ( { \textstyle \frac { { { n_k } - 1 } } { 2 } } ) - \ln ( { \textstyle \frac { { { n_k } - 1 } } { 2 } } ) ) + { \textstyle \frac { { \lambda p } } { K } } \sum \nolimits_ { j = 1 } ^K ( \Psi ( { \textstyle \frac { { { n_j } - 1 } } { 2 } } ) - \ln ( { \textstyle \frac { { { n_j } - 1 } } { 2 } } ) )$$ \end{document} . By (5.16), we propose the following unbiased estimator for \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${L_{k2}}$$ \end{document} : \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} { \mathord{\breve{L}} _{k2}} = \mathop \sum \limits_{i = 1}^p \ln \left[ {{{ ( s_{ki}^2 ) }^{1 - \lambda }}{{ ( s_{pool , i}^2 ) }^ \lambda }} \right] - {E_k}. \tag{5.17} \end{align*} \end{document}

Finally, by the proposed bias-corrected estimators (5.15) and (5.17), we get the bias-corrected discriminant score as follows: \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} \mathord{\breve{d}} _k^{ R} ( \textbf{\textit{y}} ) = \;{ \mathord{\breve{L}} _{k1}} + {\mathord{\breve{L}} _{k2}} - 2 \ln { \pi _k} , \end{align*} \end{document}

This completes the proof of the theorem.