Integrating generalized domain adaptation and Fisher discriminative learning: A unified framework for face recognition with single sample per person

Abstract

In this paper, an enhanced discriminative feature learning (EDFL) method is proposed to address single sample per person (SSPP) face recognition. With a separate auxiliary dataset, EDFL integrates Fisher discriminative learning and domain adaptation into a unified framework. The separate auxiliary dataset and the gallery/probe dataset are from two different domains (named source and target domains respectively) and have different data distributions. EDFL is modeled to transfer the discriminative knowledge learned from the source domain to the target domain for classification. Since the gallery set with SSPP contains scarce number of samples, it is hard to accurately represent the data distribution of the target domain, which hinders the adaptation effect. To overcome this problem, the generalized domain adaption (GDA) method is proposed to realize good overall domain adaptation when one domain contains limited samples. GDA considers the both global and local domain adaptation effect at the same time. Further, to guarantee that the learned domain adaptation components are optimal for discriminative learning, the domain adaptation and Fisher discriminant model learning are unified into a single framework and an efficient algorithm is designed to optimize them. The effectiveness of the proposed approach is demonstrated by extensive evaluation and comparison with some state-of-the-art methods.

Keywords

Single sample per person domain adaptation discriminative feature learning feature selection

1 Introduction

Single sample per person (SSPP) face recognition is a great challenge in artificial intelligence and pattern recognition, where only single sample per person (SSPP) is available in the training set, and approaches developed for multiple samples per person (MSPP) are not effective. In real world applications, SSPP face recognition is quite realistic and important. Mostly, in such scenarios, the training face samples are limited, and it is difficult to collect more samples per person. For example, in the application of passport identification, law enforcement, and gate ID identification; only one profile face image per individual is archived in the database. This leads to SSPP problem. Numerous techniques have been exclusively designed to deal with SSPP face recognition. These approaches can be categorized into three main classes: image partition [2, 7], virtual sample generation [25, 26] and generic learning [17, 28].

To address SSPP face recognition, in this paper we employ an auxiliary set which could be different from the gallery set and propose enhanced discriminative feature learning (EDFL) method. The proposed EDFL incorporates generic learning and domain adaptation. In EDFL, the auxiliary set is treated as the source domain, the gallery and probe sets are treated as the target domain. EDFL aims to learn the discriminative knowledge from the source domain which is optimal for the classification task in the target domain.

1.1 Related work

As the proposed method is also a generic learning method, we provide a detailed survey about the third class. More interested readers are encouraged to consult [11, 16] for detailed reviews about image partition and virtual sample generation for SSPP. Essentially, the image partition-based and virtual sample generation-based methods make the best use of the one face sample for each person. Generally, these methods are incompetent to handle complex variations common in faces such as extreme expression and pose, varied illumination and occlusion. Except for the gallery set, generic learning-based methods utilize a new training data set, i.e., generic set, which includes multiple samples and sample labels for each subject. With the generic set, the intra-personal as well as the inter-personal variations of each subject can be explored in the training process. The generic set used in generic learning-based methods can be either dependent or independent of the gallery set. Depending upon the relationship between the generic set and the gallery set, we further divide the generic learning-based methods into two sub-classes, i.e., approaches utilizing dependent generic sets and separate generic sets.

Deng et al. [5] proposed extended sparse representation based classification (ESRC) to deal with SSPP face recognition. The main idea of ESRC is to develop an intra-class variant dictionary from the generic training set, the the dictionary can be used to encode the possible variations between the gallery and test images. Li et al. [14] designed a sparse representation model which ensures not only the sparsity of representation coefficient but also the robustness for the variational information. Aiming at encoding the occlusion information in SSPP face recognition, Yu et al. [31] developed discriminative multi-scale sparse coding technique via a learned dictionary from the generic set. Wei et al. [22] proposed the robust auxiliary dictionary learning (RADL) framework, which simultaneously achieves the goal of auxiliary dictionary learning and robust sparse coding. Gao et al. [7] presented regularized patch-based representation for SSPP to encode possible variations in the test images. These methods assume that the generic set and the gallery set share the similar intra-class and inter-class variations, which does not necessarily hold in all cases. For instance, the generic face dataset may be a new database containing face images captured from different ethnicities or under different imaging conditions compared to the gallery set. In such situations, the above-mentioned methods may not perform that well.

To advance the generalization of generic learning, methods employing separate generic dataset (i.e., the new auxiliary set independent of the gallery) are proposed. As the auxiliary set is distinctive from the gallery set, the relationship between them must be considered. For example, both Su et al. [21] and Kan et al. [12] proposed to infer the within-class and between-class scatter matrix of gallery set by using the intra-class and inter-class scatter matrices computed from the auxiliary set. While the former work adopts a coupled linear representation, the latter one leverages k-NN regression or Lasso regression to characterize the relationship. Besides, dictionary learning technique has also been explored. Yang et al. [27] proposed the sparse variation dictionary learning (SVDL) to learn the gallery set adaptive dictionary.

In order to capture the non-linear variations caused by poses, Mokhayeri and Granger [17] proposed a paired sparse representation model by jointly using variational dictionary and gallery dictionary augmented with a set of synthetic images. Recently, Cuculo et al. [3] considered to adopt the large datasets of images acquired in the wild, then proposed sparsity-driven sub-dictionary learning framework to handle the various variations in the probe faces. Convolutional neural networks (CNN) based deep learning framework have also been developed to address SSPP face recognition. These methods naturally require a larger sized and independent generic dataset to train the model. Yang et al. [28] proposed to extract local adaptive convolution features of images by embedding joint and collaborative representation into CNN framework. Further, Yang et al. [29] developed a novel class-level joint representation framework. As the data augmentation techniques, generative adversarial neural networks (GANs) are also used for addressing SSPP face recognition. Zhang et al. [32] proposed to learn illumination variations with GANs, and developed a model to specifically deal with SSPP face recognition with different illuminations; similarly, Wu et al. [24] proposed the multi-domain dictionary learning (MDDL), which employs GANs to learn different data styles, such as smile face, occlusion face, from the source domain data, and augmented target data with these learned data styles. Our wok is somewhat similar to MDDL, since both use domain adaptation to capture the probe face variations. Nevertheless, the proposed EDFL is quite different from MDDL in the way to achieve domain adaptation. MDDL adds the styles learned from the source domain to the gallery set, so that the target domain owns the similar styles of source domain; while EDFL uses learned projections to map both the source domain and target domain in a common subspace, in which the data distributions of these two domains are minimized

1.2 Motivation and contribution

1.2.1 Motivation

Most of the above generic learning methods rely on dictionary learning to characterize the possible face variations of probe face, which require that the gallery dataset and the auxiliary dataset share the same or similar data distribution. In practice, however, a separate dataset which is independent of the gallery set is more available. When this kind of generic sets have quite different distributions from the gallery set, the performance will dramatically drop. Kan et al. [13] demonstrated that the recognition accuracy drops from 96%to 59%, when use the face recognition model trained from Mongolian face images to recognize Caucasian faces. This is because the two datasets own different distributions, i.e., they come from distinct domains (named source and target domain, respectively). Consequently, the knowledge learned from the original subspace of generic set (source domain) might not be best for conducting matching between the gallery and probe images (target domain).

To make full use of information embedded in the source domain, reducing the discrepancy between the source and target domain is essential. Domain adaptation or transfer learning has been a popular technique which is able to hold the distribution mismatch between two domains. Moreover, to achieve the goal of face recognition, discriminative learning is indispensable. As a result, there will be two subspaces to be learned, one is for domain adaptation and the other is for discriminative learning. Motived by [9], learning the two subspaces simultaneously may make them best for each other, thus produce good face recognition performance. Therefore, in this paper, we integrate domain adaptation and Fisher discriminative learning in a unified framework to address SSPP face recognition.

1.2.2 Contribution

The main contribution of this work is as follows. Firstly, we design a generalized domain adaptation (GDA) model to realize good overall domain adaptation between the gallery set containing limited samples (target domain) and the auxiliary set (source domain). Since the gallery set of SSPP face recognition contains only one sample per person, it is difficult to accurately represent the real distribution of face images in the probe set (target domain). The proposed GDA seeks both the local and global structure information across domains to minimize the distribution mismatch between the auxiliary set and gallery set. GDA guarantees that the discriminative information exploited from the generic set can be compatible with the gallery and probe set. GDA is helpful to achieve better domain adaptation for situations where one domain has too limited samples to characterize the latent real distribution of the domain. Secondly, we consider the discriminative learning of the generic set, and integrate it with GDA into a unified learning framework to deal with SSPP face recognition, i.e., enhanced discriminative feature learning (EDFL). In EDFL, domain adaptation and discriminative learning can be accomplished simultaneously. Thirdly, we conduct extensive analysis and evaluation of EDFL to show its effectiveness. We also demonstrate that EDFL works well on deep features.

The remainder of this paper is organized as follows. Section 2 presents the details of the proposed EDFL method, including its formulation and optimization. In Section 3, we evaluate EDFL on three widely used databases under two scenarios, i.e., (i) enhanced discriminative feature learning across ethnicity and (ii) enhanced discriminative feature learning across imaging condition. Finally, we conclude the work in Section 4.

2 Enhanced discriminative feature learning

2.1 Notations

Let $X^{s} \in ℝ^{d \times n_{s}}$ be a dataset of training images from the source domain, d is the dimension of samples and n_s is the total number of training images in the source domain. X^s consists of face images from c₁ classes with each class having l samples; all samples are labeled. As an auxiliary dataset, face images in X^s involve diverse variations, such as illumination, expression, and occlusion. Similarly, let $X^{t} \in ℝ^{d \times n_{t}}$ be the dataset of unlabeled training images from target domain, n_t is the total number of training images in the training set of target domain. For SSPP face recognition problem, the target training dataset X^t is usually a gallery set which contains only one face sample for each individual. Thus n_t is equal to the number of classes c₂ in target dataset. Unless otherwise stated, the symbols s and t used in the superscript or subscript mean the source and target domain, respectively.

2.2 Generalized domain adaptation

In general, face images in the source domain and the target domain are captured from different ethnicities, under different periods or different imaging conditions. This causes the two domains to have different feature spaces or different marginal probability distributions. To reduce the domain difference, Pan et al. [19] proposed transfer component analysis (TCA) to project source and target data onto a learned transferring subspace for domain adaptation. TCA diminishes the distance between different distributions across domains by minimizing the maximum mean discrepancy of transformed features in the latent subspace, meanwhile preserving the property of source and target data by maximizing the data variance. The formulation of TCA can be expressed by the following: $min_{R} {∥ \frac{1}{n_{s}} R^{T} X^{s} 1^{s} - \frac{1}{n_{t}} R^{T} X^{t} 1^{t} ∥}_{F}^{2} + λ {∥ R ∥}_{F}^{2},$ (1)

$s . t . \begin{matrix} \end{matrix} R^{T} {XHX}^{T} R = I_{m},$ (2) where $X = [X^{s}, X^{t}] \in ℝ^{d \times (n_{s} + n_{t})}$ , $R \in ℝ^{d \times m} (m < d)$ is the learned latent transferring subspace $1^{s} \in ℝ^{n_{s} \times 1}$ , $1^{t} \in ℝ^{n_{t} \times 1}$ , $H \in ℝ^{n \times n}$ , n = n_s + n_t, H = I_n - (1/n) 11^T, I denotes the identity matrix, λ is a scalar. The regularization term $λ {∥ R ∥}_{F}^{2}$ is used to control the complexity of learned R.

In Equation (1), $\frac{1}{n_{s}} R^{T} X^{s} 1^{s}$ and $\frac{1}{n_{t}} R^{T} X^{t} 1^{t}$ are the means of transformed features from the source and target domain respectively. In the latent subspace of TCA, it reduces the distribution mismatch between two domains by reducing the distance of the means of their transformed features. Therefore, the adaptation performance is closely bound by the sample means.

The gallery set of SSPP face recognition comprises of only one sample per person, it lacks substantial images which involve various variations of faces. Therefore, the mean of the target training set cannot characterize the distribution of real face images well. In such situation, TCA will fail to reduce the distribution mismatch between two domains and domain adaptation is far from being realized. To address this issue, we propose the generalized domain adaptation (GDA) method, which utilizes the partial information exploited from the source domain data to re-estimate the real mean feature of the target domain, and reduces the distribution mismatch between two domains both globally and locally. Figure 1 illustrates the pipeline of GDA, it consists of two parts, which are designed to realize global and local domain adaptation, respectively.

Fig. 1

The pipeline of GDA (Take κ = 3 as an example).

In Fig. 1, Step ding172–ding175 are used to generate virtual face images according to the gallery set in the target domain and auxiliary dataset in the source domain, which means to augment the sample size of target domain. Figure 2 shows four gallery faces (the upper two and bottom two are from FERET and LFW dataset, respectively) and their corresponding virtual face images, each person has four virtual images and each row represents one subject. The virtual face images are produced when CAS-PEAL dataset is used as the source data. The reason for generating virtual face images is inferring the possible face variations in target domain. Specifically, as the gallery set of SSPP is hard to represent the distribution of the target domain, we utilize the face variations in the source domain to roughly infer the variations in target domain, so that we can also roughly estimate the data distribution of target domain. It is obvious that the estimated distribution by virtual images is not the same as the underlying true data distribution of target domain, however, as the virtual images do contain some face variations, the estimated distribution should be more reasonable than that one reflected by the original gallery set, which contains single sample per person. Therefore, it is helpful for the global domain adaptation.

Fig. 2

The gallery face images (1^st column) and corresponding virtual face images (2nd –5th column).

Specifically, the proposed GDA works as follows.

Compute the class mean $m_{ς} = \frac{1}{l} \sum_{r = 1}^{l} x_{ς r}^{s}$ for every class in the source domain.

Obtain the sample variation $v_{ς r} = x_{ς r}^{s} - m_{ς}$ for each sample in the source domain, where ς = 1, ⋯ , c₁ and r = 1, ⋯ , l.

For each single sample $x_{ξ}^{t} \in X^{t}$ in the target domain, find its κ-nearest neighbors in the source domain. Further, find out and collect the sample variations of these κ-nearest neighbors, $V_{ξ} = \underset{κ}{\underset{︸}{[v_{ξ 1}, v_{ξ 2}, \dots, v_{ξ κ}]}} \subset \underset{c_{2} l}{\underset{︸}{[v_{11}, \dots, v_{ς r}, \dots, v_{c_{2} l}]}}$ , in which ξ = 1, ⋯ , n_t. Then, for the κ selected sample variations, assign them various weights w_ξτ according to the similarity between $x_{ξ}^{t}$ and its κ-nearest neighbors in X^s, where τ = 1, ⋯ , κ.

Use X^t and ‘Variations’ of source domain to create an augmented dataset X^(T) in the target domain, each sample in X^(T) is obtained by $x_{ξ τ}^{(T)} = x_{ξ}^{t} + w_{ξ τ} v_{ξ}^{τ}$ , in which ξ = 1, ⋯ , n_t and τ = 1, ⋯ , κ. As a result, X^(T) contains κ samples for each class, κn_t samples in total. In this paper, we employ sparse representation to reconstruct $x_{ξ}^{t}$ with all samples in X^s. The samples corresponding to the first κ largest reconstruction coefficients are picked out as the κ-nearest neighbors; meanwhile, these κ rcording to the similarity betweeneights w_ξτ of sample variation.

Global domain adaptation: Use the new dataset in the target domain X^(T) and the original dataset in source domain X^s to seek global domain adaptation. Note that there are multiple samples per subject in each domain.

Local domain adaptation: Use the new dataset X^(S) = [m₁, m₂ ⋯ , m_{c
₁}] in the source domain and the original dataset X^t in target domain to realize local domain adaptation. For the i - th sample in X^t, let $X_{i}^{t}$ contains its k-nearest neighbors in X^t, $X_{i}^{(S)}$ contains its k nearest neighbors in X^(S), $X_{i}^{t} \subset X^{t}$ and $X_{i}^{(S)} \subset X^{(S)}$ . The neighbor relationship of data is widely used in characterizing the manifold structure of data [23]. In this part, each sample in X^t is regarded as a bridge which correlates two subsets containing its neighbors in different domains. GDA employs these coupled subsets to preserve local domain adaptation.

Generalized domain adaptation: By combining global and local domain adaptation in the above, the GDA model can be formulated as the following: $\begin{matrix} min_{R} {∥ \frac{1}{n_{s}} R^{T} X^{s} 1^{s} - \frac{1}{κ n_{t}} R^{T} X^{(T)} 1^{(T)} ∥}_{F}^{2} \\ + \begin{matrix} \end{matrix} \sum_{i = 1}^{n_{t}} {∥ \frac{1}{k} R^{T} X_{i}^{(S)} 1_{o} - \frac{1}{k} R^{T} X_{i}^{t} 1_{0} ∥}_{F}^{2} + λ {∥ R ∥}_{F}^{2} \end{matrix},$ (3) $s . t . \begin{matrix} \end{matrix} R^{T} {XHX}^{T} R + R^{T} (\sum_{i = 1}^{n_{t}} X_{i} H_{o} (X_{i})^{T}) R = I_{m},$ (4)

where

R \in ℝ^{d \times m} (m < d)

is the learned latent transferring subspace,

1^{s} \in ℝ^{n_{s} \times 1}

1^{(T)} \in ℝ^{κ n_{t} \times 1}

1_{o} \in ℝ^{k \times 1}

, and ∥ • ∥ _F denotes Frobenius norm of matrix;

X = [X^{s}, X^{(T)}] \in ℝ^{d \times n}

, n = n_s + κn_t,

H \in ℝ^{n \times n}

H = I_{n} - \frac{1}{n} 11^{T}

;

X_{i} = [X_{i}^{(S)}, X_{i}^{t}] \in ℝ^{d \times 2 k}

H_{o} = I_{2 k} - \frac{1}{2 k} 11^{T}

The first and second term of Equation (3) are used to reduce the distance between the global means of two domains, the distances between all coupled local means across two domains, respectively. The left two terms of Equation (4) are applied to preserves the maximum covariance of two domains globally and locally, respectively. Consequently, GDA attempts to minimize the distribution mismatch meanwhile preserves the maximum covariance between two domains from both global and local perspective.

Note X^(T) is an augmented dataset of X^t, each sample in X^(T) is obtained by adding a variation $v_{ς r} = x_{ς r}^{s} - m_{ς}$ extracted from X^s. By using X^t and V_ξ = [v_ξ1, v_ξ2, ⋯ , v_ξκ], the first term of Equation (3) can be rewritten as $\begin{matrix} ∥ \frac{1}{n_{s}} R^{T} X^{s} 1^{s} - \\ {(\frac{1}{κ n_{t}} R^{T} (κ X^{t}) 1^{κ n_{t}} + \frac{1}{κ n_{t}} R^{T} (\sum_{ξ}^{n_{t}} \sum_{τ}^{κ} w_{ξ τ} v_{ξ τ}) 1^{κ n_{t}}) ∥}_{F}^{2} . \end{matrix}$

Hence, in the global domain adaptation, GDA appends the variations to the target domain, i.e., $\sum_{ξ}^{n_{t}} \sum_{τ}^{κ} w_{ξ τ} v_{ξ τ}$ , to estimate new distribution center of the target domain. Compared with the first term in Equation (1), the new estimated mean of target domain considers the possible sample variations. As the added variations are exploited from the source domain only, the global domain adaptation may be coarse in the adaptation result. To offset the drawback, we further propose the local domain adaptation, which engenders the local similarity of two domains. With the global and local adaptation together, GDA guarantees promising domain adaptation effect.

2.3 Formulation of EDFL

After obtaining the transformation matrix R for domain adaptation, we can get the new representations for source and target data in the latent transferring subspace by Y^s = R^TX^s, Y^t = R^TX^t. In the latent subspace, Y^s and Y^t have the same marginal distribution or own the same feature subspace. To further conduct classification task, we can learn a discriminative subspace $W \in ℝ^{m \times p}$ using Y^s and Y^t according to the objective function of LDA: $max_{W} \begin{matrix} \end{matrix} tr (W^{T} S_{b} W) / tr (W^{T} S_{w} W),$ (5) where tr (•) denotes the trace of a matrix, S_b and S_w are the between-class and within-class scatter matrices computed by all sample features in the latent subspace, respectively. $S_{b} = S_{b}^{(s)} + S_{b}^{(t)}$ , and $S_{b}^{(s)}$ , $S_{b}^{(t)}$ are the between-class scatter matrix computed by source data and target data in the latent subspace, respectively; $S_{w} = S_{w}^{(s)} + S_{w}^{(t)}$ , and $S_{w}^{(s)}$ , $S_{w}^{(t)}$ are the within-class scatter matrix computed by source data and target data in the latent subspace, respectively. As there is only one sample per subject in the target training dataset, $S_{w}^{(t)} = 0$ ; $S_{b}^{(s)}$ , $S_{w}^{(s)}$ and $S_{b}^{(t)}$ are calculated by the following formulations. $\begin{matrix} S_{b}^{(s)} = \sum_{i = 1}^{c_{1}} ({\bar{Y}}_{i}^{s} - {\bar{Y}}^{s}) {({\bar{Y}}_{i}^{s} - {\bar{Y}}^{s})}^{T} \\ = R^{T} (\sum_{i = 1}^{c_{1}} ({\bar{X}}_{i}^{s} - {\bar{X}}^{s}) ({\bar{X}}_{i}^{s} - {\bar{X}}^{s})) R = R^{T} G_{b}^{(s)} R \end{matrix},$ (6)

$\begin{matrix} S_{w}^{(s)} = \sum_{i = 1}^{c_{1}} \sum_{j = 1}^{l} (Y_{ij}^{s} - {\bar{Y}}_{i}^{s}) {(Y_{ij}^{s} - {\bar{Y}}_{i}^{s})}^{T} \\ = R^{T} (\sum_{i = 1}^{c_{1}} \sum_{j = 1}^{l} (X_{ij}^{s} - {\bar{X}}_{i}^{s}) {(X_{ij}^{s} - {\bar{X}}_{i}^{s})}^{T}) R = R^{T} G_{w}^{(s)} R \end{matrix},$ (7) $\begin{matrix} S_{b}^{(t)} = \sum_{i = 1}^{c_{2}} ({\bar{Y}}_{i}^{t} - {\bar{Y}}^{t}) {({\bar{Y}}_{i}^{t} - {\bar{Y}}^{t})}^{T} \\ = R^{T} (\sum_{i = 1}^{c_{2}} ({\bar{X}}_{i}^{t} - {\bar{X}}^{t}) {({\bar{X}}_{i}^{t} - {\bar{X}}^{t})}^{T}) R = R^{T} G_{b}^{(t)} R \end{matrix} .$ (8)

In Equation (6–8), $S_{b}^{(s)}$ and $S_{b}^{(t)}$ are the between-class scatter matrix computed by source and target data in the original subspace, respectively; $S_{w}^{(s)}$ is the within-class scatter matrix computed by source data in the original subspace.

Note that the discriminative subspace W is learned after projecting the original data into the learned transferring subspace R, i.e., R and W are learned sequentially. There is one problem of this scheme: sequential learning of R and W means that W is learned after finishing learning R, which cannot ensure the new representations obtained by the transferring matrix R are optimal for generating the discriminative subspace. To solve this problem, we propose to combine the learning of R and W into a unified framework, which is able to simultaneously learn the optimal R and W. This framework can guarantee that the learned R is best for W, and vice versa. Also, to enable the discriminative subspace W to be robust to outliers and noise, we adopt the ℓ₂₁ norm of W as a sparsity regularizer for feature selection. By considering joint generalized domain adaptation and discriminative subspace learning, we formulate the objective function of EDFL as $min_{R, W} {\begin{matrix} {∥ \frac{1}{n_{s}} R^{T} X^{s} 1^{s} - \frac{1}{κ n_{t}} R^{T} X^{(T)} 1^{(T)} ∥}_{F}^{2} \\ + \sum_{i = 1}^{n_{t}} {∥ \frac{1}{k} R^{T} X_{i}^{(S)} 1_{o} - \frac{1}{k} R^{T} X_{i}^{t} 1_{0} ∥}_{F}^{2} + λ {∥ R ∥}_{F}^{2} \\ - \frac{tr (W^{T} R^{T} G_{b} RW)}{tr (W^{T} R^{T} G_{w} RW)} + γ {∥ W ∥}_{2, 1} \end{matrix}},$ (9) $s . t . \begin{matrix} \end{matrix} R^{T} {XHX}^{T} R + R^{T} (\sum_{i = 1}^{n_{t}} X_{i} H_{o} (X_{i})^{T}) R = I_{m} .$ (10)

In Equation (9), γ is a regularization coefficient controlling the extent of feature selection, a larger value of γ means that more rows of W shrink to zero, so that fewer features are selected.

To make the objective function solvable, we relax the above optimization problem to be the following one: $\begin{matrix} min_{R, W} R^{T} ({XLX}^{T} + \sum_{i = 1}^{n_{t}} X_{i} L_{o} (X_{i})^{T}) R + λ {∥ R ∥}_{F}^{2} \\ + tr (W^{T} R^{T} G_{w} RW) - tr (W^{T} R^{T} G_{b} RW) + γ {∥ W ∥}_{2, 1} \end{matrix},$ (11) $\begin{matrix} s . t . \begin{matrix} \end{matrix} R^{T} ({XHX}^{T} + \sum_{i = 1}^{n_{t}} X_{i} H_{o} (X_{i})^{T}) R = I_{m} \\ \begin{matrix} \end{matrix} W^{T} W = I_{p} \end{matrix} .$ (12) where L and L_o are indicator matrices: L_(ij) = 1/(n_s) ² if i ⩽ n_s and j ⩽ n_s, L_(ij) = 1/(κn_t) ² if i > n_s and j > n_s, otherwise, L_(ij) = -1/κn_tn_s; L_o(ij) = 1/k² if i ⩽ k and j ⩽ k, or i > k and j > k, otherwise, L_o(ij) = -1/k². We add a constraint W^TW = I_p to the relaxed optimization problem, this new constraint is used to get orthogonal W and meanwhile to avoid some trivial solutions of W. Unfortunately, the relaxed objective function is still not easy to solve due to the first constraint in Equation (12). For ease of optimization, we follow the work of [18] and introduce an intermediate variable Q. Then by adding a coherent regularizer ${∥ Q^{T} R ∥}_{F}^{2}$ , we can promote the coherence between R and Q: R is more coherent to Q if the value of ${∥ Q^{T} R ∥}_{F}^{2}$ is larger. As a result, the complete formulation of the proposed EDFL becomes: $\begin{matrix} min_{R, W, Q} R^{T} ({XLX}^{T} + \sum_{i = 1}^{n_{t}} X_{i} L_{o} (X_{i})^{T}) R + λ {∥ R ∥}_{F}^{2} \\ - {∥ Q^{T} R ∥}_{F}^{2} + tr (W^{T} Q^{T} (μ G_{w} - G_{b}) QW) + γ {∥ W ∥}_{2, 1} \end{matrix},$ (13) $\begin{matrix} s . t . \begin{matrix} \end{matrix} R^{T} ({XHX}^{T} + \sum_{i = 1}^{n_{t}} X_{i} H_{o} (X_{i})^{T}) R = I_{m} \\ \begin{matrix} \end{matrix} W^{T} W = I_{p} \end{matrix} .$ (14)

2.4 Optimization of EDFL

As the optimization problem of Equation (13) is not convex for all variables, we exploit an alternative optimization strategy to iteratively solve the projection matrices (R, W, Q) until the objective value of Equation (13) converges. It can be solved by augmented Lagrangian multipliers (ALM) method. The detailed solution process can be found in supplementary.

For clarity, the overall algorithm to solve the optimization problem of EDFL formulation is summarized in Algorithm 1. The objective value of Equation (13) monotonically decreases as the number of iterations increases, and usually converges within 20 iterations in our experiments. By unifying R and W into the same optimization problem and optimizing them alternatively, we can assure that the transferring learning subspace R is optimal for transferring learning, meanwhile W is the best discriminative projection for the new representations in the transferring subspace. For samples in X^t, the features are extracted by fea^(t) = W^TR^TX^t, and ${fea}^{(t)} \in ℝ^{p \times n_{s}}$ .

Table 7

Algorithm 1 Enhanced Discriminative Feature

Learning (EDFL)

1:     Input: The source data X^s, the target data X^t.

2:     Initialize the dimension of transferring projection

        matrix m, the dimension of discriminative projection

        matrix p, Q = 10 × I_m×m, λ = 1e2, γ = 1e3, μ = 1,

        η = 1e - 2.

3:    While not converge do

4:     Solve the eigen-decomposition problem (XLX^T + λI_d - QQ^T) R = (XHX^T) RΛ.

5:     Update R = [r₁, r₂, ·· · , r_m], where r₁, r₂, ·· · , r_m are the eigenvectors corresponding to the first m smallest eigenvalues.

6:     Initialize D = I_d×d.

7:     While not converge do

8:     Solve the generalized eigen-decomposition problem (A + γD) W = WΛ.

9:    Update W = [w₁, w₂, ·· · , w_p], where w₁, w₂, ·· · , w_p are the eigenvectors corresponding to the first p smallest eigenvalues.

10: Update the diagonal matrix D, whose i-th diagonal element is 1/(2 ∥ W_i,• ∥ ₂).

11: End while

12: Initialize Q = I_m×m.

13: While not converge do

14: Compute the gradient g = AQWW^T - RR^TQ.

15: Update Q = Q - η * (g/∥ g ∥ ₂).

16: End while

17: End while

18: Output: The transferring projection matrix R and the

        discriminative projection W.

Algorithm 1 Enhanced Discriminative Feature
1: Input: The source data X^s, the target data X^t.
2: Initialize the dimension of transferring projection
matrix m, the dimension of discriminative projection
matrix p, Q = 10 × I_m×m, λ = 1e2, γ = 1e3, μ = 1,
η = 1e - 2.
3: While not converge do
4: Solve the eigen-decomposition problem (XLX^T + λI_d - QQ^T) R = (XHX^T) RΛ.
5: Update R = [r₁, r₂, ·· · , r_m], where r₁, r₂, ·· · , r_m are the eigenvectors corresponding to the first m smallest eigenvalues.
6: Initialize D = I_d×d.
7: While not converge do
8: Solve the generalized eigen-decomposition problem (A + γD) W = WΛ.
9: Update W = [w₁, w₂, ·· · , w_p], where w₁, w₂, ·· · , w_p are the eigenvectors corresponding to the first p smallest eigenvalues.
10: Update the diagonal matrix D, whose i-th diagonal element is 1/(2 ∥ W_i,• ∥ ₂).
11: End while
12: Initialize Q = I_m×m.
13: While not converge do
14: Compute the gradient g = AQWW^T - RR^TQ.
15: Update Q = Q - η * (g/∥ g ∥ ₂).
16: End while
17: End while
18: Output: The transferring projection matrix R and the
discriminative projection W.

3 Experiments

3.1 Datasets

FERET [20] dataset. It consists of 13,539 facial images corresponding to 1,565 subjects, who are diverse across ethnicity, gender, and age. Most of the face images in FERET dataset are of individuals of Caucasian race. In the following experiments, we select five subsets from the FERET dataset, i.e., training set, fa, fb, dup1 and dup2. The training set contains 540 images from 180 individuals, each individual has 3 images; fa includes 994 images of 994 persons, one image per person; fb contains 812 images captured under another facial expression; dup1 includes 736 face images captured from different period; dup2 contains 228 image, it is a subset of dup1 and images were captured over the period of one year. The FERET data subset can be used as either source data or target data in the experiments. If it is used as source data, then only the training set is selected; if it is used as target data, then fa is selected as the gallery set, while fb, dup1 and dup2 are used as three probe sets. Sample images in this database are shown in Fig. 3(a).

Fig. 3

Sample images from (a) FERER database, (b) CAS-PEAL-R1 database, (c) LFW database.

CAS-PEAL-R1 [8] (CAS-PEAL in the following) database. It is the largest Chinese face database for training and evaluating face recognition methods. The face images in this data set were taken with various variations such as expression, accessory, lighting, etc. The training set consists of 1200 images of 300 subjects, 4 images for each individual; the gallery set contains 1040 images from 1040 subjects with a normal condition; for the probe set, five subsets are selected in our experiments: accessory (2285 images), expression (1570 images), lighting (2243 images), background (553 images) and distance (275 images) subset. When the CAS-PEAL dataset is selected as source data, it means the training set is used; whereas when it is selected as target data, it indicates that the gallery set and five probe sets are used for evaluating face recognition methods. Figure 3(b) shows some sample images in this database.

LFW [10] dataset. It contains images of 5,749 individuals taken under an unconstrained setting. Face images in LFW database involves diverse variations such as pose, scale, lighting, hairstyle, expression, partial occlusion. The complex surroundings of image capturing and inaccurate alignment of faces makes the LFW data quite challenging for face recognition in the SSPP setting. LFW-a is a subset of the LFW dataset, and the images in LFW-a have been aligned with a commercial software tool. In this subset, we gather the subjects containing no less than ten samples and then get a dataset with 158 subjects from LFW-a database. LFW subset is used as either source training set or target probe set. Since there are no frontal face images in this dataset, the mean face of each person is used to construct the gallery set. We illustrate some sample images in Fig. 3(c).

3.2 Basic experimental settings

In this work, we focus on the unsupervised domain adaptation, hence the source domain training data is labeled, while the target domain training data is unlabeled. In the evaluation of face recognition methods, the nearest neighbor classifier with cosine distance is adopted for classification, and rank-1 recognition rate is reported for each experiment.

In the experiments, each database is treated as either the source data or the target data. The target data contains gallery set and probe set, the source data is seen as the auxiliary dataset or generic set. For SSPP face recognition, the gallery set together with the auxiliary set is used for training the EDFL model, then the model is tested on the probe set.

In all experiments, the face images are aligned according to the labeled eye locations, and then resized to 64×64 pixels for all datasets. PCA is applied to reduce the dimensionality of input raw features with 99.99%energy maintained. As the first 3 principal components of PCA usually contains more information about variations, we eliminate them in the following experiments to improve the recognition performance. Please note that PCA is learned from the source data and then directly used on target data. The parameters of all involved methods are tuned to report the best results unless otherwise specified. In GDA, two parameters, i.e., κ and k, are used to find nearest neighbors, we set κ = [c₁l/c₂], where [] denotes rounding up to an integer. The value of κ makes the size of datasets in the source and target domain nearly equal. We tune k = 30 in all experiments. The parameters in the proposed algorithm, i.e., μ, λ and γ are empirically set to 1, 0.01 and 1000, respectively.

3.3 Analysis of EDFL

To measure EDFL’s effectiveness for SSPP face recognition task, we have conducted three sets of experiments. The three groups of experiments are: F⟶C, C⟶L and L⟶F. F⟶C denotes FERET and CAS-PEAL dataset are used as the source and target domains respectively. Similarly, C⟶L identifies CAS-PEAL and LFW dataset are used as the source and target domain respectively and in L⟶F group of experiments, LFW and FERET dataset are used as the source and target domain respectively.

3.3.1 The effect of generalized domain adaptation (GDA)

We validate the essentiality of GDA of SSPP face recognition by investigating the following four designs, i.e., EDFL using GDA and EDFL without using GDA (denoted by EDFL_wot). Figure 4 displays the respective recognition rates. As seen in the figure, EDFL outperforms EDFL_wot significantly. For EDFL_wot, using the global information of gallery set in the target domain is unable to capture the accurate distribution of the target domain, which leads to poor adaptation across domains. Therefore, EDFL has produced good domain adaptation results, which boost their performance on SSPP face recognition.

Fig. 4

The recognition rate of EDFL with and without using generalized domain adaptation.

3.3.2 Influence of parameter k in EDFL

In EDFL, the domain adaptation is achieved by global and local domain adaptation. The local domain adaptation is one novelty of this work, and it plays an important role in the overall domain adaptation. The parameter k involved in EDFL is used to determine the size of local domains, which affects the effectiveness of domain adaptation. To test the influence of k on the recognition performance, we have conducted experiments on three databases. Let the value of k varies from 20 to 300 with an increment of 20, we investigate the changes of recognition accuracy of EDFL on three databases. The results are shown in Fig. 5. It can be observed that the performance roughly increases first and then decreases or keeps stable with the increase of k. For all three databases, EDFL is able to achieve promising results when k = 120. This implies that local domains with such size is sufficient to cover the local structure information across domains.

Fig. 5

The performance of EDFL w.r.t. different value of k.

3.3.3 Convergence and complexity analysis of the optimization for EDFL

Since the optimization problem is solved in an alternative fashion, and the sub-optimizations of W and Q also utilize iterative strategy, we have explored the convergence of the algorithms involved in the optimization. While it is hard to prove its convergence theoretically, we have shown some empirical evaluation. Here, we show the objective values corresponding to the optimization of W, Q and the overall EDFL. Meanwhile, we also display the variations of recognition accuracy of EDFL with the iterations of the Algorithm 1. The results are shown in Fig. 6. As seen, all algorithms steadily converge after a number of iterations. Specifically, the sub-optimizations of W and Q converge within 10 iterations, while the overall optimization of EDFL converges in about 5 iterations, and the recognition performance of EDFL keeps stable in less than 10 iterations.

Fig. 6

The convergence curve of sub-optimization of W, Q and over-all optimization in EDFL on three databases.

The main computational cost of Algorithm 1 is spent on solving eigen-decomposition (line 4) and computing the gradient (line 14).We make runtime test of training process using a PC with Intel Core(TM) i7 2.3 GHz CPU, 16G RAM and Matlab 2015a software. The max iteration in the algorithm is set to 30. The CAS-PEAL database is treated as the target domain, in which the gallery set contains 1040 samples, when the FERET (540 training samples) and LFW database (1580 training samples) are employed as the source data, respectively, the training time of EDFL is 65.11s and 109.83s, respectively; the highest training time of other compared generic learning methods is 52.25s (DTL method) and 90.26s (DTL method). Thus, the time complexity of our proposed EDFL is relatively high and does not own the advantage over other methods.

According to the analysis about time complexity, we clarify that when EDFL applied in a wide range, it is recommended EDFL is used for closed set face recognition, where the training process can be done offline.

3.4 Comparison with the existing works

To further validate our method, we conduct more experiments and compare it with several existing approaches which are designed for SSPP face recognition problem. All of these compared approaches are generic learning based, i.e., they adopt an auxiliary dataset containing multiple samples per person to address SSPP problem. There are Fisher’s Linear Discriminant (FLD) [1], Adaptive Generic Learning (AGL) [21], Adaptive Discriminative Learning (ADL) [12], Extended Sparse Representation Classification (ESRC) [5], Sparse Variation Dictionary Learning (SVDL) [27], Discriminative Transfer Learning (DTL) [9], Low-rank Regularized Generic Representation with Block-Sparse Structure (LRGR-BSS) [15] and k-Nearest Neighbor based Multi-manifold Discriminant Learning (kNNMDL) [6].

Additionally, we compare the above methods with two state-of-the-art deep learning techniques, i.e., RegularFace [33] and ArcFace [4].These two techniques are not specifically designed but can be used for SSPP face recognition. We do not modify the network structures of of RegularFace and ArcFace, and pre-train them with CASIA-Webface [30] dataset. We use the two deep learning methods in the following two ways. (a) We input the pre-trained RegularFace and ArcFace with gallery set and probe set to extract deep facial features, then conduct recognition with 1-NN classifier. (b) We input the pre-trained two deep networks with generic set, gallery set and probe set to extract deep features respectively, then feed them to DTL and EDFL directly for SSPP face recognition, which derives four methods, denoted by RF-DTL, AF-DTL, RF-EDFL, AF-EDFL, respectively.

3.4.1 Performance comparison on FERET database

We first treat FERET database as the target domain, and use CAS-PEAL training set and LFW subset as source data to simulate enhanced discriminative feature learning across ethnicity and image condition, respectively. Note that most of face images in FERET database are that of Caucasian race, though it includes diverse ethnicities. We apply FERET database to roughly represent an ethnicity which is different from Chinese people. Tables 2 list the recognition rates of various methods on FERET database when using generic dataset across ethnicity and imaging condition, respectively.

Table 1
Performance comparison on FERET database when CAS-PEAL database is used as source data

Fb Dup1 Dup2 Ave.

FLD 66.63 37.41 32.02 45.35

AGL 71.28 48.24 47.62 55.71

ADL 73.28 52.37 58.77 61.47

DTL 70.69 45.14 49.12 54.98

ESRC 67.37 50.12 56.56 58.02

SVDL 69.21 54.69 62.72 62.21

kNNMDL 70.36 48.56 57.36 58.76

LRGR-BSS 72.17 52.37 59.06 61.20

EDFL 76.97 55.57 62.28 64.94

RegularFace 78.45 58.97 66.67 68.03

ArcFace 80.05 61.14 73.68 71.62

RF-DTL 78.94 59.24 67.11 68.43

AF-DTL 79.80 61.14 74.56 71.84

RF-EDFL 80.54 60.73 72.37 71.21

AF-EDFL 81.28 61.55 75.88 72.90

	Fb	Dup1	Dup2	Ave.
FLD	66.63	37.41	32.02	45.35
AGL	71.28	48.24	47.62	55.71
ADL	73.28	52.37	58.77	61.47
DTL	70.69	45.14	49.12	54.98
ESRC	67.37	50.12	56.56	58.02
SVDL	69.21	54.69	62.72	62.21
kNNMDL	70.36	48.56	57.36	58.76
LRGR-BSS	72.17	52.37	59.06	61.20
EDFL	76.97	55.57	62.28	64.94
RegularFace	78.45	58.97	66.67	68.03
ArcFace	80.05	61.14	73.68	71.62
RF-DTL	78.94	59.24	67.11	68.43
AF-DTL	79.80	61.14	74.56	71.84
RF-EDFL	80.54	60.73	72.37	71.21
AF-EDFL	81.28	61.55	75.88	72.90

Table 2

Performance comparison on FERET database when LFW database is used as source data

	Fb	Dup1	Dup2	Ave.
FLD	60.44	44.60	49.12	51.39
AGL	64.74	46.28	52.36	54.46
ADL	65.52	52.52	57.28	58.44
DTL	65.89	52.27	55.18	57.78
ESRC	64.72	52.41	60.96	59.36
SVDL	70.29	56.81	62.28	63.13
kNNMDL	62.38	45.42	59.76	55.85
LRGR-BSS	71.18	54.62	58.46	61.42
EDFL	77.34	55.84	62.72	65.30
RegularFace	78.45	58.97	66.67	68.03
ArcFace	80.05	61.14	73.68	71.62
RF-DTL	78.94	59.51	66.67	68.37
AF-DTL	80.05	61.68	75.88	72.54
RF-EDFL	80.79	61.01	73.68	71.83
AF-EDFL	82.02	61.82	76.75	73.53

As observed from the results in the two tables, FLD performs worst since it directly applies the source domain model to the target domain without any adaptation. Among the non-deep features-based methods, those specially designed for SSPP with using the generic data perform much better than FLD. Though SVDL obtains the best results in some situations, EDFL achieves the highest overall recognition accuracy. Besides, by observing results in Tables 1 2, we find that AGL and ADL obtain lower recognition rates when LFW is used as the source data, though there are 10 samples per person in LFW database. This reveals that the complex imaging condition negatively impacts AGL and ADL and hinders them to take the advantages of within-class variation explored from 10 samples. By comparison, other methods can utilize the advantage of big sample size to boost the performance. No matter whether the domain adaptation is performed across ethnicity or imaging condition, our proposed EDFL can achieve competitive results on FERET database.

Compared with non-deep features, the deep-features perform obviously better. RegularFace and ArcFace are not especially developed for SSPP face recognition, but the pre-trained models produce higher recognition rates than those specifically modeled non-deep featured based methods, which implies the good feature representation of deep learning networks. With the deep features extracted from RegularFace and ArcFace, both DTL and EDFL can boost the recognition performance, and EDFL benefits more from deep features than DTL. Among all the compared methods, AF-EDFL, i.e., EDFL with ArcFace deep features, achieves the highest recognition rates.

3.4.2 Performance comparison on CAS-PEAL database

Similar to the evaluation on FERET dataset, we test each method under two scenarios, i.e., domain adaptation across ethnicity and across image condition, respectively. The target domain is CAS-PEAL dataset, which consists of face images taken from Chinese people. The evaluations on CAS-PEAL database are shown in Table 3 and Table 4, from which similar conclusions as before can be drawn. FLD obtains the lowest recognition accuracy, other methods perform better than it, and the proposed EDFL performs the best among non-deep methods under two scenarios. For the deep-features based methods, most outperform non-deep methods, but when CAS-PEAL database is used as the generic set, EDFL obtains competitive recognition rates to RegularFace and ArcFace. AF-EDFL is still the best one.

Table 3
Performance comparison on CAS-PEAL database when FERET database is used as source data

Accessory Expression Background Distance Ave.

FLD 38.60 54.33 81.19 81.45 63.89

AGL 38.24 56.11 83.04 85.15 65.64

ADL 39.67 59.87 82.10 85.09 66.68

DTL 43.94 66.31 84.63 86.91 70.45

ESRC 42.10 64.78 87.07 87.32 64.78

SVDL 46.33 65.10 88.43 86.55 71.60

kNNMDL 45.33 62.28 86.46 85.31 69.85

LRGR-BSS 47.39 65.10 89.73 87.44 72.42

EDFL 49.02 67.20 91.14 90.91 74.57

RegularFace 51.33 68.15 91.50 92.00 75.75

ArcFace 52.30 70.25 93.49 96.00 78.01

RF-DTL 51.42 68.03 91.68 92.36 75.87

AF-DTL 52.30 70.38 94.03 96.36 78.27

RF-EDFL 51.55 68.28 91.86 93.09 76.20

AF-EDFL 52.47 71.02 94.76 98.55 79.20

	Accessory	Expression	Background	Distance	Ave.
FLD	38.60	54.33	81.19	81.45	63.89
AGL	38.24	56.11	83.04	85.15	65.64
ADL	39.67	59.87	82.10	85.09	66.68
DTL	43.94	66.31	84.63	86.91	70.45
ESRC	42.10	64.78	87.07	87.32	64.78
SVDL	46.33	65.10	88.43	86.55	71.60
kNNMDL	45.33	62.28	86.46	85.31	69.85
LRGR-BSS	47.39	65.10	89.73	87.44	72.42
EDFL	49.02	67.20	91.14	90.91	74.57
RegularFace	51.33	68.15	91.50	92.00	75.75
ArcFace	52.30	70.25	93.49	96.00	78.01
RF-DTL	51.42	68.03	91.68	92.36	75.87
AF-DTL	52.30	70.38	94.03	96.36	78.27
RF-EDFL	51.55	68.28	91.86	93.09	76.20
AF-EDFL	52.47	71.02	94.76	98.55	79.20

Table 4

Performance comparison on CAS-PEAL database when LFW database is used as source data

	Accessory	Expression	Background	Distance	Ave.
FLD	42.48	53.38	82.04	83.09	65.25
AGL	46.08	55.42	85.47	82.56	67.38
ADL	48.97	60.06	87.88	82.18	69.77
DTL	46.44	57.78	84.63	84.73	68.40
ESRC	44.27	59.44	84.07	83.46	67.81
SVDL	45.34	61.27	84.81	86.36	69.45
kNNMDL	43.26	56.14	83.36	83.09	66.46
LRGR-BSS	46.76	62.28	85.17	87.56	70.44
EDFL	52.08	69.11	92.22	93.45	76.72
RegularFace	51.33	68.15	91.50	92.00	75.75
ArcFace	52.30	70.25	93.49	96.00	78.01
RF-DTL	55.05	72.55	95.48	96.36	79.86
AF-DTL	56.15	73.57	96.93	97.45	81.03
RF-EDFL	56.06	73.57	96.75	97.45	80.96
AF-EDFL	56.24	73.69	97.47	97.82	81.31

Comparing Tables 4, it can be observed that different generic set has a noticeable impact on the recognition performance, especially when conducting using LFW as the generic set, EDFL shows significant superiority over other non-deep features based methods. Nearly all approaches obtain much higher accuracy after switching the generic data from FERET to LFW, the reason might be that the sample size of generic dataset (FERET) is too small (only 3 image per person) to explore more variations.

3.4.3 Performance comparison on LFW database

In real-world applications, the labeled face images are usually collected in laboratory under controlled conditions, i.e., laboratory images, but the unknown faces are captured in unstrained environment, i.e., real environment images. Thus, applying the laboratory images to boost the recognition performance on real environment images is quite important for a face recognition algorithm. To assess the performance of different methods in this setting, we conducted two groups of experiments: LFW dataset is used as target, CAS-PEAL and FERET database are used as source domains respectively. Table 5 and Table 6 illustrate the corresponding performance comparison, respectively. Among all non-deep techniques, EDFL performs the best and takes superiority over the second best one by a large margin of about 10 and 7 percentages respectively in the two experiments. Moreover, applying CAS-PEAL database to be the generic set achieves higher recognition rates for all methods.

Table 5
Performance comparison on LFW database when CAS-PEAL database are used as source data

FLD AGL ADL DTL ESRC SVDL kNNMDL LRGR-BSS EDFL RegularFace ArcFace RF-DTL AF-DTL RF-EDFL AF-EDFL

CASPEAL→LFW 59.68 72.46 76.08 70.38 73.15 71.46 59.76 70.56 83.02 70.62 71.11 85.66 87.18 85.89 87.25

	FLD	AGL	ADL	DTL	ESRC	SVDL	kNNMDL	LRGR-BSS	EDFL	RegularFace	ArcFace	RF-DTL	AF-DTL	RF-EDFL	AF-EDFL
CASPEAL→LFW	59.68	72.46	76.08	70.38	73.15	71.46	59.76	70.56	83.02	70.62	71.11	85.66	87.18	85.89	87.25

Table 6

Performance comparison on LFW database when FERET database are used as source data

	FLD	AGL	ADL	DTL	ESRC	SVDL	kNNMDL	LRGR-BSS	EDFL	RegularFace	ArcFace	RF-DTL	AF-DTL	RF-EDFL	AF-EDFL
FERET→LFW	48.16	54.35	57.85	53.01	61.05	60.95	54.35	61.38	68.91	70.62	71.11	70.90	71.22	70.99	71.29

However, when CAS-PEAL database is used as the generic, the specifically designed methods for SSPP face recognition obtain higher recognition rates than two deep learning based frameworks, i.e., RegularFace and ArcFace. It indicates again that different generic sets make a difference on recognition performance, and an appropriate generic set may be more important than the models.

3.5 Discussion

As clearly seen from the above evaluation, our proposed EDFL outperforms other approaches designed for addressing SSPP face recognition. According to Table 1 –6, no matter whether domain adaptation is conducted across ethnicity or imaging condition, EDFL is able to achieve promising recognition accuracy. EDFL can boost the recognition accuracy by using features extracted from deep learning frameworks and obtains higher recognition rates than deep learning methods themselves. Moreover, DTL is the most similar method to the proposed EDFL, both attempt to achieve the following two goals: transferring knowledge from source to target domain and learning discriminative subspace. However, EDFL obtains better recognition results than DTL in every case. This is due to the fact that DTL learns only one mapping to simultaneously accomplish knowledge transferring and discriminative learning but learning single mapping may be difficult to realize the two goals well. The proposed EDFL learns one mapping for domain adaptation and the other one for discriminative learning, and it unifies the learning of two mappings into one framework, thus EDFL outperforms DLT on both goals.

4 Conclusions

To cope with face recognition with single sample per person, in this paper, we have proposed a novel enhanced discriminative feature learning (EDFL) method from the perspective of domain adaptation. EDFL employs an auxiliary face dataset in which face images are labeled and involve diverse variations. To make better use of the knowledge in the auxiliary set, we first propose generalized domain adaptation to minimize the distribution mismatch between auxiliary set and gallery set, and then integrate the domain adaptation and Fisher discriminative learning into one scheme. Further, we have developed an efficient iterative optimization algorithm to obtain the solution of EDFL. To estimate EDFL approach, extensive experiments have been conducted on three databases. By comparing with several specially designed methods for SSPP, and deep learning-based methods, the promising recognition performance of EDFL demonstrates its effectiveness for single sample per person face recognition.

In the EDFL framework, we consider only one source domain, i.e., one auxiliary dataset. Though the generalized domain adaptation can be directly used for multiple source domains, it does not guarantee to output better performance as different source domains may also require adaptation. As a good generic set is good for improving recognition results, but to a specific probe set, what kinds of generic sets are best is also an interesting problem. In future, we will work on these issues.

Footnotes

Acknowledgments

This work was partially supported by the Humanities and Social Science Research on Youth Fund Project of Ministry of Education of China (Grant No. 19YJC870003), the Natural Science Foundation of Jiangsu Province (Grant No. BK20210576), and sponsored by NUPTSF (Grant No. NY219038, NY220045).

References

Belhumeur

P.N.

, Hespanha

J.P.

and Kriegman

D.J.

Eigenfaces vs. fisherfaces: Recognition using class specific linear projection, IEEE Transactions on Pattern Analysis and Machine Intelligence 19 (1997), 711–720.

Chu

, Zhao

and Ahmad

Multiple feature subspaces analysis for single sample per person face recognition, , The Visual Computer 35 (2019), 239–256.

Cuculo

, D’Amelio

, Grossi

, Lanzarotti

and Lin

, Robust single-sample face recognition by sparsity-driven sub-dictionary learning using deep features, Sensors 19 (2019), 146.

Deng

, Guo

, Xue

and Zafeiriou

ArcFace: Additive Angular Margin Loss for Deep Face Recognition, in: Computer Vision and Pattern Recognition, 2019, pp. 4690–4699.

Deng

, Hu

and Guo

, Extended SRC: Undersampled Face Recognition via Intraclass Variant Dictionary, , IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (2012), 1864–1870.

Dong

, Wu

and Jing

, Generic Training Set based Multimanifold Discriminant Learning for Single Sample Face Recognition, , KSII Transactions on Internet and Information Systems 12 (2018), 368–391.

Gao

, Jia

, Zhuang

and Ma

, Neither global nor local: Regularized patch-based representation for single sample per person face recognition, International Journal of Computer Vision 111 (2015), 365–383.

Gao

, Cao

, Shan

, Chen

, Zhou

, Zhang

and Zhao

, The CAS-PEAL Large-Scale Chinese Face Database and Baseline Evaluations,Part A: Systems and Humans, IEEE Transactions on Systems, Man, and Cybernetics – 38 (2008), 149–161.

, Discriminative transfer learning with sparsity regularization for single-sample face recognition, Image & Vision Computing 60 (2017), 48–57.

10.

Huang

G.B.

, Mattar

, Berg

and Learned-Miller

, Labeled faces in the wild: A database forstudying face recognition in unconstrained environments, Month (2007).

11.

H.K.

, Sun

Q.S.

, Ji

Z.X.

, Yuan

Y.H.

and Zhang

G.Q.

, Collaborative probabilistic labels for face recognition from single sample per person, , Pattern Recognition 62 (2017), 125–134.

12.

Kan

, Shan

, Su

, Xu

and Chen

, Adaptive discriminant learning for face recognition, , Pattern Recognition 46 (2013), 2497–2509.

13.

Kan

, Wu

, Shan

and Chen

, Domain Adaptation for Face Recognition: Targetize Source Domain Bridged by Common Subspace, , International Journal of Computer Vision 109 (2014), 94–109.

14.

Z.M.

, Huang

Z.H.

and Shang

, A Customized Sparse Representation Model With Mixed Norm for Undersampled Face Recognition, IEEE Transactions on Information Forensics & Security 11 (2016), 2203–2214.

15.

Liu

, Ding

, Xu

and Ye

, Learning Low-Rank Regularized Generic Representation With Block-Sparse Structure for Single Sample Face Recognition, , IEEE Access 7 (2019), 30573–30587.

16.

, Tan

Y.-P.

and Wang

, Discriminative multimanifold analysis for face recognition from a single training sample per person, , IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (2013), 39–51.

17.

Mokhayeri

and Granger

, A paired sparse representation model for robust face recognition from a single sample, Pattern Recognition 100 (2020), 107129.

18.

Niu

, Cai

and Xu

, Domain Adaptive FisherVector for Visual Recognition, in: European Conference on Computer Vision, Springer International Publishing, 2016.

19.

Pan

S.J.

, Tsang

I.W.

, Kwok

J.T.

and Yang

, Domain Adaptation via Transfer Component Analysis, IEEE Transactions on Neural Networks 22 (2011), 199–210.

20.

Phillips

P.J.

, Moon

, Rizvi

S.A.

and Rauss

P.J.

, The FERET evaluation methodology for face-recognition algorithms, , IEEE Transactions on Pattern Analysis and Machine Intelligence 22 (2000), 1090–1104.

21.

, Shan

, Chen

and Gao

, Adaptive generic learning for face recognition from a single sample per person, in: IEEE Conference on Computer Vision and Pattern Recognition, 2010.

22.

Wei

C.-P.

and Wang

Y.-C.F.

, Undersampled face recognition via robust auxiliary dictionary learning, , IEEE Transactions on Image Processing 24 (2015), 1722–1734.

23.

Wei

and Jin

, Locality sensitive discriminant projection for feature extraction and face recognition, , Journal of Electronic Imaging 28 (2019), 043028.

24.

C.Y.

and Neumann

, Efficient Multi-Domain Dictionary Learning With GANS, in: 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP), IEEE, 2019, pp. 1–5.

25.

, Fang

, Li

, Yang

, You

, Liu

and Teng

, Data uncertainty in face recognition, , IEEE Transactions on Cybernetics 44 (2014), 1950–1961.

26.

, Li

, Zhang

, Yang

and You

, Sample diversity, representation effectiveness and robust dictionary learning for face recognition, , Information Sciences 375 (2017), 171–182.

27.

Yang

, Van Gool

and Zhang

, Sparse variation dictionary learning for face recognition with a single training sample per person, in: Proceedings of the IEEE International Conference on Computer Vision, 2013, pp. 689–696.

28.

Yang

, Wang

, Zeng

and Shen

, Joint and Collaborative Representation with Local Adaptive Convolution Feature for Face Recognition with Single Sample Per Person, , Pattern Recognition 66 (2016), 117–128.

29.

Yang

, Wen

, Wang

, Shen

and Gao

, Adaptive convolution local and global learning for class-level joint representation of facial recognition with a single sample per data subject, IEEE Transactions on Information Forensics and Security 15 (2020), 2469–2484.

30.

, Lei

, Liao

and Li

S.Z.

, Learning Face Representation from Scratch, arXiv: Computer Vision and Pattern Recognition, (2014).

31.

Y.F.

, Dai

D.Q.

, Ren

C.X.

and Huang

K.K.

, Discriminative multi-scale sparse coding for single-sample face recognition with occlusion ✩, Pattern Recognition 66 (2017), 302–312.

32.

Zhang

, Hu

and Lu

, IL-GAN: Illumination-invariant representation learning for single sample face recognition, , Journal of Visual Communication and Image Representation 59 (2019), 501–513.

33.

Zhao

, Xu

and Cheng

, RegularFace: Deep Face Recognition via Exclusive Regularization, in: Computer Vision and Pattern Recognition, 2019, pp. 1136–1144.

Integrating generalized domain adaptation and Fisher discriminative learning: A unified framework for face recognition with single sample per person

Abstract

Keywords

1 Introduction

1.1 Related work

1.2 Motivation and contribution

1.2.1 Motivation

1.2.2 Contribution

2 Enhanced discriminative feature learning

2.1 Notations

2.2 Generalized domain adaptation

3.1 Datasets

3.3 Analysis of EDFL

3.3.1 The effect of generalized domain adaptation (GDA)

3.4.1 Performance comparison on FERET database

Table 5 Performance comparison on LFW database when CAS-PEAL database are used as source data FLD AGL ADL DTL ESRC SVDL kNNMDL LRGR-BSS EDFL RegularFace ArcFace RF-DTL AF-DTL RF-EDFL AF-EDFL CASPEAL→LFW 59.68 72.46 76.08 70.38 73.15 71.46 59.76 70.56 83.02 70.62 71.11 85.66 87.18 85.89 87.25

4 Conclusions

Footnotes

Acknowledgments

References

Table 5
Performance comparison on LFW database when CAS-PEAL database are used as source data

FLD AGL ADL DTL ESRC SVDL kNNMDL LRGR-BSS EDFL RegularFace ArcFace RF-DTL AF-DTL RF-EDFL AF-EDFL

CASPEAL→LFW 59.68 72.46 76.08 70.38 73.15 71.46 59.76 70.56 83.02 70.62 71.11 85.66 87.18 85.89 87.25