Adapted discriminative coupled mappings for low-resolution face recognition with one-shot

Abstract

Low-resolution face recognition with one-shot is a prevalent problem encountered in law enforcement, where it generally requires to recognize the low-resolution face images captured by surveillance cameras with the only one high-resolution profile face image in the database. The problem is very tough because the available samples is quite few and the quality of unknown images is quite low. To effectively address this issue, this paper proposes Adapted Discriminative Coupled Mappings (AdaDCM) approach, which integrates domain adaptation and discriminative learning. To achieve good domain adaptation performance for small size dataset, a new domain adaptation technique called Bidirectional Locality Matching-based Domain Adaptation (BLM-DA) is first developed. Then the proposed AdaDCM is formulated by unifying BLM-DA and discriminative coupled mappings into a single framework. AdaDCM is extensively evaluated on FERET, LFW, and SCface databases, which includes LR face images obtained in constrained, unconstrained, and real-world environment. The promising results on these datasets demonstrate the effectiveness of AdaDCM in LR face recognition with one-shot.

Keywords

Domain Adaptation discriminative learning low-resolution face recognition one-shot

1 Introduction

For the purpose of security in public places, nowadays increasing number of surveillance cameras are installed everywhere. For example, it is estimated by IHS Markit that more than 176 million public and private surveillance cameras have been installed in China, and it was predicted that the number will increase to 450 million by the end of 2020. With the prevalence of cameras, incomputable face images are captured. As surveillance cameras are generally installed at a specific height above the pedestrians, the face images taken by these cameras are generally small sizes or low-resolution (LR). Recognizing such LR faces effectively has become an important research dimension in computer vision communities. LR face recognition is typically to identify LR face images based on available high-resolution (HR) face images. In addition, in some practical applications, such as law enforcement, there is only one HR profile face image for every person in the database of security departments. When the public security organizations attempt to determine the true identity of a suspect or terrorist in public according the face images captured by surveillance cameras, they are facing LR face recognition problem with one-shot. This problem is quite challenging because it is difficult to directly find a match between LR and HR images and to explore face variations with one-shot.

To deal with LR face recognition, super-resolution (SR) based and coupled mappings (CM) based methods are proposed by researchers. The basic idea of the SR-based methods is using the HR images recovered from the corresponding LR images to recognize an unknown person. Based on example pairs of input LR and output HR images, a map from input images to target HR images can be learned by using kernel ridge regression [20]. With an over-complete dictionary [35], proposed to use sparse representation to generate the HR versions of LR images. Recently, deep networks are also employed to learn the relationship between LR and HR images [8, 9 32]. But these SR approaches are mostly devised for enhancing the vision performance, not for improving recognition or classification performance. To well tackle LR face recognition with SR technique [17], proposed to perform SR on coherent feature domain between HR and LR features. Differently [3], focused on the identity-aware face features and utilized the magnitude and angle of features to recover the identity information of LR images by designing a super-resolution network. However [22], compared the performance of various super-resolution methods on LR face recognition, including those based on deep learning, the results showed that the deep learning super-resolution methods cannot output expected recognition accuracy. When the resolution of images is quite low, using the limited number of pixels to recover sufficient meaningful information by SR is very difficult.

Coupled mappings (CM)-based methods learn one mapping for HR and LR images, respectively. With the learned coupled mappings, HR and LR images are projected into a common subspace for matching. Coupled locality preserving mappings (CLPM) [21], coupled marginal fisher analysis (CMFA) [30], and simultaneous discriminant analysis (SDA) [37] are the typical CM-based approaches. Based on CM, [14] proposed discriminant correlation analysis (DCA) recently, which finds the projections that maximizes the pair-wise correlations between the HR and LR feature sets and separate the classes within each set at the same time. To improve the discriminative ability of the common subspace, a semi-coupled locality-constrained representation approach is developed to simultaneously learn the discriminative representations and the mapping relationship between LR and HR features [23]. Besides, the kernel tricks are also integrated with CM based models to address LR face recognition, such as coupled kernel fisher discriminative analysis [31], coupled kernel embedding [27], and kernel coupled distance metric learning [28]. Multidimensional scaling (MDS), as a technique for nonlinear dimension reduction, is also employed for LR face recognition [1 , 34]. MDS based methods learn two mappings via MDS, and aim to make the distances of LR and HR images in the projected subspace close to the distance of two corresponding HR images. Different from MDS, to deal with multi-resolution face images, [24] proposed a multi-resolution dictionary learning (MRDL) method providing multiple dictionaries associated with diverse resolutions.

Recently, deep network has been introduced into CM based models to address LR face recognition [36]. Proposed to use the deep network to design the coupled mappings, so that the nonlinear transformations are capture when projecting the HR and LR images into the common subspace. But the proposed method is developed for LR face recognition with multiple training samples. Most recently, based on successive subspace learning [29], proposed an explainable non-parametric model for LR face recognition even there is a small number of labeled data samples. Furthermore, both [18] and [12] proposed to use the external dataset to boost the LR face recognition performance, while the former one focuses on generating high-quality images by a dual domain adaptive structure, the latter one aims at simplifying a pre-trained complex model into a light one for LR face recognition via a two-step distillation.

The above methods can cope with LR face recognition; however, they depend on multiple training samples for each subject. Based on few-shot learning [11], proposed a few-shot knowledge distillation approach for LR face recognition with few-shot. To deal with LR face recognition with one-shot [4], proposed cluster-based regularized simultaneous discriminant analysis (C-RSDA), which enriches the intra-class and inter-class variation by using the intra-cluster and inter-cluster variation, respectively. Besides [6], and [5] adopt domain adaptation to improve the performance of LR face recognition, however, as the gallery set contains single sample per subject, the domain adaptation result is not promising. Though locality preserving domain adaptation (LPDA) proposed in [6] also involves the local structure information across domains, it considers the locality information according to the samples in source domain only. When the data size of source domain is much larger than that of target domain, LPDA will overexploit the local information of target domain. Thus, these methods do not make full use of the auxiliary database to improve the performance of LR face recognition, which is our focus in this presented work.

To explore more variation and design robust LR face recognition algorithm with one-shot, this paper introduces an auxiliary database which contains multiple samples per person, and employs the discriminative subspace learned by the auxiliary set to improve LR face recognition with one-shot. Nevertheless, due to the auxiliary set and the probe set may share quite different data distributions, if the learned discriminative subspace is directly used for classification task in the probe set, there will be dramatic performance drop, just as [19] have shown. To solve the issue, this paper integrates discriminative coupled mappings and domain adaptation together, and designs the Adapted Discriminative Coupled Mappings (AdaDCM). Figure 1 illustrates the outline of the proposed approach. AdaDCM simultaneously learns a mapping for HR and LR images, respectively. By learning these two mappings, AdaDCM tries to achieve two objectives i.e., domain adaptation and discriminative learning.

Fig. 1

The illustration of the proposed Adapted SDA.

Specifically, the contribution of this work is twofold. First, for domain adaptation, we design a new domain adaptation technique for small size of samples, named as bidirectional locality matching-based domain adaptation (BLM-DA). BLM-DA finds the locality matching for every sample in both the source and target domain, and then push the overall domain adaptation according to all the local domain adaptation among the matching localities, which is able to guarantee domain adaptation performance for small size of samples, even if the data sizes of two domains are significantly different. Second, for discriminative learning, based on discriminative coupled mappings we combine the knowledge exploited from the source and the target domain together, and unify BLM-DA and discriminative coupled mappings into one scheme to efficiently and simultaneously achieve domain adaptation and discriminative learning. This framework adapts for low-resolution face recognition with one-shot, and can be generalized by other coupled mapping approaches.

The rest of the paper is organized as follows. Section 2 describes the proposed AdaDCM model, Section 3 provides the experimental evaluation, and conclusions are finally drawn in Section 4.

2 Proposed approach

Let H = [h₁, h₂, ⋯ , h_N] be a set of HR face images, in which $h_{i} \in R^{M}$ , N is the number of samples in H, M is the feature dimension of HR images. Similarly, let L = [l₁, l₂, ⋯ , l_N] be a set of LR face images, in which $l_{i} \in R^{m}$ , m is the feature dimension of LR images, and m << M. To reduce the difference of feature dimension between HR images and LR images, based on coupled mappings framework, the proposed AdaDCM aims to learn two mappings, i.e., $P_{H} \in R^{M \times d}$ for HR images, and $P_{L} \in R^{m \times d}$ for LR images. With the learned two mappings, HR and LR face images are projected into a common subspace, in which the similarity between HR gallery projections and probe LR projections can be measured. The projections of HR and LR samples in the common subspace, i.e., $H_{j}^{△}$ and $L_{j}^{△}$ , can be expressed by $H_{j}^{△} = P_{H}^{T} h_{j}$ and $L_{j}^{△} = P_{L}^{T} l_{j}$ , where j = 1, 2, ⋯ , N. Assume that samples in H and L are from K classes. Let H_i denote a set that contains HR samples from the i - th class, c (h_j) denote the class label of h_j, and c (h_j) ∈ (1, 2, ⋯ , K).

2.1 Bidirectional Locality Matching-based Domain Adaptation

The auxiliary set and the probe set are from different domains, we name them as source domain and target domain respectively, and may share quite different data distributions. To make the full use of the auxiliary set, reducing the discrepancy between domains is required.

Let $X^{s} \in R^{d \times n_{s}}$ denote a dataset in the source domain, d is the dimension of samples and n_s is the total number of training samples in the dataset. Similarly, let $X^{t} \in R^{d \times n_{t}}$ denote the dataset in the target domain, n_t is the total number of training samples in the dataset. The symbols s and t are used in the superscript or subscript to denote the source domain and target domain, respectively. To minimize the difference of data distributions between X^s and X^t, [25] proposed transfer component analysis (TCA) model which learns a transfer subspace by the following model, in which domain adaptation between two domains is achieved:

$\begin{matrix} W^{*} = arg min_{W} {∥ \frac{1}{n_{s}} W^{T} X^{s} 1^{s} - \frac{1}{n_{t}} W^{T} X^{t} 1^{t} ∥}_{F}^{2} \\ + λ {∥ W ∥}_{F}^{2} \\ \begin{matrix} \end{matrix} \begin{matrix} \end{matrix} = arg min_{W} tr (W^{T} [X^{s} X^{t}] K {[X^{s} X^{t}]}^{T} W) \\ + λ tr (W^{T} W) \\ \begin{matrix} \end{matrix} \begin{matrix} \end{matrix} = arg min_{W} tr (W^{T} {XKX}^{T} W) \\ + λ tr (W^{T} W) \end{matrix}$ (1)

$s . t . \begin{matrix} \end{matrix} W^{T} {XRX}^{T} W = I_{m}$ (2) where ∥ * ∥ _F is the Frobenius norm of a matrix, $X = [X^{s}, X^{t}] \in R^{d \times (n_{s} + n_{t})}$ , and $W \in R^{d \times m} (m < d)$ is the transform matrix of domain adaptation; $1^{s} \in R^{n_{s} \times 1}$ and $1^{t} \in R^{n_{t} \times 1}$ are the column vectors in which all elements are 1; K is an indicator matrix, K_ij = 1/(n_s) ² if i ⩽ n_s and j ⩽ n_s, else K_ij = 1/(n_t) ² if i > n_t and j > n_t; otherwise, K_ij = -1/(n_sn_t); $R \in R^{n \times n}$ , $R = I_{n} - \frac{1}{n} 11^{T}$ is a centering matrix, n = n_s + n_t; I is the identity matrix, λ is a scalar and $λ {∥ W ∥}_{F}^{2}$ is a regularization term that controls complexity of W. Note that the purpose of Equation (1) is to reduce the cross-domain difference in the transfer subspace, and the purpose of Equation (2) is to maintain the data variance as much as possible.

In the above model, when TCA reduces the distribution mismatch across domains, it in fact reduces the distance between the mean vectors of the two domains. Consequently, TCA is quite effective when the available datasets in both domains contain sufficient samples, where the mean vector of each dataset is very close to the real mean vector of the domain. That is to say, TCA is designed and effective for large-scale datasets. However, for face recognition with one-shot, there is only single face image per person in the training set of target domain, and the mean vector of the training set cannot well represent the true mean vectors of the target domain. In this scenario, TCA cannot ensure a good domain adaptation result. To address this problem, we consider the local information of samples across the domains, and develop bidirectional locality matching-based domain adaptation (BLM-DA), which is formulated by

$\begin{matrix} W^{*} = arg min_{W} \sum_{i = 1}^{n_{s}} {∥ \frac{1}{k} W^{T} X_{i}^{s} 1_{o} - \frac{1}{k} W^{T} X_{i}^{t} 1_{0} ∥}_{F}^{2} \\ + \sum_{j = 1}^{n_{t}} {∥ \frac{1}{k} W^{T} Y_{j}^{t} 1_{o} - \frac{1}{k} W^{T} Y_{j}^{s} 1_{0} ∥}_{F}^{2} + λ {∥ W ∥}_{F}^{2} \\ \begin{matrix} \end{matrix} \begin{matrix} \end{matrix} = arg min_{W} tr (W^{T} (\sum_{i = 1}^{n_{s}} [X_{i}^{s} X_{i}^{t}] K_{o} {[X_{i}^{s} X_{i}^{t}]}^{T}) W \\ + W^{T} (\sum_{j = 1}^{n_{t}} [Y_{j}^{t} Y_{j}^{s}] K_{o} {[Y_{j}^{t} Y_{j}^{s}]}^{T}) W) \\ + λ tr (W^{T} W), \\ \begin{matrix} \end{matrix} \begin{matrix} \end{matrix} = arg min_{W} tr (W^{T} (\sum_{i = 1}^{n_{s}} X_{i} K_{o} X_{i}^{T} \\ + \sum_{j = 1}^{n_{t}} Y_{i} K_{o} Y_{i}^{T}) W) + λ tr (W^{T} W) \end{matrix}$ (3)

$\begin{matrix} s . t . \begin{matrix} \end{matrix} W^{T} (\sum_{i = 1}^{n_{s}} X_{i} R_{o} (X_{i})^{T} + \sum_{j = 1}^{n_{t}} Y_{i} R_{o} (Y_{i})^{T}) \\ W = I_{m}, \end{matrix}$ (4)

for the i - th sample in the source domain, $X_{i}^{s}$ and $X_{i}^{t}$ contains its k nearest neighbors in the source domain and target domain, respectively; for the j - th sample in the target domain, $Y_{j}^{t}$ and $Y_{j}^{s}$ contains its k nearest neighbors in the target domain and source domain, respectively; $X_{i}^{s} \subset X^{s}$ and $X_{i}^{t} \subset X^{t}$ ; $X_{i} = [X_{i}^{s}, X_{i}^{t}] \in R^{d \times 2 k}$ , $1_{o} \in R^{k \times 1}$ , $R_{o} = I_{2 k} - \frac{1}{2 k} 11^{T}$ ; K_o is an indicator matrix, K_o(ij) = 1/k² if i ⩽ k and j ⩽ k, or i > k and j > k, otherwise, K_o(ij) = -1/k². BLM-DA first finds n_s matching local domains between the source and target domains, then reduces the domain distribution mismatch by minimizing the distance across all of the local means. Meanwhile, the covariance of each local domain is maximum to keep the original characteristic of each domain. By this way, BLM-DA alleviates the bias produced by global mean vectors and better domain adaptation is realized.

Let $\sum_{i = 1}^{n_{s}} X_{i} K_{o} X_{i}^{T} + \sum_{j = 1}^{n_{t}} Y_{i} K_{o} Y_{i}^{T} = \tilde{K}$ and $\sum_{i = 1}^{n_{s}} X_{i} R_{o} (X_{i})^{T} + \sum_{j = 1}^{n_{t}} Y_{i} R_{o} (Y_{i})^{T} = \tilde{R}$ , the model characterized by Equations (3-4) can be expressed by the following:

$W^{*} = arg min_{W} tr (W^{T} (\tilde{K} + λ I_{d}) W),$ (5)

$s . t . \begin{matrix} \end{matrix} W^{T} \tilde{RW} = I_{m} .$ (6)

2.2 Adapted discriminative coupled mappings

For LR face recognition, each domain has two part, one is for HR images and the other is for LR images. Let ${X_{H}^{t}}$ and ${X_{L}^{t}}$ represent the HR target sub-domain and LR target sub-domain, respectively; then the complete target domain is denoted as $= {{X_{H}^{t}}; {X_{L}^{t}}}$ Similarly, $= {{X_{H}^{s}}; {X_{L}^{s}}}$ denotes the complete source domain, including HR source sub-domain ${X_{H}^{s}}$ and LR source sub-domain ${X_{L}^{s}}$ . The datasets in the source domain contain several training samples per subject, while the datasets in the target domain include only one sample per person. In the HR and LR source datasets, we assume that every person has the same number of samples.

The basic idea of this method is to learn an interlaced common subspace for the source and target domain, where the samples are clustered according to their class labels. Both the source and target domains contain HR and LR sub-domains, but the HR and LR sub-domains differ greatly from each other. To reduce this difference, the coupled projections are learned, represented as P_H and P_L, which projects HR images and LR images from different sub-domains into a common subspace. To well transfer the knowledge embedded in the source domain to the target domain and improve the recognition performance of LR face recognition in the target domain, the common subspace should meet the following requirements:

1. Good domain adaptation across domains should be realized.

In this paper, BLM-DA is developed to achieve this goal for LR face recognition with one-shot. BLM-DA has two parts, which are described below.

(1) Minimization of the data distribution mismatch between ${X_{H}^{t}}$ and ${X_{H}^{s}}$ , ${X_{L}^{t}}$ and ${X_{L}^{s}}$ , simultaneously.

When using the mapping P_H to project $X_{H}^{t}$ and $X_{H}^{s}$ to the common subspace, the distribution of their projections should be minimized; also, when using the mapping P_L to project $X_{L}^{t}$ and $X_{L}^{s}$ into the common subspace, the distribution of their projections should be minimized. Let $X_{H} = [X_{H}^{s}, X_{H}^{t}]$ and $X_{L} = [X_{L}^{s}, X_{L}^{t}]$ be the dataset contains all HR images and LR images from two domains, respectively. According to Equation (5), we get

$\begin{matrix} [P_{H}^{*}, P_{L}^{*}] \\ = arg min_{P_{H}, P_{L}} tr (P_{H}^{T} {\tilde{K}}_{H} P_{H}) + tr (P_{L}^{T} {\tilde{K}}_{L} P_{L}) \\ = arg min_{P_{H}, P_{L}} tr ([\begin{matrix} P_{H}^{T} & P_{L}^{T} \end{matrix}] [\begin{matrix} {\tilde{K}}_{H} & 0 \\ 0 & {\tilde{K}}_{L} \end{matrix}] [\begin{matrix} P_{H} \\ P_{L} \end{matrix}]) \\ = arg min_{P_{H}, P_{L}} tr (P^{T} K_{HL} P) \end{matrix}$ (7) where ${\tilde{K}}_{H}$ and ${\tilde{K}}_{L}$ are computed by using samples in X_H and X_L, respectively. In the computation of ${\tilde{K}}_{H}$ , we use the neighbor relationship across domains. Specifically, for each sample in the HR source dataset $X_{H}^{s}$ , its k nearest neighbors in $X_{H}^{s}$ and $X_{H}^{t}$ are found out, respectively; meanwhile for for each sample in the HR target dataset $X_{H}^{t}$ , its k nearest neighbors in $X_{H}^{s}$ and $X_{H}^{t}$ are also found out, respectively. We call this ‘bidirectional-matching’ because not only the local matching information of a sample in source domain is considered, but also that of a sample in target domain is considered. Note that when computing $K_{L}^{△}$ , we use the same neighbor relationship obtained via X_H to find out the neighbors in $X_{L}^{s}$ and $X_{L}^{t}$ , since the HR images involve more information than LR images.

(2) Maximal preservation of the variance of the data.

Besides reducing the discrepancy between the source and target domains, the structures of both domains in the common subspace are also expected to be preserved to the largest extent. That is to say, when using the mapping P_H to project $X_{H}^{t}$ and $X_{H}^{s}$ into the common subspace, as well as using the mapping P_L to project $X_{L}^{t}$ and $X_{L}^{s}$ into the common subspace, the data variances of X_H and X_L should be simultaneously preserved as much as possible. According to Equation (6), we get

$\begin{matrix} [P_{H}^{*}, P_{L}^{*}] \\ = arg max_{P_{H}, P_{L}} tr (P_{H}^{T} {\tilde{R}}_{H} P_{H}) + tr (P_{L}^{T} {\tilde{R}}_{L} P_{L}) \\ = arg max_{P_{H}, P_{L}} tr ([\begin{matrix} P_{H}^{T} & P_{L}^{T} \end{matrix}] [\begin{matrix} {\tilde{R}}_{H} & 0 \\ 0 & {\tilde{R}}_{L} \end{matrix}] [\begin{matrix} P_{H} \\ P_{L} \end{matrix}]) \\ = arg max_{P_{H}, P_{L}} tr (P^{T} R_{HL} P) \end{matrix}$ (8) where ${\tilde{R}}_{H}$ and ${\tilde{R}}_{L}$ are computed by using samples in X_H and X_L, respectively. In the computation of ${\tilde{R}}_{H}$ and ${\tilde{R}}_{L}$ , we use the same rule as the computation of ${\tilde{K}}_{H}$ and ${\tilde{K}}_{L}$ .

2. Good discriminative power of the common subspace should be ensured.

With the coupled mappings projecting HR and LR images to the common subspace, for both the source domain and target domain, the projections from the same class should be as close as possible while the projections from different classes should be as distant as possible. A typical coupled mappings based formulation for LR face recognition with multiple training samples per person is

$[P_{H}^{*}, P_{L}^{*}] = arg max_{P_{H}, P_{L}} \frac{tr (P^{T} G_{b} P)}{tr (P^{T} G_{w} P)},$ (9) where G_b and G_w are the mensuration used to measure the inter-class and intra-class difference, respectively. Considering samples from source and target domain, the discriminative learning model should be formulated as:

$\begin{matrix} [P_{H}^{*}, P_{L}^{*}] \\ = arg max_{P_{H}, P_{L}} \frac{tr (P^{T} G_{b}^{s} P) + tr (P^{T} G_{b}^{t} P)}{tr (P^{T} G_{w}^{s} P) + tr (P^{T} G_{w}^{t} P)}, \end{matrix}$ (10) where $G_{b}^{s}$ and $G_{w}^{s}$ are computed by using HR and LR samples in the source domain $= {{X_{H}^{s}}; {X_{L}^{s}}}$ ; similarly, $G_{b}^{t}$ and $G_{w}^{t}$ are calculated by using HR and LR face images in the target domain $= {{X_{H}^{t}}; {X_{L}^{t}}}$ .

For the proposed AdaDCM, its goal is to find a common subspace simultaneously satisfying the above two requirements. Therefore, we integrate BLM-DA and Coupled Mappings together, and formulate AdaDCM model as:

$\begin{matrix} [P_{H}^{*}, P_{L}^{*}] = arg max_{P_{H}, P_{L}} \\ \frac{α tr (P^{T} G_{b}^{s} P) + (1 - α) tr (P^{T} G_{b}^{t} P) + τ tr (P^{T} R_{HL} P)}{β tr (P^{T} G_{w}^{s} P) + (1 - β) tr (P^{T} G_{w}^{t} P) + γ tr (P^{T} K_{HL} P)} \\ = arg max_{P_{H}, P_{L}} \frac{tr (P^{T} (α G_{b}^{s} + (1 - α) G_{b}^{t} + τ R_{HL}) P)}{tr (P^{T} (β G_{w}^{s} + (1 - β) G_{w}^{t} + γ K_{HL}) P)}, \end{matrix}$ (11)

in which we add four parameters: α and β are used to balance the contribution between the source and target domain; τ and γ are used to control the contribution between data variance preservation and distribution mismatch reduction, respectively. This formulation clear shows that domain adaptation and discriminative coupled mappings are jointly optimized. This model utilizes an auxiliary dataset to enhance the performance of discriminative coupled mappings, making discriminative coupled mappings adapt for LR face recognition with one-shot, hence we call it Adapted Discriminative Coupled Mappings (AdaDCM).

In the following of this paper, we select SDA, which is one of the coupled mappings based methods for LR face recognition, as the example to elaborately analyze the proposed AdaDCM. We integrate SDA [37] with BLM-DA, and obtain AdaSDA, which is able to tackle LR face recognition with one-shot. According the formulation of SDA [37] and AdaDCM framework, AdaSDA is characterized by

$\begin{matrix} [P_{H}^{*}, P_{L}^{*}] \\ = arg max_{P_{H}, P_{L}} \frac{tr (P^{T} (α S_{b}^{s} + (1 - α) S_{b}^{t} + τ R_{HL}) P)}{tr (P^{T} (β S_{w}^{s} + (1 - β) S_{w}^{t} + γ K_{HL}) P)}, \\ = arg max_{P_{H}, P_{L}} \frac{tr (P^{T} Θ P)}{tr (P^{T} Ω P)} \end{matrix}$ (12)

where $S_{b}$ and $S_{b}$ denote inter-class and intra-class scatter matrix, respectively. According to Equation (12), AdaSDA learns only one couple of mapping to realize discriminative learning and domain adaptation simultaneously. Compared with the solution learning two couples of mappings for domain adaptation and discriminative learning respectively, AdaSDA avoids more information loss caused by another dimension reduction. The optimal $P = {[\begin{matrix} P_{H}^{T} & P_{L}^{T} \end{matrix}]}^{T}$ consists of the eigenvectors of $Ω_{-1} Θ$ corresponding to its first largest d eigenvalues, in which $Θ = α S_{b}^{s} + (1 - α) S_{b}^{t} + τ R_{HL}$ and $Ω = β S_{w}^{s} + (1 - β) S_{w}^{t} + γ K_{HL}$ .

According to Equation (12), the computation costs of the proposed method includes two parts: the computation of six square matrices, i.e., $S_{b}^{s}, S_{w}^{s}, S_{b}^{t}, S_{w}^{t}, R_{HL}, K_{HL}$ , their computation time complexity is O ((n_s + n_t) ²); the computation eigenvalue decomposition, i.e., $Ω_{-1} Θ$ , its computation time complexity is O (d (n_s + n_t) ²), in which d is the required eigenvectors. Therefore, to get the projection matrices, the proposed method requires only an eigenvalue decomposition, and the total computation time complexity is O (d (n_s + n_t) ²), it is efficient. We employ a PC with Intel Core(TM) i7 2.3 GHz CPU, 16 G RAM and Matlab 2015a software to conduct experiments. When the size of HR dataset and LR dataset in the source domain are 1200, the size of HR gallery and LR gallery dataset in the target domain is 158, the training time of the 2.238 s.

3 Experimental analysis

AdaSDA is evaluated by extensive experiments on four popularly used face databases, i.e., CAS-PEAL-R1[10], FERET [26], LFW [16], and SCface [13]. Face images in the first two databases are taken in constrained settings, while images in the last two databases are captured under uncontrolled environment. Noted that in Section 4.5, we also report the performance of other adapted coupled mappings methods, including AdaCLPM and AdaCMFA.

3.1 Dataset description

The CAS-PEAL-R1 (short for CAS in the following) database is the largest Chinese face database, which contains various face variations. We use this image set as a source database. The FERET database consists of images of 1565 subjects. Five subsets are selected in this work, i.e., training set, Fa, Fb, Dup1 and Dup2. The training set is used as the source data, while other subsets are used as target data. The LFW database includes images taken in a wild environment, which involves diverse variations such as pose, scale, lighting, hairstyle, expression, partial occlusion. Similar to [4], we utilize a subset containing faces of 158 subjects from LFW-a database in the experiments. For this database, the HR gallery set contains the most frontal face image of every subject; the HR probe set contains 5 no-frontal face images of each individual.

The SCface database consists of 130 persons’ face images pictured in a simulating real-world environment. The face images were taken by five video surveillance cameras at three different distances (4.20, 2.60 and 1.00 m). As the LR images are very close to images captured in the real-world environment, recognizing the LR faces in SCface database is much more challenging. In this work, five sections of images in this database are selected: the first section contains HR face images pictured by a high-quality camera; the other four sections contain LR face images captured at 3 different distances by camera 1–4.

Table 1 list the detailed information of the datasets used in the experiments. The sample HR and LR face images of these dataset are shown in Fig. 2. Specially, in Fig. 2(d), the ones in the top two rows and bottom two rows are the original images and their corresponding cropped images, respectively.

Table 1
The information of the datasets used in the experiments

Source data Target data

Gallery set Probe set

CAS training set 1200 images of 300 subjects, 4 images for each individual FERET Fa: 994 images of 994 persons Fb: 992 LR images

Dup1: 736 LR images

Dup2: 228 LR images

FERET training set 540 images of 180 persons, 3 images per subject LFW 158 images of 158 persons 790 LR images

LFW training set 790 images of 158 persons, 5 images for each person SCface 65 images of the first 65 persons 12 sets (3 distances×4 cameras), each set contains 65 LR images of the last 65 persons

Source data	Target data
CAS training set	1200 images of 300 subjects, 4 images for each individual	FERET	Fa: 994 images of 994 persons	Fb: 992 LR images
				Dup1: 736 LR images
				Dup2: 228 LR images
FERET training set	540 images of 180 persons, 3 images per subject	LFW	158 images of 158 persons	790 LR images
LFW training set	790 images of 158 persons, 5 images for each person	SCface	65 images of the first 65 persons	12 sets (3 distances×4 cameras), each set contains 65 LR images of the last 65 persons

Fig. 2

Sample face images: (a) CAS, (b) FERET, (c) LFW and (d) SCface database.

3.2 Experimental setting

The basic experimental setting follows [4, 6]. The HR face images are 64×64 pixels; the LR face images in SCface dataset are cropped to 16×16 pixels; and LR face images in other datasets are down-sampled from the HR images to 8×8, 12×12 and 16×16 pixels. Note that there are only one HR sample and one LR sample for each subject in the training set of the target domain.

There are five parameters to be pre-specified in the AdaSDA model, i.e., k in Equation (3), (α, β) and (τ, γ) in Equation (11). Unless otherwise specified, the values of these parameters are set as the following. When FERET dataset is used as target data, we set k = 300, (τ, γ) = (1, 10); when LFW and SCface dataset are used as target data, we set k = 80 and k = 30 respectively, and set (τ, γ) = (1e^-3, 1e^-3). In all experiments, we set (α, β) = (0.9, 0.1). For classification purpose, the nearest neighbor classifier is used, and the Rank-1 recognition rate is reported.

3.3 Analysis of adapted SDA

3.3.1 The Importance of BLM-DA

To employ the auxiliary dataset to address LR face recognition with one-shot, we proposed BLM-DA technique to achieve good domain adaptation across domains, and integrate BLM-DA into the SDA to boost the recognition performance of SDA. BLM-DA plays an essential role in the model of the proposed AdaSDA. We examine the importance of BLM-DA in the following by conducting experiments on the FERET database and making comparisons between several variants of SDA based methods. The detail of these variants is listed in Table 2. In Table 2, ‘BLM-DA’ is the proposed new domain adaptation technique, ‘DA’ represents traditional domain adaptation technique; ‘One-step’ means learning the coupled mapping to simultaneously realize domain adaptation and discriminative analysis, while ‘two-step’ means first learning coupled mappings for domain adaptation, then unifying the source and target samples in the transferring subspace to further learn discriminative subspace.

Table 2
The variants of SDA

The variant approach Discrim-inative learning (DL) method The dataset in source domain used for training The dataset in target domain used for training Domain adaptation (DA) technique The way combine DL and DA

SDA_st_BL_1 SDA ✓ ✓ BLM-DA One-step

SDA_st_BL_2 SDA ✓ ✓ BLM-DA Two-step

SDA_st_DA_1 SDA ✓ ✓ DA One-step

SDA_st_DA_2 SDA ✓ ✓ DA Two-step

SDA_st SDA ✓ ✓ x —

SDA_s SDA ✓ x x —

SDA_t SDA x ✓ x —

The variant approach	Discrim-inative learning (DL) method	The dataset in source domain used for training	The dataset in target domain used for training	Domain adaptation (DA) technique	The way combine DL and DA
SDA_st_BL_1	SDA	✓	✓	BLM-DA	One-step
SDA_st_BL_2	SDA	✓	✓	BLM-DA	Two-step
SDA_st_DA_1	SDA	✓	✓	DA	One-step
SDA_st_DA_2	SDA	✓	✓	DA	Two-step
SDA_st	SDA	✓	✓	x	—
SDA_s	SDA	✓	x	x	—
SDA_t	SDA	x	✓	x	—

On FERET dataset, CAS database is used as the source domain. Figure 3 displays the comparison of the seven models. We can observe that SDA_st_BL_1 performs best in every case. In most cases (5 out 6), SDA_s performs worst, SDA_st performs worse than BLM-DA based methods, the one-step methods (both DA-based and BLM-DA based) achieve higher accuracy than two-step methods. We can get three insights from these findings. First, when a model learned from one domain is applied to another domain, its performance degrades obviously. Second, simultaneously realizing the goals of domain adaptation and discriminative analysis is better than sequentially achieving these goals. Third, BLM-DA is beneficial for LR face recognition with one-shot.

Fig. 3

Recognition performance on FERET database: (a) LR images are 16×16, (b) LR images are 8×8.

3.3.2 The influence of parameter pair (τ, γ)

To gain more insights for the proposed AdaSDA approach, experiments are conducted to investigate the role of the parameter τ and γ, which are applied to balance the importance of two component of BLM-DA, i.e., maximizing data variance and minimizing distribution mismatch, respectively. In the following experiments, CAS and FETET database are used as source and target data, respectively. LR faces used are of 16×16 and 8×8 pixels.

We keep other parameters fixed and vary both τ and γ according to the vector [1e^-3, 1e^-2, 1e^-1, 1, 0, 1e¹, 1e², 1e³]. We conduct six experiment on the FERET dataset. Figure 4 illustrates the Rank-1 accuracy versus different values of τ and γ. As can be seen, the shapes of figures in the top row are similar to those in the bottom row. Specifically, when the value of γ is fixed, the various values of τ produce similar recognition accuracy; while when the value of τ is fixed, the changes of γ have an obvious influence on the recognition results, and when the value of γ is larger than 10, in all subfigures the recognition accuracy will drop significantly. It implies that minimizing the data distribution mismatch plays more important role than maximizing the data variance in BLM-DA.

Fig. 4

Recognition accuracy of AdaSDA versus various values of τ and γ on the FERET database. Top row: LR images are 16×16, bottom row: LR images are 8×8. Three columns denote probe Fb, Dup1 and Dup2, respectively.

3.4 Recognition performance on unconstrained LR face images

With the LFW database and SCface database, the LR face recognition performance under uncontrolled environment of AdaSDA is evaluated. In the following experiments, two classes of methods are compared. The first class of methods includes DCA [14], MRDL [24], E-SDA and the proposed AdaSDA, which employ the provided auxiliary database. E-SDA and AdaSDA are domain adaptation based ones, while DCA and MRDL use the auxiliary database for training the model only. The second class of methods includes CLPM [21], CMFA [30], SDA [37], C-RSDA [4], VGGFace [33] and ResNet [15], which do not use the provided auxiliary database. CLPM, CMFA, SDA and C-RSDA are trained with the training set in the target domain only, and C-RSDA is specifically designed for recognizing LR face images with one-shot. VGGFace and ResNet are deep learning based methods, they are pre-trained on ImageNet [7]. When the deep learning based methods are used to classify LR images, the LR images are rescaled to the required input size of the network by bicubic interpolation. Since VGGFace and ResNet are only pre-trained on ImageNet, and the data size of the provided auxiliary set is too small to train them, we categorize the two deep learning based methods as those which are trained without the auxiliary set.

3.4.1 Evaluation on LFW dataset

The LFW dataset is treated as the target domain, while the CAS and FERET database are treated as the source domain respectively. The performance of SDA_st_BL_1 (i.e., AdaSDA), SDA_st_BL_2, SDA_st_DA_1 (i.e., E-SDA), SDA_st_DA_2, are also evaluated. Table 3 posts the recognition rates of diverse approaches when classify LR face images of different resolutons. According to the average accuracy in Table 3, domain adaptation methods (E-SDA and AdaSDA) perform obviously better than other methods. Though VGGFace and RetNet are pre-trained by large-scale database, they perform worse than those specifically designed methods for LR face recognition. DCA and MRDL do not outperform E-SDA and AdaSDA, which may imply that the models trained by source domain databases is not well suitable for target domain. Besides, the performance of C-RSDA is better than SDA_st_DA_2 and SDA_st_BL_2, but still worse than E-SDA and AdaSDA. With the source data, the one-step methods outperform two-step methods by a margin of at least three percent. This implies that achieving domain adaptation and discriminative analysis at the same time by a coupled mappings is quite beneficial for LR face recognition. Furthermore, AdaSDA obtains better performance than E-SDA in each case, implying that BLM-DA technique is effective in domain adaptation. According to the rightmost column of Table 3, it can be found that using different auxiliary datasets affects not only the recognition accuracy, but also the recognition stability of different methods. Compared with other auxiliary dataset based methods, the standard deviations of AdaSDA is least, implies its stability on recognizing different LR faces.

Table 3
Recognition accuracy (%) on LFW database

8×8 12×12 16×16 Ave.±Std.

Without auxiliary set CLPM [21] 4.10 6.58 7.47 6.05±1.43

CMFA [30] 9.49 10.38 10.89 10.25±0.58

SDA [37] 4.94 6.96 7.85 6.58±1.22

C-RSDA [4] 12.28 14.43 15.06 13.92±1.19

VGGFace [33] 9.11 9.87 12.15 10.38±1.29

ResNet [15] 9.87 10.89 14.05 11.60±1.78

CAS → LFW SDA_st_DA_2 8.61 12.41 15.57 12.19±2.85

SDA_st_BL_2 8.35 11.39 15.57 11.77±2.96

DCA [14] 8.48 12.15 16.08 12.24±3.10

MRDL [24] 7.97 10.63 14.43 11.01±2.65

E-SDA [5] 12.53 15.44 18.86 15.61±2.59

AdaSDA 13.16 16.84 19.37 16.46±2.55

FERET → LFW SDA_st_DA_2 8.61 11.39 11.90 10.63±1.45

SDA_st_BL_2 8.48 11.27 13.80 11.18±2.17

DCA 8.48 10.89 15.57 11.65±2.94

MRDL 8.23 9.87 13.42 10.51±2.17

E-SDA 12.15 14.56 15.06 13.92±1.27

AdaSDA 13.29 15.19 15.19 14.56±0.90

		8×8	12×12	16×16	Ave.±Std.
Without auxiliary set	CLPM [21]	4.10	6.58	7.47	6.05±1.43
	CMFA [30]	9.49	10.38	10.89	10.25±0.58
	SDA [37]	4.94	6.96	7.85	6.58±1.22
	C-RSDA [4]	12.28	14.43	15.06	13.92±1.19
	VGGFace [33]	9.11	9.87	12.15	10.38±1.29
	ResNet [15]	9.87	10.89	14.05	11.60±1.78
CAS → LFW	SDA_st_DA_2	8.61	12.41	15.57	12.19±2.85
	SDA_st_BL_2	8.35	11.39	15.57	11.77±2.96
	DCA [14]	8.48	12.15	16.08	12.24±3.10
	MRDL [24]	7.97	10.63	14.43	11.01±2.65
	E-SDA [5]	12.53	15.44	18.86	15.61±2.59
	AdaSDA	13.16	16.84	19.37	16.46±2.55
FERET → LFW	SDA_st_DA_2	8.61	11.39	11.90	10.63±1.45
	SDA_st_BL_2	8.48	11.27	13.80	11.18±2.17
	DCA	8.48	10.89	15.57	11.65±2.94
	MRDL	8.23	9.87	13.42	10.51±2.17
	E-SDA	12.15	14.56	15.06	13.92±1.27
	AdaSDA	13.29	15.19	15.19	14.56±0.90

3.4.2 Evaluation on SCface dataset

The LR images in SCface dataset are taken by the surveillance cameras. Thus, these LR images are real. Different from the above experiments, the gallery and probe set of SCface dataset do not contain any overlapping persons. In this subsection, we adopt the CAS and LFW dataset as the source domain, respectively. As two-step methods fail to show better performance than one-step methods, in this subsection we only access the one-step domain adaptation methods. Tables 4 –6 displays the comparison among various approaches in three situations: the disatances betweeen the persons and cameras are 1 meter, 2.6 meters and 4.2 meters. It can be observed that the domain adaptation based methods still outperform other compared methods in overall performance.

Table 4
Recognition accuracy (%) on SCface database (Distance = 1 m)

Cam1 Cam2 Cam3 Cam4 Ave.±Std.

Without Auxiliary set CLPM 6.15 1.54 1.54 3.08 3.08±1.88

CMFA 10.00 4.62 7.69 8.46 7.69±1.96

SDA 16.92 12.31 20.00 16.92 16.54±2.75

C-RSDA 20.00 10.77 23.08 20.00 18.46±4.62

VGGFace 15.38 16.92 16.92 20.00 17.31±1.68

ResNet 16.92 16.92 18.46 21.54 18.46±1.89

CAS → SCface DCA 15.38 12.31 21.54 27.69 19.23±5.91

MRDL 10.77 9.23 15.38 13.85 12.31±2.43

E-SDA 21.54 18.46 20.00 24.62 21.15±2.28

AdaSDA 23.08 20.00 21.54 24.62 22.31±1.72

LFW → SCface DCA 16.92 21.54 23.08 21.54 20.77±2.31

MRDL 12.31 18.46 16.92 20.00 16.92±2.88

E-SDA 18.46 27.69 21.54 26.15 23.46±3.67

AdaSDA 20.00 27.69 23.08 26.15 24.23±2.95

		Cam1	Cam2	Cam3	Cam4	Ave.±Std.
Without Auxiliary set	CLPM	6.15	1.54	1.54	3.08	3.08±1.88
	CMFA	10.00	4.62	7.69	8.46	7.69±1.96
	SDA	16.92	12.31	20.00	16.92	16.54±2.75
	C-RSDA	20.00	10.77	23.08	20.00	18.46±4.62
	VGGFace	15.38	16.92	16.92	20.00	17.31±1.68
	ResNet	16.92	16.92	18.46	21.54	18.46±1.89
CAS → SCface	DCA	15.38	12.31	21.54	27.69	19.23±5.91
	MRDL	10.77	9.23	15.38	13.85	12.31±2.43
	E-SDA	21.54	18.46	20.00	24.62	21.15±2.28
	AdaSDA	23.08	20.00	21.54	24.62	22.31±1.72
LFW → SCface	DCA	16.92	21.54	23.08	21.54	20.77±2.31
	MRDL	12.31	18.46	16.92	20.00	16.92±2.88
	E-SDA	18.46	27.69	21.54	26.15	23.46±3.67
	AdaSDA	20.00	27.69	23.08	26.15	24.23±2.95

Table 5

Recognition accuracy (%) of the various method on SCface database (Distance = 2.6 m)

		Cam1	Cam2	Cam3	Cam4	Ave.±Std.
Without Auxiliary set	CLPM	3.08	6.15	4.62	3.08	4.23±1.27
	CMFA	5.38	5.38	7.69	5.38	5.96±1.00
	SDA	15.38	18.64	10.77	23.08	16.97±4.50
	C-RSDA	16.92	20.00	10.77	24.62	18.08±5.03
	VGGFace	9.23	10.77	12.31	15.38	11.92±2.27
	ResNet	10.77	12.31	13.85	16.92	13.46±2.27
CAS → SCface	DCA	12.31	13.85	16.92	20.00	15.77±2.95
	MRDL	9.23	10.77	10.77	12.31	10.77±1.09
	E-SDA	21.54	20.00	16.92	23.08	20.38±2.28
	AdaSDA	23.08	21.54	18.46	24.62	21.92±2.28
LFW → SCface	DCA	12.31	16.92	18.46	20.00	16.92±2.88
	MRDL	10.77	12.31	10.77	13.85	11.92±1.28
	E-SDA	23.08	21.54	15.38	24.62	21.16±3.51
	AdaSDA	24.62	21.54	16.92	26.15	22.31±3.53

Table 6

Recognition accuracy (%) of the various method on SCface database (Distance = 4.2 m)

		Cam1	Cam2	Cam3	Cam4	Ave.±Std.
Without Auxiliary set	CLPM	3.08	4.62	3.08	3.08	3.47±0.67
	CMFA	4.62	4.62	4.62	3.08	4.24±0.67
	SDA	12.31	12.31	18.46	12.31	13.85±2.66
	C-RSDA	13.85	13.85	20.00	15.38	15.77±2.52
	VGGFace	7.69	6.15	9.23	7.69	7.69±1.09
	ResNet	6.15	7.69	10.77	9.23	8.46±1.72
CAS → SCface	DCA	12.31	9.23	7.69	10.77	10.00±1.72
	MRDL	7.69	6.15	6.15	7.69	6.92±0.77
	E-SDA	13.85	12.31	15.38	13.85	13.85±1.09
	AdaSDA	16.53	13.85	18.46	16.92	16.44±1.66
LFW → SCface	DCA	10.77	9.23	10.77	10.77	10.38±0.67
	MRDL	6.15	7.69	4.62	6.15	6.15±1.09
	E-SDA	18.46	12.31	18.46	13.85	15.77±2.74
	AdaSDA	16.92	13.85	18.46	16.92	16.54±1.67

According to the averaged performance in Table 4, with the source data, both E-SDA and AdaSDA obviously outperform other compared methods. Compared with C-RSDA, domain adaptive methods demonstrate big advantage on recognizing faces captured by Cam2 and Cam4, achieve only competitive performance on recognizing faces captured by Cam1 and Cam3, and display lower standard deviations of recognition accuracy. According to Table 5, E-SDA and AdaSDA also achieve higher recognition accuracy than other methods. But compared with C-RSDA, domain adaptive methods show apparent superiority on recognizing faces captured by Cam1 and Cam3, and obtain only competitive performance on recognizing faces captured by Cam2 and Cam4, but the standard deviations are higher. It is contrary to what we find in Table 4. The possible reason is that the distribution of face data collected by various cameras at different distances may vary a lot, and distribution mismatch between the target domain and the source domain cannot be reduced by equal effect using BLM-DA, which makes LR face recognition more difficult than the case of ‘Distance = 1m’. According to these two tables, AdaSDA obtains the best overall performance and low standard deviations in each case, and performs best in nearly all experiments, which further demonstrates its effectiveness.

Table 6 illustrates the performance when the distance is 4.2 meters. The large distance makes the quality of the captured face images quite low, and the recognition is much worse than before. With the distances of cameras varying from 1.0 meter to 4.2 meters, the performance of two deep learning based methods degrades greatly. In Table 6, though E-SDA obtains better performance than CLPM, CMFA and SDA, it fails to surpass C-RSDA. This is different from what we observe before. Unsurprisingly, AdaSDA still achieves the highest overall accuracy and quite low standard deviation, though it achieves only a small margin over C-RSDA in average accuracy.

According to Tables 4 –6, compared with CAS database, when the LFW database is employed as the source domain, the better recognition accuracy is obtained. The reason may be twofold. First, LFW dataset contains more samples for each subject. Second, images in the LFW database involve more variations than those in the CAS database. However, no matter which database is used as source domain, the proposed AdaSDA approach obtains better performance than other methods. Therefore, AdaSDA is effective in recognizing LR face images when there is only one sample per person in the gallery set.

3.5 Robustness analysis

In the experiments, we evaluate the performance of the proposed method on recognizing different types of LR face images. The rightest column of Table 3 lists the average performance of various methods on recognizing 8×8, 12×12, 16×16 LR face images. Compared with other methods, it shows that AdaSDA obtains the highest accuracy with small standard deviation, implying that the proposed method is robust to the image resolutions. The rightest columns of Tables 4 –6 show the average performance on recognizing LR face images captured by different cameras or picture conditions. The best performance and relatively small standard deviates indicate the robustness of AdaSDA to the picture conditions of LR images.

Moreover, to investigate the robustness of AdaSDA to image quality of face images captured from the unconstrained environment, we also present the average recognition accuracy of 3 distances on SCface database, as Table 7 shows. According to this table, compared with the second best method, AdaSDA achieves higher average accuracy and smaller or comparative standard deviation, demonstrating the robustness of the proposed method.

Table 7
The average recognition accuracy (%) from 3 distances on SCface database (Ave.±Std.)

Cam1 Cam2 Cam3 Cam4

Without Auxiliary se CLPM 4.1±1.45 4.1±1.92 3.08±1.26 3.08±0

CMFA 6.67±2.38 4.87±0.36 6.67±1.45 5.64±2.2

SDA 14.87±1.92 14.42±2.98 16.41±4.04 17.44±4.41

C-RSDA 16.92±2.51 14.87±3.84 17.95±5.23 20±3.77

VGGFace 10.77±3.32 11.28±4.41 12.82±3.16 14.36±5.08

ResNet 11.28±4.41 12.31±3.77 14.36±3.16 15.9±5.08

CAS → SCface DCA 13.33±1.45 11.8±1.92 15.38±5.76 19.49±6.92

MRDL 9.23±1.26 8.72±1.92 10.77±3.77 11.28±2.62

E-SDA 18.98±3.63 16.92±3.32 17.43±1.92 20.52±4.76

AdaSDA 20.9±3.09 18.46±3.32 19.49±1.45 22.05±3.63

LFW → SCface DCA 13.33±2.61 15.9±5.08 17.44±5.08 17.44±4.76

MRDL 9.74±2.62 12.82±4.41 10.77±5.02 13.33±5.67

E-SDA 20±2.18 20.51±6.32 18.46±2.51 21.54±5.47

AdaSDA 20.51±3.16 21.03±5.66 19.49±2.62 23.07±4.35

		Cam1	Cam2	Cam3	Cam4
Without Auxiliary se	CLPM	4.1±1.45	4.1±1.92	3.08±1.26	3.08±0
	CMFA	6.67±2.38	4.87±0.36	6.67±1.45	5.64±2.2
	SDA	14.87±1.92	14.42±2.98	16.41±4.04	17.44±4.41
	C-RSDA	16.92±2.51	14.87±3.84	17.95±5.23	20±3.77
	VGGFace	10.77±3.32	11.28±4.41	12.82±3.16	14.36±5.08
	ResNet	11.28±4.41	12.31±3.77	14.36±3.16	15.9±5.08
CAS → SCface	DCA	13.33±1.45	11.8±1.92	15.38±5.76	19.49±6.92
	MRDL	9.23±1.26	8.72±1.92	10.77±3.77	11.28±2.62
	E-SDA	18.98±3.63	16.92±3.32	17.43±1.92	20.52±4.76
	AdaSDA	20.9±3.09	18.46±3.32	19.49±1.45	22.05±3.63
LFW → SCface	DCA	13.33±2.61	15.9±5.08	17.44±5.08	17.44±4.76
	MRDL	9.74±2.62	12.82±4.41	10.77±5.02	13.33±5.67
	E-SDA	20±2.18	20.51±6.32	18.46±2.51	21.54±5.47
	AdaSDA	20.51±3.16	21.03±5.66	19.49±2.62	23.07±4.35

Compared with other methods, the proposed AdaSDA improves the recognition accuracy of LR face recognition with one-shot. However, the improvement is limited, and the recognition accuracy is still too low to be used in the unconstrained real-world applications. The reason is that our proposed BLM-DA technique considers only the data distribution mismatch between two domains, ignores other factors such as the essential face feature differences in races, which also induce domain discrepancy. Besides, AdaSDA employs coupled mappings to match images of different resolutions, but does not consider the domain adaptation of the same low-resolutions across domains, which may affect the recognition performance.

3 Discussion

We demonstrate that except SDA other coupled mappings based methods can be embedded into AdaDCM framework, to tackle LR face recognition with one-shot. In this subsection, we report the performance of AdaCLPM and AdaCMFA, which integrate BLM-DA with CLPM, CMFA, respectively. Besides, we further compare the one-step and two-step framework. These experiments are conducted on the LFW dataset by using the CAS database and FERET database as the source domain, respectively. Each coupled mappings based methods has four variants. The recognition results of all variant methods are listed in Table 8. We can conclude two things from this table. Frist, the one-step framework is more helpful than two-step framework in LR face recognition task using auxiliary dataset, because in 23 out of 24 cases, the one-step methods obtain higher accuracy than the two-step ones. The only one exception is that in ‘FERETàLFW’ case, CMFA_DA_2 obtains higher accuracy than CMFA_DA_1. Compared with other results, this implies that traditional domain adaptation (DA) is not stable for small size datasets. Second, BLM-DA is much more effective in the one-step framework than DA when the number of samples in the training set of target domain is quite limited, all one-step methods using BLM-DA perform better than those using DA, and the standard deviations of these methods are lower, implying that BLM-DA is able to output stable performance for small size datasets.

Table 8
Performance comparison among three classical coupled mappings based methods

8×8 12×12 16×16 Ave.±Std.

CAS → LFW CLPM_DA_2 10.00 14.05 14.94 13.00±2.15

CLPM_DA_1 11.14 13.42 14.94 13.16±1.56

CLPM_BL_2 9.11 13.80 14.56 12.49±2.41

AdaCLPM 12.91 14.56 15.82 14.43±1.19

CMFA_DA_2 11.90 15.70 15.82 14.47±1.82

CMFA_DA_1 13.29 14.43 15.82 14.51±1.03

CMFA_BL_2 12.28 15.95 17.34 15.19±2.13

AdaCMFA 13.04 16.20 17.34 15.53±1.82

SDA_DA_2 8.61 12.41 15.57 12.19±2.85

SDA_DA_1 12.53 15.44 18.86 15.61±2.59

SDA_BL_2 8.35 11.39 15.57 11.77±2.96

AdaSDA 13.16 16.84 19.37 16.46±2.55

FERET → LFW CLPM_DA_2 10.38 12.15 13.29 11.94±1.20

CLPM_DA_1 10.76 12.78 14.68 12.74±1.60

CLPM_BL_2 9.49 11.77 14.68 11.98±2.12

AdaCLPM 11.90 13.80 15.82 13.84±1.60

CMFA_DA_2 11.90 14.18 15.44 13.84±1.47

CMFA_DA_1 11.01 14.05 14.43 13.16±1.53

CMFA_BL_2 12.28 15.44 16.96 14.90±1.95

AdaCMFA 13.29 15.19 17.22 15.23±1.60

SDA_DA_2 8.61 11.39 11.90 10.63±1.45

SDA_DA_1 12.15 14.56 15.06 13.92±1.27

SDA_BL_2 8.48 11.27 13.80 11.18±2.17

AdaSDA 13.29 15.19 15.19 14.56±0.90

		8×8	12×12	16×16	Ave.±Std.
CAS → LFW	CLPM_DA_2	10.00	14.05	14.94	13.00±2.15
	CLPM_DA_1	11.14	13.42	14.94	13.16±1.56
	CLPM_BL_2	9.11	13.80	14.56	12.49±2.41
	AdaCLPM	12.91	14.56	15.82	14.43±1.19
	CMFA_DA_2	11.90	15.70	15.82	14.47±1.82
	CMFA_DA_1	13.29	14.43	15.82	14.51±1.03
	CMFA_BL_2	12.28	15.95	17.34	15.19±2.13
	AdaCMFA	13.04	16.20	17.34	15.53±1.82
	SDA_DA_2	8.61	12.41	15.57	12.19±2.85
	SDA_DA_1	12.53	15.44	18.86	15.61±2.59
	SDA_BL_2	8.35	11.39	15.57	11.77±2.96
	AdaSDA	13.16	16.84	19.37	16.46±2.55
FERET → LFW	CLPM_DA_2	10.38	12.15	13.29	11.94±1.20
	CLPM_DA_1	10.76	12.78	14.68	12.74±1.60
	CLPM_BL_2	9.49	11.77	14.68	11.98±2.12
	AdaCLPM	11.90	13.80	15.82	13.84±1.60
	CMFA_DA_2	11.90	14.18	15.44	13.84±1.47
	CMFA_DA_1	11.01	14.05	14.43	13.16±1.53
	CMFA_BL_2	12.28	15.44	16.96	14.90±1.95
	AdaCMFA	13.29	15.19	17.22	15.23±1.60
	SDA_DA_2	8.61	11.39	11.90	10.63±1.45
	SDA_DA_1	12.15	14.56	15.06	13.92±1.27
	SDA_BL_2	8.48	11.27	13.80	11.18±2.17
	AdaSDA	13.29	15.19	15.19	14.56±0.90

4 Conclusions

For public security purpose, recognizing the low-resolution face images with single sample per person is necessary and important. To tackle this challenging problem, this paper proposed AdaDCM approach, which utilizes the auxiliary dataset to encode more face variations. In order to make good use of the auxiliary dataset, the key idea of the proposed method is that it adopts and unifies domain adaptation and discriminative coupled mappings together, so that the face variations exploited from the auxiliary database can closely characterize the possible face variations of unknown face images. The extensive experiments and comparisons on the widely used face databases show the effectiveness of the proposed AdaDCM in recognizing LR face images with one-shot.

AdaDCM employs only one auxiliary database as the source data, while multiple auxiliary databases could be more helpful for extracting discriminative features. Therefore, unifying multiple domain adaptations with coupled mappings will be an interesting research direction about LR face recognition. Besides, AdaDCM still follows the traditional discriminative learning framework, in the future, we will consider to integrate AdaDCM with deep learning-based framework.

Footnotes

Acknowledgments

This research was partially supported by the Humanities and Social Science Research on Youth Fund Project of Ministry of Education of China (Grant No. 19YJC870003), the Natural Science Foundation of Jiangsu Province (Grant No. BK20210576), and sponsored by NUPTSF (Grant No. NY219038, NY220045).

References

Biswas

, Aggarwal

, Flynn

P.J.

and Bowyer

K.W.

, Pose-robust recognition of low-resolution face images, IEEE Transactions on pattern analysis and machine intelligence 35 (2013), 3037–3049.

Biswas

, Bowyer

K.W.

and Flynn

P.J.

, Multidimensional scaling for matching low-resolution face images, IEEE Transactions on pattern analysis and machine intelligence 34 (2012), 2019–2030.

Chen

, Chen

, Wang

, Liang

and Lin

C.-W.

, Identity-aware face super-resolution for low-resolution face recognition, IEEE Signal Processing Letters 27 (2020), 645–649.

Chu

, Ahmad

, Bebis

and Zhao

, Low-resolution face recognition with single sample per person, Signal Processing 141 (2017), 144–157.

Chu

, Su

and Zhao

, Unifying Domain Adaptation into Simultaneous Discriminant Analysis: A Method for Low-Resolution Face Recognition with Single Sample Per Person, in: International Conference on Artificial Intelligence and Security, Springer Singapore, Singapore, (2020), 461–471.

Chu

, Zhao

, Ahmad

and Zhao

, Low-Resolution Face Recognition with Single Sample per Person via Domain Adaptation, International Journal of Pattern Recognition and Artificial Intelligence 33 (2019), 1956005.

Deng

, Dong

, Socher

, Li

L.-J.

, Li

and Fei-Fei

, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, (2009), 248–255.

Dong

, Loy

C.C.

, He

and Tang

, Learning a deep convolutional network for image super-resolution, in: European Conference on ComputerVision, Springer, (2014), 184–199.

Dong

, Loy

C.C.

, He

and Tang

, Image super-resolution using deep convolutional networks, IEEE Transactions on pattern analysis and machine intelligence 38 (2016), 295–307.

10.

Gao

, Cao

, Shan

, Chen

, Zhou

, Zhang

and Zhao

, The CAS-PEAL Large-Scale Chinese Face Database and Baseline Evaluations, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 38 (2008), 149–161.

11.

, Zhao

, Gao

and Li

, Fewer-shots and lower-resolutions: Towards ultrafast face recognition in the wild, in: Proceedings of the 27th ACM International Conference on Multimedia, (2019), 229–237.

12.

, Zhao

, Li

, Zhang

and Li

, Efficient low-resolution face recognition via bridge distillation, IEEE Transactions on Image Processing 29 (2020), 6898–6908.

13.

Grgic

, Delac

and Grgic

, SCface–surveillance cameras face database, Multimedia tools and Applications 51 (2011), 863–879.

14.

Haghighat

and Abdel-Mottaleb

, Low Resolution Face Recognition in Surveillance Systems Using Discriminant Correlation Analysis, in: IEEE International Conference on Automatic Face & Gesture Recognition, (2017), 912–917.

15.

, Zhang

, Ren

and Sun

, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), 770–778.

16.

Huang

G.B.

, Mattar

, Berg

and Learned-Miller

, Labeled faces in the wild: A database forstudying face recognition in unconstrained environments, Month (2007).

17.

Huang

and He

, Super-resolution method for face recognition using nonlinear mappings on coherent features, IEEE Transactions on Neural Networks 22 (2011), 121–130.

18.

Jiao

, Li

, Cao

, Zhong

, Wu

and Wong

H.-S.

, DDAT: Dual Domain Adaptive Translation for Low-Resolution Face Verification in the Wild, Pattern Recognition (2021), 108107.

19.

Kan

, Wu

, Shan

and Chen

, Domain Adaptation for Face Recognition: Targetize Source Domain Bridged by Common Subspace, International Journal of Computer Vision 109 (2014), 94–109.

20.

Kim

K.I.

and Kwon

, Single-image super-resolution using sparse regression and natural image prior, IEEE Transactions on pattern analysis and machine intelligence 32 (2010), 1127–1133.

21.

, Chang

, Shan

and Chen

, Low-resolution face recognition via coupled locality preserving mappings, IEEE Signal Processing Letters 17 (2010), 20–23.

22.

, Prieto

, Mery

and Flynn

P.J.

, On low-resolution face recognition in the wild: Comparisons and new techniques, IEEE Transactions on Information Forensics and Security 14 (2019), 2000–2012.

23.

, Yang

, Zhang

, Li

and Xiong

, Very Low-Resolution Face Recognition via Semi-Coupled Locality-Constrained Representation, in: IEEE International Conference on Parallel and Distributed Systems, (2017), 362–367.

24.

Luo

, Xu

and Yang

, Multi-resolution dictionary learning for face recognition, Pattern Recognition 93 (2019), 283–292.

25.

Pan

S.J.

, Tsang

I.W.

, Kwok

J.T.

and Yang

, Domain Adaptation via Transfer Component Analysis, IEEE Transactions on Neural Networks 22 (2011), 199–210.

26.

Phillips

P.J.

, Moon

, Rizvi

S.A.

and Rauss

P.J.

, The FERET evaluation methodology for face-recognition algorithms, IEEE Transactions on pattern analysis and machine intelligence 22 (2000), 1090–1104.

27.

Ren

C.-X.

, Dai

D.-Q.

and Yan

, Coupled kernel embedding for low-resolution face image recognition, IEEE Transactions on Image Processing 21 (2012), 3770–3783.

28.

Ren

, Dai

, Huang

and Lai

, Transfer Learning of Structured Representation for Face Recognition, IEEE Transactions on Image Processing 23 (2014), 5440–5454.

29.

Rouhsedaghat

, Wang

, Hu

, You

and Kuo

C.-C.J.

, Low-resolution face recognition in resource-constrained environments, Pattern Recognition Letters (2021).

30.

Siena

, Boddeti

V.N.

and Kumar

B.V.

, Coupled marginal fisher analysis for low-resolution face recognition, in: European Conference on Computer Vision Workshops and Demonstrations, Springer, (2012), 240–249.

31.

Wang

, Liu

and Hu

, Coupled Kernel Fisher Discriminative Analysis for Low-Resolution Face Recognition, in: Biometric Recognition, Springer, (2013), 81–88.

32.

Wang

, Liu

, Yang

, Han

and Huang

, Deep networks for image super-resolution with sparse prior, in: Proceedings of the IEEE International Conference on Computer Vision, (2015), 370–378.

33.

Wen

, Zhang

, Li

and Qiao

, A discriminative feature learning approach for deep face recognition, in: European conference on computer vision, Springer, (2016), 499–515.

34.

Yang

, Yang

, Gao

and Liao

, Discriminative Multidimensional Scaling for Low-Resolution Face Recognition, IEEE Signal Processing Letters 25 (2018), 388–392.

35.

Yang

, Wright

, Huang

T.S.

and Ma

, Image super-resolution via sparse representation, IEEE Transactions on Image Processing 19 (2010), 2861–2873.

36.

Zangeneh

, Rahmati

and Mohsenzadeh

, Low resolution face recognition using a two-branch deep convolutional neural network architecture, Expert Systems With Applications 139 (2020), 112854.

37.

Zhou

, Zhang

, Yi

, Lei

and Li

S.Z.

, Low-resolution face recognition via simultaneous discriminant analysis, in: International Joint Conference on Biometrics, IEEE, (2011), 1–6.

Source data		Target data
			Gallery set	Probe set
CAS training set	1200 images of 300 subjects, 4 images for each individual	FERET	Fa: 994 images of 994 persons	Fb: 992 LR images
				Dup1: 736 LR images
				Dup2: 228 LR images
FERET training set	540 images of 180 persons, 3 images per subject	LFW	158 images of 158 persons	790 LR images
LFW training set	790 images of 158 persons, 5 images for each person	SCface	65 images of the first 65 persons	12 sets (3 distances×4 cameras), each set contains 65 LR images of the last 65 persons