A feature map aggregation network for unconstrained video face recognition

Abstract

Unconstrained video face recognition is an extension of face recognition technology, and it is an indispensable part of intelligent security and criminal investigation systems. However, general face recognition technology cannot be directly applied to unconstrained video face recognition, because the video contains fewer frontal face image frames and a single image contains less face feature information. To address the above problems, this work proposes a Feature Map Aggregation Network (FMAN) to achieve unconstrained video face recognition by aggregating multiple face image frames. Specifically, an image group is used as the input of the feature extraction network to replace a single image to obtain a multi-channel feature map group. Then a quality perception module is proposed to obtain quality scores for feature maps and adaptively aggregate image features from image groups at the feature map level. Finally, extensive experiments are conducted on the challenging face recognition benchmarks YTF, IJB-A and COX to evaluate the proposed method, showing a significant increase in accuracy compared to the state-of-the-art.

Keywords

Video face recognition Aggregation Deep convolutional neural network Feature map

1 Introduction

With the wide application of deep learning in the field of computer vision, many computer vision application technologies have been developed, such as face recognition [1, 2], face anti-spoofing [3], self-driving cars [4], pedestrian re-identification [5], and facial expression recognition [6]. Among them, face recognition, as a basic application, has been paid much attention by researchers. Early face recognition methods were mainly for static images, and a series of challenging data sets emerged, such as [7 –12]. However, face recognition methods based on static images have poor performance for video faces. Compared with static face images, unconstrained moving faces in videos can cause huge posture changes and motion blur, and most video frames have incomplete face information. Therefore, it is an effective method to use the multi-view and time information of the video face sequence to characterize the video face [13, 14].

Fig. 1

Similarity measurement. The left part shows a frontal static face image and a set of large pose face image sets of the same person, and the aggregated feature has a smaller distance from the static image feature. The right part shows two sets of images of different people, with a larger distance between the aggregated features.

Compared with static face images, unconstrained moving faces have problem such as pose changes, motion blur, and incomplete face information. Using video face sequences to represent video faces is a feasible solution because it contains multiple views and time information. Video face recognition scenes include: video to static (V2S), static to video (S2V), and video to video (V2V). Specifically, in V2S, the face sample library is composed of static images, and the target face is a video sequence. The image-to-image matching problem has evolved into a more challenging image-to-video, video-to-video matching problem. For the above problem, an intuitive solution is to convert the video face into the same representation as the image face. For example, the key frames in the video sequence are used as the representation of the video face. However, incomplete video face information limits its performance. Therefore, aggregating multiple face images into a unique video face representation is more conducive to the video face recognition method.

There has been some outstanding work in video face aggregation [15 –19], aggregate multiple video frames to realize the representation of video faces on images or highly compact feature vectors. The above method has been successful to a certain extent. However, we argue that aggregation at the image level or feature vector level has certain limitations. The aggregation noise at the image level is too large, which affects the aggregation effect [20, 21]. The level of feature vector aggregation makes the difference of different images in each feature channel ignored [15 , 23]. Therefore, it may be a better solution to consider feature mapping and aggregation at the high feature level of each feature channel.

Based on the above analysis, this paper proposes a feature map aggregation network (FMAN) for video face recognition. The purpose of FMAN is to aggregate face images of multiple video frames at the feature map channel level. In detail, a quality perception module is used to obtain the quality score of each feature map channel, and then the quality score is used as a weight to adaptively aggregate the feature maps of multiple images into a more effective feature map in the same channel. In this way, the aggregated features of multiple images are consistent with the feature types of a single image. We measured the similarity between the image and the image set, and the image set and the image set (see Fig. 1). The results show that the aggregated vector is more beneficial for face representation. More convincingly, the evaluation results on three differently distributed face data sets show that the FMAN method is more effective than other aggregation methods.

In summary, the contributions of the paper are as follows:

A Feature Map Aggregation Network (FMAN) is proposed to aggregate a set of images into the same feature space as image-based face recognition. The aggregated features have richer face information and can be applied to video face recognition.

A quality-aware module is proposed to adaptively learn quality scores for different feature maps. Makes more parameters available for learning aggregation and better independence of aggregations in each feature channel.

We evaluate the proposed method on the challenging datasets YTF, IJB-A, and COX, and the results show a competitive performance.

The flow of remaining paper is as follows. Section 2 describes related work, including general face recognition and video face recognition. Section 3 details our motivation and proposed method. Section 4 reports the details of the experiments, including the dataset, implementation details, evaluation results, and ablation studies. Finally, Section 5 shows the conclusions of this paper.

2 Related work

2.1 Deep face recognition

In the past, face recognition developed slowly and mainly based on traditional methods [24]. Driven by the deep learning technology used by AlexNet in Imagenet competition in 2012 [25], DeepFace [2] achieved the best performance in the famous LFW benchmark in 2014 and reached the level close to human for the first time (DeepFace: 97.35 vs. human: 97.53). Since then, the focus of face recognition has turned to the method of deep learning, which has made great progress in recent years.

The representative network include AlexNet [25], VGGNet [12], GoogleNet [26], ResNet [27]. Currently, some mainstream face recognition algorithms are based on the above-mentioned network. Among them, ResNet solves the problem that some parameters cannot be effectively learned due to too deep network depth, and has achieved better performance. As for the research on loss function, Euclidean distance loss is the main direction of the initial research, such as triplet loss [28], center loss [7], and coco loss [8], etc. Among them, triplet loss guides the model to learn a complex feature map, so that the feature distance from triplet anchors to negative samples is larger than that of positive samples. Center loss provides a category center for each category, minimizes the distance between each sample in the min-batch and the corresponding category center, and achieves the purpose of reducing the intra-class distance. Coco loss introduces cosine distance, which increases computational complexity and gains improvements. After 2016, cosine-based loss has gained more attention. Such as A-Softmax [9] expresses softmax loss as cosine loss and introduces cosine spacing to further maximize the spacing in angular space. CosFace [10] uses the L2 normalization of the feature and the weight vector to remove the influence of the radial difference of the feature. A cosine margin is introduced to minimize intra-class variance and maximize inter-class variance. ArcFace [11] uses angular space instead of cosine space in Cosineface to maximize classification bounds. In addition to these mainstream methods, other special depth algorithms also performed well under specific constraints, such as lightweight network SqueezeNet [29] and MobileNet [30], which are more suitable for mobile devices due to fast inference speed. The performance of the above methods is shown in Table 1.

Table 1
Performance of deep face recognition methods

Method Years Benchmark Acc (%)

Triplet loss 2015 LFW 98.87

Center loss 2016 LFW 99.28

Coco loss 2017 LFW 99.86

A-Softmax 2017 LFW 99.42

CosFace 2018 LFW 99.73

ArcFace 2019 LFW 99.41

SqueezeNet 2016 ImageNet 57.5

MobileNet 2017 ImageNet 70.6

Method	Years	Benchmark	Acc (%)
Triplet loss	2015	LFW	98.87
Center loss	2016	LFW	99.28
Coco loss	2017	LFW	99.86
A-Softmax	2017	LFW	99.42
CosFace	2018	LFW	99.73
ArcFace	2019	LFW	99.41
SqueezeNet	2016	ImageNet	57.5
MobileNet	2017	ImageNet	70.6

2.2 Video face recognition

The existing video face recognition methods are mainly divided into two categories: video-based and image set based [20]. In the video-based method, each video is regarded as a set of ordered image sequences, and image information and video frame related information are used for identification at the same time [19 , 32]. This method solves the general video face recognition problem by modeling the relationship between frames. In the method based on image sets, each video is regarded as a group of images, and the relationship between image and video, video and video is regarded as a point-to-set and set-to-set relationship, respectively [16 , 34].

2.3 Motivation

Recently, due to the excellent performance of CNN in feature extraction, many studies have tried to aggregate the facial features of video frames extracted by CNN to obtain a unique representation of the video or image set. Most of these methods are aggregated compact face representations extracted from deep networks, and the most direct method is to use the largest or average pool [31 , 36]. However, the performance of the above method is limited by the number of samples and noise. Therefore, weighted aggregation of feature vectors by predicting quality scores has become a more effective method [38, 39]. For example, Ranjan [15] introduced face detection scores into quality discrimination, Yang [37] and Liu [18] predicted quality scores through input images, and then weighted each feature vector. Recently, Gong [23, 25] has achieved aggregation through multi-mode recurrent networks that pay more attention to high-quality feature vectors, and achieved better performance. The aggregation network proposed in this paper is similar to these works, but instead of aggregating at the feature vector level, we aggregate at the feature map level where the features are relatively independent and more parameters can be learned.

Fig. 2

Training and testing process based on DCNN. The upper part is the training process, and the lower part is the inference process.

3 Methodology

We summarized the general flow of the face verification system from model training to face verification (see Fig. 2). The face image is input into the deep network to obtain the vector representation of the specified dimension, and then the loss is obtained through the fully connected layer and the loss function. Finally, gradient descent and backpropagation techniques are used to update the network parameters, and the face representation model is obtained. In inference, the face image is represented by a vector through the model, and then the face is verified according to the distance measurement and threshold setting.

However, in more unconstrained real scenes, a single face image only contains part of the face information, such as partial face occlusion, side face, etc. In this case, even the best face representation model cannot obtain sufficient face features. Therefore, aggregating multiple face images has become a popular solution. An intuitive method is to average the feature vectors of a group of face images, which is called the average pool [31, 35]. However, the average pool can offset the noise only when the sample size is large enough. Therefore, [37] and [18] reduce the impact of noise in aggregation by learning the weights of each feature vector, and the aggregation method of learning quality scores can be expressed by the following formula:

$γ = \frac{1}{N} \sum_{i} w_{i} (μ + ɛ_{i}) = μ + \frac{1}{N} w_{i} ɛ_{i},$ (1) where, γ is the aggregated feature vector, w_i is the mass fraction, μ is the standard vector representation of the face without noise, ɛ_i is the corresponding face noise, and N is the number of aggregated face images.

However, low-quality face has a strong representation on some features, and the overall weighting of feature vectors will weaken the useful features of low-quality face images. Therefore, [23] performs feature aggregation by learning the weight of each feature vector component, thereby enhancing the facial feature representation and weakening the influence of noise. The aggregation method is expressed as follows:

$γ_{j}^{'} = \frac{1}{N} \sum_{i} w_{ij} (μ_{j} + ɛ_{ij}) = μ_{j} + \frac{1}{N} w_{ij} ɛ_{ij} .$ (2)

Fig. 3

Overall framework of the aggregation network.

The above methods have gained benefits, but we believe that the method of directly aggregating compact feature vectors obtains a compact face representation, but it is not conducive to the aggregation of face features. Feature maps have lower-level feature information than feature vectors and are more independent. Therefore, this paper proposes a feature map level aggregation method. Considering the independence of each channel, we introduce quality perception to aggregate the input image set of each channel separately. The aggregated feature map has the same structure and meaning as the feature map of a single image, due to the quality score is converted into a probability representation. With reference to image-based processing methods, after stacking appropriate convolutional layers, global convolution is used to obtain a face representation that incorporates multiple image features.

3.1 Overall framework

The network structure of FMAN is shown in Fig. 3. First, the deep convolutional neural network is used to obtain the multi-channel features of the input face image, and the output features are used as the input of the aggregation network; In the aggregation module, the multi-channel features corresponding to multiple faces are reconstructed; The aggregated multi-channel feature map has the same form and meaning as a single image. In addition, similar to image processing, the aggregated multi-channel feature map undergoes two convolutional layers and global convolution to obtain a more compact vector representation.

3.2 Feature map extraction module

FMAN’s feature map extraction module is a popular deep convolutional neural network, and this paper chose ResNet101 due to its outstanding performance [27]. The difference is that the feature output of FMAN is a multi-channel feature map. In addition, we reserve the conv1-conv5 part of ResNet101, and discard the average pool, 1000-d fc and softmax layers after conv5. Among them, the multi-channel feature map output by the conv5 part is used as the input of the aggregation module.

3.3 Aggregation module

Suppose T ={ I₁, I₂, …, I_N } is a group of face images to be aggregated, H (·) be the basic DCNN module of FMAN. For a single face image, its feature mapping is F ={ f₁, f₂, …, f_N }. Where, f_i is the three-dimensional data with the number of channels D and the size w × h. Each channel of f_i represents part of the features extracted by the basic DCNN module to distinguish faces. Therefore, we separately aggregate the features of each channel.

In order to aggregate the characteristics of each channel, the feature map set is first reconstructed according to the number of channels. Suppose F′ is the set of feature maps after reconstruction, then $F^{'} = {f_{1}^{'}, f_{2}^{'}, \dots, f_{D}^{'}}$ . Where, $f_{i}^{'}$ is the three-dimensional data with depth of N and size of w × h, and represents the feature map set of N aggregated images in channel i.

A feature map quality perception module is designed by us to perceive the quality score of the feature map. Since the quality score of the feature map depends on all the feature maps of the channel where the feature map is located, the quality perception module takes $f_{i}^{'}$ as the perception input, and the output is the quality score corresponding to N feature maps in $f_{i}^{'}$ . Let A (·) be the perception module, the specific design is described in section 3.5. Therefore, the aggregation method in the k channel is as follows:

Fig. 4

Quality perception module of feature maps.

$C_{K} = \sum_{i} (A (f_{k}^{'}) \times f_{ki}^{'})$ (3) where, C_k is the feature map obtained by aggregation in the k channel, $f_{k}^{'}$ is all the feature maps of the k channel, and $f_{ki}^{'}$ is the k-th feature map of the i channel. Therefore, the aggregated multi-channel feature map can be represented as F″ ={ C₁, C₂, …, C_D }.

Since the aggregated F″ has the same meaning and size as the feature map f_i of a single face image, some methods for image features are used to further process the feature representation of F″ to obtain a more compact feature representation. We add two convolutional layers after the aggregation layer to enhance the representation of the aggregation features, and use the Relu function at an appropriate location to increase the nonlinear expression. In addition, a fully convolutional layer and a fully connected layer replace a single fully connected layer in the last layer of the network in order to obtain a compact feature vector representation and a specified dimension output respectively.

3.4 Quality perception module of feature maps

The quality perception module reads a set of feature maps in the reconstructed single channel as the perception object, and generates the corresponding quality score for each feature map. We believe that the richness of local face information contained in a feature map should not only be obtained through the feature map itself, but also need to balance all the feature maps in the channel so that the feature map with less information is suppressed, while the feature map with more information gets more attention. To this end, we first designed a shallow neural network to obtain the mapping of the feature map in the mass space, and then obtained the corresponding probability score of all the mapping values through a softmax operation.

The quality score learning module we designed is shown in Fig. 4, including 3 convolutional layers, 3 densely connected layers, and a softmax layer. The input feature map first passes through two convolutional layers in parallel, and then the dimension of each channel is reduced to a set of feature values using the global convolutional layer. Finally, three fully connected layers are designed to reduce the eigenvalue set to one eigenvalue. As a result, the input feature map set is converted into a feature value set containing quality information. As a result, the input feature map set is converted into a feature value set containing quality information. In addition, our output feature values are converted into quality scores through softmax, so as to facilitate weighted aggregation in the channel. The specific formula is as follows: $S_{k} = \frac{\exp (q_{k})}{\exp (q_{i})}$ (4) where, S_k represents the quality score of the k-th feature map in the channel, and q_i represents the i-th feature value. Since there is ∑_kS_k = 1 for the generated quality scores {S_k}, {S_k} can be used as a weight for subsequent aggregation.

3.5 Network training

As an end-to-end network, there are two parameter learning modes that can be selected: Train the whole FMAN, and the parameters of each part are updated synchronously; Use the static face image data set to train the basic DCNN, use the pre-trained model to extract the feature map, and finally train the aggregation network separately based on the feature map.

This paper chooses the second mode for experimentation, and there are two main reasons:

The focus of this paper is not to extract facial features, but to aggregate different facial images to obtain more accurate facial representations.

It is not conducive to analyzing the performance of the aggregation module, when the DCNN module is trained together with the aggregation module.

Based on the above analysis, we first train the basic DCNN module. In order to obtain a more accurate feature representation, ResNet101 is selected as the main network architecture. Since the latest ArcFace directly maximizes the classification distance in angular space [11], making it more suitable for deep network structure and multi-class output than others. Therefore, ArcFace is used as the loss function of the network, and ArcFace can be expressed as:

$L = - \frac{1}{N} \sum_{i = 1}^{N} \log \frac{e^{s (\cos (θ_{y_{i}} + m))}}{e^{s (\cos (θ_{y_{i}} + m))} + \sum_{j = 1, j \neg = y_{i}}^{n} e^{scos (θ_{j})}},$ (5) where, N is the number of samples, m is the angular margin penalty, y_i is the category, n is the number of categories, θ is the Angle, and s is the characteristic scale.

For the training of the aggregation module, we chose the basic softmax loss function due to its shallow network structure. Softmax loss can be formulated as:

$L = - \frac{1}{N} \sum_{i = 1}^{N} \log (\frac{e^{z_{i}}}{\sum_{j} e^{z_{j}}}),$ (6) where, i represents the i-th sample, j represents the j-th category, N is the number of samples, Z_i represents the score of the category to which the i-th sample belongs, and L represents the loss value.

4 Experiments

In this section, we first introduces the datasets and protocols. Then, the baseline method and implementation details are described. Finally, the evaluation results of the FMAN network on three video face recognition datasets (YTF [40], IJB-A [41], and COX [42]) are reported.

4.1 Datasets and protocols

MS-Celeb-1M. The MS-Celeb-1M dataset is a data set composed of 1 million celebrities all over the world. It is mainly used for identification research. The training set collects the top 100,000 celebrities in the list, and then uses the search engine to provide about 100 images for each celebrity, and then generates 10 million network images. The dataset was released by Microsoft in 2016, and the main publishers are Lei Zhang, Jianfeng Gao, and so on [43].

CASIA-WebFace. CASIA-WebFace is a large-scale face data set, which is mainly used for identification and face recognition. It contains 10575 subjects and 494414 images. This data set uses a semi-automatic method to collect Internet face images and establish large-scale data sets. The data set was released in 2013 by Dong Yi of Vincencer and Zhen Lei of the Chinese Academy of Sciences [44].

YTF. The YouTube Face (YTF) dataset contains 3425 videos of 1595 people. The shortest and longest video lengths are 48 frames and 6,070 frames respectively, with an average of 181 frames per video. We tested against the 1:1 face verification protocol, and the average accuracy is obtained from the 5000 pairs of videos given by the protocol by using ten-fold cross-verification [40].

IJB-A. The IJB-A dataset is a large face dataset published by the National Institute of Standardization (NIST), which includes static face images and videos collected from the Internet. The dataset contains 1,845 objects, a total of 11,754 still images, and 55,026 video frames. Since the collected face data is unconstrained, IJB-A is more challenging and diverse than YTF [41].

COX. COX dataset is released by the visual information processing and learning group (VIPL) of the institute of computing technology of the Chinese academy of sciences. It contains static and video faces, aiming to study three typical situations of face recognition: video to static (V2S), static to video (S2V), and video to video (V2V). In addition, it covers 1000 identities, and each identity contains 1 high-quality still image and 3 videos taken from different angles [42].

We first train the basic DCNN module using a large static face dataset (MS-Celeb-1M) to obtain the face representation of a single face image. On this basis, the aggregation network is pre-trained with the CASIA-WebFace dataset to obtain the aggregation performance. Finally, in the evaluation of three widely used public datasets (YTF, IJB-A, and COX), the aggregation network is fine-tuned on the training set specified in the protocol and evaluated on the test set.

For the MS-Celeb-1M data set, the basic DCNN module is learned using conventional methods. For the CASIA-WebFace dataset, we reallocate the training data. Since the purpose of the aggregation network is to aggregate multiple face vectors to obtain stronger characteristics, this means that the training data of the module should be a group of face images instead of a single image.

In summary, the details of all datasets are shown in Table 2 below.

Table 2
The video face is the brief information of the dataset

Dataset Years Identity Images Videos

MS-Celeb-1M 2016 100000 10000000 -

CASIA-WebFace 2014 10575 494414 -

YTF 2015 1595 - 3425

IJB-A 2017 1845 11754 55026

COX 2015 1000 1000 3000

Dataset	Years	Identity	Images	Videos
MS-Celeb-1M	2016	100000	10000000	-
CASIA-WebFace	2014	10575	494414	-
YTF	2015	1595	-	3425
IJB-A	2017	1845	11754	55026
COX	2015	1000	1000	3000

4.2 Implementation details

Preprocessing. Firstly, we use MTCNN algorithm to detect the face of all the data sets used for training and testing [45]. According to the obtained face position information and five face key point information, the affine transformation is carried out according to the method used in [11]. Finally, the affine face image is normalized to 112×112 to reduce the background information. In addition, the images with multiple faces are further filtered according to certain rules to ensure that the detected faces can represent the corresponding subject identity.

Training. As described in Section 3.5, a separate training strategy requires training the basic DCNN module and the aggregation module separately. We first learn a face recognition model based on static images on the cleaned MS-Celeb-1M dataset and use the model parameters as the basic DCNN model parameters. To ensure a more accurate mapping, all parameter settings follow the settings of [11]. As for the pre-training of the aggregation network, we use 70% of the CASIA-WebFace dataset after reassignment described in Section 4.1 to learn and use the remaining 30% as the verification set for debugging hyperparameters, with training and verification datasets randomly allocated in proportion. Specifically, we use a small batch stochastic gradient (SGD) of 64 batches and use error backpropagation for training. As for the setting of the learning rate, 0.1 is used as the initial learning rate and decreases with the increase of epoch to 50% of the current learning rate. In addition, we used optimization techniques such as batch normalization and dropout for optimization. Finally, after 12 epochs of learning, we obtained the optimal model for testing. Fig. 5 visualizes the loss during training.

Fig. 5

The case of loss during training.

Testing. We use the trained FMAN network to obtain the feature vectors of the test set images and videos, and then measure the similarity of the two samples by the vector distance. In all evaluation experiments, the cosine distance is used to represent the similarity of two feature vectors.

Experimental environment. Our experiment is based on the python language and the TensorFlow framework in the ubuntu16.04 environment. The main hardware configuration includes 2.4GHz CPU and 64GB RAM, and on this basis, two NVIDIA 1080Ti are used to accelerate training.

4.3 Baseline methods

To show the advantages of aggregation at the feature map level more intuitively and comparatively, we design four baseline experiments to illustrate FMAN’s aggregation performance, including the basic model, average pool, quality score, and component aggregation. To ensure fairness, all baseline experiments use basic DCNN models that have been trained in FMAN to extract features. In the following paragraphs, we will specify how to obtain the distance between the two groups of face images in each baseline experiment. Let T¹ and T² be two sets of face images, where the number of images in T¹ and T² is N and M respectively, and F is the corresponding feature vector set extracted from T through basic DCNN, where $F^{1} = {f_{1}^{1}, f_{2}^{1}, \dots, f_{N}^{1}}$ and D (f_i, f_j) represents the distance between the feature vector f_i and f_j.

Basic model. The basic model uses the minimum distance of pairwise comparison of images as the distance between two sets of images, therefore, the distance between T¹ and T² is expressed as: $S_{T^{1}, T^{2}} = \min {D (f_{i}^{1}, f_{j}^{2})}$ (7)

Average pool. The average pool averages the feature vectors of a group of images into one feature vector. $S_{T^{1}, T^{2}} = D (\frac{1}{N} \sum_{i = 1}^{N} f_{i}^{1}, \frac{1}{M} \sum_{i = 1}^{M} f_{i}^{2})$ (8)

Quality score. Each image in the image group is given a quality score according to the face quality prediction, and the quality score is used as the weight of corresponding feature vectors to adaptively aggregate the corresponding feature vectors of a group of images into a feature vector. $S_{T^{1}, T^{2}} = D (\frac{1}{N} \sum_{i = 1}^{N} w_{i} f_{i}^{1}, \frac{1}{M} \sum_{i = 1}^{M} w_{i} f_{i}^{1})$ (9) where, w is the quality score.

Component aggregation (CA). Each component of the image feature vector is predicted to obtain a weight, and the feature vectors corresponding to the image group is aggregated into a vector according to the weights. $S_{T^{1}, T^{2}} = D (\frac{1}{N} \sum_{i = 1, j = 1}^{N, L} w_{ij} f_{ij}^{1}, \frac{1}{M} \sum_{i = 1, j = 1}^{M, L} w_{ij} f_{ij}^{2})$ (10) where, N is the feature vector dimension and w is the weight of the feature vector component.

4.4 Performance evaluation on YTF dataset

We experiment our method on the YTF dataset and compare with popular methods. The experimental results in Table 3 show that our method achieved an accuracy rate of 96.64%. Compared with the four baseline methods, FMAN has a higher accuracy rate, which shows that our aggregation method is more effective. Compared with other latest methods, FMAN is only lower than InsightFace with the highest accuracy, while InsightFace is an image-based face recognition algorithm, which can be embedded in FMAN as a basic DCNN module to obtain higher accuracy. Compared with the aggregation-based recognition algorithm, FMAN improves the best performing C-FAN by 0.15%. Fig. 6 shows the ROC curves for FMAN and the four baseline methods.

Table 3
Performance comparison of FMAN and other methods on YTF dataset (: face recognition method based on aggregated video or image set)

Method Accuracy Method Accuracy

VGG Face[12] 91.6±- DAN[18] 94.28±0.69

DeepID2+[32] 93.2±0.2 NAN[35]* 95.72±0.64

FaceNet[7] 95.52±0.06 C-FAN[21]* 96.50±0.90

SphereFace[9] 95.0±- MARN[20]* 96.44±0.99

InsightFace[11] 98.02±- TBE-CNN[13]* 94.96±0.31

Basic model 95.34±0.02 Average pool 95.30±0.02

Quality score 95.82±0.45 CA 96.11±0.25

FMAN(ours) 96.64±0.13

Method	Accuracy	Method	Accuracy
VGG Face[12]	91.6±-	DAN[18]*	94.28±0.69
DeepID2+[32]	93.2±0.2	NAN[35]*	95.72±0.64
FaceNet[7]	95.52±0.06	C-FAN[21]*	96.50±0.90
SphereFace[9]	95.0±-	MARN[20]*	96.44±0.99
InsightFace[11]	98.02±-	TBE-CNN[13]*	94.96±0.31
Basic model	95.34±0.02	Average pool	95.30±0.02
Quality score	95.82±0.45	CA	96.11±0.25
FMAN(ours)	96.64±0.13

Fig. 6

ROC curves of different baseline methods and our FMAN.

Table 4

1: 1 face verification performance evaluation on the IJB-A dataset

Method	FAR=0.001	FAR=0.01	FAR=0.1
LSFS[46]	0.514±0.060	0.733±0.034	0.895±0.013
Adaptation[47]	-	0.939±0.013	0.979±0.004
C-FAN[23]	0.916±0.001	0.940±0.008	-
QAN[18]	0.893±0.039	0.942±0.015	0.980±0.006
NAN[37]	0.881±0.011	0.941±0.008	0.978±0.003
Basic model	0.879±0.035	0.930±0.005	0.974±0.012
Average pool	0.877±0.005	0.929±0.025	0.974±0.013
Quality score	0.894±0.027	0.937±0.028	0.978±0.008
CA	0.902±0.008	0.940±0.016	0.980±0.004
FMAN(ours)	0.912±0.024	0.943±0.010	0.982±0.007

Table 5

1: N face recognition performance evaluation on the IJB-A dataset

Method	FPIR=0.01	FPIR=0.1	Rank-1	Rank-5	Rank-10
LSFS[46]	0.383±0.063	0.613±0.032	0.820±0.024	0.929±0.013	-
VGG Face[12]	0.461±0.077	0.670±0.031	0.913±0.011	-	0.981±0.005
DCNNfusion[48]	0.577±0.094	0.790±0.033	0.903±0.012	0.965±0.008	0.977±0.007
DAC[39]	0.852±0.048	0.931±0.012	0.970±0.011	-	-
NAN[37]	0.804±0.036	0.909±0.013	0.954±0.007	0.978±0.004	0.984±0.003
Basic model	0.832±0.026	0.904±0.014	0.941±0.015	0.967±0.009	0.971±0.011
Average pool	0.831±0.012	0.902±0.010	0.939±0.012	0.962±0.013	0.972±0.005
Quality score	0.842±0.011	0.915±0.008	0.958±0.008	0.974±0.012	0.981±0.010
CA	0.850±0.022	0.927±0.012	0.961±0.015	0.980±0.008	0.985±0.007
FMAN(ours)	0.854±0.022	0.931±0.017	0.968±0.004	0.982±0.006	0.988±0.004

Table 6

Rank-1 recognition rate under different V2S / S2V settings on the COX dataset

Method	V2S face recognition rate			S2V face recognition rate
	V1-S	V2-S	V3-S	S-V1	S-V2	S-V3
PSCL[42]	38.60±1.39	33.20±1.77	53.26±0.80	36.39±1.61	30.87±1.77	50.96+1.44
LERM[49]	45.71±2.05	42.80±1.86	58.37±3.31	49.07±1.53	44.16±0.94	63.83±1.58
VGG Face[12]	88.36±1.02	80.46±0.76	90.93±1.02	69.61±1.46	68.11±0.91	76.01±0.71
TBE-CNN[15]	93.57±0.65	93.69±0.51	98.96±0.17	88.24±0.40	87.86±0.85	95.74±0.67
Basic model	89.41±0.45	85.23±1.63	95.25±0.77	81.66±0.87	79.45±1.13	91.97±1.35
Average pool	88.36±1.03	85.35±0.99	94.89±0.63	81.69±0.91	79.32±1.33	92.24±1.28
Quality score	93.46±0.96	90.84±0.25	98.97±0.84	88.62±1.03	85.78±0.61	95.42±0.76
CA	94.11±0.88	92.02±0.38	99.08±0.34	89.36±1.20	86.01±0.74	96.01±0.62
FMAN(ours)	94.42±0.41	92.55±0.62	99.12±0.16	90.14±0.55	87.26±0.68	96.39±0.39

4.5 Performance evaluation on IJB-A dataset

We evaluate our method on the IJB-A dataset with 1:1 and 1:N search protocols, respectively. For the 1:1 face verification task, Table 4 shows the relationship between true acceptance rate (TAR) and false positive rate (FAR). For the 1:N face recognition task, Table 5 shows the relationship between true positive recognition rate (TPAR) and false positive recognition rate (FPAR) as well as the accuracy of Rank-N. Additionally, we report the numerical results of the proposed FMAN method and baseline method, and compare them with other methods.

According to the experimental data, for 1:1 face verification, FMAN achieved the best performance at FAR=0.1 and FAR=0.01. For 1:N face recognition, FMAN has better or comparable performance compared with other methods under most indexes, only slightly lower than DAC under the Rank-1 index. In addition, compared with the evaluation results on the YTF dataset, FMAN has a more obvious performance improvement on IJB-A with greater face changes (YTF: 1.36% vs IJB-A: 3.75% (FAR = 0.001)). Therefore, FMAN is more suitable for video face tasks with richer changes such as face poses.

Table 7
Rank-1 recognition rate under different V2V settings on the COX face dataset

Method V2-V1 V3-V1 V3-V2 V1-V2 V1-V3 V2-V3

PSCL[42] 57.70±1.40 73.17±1.44 67.70±1.70 62.77±1.02 78.26±0.97 68.91±2.28

LERM[49] 65.94±1.97 78.24±1.32 70.67±1.88 64.44±1.55 80.53±1.36 72.96±1.99

VGG Face[12] 94.51±0.47 95.34±0.32 96.39±0.42 93.39±0.56 96.10±0.27 96.60±0.52

TBE-CNN[17] 98.07±0.32 98.16±0.23 97.93±0.20 97.20±0.26 99.30±0.16 99.33±0.19

Basic model 95.21±0.89 95.69±0.11 96.42±0.12 94.53±0.56 96.73±0.17 96.81±0.12

Average pool 95.08±0.74 96.03±0.13 96.25±0.20 94.72±1.05 96.70±0.13 96.76±0.17

Quality score 98.32±0.32 98.78±0.05 98.41±0.31 97.88±0.09 99.23±0.10 98.16±0.08

CA 99.01±0.03 98.86±0.10 98.63±0.08 98.04±0.22 99.39±0.07 99.33±0.04

FMAN(ours) 99.15±0.21 99.03±0.09 98.88±0.06 98.16±0.14 99.48±0.04 99.41±0.10

4.6 Performance evaluation on COX dataset

We follow the COX dataset protocol to evaluate the performance of FMAN. Table 6 and Table 7 list the Rank-1 recognition rate of FMAN and other advanced algorithms. In Table 6, Vi-S(S-Vi) indicates that the i-th video set is used as a probe (gallery) and the static face set S is used as a gallery (probe). Similarly, Vi-Vj in Table 7 represents independent experiments conducted with the i-th video set as a probe and the j-th video set as a gallery.

It can be seen from Table 6 and Table 7 that for different experimental settings, V2S and S2V recognition tasks are more difficult than V2V, which shows that the difference between the image and video has a certain interference to the recognition task, which affects the performance of FMAN in V-S and S-V tasks. As for the performance of FMAN, not only did FMAN achieve better performance than the baseline method in all experimental Settings, but compared with other methods, FMAN performed better than the comparison algorithm in the other four V2S/S2V experimental Settings except V2-S and S-V2, as well as the six V2V experimental Settings with the same distribution.

5 Conclusions

To better realize unconstrained video face recognition, a feature map aggregation network (FMAN) is proposed. FMAN can adaptively aggregate the feature maps of each feature channel in the image group. Compared with the multi-channel feature map in a single image, it has the same form and meaning, but contains more comprehensive information. By reducing the dimension of each channel after aggregation, multiple images can be represented as compact feature vectors. The proposed method achieves competitive performance on YTF, IJB-A, and COX datasets, confirming the effectiveness of the method. The ablation experiment also confirms that the more independent features and the larger the parameters are, the aggregation at the feature map level is more conducive to the aggregation of face features.

Furthermore, since aggregated feature vectors and image-based feature vectors share the same feature space, the proposed FMAN can be uniformly applied to the tasks of image, image set, and video face representation. Our future work will try to use a more precise attention mechanism at the feature map level to improve the aggregation performance.

Footnotes

Acknowledgements

This work was supported by Nondestructive Detection and Monitoring Technology for High Speed Transportation Facilities, Key Laboratory of Ministry of Industry and Information Technology, and the Fundamental Research Funds for the Central Universities (NO.NJ2022016), Natural Science Research of Jiangsu Higher Education Institutions of China (Grants No. 22KJD140004).

References

Zhang Geguang

et al., Generating adversarial examples for DNNusing pooling layers, Journal of Intelligent and Fuzzy Systems 37(12) (2019), 1–6.

Wang

, Wang

, Yu

et al., Age-Invariant Face Recognitionby Multi-Feature Fusionand Decomposition with Selfattention, ACM Transactions on Multimedia Computing, Communications, andApplications (TOMM) 18(1s) (2022), 1–18.

Wen

, Zhang

, Li

et al., Domain Generalization via Shuffled Style Assembly for Face Anti-Spoofing, Computer Vision and Pattern Recognition (CVPR) (2022), 4123–4133.

, Shen

, Chen

et al., Rethinking FeatureDiscrimination and Polymerization for Large-scale Recognition, IEEE Transactions on Instrumentation and Measurement 71(2022), 1–14.

Kokila

, Gomathi

A unique approach of person reidentification using auto track regression framework, Journal of Intelligent & Fuzzy Systems (2022), 1–18.

Baffour

, Qin

, Zhu

et al., Facial expression recognition via coarse-grained and fine-grained feature representation, Journal of Intelligent & Fuzzy Systems (2022), 1–13.

Wen

, Zhang

, Li

A Discriminative Feature Learning Approach for Deep Face Recognition, European Conference on Computer Vision (ECCV) (2016), 499–515.

Liu

, Li

, Wang

Rethinking Feature Discrimination and Polymerization for Large-scale Recognition, CoRR, (2017), 1710.00870.

Liu

, Wen

, Yu

et al., SphereFace: Deep Hypersphere Embedding for Face Recognition, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017), 6738–6746..

10.

Wang

, Cheng

, Liu

et al., Additive Margin Softmax forFace Verification, IEEE Signal Processing Letters, (2018), 1–1.

11.

Deng

, Guo

, Xue

et al., ArcFace: Additive Angular Margin Loss for Deep Face Recognition, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019), 4690–4699.

12.

Simonyan

, Zisserman

Very Deep Convolutional Networks for Large-Scale Image Recognition, International Conference on Learning Representations (ICLR), 2015.

13.

Zhao

, Shi

, Tong

G-FAN: Graph-Based Feature Aggregation Network for Video Face Recognition, InternationalConference on Pattern Recognition (ICPR) (2021), 1672–1678.

14.

, Huang

et al., Learning features fromcovariance matrix of gabor wavelet for face recognition underadverse conditions, Pattern Recognition 119 (2021), 108085.

15.

Ranjan

, Bansal

, Xu

et al., Crystal Loss and Quality Pooling for Unconstrained Face Verification and Recognition, CoRR, (2018), 1804.01159.

16.

Zhao

and Zhai

, Ensemble of multiresolution probabilisticneural network classifiers with fuzzy integral for face recognition, Journal of Intelligent & Fuzzy Systems: Applications inEngineering and Technology 31(1) (2016), 405–414.

17.

Liu

, Tan

Eqface:Asimple explicit quality network for face recognition, IEEE Conference on Computer Vision and Pattern Recognition(CVPR) (2021), 1482–1490.

18.

Liu

, Yan

, Ouyang

Quality Aware Network or Set to Set Recognition, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 5216–5225.

19.

Rao

, Lu

, Zhou

Attention-Aware Deep Reinforcement Learning for Video Face Recognition, International Conference on ComputerVision (ICCV) (2017), 3951–3960.

20.

Rao

, Ji

, Lu

et al., Learning Discriminative Aggregation Network for Video-Based Face Recognition, IEEE International Conference on Computer Vision (ICCV) (2017), 3801–3810.

21.

, Zhang

, Wu

et al., Dynamic Scene Video Deblurring UsingRobust Incremental Weighted Fourier Aggregatio, IEEE SignalProcessing Letters 28 (2021), 1565–1569.

22.

Gong

, Shi

, Jain

A.K.

Low Quality Video Face Recognition: Multi-Mode Aggregation Recurrent Network (MARN), IEEE International Conference on Computer Vision (ICCV), (2019), 1027–1035.

23.

Wang

, Wang

, Deng

et al., Learning multigranularity temporal characteristics for face anti-spoofing, IEEE Transactions on Information Forensics and Security, 2022.

24.

Jin

, Raun

, Wang

et al., Expression robust three dimensional face recognition based on gaussian filter and dualtree complex wavelet transform, Journal of Intelligent & fuzzy systems: Applications in Engineering and Technology, 2014.

25.

Krizhevsky

, Sutskever

and Hinton

, ImageNet Classification withDeep Convolutional Neural Networks, Advances in NeuralInformation Processing Systems 25(3) (2012), 1106–1114.

26.

Szegedy

, Liu

, Jia

Going deeper with convolutions, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2015), 1–9.

27.

, Zhang

, Ren

Deep Residual Learning for Image Recognition, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 770–778.

28.

Florian

, Dmitry

, James

FaceNet: A Unified Embedding for Face Recognition and Clustering, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015), 815–823.

29.

Iandola

F.N.

, Han

, Moskewicz

M.W.

SqueezeNet: AlexNetlevel accuracy with 50x fewer parameters and <0.5MB model size, CoRR, (2016), 1602.07360.

30.

Howard

A.G.

, Zhu

, Chen

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, CoRR, (2017), 1704.04861.

31.

Ding

and Tao

, Trunk-Branch Ensemble Convolutional NeuralNetworks for Video-based Face Recognition, IEEE Transactions onPattern Analysis & Machine Intelligence 40(4) (2018), 1002–1014.

32.

Sun

, Wang

, Tang

Deeply learned face representations are sparse, selective, and robust, IEEE Conference on Computer Vision and Pattern Recognition(CVPR) (2015), 2892–2900.

33.

Wang

, Shan

, Chen

et al., Manifold-Manifold Distance andIts Application to Face Recognition With Image Sets, IEEETransactions on Image Processing 21(10) (2012), 4466–4479.

34.

Cevikalp

, Triggs

, Cevikalp

, Triggs

IEEE Conference on Computer Vision and Pattern Recognition(CVPR), (2010), 2567–2573.

35.

Chen

, Ranjan

and Sankaranarayanan

, UnconstrainedStill/Video-Based Face Verification with Deep Convolutional NeuralNetworks, Networks, International Journal of Computer Vision (IJCV), 126 (2) (2018), 272–291.

36.

Roy

C.A.

, Lin

, Maji

et al., One-to-many face recognition with bilinear CNNs, IEEE Winter Conference on Applications of Computer Vision (WACV), (2016), 1–9.

37.

Yang

, Ren

, Zhang

Neural Aggregation Network forVideo Face Recognition, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), 5216–5225.

38.

Sankaran

, Tulyakov

, Setlur

Metadata-Based Feature Aggregation Network for Face Recognition, International Conference on Biometrics (ICB) (2018), 118–123.

39.

Liu

, Kumar

, Yang

Dependency-Aware Attention Control for Unconstrained Face Recognition with Image Sets, European Conference on Computer Vision (ECCV) (2018), 573–590.

40.

Liu

, Deng

, Bai

et al., Targeting Ultimate Accuracy: Face Recognition via Deep Embedding, CoRR, 2015.

41.

Whitelam

, Taborsky

, Blanton

IARPA Janus Benchmark-B Face Dataset, IEEE Conference on Computer Vision and Pattern Recognition(CVPR) (2017), 592–600.

42.

Huang

, Shan

, Wang

et al., A Benchmark and ComparativeStudy of Video-Based Face Recognition on COX Face Database, IEEE Transactions on Image Processing 24(12) (2015), 5967–5981.

43.

Guo

, Zhang

, Hu

et al., MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition, European Conference on Computer Vision (ECCV) (2016), 87–102.

44.

Dong

, Zhen

, Liao

Learning Face Representation from Scratch, CoRR (2014), 1411.7923.

45.

Zhang

, Zhang

and Li

, Joint Face Detection and AlignmentUsing Multitask Cascaded Convolutional Networks, IEEE SignalProcessing Letters 23(10) (2016), 1499–1503.

46.

Wang

, Otto

, Jain

A.K.

Face Search at Scale: 80 Million Gallery, CoRR (2015), 1507.07242.

47.

Crosswhite

, Byrne

and Stauffer

, Template adaptation forface verification and identification, Image Vis Comput, 79 (2018), 35–48.

48.

Chen

, Patel

, Chellappa

Unconstrained face verification using deep CNN features, IEEE Winter Conference on Applications of Computer Vision (WACV) (2016), 1–9.

49.

Huang

, Wang

, Shan

Learning Euclidean-to- Riemannian Metric for Point-to-Set Classification, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014), 1677–1684.

Method	V2-V1	V3-V1	V3-V2	V1-V2	V1-V3	V2-V3
PSCL[42]	57.70±1.40	73.17±1.44	67.70±1.70	62.77±1.02	78.26±0.97	68.91±2.28
LERM[49]	65.94±1.97	78.24±1.32	70.67±1.88	64.44±1.55	80.53±1.36	72.96±1.99
VGG Face[12]	94.51±0.47	95.34±0.32	96.39±0.42	93.39±0.56	96.10±0.27	96.60±0.52
TBE-CNN[17]	98.07±0.32	98.16±0.23	97.93±0.20	97.20±0.26	99.30±0.16	99.33±0.19
Basic model	95.21±0.89	95.69±0.11	96.42±0.12	94.53±0.56	96.73±0.17	96.81±0.12
Average pool	95.08±0.74	96.03±0.13	96.25±0.20	94.72±1.05	96.70±0.13	96.76±0.17
Quality score	98.32±0.32	98.78±0.05	98.41±0.31	97.88±0.09	99.23±0.10	98.16±0.08
CA	99.01±0.03	98.86±0.10	98.63±0.08	98.04±0.22	99.39±0.07	99.33±0.04
FMAN(ours)	99.15±0.21	99.03±0.09	98.88±0.06	98.16±0.14	99.48±0.04	99.41±0.10

A feature map aggregation network for unconstrained video face recognition

Abstract

Keywords

1 Introduction

2.1 Deep face recognition

Table 1 Performance of deep face recognition methods Method Years Benchmark Acc (%) Triplet loss 2015 LFW 98.87 Center loss 2016 LFW 99.28 Coco loss 2017 LFW 99.86 A-Softmax 2017 LFW 99.42 CosFace 2018 LFW 99.73 ArcFace 2019 LFW 99.41 SqueezeNet 2016 ImageNet 57.5 MobileNet 2017 ImageNet 70.6

2.3 Motivation

3.2 Feature map extraction module

3.3 Aggregation module

4.1 Datasets and protocols

Table 2 The video face is the brief information of the dataset Dataset Years Identity Images Videos MS-Celeb-1M 2016 100000 10000000 - CASIA-WebFace 2014 10575 494414 - YTF 2015 1595 - 3425 IJB-A 2017 1845 11754 55026 COX 2015 1000 1000 3000

5 Conclusions

Footnotes

Acknowledgements

References

Table 2
The video face is the brief information of the dataset

Dataset Years Identity Images Videos

MS-Celeb-1M 2016 100000 10000000 -

CASIA-WebFace 2014 10575 494414 -

YTF 2015 1595 - 3425

IJB-A 2017 1845 11754 55026

COX 2015 1000 1000 3000