Regional parallel structure based CNN for thermal infrared face identification

Abstract

The convolutional neural network, based on multi-scale features, is introduced to thermal infrared face identification in this paper. A novel CNN structure is proposed based on characteristics of thermal infrared faces. To enhance and extract inconspicuous thermal infrared facial features for identification, convoluted edges are taken as the initial features. A regional parallel structured CNN algorithm (RPS net) is proposed to obtain multi-scale features based on edge information. Extensive experiments are conducted and analyzed, including statistical test with various classifiers, feature vector property, accuracies of each class and robustness against various noise. The experimental result indicates that RPS net overtakes algorithms based on traditional features (HoG, Fisherface and LBP) and some CNN algorithms (Alex net, VGG net, DeepID net and TFR net), with high quality features. Therefore, RPS net is effective and robust for thermal infrared face identification.

Keywords

Thermal infrared face identification convolutional neural network regional parallel structure

1. Introduction

Face identification is one of the most meaningful issues in pattern recognition field. Due to the rapid development of thermal infrared sensors, face identification in thermal infrared images has been widely used, especially in some extreme environments where visible face image is invalid for identification. For example, full-time system, in which visible face recognition could not achieve high accuracy at night, could be greatly improved by thermal infrared imaging and recognition. Another example is that skilled makeup technique causes great change in human appearance, which increases the difficulty of visible face recognition. However, this has little effect on thermal infrared face recognition. We believe that thermal infrared face imaging and recognition could provide supplementary information for visible image systems, especially in some extreme environments. Multi-sensor imaging and recognition, such as thermal infrared and visible images based systems, will be the trend of intelligent applications.

Figure 1.

Face images in different imaging sensors. (a) shows visible color face images. (b) shows visible grayscale face images. (c) shows corresponding thermal infrared face images. Best viewed in color.

Figure 2.

Edge maps of visible face and thermal infrared face by Roberts, Sobel and Prewitt edge detectors.

Different from visible images, thermal infrared faces involve more temperature information than edge details, as shown in Fig. 1. To show the information difference of visible face and thermal infrared face, various edge detections, by Roberts [1], Sobel [2] and Prewitt methods [3], are deployed with results shown in Fig. 2. In general, visible faces have rich information in facial organs, such as eyes and nose. The variation of thermal infrared face locates in hair and mouse regions, where temperature changes. Moreover, temperature distribution in facial area changes over time, environment, emotion and so on [4]. In other words, the thermal infrared face images could be very different from each other, even though they are taken from the same person. Compared to faces in visible and near infrared images, thermal infrared faces contain worse inter-class difference and intra-class consistency. This increases the difficulty of distinguishing thermal infrared faces from each other. Usually, identification consists of two procedures: feature extraction and face classification. Extracting features with enough discrimination is the key to solving thermal infrared face identification problem.

The rest of this paper contains four main sections. In Section 2, some related works will be presented and analyzed, including traditional feature based algorithms and some popular CNN algorithms. In Section 3, the proposed RPS net is given in detail, including the regional parallel structure and the convolution parameters. In Section 4, comparison with other algorithms is conducted on the thermal infrared face dataset. The test accuracies, robustness against noise, and the property of feature vectors are compared and analyzed. In Section 5, the characteristics of the proposed RPS net is concluded. Multi-scale features, extracted through regional parallel structure, turn out to be effective for thermal infrared face identification.

Figure 3.

The convolutional neural networks with regional parallel structure. Best viewed in color.

2. Related work

Some enlightening algorithms for face identification [5, 6, 7, 8, 9, 10] are proposed in recent years. In [5], lattice is computed as the feature, and k-nearest neighbor (KNN) algorithm [11] is taken as classifier for infrared face identification. In [6], Zernike moments (ZMs) and Hermite kernels (HKs) are performed to generate local and global features for recognition. By using principal component analysis followed by linear discriminant analysis as classifier, the algorithm has good accuracy and robustness. Celebrity-1000 [7], containing 1000 celebrities sampled from YouTuBe and Youku, is set up as a large-scale video database in an unconstrained environment. The revised multitask joint sparse representation (MTJSR) [7] achieves high accuracy on the Celebrity-1000 dataset. Actually, most face identification algorithms base on visible images and near infrared images, in which there are rich edge and texture information to generate stable patterns for identification. Therefore, facial feature based algorithms are still effective for these images. Recently, higher accuracy of identification has been achieved by deep learning algorithms. It is because these images usually contain clear facial features, such as the region of eyes, nose and mouth, which could provide stable patterns for identification.

The convolutional neural network (CNN) has been fast developed in recent 10 years since Hinton’swork [12]. Actually, CNN has been introduced to various areas, such as semantic segmentation [13, 14], object recognition [15, 16], damage detection [17, 18, 19] and transportation problems [20, 21], with good real-time performance and robustness [22, 23]. Lots of efficient face verification and identification algorithms are proposed [24, 25, 26, 27, 28, 29, 30, 31, 32, 33]. DeepFace [24] and DeepID [26] are two of the state-of-the-art algorithms for face identification and verification. DeepFace system [24] contains two steps for identity verification of visible face images. The first step is face frontalization based on 3D models and fiducial-points, which provides the front face with more features for classification. The second step is a CNN classifier, including three locally connected layers which contain the local facial features. Actually, face frontalization reduces the difficulty of feature extraction by localizing different regions of face. However, the lack of regional textures in thermal infrared face images would affect the accuracy of frontalization, and introduce extra noise while training the network. DeepID net [26] is an efficient CNN for face verification with serial structure, which has achieved high accuracy on LFW dataset. Up to now, the DeepID team has proposed the third generation [32] with higher accuracy, which combines some basic elements from VGG net [34] and GoogLeNet [35]. Thermal face recognition algorithm [33] is proposed by adopting serial structure CNN for thermal infrared recognition, according to various head rotation, expression and illumination. For face identification, most algorithms base on manually designed features, such as HoG [36], LBP [37], Fisherface (LDA) [38] and Eigenface (PCA) [39, 40]. Followed with classifier, such as SVM [41], reasonably designed features could effectively present the discrimination among faces from different persons. However, thermal infrared face images could not be accurately distinguished by these features, as discussed in Section 3. This is because variation of facial temperature could not be fully described as the difference among different identities. In other words, the resolution of manually designed features is not enough for thermal infrared face images.

To achieve high accuracy of face identification in thermal infrared images, we propose a convolutional neural network with regional parallel structure (RPS net). Different from serial structure based CNN, such as DeepID [26], RPS net contains a regional parallel structure as one of the main modules to extract features in different region sizes. In this way, multi-scale features are generated to learn the difference among face identities. The experimental result indicates that RPS net achieves high identification accuracy with good robustness for thermal infrared face images, due to the extracted high quality feature vector.

3. Algorithm

In general, pixel intensities in thermal infrared images describe regional distribution of temperature on face. There are two kinds of information, absolute intensities and relative pattern. Actually, absolute intensities contribute a lot to the face detection from background, while relative pattern is the key to distinguishing faces from each other in foreground. As shown in Figs 1 and 2, some differences between thermal infrared faces and visible faces could be found out. Firstly, regional textures and details are not clear enough, especially the area of eyes and mouth. Secon- dly, no features are stable enough for facial expression. In other words, intensity distribution in some regions, such as the nose area in Fig. 1(c), might change due to time, environment, emotion and other temperature-related reasons. This leads to an unstable relative relationship between different facial areas. In general, the features for identification in thermal infrared face images are not as effective as visible face images. To extract features representing the difference, multi-scale information is introduced by using a novel CNN with regional parallel structure.

Figure 4.

Some examples of the learned convolution kernels for initial edge feature extraction.

The structure of the proposed RPS net is shown in Fig. 3, which contains three cascaded parts, initial edge feature extraction, multi-scale feature extraction and feature vector classification. Firstly, initial edge feature extraction module consists a cascaded convolution and max-pooling layer, in Section 3.1. Secondly, convolutions with different kernels are design as three parallel channels to generate multi-scale features, in Section 3.2. Finally, the convolutional feature maps are transformed into feature vector by fully connected layer and softmax loss, as presented in Section 3.3 and analyzed in Section 4.2.2 with experiments. These make up an end-to-end thermal infrared face identification algorithm.

3.1 Initial edge feature extraction

Regardless of sensor types and imaging principles, most images are strongly spatial correlated. In thermal infrared face images, the macroscopical pattern of facial region could not be extracted due to low contrast and noise. To find features which could describe the difference among facial identities, initial feature maps in the first layer are generated through 128 convolution kernels in shape 2 $\times$ 2. Actually, this convolution is not widely used in CNN algorithms for visible image processing due to its small receptive field. This is because visible images contain rich texture and detail information, which could be extracted by convolution kernels with large receptive field, such as 5 $\times$ 5 and 7 $\times$ 7. For thermal infrared images, there are not enough texture and detail information. This means the patterns based on texture and detail could not generate identity variance for classification. To enhance regional edge details, we introduce 2 $\times$ 2 convolution kernels as initial feature extractor. In this way, the pattern of thermal infrared faces could be learned by applying convolutions on edge-like feature maps instead of original image. The reason to choose this size is that the inner product $c_{ij}$ between convolutional weights $w$ and original intensities $x_{ij}$ , as given in Eq. (1), extracts edge features inside the nearest neighborhood.

$\displaystyle c_{ij}=\sum\begin{bmatrix}{w_{11}}&{w_{12}}\\ {w_{21}}&{w_{22}}\\ \end{bmatrix}\cdot\begin{bmatrix}{x_{i,j}}&{x_{i,j+1}}\\ {x_{i+1,j}}&{x_{i+1,j+1}}\\ \end{bmatrix}$ (1)

Figure 5.

(a) are the initial edge feature maps after convolution, ReLU and max-pooling. (b)–(d) are some response examples to specific convolution kernels shown in 3D from (a). Best viewed in color.

Figure 6.

Extracted feature maps of the three channels in regional parallel layer. Channel (a) is the features in neighbor region.Channel (b-c) are the features in local region. Feature maps in Channel (b) are convoluted in shape 3 $\times$ 3 and ReLU. Feature maps in Channel (c) are convoluted in shape 5 $\times$ 5 and ReLU. Best viewed in color.

In this way, regional edge information which is very important for thermal infrared face identification, could be maintained in the initial feature maps. The convolution operation in this layer is similar to edge detection filtering with Roberts cross operator [1], as shown below.

$\begin{bmatrix}{+{\rm 1}}&{{\rm 0}}\\ {{\rm 0}}&{{\rm-1}}\\ \end{bmatrix}{\rm and}\begin{bmatrix}{{\rm 0}}&{+{\rm 1}}\\ {{\rm-1}}&{{\rm 0}}\\ \end{bmatrix}$

Actually, Roberts cross operator is an extreme form of convolution in shape 2 $\times$ 2, by filling 0 and 1 in weights $w$ . Therefore, any convolution in shape 2 $\times$ 2 could be considered as a kind of edge detection. In this paper, feature maps with various edge information could be acquired due to the adjustable weights $w$ . Usually, edge information is the key to thermal infrared image processing, such as pedestrian segmentation [42, 43]. For thermal infrared face identification, various edge representations could help distinguish faces from each other. Some of the learned weights in this layer are shown in Fig. 4.

The ReLU [44] nonlinearity is added after each convolution in our net. In this way, the convolution operation could be expressed as Eq. (2).

$y^{k}=\max(0,c^{k}+b^{k})$ (2)

$c^{k}$ and $y^{k}$ are the $k$ th convoluted map and output map. $b^{k}$ is the corresponding bias after convolution with kernel $w^{k}$ . After nonlinearity activation by ReLU, a max-pooling operation is applied to down-sample the initial feature maps with mask in shape 2 $\times$ 2 and stride 2. The down-sampled feature maps, as shown in Fig. 5, are taken as the input of followed parallel convolutions.

Figure 7.

The final feature vectors and the cosines to mean vectors. The feature vector from the same person are close to each other with large cosine value to the mean vector. Best viewed in color.

In Fig. 5, (a) is the generated feature maps through all 128 convolution kernels in the current layer. (b, c, d) are the 2nd, 18th and 37th channels from feature map (a). (b) represents the response for face contour from background. (c) describes the distribution of facial objects organs, such as location of the eyes, nose and mouse. In (d), some key points for identification are emphasized. These feature maps with rich edge information are individually shown as 3-D meshes for better visual effect, as shown in Fig. 5(b)–(d).

3.2 The regional parallel layer

After the initial edge feature maps are extracted, followed operations are designed to find the pattern of edges in neighborhood. Different from the structure of early proposed CNN algorithms, such as Alex net [44] and VGG net [34], regional parallel convolutions with different kernel sizes and strides are designed as different channels in one layer. As shown in Fig. 3, the regional parallel layer is combined by the initial feature maps and two feature maps extracted through different convolution kernels. In this way, patterns of initial edge features could be learned in different scales. The multi-scale features could provide enough information for thermal infrared face identification. The proposed three channels for feature extraction could be considered as two types, features in neighbor region and features in local region. Details of the structure and parameters are given below.

3.2.1 The features in neighbor region

The key to thermal infrared face identification is finding the hidden relationship of edge features or gradients. In particular, edge itself is a kind of features for identification. As the source of edge pattern learning, the initial edge features could be considered as the minimum-scale pattern. Therefore, the initial feature maps, as shown in Channel (a) from Fig. 6, are taken as features in neighbor region to generate the final feature vector. Actually, some similar strategies are adopted in many CNN algorithms, such as DeepID [26] net. In this way, information in neighbor region could be included in the final feature vector.

3.2.2 The features in local region

To obtain information in middle and large receptive field, convolutions in local region are designed as two paralleled channels, as shown in Fig. 3. In this paper, a convolution with 64 kernels in shape 3 $\times$ 3 and stride 1, is added as an independent channel to learn middle scale pattern of the initial edge features. A convolution with 64 kernels in shape 5 $\times$ 5 and stride 2, is designed as another parallel channel to extract the pattern in a relatively large scale.

As shown in Fig. 6, the feature maps in Channel (b) and (c) are responses of edge variety in middle scale and large scale. With the feature maps in neighbor region, three channels are taken as the regional parallel layer, which could extract multi-scale features for thermal infrared faces.

3.3 The feature vector of thermal infrared face

Multi-scale feature maps from regional parallel layer describe the patterns in different region sizes, which contain valuable information for the final identification of thermal infrared faces. Actually, there are 128 $\times$ 32 $\times$ 32 neuron nodes in neighbor region features, 64 $\times$ 30 $\times$ 30 and 64 $\times$ 14 $\times$ 14 neuron nodes in local region features. This is massive data with much redundancy. Therefore, it is unnecessary and unreasonable to take multi-scale features as the directly inputs of identification classifier. Actually, a fully connected layer with 512 neuron nodes activated through ReLU, is added as the feature vector of thermal infrared face. The final feature vectors with different original thermal infrared faces are shown in Fig. 7. Cosine values of the angles between sample vectors and mean vectors of related classes are calculated and also shown in Fig. 7. This indicates that feature vectors from the same class have high similarity. In other words, RPS net features have good intra-class consistency. The performance and characteristics of the final feature vector are tested and compared in Section 4.2.2. In order to limit over-fitting, a dropout layer with parameter 50% is adopted following the fully connected layer and ReLU.

Dropout method [45] is to invalidate some components of feature vector randomly while training model. Due to randomness of dropout, each two nodes could be possibly not working at the same time. This makes the followed layer less affected by the correlation of nodes in current layer. Therefore, the components of feature vector could be trained with weak correlation for classification. In this way, classifier will be more sensitive to components themselves rather than the collective effect of some components. Experiments [45] indicate that dropout method is effective for limiting over-fitting.

3.4 Discussion

The CNN algorithm in this paper is quite different from the networks for object or face recognition, such as DeepID net [26], VGG net [34] and Alex net [44]. Firstly, kernel size of initial convolution is much smaller. Despite the influence of image sizes, kernel size of initial convolution layer decides the basic scale of local region. Large kernel size, such as 4 $\times$ 4 for DeepID net or 11 $\times$ 11 for Alex net, is effective for convolving texture pattern in a large local region of visible images. For thermal infrared face identification, regional patterns based on edge and gradient are the crucial information due to the lack of clear textures and details. Hence, 2 $\times$ 2 is taken as the kernel size of initial convolution layer, to generate various edges as the basis of pattern learning in different scales. Secondly, regional parallel structure is designed as parallel layer following initial convolution layer. Multi-scale features extracted through parallel layer could be considered as the edge patterns with different region sizes, which provide rich information for the identification of thermal infrared faces.

Figure 8.

Some examples from our thermal infrared face dataset with various movement and expression.

Due to regional parallel structure, information from initial edge features could be reserved and trained as the patterns in different region sizes. In this way, multi-scale features could be extracted to generate the final feature vector for classification. Different from serial structure based CNNs, RPS net has only four convolution layers and fully connected layers cascaded, but involves more edge features and edge patterns. These are crucial for thermal infrared face identification. Therefore, the feature vector extracted through RPS net has better inter-class difference and intra-class consistency, as analyzed in Section 4.2.2.

4. Experiments

There are some thermal infrared face image data- bases publicly available for different applications. IRIS (Imaging, Robotics and Intelligent System) Thermal/Visible Face Database [46] and Terravic Facial IR Database [47] have 30 and 20 subjects with various expressions and movements, respectively. These publicly available thermal infrared face datasets are not suitable for CNN based experiments for the small number of samples and subjects. The thermal infrared face images used in our experiments are 64 $\times$ 64 grayscale images sampled through a thermal infrared camera with wavelength in 7–14 $\mu$ m. Totally, there are 14873 face images from 93 identities with various movements and expressions in our dataset, as shown in Fig. 8.

To evaluate RPS net comprehensively, three traditional feature based algorithms, HoG [36], LBP [37], Fisherface (LDA) [38] and four convolutional neural network based methods, DeepID net [26], VGG net [34], Alex net [44], Thermal face recognition net (TFR net) [33] are conducted with the public Caffe toolbox [48] on this thermal infrared face image dataset. Actually, DeepID net is proposed for face verification with the Joint Bayesian [49] technique. In this paper, DeepID net is trained as an identification issue with a softmax loss layer to get the predicted classes. The softmax loss $J(\theta)$ is defined in Eq. (4), $y(i)$ is the predicted label of the $i$ th training sample feature vector $x(i)$ , $m$ is the number of training samples, $j$ is the standard identity label for training, and $\theta_{j}^{T}$ is the training parameter for feature vector in model. $\lambda$ is the weight decay coefficient to limit weights for training, which is set as 0.0005 in this paper. The initial learning rate is set to 0.001 and decreased by 0.9 after several iterations. There are totally 300 K iterations and 5 decreases of learning rate for training RPS net. About 10% of training samples are taken as validation dataset to adjust hyper parameters, such as learning rates and weight decay rates. As the scales of compared CNN algorithms are similar, these hyper parameters are set as the same for all compared CNN algorithms.

$\displaystyle J(\theta)=-\frac{1}{m}\left[\sum\limits_{i=1}^{m}\sum\limits_{j=% 1}^{k}1\{y^{(i)}=j\}\right.$ $\displaystyle\quad∼{}\left.\log\frac{e^{\theta_{j}^{T}x^{(i)}}}{\sum\nolimits_% {l=1}^{k}{e^{\theta_{l}^{T}x^{(i)}}}}\right]+\frac{\lambda}{2}\sum\limits_{i=1% }^{k}{\sum\limits_{j=0}^{n}{\theta_{ij}^{2}}}$ $\displaystyle\quad∼{}1\{a=b\}=\begin{cases}1&a=b;\\ 0&a\neq b.\\ \end{cases}$ (3)

The complexity of CNN algorithms is mainly decided by convolutions. Without accelerating strategy, convolution complexity is $i\times C\times N\times H\times W\times K_{h}\times K_{w}$ . $i$ is the iteration number, $C$ is the input channel number, $N$ is the kernel number, $H$ and $W$ are the height and width of input feature map, $K_{h}$ and $K_{w}$ are height and width of convolution kernel. With the help of vectorization in Caffe toolbox [48] and GPU, convolution operation could be greatly accelerated.

Table 1

The accuracies on the test datasets

Methods	Test accuracy(%)
HoG $+$ SVM	88.67
Fisherface	81.58
LBP	86.75
Alex net	91.78
VGG net	94.18
DeepID net	94.28
TFR net	90.14
RPS net	95.97

In our experiment, the dataset is divided into five equal parts with no intersections to each other. Specifically, there are 3007, 2978, 2969, 2978 and 2941 images in these subsets respectively. For comprehensive comparison, all subsets are chosen as test set with model trained on the other four subsets. The average accuracy of five test results is taken as the final identification accuracy. Test accuracies of the proposed algorithm and comparison algorithms are listed in Table 1. Our algorithm exceeds all the compared algorithms by average accuracy. This indicates that RPS net is effective for thermal infrared face identification, in general.

To evaluate and analyze algorithms comprehensively on our thermal infrared dataset, statistical analysis for the compared CNN algorithms with various classifiers, property analysis of extracted feature vectors, accuracy analysis based on each class and robustness analysis based on top- $n$ accuracy and various noise are introduced and conducted as below in detail.

4.1 Comparison to the traditional feature based algorithms

Compared to traditional feature based algorithms, such as HoG, Fisherface and LBP, CNN algorithms achieve higher identification accuracies. HoG feature descriptor is based on gradient statistics in regional blocks, which is effecitve for the detection of specific objects. With the performance of classifier, such as SVM [50], HoG feature works better for detecting objects from background than distinguishing faces from each other. Fisherface or Linear Discriminant Analysis (LDA), similar to Eigenface algorithms, is based on analysis of whole dataset. Therefore, identification is affected by not only inter-class difference but also intra-class consistency. The issue which reduces intra-class consistency, such as noise, could influence identification accuracy easily. LBP feature is a fast and robust texture descriptor in regional area. Based on the relation to central pixel, LBP feature is not sensitive to illumination. However, facial pattern of thermal infrared face images is quite different from visible face images. As description of temperature distribution on face, thermal infrared face images have no clear textures and details stable enough to represent the difference among various identities. Therefore, manually designed features are not sufficient for identification. CNN algorithms, including the proposed RPS net, achieve higher accuracies due to efficient feature learning strategy.

4.2 Comparison to CNN algorithms

Compared to traditional feature based algorithms, CNN algorithms achieve higher accuracies due to good resolution of extracted feature vectors. Actually, CNN algorithms achieve close test accuracies to each other. To compare the detail difference of these algorithms, test accuracies for each person are shown in Fig. 9.

In Fig. 9, the class labels in horizontal axis are sorted by test accuracies from low to high for each algorithm. As no algorithms could ensure accuracy advantage over other algorithms for every class, the accuracy curves for all classes are rearranged by accuracies as shown in Fig. 9. In general, RPS net exceeds others based on particular accuracy for each person.

Table 2
The paired-sample $t$ -test result

H $=$ 0: X $=$ Y (Accuracy)	H	$p$ -value
RPS: Alex	1	1.20e-06
RPS: VGG	1	4.06e-19
RPS: DeepID	1	4.15e-05
RPS: TFR	1	1.65e-10

Figure 9.

The sorted test accuracies of each class.

To show the significance of the proposed algorithm over the other four CNN algorithms, paired-sample $t$ -test is conducted based on the particular accuracies for each person, as shown in Table 2.

In Table 2, the Hs with low $p$ -values indicate that accuracy of RPS net is significantly higher than the other four algorithms, statistically.

To compare these CNN algorithms with RPS net comprehensively, robustness, layer depth and properties of the feature vector are analyzed with more experiments below.

4.2.1 Robustness analysis and layer depth

To measure the robustness of the proposed algorithm, two kinds of experiments are conducted. One is top-1, 3, 5, 7 and 9 accuracies of the compared CNN algorithms, as shown in Fig. 10. The other is predicting performance of CNN algorithms based on thermal infrared face images with various Gaussian or salt & pepper noise, as shown in Figs 12 and 13. To compare the network complexity and layer depth, numbers of convolutions and fully connected layers in depth are listed in Table 3. These indicates that RPS net performs better for identification of thermal infrared faces robustly with less layers.

Table 3
The numbers of convolution and fully connected layers in depth

Methods	Conv number	FC number
Alex net	5	3
VGG net	13	3
DeepID net	4	2
TFR net	3	1
RPS net	2	2

Figure 10.

The top-1, 3, 5, 7 and 9 test accuracies of the CNN algorithms. Best viewed in color.

In Fig. 10, top-1, 3, 5, 7 and 9 test accuracies of CNN algorithms are presented. It indicates that the proposed algorithm exceeds the other CNN algorithms at all top- $n$ measure. Moreover, RPS net achieves close 100% accuracies for top-9, which means higher recall robustness.

To measure the algorithm robustness against various environment and noise, Gaussian and salt & pepper noise with various intensity are introduced on the test dataset. The Gaussian noise is with 0 mean value and 0.01, 0.015, 0.02, 0.025 and 0.03 variances. The salt & pepper noise density varies in 0.01, 0.03, 0.05, 0.07 and 0.09. The noised thermal infrared face image with different noise types and intensities are shown in Fig. 11.

The predicting accuracies of CNN algorithms based on various Gaussian noised images are shown in Fig. 12. With larger Gaussian variance, noised images become more difficult to identify, for all compared algorithms. In Fig. 12, accuracy of RPS net decreases slightly, while exceeding the others. It indicates that RPS net has high accuracy and good robustness against Gaussian noise.

Figure 11.

Noised thermal infrared face images with different noise types and intensities.

Figure 12.

The test accuracies on images noised by various Gaussian noise. Best viewed in color.

As shown in Fig. 11, salt & pepper noise has more randomness than Gaussian noise. Because Gaussian noise affects all pixels of image, while salt & pepper noise randomly affects some pixels. Gaussian noised image is more difficult to identify by human eyes, but easier for computer-based image classifier. The situation for salt & pepper noise is on the contrary. The accuracies of CNN algorithms for various salt & pepper noised images are shown in Fig. 13.

In Fig. 13, some algorithms are highly affected by salt & pepper noise, such as Alex net and VGG net. Actually, Alex net and VGG net are classic serial structured CNN algorithms. Although little influenced by noise, TFR net could not achieve high accuracy for identification. DeepID net has close robustness to the proposed RPS net, due to its regional parallel layer before fully connected layer. In conclusion, parallel channels could greatly improve the robustness of CNN features. Therefore, RPS net achieves high accuracy with good robustness by taken regional parallel structure as one of the main modules.

Actually, the four compared algorithms are basically serial structure based CNNs, aiming at extracting deeper features through sequentially cascaded convolutions. By adding deeper layers, CNN could extract more abstract features for learning patterns. Different from the strategy of “going deeper”, parallel convolutions are adopted in the proposed RPS net based on edge information. We believe multi-scale features of thermal infrared face are more suitable and robust for identification than abstract features. Therefore, serial convolution layers in depth are changed into a parallel-structured convolution, which contains three different channels of convolutional features in one layer. In this way, RPS net contains only two layers cascaded before fully connected layer, initial edge convolution layer and regional parallel layer, to acquire the feature vector for classification. Although layer depth is shorter as shown in Table 3, edge information is enhanced and involved in multi-scale features, ensuring correctness and robustness of thermal infrared face identification.

Figure 13.

The test accuracies on images noised by various salt & pepper noise. Best viewed in color.

Figure 14.

The feature vectors extracted through compared CNNs. The feature vectors for Alex net and VGG net are 4096-dimensional and shown in size 128 $\times$ 32. The feature vectors for DeepID net are 160-dimensional and shown in size 20 $\times$ 8. The feature vectors for TFR net are 6272-dimensional and shown in size 128 $\times$ 49. The feature vectors for RPS net are 512-dimensional and shown in size 32 $\times$ 16. $M_{\min}$ and $M_{\rm mean}$ are measures for vector intra-class consistency. RPS net feature vectors have best intra-class consistency.

4.2.2 The property of feature vectors

The extracted feature vectors, with intra-class consistency measures for identification extracted through four CNN algorithms, are shown in Fig. 14. As the final representation of thermal infrared face, feature vector could be considered as the key to evaluating the performance of different algorithms. Good feature vector should have high inter-class difference and intra-class consistency at the same time. Inter-class difference could be directly represented by classification accuracy, while intra-class consistency is not easy to describe. Usually, vector distance, such as Euclidean distance between two vectors, is taken as the similarity measure. However, distance measure is not reasonable for vectors in different dimensions. For example, the distance between 1000-dimensional vectors is quite different from the distance between 2-dimensional vectors. Therefore, non-dimensional measures are more reasonable than dimensional measures, such as vector distance. In our paper, cosine values, based on sample feature vectors from the same class, are calculated and compared as the measure of intra-class consistency. Larger cosine value indicates better intra-class consistency in the corresponding class. To evaluate the general intra-class consistency, two measures $M_{\min}$ and $M_{\rm mean}$ are defined as Eq. (4.2.2).

$\displaystyle M_{\min}\!\!\!\!\!=\!\!\min\limits_{\textit{class\_all}}(\min% \limits_{\textit{class}\_i}(\cos(v_{\textit{class}\_i},{\rm mean}(v_{\textit{% class}\_i}))))$ $\displaystyle M_{\rm mean}\!\!\!\!\!=\!\!\min\limits_{\textit{class\_all}}(% \operatorname*{mean}\limits_{\textit{class}\_i}(\cos(v_{\textit{class}\_i},{% \rm mean}(v_{\textit{class}\_i})\mspace{-1.0mu })\mspace{-1.0mu })\mspace{-1.0% mu })v_{\textit{class}\_i}=\{v|\textit{class}(v)=\textit{class}\_i\}$ (4)

In Eq. (4.2.2), $v_{\textit{class}\_i}$ is one feature vector from the vector set in the same class $i$ . $M_{\min}$ is the all-class minimum of single-class minimum cosine values. $M_{\rm mean}$ is the all-class minimum of single-class mean cosine values. $M_{\min}$ and $M_{\rm mean}$ of compared CNN algorithms are listed in Fig. 14. The minimum cosines, $M_{\min}$ and $M_{\rm mean}$ , from RPS net feature vectors are still larger than other CNN algorithms. It means that the RPS feature vectors are more close to each other in the same class. In other words, the proposed algorithm has better robustness due to high intra-class consistency.

Table 4

The predicting accuracies using different classifiers

Methods	SVM (%)	KNN (%)	Random forests (%)
Alex net	92.24	90.34	89.48
VGG net	93.77	92.50	90.64
DeepID net	94.74	94.38	91.88
TFR net	89.09	84.04	84.04
RPS net	96.19	95.69	94.24

Actually, classification accuracy is mainly affected by inter-class difference and intra-class consistency, despite various classifier based on different theories. On the other hand, vectors for classification with better inter-class difference and intra-class consistency, could achieve higher accuracy regardless of classifier properties. In other words, the performance of vectors for classification could be induced by comparing accuracies through different classifiers. To this purpose, several classical classifiers, SVM [50], KNN [11] and Random Forests [51] are introduced to train and test on the feature vectors extracted through different CNN algorithms. Actually, 10% of training images are separated as validation dataset to adjust hyper parameters, such as $k$ of KNN. The SVM model deployed in this paper is linear SVM with $C=$ 0.8 and gamma $=$ 5. The $k$ for KNN is 3. The rest parameters for these classifiers are set as default in OpenCV. The average test accuracies by these classifiers are listed in Table 4.

Table 4 indicates that the features of RPS net achieve higher accuracies than the other compared algorithms using different classifiers, in general. For further comparison, accuracies for each class are rearranged according to their values from low to high, as shown in Figs 15–17 by different classifiers.

Similar to the result shown in Fig. 9, Figs 15–17 indicate that RPS net achieves higher single-class accuracies in general. For statistical comparison based on the accuracy of each person, paired-sample $t$ -test is also conducted for all three classifiers. Test result is shown in Table 5.

Table 5

The paired-sample $t$ -test result for three classifiers

H $=$ 0: X $=$ Y	SVM	KNN	Random forests
(Accuracy)	(H/ $p$ -value)	(H/ $p$ -value)	(H/ $p$ -value)
RPS: Alex	1/6.74e-10	1/1.52e-12	1/1.17e-08
RPS: VGG	1/3.06e-05	1/1.13e-08	1/1.35e-08
RPS: DeepID	1/1.13e-04	1/1.54e-04	1/8.26e-07
RPS: TFR	1/2.14e-15	1/1.83e-17	1/5.49e-16

Figure 15.

The sorted accuracies of each class by SVM.

Figure 16.

The sorted accuracies of each class by KNN.

Figure 17.

The sorted accuracies of each class by Random Forests.

As shown in Table 5, all Hs are rejected due to low $p$ -values. This indicates that the proposed RPS features achieve significant higher accuracy than the other CNN algorithms with different classifiers, statistically.

The experiments for intra-class consistency of feature vector and performance with different classifiers, improved that the features extracted by RPS net have good inter-class difference and intra-class consistency. This is because the multi-scale features from RPS net contain more meaningful information for thermal infrared face identification. Therefore, the proposed RPS net has better performance for thermal infrared face identification.

4.2.3 Discussion

In serial structure based CNN algorithms, convolution layers are cascaded as a sequence to acquire abstract features from deeper layers. In any serial system especially CNNs, the current layer should have more parameters to enhance information from the previous layer. Therefore, lots of pooling or down-sampling methods are proposed and adopted to limit the parameter number without losing features very much. One of the problems while designing serial structured CNN, is to deal with the contradiction between the increasing parameters and decreasing features by down-sampling. Compared to serial structure algorithms, features extracted by parallel structure usually have better property and higher robustness [52, 53, 54, 55], which could avoid losing information between cascaded modules.

By limiting depth and adding regional parallel convolutions, RPS net could generate multi-scale features which is more effective for thermal infrared face identification. This is because that multi-scale features are extracted from low-level information, which is very important for thermal infrared images. Comparing to serial structured CNN features, multi-scale features contain more information than high-level abstract features for thermal infrared face identification.

Due to regional parallel structure, RPS net could extract high quality features with less layers. Therefore, higher accuracy of thermal infrared face identification could be achieved robustly by using RPS net.

5. Conclusions

In this paper, a novel convolutional neural network is proposed and introduced to thermal infrared face identification. The convoluted edges are taken as initial features, based on the characteristics of thermal infrared face images. A regional parallel structure is proposed to extract multi-scale features based on edge information. In this way, the enhanced edges could be represented by multi-scale features including neighbor and local region information, which perform better than abstract features extracted by serial structured CNNs.

Extensive experiments are conducted and analyzed for comprehensive evaluation. The statistical test results with various classifiers indicate that features extracted by RPS net achieves high accuracy with different classifiers, statistically. Feature vector property experiment demonstrates that RPS net features have good inter-class difference and intra-class consistency. Moreover, the robustness of RPS net has been proved, by introducing Gaussian and salt & pepper noise for test. The experimental result indicates that RPS net overtakes algorithms based on traditional features (HoG, Fisherface and LBP) and some convolutional neural networks (Alex net, VGG net, DeepID net and TFR net), with high quality features.

In conclusion, the proposed RPS net is more effective and robust than some traditional feature based algorithms and serial structured neural networks for thermal infrared face identification.

Footnotes

Acknowledgments

This work is supported by the National Natural Science Foundation of China under Grant U1736217, the Program for New Century Excellent Talents in Universities under Grant NCET-13-0020, the Fundamental Research Funds for the Central Universities under Grant YWF-17-BJ-Y-69.

References

Roberts

. Machine perception of three-dimensional solids. Massachusetts Institute of Technology, 1963.

Sobel

. Neighborhood coding of binary images for fast contour following and general binary array processing. Computer Graphics & Image Processing 1978; 8(1): 127-135.

Prewitt

. Object enhancement and extraction. Picture Processing and Psychopictorics 1970; 75-149.

Ghiass

, et al. Infrared face recognition: A comprehensive review of methodologies and databases. Pattern Recognition 2014; 47(9): 2807-2824.

Papakostas

, et al. Thermal infrared face recognition based on lattice computing (LC) techniques. IEEE International Conference on Fuzzy Systems (FUZZ) 2013; 1-6.

Farokhi

, et al. Near infrared face recognition using Zernike moments and Hermite kernels. Information Sciences 2015; 316: 234-245.

Liu

, et al. Toward large-population face identification in unconstrained videos. IEEE Transactions on Circuits and Systems for Video Technology 2014; 24(11): 1874-1884.

Lenc

Král

. Local binary pattern based face recognition with automatically detected fiducial points. Integrated Computer-Aided Engineering 2016; 23(2): 129-139.

Tran

Kantartzis

Liatsis

. Improving fusion with optimal weight selection in face recognition. Integrated Computer-Aided Engineering 2012; 19(3): 229-237.

10.

, et al. Learning compact feature descriptor and adaptive matching framework for face recognition. IEEE Transactions on Image Processing 2015; 24(9): 2736-2745.

11.

Cover

Hart

. Nearest neighbor pattern classification. IEEE Transactions on Information Theory 1967; 13(1): 21-27.

12.

Hinton

, et al. Reducing the dimensionality of data with neural networks. Science 2006; 313(5786): 504-507.

13.

Long

Shelhamer

Darrell

. Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015; 3431-3440.

14.

Chen

, et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv: 1606.00915, 2016.

15.

Liu

, et al. SSD: Single shot multibox detector. European Conference on Computer Vision 2016; 21-37.

16.

, et al. Mask r-cnn. arXiv preprint arXiv: 1703.06870, 2017.

17.

Lin

Nie

. Structural damage detection with automatic feature-extraction through Deep Learning. Computer-Aided Civil and Infrastructure Engineering, 2017.

18.

Rafiei

Adeli

. A novel machine learnin-based algorithm to detect damage in high-rise building structures. The Structural Design of Tall and Special Buildings 2017; 26(18).

19.

Rafiei

Khushefati

Demirboga

, et al. Supervised Deep Restricted Boltzmann Machine for Estimation of Concrete. ACI Materials Journal 2017; 114(2).

20.

Chen

, et al. Turn signal detection during nighttime by CNN detector and perceptual hashing tracking. IEEE Transactions on Intelligent Transportation Systems, 2017.

21.

Liao

. On-line Vehicle Routing Problems for Carbon Emissions Reduction. Computer-Aided Civil and Infrastructure Engineering, 2017.

22.

Ortega-Zamorano

Jerez

Gómez

, et al. Layer multiplexing FPGA implementation for deep back-propagation learning. Integrated Computer-Aided Engineering 2017; 24(2): 171-185.

23.

Koziarski

Cyganek

. Image recognition with deep neural networks in presence of noise-Dealing with and taking advantage of distortions. Integrated Computer-Aided Engineering 2017; 24(4): 337-349.

24.

Taigman

, et al. Deepface: Closing the gap to human-level performance in face verification. IEEE Conference on Computer Vision and Pattern Recognition 2014; 1701-1708.

25.

Sun

, et al. Deep Learning face representation by joint identification-verification. Advances in Neural Information Processing Systems 2014; 1988-1996.

26.

Sun

, et al. Deep Learning face representation from predicting 10,000 classes. IEEE Conference on Computer Vision and Pattern Recognition 2014; 1891-1898.

27.

Zhu

, et al. Deep Learning identity-preserving face space. IEEE International Conference on Computer Vision 2013; 113-120.

28.

Xiong

, et al. Convolutional fusion network for face verification in the wild. IEEE Transactions on Circuits and Systems for Video Technology 2016; 26(3): 517-528.

29.

, et al. Reconstruction-based metric learning for unconstrained face verification. IEEE Transactions on Information Forensics and Security 2015; 10(1): 79-89.

30.

Chen

, et al. Unconstrained face verification using deep cnn features. IEEE Winter Conference on Applications of Computer Vision 2016; 1-9.

31.

Xiong

, et al. Convolutional fusion network for face verification in the wild. IEEE Transactions on Circuits and Systems for Video Technology 2016; 26(3): 517-528.

32.

Sun

, et al. Deepid3: Face recognition with very deep neural networks. arXiv preprint arXiv: 1502.00873, 2015.

33.

Peng

Chen

. Thermal face recognition using convolutional neural network. IEEE International Conference on Optoelectronics and Image Processing 2016; 6-9.

34.

Simonyan

Zisserman

. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.1556, 2014.

35.

Szegedy

, et al. Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition 2015; 1-9.

36.

Dalal

, et al. Histograms of oriented gradients for human detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2005; 886-893.

37.

Ojala

, et al. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002; 24(7): 971-987.

38.

Scholkopft

Mullert

. Fisher discriminant analysis with kernels. Neural Networks for Signal Processing IX 1999; 1(1): 1.

39.

Zhang

, et al. Face recognition: Eigenface, elastic matching, and neural nets. Proceedings of the IEEE 1997; 85(9): 1423-1435.

40.

Gunturk

, et al. Eigenface-domain super-resolution for face recognition. IEEE Transactions on Image Processing 2003; 12(5): 597-606.

41.

Cortes

Vapnik

. Support-vector networks. Machine Learning 1995; 20(3): 273-297.

42.

Bai

Wang

Zhou

. Pedestrian segmentation in infrared images based on circular shortest path. IEEE Transactions on Intelligent Transportation Systems 2016; 17(8): 2214-2222.

43.

Bai

Wang

Liu

, et al. Symmetry information based fuzzy clustering for infrared pedestrian segmentation. IEEE Transactions on Fuzzy Systems, 2017.

44.

Krizhevsky

Sutskever

Hinton

. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 2012; 1097-1105.

45.

Hinton

, et al. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 1207.0580, 2012.

46.

OTCBVS homepage on the Internet. IRIS Thermal/Visible Face Database; 2017 cited 3 Nov 2017. Available from: http://vcipl-okstate.org/pbvs/bench/Data/02/download.html

47.

OTCBVS homepage on the Internet. Terravic Facial IR Database; 2017 cited 3 Nov 2017. Available from: http://vcipl-okstate.org/pbvs/bench/Data/04/download.html

48.

Jia

Shelhamer

Donahue

, et al. Caffe: Convolutional architecture for fast feature embedding. Proceedings of the 22nd ACM International Conference on Multimedia 2014; 1: 675-678.

49.

Chen

, et al. Bayesian face revisited: A joint formulation. European Conference on Computer Vision 2012; 566-579.

50.

Chang

, et al. LIBSVM: A Library for Support Vector Machines. ACM Transactions on Intelligent Systems & Technology 2006; 2(3): 389-396.

51.

Breiman

. Random forests. Machine Learning 2001; 45(1):5-32.

52.

Hung

Adeli

. Parallel backpropagation learning algorithms on Cray Y-MP8/864 supercomputer. Neurocomputing 1993; 5(6): 287-302.

53.

Park

Adeli

. Distributed neural dynamics algorithms for optimization of large steel structures. Journal of Structural Engineering 1997; 123(7): 880-888.

54.

Adeli

. Supercomputing in Engineering Analysis. Marcel Dekker, New York, 1992.

55.

Adeli

. Parallel Processing in Computational Mechanics. Marcel Dekker, New York, 1992.

Regional parallel structure based CNN for thermal infrared face identification

Abstract

Keywords

1. Introduction

3. Algorithm

3.2.1 The features in neighbor region

3.2.2 The features in local region

3.3 The feature vector of thermal infrared face

3.4 Discussion

4.2 Comparison to CNN algorithms

Table 2 The paired-sample t -test result

Table 3 The numbers of convolution and fully connected layers in depth

5. Conclusions

Footnotes

Acknowledgments

References

Table 2
The paired-sample $t$ -test result

Table 3
The numbers of convolution and fully connected layers in depth