Facial expression recognition with fused handcraft features based on pixel difference local directional number pattern

Abstract

Facial expression recognition (FER) has been an active research area in recent years, which plays a vital role in national security and human-computer interaction. Due to the lacking of sufficient expression features and facial images, it is challenging to automatically recognize facial expression with high accuracy. In this paper, we propose a fusion handcraft feature method to improve FER from images. Firstly, a new texture feature extraction method PD-LDN (Pixel Difference Local Directional Number pattern) is proposed, which can extract more local information, reduce noise disturbance and feature dimension. Secondly, the handcrafted features including PD-LDN texture features, geometric features, and BOVW (Bag of Visual Words) semantic features are connected in parallel to an improved autoencoder network for fusion. Finally, the fused features are input into the softmax classifier for recognizing facial expression. We conduct extensive experiments on JAFFE and CK+datasets. Our proposed method shows superior performance than the state-of-the-art approaches on recognizing facial expressions.

Keywords

Facial expression recognition LDN feature fusion softmax

1 Introduction

Facial expressions can intuitively reflect the inner activities of human beings and play an important role in social intercourse. Studies have shown that in information transmission, the amount of information conveyed by facial expressions is as much as 55% [1]. In recent years, FER has become a research hotspot in the field of computer vision. Due to the complex changes in facial expressions and the recognition process involving many interdisciplinary subjects such as psychology, computer vision, and machine learning, FER is a challenging task [2 –4]. The most important procedure for recognizing facial expression is to extract discriminative features so as to successfully distinguish different emotions. Researchers from various countries have proposed different feature extraction methods to classify facial expressions from images.

There are handcraft feature-based methods such as: texture feature-based methods [5 , 11–15], geometric feature-based methods [16 –18], and semantic feature-based methods [19]. Furthermore, considering the combination of various features can provide more effective facial representation, many researchers began to fuse these handcraft features and the performance is superior to individual feature-based methods [20–25 , 43]. With the rapid development of deep learning in recent years, many novel neural networks have been applied to expression recognition [7–9 , 40–44]. The end-to-end neural network model can obtain high-level features through learning the low-level features of the images. However, the training and testing stage of deep learning networks require a large amount of data, and the existing expression datasets are too small to meet the need [36]. It will lead to the problem of over-fitting.

In this paper, we concentrate on recognizing facial expressions from images. Inspired by numerous successful applications of handcraft features in FER, we propose a method with fused handcraft features including texture, geometric, and semantic features. Besides, we propose PD-LDN to exploit local texture features of facial expression image. Whereas, the seven-order moment is good at denoting the locations and shape of facial components, thus reflecting the geometric characteristics of different expressions. In this paper, we adopt the seven-order moment to extract geometric features of facial expression images. Also, due to the eyes, nose, and mouth make a great contribution to the facial expression [10], we propose the approach based on the key areas of the face and BOVW to extract the representative semantic feature. Then, we fuse these handcraft features through an autoencoder network to obtain the final characteristic feature of facial expression. By fusion with the autoencoder network, the information and correlation among handcraft features can be exploited and reconstructed effectively. Meanwhile, the method shows stronger robustness and thus improves the accuracy of recognizing facial expressions.

The novelty and contribution of our work are primarily: (1) A novel PD-LDN is proposed, which can extract the detailed local features of facial expression image by calculating the edge response value of the local neighborhood and adjusting the pixel in coding direction through an adaptive threshold. Besides, PD-LDN has high robustness on noise and reduces the feature dimension effectively. (2) We propose to extract the semantic feature on key regions of a facial image by the BOVW model which introduces DSIFT (Discrete SIFT) to describe the key points of the facial expression. (3) Unlike the existing fusion methods by connecting the features in series, we propose a feature fusion model with an improved autoencoder network. In which, three hidden layer vectors are assigned to represent the texture features, geometric features, and semantic features respectively. The final fused features are obtained through the unified decoding of the hidden layer nodes. The proposed method can distinguish the changes of expressions from the aspect of appearance, structure, and intensity of a face image.

The paper is organized as follows: Section II discusses the related works. Section III introduces the proposed method. Section IV discusses the experimental results. The paper is concluded in Section V.

2 Related work

The texture feature is vital for pattern analysis of images. Jabid et al. [11] defined LDP (Local Direction Pattern) to classify facial expression. Wang et al. [12] put forward LPQ (Local Phase Quantization) and sparse representation to extract the texture features of expression. LDN [32] was presented to extract the texture feature of facial expression, which can obtain a set of direction codes according to the edge response value of the pixel, and represent the local edge texture information well. Shokoohi et al. [13] proposed to extract texture features of face images by using GLTP (Gradient Local pattern and gradient-based ternary Texture Patterns). Ding et al. [14] used LBP and Taylor expansion to perform texture feature extraction on facial expression images. Gu et al. [15] adopt a local Gabor feature radial coding mode to recognize expression. Considering the shape variables of the facial features, [16 –18] proposed geometric feature-based methods.

To make full use of different methods, Harrati et al. [20] combined BOVW and HOG (Histogram of Oriented Gradient) to extract high-dimensional features for recognizing expressions. Kumari et al. [21] extracted LBP, HOG, and LDP features in series to form a fusion feature and then used SVM to classify facial expressions. [22 –24] utilized texture and geometric features to recognize expressions. In [25], Yang et al. introduced three texture features of LBP, LDN, and EOH in series to perform FER. The recognition rate of this method increasing a lot compared to [22 –24]. Ghimire et al. [42] and Happy et al. [43] combined shape and appearance features for FER to achieve better accuracy.

Several deep learning networks have recently been used for FER. Chen et al. [26] adopted Cellular Neural Network (CNN) to recognize facial expressions, which extracted the implicit features of images through the convolutional layer of the network and used the pooling layer to reduce features. Liu et al. [27] proposed a BDBN (Bilinear Deep Belief Network) for expression recognition. BDBN consists of a set of weak classifiers, each of which is responsible for the classification of an expression. However, the high accuracy of this method is at the cost of complex computational and long training time. Lopes et al. [28] proposed to combine CNN and data preprocessing to classify facial expression. In [29], the weighted mixture deep neural network (WMDNN) is used to extract the facial expression features with the dual-stream. Literature [30] proposed to use FRR-CNN (Feature Redundancy Reduced Convolutional Neural Network) for FER. These two approaches achieve a better result on the FER compared to other deep learning methods.

Autoencoder is a symmetrical neural network that can learn the features of input data by minimizing the discrepancy between the original data and its reconstruction. Recently, autoencoder networks have been applied to FER. Lv et al. [40] adopted HOG, DBN, and Gabor to extract the features, and utilized stacked autoencoder (SAE) to recognize expressions. Zeng et al. [41] proposed deep sparse autoencoder (DSAE) to recognize the facial expression, which can learn robust and discriminative features from data. This method introduced geometric and appearance features to compose a high-dimensional feature and got an accuracy of 95.79%.

3 Proposed method

3.1 Method review

We propose a feature fusion method for FER. The method with two key schemes: handcraft features extraction and features fusion. Figure 1 illustrates the framework of the proposed method.

Fig. 1

Illustration of the proposed method for facial expression recognition. First, preprocess the original images to irregular interest blocks. For each block, texture features, geometric feature and semantic feature are extracted by the proposed PD-LDN, seven-order moment and BOVW respectively. Then, the final features are fused by the improved autoencoder network. Finally, the features are sent to the softmax classifier for facial expression recognition.

As we can see in Fig. 1, the facial images are blocked firstly. We employ the AAM (Active Appearance Model) [31] to mark feature points on the face image. Then, the face image can be divided into several irregular blocks according to the connection of these feature points. These blocks can segment the expression features well, and the size of the block can change with expression.

Then, we perform feature extraction which consists of PD-LDN to extract texture features, seven-order moment to extract geometric features, and improved BOVW to extract semantic features. Finally, the features are fused by the autoencoder network and then classified by the softmax to recognize facial expression. Below, we present the details of the key schemes, the proposed texture feature extraction method PD-LDN, semantic feature extraction method - improved BOVW, and feature fusion model.

3.2 PD-LDN

The LDN is a typical texture feature-based method. The directional and edge feature of the local neighborhood are extracted, which can reflect the local texture information of the image well and overcome the shortcomings of LBP on the influence of illumination and noise. Whereas, some local information may be lost as its operator performs mask calculation only on the local neighborhood, ignoring the pixels of the neighborhood center. Besides, LDN performs mask calculation in 8 directions for each neighborhood, resulting in high computation complexity.

To solve these problems, this paper proposes a local description operator PD-LDN. PD-LDN can calculate the edge response value of local neighborhood only in two main directions, and obtain the pixel value in the coding direction through the adaptive threshold, to obtain more detailed local texture information and representative features.

Figure 2 shows the Robinson masks [33]. M₀ ∼ M₇ are the template matrix of 8 directions. Due to the symmetry property of the Robinson matrix, the edge response value calculated in the M₀ ∼ M₃ and M₄ ∼ M₇ are opposite to each other. We only conduct mask calculation in these four directions M₀ ∼ M₃ to obtain the edge response value of the local neighborhood. The calculation reduces the computation time to a certain extent.

Fig. 2

The Robinson mask.

The mask value of the center pixel is 0 in LDN, which means that the center pixel is ignored and much representative information is missed. The PD-LDN takes into account the influence of the central pixel on the process of coding. As shown in Fig. 3, PD-LDN puts each pixel on the center of the 3*3 neighborhood and uses the Robinson mask to calculate the edge response of this pixel. As shown in Equation (1): $R_{i} = M_{i} * G, 0 ⩽ i ⩽ 3$ (1) where G is the 3*3 neighborhood, M_i and R_i are the Robinson mask and the edge response value in the direction i. The edge texture information in the neighborhood can be represented well by calculating the edge response value.

Fig. 3

The coding process of PD-LDN. M₀, M₁, M₂, and M₃ are the Robinson matrix. D₁ and D₂ are the coding directions with the largest absolute edge response value. B₁ and B₂ are binary patterns.

The two largest absolute edge response values are taken as the coding direction D₁ and D₂, that is: $D_{i} = {argmax}_{i}^{j} {| R_{i} | : 0 ⩽ i ⩽ 3, j \in {1, 2}}$ (2)

The corresponding pixel values of D₁ and D₂ are subtracted from the central pixel value respectively to obtain difference values. The difference values are binarized to obtain two binary patterns B₁ and B₂ by Equation (3): $B_{1} = {\begin{matrix} 1, | g_{1} - g_{c} | > δ \\ 0, | g_{1} - g_{c} | ⩽ δ \end{matrix}, B_{2} = {\begin{matrix} 1, | g_{2} - g_{c} | > δ \\ 0, | g_{2} - g_{c} | ⩽ δ \end{matrix}$ (3) where g₁ and g₂ are the pixel values corresponding to two coding directions D₁ and D₂. g_c is the center pixel value of the neighborhood. δ is the threshold.

The threshold δ is adaptively adjusted according to the pixel value in the neighborhood, and the calculation process is as follows:

The mean value of all pixels $\bar{g}$ in the neighborhood of 3×3 is calculated as Equation (4): $\bar{g} = (\sum_{v = 0}^{T - 1} g_{v}) / T, v = 0, \dots T - 1$ (4)

The contrast value g_v is obtained by subtracting each pixel value in the neighborhood from the mean value $\bar{g}$ : $g_{v} = g_{v} - \bar{g}, v = 0, \dots T - 1$ (5)

The mean value $\bar{g}$ is: $\bar{g} = (\sum_{v = 0}^{P - 1} g_{v}) / T, v = 0, \dots T - 1 .$ (6)

Finally, the adaptive threshold δ can be obtained by: $δ = {(\sum_{v = 0}^{P - 1} g_{v} - \bar{g})}^{2} / T$ (7)

In Equations (4)–(7), T is the number of pixels in the neighborhood, g_v is the pixel value.

Then, we combine the direction codes D₁ and D₂ and the binary codes B₁ and B₂ to get the PD-LDN code of the point (x, y) in the image. The value of PD-LDN can be obtained by converting the binary PD-LDN into decimal PD-LDN: $\begin{matrix} PD - LDN (x, y) = 2^{4} D_{1} (x, y) + 2^{3} B_{1} (x, y) \\ + 2 D_{2} (x, y) + B_{2} (x, y) \end{matrix}$ (8)

As shown in Fig. 4, the PD-LDN histogr of each block can be got by performing PD-LDN calculation on each pixel. The PD-LDN histograms of 15 blocks are connected in series to obtain the feature histogram of the entire expression image. The dimension of each block is 26, and the PD-LDN feature dimension of an image is 26 *15 = 960.

Fig. 4

The feature extraction process of PD-LDN.

3.3 Semantic feature extraction

As the local feature of an image has certain similarities with the vocabulary of natural language, it can be treated as a visual vocabulary [34]. Due to the BOVW model performs feature extraction on the whole expression image, it will make some image areas fragmentize, resulting in the low expression recognition rate [10]. Therefore, we propose a method based on the key areas of the face and BOVW. Firstly, face images are segmented according to the feature points located by AAM. The blocks of eyes, nose, and mouth are segmented to form key regions. These key regions are normalized to 41*130. Then, the semantic feature is extracted by the BOVW model. The process is illustrated in Fig. 5.

Fig. 5

Schematic diagram of construct the BOVW model. Three dictionaries consist of k words are clustered by DSIFT features. The cluster centers are denoted as triangles, squares, hexagons, and circles. The statistical word distribution can finally be represented as the histogram of word visual frequency, that is, the semantic feature of an expression.

The BOVW model usually adopts SIFT (Scale Invariant Feature Transform) to represent the key points. This mode may extract various numbers of key points for different expressions, which may cause the difference of feature dimensions extracted from every image. It has an adverse effect on the subsequent clustering operations. This paper introduces DSIFT to perform facial expression images.

The extracted DSIFT features of key regions are clustered by k-means to obtain three visual dictionaries. One word in the dictionary is regarded as a cluster center and represented as a 128-dimensional vector. Each dictionary consists of k words. The Euclidean distance is used to calculate the distance between the DSIFT feature vector and each word in the dictionary and map it to the word with the smallest distance. Finally, the semantic feature of an expression can be obtained by the statistical word distribution of a map.

3.4 Autoencoder network for feature fusion

As a kind of neural network, autoencoder can explore the deep information and correlation existing in the features [35]. To improve its representation, robustness and achieve better effects of feature fusion, this paper improves the traditional autoencoder network, as shown in Fig. 6.

Fig. 6

The proposed autoencoder structure. X_p, X_g, and X_b are input vectors of PD-LDN, seven-order moment, and BOVW. W_p, W_g, and W_b are the weight matrix. h_p, h_g, and h_b are hidden layer vectors.

Considering that the number of autoencoder network layers is higher than 3, it will lead to the problems of over-fitting [36]. To achieve better fusion features, we designed a 3 layers network model.

In Fig. 6, Layer 1 is the input layer, Layer 2 is the hidden layer, Layer 3 is the output layer. X_p, X_g, and X_b are three input vectors, which represent the extracted PD-LDN texture feature, seven-order moment feature, and BOVW semantic feature. W_p, W_g, and W_b are the weight matrix. h_p, h_g and h_b are hidden layer vectors.

The process of network coding can be expressed as: $h_{j}^{z} (x) = f (x) = \frac{1}{1 + e^{- a_{j}^{z} (x)}}$ (9) where f (x) is the sigmoid function, $h_{j}^{z} (x)$ is the jth node value, z is the type of feature, $a_{j}^{z}$ is the output of the jth hidden layer node, $a_{j}^{z}$ can be expressed as follows: $a_{j}^{z} = b_{j}^{z} + \sum_{k} W_{jk}^{z} x_{k}$ (10) where $b_{j}^{z}$ is the offset vector of the jth node, $W_{jk}^{z}$ is the weight, j and k are the row and column coordinates of the weight matrix.

Then, all of the hidden layer nodes are combined to form a hidden layer vector for decoding. The hidden layer vector x₂ can be expressed as: $x_{2} = [h_{1}^{p}, \dots, h_{M}^{p}, h_{1}^{g}, \dots h_{N}^{g}, h_{1}^{b}, \dots, h_{L}^{b}]$ (11) where h is the node value. M, N, and L are the hidden layer node number of texture, geometry, and semantic feature respectively. The fused feature vector $\hat{X}$ is: $\hat{X} = g (x_{2})$ (12) where g (.) is the decoding function, as shown in (13): $g (h) = f (wh + b)$ (13) where f (.) is the sigmoid function, w is the weight matrix, b is the offset vector.

The weight matrix and the offset vector are adjusted by the gradient descent method as (14) and (15): $W^{(Z)} (i + 1) = W^{(z)} (i) - α \frac{\partial}{\partial W^{(z)} (i)} J (W^{(z)}, b^{z})$ (14) $b^{(z)} (i + 1) = b^{(z)} (i) - α \frac{\partial}{\partial b^{(z)} (i)} J (W^{(z)}, b^{(z)})$ (15) where W^(z) and b^(z) are the weight matrix and the offset vector. J (W^(z), b^(z)) is the loss function, as shown in (16): $\begin{matrix} J (W^{(z)}, b^{z}) = \frac{1}{2} ∥ W^{(z) T} h (x) - x ∥^{2} \\ - \frac{λ}{2} \sum_{j} \sum_{k} (W_{jk}^{(z)})^{2} \end{matrix}$ (16) where λ is the penalty coefficient of weight. Finally, the fused feature is classified by softmax.

4 Results and discussion

In this section, we provide a detailed experimental analysis of the proposed method and compare it with some state-of-the-art methods.

4.1 Datasets

4.1.1 JAFFE

The JAFFE dataset [37] contains 213 facial expression images of 10 women, including 7 expressions: angry, scared, happy, disgusted, neutral, sad, and surprised. Each image has a resolution of 256*256. In this paper, all the images in JAFFE are selected for the experiment, and the expression labels are assigned as: 1-fear, 2-disgust, 3-happy, 4-sad, 5-angry, 6-neutral, 7-surprise.

4.1.2 CK+

The CK+(Extended Cohn Kanade) dataset [38] was released in 2010. CK+contains 123 people, a total of 593 expression image sequences. The resolution of frames is 640*490. This paper selects 312 sequences of the CK+dataset, a total of 5200 images for the experiment. The labels of all expression images are assigned the same as the JAFFE.

4.2 PD-LDN results and analysis

The results include the recognition rate influenced by block mode, threshold, and different detailed experiments of the proposed PD-LDN. We adopt SVM as a classifier.

4.2.1 The impact of the block mode on PD-LDN

The images are divided by different block modes firstly. The average 10 accuracies are taken as the result of each mode, as shown in Table 1. It can be seen that the irregular block mode gets a better result than the regular mode.

Table 1
Comparison of different block mode

Block mode Recognition rate (%)

JAFFE CK+

33 78.35 74.33

44 79.61 77.69

55 80.47 79.15

66 85.14 83.25

8*8 79.26 78.45

Irregular mode 93.25 91.64

Block mode	Recognition rate (%)
3*3	78.35	74.33
4*4	79.61	77.69
5*5	80.47	79.15
6*6	85.14	83.25
8*8	79.26	78.45
Irregular mode	93.25	91.64

4.2.2 PD-LDN threshold selection

We compare the dynamical threshold and the fixed threshold according to the pixel distribution in the neighborhood of PD-LDN. From Fig. 7 it can be seen with a fixed threshold, PD-LDN can achieve a better result when the threshold is set to 40 in JAFFE and 60 in CK+.

Fig. 7

The PD-LDN performance with different thresholds.

We conduct the experiment 10 times with adaptive and fixed threshold respectively. The average recognition rates are taken as the final result. In Table 2, PD-LDN_δ = 40 and PD-LDN_δ = 60 represent the fixed threshold 40 and 60, PD-LDN_K means using an adaptive threshold in PD-LDN. It can be seen that the adaptive threshold can get a better recognition rate. Therefore, an adaptive threshold is used in this paper.

Table 2

Comparison of PD-LDN with different threshold

Threshold	Recognition rate (%)
	JAFFE	CK+
PD-LDN_δ = 40	89.60	73.23
PD-LDN_δ = 60	86.17	80.40
PD-LDN_K	93.25	91.64

4.2.3 Comparison with texture feature-based methods

We compare PD-LDN with several traditional texture feature-based methods. The experimental results are shown in Table 3. It suggests that the accuracy obtained by the proposed PD-LDN is higher than other methods.

Table 3
Comparison with different texture feature-based approaches

Approaches Recognition rate (%)

JAFFE CK+

LBP [14] 81.35 79.33

LDP [11] 87.70 86.36

LPQ [12] 83.47 79.15

Gabor [15] 89.35 87.17

LDN [32] 91.60 89.30

PD-LDN 93.25 91.64

Approaches	Recognition rate (%)
LBP [14]	81.35	79.33
LDP [11]	87.70	86.36
LPQ [12]	83.47	79.15
Gabor [15]	89.35	87.17
LDN [32]	91.60	89.30
PD-LDN	93.25	91.64

4.2.4 Anti-noise experiment of PD-LDN

The facial expression images are added with various Gaussian noise. Different methods are adopted to extract the feature of the facial expression image with noise. The recognition rate is showed in Fig. 8. We can see that under the influence of different Gaussian noises, the results of the other texture feature-based methods have significantly reduced. PD-LDN shows a remarkable anti-noise ability.

Fig. 8

The recognition rate (%) of different approaches with different Gauss noise.

4.3 Feature fusion results and analysis

4.3.1 Parameter settings

The number of hidden layer nodes in the autoencoder network has a certain influence on the effect of feature fusion. If the number of hidden layer nodes is too small, the network can not have the learning ability; on the contrary, too many nodes may greatly increase the complexity of the network structure, the network tends to fall into a local minimum in the learning process.

So we set the number of hidden layer nodes by the controlling variables method. As the nodes of one feature are changed, the other two are controlled to be unchanged. The dimension of the PD-LDN feature is 15*64 = 960, which of the seventh-order moment is 7*15 = 105, and the BOVW has a feature dimension of 300. As adjusting one feature node, the nodes of the other two features are set to half of its input number. The nodes are: hidden layer nodes of PD-LDN h_p, nodes of seven-order moment feature h_g, and nodes of the BOVW feature h_b. According to experiments shown in Fig. 9, the optimal hidden layer nodes are assigned h_p = 600, h_g = 75, and h_b = 210.

Fig. 9

Experiments on the number of hidden layer nodes. h_p, h_g, h_b are the hidden node of PD-LDN, seven-order moment, and BOVW feature respectively.

In the process of semantic feature extraction, the optimal parameters of the DSIFT are the number of words k and the sampling density m in the visual dictionary, which are all selected through experiments. We set the value of k are 100, 200, 300, 400; m are 2, 4, 6, 8. The experiments are taken 3 times on JAFFE and CK+, respectively, and the average value is taken as the final result. As we can see in Fig. 10, the recognition can get the best accuracy as the sampling density m is 2 and the number of words k in the visual dictionary is 300.

Fig. 10

Recognition rates of DSIFT with different k (the number of words) and m (the sampling density) (a) On JAFFE. (b) On CK+.

4.3.2 Results in recognizing different expressions

To validate the effectiveness of the proposed method, we conduct the confusion matrix in the experiments. Figure 11 is the confusion matrix of the proposed method on JAFFE. As can be seen that the recognition rate of anger, happiness, and surprise is higher because of the large variation of facial muscles. The expression of neutral usually confuse with other expressions due to it has no remarkable change and its feature distribution contains the largest area in the feature space, so the recognition accuracy of neural is reduced.

Fig. 11

Confusion matrix on JAFFE. AN, NO, DI, FE, HA, SA, SU denote anger, neutral, disgust, fear, happiness, sadness and surprise respectively.

Figure 12 is the confusion matrix of the proposed method on CK+. The recognition rates of sadness and neutral expression are lower due to the eyes and mouths are similar, meanwhile, the features of eyes and mouth are difficult to extract properly to distinguish them. The category of anger, surprise, and happiness can be always predicted correctly because these images are all with larger facial changes and the features are obvious.

Fig. 12

Confusion matrix on CK+. AN, NO, DI, FE, HA, SA, SU denote anger, neutral, disgust, fear, happiness, sadness and surprise respectively.

4.3.3 Comparison with state-of-the-art methods

The recognition results of the three single feature methods and different feature fusion methods are compared by classifier softmax in Table 4. According to the experiment, the proposed fusion features can assist in improving the performance of expression recognizing and work best.

Table 4
Comparison with the individual component of combination

Approaches Recognition rate (%)

JAFFE CK+

PD-LDN 93.25 91.64

Seven Order Moment(SOM) 80.35 79.26

BOW 91.23 89.35

PD-LDN+BOW 93.56 91.93

PD-LDN+SOM 94.52 92.73

SOM+BOW 91.3 90.15

PD-LDN+SOM+BOW 95.36 92.87

Proposed method 98.76 97.58

Approaches	Recognition rate (%)
PD-LDN	93.25	91.64
Seven Order Moment(SOM)	80.35	79.26
BOW	91.23	89.35
PD-LDN+BOW	93.56	91.93
PD-LDN+SOM	94.52	92.73
SOM+BOW	91.3	90.15
PD-LDN+SOM+BOW	95.36	92.87
Proposed method	98.76	97.58

To further validate the effectiveness of the proposed method, Table 5 shows the comparison of the recognition rate with the state-of-the-art FER methods.

In literature [13], the GLTP is lack of facial expression information due to its incomplete feature. The literature [25] connected LBP, LDN, and EOH features in series for recognition. Since all of the three features belong to the texture feature-based method, the facial expression features can not be represented in multiple ways. Besides, the three features are only fused simply, and no deep information is discovered. Some deep learning approaches are introduced and compared, such as CNN, DNN, and FRR-CNN in [26, 29] and [30]. They are trained with our training data for a fair comparison. The results in Table 5 show that our approach performs better than others. The proposed method fulfills the extraction of texture, geometric, and semantic information by different features, which are beneficial to the recognition of expressions.

Table 5

Comparison with state-of-the-art approaches (%)

Approaches	JAFFE	CK+
GLTP [13]	93.25	91.64
LBP+LDN+EOH [25]	80.35	79.26
BOW+HOG[20]	91.3	90.15
Geometric features+LBP [42]	N/A	90.08
PHOG+LBP [43]	87.43	94.63
CNN [26]	91.23	89.35
WMDNN [29]	93.56	91.93
FRR-CNN [30]	94.52	92.73
DNN [39]	N/A	91.70
Boosted DBN [44]	N/A	96.70
HOG +DBN+Gabor+SAE [40]	N/A	91.11
DASE [41]	N/A	95.79
Proposed method	98.76	97.58

5 Conclusions

In this paper, we have presented a novel approach for FER by feature fusion method, which can recognize the expressions with high accuracy. Particularly, a novel texture feature extraction method PD-LDN has been proposed to get robust and low-dimension features. Both the handcrafted feature including texture feature, geometric feature, and semantic feature have been introduced to compose a multi-feature and fused by an autoencoder network to identify the expressions successfully by learning discriminative features from images. In the end, the experiment results have demonstrated that the proposed approach outperforms the other state-of-the-art methods in terms of the 7-class expression recognition.

References

Donato

, Bartlett

M.S.

and Hager

J.C.

, Classifying facial actions, IEEE Transactions on Pattern Analysis and Machine Intelligence 21(10) (1999), 974–989.

Stöckli

, Schulte-Mecklenbeck

and Borer

, Facial expression analysis with AFFDEX and FACET: A validation study, Behavior Research Methods 50(4) (2018), 1446–1460.

Danelakis

, Theoharis

and Pratikakis

, A survey on facial expression recognition in 3D video sequences, Multimedia Tools and Applications 74(15) (2015), 5577–5615.

Khan

S.A.

, Hussain

and Usman

, Facial expression recognition on real world face images using intelligent techniques: A survey, Optik 127(15) (2016), 6195–6203.

Holder

R.P.

and Tapamo

J.R.

, Improved gradient local ternary patterns for facial expression recognition, EURASIP Journal on Image and Video Processing 17(1) (2017), 42–49.

Liu

, Li

and Ma

, Facial expression recognition with fusion features extracted from salient facial areas, Sensors 17(4) (2017), 712–718.

Zhang

, Zheng

and Cui

, A deep neural network-driven feature learning method for multi-view facial expression recognition, IEEE Transactions on Multimedia 18(12) (2016), 2528–2536.

Chen

, Takiguchi

and Ariki

, Rotation-reversal invariant HOG cascade for facial expression recognition, Signal, Image and Video Processing 11(8) (2017), 1485–1492.

Zhao

, Liang

and Liu

, Peak-piloted deep network for facial expression recognition, European Conference on Computer Vision (2016), 425–442.

10.

Kwon

, Kim

and Lee

, ROI analysis for remote photoplethysmography on facial video, 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (2015), 4938–4941.

11.

Jabid

, Kabir

M.H.

and Chae

, Facial expression recognition using local directional pattern, IEEE International Conference on Image Processing (2010), 1605–1608.

12.

Wang

and Ying

, Facial expression recognition based on local phase quantization and sparse representation, 8th International Conference on Natural Computation (2012), 222–225.

13.

Shokoohi

, Bahmanjeh

and Faez

, Expression recognition using directional gradient local pattern and gradient-based ternary texture patterns, 2nd International Conference on Pattern Recognition and Image Analysis (2015), 1–7.

14.

Ding

, Zhao

and Li

, Facial expression recognition from image based on LBP and Taylor expansion, IEEE Access 5(3) (2017), 19409–19419.

15.

, Xiang

and Venkatesh

Y.V.

, Facial expression recognition using radial encoding of local Gabor features and classifier synthesis, Pattern Recognition 45(1) (2012), 80–91.

16.

Khan

S.A.

, Hussain

and Usman

, Facial expression recognition on real world face images using intelligent techniques: A survey, Optik 127(15) (2016), 6195–6203.

17.

Durmuşoğlu

and Kahraman

, Facial expression recognition using geometric features, International Conference on Systems, Signals and Image Processing (2016), 1–5.

18.

Ghimire

and Lee

, Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines, Sensors 13(16) (2013), 7714–7734.

19.

B.F.

, Huang

Y.C.

and Chen

B.X

, A novel facial expression recognition method based on semantic knowledge of analytical hierarchy process, Journal of Image and Graphics 16(3) (2018), 420–426.

20.

Harrati

, Bouchrika

and Tari

, Automated classification of facial expressions using bag of visual words and texture-based features, 16th International Conference on Sciences and Techniques of Automatic Control and Computer Engineering (2015), 363–367.

21.

Kumari

, Rajesh

and Kumar

, Fusion of features for the effective facial expression recognition, International Conference on Communication and Signal Processing (2016), 0457–0461.

22.

, Teng

W.D.

and Wang

X.H.

, Facial expression recognition based on local texture and shape features, Journal of Electronics & Information Technology 40(6) (2018), 1338–1344.

23.

Chen

, Chen

and Chi

, Dynamic texture and geometry features for facial expression recognition in video, IEEE International Conference on Image Processing (2015), 4967–4971.

24.

Wang

X.H.

, Li

R.J.

and Hu

, Occluded facial expression recognition based on the fusion of local features, Journal of Image and Graphics 21(9) (2016), 1473–1482.

25.

Yang

, Ban

and Li

, Multiple features fusion for facial expression recognition based on ELM, International Journal of Embedded Systems 10(3) (2018), 181–187.

26.

Chen

, Yang

and Wang

, Convolution neural network for automatic facial expression recognition, International Conference on Applied SystemInnovation (2017), 814–817.

27.

Liu

, Han

and Meng

, Facial expression recognition via a boosted deep belief network, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014), 1805–1812.

28.

Lopes

A.T.

, Aguiar

and Souza

A.F.

, Facial expression recognition with convolutional neural networks: coping with few data and the training sample order, Pattern Recognition 61(8) (2017), 610–628.

29.

Yang

, Cao

and Ni

, Facial expression recognition using weighted mixture deep neural network based on double-channel facial images, IEEE Access 8(6) (2018), 4630–4640.

30.

Xie

and Hu

, Facial expression recognition with FRR-CNN, Electronics Letters 53(4) (2017), 235–237.

31.

Gunawan

A.S.

, Face expression detection on kinect using activeappearance model and fuzzy logic, Procedia Computer Science 59(5) (2015), 268–274.

32.

Rivera

A.R.

, Castillo

J.R.

and Chae

, Local directional number pattern for face analysis: Face and expression recognition, IEEE Transactions on Image Processing 22(5) (2013), 1740–1752.

33.

Güner

and Robinson , Edge detection by compass gradient mask, Computer Graphics and Image Processing 6(5) (1977), 492–501.

34.

, Imai

and Kaneko

, Face and expression recognition based on bag of words method considering holistic and local image features, Communications and Information Technologies (2010), 1–6.

35.

Blaschke

, Olivecrona

and Engkvist

, Application of generative autoencoder in denovo molecular design, Molecular Informatics 37(2) (2018), 170–175.

36.

Jiang

, Teng

and Liu

X.B.

, Diagnosis of electrical corrosion fault in wind turbine generator bearing based on vibration signal analysis, Electric Power 52(6) (2019), 128–133.

37.

Lyons

, Hamamatsu

and Kamachi

, Coding facial yexpressions with gabor wavelets, Proceedings 3rd IEEE International Conference on Automatic Face and Gesture Recognition (1998), 200–205.

38.

Lucey

, Cohn

J.F.

and Kanade

, The extended cohn-kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression, IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops (2010), 94–101.

39.

and Lam

, Facial expression recognition using deep neural networks, IEEE International Conference on Imaging Systems and Techniques (2015), 1–6.

40.

, Feng

and Xu

, Facial expression recognition via deep learning, International Conference on Smart Computing. IEEE Computer Society (2015), 303–308.

41.

Zeng

, Zhang

and Song

, Facial expression recognition via learning deep sparse autoencoders [J], Neurocomputing (2018), 643–649.

42.

Ghimire

, Jeong

, Yoon

, et al., Facial expression recognition based on region specific appearance and geometric features, 10th International Conference on Digital Information Management (2016), 142–147.

43.

Happy

S.L.

and Routray

, Robust facial expression classification using shape and appearance features, 8th International Conference on Advances in Pattern Recognition (2015), 1–5.

44.

Liu

, Han

, Meng

, et al., Facial expression recognition via a boost deep belief network, IEEE Conference on Computer Vision and Pattern Recognition (2015), 1805–1812.

Facial expression recognition with fused handcraft features based on pixel difference local directional number pattern

Abstract

Keywords

1 Introduction

2 Related work

3 Proposed method

3.1 Method review

4.1 Datasets

4.1.1 JAFFE

4.1.2 CK+

4.2 PD-LDN results and analysis

4.2.1 The impact of the block mode on PD-LDN

Table 1 Comparison of different block mode Block mode Recognition rate (%) JAFFE CK+ 3*3 78.35 74.33 4*4 79.61 77.69 5*5 80.47 79.15 6*6 85.14 83.25 8*8 79.26 78.45 Irregular mode 93.25 91.64

Table 3 Comparison with different texture feature-based approaches Approaches Recognition rate (%) JAFFE CK+ LBP [14] 81.35 79.33 LDP [11] 87.70 86.36 LPQ [12] 83.47 79.15 Gabor [15] 89.35 87.17 LDN [32] 91.60 89.30 PD-LDN 93.25 91.64

4.3.1 Parameter settings

Table 4 Comparison with the individual component of combination Approaches Recognition rate (%) JAFFE CK+ PD-LDN 93.25 91.64 Seven Order Moment(SOM) 80.35 79.26 BOW 91.23 89.35 PD-LDN+BOW 93.56 91.93 PD-LDN+SOM 94.52 92.73 SOM+BOW 91.3 90.15 PD-LDN+SOM+BOW 95.36 92.87 Proposed method 98.76 97.58

References

Table 1
Comparison of different block mode

Block mode Recognition rate (%)

JAFFE CK+

33 78.35 74.33

44 79.61 77.69

55 80.47 79.15

66 85.14 83.25

8*8 79.26 78.45

Irregular mode 93.25 91.64

Table 3
Comparison with different texture feature-based approaches

Approaches Recognition rate (%)

JAFFE CK+

LBP [14] 81.35 79.33

LDP [11] 87.70 86.36

LPQ [12] 83.47 79.15

Gabor [15] 89.35 87.17

LDN [32] 91.60 89.30

PD-LDN 93.25 91.64