Spatiotemporal features selection for spontaneous micro-expression recognition

Abstract

Micro-expressions reveal a person’s true feelings and motivations, and they can be so subtle that they are difficult to recognize. Micro-expression recognition can be used for criminal justice and national security. The existing methods usually extract the same micro-expression features from three different orthogonal planes, but cannot accurately reflect the characteristics of different space features, so the actual recognition rate is low. According to the motion and appearance information of micro-expression sequences in the temporal domain and spatial domain, this paper proposes a micro-expression recognition method based on spatiotemporal features selection. The Improved Local Directional Number Pattern (ILDNP) in the spatial domain and the Pyramid of Histograms of Orientation Gradients without Edge Extraction (PHOG-WEE) in the temporal domain are extracted individually and fused in the spatiotemporal domain to be a micro-expression feature, and then classified by the Support Vector Machine (SVM). Compared with the state-of-the-art LBP-TOP, HOG-TOP, HIGO-TOP, LBPSIP and Gabor algorithms on the SMIC, CASME and CASME II databases, experiments show that the proposed method has a better performance in describing the texture features of different planes, as well as a higher recognition rate on micro-expressions.

Keywords

Micro-expression spatiotemporal features PHOG-WEE ILDNP

1 Introduction

Face recognition and facial expression recognition have been widely studied in the field of pattern recognition [1 –4], and micro-expression is a special type of facial expression. The main difference between micro-expressions and ordinary facial expressions is that the former are spontaneous and thus cannot be subjectively controlled. They may be the result of emotional suppression and repression which a person may not be able to perceive themselves. Because of these characteristics, micro-expression recognition has received tremendous attention in the field of psychology, image process and pattern recognition. However, due to its short duration and low intensity, research of automatic micro-expression detection and recognition is still a great challenging task [5]. Compared with ordinary facial expression, micro-expressions usually appear in a specific part of the face, and do not appear in the upper half and lower half of the face simultaneously [6]. Therefore, the establishment of micro-expression databases is quite difficult. The existing micro-expression databases are USF-HD [7], Polikovsky [8], SMIC [9], CASME [10] and CASME II [11]. USF-HD and Polikovsky which are posed micro-expression databases built by participants imitating micro-expressions. SMIC, CASME and CASME II are databases of spontaneous micro-expressions established through a specific inducing mechanism. The establishment of these databases promotes the development of micro-expression recognition. On the SMIC, Zhao et al. [9] used the Temporal Interpolation Model (TIM) and Local Binary Pattern from Three Orthogonal Planes (LBP-TOP) to extract micro-expression features from the image sequences, and classified them by Support Vector Machine (SVM). As a result, the recognition rate was 48.78%. On the CASME II, Fu et al. [11] used Active Shape Model (ASM) and LBP-TOP to do the same experiments, and the recognition rate was 63.41%. These two are the baseline of the micro-expression recognition.

Micro-expression recognition methods usually extract optical flow features or the spatiotemporal features and then use the classifier to recognize micro-expressions. By calculating the optical flow field, the optical flow features are extracted which characterize the facial changes. The representative methods are the Main Directional Mean Optical-flow (MDMO) [12] and Facial Dynamics Map (FDM) [13]. The MDMO method extracts the optical flow field of each facial Region of Interest (ROI) and then uses the average optical flow in the main direction of the optical flow in each ROI as the feature to identify and classify micro-expressions. The FDM method calculates the dense optical flow field between two frames in the video sequence and corrects it in the horizontal and vertical directions, respectively. Furthermore, the principal direction of optical flow of each spatiotemporal cuboid is calculated iteratively, and the angle is quantified. Finally, the third-order tensor FDM is formed to represent the feature of micro-expressions. These methods based on optical flow are sensitive to illumination and have high computational complexity.

LBP-TOP is a typical method which extracts feature in the spatiotemporal domain, and it is an extension of Local Binary Pattern (LBP) from two-dimensional space to three-dimensional space. Besides the X-orientation and Y-orientation of a single frame image, a video or image sequence has an orientation along the axis of time, i.e., T-orientation. LBP-TOP extracts LBP features from three orthogonal planes (XY, XT and YT planes), and concatenates them to be a feature of micro-expressions. LBP-TOP not only uses the texture information of XY plane but also records the dynamic texture information of XT and YT planes. Based on LBP-TOP features, SVM, Random Forest (RF) and Multiple Kernel Learning (MKL) are used to detect and classify micro-expressions. This is the initial attempt at recognizing spontaneous micro-expressions [14]. Some researchers improved LBP-TOP on local descriptor methods [15 –17]. Because of the simplicity and efficiency of the local descriptor methods in the spatiotemporal domain, they have become the current leading research in micro-expression recognition.

In addition to the optical flow and spatiotemporal features, Zhang et al. [18] used Gabor to extract micro-expression features. The Principal Components Analysis (PCA) and Linear Discriminant Analysis (LDA) are used to reduce the feature dimensions for recognizing micro-expressions. Gabor can extract texture features on a multi-scale and in a multi-direction with a variety of coarse and fine granularity; it closely approximates to the human visual system in the frequency and direction, and Gabor is particularly suitable for texture representation and discrimination. But it is difficult for the human eye to recognize micro-expressions, so Gabor has limitations in this area. Moreover, the recognition efficiency is affected by large feature dimensions, because Gabor parameters setting is more complex. The existing hand-crafted features have a limitation in a sense, because they rely on some prior knowledge, Kim et al. [19] used Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) to learn the spatial and temporal features of micro-expressions. CNN and LSTM require a large number of samples for training and learning, so the existing micro-expression databases cannot meet this need.

Most of the spatiotemporal feature extraction methods simply extend a feature into three-dimensional space by extracting the same feature from the XY, XT, and YT planes, and concatenating them into a feature histogram with setting the same or different coefficients. For example, the radius, or the weight of LBP feature, is set differently in three planes. Such methods do not differentiate the information between the spatial domain (XY plane) and the temporal domain (XT and YT planes). For micro-expression recognition, single features cannot reflect the changes in all planes. In the image sequence, the spatial and the temporal domain reflect the appearance and motion information of the object, respectively. This paper proposes a spatiotemporal features selection method by fusing different features to recognize micro-expressions. The contribution of this paper is as follows:

Two kinds of features are proposed: the Pyramid of Histograms of Orientation Gradients without Edge Extraction (PHOG-WEE) from the XY plane and the Improved Local Directional Number Pattern (ILDNP) from the XT and YT planes, respectively.

PHOG-WEE reflects the appearance information, and ILDNP reflects the direction and intensity information, these two features from the three planes are concatenated into a single micro-expression feature histogram. The proposed method can reflect the changes in micro-expressions from the appearance of the face, the direction, and the intensity.

The remainder of the paper is organized as follows. Section 2 briefly reviews the related work. Section 3 introduces the proposed ILDNP and PHOG-WEE features. In Section 4, the micro-expression recognition method based on spatiotemporal features selection is described. Experiment results and analysis are given in Section 5. The paper is summarized with a conclusion in Section 6.

2 Related work

Micro-expression recognition methods based on spatiotemporal features generally extract the same features from three orthogonal planes and recognize them by classifiers.

Due to the short duration and low intensity of micro-expressions, the data is sparse, and so the video sequence is usually preprocessed and magnified by Eulerian Video Magnification (EVM) [20]. Different features are extracted on the preprocessed micro-expression sequences. After using EVM to magnify the sequences, Wang et al. [21] extracted LBP-TOP feature, and Li et al. [22] extracted Histograms of Oriented Gradients from Three Orthogonal Planes (HOG-TOP) and Histograms of Image Gradient Orientation from Three Orthogonal Planes (HIGO-TOP) features to recognize micro-expressions. According to the literature [15], other researchers did not use EVM but directly applied Robust Principal Component Analysis (RPCA) to extract sparse information, and then extracted Local Spatiotemporal Directional Features (LSTD) from the subtle motion video sequences. LSTD can extract more information in the spatiotemporal domain than LBP-TOP, but it considered only the horizontal and vertical intensity differences, without considering diagonal and other directions. However, the muscle motion on the human face is not always along horizontal or vertical directions.

As mentioned above, LBP-TOP is a simple and efficient feature for micro-expression recognition, but it only extracts sign information between pixels, without considering magnitude and direction information, and thus leads to a low recognition rate. Some researchers try different strategies to improve it. Huang et al. [16] proposed Completed Local Quantization Patterns (CLQP) by extracting the information about the sign, magnitude, and orientation from the center pixel and its neighboring pixels based on the LBP algorithm. Similar to LBP-TOP, the CLQP features from XY, XT, and YT planes are extracted and then concatenated into one feature histogram, which is called SpatioTemporal Completed Local Quantization Patterns (STCLQP). Then, vector quantization and codebook selection are used to code the three components of CLQP in each plane to analyze micro-expressions, the result is that STCLQP has a better performance. In the LBP-TOP, six intersections of three orthogonal planes are used twice and obtain redundant information. Wang et al. [17] proposed the algorithm of LBP with Six Intersection Points (LBP-SIP) to reduce the redundancy of LBP-TOP, the six intersections of three orthogonal planes are divided into a set of spatial neighborhood points and a set of temporal neighborhood points, and each point is used only once to extract the spatial and temporal texture features. With this method, the feature dimensions are significantly reduced and the time efficiency is improved.

Considering the shape information of micro-expressions, Huang et al. [23] proposed SpatioTemporal Local Binary Pattern with Integral Projection (STLBP-IP) by combining the integral projection with the LBP. STLBP-IP calculated horizontal and vertical integral projections on difference images for extracting shape attribute. One-dimensional local binary pattern (1D-LBP) is calculated on the integral projection in an image, and the LBP feature is extracted from the texture images which are obtained by formulating the horizontal integral projections from all difference images in a video clip. STLBP-IP not only preserves the shape attribute of the micro-expressions but also extracts the appearance and motion information, and enhances the micro-expression recognition ability. Further, Huang et al. proposed Discriminative Spatiotemporal Local Binary Pattern with Revisited Integral Projection (DiSTLBP-RIP) [24]. The DiSTLBP-RIP feature is extracted by RPCA to get subtle motion images, and then the group feature selection based on Laplacian method is used to improve the discriminating power of the features.

Combining LBP-TOP with other features is also a common method for micro-expression recognition. Liong et al. [25] used the Optical Strain Features (OSF) to describe the pixel-level motion features and the weighted block-based LBP-TOP to encode the dynamic texture features of face image blocks, then concatenated the two kinds of features to recognize micro-expressions.

Wang et al. [26] extracted the LBP-TOP in different color spaces and proposed the tensor independent color space (TICS), in which LBP-TOP features are extracted from the three color channels of TICS. The recognition rate of this method increases, in comparison with gray and RGB color spaces, but is no more superior to CIELab and CIELuv color spaces.

The existing micro-expression recognition methods based on spatiotemporal features extract the same feature in the spatial domain and the temporal domain, but cannot catch the real micro-expression changes, and thus they lead to a low recognition rate. Therefore, in this paper, a micro-expression recognition method based on spatiotemporal features selection was proposed which extracts different features in the spatial and temporal domain according to the different characteristics of micro-expressions.

3 Feature extraction

In this paper, two kinds of features are extracted to recognize micro-expressions: they are (1) the ILDNP, which extracts both the direction and intensity information from the XT, YT planes, and (2) the PHOG-WEE, which extracts appearance information from the XY plane.

3.1 ILDNP

LBP is a first-order derivative local descriptor, which encodes the relationship between a pixel and its neighboring pixels. LBP is not sensitive to direction, but it is susceptible to non-monotonic illumination, random noise, and pose. In order to overcome these drawbacks, Jabid et al. [27] improved LBP and proposed the Local Directional Pattern (LDP). There are three steps for LDP calculation: (1) Kirsch edge operators (see Fig. 1) are applied to each pixel to get eight edge response values; (2) the edge response values are arranged in descending order by their absolute value, and the first t edge response values are set to be 1, while the others are set to be 0; (3) a binary encoding of 8-bit is obtained by bit string coding. LDP uses the absolute value to code the edge response values with binary strings and ignores the information contained in the response sign, resulting in a loss of direction information. To solve these problems, Ramirez et al. [28] proposed a local descriptor, the Local Directional Number Pattern (LDNP), which encodes direction information of the facial texture in a more concise manner using the main direction number and its sign to form a 6-bit binary pattern. LDNP enables to encode different textures with similar structure patterns. Thus LDNP is widely used in facial expression recognition.

Fig.1

Kirsch edge operators in eight directions.

The calculation of LDNP is shown in Fig. 2, and the procedure includes two steps:

Fig.2

LDNP edge response values calculation. (a) LDNP computation; (b) The LDNP code of center pixel g_c.

Firstly, calculate the eight edge response values m₀ ∼ m₇ by convoluting the 3×3 window with the Kirsch edge operators M⁰ ∼ M⁷, then mark their positions as 0 to 7, as shown in Fig. 2(a);

Secondly, encode the position of the maximum positive edge response value maxIndex and the position of the minimum negative edge response value minIndex by the position information. The LDNP value is defined as follows: $LDNP (g_{c}) = 8 \times maxIndex + minIndex$ (1) where g_c is the central pixel. As shown in Fig. 2(b), the maximum value is 2340, which is position 7 (the binary value is 111), the minimum value is – 1444 which is position 4 (the binary value is 100), so the LDNP code and value of g_c is 111100 and 60, respectively.

LDNP implicitly stores the local direction microstructure information of the image by recording the position information of the principal edge responses, which is consistent with the subtle variation in the intensity of the micro-expressions. Therefore, LDNP is used to extract the micro-expression features in this paper. However, since LDNP only encodes the direction information, using this value to represent the central pixel value, it ignores the relationship of intensity information between the two principal direction pixels and the central pixel.

In this paper, LDNP is improved by fusing the direction and intensity information. The calculation of ILDNP consists of two parts: direction information ILDNP_direction and intensity information ILDNP_intensity. The calculation of the ILDNP_direction is the same as that of LDNP, as shown in Equation (2). ${ILDNP}_{direction} (g_{c}) = LDNP (g_{c})$ (2)

The ILDNP_intensity is calculated as follows: ${ILDNP}_{intensity} (g_{c}) = 2 \times U (d_{\max}) + U (d_{\min})$ (3)

Where U (x) is a Unit Step Function, d_max and d_min are the pixel difference between the maximum and minimum with the central pixel g_c, respectively.

There are two strategies when fusing the direction and intensity information: one is early fusion which is denoted as ILDNP_early, and the other is late fusion which is denoted as ILDNP_late.

Early fusion means fusing the direction and intensity features at the pixel level and concatenating the binary codes of the two features into a binary code. Late fusion brings together the direction and intensity features at the image level and computes the two histograms of the two features, respectively, and then concatenates them into a single histogram.

Early fusion is an 8-bit binary string which is obtained by concatenating the 6-bit binary string of direction information and 2-bit binary string of intensity information for each pixel. The ILDNP_early value of each pixel is calculated by the Equation (4), and its histogram feature is calculated by the Equation (5): $\begin{matrix} {ILDNP}_{early} (g_{c}) = 4 \times {ILDNP}_{direction} (g_{c}) \\ + {ILDNP}_{intensity} (g_{c}) \end{matrix}$ (4) $\begin{matrix} {Hist}_{early} (k_{1}) = \sum_{i = 1}^{H} \sum_{j = 1}^{W} f ({ILDNP}_{early} (g_{c}), k_{1}), \\ k_{1} \in [0, 255] \end{matrix}$ (5)

where

$\begin{matrix} f ({ILDNP}_{early} (g_{c}), k_{1}) \\ = {\begin{matrix} 1, & if {ILDNP}_{early} (g_{c}) = k_{1} \\ 0, & otherwise \end{matrix} \end{matrix}$ (6)

Where k₁ is the value of ILDNP_early (g_c) and k₁ ∈ [0, 255]. H and W are the height and width of images.

Late fusion calculates the direction and intensity information histogram by using the Equations (7 and 8), respectively and then concatenate them into a single feature histogram. $\begin{matrix} {Hist}_{direction} (k_{2}) = \sum_{i = 1}^{H} \sum_{j = 1}^{W} f ({ILDNP}_{direction} (g_{c}), k_{2}), \\ k_{2} \in [0, 63] \end{matrix}$ (7) $\begin{matrix} {Hist}_{intensity} (k_{3}) = \sum_{i = 1}^{H} \sum_{j = 1}^{W} f ({ILDNP}_{intensity} (g_{c}), k_{3}), \\ k_{3} \in [0, 3] \end{matrix}$ (8) where $\begin{matrix} f ({ILDNP}_{direction} (g_{c}), k_{2}) \\ = {\begin{matrix} 1, & if {ILDNP}_{direction} (g_{c}) = k_{2} \\ 0, & otherwise \end{matrix} \end{matrix}$ (9) $\begin{matrix} f ({ILDNP}_{intensity} (g_{c}), k_{3}) \\ = {\begin{matrix} 1, & if {ILDNP}_{intensity} (g_{c}) = k_{3} \\ 0, & otherwise \end{matrix} \end{matrix}$ (10) where k₂ and k₃ are the values of ILDNP_direction (g_c) and ILDNP_intensity (g_c), respectively.

ILDNP effectively exploits the intensity and direction difference information between the two principal pixels and the central pixel, so it can better describe the texture feature of micro-expressions. The efficiency of two fusion strategies are discussed in the experiments.

3.2 PHOG-WEE

Anna Bosch et al. [29] proposed Pyramid of Histograms of Orientation Gradients (PHOG), which described the shape attribute of objects with spatial and gradient information. The PHOG calculation principle is as follows:

Use Canny operator to calculate the edge of the image;

Divide the edge image into l levels (l = 0, 1, 2, …, L - 1), the original image in the lth level is divided into 2^l × 2^l block regions;

Calculate the orientation gradients using a 3×3 Sobel mask without Gaussian smoothing for each sub-region at each level. The orientations are in the range of [0°, 180°] or [0°, 360°], and are quantized the orientations into K bins. Each bin in the histogram depends on the gradient magnitudes that have orientations within a certain angular range. The final PHOG feature for the image is a concatenation of all the histograms.

PHOG describes the gradient feature and the shape attribute of the image. It is widely used in image retrieval, target detection, scene classification and other fields. Because PHOG can calculate the distribution of gradient direction at different levels, it can directly reveal the local and spatial gradient structure information of the facial image, and the gradient information is not sensitive to illumination change. Therefore, PHOG is used to extract the XY plane feature of micro-expressions in this paper. Due to the subtle variation of micro-expressions, it is difficult to distinguish by shape features, so the Canny edge detection in step 1 is removed when calculating the PHOG, namely PHOG-WEE. Then pyramid segmentation is performed on each facial micro-expression image, and the gradient features of all pixels are calculated in each sub-region to obtain the PHOG-WEE feature of micro-expressions.

4 Spatiotemporal features selection

Micro-expression recognition usually has three steps: preprocessing, feature extraction, and classification. In 2012, Wu et al. [20] applied spatial decomposition followed by temporal filtering for each frame to propose EVM algorithm, and the obtained signal was amplified to reveal hidden information. The EVM can capture tiny changes in the video that are invisible to the naked eye. In this paper, EVM algorithm is used to preprocess micro-expression videos and then the videos are decomposed into image sequences. The faces are cut out, and the median filter is further used to reduce the noise caused by EVM.

As mentioned above, micro-expression recognition methods based on spatiotemporal features, such as LBP-TOP, usually extend LBP to spatiotemporal three-dimensional space (XY, YT, and XT planes) [30]. However, in a micro-expression video sequence, the XY plane reflects the changes in the spatial domain, but the changes are not obvious. The XT and YT planes are the changes in the temporal domain. The motion feature has larger changes than that of the appearance feature. A single feature cannot reflect the appearance and motion characteristics at the same time. In this paper, the proposed method extracts the ILDNP feature for the motion feature and the PHOG-WEE feature for the appearance feature. Firstly, in the micro-expression sequence, the PHOG-WEE feature is extracted from the XY plane; the pyramid segmentation is performed on the facial expression images in the XY plane, and the gradient features of all the pixels in each sub-region are calculated to obtain the feature histogram of XY plane. Secondly, the ILDNP values of all pixels from the XT/YT plane of the micro-expression sequence are calculated by Equation (4), and the histogram of the XT/YT plane is calculated by Equation (5). Finally, the feature histograms of the three planes are concatenated into a single histogram as the feature of micro-expression sequence. The PHOG-WEE extracts appearance feature at different levels and describes the spatial layout feature, which has the reliable robustness to noise and rotation of the face.

The schematic diagram of the feature extraction method in this paper is shown in Fig. 3. Figure 3(a) is a micro-expression image sequence; the first frame is the XY plane which characterizes the appearance of the micro-expressions. The upper part of Fig. 3(b) is the XT plane which characterizes the horizontal movement of the micro-expressions when the Y-orientation is fixed. The lower part of Fig. 3(b) is the YT plane which characterizes the vertical movement of the micro-expressions when X-orientation is fixed. Figure 3(c) are the histograms of the XY, XT and YT planes, which are obtained by extracting PHOG-WEE feature from XY plane, ILDNP feature from XT and YT planes. And Fig. 3(d) is the concatenated feature histogram formed by the three feature histograms in Fig. 3(c). It can be seen that the XY plane is the appearance image of the human face. The XT plane is the image, which is formed by the pixels on each line of the XY plane varying with time, and the YT plane is the image, which is formed by the pixels on each column of the XY plane varying with time. The texture of the XY plane is different from the texture of the XT and YT planes. It is necessary to select different feature extraction methods for different planes. The concatenated feature histogram describes the appearance and motion information and provides more details for micro-expression recognition. Thus the proposed method is summarized in Algorithm 1.

Fig.3

Feature extraction of micro-expression based on spatiotemporal features selection. (a) A micro-expression image sequence; (b) The XT and the YT planes of the micro-expression image sequence; (c) The PHOG-WEE feature histogram of XY plane and ILDNP feature histograms of XT, YT planes; (d) The final concatenated feature histogram formed by the feature histograms of XY, XT and YT planes.

Algorithm 1

Algorithm 1 Spatiotemporal Features Selection for Spontaneous Micro-expression Recognition

Step 1:	Amplify the micro-expression database with EVM, and locate, crop, normalize and denoise these images. Group the preprocessed images into the training samples and testing samples.
Step 2:	Extract PHOG-WEE feature from the XY plane of the training samples and testing samples; then obtain the feature histogram of each sample.
Step 3:	Extract ILDNP feature from the XT and YT planes of the training samples and testing samples; then obtain the feature histograms of each sample.
Step 4:	Connect the three histograms obtained in Steps 2 and 3 as the final feature vectors.
Step 5:	Input the feature vectors of training samples to the SVM and select the best parameters.
Step 6:	Put the feature vectors of the testing samples into the trained SVM model for classifying and output the results.

5 Experiments

5.1 Databases

5.1.1 SMIC

The SMIC database contains 16 participants and 164 micro-expression samples, which are recorded by a high-speed camera with a resolution of 640×480 and a frame rate of 100 fps. The videos are divided into three categories: Positive, Surprise and Negative. Figure 4 shows a Negative micro-expression sequence in the SMIC database, using cropped faces. Obviously, the changes are very subtle to the human eye, and the naked eye hardly finds any changes in the sequence.

Fig.4

A Negative micro-expression sequence in SMIC.

5.1.2 CASME

The CASME database contains 195 samples of micro-expressions, which are labeled onset, apex and offset, as well as action unit (AUs) and micro-expression category. CASME is divided into two parts, CASME-A and CASME-B. The CASME-A is recorded by a BenQ M31 camera with a resolution of 1280×720 and a frame rate of 60 fps. The CASME-B is recorded by a Point Gray GRAS-03K2C camera with a resolution of 640×480 and a frame rate of 60 fps. The samples of CASME are divided into seven categories: Positive (Happiness), Negative (Sadness, Disgust, Fear), Surprise, Tense (Repression, Tense). Figure 5 is a Happiness micro-expression sequence in CASME, the first image is the onset frame, the 14th and 21st images are the apex frames, and the 26th image is the offset frame. For the human eye, there may be no difference between these images.

Fig.5

A Happiness micro-expression sequence in CASME.

5.1.3 CASME II

In 2014, Fu’s team of the Chinese Academy of Sciences also released an enhanced version of CASME, called CASME II, which contains 247 micro-expression samples recorded at a resolution of 640×480 and a frame rate of 200 fps, and also marks the onset, apex, offset, action unit (AUs) and micro-expression category. The micro-expression samples are divided into five categories: Happiness, Disgust, Surprise, Repression, and Others. Figure 6 is a micro-expression sequence of Repression in CASME II. As in Fig. 6, the first image is the onset frame, the 11th image is the apex frame, and the 26th image is the offset frame.

Fig.6

An Others micro-expression sequence in CASME II.

5.2 Experiments and discussion

The experiments are carried out from three aspects: (1) experiments on the fusion strategies when fusing direction and intensity information of ILDNP; (2) experiment parameters selection of PHOG-WEE; (3) experimental comparison on the proposed algorithm and LBP-TOP, LBP-SIP, Gabor, HOG-TOP, and HIGO-TOP. All experiments are on three publicly available databases SMIC, CASME and CASME II. In the experiments, micro-expression samples of each category are randomly selected: on third as the test set, the remaining two thirds as the training set, and the classifier is SVM. All experiments are in Windows 7 system, 6G RAM, processor for Intel (R) Core (TM) i5-2410M CPU @ 2.30GHz, on the MATLAB R2014b platform.

5.2.1 Fusion experiments of ILDNP

In order to validate the recognition effect of ILDNP on micro-expressions, experiments were carried out on LDNP-TOP and ILDNP-TOP with two fusion strategies, ILDNP_early-TOP, ILDNP_late-TOP. ILDNP_early-TOP means to combine direction and intensity features at the pixel level, while ILDNP_late-TOP refers to combine the direction and intensity features at the image level. The recognition rates are shown in Table 1.

Table 1
Recognition rates of LDNP-TOP, ILDNP_early-TOP, and ILDNP_late-TOP

Methods SMIC CASME CASME II

LDNP-TOP 54.39% 64.62% 74.39%

ILDNP_early-TOP 49.12% 72.31% 69.51%

ILDNP_late-TOP 56.14% 76.92% 75.61%

Methods	SMIC	CASME	CASME II
LDNP-TOP	54.39%	64.62%	74.39%
ILDNP_early-TOP	49.12%	72.31%	69.51%
ILDNP_late-TOP	56.14%	76.92%	75.61%

From Table 1 it can be seen that the late fusion method has the highest recognition in the three methods. Therefore, ILDNP_late is used in this paper.

To validate the recognition rate of ILDNP_late in different planes, the ILDNP_late feature is extracted from the XY, XT and YT planes, respectively. Table 2 is the recognition rate of ILDNP_late on the SMIC, CASME, CASME II with different planes.

Table 2

Recognition rates of ILDNP_late with different planes

SMIC	CASME	CASME II
XY	40.35%	70.77%	63.41%
XT	57.37%	75.38%	70.73%
YT	57.89%	75.38%	69.51%

It can be seen from Table 2 that on the three databases, the recognition rates of ILDNP_late on XT and YT planes are higher than that of XY plane, while the ILDNP_late performs relatively poor in the XY plane. So ILDNP_late is selected to extract features in XT and YT planes, not in XY plane.

5.2.2 PHOG-WEE parameters selection

In this paper, the angle of PHOG-WEE is set to [0°, 360°], the number of levels L and the number of quantified angle K need to be determined experimentally. Since PHOG-WEE is used to extract the XY plane feature, the experiments of the PHOG-WEE parameters are performed on the XY plane on CASME II. Table 3 shows the recognition rates of PHOG-WEE from the XY plane on the CASME II when L takes 0, 1, 2, 3, 4, 5 and K takes 3, 6, 9, 12, 15.

Table 3
Recognition rates of PHOG-WEE from the XY plane with different L and K

K\L 0 1 2 3 4 5

3 45.12% 58.54% 53.66% 63.41% 60.98% 60.98%

6 51.22% 50.00% 54.88% 56.10% 60.98% 60.98%

9 56.10% 60.98% 60.98% 60.98% 47.56% 46.34%

12 63.42% 65.85% 65.85% 67.07% 59.76% 59.76%

15 63.42% 58.54% 58.54% 58.54% 58.54% 58.54%

K\L	0	1	2	3	4	5
3	45.12%	58.54%	53.66%	63.41%	60.98%	60.98%
6	51.22%	50.00%	54.88%	56.10%	60.98%	60.98%
9	56.10%	60.98%	60.98%	60.98%	47.56%	46.34%
12	63.42%	65.85%	65.85%	67.07%	59.76%	59.76%
15	63.42%	58.54%	58.54%	58.54%	58.54%	58.54%

As can be seen from Table 3, when L is fixed and K = 12, the four highest recognition rates are obtained. When K is fixed and L = 3, the three highest recognition rates are obtained. The parameters L and K of the PHOG-WEE take 3 and 12, respectively.∥In order to validate the effectiveness of PHOG-WEE in the XY plane, this paper extracted the PHOG-WEE from the XY, XT and YT planes, respectively. Table 4 shows the recognition rates of PHOG-WEE with different planes.

Table 4

Recognition rates of PHOG-WEE with different planes

Plane\Database	SMIC	CASME	CASME II
XY	42.11%	72.31%	67.07%
XT	36.84%	49.23%	52.44%
YT	47.37%	63.01%	59.76%

From Table 4 it can be seen that on the CASME and CASME II, the recognition rate of the PHOG-WEE in the XY plane is higher than elsewhere. On the SMIC, although the recognition rate of PHOG-WEE in the XY plane is lower than that of the YT plane, it is higher than that of XT plane. From Tables 2 and 4, on SMIC, CASME and CASME II, the recognition rate of PHOG-WEE is higher than that of ILDNP from the XY plane. Therefore, PHOG-WEE is used to extract the XY plane features of micro-expressions.

5.2.3 Comparison with the state-of-the-art methods

To validate the effectiveness of the method, we add the confusion matrix in the experiments. Figure 7–9 are the confusion matrices of the proposed method on SMIC, CASME and CASME II, respectively. It can be seen that on the SMIC, it did not perform well in the Positive class which is the result of the broad classification in this database. On the CASME, the proposed method also has a poor recognition rate in the Positive class. Most of the Positive class has been recognized as an Others class, because the Others class represents uncertain emotions, and further inference is needed in CASME. On CAMSE II, the recognition rate in Disgust class is low. Most of the Disgust class is recognized as Surprise and Repression, because these three classes have similar expression changes. As a whole, good recognition rates are achieved.

Fig.7

Confusion matrix of the proposed method on SMIC.

Fig.8

Confusion matrix of the proposed method on CASME.

Fig.9

Confusion matrix of the proposed method on CASME II.

To further validate the effectiveness of the proposed method, it is compared with state-of-the-art micro-expression recognition methods. LBP-TOP [21] is a classical representative micro-expression recognition algorithm in the spatiotemporal domain. HOG-TOP [22] and HIGO-TOP [22] are micro-expression automatic recognition methods based on the gradient. LBP-SIP [17] is successful in many improved methods of LBP-TOP. The frequency and direction of Gabor filters are similar to those of simple cells in the human visual system, and so they are particularly suitable for texture representation and discrimination [18]. Since these methods do not publicly make available the source code, all the comparative methods adopt the same preprocessing as the proposed method in this paper and use SVM as a classifier. All codes are encoded by the authors. These codes are available at https://github.com/DuanHuiyan/CODES. Table 5 shows the comparison of the proposed method and other state-of-the-art methods on SMIC, CASME, and CASME II. It can be seen that on SMIC, the proposed method has the same recognition rate as LBP-TOP, and is higher than that of other algorithms. On CASME and CASME II, the proposed method has the highest recognition rate. LBP-TOP, HOG-TOP, and HIGO-TOP extend a feature into three-dimensional space and extract the same feature from XY, XT and YT planes, respectively, but the same feature may not be suitable for all planes. LBP-SIP only use the six intersections of three orthogonal planes which can be seen as four neighborhoods of LBP features from the XY plane and two neighborhoods of LBP features using the previous frame and the latter frame at the same location; thus some information in the temporal domain is lost. Gabor’s multi-scale and multi-direction cause high-dimensional micro-expression features, resulting in low classification efficiency. The proposed method extracts different features from the XY plane and XT, YT planes, and experiments show that the extraction of appearance and motion information by different features is beneficial to the recognition of micro-expressions.

Table 5

Comparison with other algorithms on SMIC, CASME, CASME II

Methods	SMIC	CASME	CASME II
LBP-TOP [21]	63.16%	72.31%	75.61%
LBPSIP [17]	54.39%	70.77%	71.95%
Gabor [18]	52.63%	75.38%	75.61%
HOG-TOP [22]	54.39%	66.15%	67.07%
HIGO-TOP [22]	56.92%	67.69%	73.17%
Proposed method	63.16%	81.54%	86.59%

6 Conclusion

The existing micro-expression recognition methods based on the spatiotemporal domain usually extract the same feature from three orthogonal planes and cannot reflect the motion and appearance feature. This paper proposes a new spatiotemporal features selection method where the spatial domain (XY plane) features and the temporal domain (XT and YT planes) features are extracted respectively. Because the XY plane describes the facial appearance, and the XT and YT planes describe the facial movement of the micro-expressions, such as the information about how the eyes, mouth, eyebrows, etc., change over time. PHOG-WEE is extracted from the XY plane and ILDNP which fuses direction and intensity information is extracted from the XT and YT planes. Then we use SVM for micro-expression recognition, achieving a higher recognition rate than existing methods. The method of calculating the edge response takes a long time in this paper. Also, the same feature is extracted from the XT and YT planes. However, the XT and YT planes express horizontal and vertical motion information, respectively and perhaps should be treated differently. Exploring new feature extraction methods and trying to extract novel feature extraction methods in three planes will be the focus of future research.

Footnotes

Acknowledgments

This work was granted by the National Natural Science Foundation of China (Grant No. 60302018), Tianjin Sci-tech Planning Projects (Grant No. 14RCGFGX00846), Tianjin Sci-tech Planning Projects (14JCYBJC18500), Tianjin Sci-tech Planning Projects (Grant No. 15ZCZDNC00130), Tianjin Sci-tech Planning Projects (Grant No. 17ZLZDZF00040) and the Natural Science Foundation of Hebei Province, China (Grant No. F2015202239). The authors would like to thank the editors and anonymous reviewers for their helpful comments and suggestions.

References

Zhai

, Zhao

, Ensemble of multiresolution probabilistic neural network classifiers with fuzzy integral for face recognition, Journal of intelligent & Fuzzy Systems 31(1) (2016), 405–414.

Nazir

, Jan

, Sajjad

, Facial expression recognition using weber discrete wavelet transform, Journal of Intelligent & Fuzzy Systems 33(1) (2017), 479–489.

Khan

S.A.

, Hussain

, Usman

, Reliable facial expression recognition for multi-scale images using weber local binary image based cosine transform features, Multimedia Tools & Applications 12 (2017), 1–33.

Munir

, Hussain

, Khan

S.A.

et al., Illumination invariant facial expression recognition using selected merged binary patterns for real world images, Optik - International Journal for Light and Electron Optics 158 (2018), 1016–1025.

Ben

, Yang

, Zhang

, Li

, Survey on automatic micro expression recognition methods, Journal of Computer-Aided Design & Computer Graphics 26(9) (2014), 1385–1395.

, Zhang

, Facial microexpression recognition: A survey, Acta Automatica Sinica 43(3) (2017), 333–348.

Shreve

, Godavarthy

, Goldgof

, Sarkar

, Macro- and micro-expression spotting in long videos using spatio-temporal strain, Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition and Workshops, 2011, pp.51–56.

Polikovsky

, Kameda

, OhtA

, Facial micro-expression detection in hi-speed video based on facial action coding system (facs), Ieice Transactions on Information & Systems 96(1) (2013), 81–92.

, Pfister

, Huang

, Zhao

, Pietikainen

, A Spontaneous Micro-expression Database: Inducement, collection and baseline, Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 2013, pp.1–6.

10.

Yan

W.-J.

, Wu

, Liu

Y.-J.

, Wang

S.-J.

, Fu

, CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces, Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 2013, pp. 1–7.

11.

Yan

W.-J.

, Li

, Wang

S.-J.

, Zhao

, Liu

Y.-J.

, Chen

Y.-H.

, Fu

, Casme ii: An improved spontaneous micro-expression database and the baseline evaluation, Plos One 9(1) (2014), e86041.

12.

Liu

Y.-J.

, Zhang

J.-K.

, Yan

W.-J.

, Wang

S.-J.

, Zhao

, Fu

, A main directional mean optical flow feature for spontaneous micro-expression recognition, IEEE Transactions on Affective Computing 7(4) (2016), 299–310.

13.

, Zhang

, Wang

J.Z.

, Microexpression identification and categorization using a facial dynamics map, IEEE Transactions on Affective Computing 8(2) (2017), 254–267.

14.

Pfister

, Li

, Zhao

, Pietikainen

, Recognising spontaneous facial micro-expressions, Proceedings of the IEEE International Conference on Computer Vision, 2011, pp. 1449–1456.

15.

Wang

S.-J.

, Yan

W.-J.

, Zhao

, Fu

, Zhou

C.-G.

, Micro-Expression Recognition Using Robust Principal Component Analysis and Local Spatiotemporal Directional Features, European Conference on Computer Vision, Proceedings of the ECCV 2014 – European Conference on Computer Vision, 2014, pp. 325–338.

16.

Huang

, Zhao

, Hong

, Zheng

, Pietikäinen

, Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns, Neurocomputing 175 (2016), 564–578.

17.

Wang

, See

, Phan

R.C.-W.

, Oh

Y.-H.

, Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition, Plos One 10(5) (2015), e0124674.

18.

Zhang

, Ben

, Yan

, Wu

, Guo

, Micro-expression recognition system, Optik - International Journal for Light and Electron Optics 127(3) (2016), 1395–1400.

19.

Kim

D.H.

, Baddar

W.J.

, Ro

Y.M.

, Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations, Proceedings of the 2016 ACM on Multimedia Conference (2016), pp. 382–386.

20.

H.-Y.

, Rubinstein

, Shih

, Durand

J.G.F.

, Freeman

, Eulerian video magnification for revealing subtle changes in the world, Acm Transactions on Graphics 31(4) (2012), 65.

21.

Wang

, See

, Oh

Y.-H.

, Phan

R.C.-W.

, Rahulamathavan

, Ling

H.-C.

, Tan

S.-W.

, Li

, Effective recognition of facial micro-expressions with video motion magnification, Multimedia Tools & Applications 76(20) (2016), 21665–21690.

22.

, Hong

, Moilanen

, Huang

, Pfister

, Zhao

, Pietikainen

, Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting and recognition methods, IEEE Transactions on Affective Computing (2017). 10.1109/taffc.2017.2667642.

23.

Huang

, Wang

S.-J.

, Zhao

, Piteikainen

, Facial Micro-Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection, Proceedings of the 2015 IEEE International Conference on Computer Vision Workshop, (2015), pp. 1–9.

24.

Huang

, Wang

S.-J.

, Liu

, Zhao

, Feng

, Pietikainen

, Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition, IEEE Transactions on Affective Computing (2017), 10.1109/taffc.2017.2713359.

25.

Liong

S.-T.

, See

, Phan

R.C.-W.

, Oh

Y.-H.

, Ngo

A.C.L.

, Wong

K.S.

, Tan

S.-W.

, Spontaneous subtle expression detection and recognition based on facial strain, Signal Processing Image Communication 47 (2016), 170–182.

26.

Wang

S.-J.

, Yan

W.-J.

, Li

, Zhao

, Zhou

C.-G.

, Fu

, Yang

, Tao

, Micro-expression recognition using color spaces, IEEE Transactions on Image Processing 24(12) (2015), 6034–6047.

27.

Jabid

, Kabir

M.H.

and Chae

Oksam

, Local Directional Pattern (LDP) for face recognition, Proceedings of the 2010 Digest of Technical Papers International Conference on Consumer Electronics, (2010), pp. 329–330.

28.

Rivera

A.R.

, Castillo

J.R.

, Chae

O.O.

, Local directional number pattern for face analysis: Face and expression recognition, IEEE Transactions on Image Processing 22(5) (2013), 1740–1752.

29.

Bosch

, Zisserman

, Munoz

, Representing shape with a spatial pyramid kernel, Proceedings of the 6th ACM International Conference on Image and Video Retrieval 2007, pp. 401–408.

30.

Zhao

, Pietikainen

, Dynamic texture recognition using local binary patterns with an application to facial expressions, IEEE Transactions on Pattern Analysis and Machine Intelligence 29(6) (2007), 915–928.

Spatiotemporal features selection for spontaneous micro-expression recognition

Abstract

Keywords

1 Introduction

2 Related work

3 Feature extraction

3.1 ILDNP

4 Spatiotemporal features selection

5.1 Databases

5.1.1 SMIC

5.2.1 Fusion experiments of ILDNP

Table 1 Recognition rates of LDNP-TOP, ILDNP early -TOP, and ILDNP late -TOP Methods SMIC CASME CASME II LDNP-TOP 54.39% 64.62% 74.39% ILDNP early -TOP 49.12% 72.31% 69.51% ILDNP late -TOP 56.14% 76.92% 75.61%

Footnotes

Acknowledgments

References

Table 1
Recognition rates of LDNP-TOP, ILDNP_early-TOP, and ILDNP_late-TOP

Methods SMIC CASME CASME II

LDNP-TOP 54.39% 64.62% 74.39%

ILDNP_early-TOP 49.12% 72.31% 69.51%

ILDNP_late-TOP 56.14% 76.92% 75.61%