Research on facial expression recognition based on wide attention and multi-scale fusion mechanism

Abstract

To solve the problem of limitations of current attention mechanisms in extracting key facial expression features and the problem of low accuracy of facial expression recognition due to insufficient consideration of feature information fusion in the receptive field by convolutional neural networks. In this paper, we propose a facial expression recognition network based on wide attention (WA) and a multi-scale fusion (MF) mechanism (wide attention and multi-scale fusion [WAMF]). WA by extracting the background information of facial expression images while focusing on texture information, thus achieving better feature extraction. The MF mechanism is added at the connection points of layers in ResNet, where features extracted from each upper layer are fused using different-sized convolutional kernels and input into the lower layer. Finally, a viewpoint-invariant Capsule Net is used as the classification network after receiving the feature maps. The proposed WAMF model was applied to two publicly available datasets, CK+ and Jaffe, achieving excellent recognition rates of 98.98% and 98.46%, respectively.

Keywords

facial expression recognition attention mechanism multi-scale fusion ResNet Capsule Net

1 Introduction

Human emotions refer to the genuine feelings experienced by an individual after being influenced by external factors, which can be expressed through various means such as facial expressions, gestures, voice, and text. Particularly, facial expressions convey up to 55% of emotional information (Pantic and Rothkrantz, 2000), making it a hot research area in the field of emotional recognition. Facial expressions are also the most natural and intuitive way to judge emotions. Facial expression recognition is a branch of computer vision that is widely used in real-world scenarios, such as fatigue driving (Jeong and Ko, 2018), assisted medical treatment (Li et al., 2019). The process of facial expression recognition includes three stages: facial expression image pre-processing, facial feature extraction, and expression classification. Currently, the basic methods for expression recognition are traditional methods and deep learning-based methods, and the distinction is based on whether or not a deep learning method is used.

Traditional methods for extracting facial expression features include hidden Markov models (Bystroff et al., 2000), bayesian classification (Vinay et al., 2018), and support vector machine feature classification (Suykens and Vandewalle, 1999). However, these methods have the limitations of difficult feature extraction and low recognition accuracy and researchers prefer deep learning research methods. After Hinton and Salakhutdinov (2006) proposed a method to solve the problem of gradient disappearance, deep learning methods have become widely used, and most researchers have turned to using deep learning methods. Deep learning techniques make it easier to extract features of facial expressions, and the accuracy of recognition has been greatly improved. Liu et al. (2021) proposed an expression recognition structure that combines long short-term memory (LSTM) networks and capsule networks, which consists of three parts: capsule encoder, capsule decoder, and LSTM module. The LSTM module can effectively extract differences and features between adjacent video frames to improve the accuracy of video representation recognition. The combination of the capsule network and LSTM achieved good results in facial expression recognition in videos. Cao et al. (2020) proposed a double-enhanced Capsule Neural Network E2-Capsnet, which includes two feature enhancement modules. The first module uses action unit Attention to focus on the important part, multiple convolutional layers are added to the capsule site to form the second module that enhances feature representation, resulting in better recognition.

Recent advancements in the field, such as the work by Xiao et al. on dense knowledge-aware networks for multivariate time series classification (Xiao et al., 2024a), and deep transformer capsule mutual distillation for multivariate time series classification (Xiao et al., 2024c), highlight the importance of advanced feature extraction techniques. Although primarily focused on time series data, the principles of these methods can be adapted to improve facial expression recognition systems by enhancing feature learning capabilities. Furthermore, Xiao et al. (2024b) proposed a deep contrastive representation learning method with self-distillation, which integrates data augmentation, deep contrastive learning, and self-distillation to address deficiencies in existing contrastive learning algorithms. This approach can significantly improve the performance of time series classification and clustering, and its principles can be applied to enhance facial expression recognition.

While deep learning methods show excellent performance in facial expression recognition, facial expression images often contain interference information from non-expression regions which can greatly affect recognition accuracy. In recent years, attention mechanisms have become a hot topic in deep learning research, and many excellent facial expression recognition networks have used attention mechanisms, as they can simulate the learning mechanism of humans for new things, mainly by focusing neural networks on key target features and increasing the weight of these features to enhance recognition. Hu et al. (2018) proposed an attention mechanism that considers the channel aspect of the feature graph called the Squeeze and Excite Network (SENet) channel attention mechanism in 2017, which enhances the weights of important channels, making the network focus on different channel feature maps with different attention weights. The convolutional block attention module (CBAM), proposed by Woo et al. (2018) in 2018, overcomes the deficiencies of the SENet approach by extracting features from two dimensions of channels and space. Hou et al. (2021) proposed a relatively novel attention coordination mechanism in 2021 as a modification of CBAM to incorporate the coordinate location information of features into the channel attention during feature extraction, thereby capturing cross-channel information, directional perception, and position-sensitive information.

Additionally, the recent work by Tarek et al. (2023) on eye detection-based deep belief neural networks and the CapMatch method by Xiao et al. (2023) emphasize the significance of robust feature extraction techniques under varying conditions, which can also be relevant to improving the robustness of facial expression recognition systems. Furthermore, the approach by ElAraby and Shams (2021) on face retrieval systems and the work by Sarhan et al. (2020) on multipose face recognition highlight the need for adaptive and efficient algorithms in face-related recognition tasks, which can provide insights into improving facial expression recognition accuracy and robustness.

In order to more effectively capture facial expression features and fuse extracted features at a deeper level to enhance the representation of relevant features, this paper proposes a novel WAMF (wide attention and multi-scale fusion) network based on the WA (wide attention) and MF (multi-scale fusion) mechanisms for facial expression recognition.

The contributions of this study include the following:

We propose a WAMF network for facial expression recognition tasks, which innovatively integrates feature extraction and fusion steps, enhancing the capture of feature information.

The WA mechanism can attentively focus on various aspects of the image, including its horizontal and vertical dimensions, texture, and background. This in-depth and comprehensive attention mechanism effectively mines the information embedded in the image, demonstrating strong performance in the image feature extraction step.

We have devised an MF mechanism for ResNet, which integrates information from shallow and deep layers at various scales, facilitating feature interaction.

Comparative and ablation experiments were conducted on publicly available facial expression datasets CK+ and Jaffe, demonstrating the effectiveness and robustness of WAMF.

2 Related work

Since the widespread popularity of convolutional neural networks (CNNs) in deep learning, they can be seen frequently in the field of image processing. ResNet is a very well-known CNN structure proposed by He et al. (2016) in 2015, stands out among various CNNs due to its multi-level superposition advantage. Qian et al. (2022) designed an expression recognition network with a backbone network of ResNet34 by performing feature extraction from the head and tail part of the network, and Jiang and Huang (2022) used ResNet50 for feature extraction at the channel level of the image. While ResNet mitigates the vanishing and exploding gradient problems, it does not address the interaction of features between shallow and deep layers. Therefore, this paper introduces a MF mechanism, which considers feature fusion in ResNet from a multiscale perspective.

Attention mechanisms, as a hot topic in deep learning research, are closely intertwined with image processing tasks. In the context of facial expression tasks, various networks tend to segment facial image features in a somewhat similar manner, yet the effectiveness of different attention mechanisms varies significantly. Various attention mechanisms are widely used in facial expression recognition tasks. Qian et al. (2022) proposed a strong attention mechanism in 2021 by adding it before and after residual modules to strengthen the extraction of key information and achieved excellent results. Zhang et al. (2022) proposed an expression recognition network based on channel attention mechanisms, mainly focusing on enhancing key feature information representation through channel attention mechanisms. Jiang and Huang (2022) introduced a ResNet50 model with multiple feature layers that have a channel attention module for facial expression recognition, utilizing channel attention mechanisms to enhance key feature channels and improve recognition accuracy. However, most attention mechanisms are not sufficiently effective for feature extraction in facial expression recognition tasks, and some CNNs lack feature fusion at the receptive field level, leading to poor facial expression recognition accuracy. Hence, we have designed an attention module to enhance the feature extraction capability of the network.

3 Methodology

3.1 Overview

We propose a network called WAMF to address the issues of insufficient feature extraction and information fusion in facial expression recognition tasks. As shown in Figure 1, WAMF consists of three components: image preprocessing, an enhanced ResNet module, and a capsule network module. Firstly, we crop facial expression images to remove irrelevant information. Next, these cropped images are fed into an enhanced ResNet-18 network. In this improved ResNet-18 network, we introduce WA mechanisms in each layer to enhance the capture of feature information, and we incorporate MF mechanisms between every two layers to facilitate information fusion. Finally, the extracted feature maps are input into a capsule network with viewpoint invariance for classification, yielding the ultimate recognition results.

Figure 1.

WAMF network structure.

Figure 2.

WA structure diagram.

3.2 Wide attention

Maximum pooling mitigates feature extraction errors resulting from shifts in the estimated mean value due to convolutional layer parameter errors, thereby preserving more texture information. On the other hand, mean pooling reduces feature extraction errors associated with an increase in estimated variance due to neighborhood size limitations, allowing for the retention of more background image information.

The formula of max-pooling is given in equation (1)

y_{k i j} = max_{(p, q) \in R_{i j}} x_{k p q}

(1)

Equation (1) represents the output of max-pooling,

y_{k i j}

denotes the output value of the max-pooling rectangular region

R_{i j}

associated with the kth feature map, and

x_{k p q}

indicates an element located in a rectangular area

R_{i j}

(p, q)

, and

| R_{i j} |

denotes the number of elements of the kth feature map in its entirety. The max-pooling selects only the maximum value in each rectangular region, thus preserving more information about the image texture.

The formula of mean-pooling is given in equation (2)

y_{k i j} = \frac{1}{| R_{ij} |} \sum_{(p, q) \in R_{i j}} x_{k p q}

(2)

Equation (2) represents the output of mean-pooling,

y_{k i j}

denotes the output value after mean-pooling with the kth feature map,

x_{k p q}

denotes the element located in the feature map region

R_{i j}

(p, q)

, and

| R_{i j} |

denotes the number of all elements of the kth feature map. Unlike max-pooling, mean-pooling extracts all the available information in the feature map and forwards it to the next layer. Therefore, mean-pooling usually expresses a more comprehensive image feature.

WA is a novel attention mechanism introduced in this paper that effectively captures both background and texture information from images. Figure 2 illustrates the internal structure of WA. Specifically, the process begins by applying average pooling and max pooling separately along the spatial axes (horizontal and vertical) of the input feature maps. Next, the results of average pooling along the horizontal and vertical directions are recombined, as are the results of max pooling. These recombined results are then separated along both horizontal and vertical axes. Subsequently, the separated results along the horizontal axis are recombined, as are the results along the vertical axis. The final step involves combining these two recombined results to calculate the weight for each pixel in the feature map. These weights are then used to perform element-wise multiplication with the input feature map, resulting in the output feature map.

Given an input feature map of dimensions $H \times W \times C$ , Each pooling operation (average pooling and max pooling) along the horizontal and vertical directions has a computational complexity of $O (H \times W \times C)$ . The resulting weights from the WA mechanism are used for element-wise multiplication with the input feature map, also with a complexity of $O (H \times W \times C)$ . N denotes the batch size. C denotes the number of channels, H and W denote the height and width. The structure of WA is shown in Figure 2. The horizontal and vertical feature maps are concatenated and convoluted, achieving channel compression through convolution. The obtained feature map is then normalized and activated, separated into horizontal and vertical directions, and concatenated again. The separated horizontal direction feature map undergoes max-pooling and mean-pooling, while the separated vertical direction feature map experiences the same process. This process ensures that texture information and background information are extracted from the input image. Finally, The final feature map is obtained by convolving and activating the feature map. The WA algorithm is summarized as follows:

WA algorithm: Our proposed algorithm for the attention mechanism

Input:

X (N \times C \times H \times W)

Step 1:

X_{h_a v g} = A d a p t i v e A v g P o o l 2 d (X, (1, W))

X_{w_a v g} = A d a p t i v e A v g P o o l 2 d (X, (H, 1))

X_{h_\max} = A d a p t i v e A v g P o o l 2 d (X, (1, W))

X_{w_\max} = A d a p t i v e A v g P o o l 2 d (X, (H, 1))

Step 2:

X_{c a t_a v g} = c o n c a t (X_{h_a v g}, X_{w_a v g}, \dim = 2)

X_{c a t_max} = c o n c a t (X_{h_max}, X_{w_max}, \dim = 2)

Step 3:

o u t_a v g = h_s w i s h (B a t c h N o r m (C o n v (X_{c a t_a v g})))

o u t_m a x = h_s w i s h (B a t c h N o r m (C o n v (X_{c a t}_m a x)))

Step 4:

X_{h} = c o n c a t (s p l i t (o u t_a v g, H), s p l i t (o u t_max, H), \dim = 1)

X_{w} = c o n c a t (s p l i t (o u t_a v g, W), s p l i t (o u t_max, W), \dim = 1)

Step 5:

attention\_h = sigmoid (Conv (X_{h}))

attention\_w = sigmoid (Conv (X_{w}))

Step 6:

O u t p u t = attention\_h \times attention\_w \times I n p u t

3.3 Multi-scale fusion mechanism

MF refers to the process of sampling input signals at different granularities to obtain distinct features and then integrating them using a specific method to extract more information. In this paper, the MF designed for ResNet involves short-circuiting four layers of ResNet. This approach allows the lower layers to receive all the feature information from the upper layers, thereby achieving the goal of fusing a broader range of facial feature information while maintaining the original channel count and image size of ResNet.

In the original ResNet architecture, the outputs of each layer are partially connected to the intermediate output and serve as input to the subsequent layer through short connections. However, as the hierarchy deepens within ResNet, the size of feature maps gradually decreases. This connection method does not take into consideration the varying sizes of the feature maps extracted at different layers and the crucial facial key points. Instead, it employs a uniform 3 × 3 convolutional kernel size. Consequently, this approach may result in incomplete extraction of important features for certain expressions. Hence, we have devised an MF mechanism that enables all deeper layers to receive information carried by all shallower layers.

As illustrated in Figure 3, the specific approach of MF involves several steps. Firstly, the output feature map of the first layer is concatenated with the input and then subjected to a 1 × 1 convolution, ensuring that the output feature map maintains its original size and channel count. For the second layer, its input undergoes a 7 × 7 kernel convolution to enlarge its receptive field, followed by concatenation with the output of the second layer and convolution, maintaining an output channel count of 128. The output feature map of the second layer is then short-circuited with the intermediate input of the third layer, resulting in an input channel count of 256 for the third layer. In the third layer, the input feature map is convolved with a 5 × 5 kernel, followed by concatenation with the output feature map, keeping the channel count at 256. The output feature map of the third layer is short-circuited with the intermediate output of the fourth layer, resulting in an input channel count of 512 for the fourth layer. Finally, the input of the fourth layer undergoes a 3 × 3 kernel convolution, followed by concatenation and convolution with the output feature map, ensuring that the final output feature map maintains a channel count of 512.

Figure 3.

Structure of the layer after adding the MF mechanism.

Given an input feature map at layer l with dimensions $H_{l} \times W_{l} \times C_{l}$ , The computational complexity for a 1 × 1 convolution is $O (H_{l} \times W_{l} \times C_{l})$ . For the second layer, the complexity of the 7 × 7 convolution is $O (H_{2} \times W_{2} \times C_{2} \times 7^{2})$ . For the third layer, the complexity of the 5 × 5 convolution is $O (H_{3} \times W_{3} \times C_{3} \times 5^{2})$ . For the fourth layer, the complexity of the 3 × 3 convolution is $O (H_{4} \times W_{4} \times C_{4} \times 3^{2})$ . Each layer's feature map is concatenated with the input, followed by additional convolutional operations to maintain the original channel count and spatial dimensions. The overall complexity is influenced by the number of layers and the chosen kernel sizes, resulting in a cumulative complexity term proportional to the number of convolutional operations.

4 Experiments and analysis of results

4.1 Experimental environment

All experiments are based on Python 3.7 and CUDA version 11.0. The network is built using PyTorch 1.7.1 for training and testing on Windows 11 operating system. The hardware conditions used are an Intel i9-12900H CPU; 16GB of memory; and an NVIDIA RTX 3070 graphics card with a memory size of 8 GB.

4.2 Dataset

The datasets used in this paper are the Jaffe (Lyons et al., 1998) and CK+ (Lucey et al., 2010), which are collected in a laboratory environment. The Jaffe dataset contains 213 images of facial expressions of 7 emotional states performed by 10 Japanese female students: anger (30 images), disgust (29 images), fear (32 images), happiness (31 images), neutral (30 images), sadness (31 images), and surprise (30 images). Figure 4 shows the sample images of expressions from the Jaffe dataset, and Figure 5 shows the data distribution of the seven expressions from Jaffe. The CK+ dataset is obtained by expanding the Cohn-Kanade dataset, and it includes 123 participants, 69% of whom are women, 81% are European-American, 13% are African-American, and 6% are from other ethnic groups. The CK+ dataset consists of 593 image sequences contributed by the participants, and 327 of these sequences are selected with labeled information. We selected three peak frames from the labeled sequences and ended up with 981 images of facial expressions, containing seven emotions: anger (135 images), contempt (54 images), disgust (177 images), fear (75 images), happiness (207 images), sadness (84 images), and surprise (249 images). Figure 6 shows the sample images of expressions from the CK+ dataset, and Figure 7 shows the data distribution of the seven expressions from CK+.

Figure 4.

Example of Jaffe dataset.

Figure 5.

Jaffe data distribution.

Figure 6.

Example of CK+ dataset.

Figure 7.

CK+ data distribution.

We divided CK+ and Jaffe into training and test sets in the ratio of 7:3. To reduce the interference of non-face areas in the images, we cropped the center regions of each image for face extraction, as it is more beneficial for feature extraction and emotion analysis, and reduces the interference of irrelevant features. Before training, the data was unified for normalization to facilitate better gradient diffusion and faster model convergence. In the experiments, the learning rate was set to 0.001, batch size was 16. For each experiment, we trained for 200 epochs.

4.3 Evaluation indicators

The evaluation metrics for this experiment are standardized measures used in image classification, including $P r e c i s i o n$ , $R e c a l l$ , $F 1$ , and $A c c u r a c y$ . The accuracy of the predicted correct positive sample is denoted as $P r e c i s i o n$ , and can be calculated using equation (3). The coverage of the predicted correct positive sample is denoted as $R e c a l l$ , and can be calculated using equation (4). The summed average of precision and recall is denoted as $F 1$ , and can be calculated using equation (5). The proportion of the predicted correct sample is denoted as $A c c u r a c y$ , and can be calculated using equation (6).

\begin{aligned} P r e c i s i o n & = \frac{T P}{T P + F P} \end{aligned}

(3)

\begin{aligned} R e c a l l & = \frac{T P}{T P + F N} \end{aligned}

(4)

\begin{aligned} F 1 & = \frac{2 P}{2 T P + F P + F N} \end{aligned}

(5)

\begin{aligned} A c c u r a c y & = \frac{T P + T N}{T P + T N + F P + F N} \end{aligned}

(6)

T P

represents positive samples with a correct prediction,

F P

negative samples representing prediction errors,

F N

represents positive samples representing prediction errors, and

T N

represents negative samples with correct prediction.

4.4 Performance comparison and analysis

In this section, we first conducted a comparison experiment and obtained the accuracy of WAMF on the CK+ dataset as 98.98%, and on the Jaffe dataset as 98.46%. Figures 8 and 9 show the final accuracy curves and loss curves of WAMF on CK+ and Jaffe, respectively. We can see that WAMF has excellent fitting ability and fast convergence speed, and the accuracy increases with the number of rounds, while the loss reduces with the decrease of rounds. There is no occurrence of overfitting, which supports the conclusion that the network designed in this study has a strong generalization ability for facial expression recognition tasks. Second, we conducted an accuracy comparison experiment to compare the performance of WAMF with other methods proposed in other papers on the CK+ and Jaffe datasets. The experimental results show that the WAMF method has better performance on these two datasets compared to the previous methods.

Figure 8.

(a) Accuracy on CK+, (b) loss on CK+.

Figure 9.

(a) Accuracy on CK+, (b) loss on CK+.

To prove the performance of WAMF, we conducted a comparative analysis with several existing networks. The classification results, presented in Tables 1 and 2, reveal the efficacy of different networks on the CK+ and Jaffe datasets, respectively.

Table 1.

Accuracy comparison with other models on the CK+ dataset.

Model	Accuracy (%)
Lopes et al. (2017)	96.76
DeRL (Yang et al., 2018)	97.30
FERAtt (Fernandez et al., 2019)	96.53
Li et al. (2020b)	97.38
Sadeghi and Raie (2019)	95.11
OAENet (Wang et al., 2021)	98.65
NTF_LRS (Liu et al., 2022)	95.57
DCNN (Mohan et al., 2021)	96.79
SCAN (Gera and Balasubramanian, 2021)	97.31
Rao et al. (2021)	98.68
WAMF (ours)	98.98

DCNN: deep convolution neural networks; DeRL: de expression residue learning; FERAtt: facial expression recognition with attention net; NTF-LRS: nonnegative tensor factorization method is proposed based on low-rank subspace; OAENet: oriented attention enable network; SCAN: spatio-channel attention net.

Table 2.

Accuracy comparison with other models on the Jaffe dataset.

Model	Accuracy (%)
ARLCP (Rubel et al., 2019)	94.41
SCNN (Khaliluzzaman et al., 2019)	93.02
Li et al. (without the attention module; Li et al., 2020a)	95.12
Avani et al. (2021)	97.48
WMDCNN (Zhang et al., 2020)	92.38
Comp-GAN-Cls (Wang et al., 2020)	96.20
SCAN (Gera and Balasubramanian, 2021)	56.33
HDG (Ayeche and Alti, 2021)	90.00
HDGG (Ayeche and Alti, 2021)	91.43
DCNN (Mohan et al., 2021)	95.63
MFF-CNN (Zou et al., 2022)	96.52
C-G-ECA-R34 (Qian et al., 2022)	97.65
WAMF (ours)	98.46

ARLCP: adaptive robust local complete pattern; C-G-ECA-R34: CBAM-global-efficient channel attention-ResNet34; COMP-GAN-CLS: compositional generative adversarial network classifier; DCNN: deep convolution neural networks; HDG: histogram of directional gradient; HDGG: histogram of directional gradient generalized; MFF-CNN: multi-feature fusion based convolutional neural network; SCAN: spatio-channel attention net; SCNN: shallow convolutional neural network; WMDCNN: weighted mixture deep convolution neural networks.

Table 1 illustrates the accuracy improvement of WAMF compared to other networks on the CK+ dataset, ranging from 0.30% to 3.87%. Notably, our proposed WA mechanism exhibits superior efficacy in key feature extraction in contrast to deep convolution neural networks (Mohan et al., 2021), which employs traditional feature extraction methods combined with deep learning for recognition. Additionally, the use of deep learning enables WAMF to extract shallow features more effectively. Rao et al. (2021) proposed a multi-scale graph CNN for facial expression recognition, the main approach of this network is to fuse the multi-scale structural graph generated by segmentation with the original graph for facial expression classification. However, this approach can only extract 98 facial features and may overlook smaller facial expression features. The WA mechanism can focus on the background and texture of the image with information and has better face feature extraction capability.

Table 2 demonstrates that the accuracy of WAMF in the Jaffe dataset surpasses that of other networks. Avani et al. (2021) divided the face region into four separate parts for feature extraction. In contrast, the MF mechanism based on ResNet, which we proposed in this paper, enables the fusion of features extracted at various scales. As a result, the information on individual feature components of facial expressions can be interpreted appropriately, thus improving the overall recognition accuracy and generalization ability of the model. Our research shows that employing ResNet18 as the backbone network yields optimal performance.

4.5 Ablation experiments

In this section, two sets of ablation experiments are designed to demonstrate the effectiveness of the proposed WA and MF mechanisms from the selection of network structure and the splitting of WAMF, respectively, and finally prove that WAMF is optimal, the experiments use the CK+ and Jaffe datasets on ResNet18 and ResNet34, respectively. After adding WA and MF, respectively, we compare the effectiveness of ResNet18 and ResNet34.

Tables 3 and 4 show the accuracy comparison of ResNet18 and ResNet34 on the CK+ dataset, and Tables 5 and 6 show the accuracy comparison of ResNet18 and ResNet34 on the Jaffe dataset. Table 3 shows that when ResNet34 is used as the backbone network, a maximum accuracy of 98.98% can be achieved on the CK+ dataset, whereas Table 4 indicates that when ResNet34 is used as the backbone network, a maximum accuracy of 98.06% can be achieved on the CK+ dataset, which is a decrease of 0.92% in comparison to the use of ResNet18. Whereas Table 5 shows that when ResNet18 is used as the backbone network, 98.46% accuracy can be achieved on the Jaffe dataset, Table 6 shows that when ResNet34 is used, only 96.92% recognition accuracy can be achieved on the Jaffe dataset, which is 1.54% lower compared to using ResNet18.

Table 3.
Accuracy of different models in the CK+ dataset under ResNet18.

Model Test accuracy (%)

ResNet18 + Capsule Net 91.86

+MF 93.56

+WA 97.97

WAMF 98.98

Model	Test accuracy (%)
ResNet18 + Capsule Net	91.86
+MF	93.56
+WA	97.97
WAMF	98.98

Table 4.

Accuracy of different models in the CK+ dataset under ResNet34.

Model	Test accuracy (%)
ResNet34 + Capsule Net	92.54
+MF	94.24
+WA	95.93
WAMF	98.06

Table 5.

Accuracy of different models in the Jaffe dataset under ResNet18.

Model	Test accuracy (%)
ResNet18+ Capsule Net	92.31
+MF	93.85
+WA	96.92
WAMF	98.46

Table 6.

Accuracy of different models in the Jaffe dataset under ResNet34.

Model	Test accuracy (%)
ResNet18 + Capsule Net	90.77
+MF	92.31
+WA	95.38
WAMF	96.92

Figures 10 and 11 show the accuracy graphs we obtained using ResNet18 and ResNet34 on the CK+ and Jaffe datasets, respectively. From Figure 10, it can be seen that on the CK+ dataset, when using ResNet34, the two ways: without adding any improvement and adding MF mechanism achieve higher accuracy than ResNet18, but when adding WA and using the WAMF method, ResNet18 achieves better results. Figure 11 shows that all four ways of ResNet18: without adding improvement, adding MF mechanism, adding WA, and using WAMF method on Jaffe dataset are due to ResNet34, so we choose ResNet18 as the backbone network.

Figure 10.

Accuracy of the two networks on CK+.

Figure 11.

Accuracy of the two networks on Jaffe.

Figure 12.

Confusion matrix for ablation experiments on the CK+ dataset. (a) Nothing, (b) MF, (c) WA, (d) WAMF.

Figure 13.

Confusion matrix for ablation experiments on the Jaffe dataset. (a) Nothing, (b) MF, (c) WA, (d) WAMF.

Figure 12(a) shows the confusion matrix obtained for ResNet18 and Capsule Net without adding any improvement mechanism, where 24 expressions were incorrectly identified. After adding the MF mechanism, the number of errors is reduced to 19, as shown in Figure 12(b). The WA mechanism, on the other hand, further improves the performance with six expressions incorrectly recognized, as can be seen in (c). The performance peaks when the WA and MF mechanisms are combined, as shown in Figure 12(d) with only three errors.

Figure 13 shows the confusion matrix obtained from our ablation experiments on the Jaffe dataset. As shown in Figure 13(a), a total of five expressions are incorrectly identified when using only the network with the combination of ResNet18 and Capsule Net. (b) Shows the results after adding MF, where one expression classified as “disgust” is correctly recognized compared to the network without any improvement. (c) Shows the results after adding WA, where there is a big improvement, with two expressions incorrectly classified. Figure 13(d) shows the results after using WAMF, where the best performance is achieved with only one expression misclassification.

Tables 7 and 8 show the precision, recall, and F1 obtained from WAMF experiments on the CK+ and Jaffe datasets, respectively, and it can be seen that WAMF achieves excellent results on the CK+ and Jaffe datasets. For the CK+ dataset, only three “sadness” expressions were misidentified as “contempt,” while only one “fear” expression was misidentified as “sadness” on Jaffe.

Table 7.

Precision, recall, and F1 under the CK+ dataset.

Expression category	Precision (%)	Recall (%)	F1 (%)
Anger	100	100	100
Contempt	84.2105	100	91.4286
Disgust	100	100	100
Fear	100	100	100
Happy	100	100	100
Sadness	100	85.7143	92.3077
Surprise	100	100	100

Table 8.

Precision, recall, and F1 under the Jaffe dataset.

Expression category	Precision (%)	Recall (%)	F1 (%)
Anger	100	100	100
Disgust	100	100	100
Fear	100	87.5	93.3333
Happy	100	100	100
Neutral	100	100	100
Sadness	88.8889	100	94.1177
Surprise	100	100	100

5 Conclusion

This paper presents a facial expression recognition network based on WA and MF mechanism. Firstly, the wide and comprehensive attention module facilitates the extraction of background and texture information, while the MF mechanism enables the fusion and interaction of deep and shallow features, leading to a more comprehensive understanding of facial expression features. WAMF achieves impressive results on publicly available facial expression datasets, achieving 98.98% accuracy on CK+ and 98.46% on Jaffe. The experimental results show that although WAMF has more excellent results in face expression recognition, the network structure is more complex, with more parameters, and the overall network is relatively bulky, so how to optimize the network for lightweight in future research is the next research direction.

Footnotes

ORCID iD

Daipeng Guo

Funding

This work is supported by the Education Department of Shaanxi Provincial Government Service Local Special Scientific Research Plan Project under grant number (No. 22JC037).

Conflict of interest

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Correction (May 2025):

Since the original publication, 2 references “Michael et al. 1998” and “Lucey et al. 2010” have been included in the article.

References

Avani

Shaila

Vadivel

(2021) Interval graph of facial regions with common intersection salient points for identifying and classifying facial expression. Multimedia Tools and Applications 80: 3367–3390.

Ayeche

Alti

(2021) HDG and HDGG: An extensible feature extraction descriptor for effective face and facial expressions recognition. Pattern Analysis and Applications 24: 1095–1110.

Bystroff

Thorsson

Baker

(2000) HMMSTR: A hidden Markov model for local sequence-structure correlations in proteins. Journal of Molecular Biology 301(1): 173–190.

Cao

Yao

(2020) E2-capsule neural networks for facial expression recognition using AU-aware attention. IET Image Processing 14(11): 2417–2424.

ElAraby

Shams

(2021) Face retrieval system based on elastic web crawler over cloud computing. Multimedia Tools and Applications 80: 11723–11738.

Fernandez

PDM

Peña

FAG

Ren

, et al. (2019) FERAtt: Facial expression recognition with attention net. In: 2019 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp.837–846. IEEE. DOI: https://doi.org/10.1109/CVPRW.2019.00112.

Gera

Balasubramanian

(2021) Landmark guidance independent spatio-channel attention and complementary context information based facial expression recognition. Pattern Recognition Letters 145: 58–66.

Zhang

Ren

, et al. (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp.770–778. IEEE. DOI: https://doi.org/10.1109/CVPR.2016.90.

Hinton

Salakhutdinov

(2006) Reducing the dimensionality of data with neural networks. Science 313(5786): 504–507.

10.

Hou

Zhou

Feng

(2021) Coordinate attention for efficient mobile network design. In: 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp.13708–13717. IEEE. DOI: https://doi.org/10.1109/CVPR46437.2021.01350.

11.

Shen

Sun

(2018) Squeeze-and-excitation networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp.7132–7141. IEEE. DOI: https://doi.org/10.1109/CVPR.2018.00745.

12.

Jeong

(2018) Driver’s facial expression recognition in real-time for safe driving. Sensors 18(12): 4270.

13.

Jiang

Huang

(2022) Improving expression recognition by fusing attention mechanisms with residual networks. Computer Technology and Development 32(5): 42–46 + 52.

14.

Khaliluzzaman

Pervin

Islam

, et al. (2019) Automatic facial expression recognition using shallow convolutional neural network. In: 2019 IEEE international conference on robotics, automation, artificial-intelligence and internet-of-things (RAAICON), pp.98–103. IEEE. DOI: https://doi.org/10.1109/RAAICON48939.2019.42.

15.

Mehta

Aneja

, et al. (2019) A facial affect analysis system for autism spectrum disorder. In: 2019 IEEE international conference on image processing (ICIP), pp.4549–4553. IEEE. DOI: https://doi.org/10.1109/ICIP.2019.8803604.

16.

Jin

Zhou

, et al. (2020a) Attention mechanism-based CNN for facial expression recognition. Neurocomputing 411: 340–350.

17.

Jin

Akram

, et al. (2020b) Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. The Visual Computer 36: 391–404.

18.

Liu

Long

, et al. (2021) An optimized Capsule-LSTM model for facial expression recognition with video sequences. Available at: https://arxiv.org/abs/2106.07564 (accessed 27 May 2021).

19.

Liu

Dai

, et al. (2022) Nonnegative tensor factorization based on low-rank subspace for facial expression recognition. Mobile Networks and Applications 27: 58–69.

20.

Lopes

Aguiar

Souza

, et al. (2017) Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order. Pattern Recognition 61: 610–628.

21.

Lucey

Cohn

Kanade

, et al. (2010) The extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE.

22.

Lyons

Akematsu

Kamachi

, et al. (1998) Coding facial expressions with Gabor wavelets. 3rd IEEE International Conference on Automatic Face and Gesture Recognition: 200–205. IEEE.

23.

Mohan

Seal

Krejcar

, et al. (2021) Facial expression recognition using local gravitational force descriptor-based deep convolution neural networks. IEEE Transactions on Instrumentation and Measurement 70: 1–12.

24.

Pantic

Rothkrantz

LJM

(2000) Expert system for automatic analysis of facial expressions. Image and Vision Computing 18(11): 881–905.

25.

Qian

Tian

, et al. (2022) Facial expression recognition based on strong attention mechanism and residual network. Multimedia Tools and Applications 82: 14287–14306.

26.

Rao

Wang

, et al. (2021) Facial expression recognition with multiscale graph convolutional networks. IEEE MultiMedia 28(2): 11–19.

27.

Rubel

Ahsan Chowdhury

Kabir

(2019) Facial expression recognition using adaptive robust local complete pattern. In: 2019 IEEE international conference on image processing (ICIP), pp.41–45. IEEE. DOI: https://doi.org/10.1109/ICIP.2019.8802911.

28.

Sadeghi

Raie

(2019) Human vision inspired feature extraction for facial expression recognition. Multimedia Tools and Applications 78: 30335–30353.

29.

Sarhan

Nasr

Shams

(2020) Multipose face recognition-based combined adaptive deep learning vector quantization. Computational Intelligence and Neuroscience 2020(1): 8821868.

30.

Suykens

JAK

Vandewalle

(1999) Least squares support vector machine classifiers. Neural Processing Letters 9: 293–300.

31.

Tarek

Shohieb

Elhady

, et al. (2023) Eye detection-based deep belief neural networks and speeded-up robust feature algorithm. Computer Systems Science and Engineering 45(3): 3195–3213.

32.

Vinay

Gupta

Bharadwaj

, et al. (2018) Unconstrained face recognition using Bayesian classification. Procedia Computer Science 143: 519–527.

33.

Wang

Sun

, et al. (2020) Learning to augment expressions for few-shot fine-grained facial expression recognition. Available at: https://arxiv.org/abs/2001.06144 (accessed January 2020).

34.

Wang

Zeng

Liu

, et al. (2021) OAENEt: Oriented attention ensemble for accurate facial expression recognition. Pattern Recognition 112: 107694.

35.

Woo

Park

Lee

(2018) CBAM: Convolutional block attention module. In: Computer vision – ECCV 2018, pp.3–19. Springer. DOI: https://doi.org/10.1007/978-3-030-01234-2_1.

36.

Xiao

Tong

, et al. (2023) Capmatch: Semi-supervised contrastive transformer capsule with feature-based knowledge distillation for human activity recognition. IEEE Transactions on Neural Networks and Learning Systems: 1–15. DOI: https://doi.org/10.1109/TNNLS.2023.3344294.

37.

Xiao

Xing

, et al. (2024a) Densely knowledge-aware network for multivariate time series classification. IEEE Transactions on Systems, Man, and Cybernetics: Systems 54(4): 2192–2204.

38.

Xiao

Xing

Zhao

, et al. (2024b) Deep contrastive representation learning with self-distillation. IEEE Transactions on Emerging Topics in Computational Intelligence 8(1): 3–15.

39.

Xiao

Xing

, et al. (2024c) DTCM: Deep transformer capsule mutual distillation for multivariate time series classification. IEEE Transactions on Cognitive and Developmental Systems 16(4): 1445–1461.

40.

Yang

Ciftci

Yin

(2018) Facial expression recognition by de-expression residue learning. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp.2168–2177. IEEE. DOI: https://doi.org/10.1109/CVPR.2018.00231.

41.

Zhang

Huang

Tian

(2020) Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture. Pattern Recognition Letters 131: 128–134.

42.

Zhang

Kong

Teng

(2022) Face expression recognition based on multi-scale feature attention mechanism. Computer Engineering and Applications 58(1): 182–189.

43.

Zou

Zhang

Lee

(2022) A new multi-feature fusion based convolutional neural network for facial expression recognition. Applied Intelligence 52: 2918–2929.

Research on facial expression recognition based on wide attention and multi-scale fusion mechanism

Abstract

Keywords

1 Introduction

2 Related work

3 Methodology

3.1 Overview

4.1 Experimental environment

4.2 Dataset

Table 3. Accuracy of different models in the CK+ dataset under ResNet18. Model Test accuracy (%) ResNet18 + Capsule Net 91.86 +MF 93.56 +WA 97.97 WAMF 98.98

Footnotes

ORCID iD

Funding

Conflict of interest

Correction (May 2025):

References

Table 3.
Accuracy of different models in the CK+ dataset under ResNet18.

Model Test accuracy (%)

ResNet18 + Capsule Net 91.86

+MF 93.56

+WA 97.97

WAMF 98.98