Real-time facial expression recognition using smoothed deep neural network ensemble

1. Introduction

The evolution of robotics and artificial intelligence has taken a qualitative leap in domains such as dynamic stability [1], autonomous navigation [2], or speech recognition [3] among others, in the last decade. This has allowed social robots, which must be designed based on human-oriented perception, to behave and adapt to complex scenarios in more advanced manners [4], involving emotional feedback during human-robot interaction (HRI) [5]. Emotion estimation is a key for such applications and by improving its performance accuracy, enhances the living conditions of several social groups such as elders with mental diseases [6], autistic [7], and hospitalized children and their families [8], even open up to other categories.

Emotion estimation is a complex task [9], and despite the variability of emotional responses and their dependence on culture [10], there is a need to narrow down the reference system for experimental research by creating an emotional model. Based on psychological studies, emotions are generally divided into six prototypical categories [11]: anger, disgust, fear, happiness, sadness, and surprise; which the neutral emotion is defined as a baseline. Emotion recognition systems are developed based on several modalities such as: facial expressions [12, 13, 14], body gesture [15], speech [16], and physiological signals [17, 18].

Facial emotion recognition (FER) systems can be designed using two different strategies: Image-based and video-based. Although, video-based systems carry a high amount of emotional information due to their dynamic properties, they require a more complex analysis which is still complicated to implement in real-time systems. For this reason, this methodology was tested first using a simplified approach on static images; however, it is worth mentioning that FER in these conditions, stills a challenging task due to multiple environment variations such as brightness, occlusions, or body posture, which can affect the information available in the processed image.

Globally, FER image-based systems consist of three major stages: face detection and pre-processing, feature extraction, and classification. The first stage aims to detect the region of interest (face) in goal of removing unnecessary information to recognize the facial expression such as the background; it becomes challenging when acquisition conditions are poor mainly the lighting. The feature extraction leads to represent the pre-processed image with a characteristic vector including the relevant features. In FER, feature extraction techniques could be divided into two main categories: appearance-based and geometric-based methods. The first methods aim to detect textural information from face images using well-known feature extractors such as Gabor filters [19] and local binary patterns (LBP) [20]. Strategies based on geometric properties use overall the detected facial landmarks to encode the geometric information like angles and distances. The extracted features are then used for classification to recognize facial emotions.

In the last few years, the use of deep learning in several applications [21, 22] has become possible, since powerful graphics processing units (GPUs) are now available. Compared to traditional methods based on the aforementioned feature extraction techniques, convolutional neural networks (CNN) based techniques have achieved the FER state of the art recognition rates [14]. However, due to the training process’s stochastic properties, different sets of weights are found on each trained neural network; therefore, an intuition could lead to train several models to combine their predictions as an ensemble, and subsequently, may lead to a reduction of the generalization error in most cases since each single CNN extracts discriminative features for a specific task, as shown in our published paper [23].

However, it is worth mentioning that training FER models face two major challenges. First, environment variations such as brightness, occlusions, or body posture, can affect the information available to recognize a facial expression, and secondly, most of databases designed for training are prelabeled by humans, which contain inherent bias due to emotion perception complexity, as shown in Fig. 1. Thus, train a model with a wrong target class may lead to learn incorrect features and subsequently decrease the accuracy performance and generalization capabilities.

OPEN IN VIEWER

This paper extends the CNN ensembling method for facial emotion recognition proposed in our previous work [23], by including the label smoothing technique to deal with the miss-labelled data. Additional computation is provided through three different hardware setup, with two possible face detection systems depending on the image acquisition complexity respect to environment variation. The methodology was tested on three challenging databases, collected in real-world conditions and in unconstrained manner, FER 2013 [24], SFEW 2.0 [25], and ExpW [26].

The rest of the paper is organized as follows: Section 2 presents the related works, the proposed method’s stages are described in details in Section 3, followed by the experimental results as well as discussions in the fourth section. Conclusions and perspectives are drawn in Section 5.

In the facial emotion recognition field, several research works have been carried out to improve the emotion estimation accuracy. The use of deep learning has outperformed the conventional approaches that use the traditional feature extraction techniques; their training methodology can generally be divided into two main categories: single network learning and ensemble network learning.

The single network learning category uses only one architecture for the feature extraction and the recognition task. For example, Mollahosseini et al. [27] designed a CNN architecture of two convolutional layers and four inception layers for FER. In [28], they used a deep neural network with relativity learning (DNNRL) of three convolutional layers followed by three inception layers and trained by the triple loss. A multi-task convolutional network for simultaneous facial landmarks detection and facial expression recognition, is proposed in [29]. Tang et al. [30] proposed the DLSVM by replacing the softmax layer with a linear support vector machine (SVM) for FER application. Lian et al. [31] studied the contribution of each face region in the facial expression recognition using the class activation mapping technique (CAM) and a visualization model based on the Densenet-BC architecture.

The ensemble network learning methods adopt the ‘divide and conquer’ approach; the input image is fed through a set of deep architectures and the final decision is obtained by the combination of each network score output. Overall, the ensemble network learning methods achieve a better performance compared to the single learning category, since each network extracts different relevant features and may lead to minimizing the generalization error; however, this methodology requires additional computing time, given that for the same task, multiple networks are used compared to the first approach.

Levi et al. [32] proposed a novel method based on a deep ensemble network. First, by applying different radius parameters, LBP codes are extracted and mapped to a 3D space using multidimensional scaling (MDS); afterwards, the original RGB images and the mapped codes are used to train a set of 20 CNN models, and the predicted emotion is then obtained by the weighted average of each model’s output. Kim et al. [33] proposed to fuse information about non-aligned and aligned faces for FER. They introduced an alignment mapping network (AMN) to estimates aligned states of non-alignable faces, then with a set of 9 deep convolutional neural networks (DCNs), the average or the majority voting rules are used to compute the emotion prediction. Similarly, Pramerdorfer et al. [34] trained an ensemble of 8 CNNs achieving the best state-of-the-art performance on the FER 2013 database [24].

Most of the previous studies do not take into account the computation time of the FER system and its integration capability in real devices; furthermore, the miss-labelled training data that drives the model to learn incorrect features are not studied. Similar to [32, 33, 34], we proposed in [23] a deep network ensemble including only three CNN, with a comparable accuracy performance.

In this paper, we propose two possible implementations, with/without dedicated GPU by providing additional computations to evaluate our proposed model under the real-time constraints and its integration capability for global use, according to three hardware configurations. In regards to the miss-labelled data, we propose to use the label smoothing technique [35] by reducing the confidence given to image labels during the training process.

OPEN IN VIEWER

The proposed framework is based on the following stages: Face detection and pre-processing, feature extraction and classification using deep neural networks, and label smoothing optimization followed by ensemble learning approach. Figure 2 shows a schematic overview of our proposed FER system.

To verify the validity of our method, we have carried out several experiments according to three phases. First, we trained our single models (A, B, C and D) on the FER 2013 database without label smoothing technique; afterwards, all possible ensembles have been built to investigate the ensembling generalization capability against single models. In the second phase, we have trained the single models using label smoothing with ϵ = 0.2, and compare their performances with the first models trained without label smoothing, and similarly, for the ensembles. Finally, according to a transfer learning approach, the trained single model’s weights on FER 2013 with/without label smoothing have been fine-tuned on the SFEW 2.0 and ExpW databases separately.

To keep our recognition performances regarding FER 2013 and SFEW 2.0 databases, comparable with those reported in the literature, we have trained the four single models with/without label smoothing using the same training subset according to each original holdout dataset split. Similarly, for the ExpW database since any particular split is initially provided, the FER 2013 ratio has been considered, 80% for training, 10% for validation, and 10% for test.

Special attention has been paid to the FER processing time in order to verify the integration capability in real devices, facing the real-time constraints. Three different hardware configurations have been tested: Multi-core i7 CPU without dedicated GPU, multi-core i7 CPU with a GeForce GTX 1080 GPU, and the Jetson Nano with a Tegra X1 GPU for embedded systems. The time processing was computed on each hardware and for each frame, through the facial detection and facial expression recognition stages independently. For the face detection step, the Viola-Jones and YOLO-v2 methods were tested. The experimental protocol was performed using an attached webcam with a resolution of 640 × 480 px.

All architectures have been implemented using keras framework, trained using the categorical cross entropy loss and Adam optimizer, with a batch size of 128 for 100 epoch each. We used the learning rate reducer and the early stopper callbacks to get the best model weights according to the validation set loss.

5. Conclusions

In this paper, we have proposed a framework for facial emotion recognition. It adopts the neural network ensemble learning with only three to four CNN architectures, for a better model generalization. Dividing and conquering has demonstrated to be a good strategy since the first stage deals only with the face detection and pre-processing in different conditions, and allow to focus the second stage only on inferring the emotional state; therefore, simplifying the whole process.

The label smoothing technique has clearly shown that reducing the confidence given to training labels improves the FER accuracy, and when combined with the ensembling learning, reaches the current state-of-the art performances according to three challenging databases, FER 2013, SFEW 2.0, and ExpW.

The proposed FER system provides an advanced solution to expand several computer vision applications and could improve human-robot interactions (HRI), mainly affective robotics, by rendering the robot accurately aware of the users’ emotional state. This could be used for therapeutic purposes such as autism’s and Alzheimer patients, and as a companion for elder people.

For future works, we plan to combine the findings on static images with the temporal information provided by videos to study in-depth the variations between the facial expression states, and further the emotional state provided by the speech intensity. This could lead to improve emotion recognition instead of using the images only.