A robust face emotion recognition approach through optimized SIFT features and adaptive deep belief neural network

Abstract

Facial expression is a significant indications for non verbal communication between individuals. The assignment of face emotion recognition is predominantly intricate for two reasons. Initial one is the non-existence of large database of training images and second issue is about classifying the emotions, which can be complex based on if the input image is static or not. Hence, this paper intends to propose a model, where two contributions are made both in the feature extraction and classification process. Initially, in feature extraction process, SIFT features are extracted as it is more associated with facial emotion features. However, numerous key point during SIFT extraction tends to provide redundant information. Hence, the key points are optimized using Whale Optimization Algorithm (WOA). Subsequently, the features extracted from the selected keypoints are multiplied with a weight factor, which has to be optimized by WOA. The resultant features are given to Deep Belief Network (DBN) for the classification of face emotion, in which the number of hidden neurons is also optimized by WOA along with feature weight. It is because of complexity that occurs with the utilization of both forward and reverse training in standard DBN model. Hence, after feature extraction and classification, the face emotion image is recognized accurately, while comparing the proposed Adaptive DBN (ADBN) using WOA over traditional classifiers and other optimization algorithms.

Keywords

Face emotion recognition feature extraction SIFT deep belief neural networks whale optimization

1. Introduction

Facial emotion recognition [1, 2] that plays a significant role in pattern recognition, human-computer interaction, and computer vision, is extensively exploited in surveillance systems, personalised healthcare, video games, multimedia, and humanoid service robots. In current studies, numerous algorithms focusing on face recognition, age, and gender estimation, and facial emotion classification have been expanded vastly [3, 4, 5, 45]. On the other hand, high dimensionality recognition is still a demanding problem for such applications. Facial emotion recognition is the most significant mode of human emotion expression [6]. For a couple of precedent decades, it has been a very imperative research field in the area of image recognition and computer vision. However, facial expression recognition remains as a demanding task [7, 8]. This is primarily connected with changing the environment, pose and lighting.

Facial emotion recognition [9, 10] in uncontrolled surroundings is an extremely demanding task owing to large intra-class distinctions caused by features such as occlusion, illumination, and head movement, and pose alterations [11]. The accurateness of a facial emotion recognition system usually depends on two significant characteristics: (a) facial features extraction which is robust beneath intra-class dissimilarities (for illustration, pose variations), however, are typical for a variety of emotions [12, 13, 53, 54, 55], and (b) model of a classifier that is proficient of differentiating dissimilar facial emotions dependent on imperfect and noisy data (for illustration, occlusion and illumination variations) [14, 15, 16].

In current years, DBN [17, 18, 19, 44] has obtained a rising consideration in artificial intelligence and machine learning, and numerous kinds of DBN associated algorithms have been productively deployed to image recognition tasks. Being diverse from a shallow learning structural design for nonlinear transformation [20], DBN techniques endeavour to discover abstract features in data at a high-level by employing hierarchical architectures [21], that have turn out to be an efficient approach for obtaining high-level characteristics from data. Nevertheless, conventional DBN experiences from inconveniences of learning effectiveness and computational complication [22, 23, 47].

Robust unbiased face recognition in real time remains as a foremost challenge for a variety of supervised learning dependent facial expression recognition techniques [24, 46]. This is owing to the actuality that supervised schemes cannot contain the entire appearance inconsistency across the faces regarding lighting, race, facial biases, pose, etc. in the restricted quantity of training data [25, 26, 43, 47]. In addition, dealing out each and every frame to categorize emotions is not necessary, as the user remains neutral for most of the time in common applications such as photo album or video chat/web browsing. Identifying neutral state at a beginning phase, thus passing those frames from emotion classification would accumulate the computational power [27, 28, 48, 49].

The paper contributes a facial emotion recognition model, in which two contributions are made in feature extraction and classification process. Primarily, in feature extraction process, SIFT features are extracted. Anyhow, several key points obtained throughout SIFT extraction tends to offer redundant information. Therefore, the key points are optimized by exploiting WOA scheme. Accordingly, the features extracted from the chosen keypoints are multiplied with a weight factor that has to be optimized by WOA. The optimal features are then subjected to DBN for the classification of face emotion, where the number of hidden neurons is further optimized by WOA together with feature weight. Moreover, the proposed ADBN method using WOA is compared with conventional classifiers, and other optimization algorithms such as Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Artificial Bee Colony (ABC), FireFly (FF) and Grey Wolf Optimization (GWO) and the results are obtained. In addition, the evaluation is done by comparing the proposed classification schemes with conventional schemes such as DBN, Neural Network (NN), Support Vector Machine (SVM) and Naives Bayes (NB) techniques and the results are attained in terms of performance measures. The paper is organized as follows. Section 2 analyzes the related works and reviews done under this topic. Section 3 describes the modeling of face emotion recognition and Section 4 portrays the results and discussion. Finally, Section 5 concludes the paper.

2. Literature review

2.1 Related works

Morita et al. [1] have introduced a scheme for detecting emotional faces for 100 ms to 34 patients with an matched and MDD controls. Stimulus faces were surrounded by a scarf and cap or by an Islamic headdress. Accordingly, the investigations were performed based speed on and accuracy. Results demonstrate that across fearful faces, groups were identified faster and with higher accuracy in the out-group than in the in-group condition. In addition, sadness was recognized more precisely in the out-group stipulation. In association, happy faces were more exactly (and tend to be quicker) recognized in the in-group state.

Liedtke et al. [2] have established an approach known as extreme sparse learning that has the capability to learn a group of basis (dictionary) and a nonlinear classification design in cooperation. The suggested scheme merges the discriminative authority of extreme learning mechanism with the modernization feature of sparse representation to facilitate precise classification when offered with noisy signals and defective data recorded in normal settings. Moreover, this paper has offered a novel local spatiotemporal descriptor which is pose-invariant and typical. The suggested structure was capable to attain the state-of-the-art identification accurateness on both spontaneous and acted facial emotion databases.

Shojaeilangari et al. [3] have presented a scheme where coupling was related with the functional connectivity among the medial prefrontal cortices and forward cingulate. The outcomes of the simulation has recommended that the condensed impact of surveillance on embarrassment persuaded by self-face images in individuals with ASD was associated to mutilation in the right anterior insula, that was concerned in producing prejudiced feelings, and the anterior cingulate cortex, that acts as a focal point for incorporating information from others throughout self-face assessment.

Wang et al. [4] have implemented a new intellectual emotion recognition arrangement. Accordingly, the proposed model employed stationary wavelet entropy to take out the features and deployed a hidden layer feed forward NN as the classifier. To avoid the training of the classifier drop into local optimum points, Jaya algorithm was established. The representation outcomes over a 20-subject 700-image dataset have demonstrated that the suggested model has attained an overall accurateness of 96.80 $\pm$ 0.14%. This implemented approach executes better than five conventional techniques in terms of overall precision.

Chen et al. [5] have proposed a Softmax regression-dependent deep sparse auto encoder network (SRDSAN) to identify facial emotion in human-robot communication. It endeavors to deal with large data in the deep learning output by means of SR. In addition, to prevail over gradient diffusion and local extrema inconveniences in the training procedure, the overall network weights were tuned finely to arrive at the overall optimum that makes the whole depth of the NN more forceful, thereby improving the presentation of facial emotion recognition. Outcomes illustrate that the average recognition accurateness of SRDSAN was superior to that of the convolutional NN and the SR.

Happy and Routray [6] have suggested a novel structural design for expression recognition by deploying appearance characteristics of chosen facial patches. A little major facial patches, depending on the location of facial landmarks, were extracted that were active throughout emotion elicitation. One-against-one classification technique was implemented by means of these characteristics. Furthermore, an automated learning-free facial landmark detection method has been suggested that accomplishes comparable performances as that of various conventional landmark detection techniques.

Chiranjeevi et al. [7] have suggested a light-weight neutral versus emotion classification mechanism that performs as a pre-processer to the conventional supervised emotion classification techniques. It enthusiastically learns neutral appearance at key emotion (KE) points by means of an arithmetical texture representation, build up by a set of reference frames for every user. The suggested system was robust to a variety of user head motions by associating with affine deformations dependent on a statistical texture design. The suggested technique, consequently, improves emotion recognition (ER) accurateness and concurrently reduces the computational complication of the ER system, as authenticated on numerous databases.

Zhang et al. [8] have implemented a facial expression recognition scheme with an alternative of evolutionary firefly algorithm for feature optimization. Initially, a customized Local Binary Pattern descriptor was implemented to generate a preliminary discriminative face demonstration. Moreover, a variant of the firefly algorithm was implemented to carry out feature optimization. From the evaluation, the implemented system attains a superior presentation and outperforms other traditional feature optimization techniques and associated facial expression recognition designs by a noteworthy margin.

2.2 Review

Table 1 shows the methods, features, and challenges of conventional techniques based on face emotion recognition protocols. At first, Pattern recognition was suggested in [1], where there was no correlation between accuracy of performance and the outcomes were stable. However, there was no consideration of personality styles in emotion platform. In addition, Extreme Learning Machine (ELM) was proposed in [2] which perform better during challenging scenarios with reduced classification cost. Anyhow, there was no contemplation of motion exaggeration schemes. NN was suggested in [3] that improves the coupling strength with minimized impact on embarrassment. However, the sample size was comparatively small, and there was gender variation in social sensitivities. Further, Jaya algorithm was suggested in [4] that offers increased accurateness, and it also includes various industrial and academic applications, but distortions arise if the photography is taken in an imperfect way. DNN was implemented in [5] that provide increased robustness with better efficiency and reduced complications. However, there were possibilities of gradient diffusion issues. LBP was suggested in [6] that offers accurate classification with reduced computational cost, but there was no accomplishment on real time recognition. In addition, LBP was presented in [7] that provide reduced performance complications, and it also minimizes the classification error. Anyhow, existence of issues related to non-availability of neutral frames. Finally, Moth-firefly optimization was proposed in [8] that handle better with illumination variations, and it also increases the convergence speed. However, there was no involvement on varied multimodal optimization issues. Thus these limitations are considered for enhancing the face emotion recognition algorithms efficiently in the recent research work.

Table 1
Features and challenges of face recognition using various techniques

Author [citation]	Adopted methodology	Features	Challenges
Morita et al. [1]	Pattern recognition	• No correlation between accuracy of performance. • The outcomes were stable.	• No consideration of personality styles in emotion platform.
Liedtke et al. [2]	ELM	• Performs better during challenging scenarios. • Reduced classification cost.	• No contemplation of motion exaggeration schemes.
Shojaeilangari et al. [3]	NN	• Improves the coupling strength. • Minimized impact on embarrassment.	• Sample size was comparatively small. • Gender variation in social sensitivities.
Wang et al. [4]	Jaya algorithm	• Increased accurateness. • Includes various industrial and academic applications.	• Distortions arises if the photography is taken in an imperfect way.
Chen et al. [5]	DNN	• Increased robustness. • Better efficiency. • Reduced complications.	• Possibility of gradient diffusion issues.
Happy and Routray [6]	LBP	• Offers accurate classification. • Reduced computational cost.	• No accomplishment on real time recognition.
Chiranjeevi et al. [7]	LBP	• Reduced performance complications. • Minimizes the classification error.	• Existence of issues related to non-availability of neutral frames.
Zhang et al. [8]	Moth-firefly optimization	• Handles better with illumination variations. • Increased convergence speed.	• No involvement on varied multimodal optimization issues.

Figure 1.

Overall architecture of the proposed face emotion recognition model.

3. Modelling proposed face emotion recognition

3.1 Proposed architecture

The overall framework of the proposed face emotion recognition model is given by Fig. 1. An image of an individual is taken for face emotion recognition analysis. The face of the image is selected and detected by means of face detector, and the keypoints are obtained using SIFT technique. The keypoints of the image are optimally selected by means of WOA. The optimized keypoints are then subjected for extracting the features. Moreover, the features obtained from the key points are multiplied with weight factors, which is equal to the length of the features. The extracted features of the image are given to D6tyuBN for classifying the respective emotions like normal, happy, sad, surprise, angry, fear and disgust. Accordingly, the weight factors along with the number of hidden neurons are optimized by WOA, which results in optimal classification of emotions. Thus the classifier recognizes certain emotions of the testing image from a set of training images.

3.2 SIFT extraction

SIFT [32] is a method for extracting and identifying local feature descriptors that are invariant to image contrast, rotation, and scaling. The SIFT technique is first implemented by Lowe [31]. In recent years, it has been enhanced and improved to offer better image extraction. SIFT characteristics have a number of significances as specified in the below points.

•
Natural characteristics can be obtained from images. They are invariant to uniform orientation, scaling and moderately invariant to contrast variations.
•
Enhanced error acceptance with fewer matches.
•
Offers characteristics with excellent effectiveness and speediness.
•
Suitable to merge and produce beneficial information.

Scale-space extrema detection: Initially, in this stage, the forgery image $I(x,y)$ is convolved with Gaussian filters at diverse scales as revealed in Eq. (1), in which, $A\left({x,y,\sigma}\right)$ denotes the convolution of image $I(x,y)$ along with the Gaussian filter $M(x,y,\sigma)$ at scale $\sigma$ . Variations among both the Gaussian images at scale $\sigma$ and $k\sigma$ are manipulated as in Eqs (1) and (2).

$\displaystyle A\left({x,y,\sigma}\right)=M\left({x,y,\sigma}\right)\ast I(x,y)$ (1) $\displaystyle C\left({x,y,\sigma}\right)=A\left({x,y,k\sigma}\right)-A(x,y,\sigma)$ (2)

The dissimilarity in both the scales is said to be the DoG (Differences of Gaussians).

Keypoint localization: Following the initial process, Interest points, which are also known to be the keypoints are recognized as local minima or maxima across scales for the DoG images. The entire pixels in the DoG images is distinguished with its corresponding eight neighbours at the similar scale as revealed by Eq. (3), where $\hat{{x}}$ is evaluated by fixing the derivative $C\left({x,y,\sigma}\right)$ to 0. Moreover, there is necessity to obtain the precise localization of keypoints by neglecting the points by a prespecified value.

$\displaystyle C\left({\hat{{x}}}\right)=C+\frac{1}{2}\frac{\partial C^{T}}{% \partial x}\hat{{x}}$ (3)

Orientation allocation: With the intention of attaining invariance to orientation, the gradient magnitude $l(x,y)$ and orientation $\theta(x,y)$ are evaluated earlier by means of Eqs (4) and (5).

$\displaystyle l(x,y)=$ (4) $\displaystyle\sqrt{A(x{+}1,y){-}A(x{-}1,y)^{2}{+}A(x,y{+}1){-}A(x,y{-}1)^{2}}$ $\displaystyle\theta(x,y)=\arctan\left({\frac{A(x,y+1)-A(x,y-1)}{A(x+1,y)-A(x-1% ,y)}}\right)$ (5)

Keypoint descriptor production: After selecting the keypoint orientation, the feature descriptor is evaluated on 4 $\times$ 4 pixel neighborhoods as a group of orientation histograms.
3.3 Deep belief network

DBN [29, 30] includes different layers, in which every layer comprise visible neurons compiling input layer and hidden neurons constitute the output layer. Moreover, the whole hidden neurons encompass a link to the neurons specified at input; however, there is a denial of association between the hidden neurons, and there exists a shortage of bond between the visible neurons. In fact, the bond between the visible and hidden neurons appears to be restricted and symmetric. The stochastic neuron model portrays the exact output for a particular input. As the output of a stochastic neuron in Boltzmann network is probabilistic, Eq. (6) points out the output and Eqs (6) and (7) indicates the sigmoid-shaped function possibility. In addition, the deterministic representation of the stochastic design is specified in Eq. (8), in which $T$ denotes the pseudo-temperature.

$\displaystyle P_{r}(\zeta)=\frac{1}{1+e^{\frac{-\zeta}{T}}}$ (6) $\displaystyle H=\begin{Bmatrix}1&\text{∼{}∼{}with∼{}}1-P_{r}(\zeta)\\ 0&\text{∼{}∼{}with∼{}}P_{r}(\zeta)\\ \end{Bmatrix}$ (7)

In DBN, feature extraction is carried out by a group of Restricted Boltzmann Machine (RBM) layers, and classification assignment is carried out by means of various perceptron layers. The numerical design depicts the Boltzmann device energy for the involvement of neuron state $c$ , as revealed by Eq. (9), in which the weight between the neurons is indicated by $W_{i,j}$ and the biases are pointed out by $\upsilon_{i}$ .

$\displaystyle\lim\limits_{T\to 0^{+}}P_{r}(\zeta)=\lim\limits_{T\to 0^{+}}% \frac{1}{1+e^{\frac{-\zeta}{T}}}=\begin{cases}0&\text{for∼{}}{\zeta<0}\\ {\frac{1}{2}}&\text{for∼{}}{\zeta=0}\\ 1&\text{for∼{}}{\zeta>0}\\ \end{cases}$ (8) $\displaystyle\Delta R\left({c_{i}}\right)=\sum\limits_{j}{c_{i}}W_{i,j}+% \upsilon_{i}$ (9)

The descriptions of energy about the joint symphony of visible and hidden neurons $(a,b)$ are characterized in Eqs (10)–(12). Depending on such descriptions, $a_{i}$ signifies the binary state of visible unit $i$ , $k_{j}$ points out the binary position of hidden unit $j$ and $u_{i}$ and $k_{i}$ symbolizes the biases deployed in the network.

$\displaystyle R(a,b)=\sum\limits_{(i,j)}{W_{i,j}}a_{i}b_{j}-\sum\limits_{i}{u_% {i}}a_{i}-\sum\limits_{j}{k_{j}}b_{i}$ (10) $\displaystyle\Delta R({a_{i},\vec{{b}}})=\sum\limits_{j}{W_{ij}}b_{j}+u_{i}$ (11) $\displaystyle\Delta R({\vec{{a}},b_{i}})=\sum\limits_{j}{W_{ij}}a_{i}+k_{j}$ (12)

The probability allotments of the input data are encoded into weight factors and are allocated as the learning representation of RBM. In reality, training of RBM can make use of the allotted probabilities, and the allocation of weight is described by means of Eq. (13). For all possible pair of hidden and noticeable vectors, the probability allotted RBM design is revealed as specified in Eq. (14), in which $Y$ specifies to the partition function as depicted in Eq. (15).

$\displaystyle W^{(m)}=\max_{W}\prod_{\vec{{a}}\in o}p\left({\vec{{a}}}\right)$ (13) $\displaystyle p({\vec{{a}},\vec{{h}}})=\frac{1}{Y}e^{-R(\vec{{a}},\vec{{b}})}$ (14) $\displaystyle Y=\sum\limits_{\vec{{a}},\vec{{b}}}{e^{-R({\vec{{a}},\vec{{b}}})}}$ (15)

As it is a complicated assignment to attain the sampling of the outlooks beneath the allocation portrayed by the format, a learning method called Contrastive Divergence (CD) is exploited. Thus the concise phases of CD performance are demonstrated as specified below.

Step 1: Step 1:

Choose the training samples $a$ and strut it into the perceptible neurons.

Step 2:

Compute the possibilities of hidden neurons $p_{b}$ , by identifying the product of weight matrix $W$ and visible vector $a$ as $p_{b}=\sigma(a.W)$ , on the basis of Eq. (Step 2:).

$\displaystyle p\left({\vec{{b}}_{j}\to 1|\vec{{a}}}\right)=\sigma\left({v_{j}+% \sum\limits_{i}{b_{i}w_{i,j}}}\right)$

Step 3:

Verify the hidden states $b$ from the probabilities $p_{b}$ .

Step 4:

Determine the external product of vectors $a$ and $p_{b}$ , regard it as a positive gradient $\phi^{+}=a.p_{b}^{T}$ .

Step 5:

Verify the reformation of the visible states ${a}^{\prime}$ from the hidden states $b$ as per Eq. (Step 5:). In addition, it is required to check the concealed states ${b}^{\prime}$ from the renovation of the noticeable states ${a}^{\prime}$ .

$\displaystyle p\left({\vec{{a}}_{j}\to 1|\vec{{b}}}\right)=\sigma\left({u_{i}+% \sum\limits_{i}{a_{j}W_{i,j}}}\right)$

Step 6:

Compute the exterior product of ${a}^{\prime}$ and ${b}^{\prime}$ , by it as negative gradient $\phi^{-}={a}^{\prime}.{b}^{\prime T}$ .

Step 7:

Find out the updated weight as exposed in Eq. (18), in which $\eta$ points out the rate of learning.

$\displaystyle\Delta W=\eta\left({\phi^{+}-\phi^{-}}\right)$ (18)

Step 8:

The weights with novel values are customized as specified in Eq. (19).

$\displaystyle W_{i,j}=\Delta W_{i,j}+W_{i,j}$ (19)

To initiate the procedure of learning dependent on MLP system, let us regard the patterns of training $\left({J^{m},S^{m}}\right)$ , in which $m$ denotes the quantity of patterns adopted for training, $1\leqslant m\leqslant U$ , $J^{m}$ and $S^{m}$ indicate the desired output vector and input vector, correspondingly. The error of each neuron in the output layer, in $j$ is exposed as specified in Eq. (20).

$\displaystyle e_{j}^{m}=J^{m}-S^{m}$ (20)

Thus, Eq. (21) offers the pattern $m$ for the squared error pursued by Mean Squared Error (MSE) in Eq. (21).

$\displaystyle R_{m}^{\textit{mean}}=\frac{1}{n_{k}}\sum\limits_{j=1}^{n_{k}}{% \left({e_{j}^{m}}\right)}^{2}=\frac{1}{n_{k}}\sum\limits_{j=1}^{n_{k}}{(J^{m}-% S^{m})}^{2}$ (21) $\displaystyle R_{\textit{avg}}=\frac{1}{U}R_{m}^{\textit{mean}}$ (22)

The course of action taking place throughout the training of DBN with the incorporation of normal training (MLP) and pre-training (RBM) is provided in the following steps.

Step 1: Step 1:

By means of biases and various related factors along with the arbitrarily chosen weights, the DBN design is initialized.

Step 2:

Primarily, the initialization of RBM representation is carried out by means of data provided at input, helping potentials in its visible neurons and holds the unsupervised learning.

Step 3:

The input to the following layer is attained by sampling the potentials that is produced in the hidden neurons of the preceding layer. In addition, it tracks the unsupervised learning.

Step 4:

These steps are recurred for a definite amount of layers. Consequently the pre-training phase by RBM is completed till it arrives the MLP layer.

Step 5:

MLP stage offers sophisticated learning by supervised format and is continued till it acquires the error rate of objective.

4. Whale optimization for keypoint selection and classification

4.1 Solution encoding

The solution encoding involves two phases. Initially, the keypoints from the SIFT technique is taken as solution for optimal selection as shown in Fig. 2. Here, $k_{i}$ indicates the keypoints from SIFT, $i=1,2,\ldots N_{k}$ , $N_{k}$ is the number of keypoints. Next, the attained features from the selected keypoints are multiplied with a weight factor, which is equal to the length of the feature. Accordingly, the corresponding weight factors along with the hidden neuron of DBN is taken as a solution, as given in Fig. 3. In this solution, $we_{j}$ is the weight factor, $j=1,2,\ldots N_{w}$ , $N_{w}$ is the number of weight and $N_{H}$ is the number of hidden neurons that can be utilized in DBN. These two solutions are subjected to WOA in order to attain the best solution.

Figure 2.

Solution encoding (Phase 1).

Figure 3.

Solution encoding (Phase 2).

4.2 Objective model

The proposed face recognition model includes two objective functions as mentioned below.

1) 1)
The objective function of the phase 1is the minimization of correlation among the keypoints that get optimized by WOA as exposed in Eq. (23). The correlation among the two key points $u$ and $v$ is shown in Eq. (24), in which $n$ denotes the number of point pairs.

$\displaystyle O_{1}=\min(\textit{correlation})$ (23) $\displaystyle\textit{Correlation}=$ (24) $\displaystyle\frac{n\sum{uv-\sum{u\sum v}}}{\sqrt{(n}\sum{u^{2}-(\sum{u)^{2})-% (n\sum{v^{2}-(\sum{v)^{2})}}}}}$
2)
The objective function of phase 2 concerns with the optimal features that are multiplied with the weight factors, which is equal to the length of features. The solution of weight factor and hidden neuron focus on the objective function as maximization of classification accuracy as shown by Eq. (25), where $A_{cc}$ is the classification accuracy.

$\displaystyle O_{2}=\textit{Max}(A_{cc})$ (25)

4.3 Whale optimization algorithm

The solutions in phases 1 and 2 are given to WOA for selecting the optimal solution. In general, whales are liable for emotions, judgment, and social performances as done by humans [33]. The major motivating thing concerning the humpback whales is the amazing hunting system. The foraging activities of them are identified as bubble-net feeding process. It is an outstanding behavior that can be perceived in only the humpback whales.

Figure 4.

Sample image data for (a) Normal (b) Smile (c) Sad (d) Surprise (e) Angry (f) Fear (g) Disgust.

They can identify the position of prey and encircle them. As the location of the most favorable model in the explore space is not recognized as a priori, the WOA process presumes that the current most excellent candidate solution is the objective prey or is closer to the most favorable. Following the description of the fine exploring agent, the erstwhile exploring agents will, therefore, attempt to modernize their locations towards the optimal agent for search. This behavior is indicated by Eqs (26) and (27), in which $t$ points out the present iteration, coefficient vectors are signified by $\vec{{B}}$ and $\vec{{C}}$ and $X^{\ast}$ signifies the position vector of the most outstanding solution attained so far, $\left|∼{}\right|$ denotes the absolute value, $X$ symbolizes the location vector, and $\cdot$ indicates an element-by-element multiplication.

$\displaystyle\vec{{V}}=\left|{\vec{{J}}.\vec{{X}}^{\ast}(t)-\vec{{X}}(t)}\right|$ (26) $\displaystyle\vec{{X}}(t+1)=\vec{{X}}^{\ast}(t)-\vec{{B}}.\vec{{V}}$ (27)

It is important to note that $X^{\ast}$ has to be updated in the entire iterations in the existence of an improved solution. The vectors $B$ and $J$ are calculated as specified in Eqs (28) and (29) in which $\vec{{f}}$ is linearly decreased from 2 to 0 over the course of iterations (in both exploration and exploitation phases) and $\vec{{r}}$ is a random vector in $[0,1]$ .

$\displaystyle\vec{{B}}=2\vec{{f}}.\vec{{r}}-\vec{f}$ (28) $\displaystyle\vec{{J}}=2.\vec{{r}}$ (29)

Exploitation phase: Shrinking encircling mechanism: This action is accomplished by minimizing the value of $\vec{a}$ in Eq. (28). Observe that the variation of $\vec{{B}}$ is further reduced by $\vec{f}$ , that is $\vec{{B}}$ is an arbitrary value that lies between $[-\vec{{f}},\vec{{f}}]$ in which $\vec{{f}}$ is reduced from 2 to 0 for more iterations.

Spiral updating position: Initially $,$ this method measures the distance among the whale positioned at $(X,Z)$ and prey at $\left({X^{\ast},Z^{\ast}}\right)$ . A spiral formula is moreover generated between the location of prey and whale as shown in Eq. (30) in which $\vec{{V}}^{\prime}=|{\vec{{X}}^{\ast}(t)-\vec{{X}}(t)}|$ and it specifies the distance of the $i^{\rm th}$ whale to the prey, the constant for determining the shape of the logarithmic spiral is pointed out by $g$ , ‘.’ is an element by element multiplication and $l$ is an alternate value lying in the interval between $[-1,1]$ .

$\displaystyle\vec{{X}}(t+1)=\vec{{V}}.e^{gl}.\cos(2\pi l)+\vec{{X}}^{\ast}(t)$ (30)

It can be further modeled as given in Eq. (31) where $p_{a}$ is an alternate number lying between [0,1].

$\displaystyle\vec{{X}}(t+1)=$ (31) $\displaystyle\begin{cases}{\vec{{X}}^{\ast}(t)-\vec{{B}}-\vec{{V}}}&\text{if∼{% }}p_{a}<0.5\\ {\vec{{V}}^{\prime}.e^{bl}.\cos(2\pi l)+\vec{{X}}^{\ast}(t)}&\text{if∼{}}p_{a}% \geqslant 0.5\\ \end{cases}$

Exploration phase: According to this phase, randomly selected exploration agent rather than the most excellent search agent is detected so far. Such mechanism and $|{\vec{{B}}}|>1$ highlights the searching and permits the WOA technique to carry out a comprehensive search. It is explained as in Eqs (32) and (33), where $\vec{{X}}_{\textit{rand}}$ is a random whale selected from the present population.

$\displaystyle\vec{{V}}=|{{\vec{{J}}.\vec{{X}}_{\textit{rand}}-\vec{{X}}}}|$ (32) $\displaystyle\vec{{X}}(t+1)=\vec{{X}}_{\textit{rand}}-\vec{{B}}.\vec{{V}}$ (33)

5. Results and discussion

5.1 Experimental setup

Figure 5.

Classification analysis of proposed and conventional methods measuring (a) Accuracy (b) Sensitivity (c) Specificity (d) Precision (e) FPR (f) FNR (g) NPV (h) FDR (i) F1-score (j) MCC.

Algorithm 1: Whale optimization algorithm
Allocate the population of whales $X_{i}\left({i=1,2,\ldots,n}\right)$
Evaluate the fitness of the entire exploring agents
$X^{\ast}$ is the best agent for exploring
While $t$ is less than the maximum amount of iterations
For the entire search agents
Update $\vec{{f}},b,j,l$ and $p_{a}$
if 1 $p_{a}<0.5$
if 2 $\left({\left\|B\right\|<1}\right)$
update the location of present search agent based on
Eq. (27)
else if 2 $\left({\left\|B\right\|\geqslant 1}\right)$
choose an alternate search agent $\vec{{X}}_{\textit{rand}}$
update the location of present search agent based on
Eq. (30)
end if 2
else if 1 $p_{a}\geqslant 0.5$
update the location of present search agent based on
Eq. (33)
end if 1
end for
Confirm whether any search agent extends above the
exploring space
Compute the fitness of exploring agent
if there exists a improved solution, update $X^{\ast}$
$t=t+1$
end while
return $X^{\ast}$

The proposed face emotion recognition model was carried out in MATLAB using JAFFE database, and the results were obtained. The analysis was made based on six face emotions such as normal, happy, sad, surprise, angry, fear and disgust as shown by Fig. 4. The proposed ADBN method using WOA was compared with conventional algorithms such as GA [38], PSO [39], ABC [40], FF [41] and GWO [42] and the outcomes were analyzed. In addition, the evaluation was done by comparing the implemented classification scheme with conventional schemes such as DBN [34], NN [35], SVM [36] and NB [37] techniques and the related measures such as accuracy, sensitivity, specificity, precision, False Positive Rate (FPR), False Negative Rate (FNR), Negative Predictive Value (NPV), False Discovery Rate (FDR), F1-score, Matthews Correlation Coefficient (MCC) were achieved.

Table 2

Impact of whale optimization over conventional optimization algorithms for face emotion recognition

Methods	GA [38]	PSO [39]	ABC [40]	FF [41]	GWO [42]	WOA
Accuracy	0.926531	0.922449	0.942857	0.946939	0.946939	0.95102
Sensitivity	0.742857	0.728571	0.8	0.814286	0.814286	0.828571
Specificity	0.957143	0.954762	0.966667	0.969048	0.969048	0.971429
Precision	0.742857	0.728571	0.8	0.814286	0.814286	0.828571
FPR	0.042857	0.045238	0.033333	0.030952	0.030952	0.028571
FNR	0.257143	0.271429	0.2	0.185714	0.185714	0.171429
NPV	0.957143	0.954762	0.966667	0.969048	0.969048	0.971429
FDR	0.257143	0.271429	0.2	0.185714	0.185714	0.171429
F1-score	0.742857	0.728571	0.8	0.814286	0.814286	0.828571
MCC	0.7	0.683333	0.766667	0.783333	0.783333	0.8

5.2 Analysis on classification

The classification analysis for proposed ADBN using WOA for face emotion recognition has been specified by Fig. 5. From Fig. 5a, the accuracy of the proposed scheme for SIFT is 16.67% better than DBN, 6.25% better than NN, 1.04% better than SVM, and 11.46% better than NB methods. Also, the suggested scheme for OSIFT is 16.33% superior to DBN, 8.16% superior to NN, 9.18% superior to SVM, and 13.27% superior to NB models. Also, from Fig. 5b, the sensitivity of the implemented method for SIFT is 50% better than DBN, 12.82% better than NN, 5.12% better than SVM, and 23.07% better than NB methods. As well, the presented system for OSIFT is 47.05% superior to DBN, 17.65% superior to NN, 15.29% superior to SVM, and 23.53% superior to NB designs. From Fig. 5c, the specificity of SIFT is 6.25% better than DBN, 1.04% better than NN, 1.04% better than SVM, and 4.16% better than NB techniques. Similarly, OSIFT is 4.21% superior to DBN, 1.05% superior to NN, 2.1% superior to SVM, and 3.16% superior to NB schemes. In addition, from Fig. 5d, the precision of SIFT is 50% better than DBN, 8.98% better than NN, 3.85% better than SVM, and 3.85% better than NB approaches. Similarly, the proposed OSIFT is 47.05% superior to DBN, 11.76% superior to NN, 23.53% superior to SVM, and 35.29% superior to NB algorithms. From Fig. 5e, FPR of the implemented scheme for SIFT is 62.38% better than DBN, 10.53% better than NN, 10.53% better than SVM, and 57.78% better than NB models. Also, the proposed OSIFT is 64.70% superior to DBN, 40% superior to NN, 51.61% superior to SVM, and 66.67% superior to NB schemes. From Fig. 5f, the FNR of SIFT is 63.93% better than DBN, 45.45% better than NN, 18.18% better than SVM, and 72.72% better than NB approaches. In addition, from Fig. 5g, NPV of SIFT is 6.25% superior to DBN, 1.04% superior to NN, 1.04% superior to SVM, and 4.16% superior to NB algorithms. Also, NPV of proposed OSIFT is 7.14% better than DBN, 2.04% better than NN, 3.06% better than SVM, and 7.14% better than NB algorithms. From Fig. 5h, FDR of SIFT is 63.93% superior to DBN, 45.45% superior to NN, 18.18% superior to SVM, and 72.72% superior to NB approaches and the proposed OSIFT is 65.38% better than DBN, 55.55% better than NN, 50% better than SVM, and 83.33% better than NB schemes. From Fig. 5i, the F1-score of the implemented scheme is 50% superior to DBN, 12.82% superior to NN, 5.12% superior to SVM, and 23.07% superior to NB systems and the suggested OSIFT is 47.05% better than DBN, 17.65% better than NN, 15.29% better than SVM, and 23.53% better than NB models. Finally, from Fig. 5j, the MCC of modelled scheme for SIFT is 60% superior to DBN, 7.14% superior to NN, 7.14% superior to SVM, and 28.57% superior to NB techniques and the proposed OSIFT is 52.5% better than DBN, 18.75% better than NN, 21.25% better than SVM, and 35% better than NB algorithms. Thus the enhancement of the implemented classification technique has been confirmed successfully from the obtained results.

5.3 Impact of whale optimization

The performance analysis of the proposed face recognition model using WOA is compared over several conventional optimization algorithms, which is shown in Table 2, where the accuracy of the suggested scheme is 2.57% better than GA, 3% better than PSO, 0.864% better than ABC, 0.43% better than FF and 0.43% better than GWO techniques. Also the sensitivity of the presented method is 10.44% superior to GA, 12.06% superior to PSO, 3.45% superior to ABC, 1.73% superior to FF and 1.73% superior to GWO methods. The specificity of the implemented method is 1.47% better than GA, 1.72% better than PSO, 0.846% better than ABC, 0.25% better than FF and 0.25% better than GWO models. Also, the precision of the suggested scheme is 10.44% superior to GA, 12.06% superior to PSO, 3.45% superior to ABC, 1.73% superior to FF and 1.73% superior to GWO methods. The FPR of the presented mechanism is 49.8% better than GA, 57.5% better than PSO, 15.5% better than ABC, 8.15% better than FF and 8.15% better than GWO algorithms. Moreover, the FNR of the proposed model is 49.9% superior to GA, 57.49% superior to PSO, 16.67% superior to ABC, 7.92% superior to FF and 7.92% superior to GWO designs. The NPV of the suggested method is 1.47% better than GA, 1.72% better than PSO, 0.46% better than ABC, 0.25% better than FF and 0.25% better than GWO techniques. The FDR of presented scheme is 49.9% superior to GA, 57.49% superior to PSO, 16.67% superior to ABC, 7.92% superior to FF and 7.92% superior to GWO approaches. Similarly, the F1-score of implemented scheme is 10.44% better than GA, 12.06% better than PSO, 3.45% better than ABC, 1.73% better than FF and 1.73% better than GWO models. Finally, the MCC of implemented scheme is 12.5% superior to GA, 14.59% superior to PSO, 4.25% superior to ABC, 2.09% superior to FF and 2.09% superior to GWO approaches. Thus the improved mechanism of the proposed method has been verified successfully.

6. Conclusion

This paper has presented a technique, in which two contributions were performed in feature extraction and classification process. Primarily, SIFT features were extracted in the process of feature extraction. Anyhow, numerous key points were achieved throughout SIFT extraction which provides redundant information. Hence, the key points were optimized by deploying WOA approach. Moreover, the features that were extracted from the chosen keypoints were factorized with a weight factor that has to be optimized by WOA scheme. The optimal features were then offered to DBN for the classification of face emotion, where the count of hidden neurons was optimized by WOA approach along with feature weight. Finally, investigations were carried out, where the proposed ADBN scheme using WOA was compared with existing classifiers and other optimization algorithms. From the classification results, the accuracy of the suggested method for SIFT was 16.67% better than DBN, 6.25% better than NN, 1.04% better than SVM, and 11.46% better than NB methods. Also, the presented scheme for OSIFT was 16.33% superior to DBN, 8.16% superior to NN, 9.18% superior to SVM, and 13.27% superior to NB approach. Also, the overall performance of the suggested method in terms of accuracy is 2.57% better than GA, 3% better than PSO, 0.864% better than ABC, 0.43% better than FF and 0.43% better than GWO techniques. Thus the improved performance of the implemented face emotion recognition was proved in a better way.

References

Morita

Kosaka

Saito

Fujii

Sadato

. Neural correlates of emotion processing during observed self-face recognition in individuals with autism spectrum disorders. Research in Autism Spectrum Disorders. 2016; 26: 16-32.

Liedtke

Kohl

Kret

Koelkebeck

. Emotion recognition from faces with in- and out-group features in patients with depression. Journal of Affective Disorders. 2018; 227: 817-823.

Shojaeilangari

Yau

Nandakumar

Teoh

. Robust representation and recognition of facial emotions using extreme sparse learning. IEEE Transactions on Image Processing. 2015; 24(7): 2140-2152.

Wang

Phillips

Dong

Zhang

. Intelligent facial emotion recognition based on stationary wavelet entropy and jaya algorithm. Neurocomputing. 2018; 272: 668-676.

Chen

Zhou

Hirota

. Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction. Information Sciences. 2018; 428: 49-61.

Happy

Routray

. Automatic facial expression recognition using features of salient facial patches. IEEE Transactions on Affective Computing. 2015; 6(1): 1-12.

Chiranjeevi

Gopalakrishnan

Moogi

. Neutral face classification using personalized appearance models for fast and robust emotion detection. IEEE Transactions on Image Processing. 2015; 24(9): 2701-2711.

Zhang

Mistry

Neoh

Lim

. Intelligent facial emotion recognition using moth-firefly optimization. Knowledge-Based Systems. 2016; 111: 248-267.

Presti

Cascia

. Boosting hankel matrices for face emotion recognition and pain detection. Computer Vision and Image Understanding. 2017; 156: 19-33.

10.

Sellaro

Gelder

Finisguerra

Colzato

. Transcutaneous vagus nerve stimulation (tVNS) enhances recognition of emotions in faces but not bodies. Cortex. 2018; 99: 213-223.

11.

Lahera

Herrera

Fernández

Bardón

Fernández-Liria

. Familiarity and face emotion recognition in patients with schizophrenia. Comprehensive Psychiatry. 2014; 55(1): 199-205.

12.

White

Abbott

Wieckowski

Capriola

Youssef

. Feasibility of automated training for facial emotion expression and recognition in autism. Behavior Therapy. 2017.

13.

Meehan

Panfilis

Cain

Antonucci

Sambataro

. Facial emotion recognition and borderline personality pathology. Psychiatry Research. 2017; 255: 347-354.

14.

Ming

. Robust regional bounding spherical descriptor for 3D face recognition and emotion analysis. Image and Vision Computing. 2015; 35: 14-22.

15.

Timmermann

Jeung

Schmitt

Boll

Herpertz

. Oxytocin improves facial emotion recognition in young adults with antisocial personality disorder. Psychoneuroendocrinology. 2017; 85: 158-164.

16.

Rieffe

Wiefferink

. Happy faces, sad faces: Emotion understanding in toddlers and preschoolers with language. Impairments Research in Developmental Disabilities. 2017; 62: 40-49.

17.

Mao

Xue

Gou

Zhan

. Learning emotion-discriminative and domain-invariant features for domain adaptation in speech emotion recognition. Speech Communication. 2017; 93: 1-10.

18.

Lucille

Bellegarde

Haskell

Duvaux-Ponter

Weiss

, Erhard. Face-based perception of emotions in dairy goats. Applied Animal Behaviour Science. 2017; 193: 51-59.

19.

Kim

Kang

. An analysis of smartphone overuse recognition in terms of emotions using brainwaves and deep learning. Neurocomputing. 2018; 275: 1393-1406.

20.

Quevedo

Harms

Sauder

Scott

Smyda

. The neurobiology of self face recognition among depressed adolescents. Journal of Affective Disorders. 2018; 229: 22-31.

21.

Yankouskaya

Stolte

Moradi

Rotshtein

Humphreys

. Integration of identity and emotion information in faces: FMRI evidence. Brain and Cognition. 2017; 116: 29-39.

22.

Balas

Huynh

Saville

Schmidt

. Orientation biases for facial emotion recognition during childhood and adulthood. Journal of Experimental Child Psychology. 2015; 140: 171-183.

23.

Hargreaves

Mothersill

Anderson

Lawless

Donohoe

. Detecting facial emotion recognition deficits in schizophrenia using dynamic stimuli of varying intensities. Neuroscience Letters. 2016; 633: 47-54.

24.

Ciaramidaro

Bölte

Schlitt

Hainz

Walter

. Transdiagnostic deviant facial recognition for implicit negative emotion in autism and schizophrenia. European Neuropsychopharmacology. 2017.

25.

Huang

Wang

. Revisiting the earliest electrophysiological correlate of familiar face recognition. International Journal of Psychophysiology. 2017; 120: 42-53.

26.

Weisgerber

Vermeulen

Peretz

Samson

Constant

. Facial, vocal and musical emotion recognition is altered in paranoid schizophrenic patients. Psychiatry Research. 2015; 229(1–2): 188-193.

27.

Marsh

Luckett

Russell

Coltheart

Green

. Effects of facial emotion recognition remediation on visual scanning of novel face stimuli. Schizophrenia Research. 2012; 141(2–3): 234-240.

28.

Wong

Cho

. A local experts organization model with application to face emotion recognition. Expert Systems with Applications. 2009; 36(1): 804-819.

29.

Tang

Liu

Lei

Song

Dong

. DeepChart: Combining deep convolutional networks and deep belief networks in chart classification. Signal Processing. 2016; 124: 156-161.

30.

Mannepalli

Sastry

Suman

. A novel adaptive fractional deep belief networks for speaker emotion recognition. Alexandria Engineering Journal. 2016.

31.

Lowe

. Object recognition from local scale-invariant features in computer vision. The Proceedings of the Seventh IEEE International Conference on. 1999; 2: 1150-1157.

32.

Sahu

Sur

ASI

. FT based video watermarking resistant to temporal scaling. Journal of Visual Communication and Image Representation. 2017; 45: 77-86.

33.

Mirjalili

Lewis

. The whale optimization algorithm. Advances in Engineering Software. 2016; 95: 51-67.

34.

Elleuch

Tagougui

Kherallah

. Optimization of DBN using regularization methods applied for recognizing arabic handwritten script. Procedia Computer Science. 2017; 108: 2292-2297.

35.

Watanabe

Hiramatsu

, Kashino. Modular representation of layered neural networks. Neural Networks. 2018; 97: 62-73.

36.

Yuan

Zhang

Luo

Ghassemlooy

Han

DSV

. M-based detection in visible light communications. Optik – International Journal for Light and Electron Optics. 2017; 151: 55-64.

37.

Arar

ÖF

Ayan

. A feature dependent naive bayes approach and its application to the software defect prediction problem. Applied Soft Computing. 2017; 59: 197-209.

38.

Vrionis

Koutiva

Vovos

. A genetic algorithm-based low voltage ride-through control strategy for grid connected doubly fed induction wind generators. in IEEE Transactions on Power Systems. 2014; 29(3): 1325-1334.

39.

Zhang

Xia

. An improved PSO algorithm for parameter identification of nonlinear dynamic hysteretic models. Journal of Sound and Vibration. 2017; 389: 153-167.

40.

Koçer

. Bollinger bands approach on boosting ABC algorithm and its variants. Applied Soft Computing. 2016; 49: 292-312.

41.

Wang

Zhou

Sun

Cui

. Firefly algorithm with neighborhood attraction. Information Sciences. 2017; 382-383, 374-387.

42.

Mirjalili

Lewis

. Grey wolf optimizer. Advances in Engineering Software. 2014; 69: 46-61.

43.

Wagh

Todmal

. Eyelids, eyelashes detection algorithm and hough transform method for noise removal in iris recognition. International Journal of Computer Applications. 2015; 112(3).

44.

Singh

Jain

Singh

. Adaptive network architecture and firefly algorithm for biogas heating model aided by photovoltaic thermal greenhouse system. Journal of Energy Environment. 2018; 1-25.

45.

Madhuri

Aniruddha

Rahul

. Identification and classification of flood prone areas using AHP, GIS and GPS. Journal of Disaster Advances. 2013; 6: 120-131.

46.

Sherifi

Senja

. Internet usage on mobile devices and their impact on evolution of informative websites in albania. 3(6): 37-43.

47.

Sable

Jondhale

. Modified double bilateral filter for sharpness enhancement and noise removal. 2010 International Conference on Advances in Computer Engineering. Bangalore. 2010; 295-297.

48.

Kumar

SBV

Rao

Sharath

Sachin

Ravi

Monica

. Review on VLSI design using optimization and self-adaptive particle swarm optimization. Journal of King Saud University – Computer and Information Sciences. 2018.

49.

Lecce

Dimauro

Guerriero

Impedovo

Pirlo

Salzo

. Classifier combination: The role of a-priori knowledge. 2000; 7: 143-152.

50.

Triantafyllou

Tsihrintzis

. Group affect recognition: Evaluation of basic automated sorting. 9th IEEE International Conference on Information. Intelligence, Systems and Applications (IISA 2018). 2018; 23-25.

51.

Triantafyllou

Tsihrintzis

. Group affect recognition: Visual – facial data collection. 29th IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2017). 2017; 6-8.

52.

Stathopoulou

Tsihrintzis

. Appearance-based face detection with artificial neural networks. Intelligent Decision Technologies. 2011; 5(2): 101-111.

53.

Stathopoulou

Tsihrintzis

. Visual affect recognition. Frontiers in Artificial Intelligence and Applications. IOS Press. 2010; 214: 1-247. ISBN 978-1-60750-596-9.

54.

Stathopoulou

Alepis

Tsihrintzis

Virvou

. On assisting a visual-facial affect recognition system with keyboard-stroke pattern information. Knowl.-Based Syst. 2010; 23(4): 350-356.

55.

Lampropoulos

Stathopoulou

Tsihrintzis

. Comparative performance evaluation of classifiers for facial expression recognition. KES IIMSS. 2009; 253-263.

A robust face emotion recognition approach through optimized SIFT features and adaptive deep belief neural network

Abstract

Keywords

1. Introduction

2. Literature review

2.1 Related works

2.2 Review

Table 1 Features and challenges of face recognition using various techniques

3.1 Proposed architecture

3.2 SIFT extraction

4.1 Solution encoding

5.1 Experimental setup

5.3 Impact of whale optimization

6. Conclusion

References

Table 1
Features and challenges of face recognition using various techniques