EmotionFusion: A unified ensemble R-CNN approach for advanced facial emotion analysis

Abstract

To assess non-verbal reactions to commodities, services, or products, sentiment analysis is the technique of identifying exhibited human emotions utilizing artificial intelligence-based technology. The facial muscles flex and contract differently in response to each facial expression that a person makes, which facilitates the deep learning AI algorithms’ ability to identify an emotion. Facial emotion analysis has numerous applications across various industries and domains, leveraging the understanding of human emotions conveyed through facial expressions, so it is very much required in healthcare, security and survelliance, Forensics, Autism and cultural studies etc,.. In this study, facially expressed sentiments in real-time photographs as well as in an existing dataset are classified using object detection techniques based on deep learning. Fast Region-based Convolution Neural Network (R-CNN) is an object detection system that uses suggested areas to categorize facial expressions of emotion in real-time. Using a high-quality video collection made up of 24 actors who were photographed facially expressing eight distinct emotions (Happy, Sad, Disgust, Anger, Surprise, Fear, Contempt and Neutral). The Fast R-CNN and Mouth region-based feature extraction and Maximally Stable Extremal Regions (MSER) method used for classification and feature extraction respectively. In order to assess the deep network’s performance, the proposed work builds a confusion matrix. The network generalizes to new images rather well, as seen by the average recognition rate of 97.6% for eight emotions. The suggested deep network approach may deliver superior recognition performance when compared to CNN and SVM methods, and it can be applied to a variety of applications including online classrooms, video game testing, healthcare sectors, and automated industry.

Keywords

Deep learning R-CNN emotion recognition facial expression computer vision

1 Introduction and literature survey

Facial emotion analysis holds great potential to enhance a wide range of applications, from improving user experiences to advancing research in psychology and neuroscience. However, it’s important to navigate ethical considerations related to privacy, consent, and potential biases when implementing emotion analysis technologies. Charles Darwin published the first scientific investigation on emotional expression on the face in 1872 [1]. His research demonstrated the consistency of various facial expressions with certain emotions and clarified why certain expressions go along with particular emotions, even if other emotions can be displayed while masking the underlying mood. It was argued that despite efforts to hide them, genuine emotions can still be shown. Dr. Paul Ekman writes in [2] that he used the Facial Action Coding System, where less than 25% of his participants were unable to consciously regulate an involuntarily generated muscle contraction, to demonstrate that Darwin’s hypothesis was right. This method, which is seen in Fig. 1, is used to identify and categorize human facial motions as well as determine which set of muscle movements are responsible for each emotion portrayed on the face.. Any movement made by any muscle has a name for each unique direction. Swedish anatomist Carl-Herman Hjortsjo is credited with creating this method [3]. He related 24 different emotional states to groups of face muscle movements. The following are a few instances of the emotional states: dejected, tortured, bewildered, pained, contemptuous, and disgusted.

Fig. 1

Facial Action Coding System (Image taken from- https://www.eiagroup.com/the-facial-action-coding-system/).

Happiness –Raised eyebrows and corners of the mouth

Surprise –Raised eyebrows but not drawn together, raised upper eyelids, jaw dropped down

Anger –Eyebrows pulled down, eyes open wide, lips pursed tightly together

Fear –Jaw dropped open, lips pressed backwards horizontally, upper eyelids raised

Sadness –frowned smile with lip corners pulled down, inner corners of the eyebrows pulled together, eyelids drooped.

Disgust –Upper lip raised in a ‘u’ like shape, eyebrows lowered, wrinkling of the nose

Contempt –Raised lip corner on one side of the face, this is the only signifier but it can also be accompanied by a smile or angry expression.

The Wheel of Emotions, devised by Robert Plutchik in 1980 to assist illustrate the Psycho evolutionary Theory of Emotion [4], is a graphic representation of eight fundamental emotions depicts in Fig. 2. According to the hypothesis, emotion is not just a state of feeling, but also a complicated series of events that, when recreated, would cause sensations that would lead to behaviors. A person would smile when they were feeling happy. The four pairs that make up the eight fundamental emotions are: joy - sadness; trust - disgust; fear - anger; and anticipation - surprise. Any two fundamental emotions placed adjacent to one another can create an emotion. On the other side of the 2D emotion wheel is the opposing feeling to the one that was formed. The challenge in facial emotion analysis lies in accurately and robustly detecting and interpreting human emotional states from facial expressions captured in varying environmental conditions. This problem is characterized by the need to develop computational models capable of discerning nuanced and contextually diverse emotional cues across individuals, cultures, and demographics. Factors contributing to the complexity of the problem include variations in lighting, pose, occlusions, facial morphologies, and the dynamic nature of emotional expressions. Moreover, addressing ethical concerns such as privacy, bias mitigation, and data security further underscores the multifaceted nature of this problem. The objective is to devise methodologies that balance sensitivity, specificity, and execution efficiency while accounting for the intricacies inherent in human emotional expression, leading to more accurate, reliable, and ethically sound facial emotion analysis solutions.

Fig. 2

Wheel of emotions.

Recognition of facial expressions fits into the pattern recognition research paradigm. The three phases of a recognition system would be face detection, feature extraction, and expression categorization. These three crucial challenges of recognition of facial expressions have been the subject of extensive investigation. Different feature classification and extraction techniques have been used to create a range of face emotion recognition systems. ANN were used as the classifier in Tian et alstudy.’s on the combination of permanent characteristics and transient features [5]. In more recent study, Sánchez et al. used Support Vector Machine (SVM) as a classification approach and optical flow-based algorithms for feature extraction [6].

A real-time facial expression detection system with good recognition rates was created by Junnan Li et al. created a multi step feature extraction and picture preparation approach that enhances classification performance [7]. To specifically eliminate illumination variance, histogram rebinding to a normally distributed is utilized after photo metric normalization. To extract resonant characteristics from the pre-processed picture, Kernel Principle Component Analysis and Gabor wavelets were used. When compared to linear PCA, the nonlinear features retrieved by Kernel PCA offer higher recognition rates. The DNN is given the features for classification. A greedy layer-wise training technique is used by the deep network to enable learning from both labeled and unlabeled input. When there is a shortage of labeled data, using unlabeled data enables the system to learn better characteristics.

A deep cascade convolution network using Fast R-CNN for face identification was suggested by Wang, Dong, and Hu in [8, 9]. In [10], Li Shang, Qin, and Chen reported a method for applying Fast R-CNN object detector to detect and identify fish species underwater. When compared to a deform-able parts model (DPM), the Fast R-CNN increases mean average precision (mAP) by 12.2%, obtaining a mAP of 82.4% and outperforming the R-CNN object detector by 85 times. Facial expression recognition using a Faster R-CNN (Faster region-based Convolution Neural Network) approach was proposed by Li, Zhang, and their colleagues in [11]. In their suggestion, the facial expression picture was first normalized before the trainable convolution kernel was used to extract implicit features. In the field of facial emotion analysis, previous articles used LBP feature extraction, CNN and RNN deep learning features, Support Vector machines (SVM), Cross cultural analysis and LSTM based dynamic emotion analysis. Challenges before the proposed work is Capturing and interpreting subtle and context-dependent facial expressions, Defining universal emotional categories and addressing inter-annotator agreement issues and Obtaining balanced datasets for training models across different emotions and intensities.

Microsoft Azure, Face++, Face Reader offers various services and tools for facial emotion analysis using machine learning and computer vision techniques, but lacks with generalizing to diverse facial expressions, especially those not well-represented in the training data. While Azure offers pre-built emotion analysis models, customization might be limited in some cases. Fine-tuning models to specific use cases might require advanced expertise in machine learning.

The remainder of the study will first go proposed method in Section 2. Results and explanation of the proposed method in comparison to the current model are included in Section 3. Section 4 provides the proposed method’s conclusion.

2 Proposed method

Figure 3 shows the entire architecture diagram of the proposed method. The input image to be taken for facial emotion analysis is given for pre-processing and features are extracted and given to R-CNN deep learning approach to classify the emotion.

Fig. 3

Architecture diagram.

Image Pre-Processing

The crucial component of a face emotion recognition system is image pre-processing. It significantly affects the system’s performance and robustness.

Pre-processing aims to improve the discriminating information present in pictures and lessen the impact of noise on feature extraction.

i. Face Detection - Cascade Object Detection:

We require scale-normalized photos of frontal faces for our facial expression detection system. The ability to locate and extract the facial region from an image is crucial. Background must be excluded in order to classify expressions accurately. Numerous face identification algorithms have been developed, including approaches based on skin colour and shape information [6, 7].

The internal image is a quick-to-compute intermediate picture representation created by the Viola-Jones technique. Combining classifier in a cascade structure that can improve detection performance while lowering computing cost is how the face detection is accomplished. Skin-color-based approaches can only be used with coloured photos, whereas shape-information-based approaches are frequently too slow for real-time detection. The Viola-Jones face detection mechanism, a reliable algorithm that can analyse photos incredibly quickly for real-time applications [8], is what we employ in our system. It works best with frontal face photographs, which is precisely the kind of image used. According to tests, the Viola-Jones algorithm successfully detects all of the photos utilized in this study. Figure 4 shows the detected face.

Fig. 4

Face detection.

ii. Binarization –Otsu Method

Three channels with frequencies ranging from 0 to 255 make up a colour image. The conversion of grayscale photos into black-and-white is one of the main characteristics of binarization (0 and 1). Binarization makes the outlines of the numerous things shown in the image more distinct and obvious. Feature extraction enhances how AI models learn. By selecting a threshold value and classifying all pixels with values above it as white and all other pixels as black, the Otsu technique performs the picture binarization process. It eliminates the background noise and displays the foreground image’s contour.

Step 1: Input an image with G gray level values

Step 2: Set the threshold value Th, which will classify the object and background.

$Object (Th) = \sum_{j = Th + 1}^{G} P (j)$ (1) $Background (Th) + Object (Th) = 1$ (2)

Where:

P(j) = Normalization of entire pixel values

G = Number of Gray levels

Object(Th) = Foreground Pixels

Background(Th) = Background Pixels

Figure 5 shows the output of the cropped and binarization process.

Fig. 5

Cropped and binarization image.

iii Filtering using RIR:

The wiener filter [5] technique is a statistical approach to remove noise from every pixel in a picture. It carries out the best possible exchange between noise smoothing and inverse filtering. It is the ideal filter to use for reducing the total MSE during the noise smoothing process. By placing an MSE restriction between the predicted and the original picture, it attempts to construct an image. Wiener filters analyse data in the frequency domain, but they may fail to recover frequency components that have been damaged by noise and the minimized error is shown by Equation (3).

$e 2 = E {J (x, y) - J \land (x, y)}$ (3)

Figure 6 shows the output for the RIR filtering process.

Fig. 6

Filtered image.

iv. Feature Extraction:

The mandatory step in facial emotion identification is applying the exact feature extraction technique to extract the features. Extracted features are given as the input to R-CNN Classifier to classify the emotions with better accuracy. The proposed work used two feature extraction techniques: Mouth region-based feature extraction and Maximally Stable Extreme Regions (MSER) method.

The region around the mouth can be extracted to identify emotions. The key procedures in calculating the mouth area include segmenting the mouth region, converting the grey mouth region to binary, and then performing morphological operations to produce the mouth region. First, input photographs are chosen from the database, and the Viola-Jones Mouth identification technique is used to segment the mouth region [12]. After the mouth region has been separated, pre-processing is done to turn the grey colored mouth regions into binary. The morphological opening function is used to both remove little objects from an image while maintaining the shape of larger objects. After opening an input image, the erosion function is used to reduce the foreground object’s size. By subtracting the opening picture from the erosion image, boundary extraction is obtained. Only the mouth region is removed in the following phase. Finally, any gaps around the mouth are filled using the fill function. The following formula is used to determine the mouth region’s area.

Region = Sum(b(:));

Area = Region * 0.2645;

The emotions are divided into happy, fear, sad, angry, disgust, and surprise categories based on the region values.

Input: Pre-Processed Image

Output: Mouth features

Algorithm:

Step 1: Read the pre-processed image

Step 2: Apply intensity based segmentation

Step 3: Find the connected components and label it using the threshold values.

C (i) = |C_i+D ∖ C_i-D|/|C_i|

Step 4: Suppress gradients near edges using the gradient operator in eq:

M (x, y) = mag (\nabla f) = \sqrt g_{x}^{2} + g_{y}^{2}

Where:

g_x: Subtract the pixel in the top row from the pixel in the bottom row

g_y: Subtract the pixel in the left column from the pixel in the right column

Step 5: Suppress the unwanted corners by using new threshold values

g (x, y) = {1 f (x, y) > T}

{0 f (x, y) < = T}

Where T is new threshold value

Step 6: maximum value of the components are selected to get the points of Interest.

In Mouth based feature extraction method mouth area is calculated and based on that value the emotions are classified. In the MSER method, the features are extracted by using connecting components and then the extracted features are given to a simple ANN for classification. By Matas et al. [12], the theory of Maximally Stable Extreme Regions was first presented. Taking threshold values between 0 and 255 and obtaining the appropriate binary pictures is the first stage in MSER. The maximally stable extreme regions are those connected regions that, after getting all the binary images, remain unchanged with altering intensity values. As MSER key points, it is assumed that the locations of the stable regions and the associated threshold values. The mathematical formula is displayed below. $C (i) = | C_{i + D} ∖ C_{i - D} | / | C_{i} |$ (4)

Where,

Ci - Threshold of Connected region

D - Grey value change

C(i) - Region changing rate

The following actions are taken to implement the MSER algorithm [13].

Figure 7 shows the output of the feature extraction process.

Fig. 7

Output for the feature extraction process.

v. Feature Classification - Fast R-CNN Algorithm:

An object detector called a fast region-based convolution neural network uses deep learning to categorize things in a picture. It employs an approach called Edge Boxes, which boosts an object detector’s computing effectiveness by producing effective region suggestions. An edge box is a straightforward measure of the objectiveness of a box that subtracts the edges that are a component of contours that cross the box’s border [17]. Algorithms for two-state detection make up the quick R-CNN. In the first step, areas in an image that could contain an item are narrowly defined (using edge boxes). The item is classified in each region in the second stage [18]. The entire image is processed by convolution layers in the first stage (feature extraction), which also processes edge boxes to produce region proposals and extract features from the image. Real-time emotion classification is achieved when face emotion detection is done using the Fast R-CNN object detector [19].

i. Training:

Using an image labeled is necessary for training object detectors like Fast R-CNN. An programme called an image labeled allows you to assign predetermined rectangular Region of interest (ROI) labels to a group of photographs.

net = vgg16;

layersTransfer = net.Layers(1:end-3);

% numClasses = numel(categories(imdsTrain.Labels));

layers = [

layersTransfer

fullyConnectedLayer(8,‘WeightLearnRateFactor’,10,‘BiasLearnRate Factor’,10)

softmaxLayer

classificationLayer];

options = trainingOptions(‘sgdm’, . . .

‘MiniBatchSize’,16, . . .

‘MaxEpochs’,12, . . .

‘InitialLearnRate’,0.2e-4, . . .

‘ValidationData’,imdsValidation, . . .

‘ValidationFrequency’,30, . . .

‘Verbose’,false, . . .

‘Plots’, ‘training-progress’)

sa3emNet = trainNetwork(imdsTrain,layers,options);

YPred = classify(sa3emNet,imdsValidation);

accuracy = mean(YPred == imdsValidation.Labels)

save(‘d:\sa3emNet’, ‘-v7.3’, ‘-nocompression’)

Explored to a MATLAB workspace once each image in a collection has had its ROI created with a label. A command like “trainingData” would return the ground truth object as a table of training data once it has been exported as a ground truth to workspace.

The VGG-16 ConvNet [20] is also the foundation for the Fast R-CNN object detector developed for this study. The VGG-16 ConvNet is also subjected to the transfer learning approach. The region for each image in its corresponding emotion label column was set to [1,1,224,224] because every image in this image data store is a cropped image of a face expressing an emotion. The last three layers of the VGG-16 model, as well as the image input layer, were changed from [224, 224, 3] to [48, 48, 3]. Additionally, the grayscale picture data set was changed from [48, 48, 1] to [48, 48, 3]. In addition, a ROI pooling layer and a bounding box regression layer have been added, converting the pretrained network model into a Fast R-CNN identification network [21].

3 Results and discussions

A. Data set: The facial expression database created with multi-ethnic people is used in this project to identify facial expressions. The collection consists of video clips of eight fundamental emotional facial expressions (happiness, surprise, anger, fear, disgust, Contempt, neutral and sadness). Our initial research demonstrates that backdrop alterations have no impact on the suggested methodology. Some other examples are:

Extended Cohn-Kanade Data set (CK+):This is a database made up of 123 subjects, 593 images, and 327 emotion labeled sequences. The images are mostly gray with a resolution of 640*490 [14].

Japanese Female Facial Expressions (JAFFE): This is a database made up of 10 subjects, 213 gray images of 256*256 resolution [15].

MMI Facial Expression Database: Is a database made up of 75 subjects, 2900 videos of 720 * 576 resolution [16].

MMI Facial Expression Database

A growing online tool for basic emotion identification from face videos is the MMI Facial Expression Database. Over 2900 movies and high-resolution still photos of 75 subjects are included in the database. It is completely annotated (event coded) for the existence of AUs in movies and partially coded (frame-level) to indicate whether an AU is in the neutral, onset, apex, or offset phase at any given time. A tiny section was marked for audio-visual chuckles. Figure 8 shows the various sample emotions present in the MMI database. The scientific community can access the database for free [22, 23].

Fig. 8

MMI facial expression database.

The MMI Facial Expression Database’s initial data consisted primarily of presentations of single Action Units(AU). There were a total of 1767 clips of the 20 contestants. All 31 AUs as well as a few more Action Descriptors were required for each participant to exhibit (ADs, also part of FACS) [8]. The subjects were requested to conduct two or three emotional states following the recording of all AUs (e.g. sleepy, happy, bored). For the purpose of increasing the dataset’s variety, the subjects were instructed to repeat each facial movement twice [29]. A mirror that faced the camera at a 45-degree angle was set up on a table adjacent to the participant while the recording was being done. 1767 images were taken for experiment from 1767 movie clips with various emotions contains 220 Happy images, 220 sad images, 220 disgust images, 220 angry images, 220 anger images, 220 surprise images, 220 fear images, 220 contempt images and 227 neutral images. Table 1 shows the properties of MMI Data set used in the proposed model.

Table 1

Properties of dataset

Data Size	Training Data	Test Data	Emotions
224×224	75%	25%	Happy, Sad, Disgust, Anger,
			Surprise, Fear, Contempt and
			Neutral

B. Performance Analysis: Existing Recognition Algorithms

The following table depicts the comparison of performance of existing facial emotion recognition algorithms like SVM, SBN-CNN, RETRaiN, CB-CNN and VGG-19 with proposed Fast R-CNN method [24, 25]. Table 2 shows the comparative analysis of recognition rate (Correctly identified Emotions). The proposed method is compared with the existing algorithms for same emotion recognition like SVM, SBN-CNN, RETRaiN, CB-CNN, and VGG19.

The proposed method achieves maximum of 98.6% as the recognition rate and average recognition rate is 97.6 which is high when compared to the existing existing algorithms (Existing achieved 96.3, 93.4, 88.9, 86.1, and 79.5 for SVM, SBN-CNN, RETRaiN, CB-CNN, and VGG19 respectively). This shows the proposed method outperforms with the highest recognition rate of 98.6%. Figure 9 shows the graphical representation of performance of the proposed method for recognition rate. Figure 10 shows the graphical representation of comparative analysis of average recognition rate of proposed with the existing methods.

Table 2

Recognition rate of various emotions for different CNN

Emotion	Fast R-CNN (Proposed)	SVM	SBN-CNN	RETRaiN	CB-CNN	VGG19
Happy	98.6	95.6	94.8	90.3	87.3	80.3
Sad	97.2	95.3	95.1	89.4	85.8	79.8
Disgust	96.9	95.1	93.1	88.6	86.4	81.7
Anger	98.2	96.1	92.8	87.4	84.3	78.6
Surprise	97.9	96.8	94.2	90.2	85.7	78.4
Fear	98.2	95.9	91.9	89.4	86.8	79.6
Contempt	96.2	97.1	93.4	86.7	85.8	80.2
Neutral	97.2	98.3	92.6	89.2	86.9	77.5
Average	97.6	96.3	93.4	88.9	86.1	79.5

Fig. 9

Recognition rate of various emotions for different CNN.

Fig. 10

Average recognition rate of proposed with existing CNN network.

To check the performance of the proposed method, the following three parameters are taken in to account. Specificity, Precision and Recall, which are specified in equations 1 through 3, are among the metrics utilized for evaluation. Four separate parameters—true negative (TN), true positive (TP), false-negative (FN), and false positive—are used to measure these measurements (FP). Accuracy is defined as the proportion of documents from all the data that were correctly classified. Precision is the proportion of the performance that is relevant. On the other hand, recall is the percentage of correctly classified results that the algorithm produced for all functional outcomes [26, 27]. $Pr ecision = \frac{TP}{FP + TP}$ (5) $SP = \frac{TN}{TN + FP}$ (6) $Re call = TP / TP + FN$ (7)

Table 3 shows the comparison of Precision, Recall and Specificity of the proposed algorithm for various emotions. Precision reached highest value of 0.898 and Recall reached 0.891 and Specificity reached 0.975. These figures shows that the proposed method achieves better values for Precision, Recall and Specificity.

Table 3

Precision, specificity and recall of the proposed method fast R-CNN

Emotion	Precision	Recall	Specificity
Happy	0.807	0.818	0.952
Sad	0.802	0.883	0.937
Disgust	0.781	0.871	0.918
Anger	0.805	0.836	0.927
Surprise	0.788	0.835	0.975
Fear	0.789	0.714	0.899
Contempt	0.742	0.835	0.932
Neutral	0.898	0.891	0.917

Figure 11 shows the graphical representation of the same analysis. It shows that precision value for the Neutral facial expression is reached higher value and contempt shows the lower value. Recall value for neutral facial expression is high and fear is very low. Specificity value for surprise facial expression is more fear is very low. It shows that identifying some important facial expressions surprise, fear contempt and normal expression neutral can be easily identified when compared to the other facia expressions.

Fig. 11

Performance analysis of different emotions (precision, recall and specificity).

C. Performance Analysis: Existing Face Recognition Tools

In recent days, many number of FER-Face Expression Recognition Tools are available in market to recognize the facial emotions. The section uses the FER tools to compare with the proposed method. The standard tools are Microsoft’s Azure, Megvii’s Face++ and Noldus’s Face Reader. The following Tables 4 to 6 depicts the Accuracy, sensitivity and precision of proposed and existing FER Tools used in real world for emotion recognition purpose. Table 4 shows the comparative analysis for specificity of the proposed method and existing FER applications. For all the emotions, the proposed method achieved better specificity or equal specificity value when compared existing method and achieved maximum of 0.98 for surprise emotion and minimum of 0.89 for fear emotion. Figure 12 shows the graphical representation of specificity achieved in proposed work which shows that proposed work achieved better values for all the eight emotions taken for consideration, so proposed work outperforms with maximum specificity value of 0.98. The higher sensitivity value leads to reduced false positives, avoids mitigation and misinterpretation and Higher specificity prevents the system from overreacting to ambiguous or non-emotional facial expressions, leading to more natural and accurate responses.

Table 4

Comparative analysis: Specificity

Emotions	Azure	Face++	Face Reader	Proposed
				System
Happy	0.80	0.80	0.79	0.95
Sad	0.75	0.75	0.77	0.93
Disgust	0.78	0.77	0.77	0.92
Anger	0.81	0.74	0.79	0.93
Surprise	0.73	0.71	0.73	0.98
Fear	0.79	0.75	0.77	0.89
Contempt	0.72	0.72	0.78	0.93
Neutral	0.63	0.70	0.82	0.92

Table 5

Comparative analysis: Precision

Emotions	Azure	Face++	Face Reader	Proposed
				System
Happy	0.80	0.80	0.79	0.81
Sad	0.78	0.77	0.78	0.80
Disgust	0.76	0.76	0.74	0.78
Anger	0.79	0.78	0.79	0.81
Surprise	0.76	0.77	0.76	0.79
Fear	0.78	0.78	0.77	0.79
Contempt	0.73	0.72	0.74	0.74
Neutral	0.88	0.87	0.88	0.90

Table 6

Comparative analysis: Recall

Emotions	Azure	Face++	Face Reader	Proposed
				System
Happy	0.80	0.81	0.79	0.82
Sad	0.81	0.82	0.83	0.88
Disgust	0.84	0.85	0.85	0.87
Anger	0.82	0.81	0.89	0.84
Surprise	0.82	0.81	0.81	0.84
Fear	0.81	0.70	0.71	0.72
Contempt	0.79	0.82	0.81	0.84
Neutral	0.87	0.85	0.88	0.89

Fig. 12

Comparative analysis of different emotion applications (Specificity).

Table 5 shows the comparative analysis for precision of the proposed method and existing FER applications. For all the emotions, the proposed method achieved better precision or equal precision value when compared existing method and achieved maximum of 0.9 for neutral emotion and minimum of 0.74 for contempt emotion. Figure 13 shows the graphical representation of precision achieved in proposed work which shows that proposed work achieved better values for all the eight emotions taken for consideration, so proposed work outperforms with maximum precision value of 0.90. The higher precision value leads to accurate emotion detection and classification, reduced false positives and suble emotion changes. Table 6 shows the comparative analysis for recall of the proposed method and existing FER applications.

Fig. 13

Comparative analysis of different emotion applications (Precision).

For all the emotions, the proposed method achieved better precision or equal precision value when compared existing method and achieved maximum of 0.89 for neutral emotion and minimum of 0.72 for fear emotion. For fear emotion Azur application achieved better recall value when compared to proposed precision value. Figure 14 shows the graphical representation of recall achieved in proposed work which shows that proposed work achieved better values for all the eight emotions taken for consideration, so proposed work outperforms with maximum Recall value of 0.89. The higher recall value leads to comprehensive emotion detection, reduced missed emotions and very effective in real world scenarios. In situations where missing an emotion could lead to incorrect decisions or judgments, higher recall is crucial to prevent unfair or biased outcomes.

Fig. 14

Comparative analysis of different emotion applications (Recall).

D. Performance Analysis: Execution Time

The best recognition were detected in proposed method, where it was obtained almost 98% in the MMI database. We saw that facial images with extreme exhibited expressions recorded the best results. The outcome shows that the arrangement of the muscles around the eyes and the mouths are the most dependable decisive for auto facial expressions emotions. This accounts why recognition in the neutral face is poor. Thus the gain in these muscle arrangement increases the quality of auto recognition. The execution time for a pixel of size 100×100 is 2.3 Secs. Figure 15 and Table 7 shows the comparative performance of execution time with other famous organization’s face emotion recognition application. Having a lesser execution time in facial emotion analysis offers several advantages that can enhance the usability, responsiveness, and efficiency of the analysis process. It helps to achieve quick decision making, Efficient resource utilization, reduced latency, and In scenarios where emotion analysis needs to be performed on a large volume of data or in a crowd, reducing execution time allows for more efficient processing and quicker insights.

Fig. 15

Comparative analysis: Execution time.

Table 7

Comparative analysis: Execution time

Images with 100×100	Proposed method Execution Time in Secs	Azur Execution Time in Secs	Face++ Execution Time in Secs	Face Reader Execution Time in Secs
1	2.5	3.1	3.4	4.1
2	2.8	3.0	3.1	4.0
3	2.9	3.2	3.2	3.9
4	2.4	3.4	3.3	4.2
5	2.2	3.5	3.5	4.3

E. Performance Analysis: Confusion Matrix

We recorded an average recognition rate of 97.6% in MMI Database. The confusion matrix shows the performance of the proposed algorithms based on the recognition rate. Table 8 and Fig. 16 shows the confusion matrix and its pictorial representation. It represents that the proposed method done good classification of all emotions with higher accuracy, precision and sensitivity for the taken MMI database.

Table 8

Confusion matrix

Emotions	Contempt	Neutral	Sad	Fear	Anger	Disgust	Happy	Surprise
Contempt	94.3	0	0	0	0	0	0	0
Neutral	0	92.23	4.31	1.11	2.35	0	0	0
Sad	0	3.85	93.9	1.3	0.97	0	0	0
Fear	2.80	0	0.95	96.08	0	0	0	0
Anger	0	2.80	0	1.1	96.10	0	0	0
Disgust	0	0	0.71	0	0.61	99.91	0	0
Happy	0	0.21	0	0.07	0	0	99.72	0
Surprise	2.91	0	0	0.1	0	0.03	0	99.87

Fig. 16

Performance analysis: Confusion matrix.

4 Conclusion

This study uses a variety of cutting-edge strategies to increase the recognition rate of the facial expression recognition system. The Cascaded Object Detection was used to carry out face detection. To lower the computational cost and prevent mis classification, features are extracted using MRBE and MSER Algorithms. A fast R-CNN classifier was fed the chosen features. Thus, sample datasets from MMI face expression databases are used to train the network. The average recognition rate in the MMI database is 97.6 %, and the execution time for a pixel with a size of 100×100 is 2.1 seconds. The performance of the suggested method is exceptional when compared to other ways. The study’s findings also demonstrate that automatic expression identification is approximately 100% accurate when detecting surprise, disgust, and happiness. The accuracy of mild emotions like sadness, fear, and contempt are lower. However, when anxiety is at its height, recognition accuracy can be quite precise due to the importance of facial deformations around the mouth and eyes. The number of neurons may be increased to accommodate expressions that recorded lower values in the future when recognition accuracy is improved. Emotions involve dynamic transitions, and relying solely on a single frame’s analysis might miss valuable information. Future work in facial emotion analysis is likely to focus on addressing existing challenges, improving accuracy, and expanding the application of emotion analysis technologies by multi-modal analysis, dynamic emotion analysis, cross cultural and age group analysis in real world applications with the help of Artificial Intelligence methods.

References

Darwin

The Expression of the Emotions in Man and Animals, London, 2008.

Ekman

Darwin, Deception, and Facial Expression, ANNALS of the New York Academy of Sciences, 2006.

Sivapatham

, Arasakumaran

, Annappan

and Chandrasekaran

, Analysis of genetic face images with respect to reflexology for prediction of diseases, Traitement du Signal 40(1) (2023).

Umamageswari

, Bharathiraja

and Irene

D.S.

, A novel fuzzy C-means based chameleon swarm algorithm for segmentation and progressive neural architecture search for plant disease classification, ICT Express 9 (2)(2023), 160–167.

Chen

, New insights into the noise reduction Wiener filter,, IEEE Trans Audio Speech Lang Process 14 (2006), 1218–1234.

Viola

and Jones

M.J.

, Robust real-time face detection,, International Journal of Computer Vision 57 (2004), 137–154.

Maalej

, Amor

, Daoudi

, Srivastava

, Berretti

Local 3D shape analysis for facial expression recognition, In: IEEE 20th International Conference on Pattern Recognition, 2010, pp. 4129–4132.

Satiyan

, Nagarajan

Recognition of facial expression using Haar-like feature extraction method, In: IEEE International Conference on Intelligent and Advanced Systems, 2010, pp. 1–4.

Wang

Y.-Q.

, An analysis of the viola-jones face detection algorithm,, Image Process Line 4 (2014), 128–148.

10.

Prasad

and Danti

, Classification of human facial expression based on mouth feature using SUSAN edge operator, Int J Adv Res Comput Sci Softw Eng 4(12) (2014), 440–443.

11.

Matas

, Chum

, Urban

and Pajdla

, Robust wide-baseline stereo from maximally stable extremal regions, Image and Vision Computing 22(10) (2004), 761–767.

12.

Chowdary

M.K.

, Hemanth

D.J.

, Angelopoulou

, Kapetanios

Feature extraction techniques for human emotion identification from face images, In 9th International Conference on Imaging for Crime Detection and Prevention, 2019, pp. 86–92. IET.

13.

Kanade

, Cohn

, Tian

Comprehensive database for facial expression analysis, in Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000, Grenoble.

14.

Pantic

, Valstar

, Rademaker

, Maat

Web based database for facial expression analysis, in 2005 IEEE International Conference on Multimedia and Expo, Amsterdam, 2005, p. 5.

15.

Deepa

, Umamageswari

, Menaka

A Novel Hand Gesture Recognition for Aphonic People Using Convolutional Neural Network, In Computer Vision and Machine Intelligence Paradigms for SDGs: Select Proceedings of ICRTAC-CVMIP 2021, Singapore: Springer Nature Singa pore, pp. 235–243.

16.

Larry Zitnick

P.D.

Edge Boxes: Locating Object Proposals from Edges, in European Conference on Computer Vision, Part V 13, 2014, pp. 391–405.

17.

Girshick

Fast R-CNN, Microsoft Research, 2015.

18.

Ganga

M.P.

, Prakash

, Gangashetty

S.V.

Bessel transform for image resizing, In: IEEE 18th International Conference on Systems, Signals and Image Processing, 2011, pp. 1–4.

19.

Jones

J.P.

and Palmer

L.A.

, An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex,, Journal of Neurophysiology 58 (2017), 1233–1258.

20.

Kharat

G.U.

, Dudul

S.V.

Emotion recognition from facial expression using neural networks. Human–Computer Systems Interaction, Berlin, Heidelberg: Springer, 2009, pp. 207–219.

21.

Kumbhar

, Jadhav

and Patil

, Facial expression recognition based on image feature,, International Journal of Computer and Communication Engineering 1 (2012), 117–119.

22.

Lades

, Vorbruggen

J.C.

, Buhmann

, Lange

and von

, der Malsburg and R.P. Wurtz, Distortion invariant object recognition in the dynamic link architecture, IEEE Transactions on Computers 42 (2009), 300–311.

23.

Lee

C.C.

, Huang

S.S.

and Shih

C.Y.

, Facial affect recognition using regularized discriminant analysis based algorithms,, EURASIP Journal on Advances in Signal Processing 1 (2010), 1–10.

24.

Lekshmi

V.P.

and Sasikumar

, Analysis of facial expression using Gabor and SVM,, International Journal of Recent Trends in Engineering 2 (2009), 47.

25.

Londhe

and Pawar

, Facial expression recognition based on affine moment invariants,, International Journal of Computer Science Issues 9 (2012), 388.

26.

and Khorasani

, Facial expression recognition using constructive feedforward neural networks,, IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 34 (2004), 1588–1595.

27.

Umamageswari

, Deepa

and Raja

, An enhanced approach for leaf disease identification and classification using deep learning techniques, Measurement: Sensors 24 (2022), 100568. ISSN2665-9174.

28.

Umamageswari

and Shiny

, Duela, D. Sara and K. Raja, An enhanced identification and classification algorithm for plant leaf diseases based on deep learning, Traitement Du Signal 39(3) (2022), 1013–1018.