Fuzzy speech emotion recognition considering semantic awareness

2.1 Traditional machine learning-based methods

Traditional machine learning-based methods for SER rely on manually designed features and annotations, then use models to recognize and classify these features into different emotions. It mainly consists of two parts, namely emotion feature extraction and emotion classification algorithm. The accuracy of emotion feature extraction determines the performance of the model. The acoustic features in speech features are directly related to emotion [11, 12], and the classification algorithm uses linear or nonlinear classifiers for feature classification. Linear classifiers included Bayesian networks and support vector machines (SVM). Yang N et al. [13] used Back Propagation Neural Network (BPNN), Extreme Learning Machine (ELM), Probabilistic Neural Network (PNN) and SVM as classifiers to extract the MFCC features for SER, and concluded that the MFCC features and the SVM classification models are highly effective in the automatic prediction of emotion. Moreover, Zhang et al. [14] proposed a hierarchical classification to classify the implicit emotions into 19 classes of four levels using an SVM, but the model is time-consuming. Nonlinear classifiers included Gaussian mixture models (GMM) and hidden Markov models (HMM). Shahin et al. [15] showed a hybrid Gaussian mixture model and deep neural network performance in the presence of noise signals. Fahad [16] proposed a novel epoch-based emotion feature and fused it with other acoustic features to improve performance. These models rely on manual feature annotations, and the quality of feature annotations determines the performance of the model. At the same time, they ignore time information, which reduces the accuracy of traditional machine learning emotion classification.

2.2 Deep learning-based methods

With the rapid development of deep learning technology, deep learning has gradually replaced machine learning as the main analysis method of SER. It uses deep learning models to automatically learn features in speech data. Compared with traditional machine learning technology for emotion recognition, it can automatically detect and extract complex structures and features in data without manual feature extraction and adjustment; extract features from raw data without feature extraction preprocessing; handle unmarked data. Among them, DNN are based on feedforward structures, which consist of one or more hidden layers between input and output. It has been successfully applied to SER’s emotion classification. Alswaidan N et al. [17] surveyed the state-of-the-art research for explicit and implicit emotion recognition in text, and concluded that deep learning models can capture explicit and implicit information in automatic feature extraction. Sun et al. [18] proposed a speech emotion recognition method based on DNN hybrid model. The model can not only accurately classify speech signals into emotions but also focus on identifying certain unique emotions to alleviate the emotional ambiguity. Recurrent Neural Networks (RNN) are very effective in SER and natural language processing. Mirsamadi et al. [19] used RNN to learn short-term acoustic features related to emotions and used local attention mechanisms to make the model focus on extracting specific areas with more prominent emotions to improve the model’s recognition rate. Although RNN can extract emotional features of short-time speech, however, when processing long sequences, RNN will be problems of gradient disappearance and gradient explosion [20]. Therefore, Long Short-Term Memory (LSTM) is introduced, which can effectively solve the problems that RNN cannot handle. Xie et al. [21] proposed a LSTM model based on the attention mechanism, which combines frame-level speech features with LSTM for SER. Convolutional Neural Networks (CNN) have been introduced into SER because they can extract abundant spatial information. Satt et al. [22] divided the speech input into smaller segments and used CNN to extract spatial information of speech through spectral information of speech, then used LSTM to obtain temporal information of speech, which can greatly improve the accuracy of emotion recognition. Hazarika et al. [23] proposed the interactive conversation memory network (ICON), which combine bidirectional LSTM with a novel pooling strategy using an attention mechanism which enables the network to focus on emotionally salient parts of a sentence. Experimental results on the benchmark dataset IEMOCAP demonstrate that the proposed model outperforms state-of-the-art net-works. Li et al. [24] proposed an attention pooling based representation learning method for SER. The model utilized two sets of convolutional filters with distinct shapes to capture temporal and frequency domain contextual information for speech data, results demonstrate a recognition performance of 71.8% weighted accuracy and 68% unweighted accuracy over four emotions. The single-modal emotion recognition has achieved good results, but the recognition results are not robust. Therefore, Yoon et al. [25] proposed a novel Deep Bidirectional Recurrent Encoder model that leverages both textual data and audio signals to better understand speech data. Specifically, the model employs dual RNNs to encode information from the audio and text sequences, and then combines this information from both sources to predict emotion categories. Transformer has better performance in natural language processing tasks and can extract long-term dependent time information. As a result, researchers began to apply it to SER [26]. Tang et al. [27] proposed a bimodal network based on Audio-Text-Interactional-Attention with ArcFace losss for speech emotion recognition, adopted GRU for textual modality, BLSTM for audio and global attention for obtaining the interactive information of two modalities. Experimental results on the benchmark dataset IEMOCAP demonstrate that the method achieved the improvement of 2.4% and 1.8% in terms of weighted accuracy and unweighted accuracy, respectively. Braunschweiler et al. [28] conducted research on the impact of speech and text on emotion recognition. The experiments have shown that combining speech and text only improves performance when there are significant variations in textual expressions across different emotions on individual corpora, performance declines when the same text expresses different emotions in fixed corpora, and models based solely on speech perform better in such cases.

3.1 Overview of the modal

To fully recognize the emotions contained in speech and improve the recognition effect of implicit emotions, we propose a semantic aware speech emotion recognition model (SASER). As shown in Fig. 1, SASER includes four modules: feature extraction module, speech embedding support module, feature fusion module, and classification module. The feature extraction module includes a speech feature extraction module and a semantic aware extraction module. In the speech feature extraction module, CNN is used to extract local features of speech, and Transformer model is used to extract global features of speech, which solves the problem that CNN ignores global features when extracting local features of speech. Then, the speech content is converted into text using speech recognition conversion technology, and the text semantic aware module uses GRU model to extract semantic features of the text. The semantic embedding support module uses fine-grained alignment between speech frames and text words to guide finely alignment of the emotional words and speech sequences to effectively solve the embedding difference problem, especially the semantic aware as an auxiliary feature to help the speech feature extraction module extract implicit emotions contained in the speech. The feature fusion module uses two key-value attention mechanisms to fuse two different modal emotion features into new features and then completes emotion recognition through the classification module.

OPEN IN VIEWER

Fig. 1

SASER model framework.

The feature extraction module includes a speech feature extraction module and a semantic aware extraction module. The speech feature extraction module includes CNN and Transformer models. Due to the asymmetry between the input features of CNN and Transformer, we connect CNN and Transformer in parallel. CNN has good feature extraction and selection functions, so the CNN structure is used to extract emotional features at a lower dimension of the model. Applying CNN to the model of SERF can overcome the diversity of speech signals by using the translational invariance of convolution. The formula for CNN is expressed as follows: v i , j xy = ∑ f ∑ w = 0 W i - 1 ∑ h = 0 H i - 1 w ijf ′ wh v ( i - 1 ) f ( x + w ) ( y + h ) + b ij(1) where v i , j xy represents the convolution result of the j-th feature map output by the i-th convolution layer at position (x, y), f represents the number of convolution kernels, W _i represents the width of the convolution kernel, and H _i represents the height of the convolution kernel. Transformer includes input module, Encoder module, Decoder module, and output module. Its core structure is multi-head attention. The low-level features of speech are encoded and projected to the corresponding Q, K, V matrix through linear layers. The corresponding Q, K, V self-attention matrix Z is calculated by the scale dot product attention module, whose calculation formula is shown below: A = QK T d k(2) Z = softmax ( A ) V(3) where A represents the similarity matrix of Q and K, and Z represents the self-attention matrix. The multiple calculated matrices Z are spliced, and the global features of the speech are obtained through linear changes. The final output formula is as follows: F = Concat ( Z 1 , Z 2 , ⋯ , Z h ) W 0(4) where h represents the number of multi-head attention, Concat (·) represents the concatenation, and W⁰ is the parameter matrix of the linear layer. The local features extracted by CNN are concatenated with the global features extracted by Transformer to obtain the local information and global information of speech features. Finally, the output of the speech feature extraction module is A = {a₁, a₂, ⋯ , a _L }.

The text semantic aware module preprocesses the text data, constructs and extracts the text representation, quantifies the text features, and then inputs them into the GRU network to complete the training of the model. GRU is a variant of LSTM with fewer parameters and lower costs. It can effectively solve the problems of gradient explosion and gradient disappearance in the long sequence training process, and reduce the training time of the model, which makes GRU perform better in the long sequence. Assuming the current input sequence is x _t , the previous hidden state is h_t-1, and the current hidden state h _t can be represented as: h t = ( 1 - z t ) ⊙ h t - 1 + z t ⊙ h t - 1 ′(5) among them, ⊙ represents Hadamard product, t ∈ [1, N], N represents sequence length; update gate z _t can control activation unit, and z _t is defined as follows function: z t = σ ( W z x t + U z h t - 1 )(6)

The candidate activation unit h t ′ is similar to the traditional recurrent unit, which is defined as follows: h t ′ = tanh ( W h x t + U h ( r t ⊙ h t - 1 ) )(7)

Reset gate r _t can be represented as: r t = σ ( W r x t + U r h t - 1 )(8) where U _z , U _h , W _z , W _h , W _r represents learnable model parameters. The output of the final semantic perception extraction module is T = [t₁, t₂, ⋯ , t _n ].

The semantic embedding support module can solve the embedding differences between two modalities of speech and text, and use semantic information as auxiliary features to help the speech feature extraction module extract implicit emotions contained in the speech. We use semantic embedding to learn the alignment between speech frames and text words, so that emotional words are finely aligned with sequences in speech rather than being forced to align, thereby mapping speech frames to the latent space of text features. Specifically, given an encoded speech embedding A and text embedding T, align the i-th speech frame with the j-th sentiment word, and normalize it to obtain the attention weight α ij ′ of the speech sequence: α ij = T j T a i(9) α ij ′ = exp ( λ α ij ) ∑ k = 1 n exp ( λ α kj )(10) where λ = 1 / d represents the scale factor used to prevent exploding gradients. Finally, the contextual features of the speech frame U = {u₁, u₂, ⋯ , u _m } are obtained according to the text T _j alignment. u j = ∑ i = 1 n α ij ′ A i(11)

The main multimodal fusion strategies for speech and text are feature-level fusion, model-level fusion, and decision-level fusion. Feature-level fusion directly concatenates, and weights, then sums the features of the two modalities to obtain a completely new feature. It has the advantages of being simple and efficient, and fully retaining the original information, but it may cause dimension explosion because of a large feature dimension; model-level fusion models the intra-modal and inter-modal relationships of multiple modalities to better explore the complementarity between multiple modalities and obtain better multimodal feature representations. Different modalities are encoded separately, and the high-level feature representations output by different modality encoders are fused, and finally emotion prediction is based on the fused features. Decision-level fusion refers to the fact that two different modalities do not affect each other, and the emotion recognition results of each modality are weighted and fused. Although decision-level fusion does not have the problem of dimension explosion, it also ignores the correlation between different modalities. Unlike the above method, the key-value attention mechanism of the feature fusion module uses text semantic features as external information to calculate the correlation between speech as internal information, using one modality to judge the importance of another modality in order to achieve fusion between two modalities. The key-value attention mechanism as shown in Fig. 2.

OPEN IN VIEWER

Fig. 2

The key-value attention mechanism.

Specifically, calculate the attention score e _ti of each key k _i with the query feature q _t , and use speech features to assist in scoring text features. W represents a learnable parameter matrix. Secondly, use the softmax function to normalize the attention score, obtain the weight α _i of each key, and weight the corresponding value α _i as the attention output. Finally, use the attention weight distribution to obtain important information in the text feature, thereby obtaining a new text feature representation M = [m₁, m₂, ⋯ , m _i ]. e ti = score ( q t , k i ) = q t T Wk i(12)

α i = Attention ( q t , K , V ) = ∑ i α ti v i(13)

M = α i · T T(14)

Use the text features to extract the important information of the speech features, and obtain a new text feature representation N = [n₁, n₂, ⋯ , n _j ], the calculation formula can be expressed as: N = α j · U T(15)

The newly learned features are input into the fully connected layer (FC) to predict the confidence score of the final emotional state. During the training phase, the target loss function is achieved by using a softmax layer with a cross-entropy function for classification, whose calculation formula is as follows: Loss = - ∑ D ∑ i ∈ N p c ′ · log p c + λ ∥ θ ∥ 2(16) where p c ′ represents the real emotional label; p _c represents the predicted emotional label; λ is the weight of L₂; θ represents all parameter sets.

4.1 Dataset

IEMOCAP [29]: The IEMOCAP dataset is an English database recorded by the University of Southern California. the dataset consisted of five groups of dialogues conducted by 10 professional speakers (5 male and 5 female). Record 12 hours of audio and video content, including improvisation and scripts, and divide emotions into anger, excitement, happiness, neutrality and sadness.

EMO-DB [30]: It contains 535 German speech samples and was recorded by ten actors (five male and five female). The dataset contains seven different emotions: anger, fear, disgust, boredom, neutral, happy, and sad. It is also a publicly available dataset in the field of SER research.

4.2 Experimental settings

We implement the proposed on SASER method with the deep learning toolbox PyTroch. The dataset was divided with 60% allocated for training, 20% for validation, and 20% for testing. During the training phase, we optimize the network using the Adma optimizer with weight decay 0.001. The GRU has a hid-den node dimension of 128. The CNN has a convolutional kernel size of 3x3, a stride of 2x2, and uses max-pooling. The Transformer has a dimension of 512, with 6 encoder layers and 8 attention heads. In addition, our model is trained for 60 epochs and a batch size is 64, the initial learning rate is 0.0001, and a dropout is 0.3, the regularization parameter is 0.01.

4.3 Evaluation metrics

In the multi-classification task, weighted average accuracy (WA) and unweighted average accuracy (UA) were used as evaluation indicators. Accuracy represents the proportion of correct classification: Accuracy = TP + TN TP + TN + FP + FN(17) where TP represents true positive samples, TN represents true negative samples, FP and FN represent false positive samples and false negative samples respectively. WA represents the weighted average accuracy of different emotion categories, and its weight is proportional to the number of samples in each category; UA represents the average accuracy of different emotion categories. WA = ∑ k = 1 K N k × Accuracy k ∑ k = 1 K N k(18) UA = ∑ k = 1 K Accuracy k K(19) where K represents the number of emotion categories in the dataset, N _k represents the number of samples of class k, and Accuracy _k represents the recognition accuracy of k samples.

In our experiments, we compared the SASER model with other models. Specifically, FSVM [12], RNN [19], ICON [23], CNN [22], CNN+LSTM [22], CNN TF Bilinear [24], MDRE [25], ATIA4 [27], SA-SER-SA, which removes the semantic aware extraction module, and SASER-Att, which removes the attention fusion mechanism. These models were tested on the IEMOCAP dataset and the EMO-DB dataset respectively. The results are shown in Table 1 and Table 2.

In the comparative models, [23], [25], [27] are multimodal emotion recognition models for textual and audio. From Table 1 and Table 2, we can find that SASER can achieve the higher scores on WA and UA metrics, which confirms that our method can improve the overall recognition rate of the model. In the IEMOCAP dataset, it can be seen that FSVM achieves a higher WA than CNN, this is because FSVM considers the intercorrelated features between emotions. In the EMO-DB dataset, the FSVM model obtained the lowest WA score, it could be attributed to the dataset was formed through actor performances. By comparing with ICON and RNN, it can be found that there is a contextual relationship in speech emotion. CNN+LSTM has a WA score that is 2.7% higher than CNN, 5% higher than ICON, and 5.3% higher than RNN, indicated that LSTM can effectively extract contextual features in speech emotion. CNN TF Bilinear outperforms CNN+LSTM and ICON in terms of UA, indicated that attention mechanisms can effectively identify distinctive information between modalities in multimodal speech emotion recognition. Comparing MDRE and ATIA3, it is evident that the interactive and fusion features between text and audio have a greater impact than the global information in multimodal settings. Com-paring SASER with CNN and its related models, it can be found that the performance of SASER is bet-ter than CNN, which proves that the SASER model can effectively make up for the shortcomings of CNN for missing global information, and can extracted the global features well. Comparing SASER with CNN TF Bilinear on the IEMOCAP dataset, the CNN TF Bilinear model has a higher UA than SA-SER, it may be that the CNN TF Bilinear model simultaneously considers contextual information in both the temporal and frequency domains for different emotions. By comparing with the Transformer model, the SASER model can better extract the emotional features in speech. In the ablation experiment, it can be seen that the semantic aware module helps SASER improve the WA metrics by 6.5%, which proves that using semantic aware information as auxiliary information to assist the model can improve the extraction of speech emotional features, and improve the recognition rate of the model for implicit emotions. By comparing the ablation experiments of SASER-Att and SASER, it is proved that the attention mechanism is effective in the feature fusion process.

From Fig. 3 we can discover that the convergence speed of the SASER model in the IEMOCAP dataset is higher than that in the EMO-DB dataset, regardless of the training set or the test set. Because the EMO-DB dataset has 7 emotion classifications, while the IEMOCAP dataset has only 4 emotion classifications; The accuracy of the SASER model in the IEMOCAP dataset is 74.3%, and the accuracy in the EMO-DB dataset is 72.5%. It is higher than that in the EMO-DB dataset.

OPEN IN VIEWER

Fig. 3

Accuracy of the training process for different data sets.

Figures 4 and 5 show the recognition accuracy of each emotion category, we can observe that. 1) SASER has the highest recognition rate of neutral emotion in IEMOCAP and EMO-DB data sets, reaching 80.5% and 75.1% respectively, which indicates that the SASER model can improve the recognition rate of neutral emotion, since it can extract hidden emotion information from emotional features and avoid mistakenly recognizing hidden emotion as neutral emotion. Compared with IEMOCAP dataset, the recognition rate of EMO-DB is lower than that of the former. It may be that IEMOCAP has more data and fewer categories than EMO-DB, which will help model train a better network.

OPEN IN VIEWER

Fig. 4

Confusion matrices of accuracy on IEMOCAP.

OPEN IN VIEWER

Fig. 5

Confusion matrices of accuracy on EMO-DB.

To select the number of CNN modules with the best generalization ability, we comparing different number of CNN modules in the dataset, which is shown in Fig. 6. The number of CNN modules is increased in turn. The indicators increase first and then start to decrease. Among them, the IEMOCAP dataset reaches the maximum value when the number is 3, and the EMO-DB dataset reaches the maximum value when the number is 2. This indicates that the increase of the number of CNN modules will increase the complexity of the model, the training time becomes longer, so continuously increase the number of CNN cannot improve the accuracy of SER. Therefore, we selected the most appropriate number of CNN in the subsequent experiments.

OPEN IN VIEWER

Fig. 6

Performances of CNN with different numbers ondataset.

In order to intuitively demonstrate the effectiveness of the proposed semantic aware module for emotion classification, we compared the SASER model with the SASER-SA model without the semantic perception module and the SASER-Att model without the feature fusion module. The confusion matrices of the emotions recognized by the models in IEMOCAP dataset are shown in Figs. 7, Fig. 8 and Fig. 9.

OPEN IN VIEWER

Fig. 7

The confusion matrix of SASER on IEMOCAP.

OPEN IN VIEWER

Fig. 8

The confusion matrix of SASER-SA on IEMOCAP.

OPEN IN VIEWER

Fig. 9

The confusion matrix of SASER-Att on IEMOCAP.

Comparing Fig. 7 and Fig. 8, we can discover that the overall recognition rate of the SASER model for the four emotions is higher than that of the SASER-SA model, especially the recognition effect of neutral emotion is improved best, which is increased by 9.3%. It indicates that the semantic aware extraction module plays an important role in the recognition of neutral emotion and verifies the effectiveness of the proposed semantic aware extraction module. Comparing Fig. 7 and Fig. 9, it can be seen that the overall performance of the SASER model is higher than that of the SASER-Att model, and the error recognition rate of other emotions in the SASER model for neutral emotion is also reduced, which indicates that semantic information assistance has solved the problem of implicit emotion in speech to a certain extent. In addition, the semantic aware extraction module has a great effect on the recognition effect of neutral and sad emotions, while although the accuracy of anger and surprise recognition has been improved, the degree of improvement is small. This may owe to anger and surprise are high-emotion-changing emotional features that are easier to extract and recognize than neutral and sad emotions, and these two emotions can be well recognized even without semantic aware extraction module.

In addition, we further compare our SASER framework with two different types of unimodal settings, i.e., the “Speech” and “Text” models. Specifically, the “Speech” model is only built on the proposed speech feature extraction module, while the “Text” model is initialized with the proposed semantic aware extraction module. Table 3 shows the comparison results with different unimodal branches. As is shown above, the worst-performing model is “Speech,” where it achieves the WA scores of 68.4% and 65.4% on IEMOCAP and EMO-DB, respectively. By contrast, the “Text” model outperforms the “Speech” model, possibly because many people prefer to euphemistically express their emotions or rely on logic inferring. The accuracy of our proposed model is better than the types of unimodal settings.