Simulation of English speech emotion recognition based on transfer learning and CNN neural network

Abstract

The difference between English and Chinese expressions is that English emphasizes the stress of syllables, so the recognition of English speech emotions plays an important role in learning English. This study uses transfer learning as the technical support to study English speech emotion recognition. The acoustic model based on weight transfer has two different training strategies: single-stage training and two-stage training strategy. By comparing the performance of the English speech emotion recognition model based on CNN neural network and the model proposed in this paper, the statistical comparison data is drawn into a statistical graph. The research results show that transfer learning has certain advantages over other algorithms in English speech emotion recognition. In the subsequent teaching and real-time translation equipment research, transfer learning can be applied to English models.

Keywords

Transfer learning English speech emotion recognition features

1 Introduction

Speech Emotion Recognition (SER) is an important branch in the field of speech research. It is a technique to determine the emotions that people belong to and involves many core issues in speech research such as signal processing, feature extraction and pattern recognition. Emotion recognition is based on the theoretical research of human emotion. Darwin, the founder of evolution, believes that emotion is a broad trait shared by all human beings across ethnic groups and cultures [1]. Some experts and scholars have done more detailed research on human emotions. In their research, all different cultures have a common expression of emotions. These emotions can be divided into happiness, sadness, anger, etc. [2], which has become the theoretical basis of the field of emotion recognition. On the other hand, emotion, as a complex psychological activity of human beings, is an expression of the true thoughts in the heart, a predictive of the person’s next move and can even reflect a person’s health level. Due to the cross-cultural and inter-ethnic characteristics of emotion and its great significance to people, the research on emotion recognition has been paid more and more attention.

Emotion, as a complex set of human physiological and psychological activities, is a research hotspot in many disciplines such as psychology and physiology. At the same time, emotion recognition, as an important part of the intelligent human-computer interaction system, is of great significance for improving machine intelligence, improving machine friendliness, and creating a smart city for us. In addition, there are many other application scenarios for emotion recognition. In the field of education, emotion recognition technology recognizes students ‘emotional changes in the learning process through signals such as students’ facial expression changes, so as to judge their concentration and understanding of the content taught by the teacher. At the same time, the judgment of the machine can also be fed back to the teacher to help them improve the teaching process [3]. For example, if the machine judges that the emotions of the students are mainly confused and frustrated, it means that the content taught by the teacher is too difficult or the teacher’s teaching method is inappropriate. At this time, the teacher should reflect on whether the content is difficult, or the teaching method needs to be improved.

The difference between English and Chinese expressions is that English emphasizes the stress of syllables, so the recognition of English speech emotions plays an important role in learning English. English speech emotion recognition is to make the machine understand the human language and convert the speech information into the corresponding text. Speech emotion recognition is the premise of intelligent information processing, has very important research value and practical significance, and brings many conveniences to people’s lives [4].

This study uses transfer learning as a technical support to study English speech emotion recognition, so as to provide reference for English learners, and also provide theoretical research reference for real-time English translation.

2 Related work

Speech recognition appears before the computer. In the 1920 s, the world’s first speech recognition machine “Radio Rex” came out. Modern automatic speech recognition technology was born in the AT & T Bell Labs in the 1950 s. Based on the formant information of digital vowels, the researchers developed a system that can recognize numbers [5], and the system can only recognize ten isolated English numbers spoken by a specific person. Later, the application of dynamic time warping (DTW) method in the field of speech recognition [6] enabled different lengths of speech to be aligned in time series, which further stimulated the rapid development of speech recognition technology, and at the same time laid a solid foundation for the development of speech recognition technology of continuous speech. Since then, research on speech recognition technology has transitioned from the recognition of isolated words to the Large Vocabulary Continuous Speech Recognition (LVCSR). From the 1980 s to the 1990 s, the method based on probability statistics replaced the method based on template matching, which marked the beginning of the rapid development of speech recognition technology. Hidden Markov Model (HMM), Gaussian Mixture Model (GMM) and n-gram language models have emerged one after another, which has gradually improved the performance of large vocabulary continuous speech recognition systems. After that, the performance of the speech recognition system with the GMM-HMM acoustic model and the n-gram language model [7] as the main frame has been a key breakthrough, and it has also achieved good results for continuous recognition of large vocabulary. After entering the 21st century, the speech recognition system based on GMM-HMM is relatively mature, but the performance of its system still cannot reach the practical level. During this period of time, the development of speech recognition technology has been unable to break through the bottleneck. With the development of deep learning and the rise of neural networks, speech recognition technology has been further breakthrough and gradually entered a climax. The literature [8] proposes a layer-by-layer initialization training method in Deep Belief Networks (DBN). The literature [9] proposed an acoustic model based on DBN-HMM, and this model fits the posterior probability of the HMM model state through DBN and takes training on the TIMIT data set and has achieved good results. The literature [10] proposed a hybrid acoustic model based on context-sensitive deep neural network (DNN) and hidden Markov. Compared with the GMM-HMM acoustic model, the performance of this model has been significantly improved. After that, the emergence of various neural networks with different structures and the widespread application in the field of speech recognition marked the official development of speech recognition technology to its peak. Long-Short Term Memory (LSTM) and Time Delay Neural Network (TDNN) can model long-term speech signals and gradually become the mainstream acoustic model in speech recognition systems. In addition, Convolutional Neural Network (CNN) can effectively overcome the diversity of speech signals through convolution calculation, which improves the recognition rate of acoustic models to a certain extent [11].

The method of “learning to learn” proposed in the literature [12] is the prototype of transfer learning thought. The Defense Advanced Research Projects Agency (DARPA) clearly defined that the classification system based on transfer learning has the ability to apply the knowledge that has been learned to other tasks. Since then, transfer learning technology has been widely researched and applied. The research in the literature [13] showed that transfer learning can effectively solve the problem of data sparseness by applying the knowledge learned from one model to another model. With the development of machine learning and deep learning, deep neural network models have been used in various classification fields. One of its advantages is that, by setting a multi-layer network structure to extract features layer by layer, these features can be migrated layer by layer as needed. The literature [14] proposed that the features learned by the hidden layer closer to the input layer (lower) are more general (general feature), and the features of the hidden layer closer to the output layer (higher) are more specific, which has played a key guiding role in the development of transfer learning. The application of transfer learning in the field of speech recognition is mainly reflected in two methods: domain adaptation and pre-training. The literature [15] proposed a domain adaptive technology based on linear hidden network (LHN) and fDLR method, which has achieved good results in speech recognition systems. The literature [16] used deep belief networks to model phoneme sequences, and proposed that the features of the middle layer of the acoustic model are not specific to the training data. The unsupervised pre-training method proposed in the literature [17] and the supervised pre-training method using the extra-domain data used in the literature [18] have been successively applied to speech recognition systems, which has further developed the application of transfer learning in speech recognition. The literature [19] used the multi-language trained DNN model as the initial model of the low-resource target language, which has achieved a significant improvement in the target domain. In terms of cross-language transfer, the DNN-based acoustic model of English speech recognition proposed in the literature [20] has achieved better results than the baseline system in the target domain French. Although the speech recognition technology has been greatly promoted by deep learning, the performance of speech recognition systems with small data sets has not yet reached the level required by people. How to improve the performance of the acoustic model of low-resource speech recognition system is the main content of this study, and transfer learning is an effective method to solve this problem.

3 Autoencoder

Autoencoder (AE) is a neural network model that learns the hidden features of data through training to make the network output equal to the input. It is currently used for pre-training of deep neural networks and is one of the main network structures of deep learning. The learning process of the autoencoder is an unsupervised learning, that is, no labeled data samples are needed for training. The process of learning from input data to express features efficiently is called encoding, and the process of reconstructing the original data through the hidden features learned is called decoding. After the feature is extracted from the data by the autoencoder, it can be sent to the supervised learning model to complete the required tasks. In this process, the role of the autoencoder is to act as a feature extractor to increase or decrease the dimension of the data to obtain a suitable and efficient hidden feature representation. The basic structure of the autoencoder is similar to a three-layer neural network, that is, it includes an input layer, a hidden layer, and an output layer, as shown in Fig. 1. An input sample is {x₁, x₂, x₃, ⋯ , x_i }, and the forward propagation formula of the autoencoder is shown in Equations (1) and (2). Among them, S is the activation function, which is generally a sigmoid function, W₁ is the weight between the input layer and the hidden layer, W₂ is the weight between the hidden layer and the output layer, b_i is the bias term, and h_ω,b (x) is the activation value of the output layer. The expected output of the self-encoder training target is equal to the input, and the final learning result is shown in formula (3) [21 –23]. $d = S (W_{1} x + b_{1})$ (1) $h_{ω, b} (x) = S (W_{2} d + b_{2})$ (2) $h_{ω, b} (x) \approx x$ (3)

Fig. 1

Basic structure diagram of autoencoder.

The self-encoder uses training to make the output equal to the input as much as possible and uses different numbers of hidden units to obtain hidden features of different dimensions, so that the original data can be dimensioned or reduced. By stacking multiple autoencoders together and adding constraints, we can learn the efficient representation of data at different levels. Therefore, the autoencoder can be used as pre-training for deep learning, extracting useful data features, which is helpful for subsequent supervised learning.

4 Recurrent neural network

The recurrent neural network is an improved version of traditional neural network, which is a neural network used to analyze time series data. The traditional artificial neural network generally refers to the feedforward neural network. In a feedforward neural network, each neuron starts at the input layer, receives the input of the previous stage, and outputs the input to the next stage, and finally outputs to the output layer. Moreover, there is no information feedback in the entire network. Due to its structural limitations, the network has no temporal characteristics, and it can only process each input sample separately, and there is no correlation between each input. Therefore, it is often not effective when dealing with time series problems. Many tasks are time-related, so we need to mine their time feature information. Based on the traditional artificial neural network, RNN introduces a directional loop, which makes hidden nodes directional connection into a loop. The internal structure of the ring helps the network to transfer time information and demonstrate dynamic timing behavior and helps the network to mine the characteristics of time series and finally make a classification or prediction.

The structure of traditional artificial neural network and RNN is shown in Fig. 2. It can be seen that the main difference between them is that RNN has an additional information transmission channel in the hidden layer, which makes the network have a time characteristic. The current time of the network can not only receive the information of the sample, but also the information of the previous time, so that the network can handle the time series problem well.

Fig. 2

Comparison of the structure of traditional neural network (left) and RNN (right).

The RNN in Fig. 2 can be expanded into Fig. 3. The forward propagation formula of the RNN can be derived from Fig. 3, as shown in formulas (4)–(7). Among them, x is the network input, y is the network output, w is the weight of the network to connect each neuron, z is the weighted sum of neurons, f is the activation function, s is the neuron activation value, t represents the current time of the network, I is the input vector dimension, i is the input vector subscript, H is the hidden layer unit dimension, h′ is the hidden layer unit vector index at time t - 1, g is the hidden layer unit vector index at time m, and o is the output vector index. From the structure diagram and the forward propagation formula, it can be derived that the value s^t of the RNN hidden layer depends not only on the current input x^t, but also on the value s^t-1 of the previous hidden layer. The current output of the RNN is affected by inputs such as x, x^t-1, x^t-2, ⋯. Therefore, the network can update the weights through training, extract the useful information from the entire time series, and obtain a good classification or prediction effect. $z_{h}^{t} = \sum_{i = 1}^{I} w_{ih} x_{i}^{t} + \sum_{h^{'} = 1}^{H} {wh}^{'} h s_{h^{'}}^{t - 1}$ (4) $s_{h}^{t} = f_{h} (z_{h}^{t})$ (5) $z_{o}^{t} = \sum_{h = 1}^{H} w_{ho} s_{h}^{t}$ (6) $\hat{y} = f_{o} (z_{o}^{t})$ (7)

Fig. 3

Expanded view of RNN structure.

$\hat{y} = f_{o} (\sum_{h = 1}^{H} \begin{matrix} w_{ho} f_{h} \\ (\sum_{i = 1}^{I} \begin{matrix} w_{ih} x_{i}^{t} + \\ \sum_{h^{'} = 1}^{H} \begin{matrix} w_{h^{'} h} f_{h^{'}} \\ (\sum_{i = 1}^{I} \begin{matrix} w_{ih} x_{i}^{t - 1} + \\ \sum_{h^{″} = 1}^{H} \begin{matrix} w_{h^{″} h^{'}} f_{h^{″}} \\ (\sum_{i = 1}^{I} w_{{ih}^{″}} x_{i}^{t - 2} + \dots) \end{matrix} \end{matrix}) \end{matrix} \end{matrix}) \end{matrix})$ (8)

5 Network training methods

For the least squares problem, the gradient descent method is the most classic optimization method. In this method, the minimum value of the function is iteratively solved step by step along the direction of gradient descent, so as to obtain the minimum loss function and the minimum model parameters. The gradient descent method can deal with the case where the loss function is clear, or the error is available. For example, for a multi-layer feedforward neural network, the parameters of the output layer can be

sealed using gradient descent and the error of the output layer. However, there are no errors in other hidden layers, so the gradient descent method cannot be used directly. In this study, the chain rule is used to propagate the error to the hidden layer in the reverse direction of the network, and then the gradient descent method is used to update the parameters. This training method is called the back propagation (BP) algorithm. This multi-layer feedforward neural network trained according to the error back propagation algorithm is currently the most widely used, also known as BP neural network. At present, for the training of various deep neural networks, the algorithms used are basically BP algorithms. The training method of RNN is Back Propagation Through Time (BPTT) over time. The idea of this method is similar to the BP algorithm. Since RNN is a neural network model based on time series, the idea of BPTT algorithm is to back-propagate the accumulated residuals from the last time. This section will introduce the back-propagation algorithm and the back propagation algorithm through time.

We assume that the training sample set is { (x₁, y₁) , ⋯ , (x_m, y_m) }, which means that there are m samples in total, and the complex nonlinear model of the neural network is represented by h_W,b (x). Among them, W, b respectively represent the weight and bias terms of the neural network, and the loss function of the network is expressed by the mean square error, as shown in formula (9). $J (W, b) = \frac{1}{m} \sum_{i = 1}^{m} \frac{1}{2} {∥ h_{W, b} (x_{i}) - y_{i} ∥}^{2}$ (9)

According to the gradient descent method, the parameter W, b is updated according to equations (10) and (11) in each iteration. η represents the learning rate of the network, and $W_{ij}^{(l)}$ represents the connection weight of the j-th neuron in the l - 1 -th layer and the i-th neuron in the l-th layer. Among them, J (W, b) in the formula represents the overall sample cost function, so the partial derivative calculation in the formula is shown in formulas (12) and (13). ${W^{'}}_{ij}^{(l)} = W_{ij}^{(l)} - η \frac{\partial}{\partial W_{ij}^{(l)}} J (W, b)$ (10) $b_{i}^{' (l)} = b_{i}^{(l)} - η \frac{\partial}{\partial b_{i}^{(l)}} J (W, b)$ (11) $\frac{\partial}{\partial W_{ij}^{(l)}} J (W, b) = \frac{1}{m} \sum_{k = 1}^{m} \frac{\partial}{\partial W_{ij}^{(l)}} J (W, b, x_{k}, y_{k})$ (12) $\frac{\partial}{\partial b_{i}^{(l)}} J (W, b) = \frac{1}{m} \sum_{k = 1}^{m} \frac{\partial}{\partial b_{i}^{(l)}} J (W, b, x_{k}, y_{k})$ (13)

Here, the chain rule for derivation of composite functions is used, as shown in Equations (14) and (15). Among them, z is the weighted sum and a is the activation value. $\begin{matrix} \frac{\partial}{\partial W_{ij}^{(l)}} J (W, b, x_{k}, y_{k}) = \frac{\partial J (W, b, x_{k}, y_{k})}{\partial z_{i}^{(l)}} \frac{\partial z_{i}^{(l)}}{\partial W_{ij}^{(l)}} \\ = \frac{\partial J (W, b, x_{k}, y_{k})}{\partial z_{i}^{(l)}} a_{j}^{(l - 1)} \end{matrix}$ (14) $\begin{matrix} \frac{\partial}{\partial b_{i}^{(l)}} J (W, b, x_{k}, y_{k}) = \frac{\partial J (W, b, x_{k}, y_{k})}{\partial z_{i}^{(l)}} \frac{\partial z_{i}^{(l)}}{\partial b_{i}^{(l)}} \\ = \frac{\partial J (W, b, x_{k}, y_{k})}{\partial z_{i}^{(l)}} \end{matrix}$ (15)

We set $δ_{i}^{(n_{l})} = \frac{\partial J (W, b, x_{k}, y_{k})}{\partial z_{i}^{(l)}}$ . If it is assumed that the network has a total of n_l layers, for the output layer, that is, when l = n_l, $δ_{i}^{(n_{l})}$ can be obtained, as shown in Equation (16). $\begin{matrix} δ_{i}^{(n_{l})} = \frac{\partial J (W, b, x_{k}, y_{k})}{\partial z_{i}^{(n_{l})}} = \frac{1}{2} \frac{\partial {∥ h_{W, b} (x_{i}) - y_{i} ∥}^{2}}{\partial z_{i}^{(n_{l})}} \\ = \frac{1}{2} \frac{\partial \sum j = 1 s_{n_{l}} {(a_{j}^{(n_{l})} - y_{i})}^{2}}{\partial z_{i}^{(n_{l})}} = \frac{1}{2} \frac{\partial \sum j = 1 s_{n_{l}} f {(z_{j}^{(n_{l})} - y_{i})}^{2}}{\partial z_{i}^{(n_{l})}} \\ = \frac{1}{2} \frac{\partial \sum j = 1 s_{n_{l}} f {(z_{j}^{(n_{l})} - y_{i})}^{2}}{\partial z_{i}^{(n_{l})}} = (a_{j}^{(n_{l})} - y_{i}) f^{'} (z_{j}^{(n_{l})}) \end{matrix}$ (16)

For other layers, namely l < n_l, using the chain rule, the following results can be obtained: $\begin{matrix} δ_{i}^{(l)} = \frac{\partial J (W, b, x_{k}, y_{k})}{\partial z_{i}^{(l)}} = \sum_{p = 1}^{S_{l + 1}} \frac{\partial J (W, b, x_{k}, y_{k})}{\partial z_{p}^{(l + 1)}} \frac{\partial z_{p}^{(l + 1)}}{\partial z_{i}^{(l)}} \\ = \sum_{p = 1}^{S_{l + 1}} δ_{p}^{(l + 1)} \frac{\partial z_{p}^{(l + 1)}}{\partial z_{i}^{(l)}} = \sum_{p = 1}^{S_{l + 1}} δ_{p}^{(l + 1)} \frac{\partial z_{p}^{(l + 1)}}{\partial a_{i}^{(l)}} \frac{\partial a_{i}^{(l)}}{\partial z_{i}^{(l)}} = \\ \sum_{p = 1}^{S_{l + 1}} δ_{p}^{(l + 1)} w_{pi}^{(l + 1)} f_{l}^{'} (z_{i}^{(l)}) \end{matrix}$ (17)

Therefore, when l < n_l, $\frac{\partial}{\partial W_{ij}^{(n_{l})}} J (W, b, x_{k}, y_{k}) = a_{j}^{(n_{l} - 1)} (a_{i}^{(n_{l})} - y_{i}) f^{'} (z_{j}^{(n_{l})})$ (18) $\frac{\partial}{\partial b_{i}^{(n_{l})}} J (W, b, x_{k}, y_{k}) = (a_{i}^{(n_{l})} - y_{i}) f^{'} (z_{j}^{(n_{l})})$ (19)

When l < n_l $\frac{\partial}{\partial W_{ij}^{(l)}} J (W, b, x_{k}, y_{k}) = a_{j}^{(l - 1)} \sum_{p = 1}^{S_{l + 1}} δ_{p}^{(l + 1)} w_{pi}^{(l + 1)} f_{l}^{'} (z_{i}^{(l)})$ (20) $\frac{\partial}{\partial n_{i}^{(l)}} J (W, b, x_{k}, y_{k}) = \sum_{p = 1}^{S_{l + 1}} δ_{p}^{(l + 1)} w_{pi}^{(l + 1)} f_{l}^{'} (z_{i}^{(l)})$ (21)

Therefore, according to the formulas (12), (13), the recursive formula of the BP algorithm is obtained, and the parameters are updated through iterative iterations during the neural network training process.

The BPTT algorithm is similar to the BP algorithm and is a time-based error back propagation algorithm. Based on formulas (4)–(7), we use vectors to represent simplified formulas and add bias terms to obtain a simplified RNN forward propagation model as shown in formulas (22)–(25). Among them, f_h is the activation function of the hidden layer, which is generally tanh, and f_o is the activation function of the output layer, which is generally softmax. Since the parameters of the RNN are shared at every moment, the parameters that need to be updated are W, U, V, b, and c. $z^{(l)} = {Wx}^{(l)} + {Uh}^{(t - 1)} + b$ (22) $h^{(t)} = f_{h} (z^{(t)})$ (23) $o^{(t)} = {Vh}^{t} + c$ (24) ${\hat{y}}^{(t)} = f_{o} (o^{(t)})$ (25)

According to the gradient descent method, the iterative update formula for W is shown in (26)-(27). Similarly, the update formula for the parameters U, V, b, and c can be obtained. $W^{'} = W - η \frac{\partial}{\partial W} J (W, U, V, b, c)$ (26) $\frac{\partial}{\partial W} J (W, U, V, b, c) = \frac{1}{m} \sum_{k = 1}^{m} \frac{\partial}{\partial W} J (W, U, V, b, c, x_{k}, y_{k})$ (27)

We set the loss function of each sample of RNN as J (W, U, V, b, c, x_k, y_k). Because RNN is a propagation process with time sequence, the network has losses at every moment. If it is assumed that the sequence has a total of T moments, then $J (W, U, V, b, c, x_{k}, y_{k}) = \sum_{t = 1}^{T} J (t)$ (28)

6 Model building

In this paper, the training steps of the acoustic model of English speech emotion recognition system based on transfer learning are as follows:

M randomly initialized hidden layers and a soft-max layer are added to the first n layers of the mi-grated pre-trained model. According to the obtained English-aligned label data and 2-state FSA topology, two different training strategies are used to train the migrated chain TDNN model. The English feature vector is used as input. The acoustic model structure of English speech emotion recognition system based on transfer learning is shown in Fig. 4:

Fig. 4

Model structure diagram.

The left is the pre-trained model, and the right is the acoustic model based on weight migration. The model includes both a migration layer and a randomly initialized hidden layer. Therefore, according to whether the migration layer and the newly added layer are simultaneously trained, the acoustic model based on weight migration has two different training strategies: single-stage training and two-stage training strategy. The two-stage training strategy includes a fine-tune to the network. The detailed elaboration is as follows: The single-stage training strategy refers to using different learning rates while training the model’s migration layer and the newly added random initialization layer. Generally, the transferred layer uses a smaller learning rate during training, and the newly added layer requires a larger learning rate. Two-stage training requires two complete training stages. In the first stage, the weights of the migration layer are fixed, and the newly added hidden layer with random initialization is trained with a larger learning rate. The second stage uses a smaller learning rate to fine-tune the entire network.

Because the original speech signal contains a lot of noise signals, it will seriously affect the classification effect. In this paper, the wavelet transform function with good directivity and translation invariance is used to denoise the signal, which can effectively smooth the noise and retain the EEG signal characteristics. While denoising, this study uses the “db5” wavelet base to perform 6-layer decomposition to decompose the original EEG signal into Q, 13, and Y bands, and then uses the sparse group lasso-granger algorithm to extract features from each band. Figure 5 shows the waveform comparison of the original EEG signal of a channel of subject 1 before and after denoising. It can be seen that the denoising signal retains the characteristics of the original signal better.

Fig. 5

Comparison of signal graphs before and after denoising.

The database of English speech emotion recognition system mainly includes two parts: English speech emotion recognition and English speech recognition database. Among them, the database is divided into speech library and text library. The voice library is an audio file of the training set and the test set, and the training set and the test set are not repeated. The text library includes the labeled corpus corresponding to the speech library, the training corpus of the language model and the pronunciation dictionary.

An English voice signal contains a lot of information. Feature extraction is to remove useless interference information and extract meaningful information for speech recognition as input to the acoustic model. Speech recognition has the following requirements for feature extraction: higher resolution in time and frequency domain, clearly distinguishing different phonemes, robust to noise and speakers.

The main point of speech production is that the sound we emit is filtered by the shape of the channel, and different shapes can produce different envelopes, namely phonemes. The essence of MFCC is to accurately represent this envelope. The extraction process is as follows: (1) Analog-to-digital conversion: English analog signals are sampled and converted into digital signals that the computer can process. (2) Pre-emphasis: High-frequency components in English digital signals are increased to improve the signal-to-noise ratio. (3) Framing and windowing: A movable fixed-length Hamming window is used to window and frame the pre-emphasized English digital signal to obtain a short-term stable voice signal. In order to make a smooth transition from frame to frame, the frame length is set to 25 ms and the frame shift is set to 10 ms. (4) Fast Fourier Transform: The signal power spectrum is obtained from the spectral information of the English digital signal after windowing. (5) Mel filter: The signal power spectrum of English is filtered and logarithmic. (6) Cepstral Coefficient: Discrete cosine transform is used to obtain the mel frequency cepstrum coefficient of English digital signal, which is the MFCC feature vector of a frame of speech.

Before the emergence of deep neural network acoustic models, acoustic models based on GMM-HMM dominated the field of speech recognition. The HMM model contains a double random process, which expresses the relationship between the speech frames to be recognized through the transition between states and accurately simulates the timing. The acoustic model of English emotion recognition based on GMM-HMM is shown in Fig. 6.

Fig. 6

Acoustic model of English emotion recognition.

7 Model performance analysis

There are some problems to be solved in current speech emotion recognition. First, deep learning methods have become more and more popular due to their powerful feature learning capabilities. Moreover, generally speaking, increasing the network depth will bring a performance improvement to the machine learning task. However, the limited size of the emotion data set greatly limits the complexity of the deep learning model. At present, most of the speech emotion recognition research is to use several layers of network models and the data set needs to be specially designed. At the same time, the models currently making progress in other fields, such as ResNet, cannot be effectively used. Secondly, the current research mainly focuses on the method of cross-sentiment data set. However, due to the difficulty of obtaining emotional tags, this method still has its limitations. In contrast, in other speech fields, labeled data is very rich, and a typical field is the speaker field. Because the collection of speaker tags is less difficult than emotional tags, it has rich labeled data. For example, the commonly used speaker data set VoxCeleb2 is much larger than the current emotion data set. If the speaker data can be used as auxiliary information for emotion recognition, the problem of insufficient emotional data scale can be greatly solved. This paper expects to use speaker labels and deep learning models in the VoxCeleb2 dataset to learn the shared parts of the speaker and emotional information. On this basis, the model test is carried out, and the performance comparison between the English speech emotion recognition model based on CNN neural network and the model proposed in this paper is performed. The model in this paper is named TEOTL.

In the experiment, through 50 sets of speech emotions, the recognition accuracy and speed were compared. The results of the recognition accuracy are shown in Table 1, and the recognition speed is shown in Table 2. The statistical graphs drawn are shown in Figs. 7 and 8, respectively.

Table 1

Statistical table of model recognition accuracy

Serial number	CNN (%)	TEOTL (%)	Serial number	CNN (%)	TEOTL (%)
1	65.0	88.6	26	65.0	89.8
2	66.7	87.2	27	65.6	87.7
3	66.5	88.5	28	69.1	94.2
4	74.7	93.2	29	74.6	83.3
5	68.5	93.4	30	71.7	90.6
6	59.4	86.9	31	72.2	87.7
7	71.2	93.5	32	74.1	93.2
8	64.4	94.7	33	73.1	94.9
9	68.7	87.0	34	74.0	92.9
10	70.0	93.2	35	69.9	83.2
11	62.8	85.7	36	60.9	86.6
12	68.1	86.5	37	63.7	93.4
13	66.6	88.7	38	72.8	93.0
14	72.3	89.8	39	59.9	85.6
15	71.8	88.5	40	60.8	84.1
16	74.7	89.7	41	66.4	88.5
17	68.5	90.1	42	71.6	91.0
18	67.4	92.8	43	70.8	93.8
19	66.6	83.2	44	65.9	86.6
20	59.3	91.3	45	64.3	85.8
21	62.7	85.8	46	65.2	84.0
22	65.5	88.2	47	60.2	86.7
23	67.4	86.0	48	66.7	93.9
24	66.4	83.3	49	72.9	86.1
25	65.6	85.9	50	64.5	88.9

Table 2

Statistical table of model recognition speed

Serial number	CNN (ms)	TEOTL (ms)	Serial number	CNN (ms)	TEOTL (ms)
1	142.3	55.2	26	149.7	64.5
2	141.0	45.3	27	143.1	56.7
3	143.0	44.7	28	131.2	56.0
4	143.0	56.2	29	149.4	62.3
5	137.1	56.2	30	134.3	46.1
6	145.4	63.7	31	131.9	53.0
7	139.0	44.4	32	138.2	60.7
8	148.6	58.1	33	146.4	41.1
9	143.9	61.9	34	138.9	50.3
10	135.3	41.5	35	146.2	55.8
11	133.8	47.1	36	141.1	50.9
12	149.1	52.3	37	139.2	50.9
13	148.9	61.6	38	131.1	52.2
14	143.1	43.2	39	144.9	53.9
15	132.1	52.7	40	145.6	49.4
16	135.8	60.6	41	136.5	59.5
17	142.3	62.1	42	149.9	48.3
18	131.5	40.6	43	140.7	57.4
19	133.2	44.2	44	147.6	59.0
20	133.7	62.7	45	133.7	42.7
21	133.0	62.7	46	142.4	60.9
22	138.6	51.5	47	141.2	50.6
23	144.5	57.3	48	133.6	40.5
24	138.5	45.1	49	136.5	55.0
25	144.8	48.7	50	130.1	56.2

Fig. 7

Statistical diagram of model recognition accuracy.

Fig. 8

Statistical diagram of model recognition speed.

According to the recognition accuracy model of Fig. 7, we can see that the model recognition accuracy curve constructed in this paper is above the CNN recognition accuracy curve, and the gap is close to 20%. It can be seen that the model proposed in this paper has a high accuracy rate, which is close to 90% and much higher than the traditional recognition model. Moreover, this shows that the model proposed in this paper has a certain practical basis, and it can be applied to practice in the future.

Through the recognition speed model of Fig. 8, we can see that the model recognition accuracy curve constructed in this paper is below the CNN recognition accuracy curve, and the gap is close to 100 ms. It can be seen that the model proposed in this paper has a faster recognition speed, which is much higher than the traditional recognition model. Moreover, this shows that the model proposed in this paper has a certain practical basis, and it can be applied to practice in the future.

As can be seen from Figs. 7 and 8, the model proposed in this paper has certain advantages over the traditional model. Moreover, transfer learning has certain advantages over other algorithms in English speech emotion recognition. In the follow-up teaching and real-time translation equipment research, we can consider applying transfer learning to the model.

Conclusion

The economic and cultural exchanges between China and the world are getting closer and closer, so the study of English speech emotion recognition system has very important significance and value. The focus of this paper is to model the acoustics of the English speech emotion recognition system and improve the performance of the acoustic model through transfer learning. M randomly initialized hidden layers and a softmax layer are added to the first n layers of the migrated pre-trained model. According to the obtained English-aligned label data and 2-state FSA topology, two different training strategies are used to train the migrated chain TDNN model. The database of English speech emotion recognition system constructed in this paper mainly includes two parts: English speech emotion recognition and English speech recognition database. Among them, the database is divided into a speech library and a text library, and the speech library is an audio file of the training set and the test set, and the training set and the test set have no duplication. Moreover, the text library includes the labeled corpus corresponding to the speech library, the training corpus of the language model and the pronunciation dictionary. The model proposed in this paper has certain advantages over traditional models, and transfer learning has certain advantages over other algorithms in English speech emotion recognition. In the subsequent teaching and real-time translation equipment research, we may consider applying transfer learning English to the English model.

References

Rhodes , Richard , Aging effects on voice features used in forensic speaker comparison, International Journal of Speech Language & the Law 24(2) (2017), 177–199.

Ngoc Duong

Q.K.

and Duong

HienThanh

, A review of audio features and statistical models exploited for voice pattern design, Computer Science 03(2) (2015), 36–39.

Sarria-Paja

, Senoussaoui

and Falk

T.H.

, The effects of whispered speech on state-of-the-art voice based biometrics systems, Canadian Conference on Electrical and Computer Engineering 2(15) (2015), 1254–1259.

Leeman

, Mixdorff

, O’Reilly

, et al., Speaker-individuality in Fujisaki model f0 features: Implications for forensic voice comparison, International Journal of Speech Language and the Law 21(2) (2015), 343–370.

Hossain

M.S.

and Muhammad

, Healthcare big data voice pathology assessment framework, IEEE Access PP(99) (2016), 1–1.

Hill

A.K.

, Cárdenas

Rodrigo A.

, Wheatley

J.R.

, et al., Are there vocal cues to human developmental stability? Relationships between facial fluctuating asymmetry and voice attractiveness, Evolution & Human Behavior 38(2) (2017), 249–258.

Woźniak

Marcin

and Połap

Dawid

, Voice recognition through the use of Gabor transform and heuristic algorithm, Nephron Clinical Practice 63(2) (2017), 159–164.

Haderlein

, Döllinger

Michael

, Matoušek

Václav

, et al., Objective voice and speech analysis of persons with chronic hoarseness by prosodic analysis of speech samples, Logopedics Phoniatrics Vocology 41(3) (2015), 106–116.

Nidhyananthan

S.S.

, Muthugeetha

and Vallimayil

, Human recognition using voice print in LabVIEW, International Journal of Applied Engineering Research 13(10) (2018), 8126–8130.

10.

Malallah

F.L.

, Saeed

K.N.Y.M.G.

, Abdulameer

S.D.

, et al., Vision-based control by hand-directional gestures converting to voice, International Journal of Scientific & Technology Research 7(7) (2018), 185–190.

11.

Sleeper

Morgan

, Contact effects on voice-onset time in Patagonian Welsh, Acoustical Society of America Journal 140(4) (2016), 3111–3111.

12.

Mohan

, Hamilton

, Grasberger

, A, et al., Realtime voice activity and pitch modulation for laryngectomy transducers using head and facial gestures, Journal of the Acoustical Society of America 137(4) (2015), 2302–2302.

13.

Kang

T.G.

and Kim

N.S.

, DNN-based voice activity detection with multi-task learning, Ieice Transactions on Information & Systems E99.D(2) (2016), 550–553.

14.

Choi

HaNa

, Byun

SungWoo

and Lee

SeokPil

, Discriminative feature vector selection for emotion classification based on speech, Transactions of the Korean Institute of Electrical Engineers 64(9) (2015), 1363–1368.

15.

Herbst

C.T.

, Hertegard

, Zangger-Borch

, et al., Freddie Mercury—acoustic analysis of speaking fundamental frequency, vibrato, and subharmonics, Logopedics Phoniatrics Vocology 42(1) (2016), 1–10.

16.

Al-Tamimi

, Revisiting acoustic correlates of pharyngealization in Jordanian and Moroccan Arabic: Implications for formal representations, Laboratory Phonology 8(1) (2017), 1–40.

17.

Laukka

, Elfenbein

H.A.

, Thingujam

N.S.

, et al., The expression and recognition of emotions in the voice across five nations: a lens model analysis based on acoustic features, Journal of Personality & Social Psychology 111(5) (2016), 686.

18.

Wang

Jie

, Gao

Hui

and Lu

Mao

, Approaches to strategic supplier selection under interval neutrosophic environment, Journal of Intelligent & Fuzzy Systems 37(2) (2019), 1707–1730.

19.

Mousavizadeh

, Maghooli

, Fatemizadeh

, et al., Liveness detection in face identification systems: using zernike moments and fresnel transformation of facial images, Indian Journal of ence and Technology 8(8) (2015), 523.

20.

Orlandi

, Garcia

C.A.R.

, Bandini

, et al., Application of pattern recognition techniques to the classification of full-term and preterm infant cry, Journal of Voice 30(6) (2015), 656–663.

21.

Uma Maheswari

, et al., Bat Optimization Algorithm with Fuzzy Based PIT Sharing (BF-PIT) Algorithm for Named Data Networking (NDN), Journal of Intelligent & Fuzzy Systems 37(1) (2019), 293–300.

22.

Hsu

C.C.

, Cheong

K.M.

, Chi

T.S.

, et al., Robust voice activity detection algorithm based on feature of frequency modulation of harmonics and its DSP implementation, IEICE Transactions on Information and Systems E98.D(10) (2015), 1808–1817.

23.

Kumar

P.H.

and Mohanty

M.N.

, Efficient feature extraction for fear state analysis from human voice, Indian Journal of Science & Technology 9(38) (2016), 1–11.