Multimedia English teaching analysis based on deep learning speech enhancement algorithm and robust expression positioning

3.1 Background subtraction

Background Subtraction (BS) is a widely used moving target detection method. By pre-establishing the background model, it uses the training image to obtain the parameters of the model and uses the current image and the background image to reduce the brightness of the same position pixel to perform foreground target detection [13]. A frame background image is first selected or constructed, the background image is updated according to a certain background model, and then the difference image between the current image and the background image is calculated, and the difference image is thresholded to obtain a foreground target. The specific algorithm flow is shown in Fig. 1:

OPEN IN VIEWER

Fig. 1

Flow chart of the Background Subtraction.

In the figure, f _k  (x, y) is the current frame, B (x, y) is the background image, I (x, y) is the grayscale image after the background difference, and R (x, y) is the image after binarization and connected domain detection. I ( x , y ) = | f k ( x , y ) - B ( x , y ) |(1)R ( x , y ) = { 0 I ( x , y ) < T 1 I ( x , y ) ≥ T(2)

The moving target is extracted by the difference between the current frame and the background image, the binarization processing, and the connected domain detection.

However, changes in the scene and light will cause the background to change every moment. Therefore, it is not advisable to select a frame with no moving target as the background from the video, and an adaptive background update model needs to be established. A more common approach is to use the image averaging model and achieve a weighted average of the image by assigning different weights to the current frame and the previously accumulated background. The specific method is as follows: B k + 1 ( x , y ) = ( 1 - α ) B k ( x , y ) + α C k ( x , y )(3)

In the formula, B _k  (x, y) is the pixel value of the point (x, y) in the current background, C _k  (x, y) is the pixel value of the point (x, y) in the current frame, and B_k+1 (x, y) is the pixel value of the point (x, y) after the update. α is the update coefficient. The larger α is, the faster the background update is. The smaller α is, the slower the background update is. Therefore, its size should be set according to the actual situation.

The background subtraction algorithm is simple and easy to understand and has good real-time performance and can extract moving targets more completely. However, in the scene with large background changes, the detection performance is poor, and the pseudo-target is often generated. Therefore, further optimization is needed on the background modeling and updating methods [14].

The interframe difference method refers to performing differential operation between consecutive image frames (between two frames or three frames), and subtracting the pixels corresponding to different frames to determine whether the absolute value of the gradation difference exceeds the threshold value to extract the moving target in the sequence image. The algorithm flow is shown in Fig. 2:

OPEN IN VIEWER

Fig. 2

Flow chart of the interframe difference method.

In the figure, f _k  (x, y) is the k-th frame, f_k-1 (x, y) is the k–1-th frame, I (x, y) is the grayscale image after the difference, and R (x, y) is the binarized image. I ( x , y ) = | f k ( x , y ) - f k - 1 ( x , y ) |(4)

The variation region of the moving object in the video image is detected by the difference calculation between two adjacent frames. For static targets, such as a stationary background behind the target, I (x, y) is zero. For a moving target in the scene, I (x, y) is not zero.

In order to better display the foreground target of the motion and remove the influence of noise on the motion foreground, the differential image will be binarized and connected domain detected. R ( x , y ) = { 0 I ( x , y ) < T 1 I ( x , y ) ≥ T(5)

In the formula, T is the threshold for binarization.

On the basis of the interframe difference method, the adjacent three frames of images are differentiated as a group to detect the moving target. The flowchart of the specific algorithm is shown in Fig. 3:

OPEN IN VIEWER

Fig. 3

Flow chart of the interframe difference method (three frames).

In the formula, f _k  (x, y) , f_k-1 (x, y) , f_k-2 (x, y) is a continuous three-frame image, and I₁ (x, y) , I₂ (x, y) is a differential image calculated by two adjacent frames. The specific algorithm process is as follows:

I 1 ( x , y ) = | f k ( x , y ) - f k - 1 ( x , y ) | I 2 ( x , y ) = | f k - 1 ( x , y ) - f k - 2 ( x , y ) |(6)

The difference image I₁ (x, y) , I₂ (x, y) is obtained by the difference between the two adjacent frames of the video frame. Then, binarization processing and connected domain detection are performed on the calculated difference image. R 1 ( x , y ) = { 0 I 1 ( x , y ) < T 1 I 1 ( x , y ) ≥ T R 2 ( x , y ) = { 0 I 2 ( x , y ) < T 1 I 2 ( x , y ) ≥ T(7)

The two binarized images calculated for each pixel point are ANDed to obtain the detected foreground target image. R ( x , y ) = { 0 R 1 ( x , y ) ∩ R 2 ( x , y ) ≠ 1 1 R 1 ( x , y ) ∩ R 2 ( x , y ) = 1(8)

Whether it is the interframe difference method or the improved three-frame difference method, the advantage is that the algorithm is simple, the operation speed is fast, and the light can be adapted to a certain extent to a certain extent, and the adaptability to the dynamic environment is good, and the influence of the target shadow is small. However, the shortcoming of this method is that the moving target area cannot be completely extracted, the positioning is not very accurate, and it is too dependent on the moving speed of the target [15]. When the target moving speed is large, voids and ghosts often appear.

The optical flow can be thought of as the instantaneous velocity field produced by the movement of pixels with grayscale on the image plane. Horn and Schunck assume that the image region is continuous and steerable both spatially and temporally, which is considered an important constraint in the calculation of optical flow. For an image sequence, we assume that the luminance value of a pixel (x, y) in the image at time t is I (x, y, t). u (x, y) and v (x, y) represent the motion components of the optical flow at the point (x, y) in the x and y directions, respectively. In a sufficiently small time dt, point (x, y) moves to point (x + dx, y + dy), dx = udt, dy = vdt. According to the assumption that the brightness is constant, that is, the corresponding pixel points in each frame along a certain motion trajectory curve have the same gray value, that is, the brightness of the corresponding point on the image is unchanged, we can obtain: I ( x + dx , y + dy , t + dt ) = I ( x , y , t )(9)

The above formula is developed by the first-order Taylor formula to obtain the following formula: I ( x , y , t ) + ∂ I ∂ x dx + ∂ I ∂ y dy + ∂ I ∂ t dt + O ( ∂ 2 ) = I ( x , y , t )(10)

In the formula, O (∂²) is a higher order trace of dx, dy, dt second or third order, which can be ignored. Taking into account the speed relationship, dx = ∂ x ∂ t dt = udt , dy = ∂ y ∂ t dt = vdt , the two sides are eliminated, so that the following formula can be obtained: I x u + I y v + I t = 0(11)∇ I · V + I t = 0(12)

In the formula, I _x , I _y , I _z represents the gray value of the point. The partial derivatives of I (x, y, t) in the x, y, t direction can be obtained from the image sequence. V = (u, v) is the velocity field, and ∇I = [I _x , I _y ]  ^T is the spatial grayscale gradient of the image pixel point (x, y). Equation (12) is the optical flow constraint equation, or the basic equation of optical flow, which represents the relationship between the grayscale spatial gradient of the image and the velocity of the optical flow. In the optical flow constraint equation, u and v are both unknowns and only one constraint equation, so the optical flow cannot be uniquely determined. Additional constraints are needed to determine u and v.

The basic idea of the introduced constraints is that the optical flow should be as smooth as possible and the smoothing constraint E _s is minimized:

E s = ∫ ∫ [ ( ∂ u ∂ x ) 2 + ( ∂ u ∂ y ) 2 + ( ∂ v ∂ x ) 2 + ( ∂ v ∂ y ) 2 ] 2 dxdy(13)

According to the basic equation I _x u + I _y v + I _t  = 0 of optical flow, the optical flow error E _c is minimized to obtain: E c = ∫ ∫ ( I x u + I y v + I t ) 2 dxdy(14)

The solution to the optical flow field can then be translated into a solution to the following problem:

E = min ∫ ∫ { ( I x u + I y v + I t ) 2 + [ [ ( ∂ u ∂ x ) 2 + ( ∂ u ∂ y ) 2 + ( ∂ v ∂ x ) 2 + ( ∂ v ∂ y ) 2 ] 2 ] } dxdy(15)

The minimization equation is solved, and the optical flow vector [16] is obtained by calculating the adjacent gray image of two frames.

For the n × n region adjacent to the pixel, if the motion trend of the pixels is assumed to be the same, the following equations can be established: [ I x 1 I y 1 I x 2 I y 2 I x 3 I y 3 · · · · · · · · ] [ u v = - [ I 1 I 2 I 3 · · · · ] ](16)

The above equations can be expressed as a vector form: u → = ( A T A ) - 1 A T b →(17)

By the least squares method, the equation can be solved and the direction and size of the optical flow vector can be obtained. However, when the above-mentioned n × n window contains more than two edges, the L-K optical flow method can ensure that the matrix is an invertible matrix, thereby ensuring that the optical flow defining equation has a solution. Therefore, the calculation process of the L-K optical flow method needs to combine the corner information and the edge information, and the optical flow vector obtained in this way is the sparse optical flow vector [17].

4.1 Forward neural network

As a computational model, the forward neural network is composed of multiple levels of neurons. Each level consists of multiple neurons connected. The structure of the forward neural network is shown in Fig. 4 [18].

OPEN IN VIEWER

Fig. 4

Single neuron structure.

The neuron has three inputs (x₁, x₂, x₃) and one output h_ω,b (x), respectively, and the computational relationship is: h ω , b ( x ) = f ( W T x ) = f ( ∑ i = 1 n ω i x i + b )(18)

In the formula, x represents the input vector, W is the weight vector of the input, b is the offset, and function f : R → R is the activation function.

The structure of the forward neural network is shown in Fig. 5. The network model can be determined by the number of specific network layers and the number of neurons in each layer. The learning process of the entire neural network is based on constantly adjusting the weights and bias coefficients of the neurons. Generally, a forward neural network has an input layer for receiving data to be processed, and then followed by multiple hidden layers or no hidden layer, receiving the output of the upper layer, after calculation by multiple neurons, the last layer Is the output layer [19].

OPEN IN VIEWER

Fig. 5

Structure of the forward neural network.

The Convolutional Layer is also a feature extraction layer. It convolves the data of the upper layer by a plurality of convolution kernels of different sizes, thereby obtaining different two-dimensional outputs. The calculation formula is as follows: y mn = f ( ∑ j = 0 J = 1 ∑ i = 0 I = 1 x m + i , n + j ω ij + b ) , ( 0 ≤ m < M , 0 ≤ n < N )(19)

In the formula, x is a two-dimensional input vector, ω _ij is a convolution kernel of size i × j, b is an offset, y _mn is the output of size m × n, and function f is the activation function. The function of the convolutional layer is to convolve the output of the upper layer by multiple convolutional kernels to obtain a plurality of new two-dimensional outputs. The two-dimensional output calculated by convolution is called a feature map.

This idea of connecting local information is also inspired by the structure of the visual system in biology. The neurons of the visual cortex receive information locally, that is, these neurons respond only to the stimulation of certain regions. The local perceptual characteristics of the convolutional neural network are shown in Fig. 6:

OPEN IN VIEWER

Fig. 6

Full connection mode (top) and local connection mode (bottom).

While reducing the amount of calculation, the robustness of the network output to displacement and deformation is also improved. The formula for the pooling operation in a certain area is: y mn = f ( ω 1 S 1 S 2 ∑ j = 0 S 2 - 1 ∑ i = 0 S 1 - 1 x m × S 1 + i , n × S 2 + j + b ) , ( 0 ≤ m < M , 0 ≤ n < N )(20)

In the formula, x_{m×S1+i,n×S2+j} is the pixel value of the input two-dimensional image at m × S₁ + i, n × S₂ + j, and y _mn is the output obtained after pooling. The pooling layer can reduce the amount of computation of the network, and makes the network have certain invariance to the translation and scaling of the object, so that the network is robust. The pooling layer is actually down-sampling on different channels. The down-sampling process of different channels is independent of each other. The process of pooling operation and convolution kernel are similar to sliding on the image. However, the pixels that are drawn by the pooling layer do not overlap each other, and the step size can be determined to determine the size of the feature map after the pooling operation. As shown in the right half of Fig. 6, the four pixels in each S₁ × S₂ region are combined into one pixel by calculating the average value, then added by the weight ω and the offset coefficient b, and finally the two-dimensional feature map is obtained by the excitation function f. Therefore, the pooling layer also plays the role of extracting features again.

Support Vector Machines (SVM) is a two-class classification model whose basic model is the linear classifier with the largest interval in the feature space. We assume that the hyperplane is ω · x + b = 0, and |ω · x + b| can relatively represent the distance of point x from the hyperplane. In general, the distance between a point and the separation hyperplane can indicate the degree of confidence in the classification prediction. For a hyperplane (ω, b) of a given training data set T, the function interval for the sample point (x _i , y _i ) is defined as: γ ˆ = y i ( ω · x + b )(21)

The function interval can indicate the correctness and certainty of the classification prediction. However, when choosing to separate hyperplanes, the equal scaling of ω and b does not change the hyperplane, but the function interval is changed by a multiple, so the unique solution cannot be found and is not suitable as the objective function of the classifier. Therefore, the normal vector ω of the hyperplane needs to be normalized, so the geometric interval is introduced: γ i = y i ( ω ∥ ω ∥ · x i + b ∥ ω ∥ )(22)

In the formula, ∥ω∥ is the L₂-norm of ω. The maximum interval separation hyperplane can be expressed as max γ ˆ . Finding the hyperplane with the largest geometric spacing for the training data set means that the training data has been classified using a sufficiently large confidence. For multi-classification problems, the correct result of SVM classification is at least a fixed margin larger than the score of the misclassified sample. The traditional SVM classifier is only for the classification task of the two classifications, and the multi-class SVM is implemented by stacking multiple two-class SVMs, and the training complexity is very high. Therefore, the Softmax classifier was proposed.

The Softmax classifier generalizes the loss function of logistic regression to multi-classification problems. When there is a task of class k, and the training set has a sample of m numbers, and the input is an n-dimensional vector, the training set can be expressed as: T = { ( x ( 1 ) , y ( 1 ) ) , ⋯ , ( x ( m ) , y ( m ) ) }(23)

In the formula, y⁽ⁱ⁾∈ { 1, 2, ⋯ , k } is the label and x⁽ⁱ⁾ ∈ Rⁿ⁺¹ is the sample. The Softmax classifier will calculate the probability that each sample belongs to k classes: P ( y = j | x ) , ( j = 1 , ⋯ k )(24)

This forms a k-dimensional output for each sample, and the form of the calculation function is as follows:

f ( x ( i ) | θ ) = [ P ( y ( i ) = 1 | x ( i ) , θ ) P ( y ( i ) = 2 | x ( i ) , θ ) ⋯ P ( y ( i ) = k | x ( i ) , θ ) ] = 1 ∑ j = 2 k e θ j T x ( i ) [ e θ 1 T x ( i ) e θ 2 T x ( i ) ⋯ e θ k T x ( i ) ](25)

In the formula, θ = [ θ 1 T θ 2 T ⋯ θ k T ] is the model parameter.

The goal of training the Softmax classifier with the training set T is to find the appropriate parameters to minimize the loss function of the classifier. The commonly used loss function is as follows: J ( θ ) = - 1 m [ ∑ i = 1 m ∑ j = 1 k 1 { y ( i ) = j } log e θ j T x ( i ) ∑ I = 1 k e θ I T x ( i ) ](26)

The smaller the value of the loss function, the more accurate the result of classifying the training set by Softmax.