Innovation of English teaching model based on machine learning neural network and image super resolution

Abstract

At present, applying image recognition technology to promote English teaching is a kind of teaching innovation that meets the needs of the times. Therefore, based on machine learning neural network and image super-resolution, this study conducted an innovative analysis of English teaching mode. This paper combined the current situation of English teaching classroom to study and analyze English classroom, combined classroom characteristics as the basis of English teaching innovation and constructed a feature recognition model suitable for current English teaching status. Moreover, this paper formed an initial high-resolution image for low-resolution image reconstruction by sparse representation method, and then established a mixed sample spine regression model to re-estimate the high-frequency components of the initial high-resolution image to realize various behavioral characteristics of students in English teaching classroom. In addition, this article builds a verification test. The research shows that the proposed algorithm has certain effects and can provide theoretical reference for subsequent related research.

Keywords

Machine learning neural network image super-resolution english teaching innovation

1 Introduction

Strengthening foreign language teaching and improving the level of foreign language teaching and teaching quality have become the top priority in the education reform of all countries, and the cultivation of international talents has become the main national policy of the 21st century. International talent = professional + cross-cultural exchange + foreign language, is the requirement of national development in the era of globalization and informatization [1]. With the continuous deepening of comprehensive reform of higher education, connotation development and quality improvement have become an important task of higher education. The development of higher education has multiple characteristics. On the one hand, it can make the young people who are about to enter the society fully prepared so that they can adapt as soon as they actually enter social work. At the same time, for individuals, it can also make individuals have a long-term application in English, improve employment opportunities, and meet social needs. On the other hand, higher education is to teach the students broader knowledge, follow the temptation, and guide them to open the door to innovation [2].

The English curriculum of colleges and universities has always been a public foundation course. As a teaching method to improve the English level and ability of college students, the importance of English education has become increasingly prominent in college teaching. The globalized economy has transformed China’s pure domestic market into a global market, and China’s foreign capital and well-known multinational companies are also increasing [3]. Therefore, the demand for more English talents in the market is not limited to the language of English itself, but also requires English talents of various professions. This requires colleges and universities to change the way of college students’ English teaching, so that students are more familiar with the study of professional English and strengthen English communication and English application [4]. Therefore, through the improvement of the effectiveness of public English classroom teaching in colleges and universities, it can help college students to establish a strong enthusiasm for English learning, and at the same time, it can promote the level of public English teaching in Chinese universities, thus enhancing the practicality and effectiveness of English.

With the continuous improvement of urban construction at home and abroad, cameras are becoming more and more popular. Moreover, many concepts such as safe city, intelligent transportation, smart home, and intelligent education have been proposed. In this context, video image-based processing technology has also made great progress. The main content of video image processing is to understand and analyze the information in the acquired video image [5]. In the field of video image processing, the detection and recognition of objects in a video scene are mainly accomplished by knowledge of multiple disciplines such as image processing and pattern recognition. The recognition of human behavior is a major research direction in the field of image processing based on video. As more and more scholars invest in it, a large number of algorithms have been proposed and applied to the field of human behavioral motion recognition.ding The main purpose of body behavior recognition is to identify and analyze human behavioral motion information in video sequence images. Generally speaking, human behavior motion recognition is the classification of human action behaviors in video scenes [6]. Based on this, applying image recognition technology to English teaching to promote English teaching is a kind of teaching innovation that meets the needs of the development of the times. Therefore, based on machine learning neural network and image super-resolution, this study conducted an innovative analysis of English teaching mode.

2 Related work

The Visual Surveillance Project (VSAM), led by Carnegie Mellon University and established by the US Defense Advanced Research Projects Agency, which is attended by many American universities and research institutes, is mainly to complete the positioning and segmentation of the human body. On this basis, it can track and detect the human body, which can be used for scene analysis and understanding of battlefield and civil scene video surveillance [7]. The Chinese Academy of Sciences is relatively deep and mature in the study of behavioral motion recognition. They applied the research results of behavioral motion recognition analysis to the daily training of national team diving, gymnastics, trampoline and other projects, and guided and corrected the athlete’s movement through the behavioral motion recognition and analysis system and achieved good results. The main content of their research is the video captured by various video equipment, positioning, tracking, identification, comprehensive analysis of moving targets in the video scene [8]. The main purpose of the research is to conduct intelligent analysis records and warnings of irregular behaviors and abnormal actions for some basic behavioral standards in daily life and work processes, and to take timely safety measures. Behavioral motion recognition based on video images involves three key steps: foreground moving target detection, behavioral action feature extraction, and behavioral action classification recognition. Moving target detection is to detect the position of the moving target from the video sequence image. Behavioral motion feature extraction refers to transforming an image in a video sequence and then acquiring features that can describe the behavioral actions in the video sequence. Behavioral action classification and recognition refers to the process of identifying and classifying behavioral actions based on the image features in the acquired video sequence. Detecting and locating the moving target area from the video sequence image is the first step of behavioral motion recognition. Moreover, the ability of the moving target area to successfully locate and detect the positioning directly affects the performance of the behavioral motion recognition system. At present, the mainstream moving target detection algorithms have the following five: background difference method, frame difference method, code table method, mixed Gaussian model, Vibe method. Moving object detection based on video sequences can be understood as image foreground detection in nature. Du Z et al. used the maximum likelihood estimation to detect and segment moving objects in the image. Stauffer et al. used a background update algorithm based on a mixed Gaussian model to model the pixels in the image. In actual use, the background model was continuously updated to obtain the foreground target [9]. Lotz J M et al. obtained the moving target by means of three frames of video image difference [10]. Mhlolo M K et al. used the inter-frame difference method and the background subtraction method to eliminate the influence of different illumination, thus achieving the adaptation of illumination [11]. With the introduction of more and more moving target detection algorithms, in the actual application scenario, the best algorithm can be selected according to the advantages and disadvantages of each algorithm and the applicable scenarios. The extraction of human behavioral actions in video sequence images is a key step in the process of behavioral motion recognition, and the performance difference of behavioral features will directly affect the effect of final classification recognition. Because of the critical role of this process in behavioral recognition, the core content of human behavioral feature extraction has been indispensable in the research work on behavior recognition in the past four decades. In the human body behavior feature extraction stage, compared with the global characteristics such as the optical flow method, the most widely used and the same effect is the local feature. There are two widely used image local representation methods: space-time interest points and local descriptors. The time-space interesting point detection algorithms commonly used in the direction of behavioral motion recognition are [12]: HarriS3D detector, Cuboid detector, convexity-based detector, Hessian detector, and so on. Unlike space-time points of interest, a local descriptor is a feature that first divides the image into individual blocks and then pairs them. Moreover, it represents the image by means of local feature description, thereby eliminating background interference or having invariance of scales such as translation, rotation and scaling. Caballero D et al. first proposed the space-time interest point STIPs [13]. In their algorithm, the essence of spatio-temporal interest points is to use the Harris3D detector to detect the corners in the image in three-dimensional space, so as to obtain a series of feature points in the image. Homer R et al. proposed the use of Cuboid detectors to detect spatiotemporal points of interest. In the video sequence, it uses the Gaussian filter in the time dimension and the Gaussian filter in the spatial dimension, and then adjusts the local minimum by changing the spatiotemporal scale in the neighborhood of the point of interest, thereby achieving the purpose of adjusting the number of points of interest in the space-time [14]. Forrest J et al. studied the detection based on convexity [15]. Wlemsb et al. propose a method for detecting spatiotemporal points of interest using the Hessian detector, which has scale invariance. The number of feature points acquired based on the detection method of space-time interest points is generally less, so it will affect the representativeness of behavioral actions. For the method of local descriptors [16], Dalai et al. first proposed the HOG descriptor. The method calculates and counts the gradient direction histogram information of the local region of the image, and then combines the features of all local regions to form the characteristics of the whole image [17]. Literature 18 proposes the use of gradient direction histogram (HOG) feature descriptors to represent behavioral actions in an image, and the size of the method features is relatively large [18]. Juneja et al. proposed the HOG3D descriptor by extending the two-dimensional HOG feature into three-dimensional space-time and used the regular polyhedron in space to quantify the orientation of the spatiotemporal gradient [19].

3 Image pyramid based on image block redundancy

Assuming I₀ is a low-resolution input image, different layers of the image gold tower can be constructed according to the following formula (1). $I_{i - 1} = (I_{i} * H_{i}) D_{σ}$ (1)

In the formula, I_i-1 represents the image of the i–1-th layer in the image pyramid, I_i represents the image of the i-th layer, * represents the convolution operation, H_i represents the fuzzy kernel, and D_σ is a down-sampling operation with a scaling factor of σ. As shown in Fig. 1, the image I_i (i = -1, - 2, ⋯ , - a) in the next layer is first created. Where the size of a is determined by the desired magnification, and then the image of the upper layer is constructed using the similarity of the image blocks within the scale and between the scales. In the figure, I₁andI₂ represent high resolution layers in the image pyramid, and I_-1andI_-2 represent low resolution layers. P_s represents a source image block in the low resolution input image, and image blocks P₁ and P₂ similar to P_s can be searched by comparing the Euclidean distances, and the image blocks P₁ and P₂ correspond to the image regions R₁ and R₂, respectively. Therefore, there are two decisive factors in establishing regions D₁ and D₂ in high-resolution layers: (1) The image area where the source image block P_s is located; (2) The layer number of the similar image block found (that is, P₁ corresponds to -1, and P₂ corresponds to -2). The pixel values of the final regions D₁ and D₂ are the pixel values obtained after the copy regions R₁ and R₂ have been enlarged.

Fig. 1

Schematic diagram of the creation of an image pyramid.

4 K-SVD dictionary learning method

4.1 Feature extraction

The image ${I_{i}}_{- a}^{a}$ in the pyramid is taken as a self-sample set, ${E_{j}}_{1}^{b}$ is the outer training image set, and b is the total number of images in the outer training image set. ${Err}_{j} = E_{j} - M_{j}$ (2)

The high frequency detail component ${{Err}_{j}}_{1}^{b}$ for training is acquired. Before the dictionary training, it is also necessary to extract meaningful features from the training set image. In the image super-resolution reconstruction algorithm, f₁ = [1, 0, - 1] , f₂ = [1, 0, - 1] ^T, f₃ = [- 1, 0, 2, 0, - 1] and f₄ = [- 1, 0, 2, 0, - 1] ^T are widely used for feature extraction. We use f₁, f₂, f₃, f₄ to convolve the image ${M_{j}}_{1}^{b}$ to obtain the gradient and Laplacian features of the image, and extract the image blocks from the feature images f₁ * M_j and Err_j respectively to obtain the feature pair ${f_{d}^{L}, f_{d}^{H}}_{1}^{n}$ . We define R∘ to represent the operation of extracting image blocks from the image, then the feature extraction can be expressed by:

$\begin{matrix} f_{d}^{L} = [\begin{matrix} R \circ (f_{1} * M_{j}) \\ R \circ (f_{2} * M_{j}) \\ R \circ (f_{3} * M_{j}) \\ R \circ (f_{4} * M_{j}) \end{matrix}], f_{d}^{H} = [R \circ {Err}_{j}] \\ d = 1, 2, \dots, n \end{matrix}$ (3)

In the formula, n represents the total number of blocks extracted.

4.2 Adaptive sample selection scheme

Considering the robustness of the sample selection, the low-pass filtering operation is performed on the self-sample image ${I_{i}}_{- a}^{a}$ and the external sample image ${E_{j}}_{1}^{b}$ before selecting the sample. Then, the image block is extracted from the filtered self-sample and the outer sample image, respectively, and the outer sample block closest to the self-sample image block is selected by comparing the Euclidean distance. G_l represents a 7 × 7 Gaussian low-pass filter with a standard deviation of 1.6. ${I_{i}}_{- a}^{a}$ and ${E_{j}}_{1}^{b}$ are filtered and the image block is extracted. $\begin{matrix} {p_{c}^{in}}_{1}^{m} = R \circ (G_{l} * I_{i}) \\ {p_{d}^{ex}}_{1}^{n} = R \circ (G_{l} * I_{j}) \end{matrix}$ (4)

In the formula (4), $p_{c}^{in}$ represents an image block extracted from the low-frequency image of the sample, m is the total number of image blocks, $p_{d}^{ex}$ represents an image block extracted from the low-frequency image of the external sample, and n is its total number. By equation (5), the outer sample most similar to the sample is selected. $δ_{c} = arg min_{d} {∥ p_{c}^{in} - p_{d}^{ex} ∥}_{2}$ (5)

In the formula (5), δ_c represents the most similar external sample number corresponding to the sample $p_{c}^{in}$ . In order to effectively solve Equation (5), we use FLANN. In fact, even in the case where the values of m and n are very large, FLANN can effectively solve Equation (5) and the calculation speed is quite fast.

4.3 Dictionary training

F^l represents the set of all $f_{d}^{L}$ , i.e. $F^{l} = {f_{d}^{L}}_{1}^{n}$ , and F^h represents the set of all $f_{d}^{H}$ , i.e. $F^{h} = {f_{d}^{H}}_{1}^{n}$ . From the feature pair ${f_{d}^{L}, f_{d}^{H}}_{1}^{n}$ generated from the external sample, the features corresponding to the number δ_c and the features corresponding to the external samples most similar to sample $p_{c}^{in}$ are taken out to form a pair {F^L, F^H} of training samples. $F^{L} = [F^{l} (δ_{c})]; F^{H} = [F^{h} (δ_{c})] c = 1, 2, \dots, m$ (6)

In the equation (6), F^l (δ_c) denotes a feature numbered δ_c extracted from all $f_{d}^{L}$ , and F^h (δ_c) denotes a feature numbered δ_c extracted from all $f_{d}^{H}$ .

Since F^L is a feature matrix composed of four features, when the selected image block is large, the dimension of F^L is too high. Therefore, it is necessary to use Principal Component Analysis (PCA) to reduce the dimensionality of F^L. The PCA projection matrix can be calculated by: $\begin{matrix} λ_{ϑ} F^{L^{Y}} F^{L} = λ_{ϑ} v_{ϑ} \\ η = arg min_{ϑ} \frac{Sort (λ)}{\sum_{ϑ = 1}^{t} λ_{ϑ}} ⩾ 0.001 \\ λ = [λ_{θ}], ϑ = 1, 2, \dots, ℓ \\ ψ = [v_{η}^{*}, v_{η + 1}^{*}, \dots v_{ℓ}^{*}] \end{matrix}$ (7)

In equation (7), λ_ϑ is the ϑ-th eigenvalue of F^{L
^T}F^L, v_ϑ is its corresponding eigenvector, λ is the vector of all eigenvalues, Sort () is an operation of sorting the vector from small to large, and ψ is the PCA projection matrix. F^L uses PCA projection matrix to reduce dimensionality and train low resolution dictionary.

$\begin{matrix} Φ^{L}, α = arg min_{Φ^{L}, α} {∥ ψ^{T} F^{L} - Φ^{L} α ∥}_{F}^{2} \\ s . t . {∥ α_{ω} ∥}_{0} \leq τ \forall ω \end{matrix}$ (8)

In the formula, Φ^L and α are the low resolution dictionary and the corresponding sparse coefficient, respectively, ∥ ∥ _F represents the Frobenius norm, the vector α_ω represents the ω-th column vector in α, and the constant τ controls the sparsity. The high-resolution dictionary Φ^H is obtained by the following formula (9). $Φ^{H} = arg min_{Φ^{H}} {∥ F^{H} - Φ^{H} α ∥}_{F}^{2}$ (9)

In the formula, g represents a matrix of sparse coefficients obtained by equation (8). The solution of the formula (9) is given by the following formula (10). $Φ^{H} = F^{H} α^{T} {(α α^{T})}^{- 1}$ (10)

Formula (10) is a pseudo inverse representation.

5 Sparse representation and adaptive mixed sample regression

5.1 Sparse representation reconstruction

The low-resolution input image Y is given. We also use the bicubic kernel to interpolate Y with the magnification factor s. Moreover, we use f₁ = [1, 0, - 1] , f₂ = [1, 0, - 1] ^T, f₃ = [- 1, 0, 2, 0, - 1] and f₄ = [- 1, 0, 2, 0, - 1] ^T to filter the interpolated image to obtain its gradient and Laplacian features, and then extract the image blocks from the feature image into a feature matrix. $F^{LR} = [\begin{matrix} R \circ (f_{1} * QY) \\ R \circ (f_{2} * QY) \\ R \circ (f_{3} * QY) \\ R \circ (f_{4} * QY) \end{matrix}]$ (11)

In the formula, Q is a bicubic interpolation operation, R∘ is an image block extraction operation defined in equation (3), and * is a convolution operation defined in Equation (1).

Since the dimension of F^LR is high, we use the PCA projection matrix ψ to reduce the dimension of F^LR, and then use the orthogonal matching pursuit (OMP) algorithm to solve the sparse coefficient on the low-resolution dictionary. $ϕ = arg min_{ϕ} {∥ ψ^{T} F_{LR} - Φ^{L} ϕ ∥}_{F}^{2} s . t . {∥ ϕ_{ρ} ∥}_{0} \leq τ \forall ρ$ (12)

In the formula, ϕ_ρ is the ρ-th column vector of the sparse coefficient matrix ϕ.

According to the sparse representation theory, the initial high resolution image $\hat{X}$ is reconstructed using the obtained sparse coefficient and the trained high resolution dictionary Φ^H. $\hat{X} = QY + β R^{- 1} \circ (Φ^{H} ϕ)$ (13)

In the formula, contrary to R∘, R^-1∘ is the operation of restoring the image block to the image, and β is the weight parameter.

5.2 Ridge regression model for adaptive mixed samples

The initial high-resolution image $\hat{X}$ reconstructed by the sparse representation still lacks detailed information and the edges of the image are not clear enough to achieve the desired reconstruction effect. In fact, although the dictionary learning method in this chapter applies both self-sample and mixed samples, it does not use self-sample for dictionary training, which means that the information in the sample is not fully used for reconstruction. Since Kernel Ridge Regression (KRR) was successfully used to learn the mapping between high and low resolution image blocks and the method based on ridge regression (ANR and A⁺) achieved good reconstruction in the field of super resolution reconstruction, we try to use the ridge regression to establish the mapping relationship between the high and low frequency components of the mixed sample, and use the complementary information of the mixed samples to improve the reconstruction effect of the high frequency components of the initial high resolution image $\hat{X}$ .

High frequency components play an irreplaceable role in super-resolution reconstruction. In order to re-estimate the high-frequency components of the initial high-resolution image $\hat{X}$ , we use a 7 × 7-Gaussian low-pass filter G_l with a standard deviation of 1.6 to filter $\hat{X}$ and obtain its high and low frequency components, respectively. $\begin{matrix} {{\hat{x}}_{o}^{l}}_{1}^{e} = R \circ (G_{l} * \hat{x}) \\ {{\hat{x}}_{o}^{h}}_{1}^{e} = R \circ (\hat{x} - G_{l} * \hat{x}) \end{matrix}$ (14)

In the formula, ${\hat{x}}_{o}^{l}$ is the low-frequency image block sampled from the low-frequency component of $\hat{x}$ , ${\hat{x}}_{o}^{l}$ is its corresponding high-frequency component, and e is the total number of image blocks sampled. According to formula (4), the image block is sampled from the high frequency image of the external sample and the self-sample. $\begin{matrix} {q_{c}^{in}}_{1}^{m} = R \circ (I_{i} - G_{l} * I_{i}) \\ {q_{d}^{ex}}_{1}^{n} = R \circ (E_{j} - G_{l} * E_{j}) \end{matrix}$ (15)

$q_{c}^{in}$ represents an image block extracted from the sample high-frequency image, m is the total number of image blocks, and $q_{d}^{ex}$ represents an image block extracted from the external sample high-frequency image, n represents its total number. The combined high and low frequency sample blocks are formed from the sample high and low frequency block pair ${q_{c}^{in}, q_{c}^{in}}_{1}^{m}$ and the outer sample high and low frequency block pair ${q_{d}^{ex}, q_{d}^{ex}}_{1}^{n}$ . Self-sample and external samples are adaptively selected according to the adaptive sample selection scheme proposed in this paper. $\begin{matrix} δ_{o}^{in} = arg min_{c} {∥ {\hat{x}}_{o}^{l} - p_{c}^{in} ∥}_{2} \\ δ_{o}^{ex} = arg min_{d} {∥ {\hat{x}}_{o}^{l} - p_{d}^{ex} ∥}_{2} \end{matrix}$ (16)

In the formula, $δ_{o}^{in}$ and $δ_{o}^{ex}$ represent the sample numbers, and $δ_{o}^{in}$ and $δ_{o}^{ex}$ represent the vectors formed by the numbers of the k-samples and the outer samples that are most similar to ${\hat{x}}_{o}^{l}$ , respectively. Equation (16) is solved by using the K-nearest neighbor search method and FLANN.

$P^{in} = {p_{c}^{in}}_{1}^{m}$ represents the set of all self-sample low frequency blocks, and $Q^{in} = {q_{c}^{in}}_{1}^{m}$ represents the set of all self-sample high frequency blocks. Similarly, $P^{ex} = {p_{c}^{ex}}_{1}^{n}$ denotes a set of all external sample low frequency blocks, and $Q^{ex} = {q_{c}^{ex}}_{1}^{n}$ denotes a set of all external sample high frequency blocks. The self-samples and external samples most similar to ${\hat{x}}_{o}^{l}$ are taken from Pⁱⁿ, Qⁱⁿ, P^ex, Q^ex, respectively, to form an adaptive mixed sample pair. $ϕ_{o}^{l} = [\begin{matrix} P^{in} (δ_{o}^{in}) \\ P^{ex} (δ_{o}^{ex}) \end{matrix}]; ϕ_{o}^{h} = [\begin{matrix} Q^{in} (δ_{o}^{in}) \\ Q^{ex} (δ_{o}^{ex}) \end{matrix}] o = 1, 2, \dots, e$ (17)

In the formula, $ϕ_{o}^{l}$ denotes the adaptive mixed sample low frequency of ${\hat{x}}_{o}^{l}$ , and $ϕ_{o}^{h}$ denotes the adaptive mixed sample high frequency of ${\hat{x}}_{o}^{l}$ .

In order to accurately describe the relationship between the adaptive hybrid sample low frequency $ϕ_{o}^{h}$ and the adaptive mixed sample high frequency $ϕ_{o}^{l}$ , we attribute the high frequency component re-estimation problem to the least squares regression problem with l₂ norm constraint. Moreover, we use the mixed samples $ϕ_{o}^{l}$ and $ϕ_{o}^{h}$ to solve the coefficients of the ridge regression and use the ridge regression coefficients to complete the high frequency re-estimation. Since the proposed model is based on mixed samples and ridge regression, the model was named Adaptive Mixed Sample Ridge Regression Model (AMSRR). First, the model coefficients are solved. $Λ_{o} = arg min_{Λ_{o}} {∥ {\hat{x}}_{o}^{l} - ϕ_{o}^{l} Λ_{o} ∥}_{2}^{2} + λ {∥ Λ_{o} ∥}_{2}^{2}$ (18)

The solution of the formula (18) is given by the following formula (19). $Λ_{o} = {(ϕ_{o}^{l^{T}} ϕ_{o}^{l} + Λ_{o})}^{- 1} ϕ_{o}^{l^{T}} {\hat{x}}_{o}^{l}$ (19)

In the formula, I is the identity matrix. After that, we use the ridge regression coefficient Λ_o and the mixed sample high frequency $ϕ_{o}^{h}$ to estimate the high frequency components of the initial high resolution image. ${\tilde{X}}^{h} = R^{- 1} \circ (ϕ_{o}^{h} Λ_{o}) o = 1, 2, \dots, e$ (20)

The high-resolution image after high-frequency re-estimation by the AMSRR model can be expressed as: $\tilde{X} = G_{l} * \hat{X} + γ R^{- 1} \circ (ϕ_{o}^{h} Λ_{o}) o = 1, 2, \dots, e$ (21)

In the formula, G_l is an 7 × 7 Gaussian low-pass filter with a standard deviation of 1.6, and γ is the weight coefficient.

5.3 Sparse representation reconstruction

The core idea of non-local mean is to search for a similar image block in a large range of neighborhoods around a target image block and establish a non-local relationship between the target block and the non-local similar image block. This section extends the search area from non-local to the entire image area to form a global mean (GLM). Assuming that ${\tilde{x}}_{o}$ is a partial image block obtained from the estimation of the current high resolution image, FLANN can be used to search for a number of image blocks ${{\tilde{x}}_{o}^{t}}_{t = 1}^{μ}$ similar to ${\tilde{x}}_{o}$ in the entire image, and these similar image blocks are used to predict ${\tilde{x}}_{o}$ . ${\tilde{x}}_{o} = \sum_{t = 1}^{μ} {\tilde{x}}_{o}^{t} g_{o}^{t}$ (22)

In Equation (22), $g_{o}^{t}$ is the weight coefficient, μ is the total number of similar image blocks searched, and ${\tilde{x}}_{o}^{t}$ is the t-th similar image block searched. The weight coefficient $g_{o}^{t}$ is calculated by the following formula (23): $g_{o}^{t} = \frac{1}{u} exp (- {∥ {\tilde{x}}_{o} - {\tilde{x}}_{o}^{t} ∥}_{2}^{2} / w)$ (23)

In the formula, w is the weight control factor. The value of u represents the sum of the weights of ownership, which is calculated by the following equation (24). $u = \sum_{o = 1}^{μ} exp (- {∥ {\tilde{x}}_{o} - {\tilde{x}}_{o}^{t} ∥}_{2}^{2} / w)$ (24)

We assume that g_o denotes a vector containing the weight $g_{o}^{t}$ , and χ_o denotes a vector containing all similar blocks ${\tilde{x}}_{o}^{t}$ . We assume that I represent the identity matrix and define Θ as follows: $Θ (o, t) = {\begin{matrix} g_{o}^{t}, g_{o}^{t} \in g_{o} \\ 0 Others \end{matrix}$ (25)

According to the above definition, the prediction error ${∥ {\tilde{x}}_{o} - g_{o}^{T} χ_{o} ∥}_{2}^{2}$ can be rewritten as ${∥ I - Θ X ∥}_{2}^{2}$ .

In addition, we use the gradient descent rule to strengthen the global reconstruction constraint, project $\tilde{X}$ to the solution space DHX = Y and add global similar regular constraints to form the following optimization problem.

$\begin{matrix} X^{*} = arg min_{X} {{∥ DHX - Y ∥}_{2}^{2} + θ {∥ X - \tilde{X} ∥}_{2}^{2} \\ + ɛ {∥ I - Θ X ∥}_{2}^{2}} \end{matrix}$ (26)

Using the gradient descent method, the solution of equation (26) can be given by iterative equation.

$\begin{matrix} X_{z + 1} = X_{z} + v \\ [\begin{matrix} H^{T} D^{T} (Y - {DHX}_{2}) + θ (X - \tilde{X}) \\ - ɛ {(I - Θ)}^{T} (I - Θ) X_{2} \end{matrix}] \end{matrix}$ (27)

6 System construction

The key technology of this system is to identify the learner’s behavior, that is, by processing, processing and analyzing the data collected by the sensor, the computer system can understand the individual actions. Due to the increasing maturity of somatosensory measurement technology, such as the Kinect somatosensory device introduced by Microsoft, it can measure the distance by infrared sensing device and provide partial bone detection information, Kinect somatosensory equipment was selected for behavior recognition in this paper. After detecting the critical behavior, the system needs to be able to capture the video image of the location of the event for post-tracking and verification. Therefore, a programmable PTZ camera with auto focus is selected for the video image.

Through the above requirements research, the overall design of the system is carried out. The overall requirements of the system are as follows: The learner behavior measurement system is based on a variety of data collected by modern devices such as body sensation instruments in the classroom, and the data is processed, identified and stored for the next step of classroom teaching evaluation statistics and analysis. At the same time, it is necessary to take into account the performance of the equipment and the functions of the application system in the measurement process, so that it is simple and easy to use. The system function diagram is shown in Fig. 2.

Fig. 2

Schematic diagram of system deployment.

According to the overall requirements of the above system, the system can be divided into three different sub-modules: data acquisition module, detection and identification module, and data analysis module. The flow and relationship between functional modules are shown in Fig. 2. The following will introduce the basis of each module division and the key technical points and difficulties of the module.

For the multi-person behavior recognition in the classroom scene, if the somatosensory device is placed directly in front of the subject, there is a problem of occlusion. The scheme of suspending the somatosensory equipment in the classroom ceiling is designed. By hanging the Kinect, measuring the coverage of the Kinect and adjusting the angle of the Kinect by the system, the occlusion problem in the multi-person situation can be effectively solved. The design of the intelligent classroom experimental scenario is based on the following requirements: The testing equipment in the classroom cannot interfere with the teaching process; the Kinect detection range covers all seats; In the case of full people, each seat in Kinect’s field of view does not block each other, and each Kinect field of view does not exceed 6 seats; the PTZ zoom camera can obtain a face with a pixel accuracy of not less than 200×200 within the zoom range. The scene diagram of the design through field measurement and calculation is shown in Fig. 4, and the equipment and layout are shown in Table 1:

Table 1

Classroom equipment layout table

Equipment ID	Equipment name	Use
1	PTZ camera	Monitor the left area of the classroom
2	PTZ camera	Monitor the right area of the classroom
3	Kinect2.0	Detect student status in the left front area of the classroom
4	Kinect2.0	Detect student status in the right front area of the classroom
5	Kinect2.0	Detect student status in the left rear area of the classroom
6	Kinect2.0	Detect student status in the right rear area of the classroom

Fig. 3

Schematic diagram of the function module.

Fig. 4

Plan view of the experimental scene.

Kinect is able to detect the bone joint points of the user within the field of view. In the scenario monitored by the classroom in this paper, it is necessary to be able to distinguish the three sitting postures of the students’ hands raised, sitting down, and squatting.

As shown in Fig. 6, the point where the palm detection accuracy is not high is eliminated, and the upper limb joint point of the human body is simplified and there are nine remaining. These nine points constitute the structure of the upper limbs of the human body and reflect the changes in the posture of the human body. Due to the difference in body height between different people, even with the same hand-raising behavior, different people have the same action upper limb structure vector in the same coordinate system. Therefore, a separate upper limb structure vector cannot be used as a feature vector, and a new auxiliary feature vector needs to be proposed for the gesture determination. By comparing three different poses, it is found that the upper limb structural vectors in different poses exhibit different correlations, especially the vector angle and the modulus ratio relationship.

Fig. 5

Schematic diagram of the skeleton in three dimensions.

Fig. 6

Schematic diagram of the sitting skeleton.

The student participation ranking table obtained on the basis of the above system monitoring is shown in Table 2. The difficulty ranking of the title is shown in Table 3.

Table 2

Student Participation Ranking Table

Student number	Number of raise hand	Number of rises	Participation	Ranking
1	1	0	0.07	6
2	3	1	0.3	2
3	5	1	0.43	1
4	1	1	0.17	5
5	2	1	0.23	4
6	3	2	0.4	2
7	1	0	0.07	6
8	0	0	0	8

Table 3

Title difficulty ranking

Title number	Fastest response time	Answer time	Difficulty value	Ranking
1	8.1	10	0.49	4
2	7.2	11	0.45	6
3	9.8	19	0.52	3
4	23.9	15	0.62	1
5	13.9	5	0.47	5
6	19.9	39	0.59	2

The time-sharing chart of the number of students attending the class and looking up is shown in Fig. 12.

The schematic diagram of the sitting skeleton feature vector and the modulus ratio are shown in Figs. 7 and 8, respectively. In this system, it involves the recognition of the low-head posture. However, by observing and calculating the bone data collected by us, it is found that if the neck joint does not move significantly downward, it is difficult for the system to judge whether it is bowing or sitting. Therefore, the head deflection angle detection function is introduced in this module, and the function depends on the interface provided by Microsoft official as shown in Fig. 9: Yaw (left and right shaking head rotation) detection range is –45°–45°, Roll (left and right deviating head rotation) detection range is –90°–90°, Pitch (bow head rotation) detection range is –45°–45°.

Fig. 7

Schematic diagram of the sitting skeleton feature vector.

Fig. 8

Schematic diagram of the modulus ratio.

Fig. 9

Kinect2.0 head deflection angle detection.

7 Test analysis

This system is a student behavior measurement system based on PC, Kinect2.0 somatosensory equipment, and Hikvision PTZ camera in a small classroom scene. Since each Kinect can only recognize 6 people’s bone information, if you can add more than one Kinect, it is also suitable for large classroom scenes, but the cost is large. The raw data collected in the experiment is transmitted and stored in the server database after being processed and identified by the PC. The actual scene of the deployed experimental scene is shown in Fig. 10.

Fig. 10

Experimental scene diagram.

Two PTZ programmable control zoom cameras are deployed in the classroom and are suspended in front of the classroom and 3 m from the ground, which is used to dynamically obtain detailed color maps. Four Kinect 2.0 somatosensory devices are suspended from the top of the classroom, and Kinect is 2.5 m above the ground. Each Kinect covers a quarter of the classroom and is used to detect student behavior in the area. A global fixed camera is mounted behind the classroom to detect the position of the classroom on the podium.

First, after the student adjusts the position in his seat and is recognized by the kinect, the simulated class answering situation is simulated, as shown in Fig. 11. From Fig. 11, we see a test interface for our student behavior detection system.

Fig. 11

Ready.

In the student lecture test, the raw data shown in Fig. 12 is subjected to statistics of time-divided periods according to the calculation method in the previous design. Then, the statistical results of time-divided periods of the number of people in the entire simulation class who are listening carefully in class are shown in Fig. 12. In Fig. 12, we can see that the number of students attending the lectures and looking up in the 90s–160 s and 210s–260 s periods is low, which is consistent with our experimental design. This experiment also shows that through our system, we can detect some negative states of students in the classroom and provide a method to detect the number of students attending classes. It has certain reference significance for the evaluation of classroom activity.

Fig. 12

Time-based statistics of the number of students attending lectures.

8 Conclusion

By reviewing the literature, this paper summarizes the development status of classroom observation based on video technology, as well as existing research methods and video analysis software. In the process of literature review, it is found that there are already mature coding systems for teachers’ coding classification methods in the classroom at home and abroad, which provides a theoretical basis and method basis for observing students’ classroom behavior. Secondly, by understanding the steps of using video technology to analyze classroom video, it not only has a deep understanding of classroom observation based on video technology, but also finds that it can be more intelligent in the process of identification. Therefore, considering the application of pattern recognition technology, this paper combines video technology and pattern recognition technology, which can simplify the process of data collection and analysis, and can greatly improve the accuracy of analysis. In addition, a classroom behavior recognition system integrating device control, data acquisition, recognition, recording, analysis and display is realized. At the same time, this paper uses the system to conduct individual experiments for learners and experiments on classroom activity evaluation and gives three experimental indicators for each learner’s personal evaluation, difficulty evaluation of each question, and overall classroom activity evaluation. Moreover, the conclusions with certain significance are obtained through experiments, which have certain reference significance for classroom teaching evaluation.

References

Islam

M.S.

, Mahmud

, Papeya

A.A.

, et al., A Combined Feature Extraction Method for Automated Face Recognition in Classroom Environment[C]// International Symposium on Signal Processing and Intelligent Recognition Systems. Springer, Cham, 2017.

Yong

, Ke’An

and Bingrui

, Feature extraction for sound source material recognition with impact sounds[J], Chinese Journal of Acoustics2017(02), 25–37.

Bin

and Zhenhua

, Interacting Feature Recognition Based on AAG Expansion and Hint[J], Journal of Graphics38(4), 2017.

Cui

and Ma

, Progress and Survey of Mobile Image Feature Recognition[J], Recent Patents on Computer Science09(999) (2017), 1.

, Wen

, Luo

, et al., A New QoS-Aware Web Service Recommendation System based on Contextual Feature Recognition at Server-Side[J], IEEE Transactions on Network and Service Management, (2017), 1.

Chengfeng

, Miao

, Keyi

, et al., An improved NBA-based STEP design intention feature recognition[J], Future Generation Computer Systems88 (2018), 357–362.

and Zhao

, High-quality point sampling for B-spline fitting of parametric curves with feature recognition[J], Journal of Computational and Applied Mathematics (2018), S0377042718301766.

Venu

, Komma

V.R.

and Srivastava

, STEP-based Feature Recognition System for B-spline Surface Features[J], International Journal of Automation and Computing15(04) (2018), 126–138.

, Chen

, Zhang

, et al., Learning Collaborative Sparsity Structure via Nonconvex Optimization for Feature Recognition[J], IEEE Transactions on Industrial Informatics (2017), 1.

10.

Lotz

J.M.

, Hoffmann

, Lotz

, et al., Integration of 3D multimodal imaging data of a head and neck cancer and advanced feature recognition[J], Biochimica et Biophysica Acta (BBA) - Proteins and Proteomics, (2016), S1570963916301807.

11.

Mhlolo

M.K.

, Regular classroom teachers’ recognition and support of the creative potential of mildly gifted mathematics learners[J], ZDM49(1) (2017), 81–94.

12.

Ishino

and Okada

, Constructing students’ deontic status by use of alternative recognitionals for student reference[J], Classroom Discourse (2017), 1–17.

13.

Caballero

, Araya

, Kronholm

, et al., ASR in Classroom Today: Automatic Visualization of Conceptual Network in Science Classrooms[J]. 2017.

14.

Homer

, Hew

K.F.

and Tan

C.Y.

, Comparing Digital Badges-and-Points with Classroom Token Systems: Effects on Elementary School ESL Students’ Classroom Behavior and English Learning[J], Educational Technology & Society (2018), 21.

15.

Forrest

, Lean

and Dunn

, Attitudes of Classroom Teachers to Cultural Diversity and Multicultural Education in Country New South Wales, Australia[J], Australian Journal of Teacher Education42(5) (2017), 17–34.

16.

Rudasill

K.M.

, Hawley

L.R.

, Crouch

L.C.

, et al., Child temperamental regulation and classroom quality in Head Start: Considering the role of cumulative economic risk[J], Journal of Educational Psychology109(1), 2017.

17.

Cinzia

, Lara

, Marta

, et al., Development of a Dog-Assisted Activity Program in an Elementary Classroom[J], Veterinary Sciences4(4) (2017), 62.

18.

Altemueller

and Lindquist

, Flipped classroom instruction for inclusive learning[J], British Journal of Special Education44(3) (2017), 341–358.

19.

Juneja

, Multiple feature descriptors based model for individual identification in group photos[J], Journal of King Saud University - Computer and Information Sciences (2017), S131915781730054X.