Performance analysis of distance teaching classroom based on machine learning and virtual reality

Abstract

In remote intelligent teaching, the facial expression features can be recorded in time through facial recognition, which is convenient for teachers to judge the learning status of students in time and helps teachers to change teaching strategies in a timely manner. Based on this, this study applies machine learning and virtual reality technology to distance classroom teaching. Moreover, this study uses different channels to automatically learn global and local features related to facial expression recognition tasks. In addition, this study integrates the soft attention mechanism into the proposed model so that the model automatically learns the feature maps that are more important for facial expression recognition and the salient regions within the feature maps. At the same time, this study performs weighted fusion on the features extracted from different branches, and uses the fused features to re-recognize student features. Finally, this study analyzes the results of this paper through control experiments. The research results show that the algorithm proposed in this paper has good performance and can be applied to the distance teaching system.

Keywords

Machine learning virtual reality distance teaching classroom performance

1 Introduction

The main purpose of behavior recognition is to analyze the types of behaviors performed by people in the video, which involves knowledge and technology in many fields such as computer vision, image processing, pattern recognition, and feature engineering. In the recognition process, the algorithm not only needs to locate and track the behavior from the image sequence, but also needs to analyze the state and trend of the movement, and infer the specific behavior category of the person based on this. Compared with images, video has a larger amount of information. Before the era of deep learning, behavior recognition was a difficult subject. In traditional machine learning, because the image or video feature extraction process is manually designed and the video has a high information content, the progress of behavior recognition research is slow. With the advent of the era of deep learning and the substantial improvement of computing power, behavior recognition has been paid attention by a large number of researchers and has achieved good results. Some objective factors of human behavior are also important reasons that hinder the development of behavior recognition, such as: individual behavior execution differences: even the same behavior, different people will produce a variety of differences when performing; change of shooting angle: different shooting angles will output different observation data, and the same motion will also produce large differences; as well as camera shake, lighting changes, and complex background. With the progress of society and the rapid development of economy, the state and society are more and more concerned about public safety, so a large number of monitoring systems are deployed in highways, schools, shopping malls and other places [1]. A large number of monitoring equipment will inevitably generate a large amount of video data, which will occupy a lot of storage resources. At the same time, these monitoring devices only provide a storage function and cannot perform intelligent analysis, so it takes a lot of manpower and material resources to perform video analysis when needed. Secondly, if an abnormal event occurs in the monitoring scene, it cannot be reported in real time, which results in the failure to take effective measures in time and delays a lot of time. Therefore, building an effective behavior recognition system plays a vital role in the field of public safety. In addition, behavior recognition has high practical value in the field of virtual reality [2]. Virtual reality is the use of a computer to generate a virtual three-dimensional world, and by collecting human sensory information such as human movement, body language, and sound in real time, it allows users to be on the scene and observe the created three-dimensional world without restrictions. Virtual reality technology has also been applied to medical, military, and educational aspects, helping these fields achieve good results. In addition, behavior recognition is also widely used in human-computer interaction, video retrieval and other fields [3].

2 Related work

Behavior recognition is one of the hot research topics in the field of computer vision, and its research topic “behavior” has multiple definitions. The literature [4] believed that Action is usually a simple action mode performed by one person and lasts for a short time (around tens of seconds), while Activity is composed of a complex sequence of actions performed by several people and several people interact with each other in a specific way. For example, running or swimming belongs to human actions, and two people shaking hands or two teams playing football matches belongs to human activities. The literature [5] defined the term “action primitive” as atomic motion that can be described at the limb level. Therefore, behavior consists of a series of behavior primitives, and a series of behaviors constitute more complex activities. For example, running behavior consists of a series of behavior primitives such as raising arms and legs, and running, kicking and other behaviors can constitute football activities. The literature [6] believed that the key to behavior is that it will bring changes or transforms to the surrounding environment. The definition in the literature [7] is more reasonable, but it needs to be clear that human behavior has a relatively clear start and end time, and can be classified as an accurate category. Multiple continuous behaviors constitute human activities, and the number of people is not the main basis for distinguishing between behaviors and activities. The main challenge of behavior recognition is that even the same behavior category has a large difference in appearance and posture. In the field of machine learning, feature extraction is an important link, so it becomes particularly important to convert behavior-containing videos into feature vectors. Behavior recognition feature extraction includes two types: global features and local features. Global features are extracted from the entire process of human behavior, which provides rich and discriminative motion information. In the whole process, background modeling needs to be carried out first, and the foreground objects of interest in each frame are extracted for feature coding to obtain global characteristics of motion behavior. The literature [8] proposed two methods of motion energy image (MEI) and motion history image (Mill), and the purpose is to encode the complete dynamic human behavior into a single image. The motion energy image is a binary image template that describes where motion occurs. The motion history image shows the time at which the motion occurred, and each pixel is related to the motion history at that location, and a larger value indicates that the motion has occurred more recently. In addition to extracting motion information from image frames, optical flow is also proposed as complementary information for image frames. Optical flow information is extracted from consecutive image frames, and it refers to calculating the movements in the horizontal and vertical directions under the premise of unchanged lighting conditions [9]. Local features only focus on the partial regions where significant motion occurs, describe the local motions that occur in the significant regions, and finally use the features in the significant regions as feature vectors. The classic methods include the recognition method based on spatiotemporal interest points [10] and the method based on motion trajectory proposed in literature [11].

In recent years, with the improvement of computer computing power and the driving of big data, deep learning methods have been widely used in computer vision, natural language processing and other fields, and researchers have gradually adopted deep learning theory to solve behavior recognition tasks. Since Convolutional Neural Network (CNN) has made important breakthroughs on ImageNet dataset, CNN-based behavior recognition method is proposed [12]. The literature [13] proposed the concept of a three-dimensional convolution kernel, which is used to simultaneously model temporal and spatial information in video. The literature [14] tried a variety of fusion methods based on the CNN model to fuse image features into video behavior features. The literature [15] proposed a dual-stream network model to separately learn the spatial and temporal information in the video and then fuse the classification probabilities of the two networks to obtain the final classification result. In addition, some top journals in the field of artificial intelligence and pattern recognition such as TPAMI, IJCV, PRL, etc. have opened behavior recognition columns. Moreover, relevant international conferences such as CVPR, ICCV, ECCV, etc. provide relevant researchers with the opportunity to display and communicate cutting-edge algorithms. In order to solve practical problems in industry, many behavior recognition related competitions have emerged, such as TRECVID event retrieval competition [16], THUMOS behavior recognition competition [17], YouTube. 8M video classification competition [18], etc.

The expression feature extraction algorithms based on static images can be divided into two categories: global feature extraction methods and local feature extraction methods. In terms of global feature extraction, the commonly used feature extraction algorithms include Principal Component Analysis (PCA), Active Appearance Model (AAM) and Independent Component Analysis (ICA). The literature [19] applied the PCA algorithm to the facial expression recognition task and verified the feasibility of PCA in the field of facial expression recognition through experiments. The literature [20] used principal component analysis algorithm to reduce the dimensionality of the original image feature matrix and used Linear Discriminant Analysis (LDA) projection matrix to classify the dimensionality-reduced features to recognize facial expressions.

3 Mixed attention subnetwork

At present, researchers have tried to use different methods to reduce the impact of cluttered background information on the effect of pedestrian re-recognition [21]. Moreover, the use of image semantic segmentation technology to obtain the pedestrian’s subject area is also a method to reduce the negative impact of cluttered background. For pedestrian re-identification, binary mask information has two advantages:

The mask can remove the cluttered background at the pixel level. In this way, the pedestrian re-identification model is more robust against different backgrounds [22].

The mask contains the shape information of the human body, which is a very important feature. According to previous studies, the human body mask is very robust to different lighting and clothing colors, which is conducive to identifying pedestrians. With the rapid development of deep learning, image semantic segmentation technology has become relatively mature, such as full convolutional neural network (FCN), Mask-RCNN and so on. Moreover, in terms of data, there are already large-scale human semantic segmentation data sets. Therefore, through these semantic segmentation techniques, an ideal body mask can be obtained, and the background can be accurately removed from the pedestrian image, as shown in Fig. 1 [23 –25].

Fig. 1

Pedestrian image and corresponding mask image.

(2) Contrast spatial attention

Usually, the spatial attention model takes the intermediate feature map of the network as input and outputs the weight map. According to this weight map, the spatial attention operation is performed on the feature map. Through such operations, the network can notice the regions on the feature map that contribute the most to the model training effect.

The input sample RGM-M is given, and the feature map f_stage-2 after two stages in the hybrid contrast attention network is obtained. f_stage-2 is used as the input of the spatial attention sub-network to obtain the human attention map. This process can be expressed as: $Φ^{+} = σ (W * f_{stage - 2} + b)$ (1)

Among them, σ (x) = 1/(1 + exp(- x)) is the sigmoid function, and W and b are the weight and deviation of the convolution kernel, respectively. Then, a contrasting attention graph Φ^- is generated and it pays attention to the opposite features. In order to ensure that Φ⁺ and Φ^- are a pair of contrasting attention maps, for each position (i, j), this attention map needs to satisfy the following constraints:

$Φ^{+} (i, j) + Φ^{-} (i, j) = 1$ (2)

These two attention maps are used separately on feature map f_stage-2 to produce a pair of contrasting feature maps:

$f_{att}^{+} = f_{stage - 2} \otimes Φ^{+}$ (3)

$f_{att}^{-} = f_{stage - 2} \otimes Φ^{-}$ (4)

Among them, ⊗ means element multiplication. For $f_{att}^{+}$ , it should obtain a higher score in the human body area, while for $f_{att}^{-}$ , it should obtain a lower score. However, without any constraints, these two attentions may learn the same weight, and there is no guarantee that $f_{att}^{+}$ can learn human features. In order to encourage Φ⁺ to pay attention to human body information, the binary body mask information is introduced as a label for human attention Φ⁺ to add a clear constraint to it. Moreover, the square loss is used between Φ⁺ and its corresponding body mask: $L_{att} = \sum_{i = 1}^{I} \sum_{j = 1}^{J} {∥ M_{(i, j)} - Φ_{(i, j)}^{+} ∥}_{2}^{2}$ (5)

Among them, M is a body mask generated in advance from a pedestrian image by an appropriate image semantic segmentation method, which is adjusted to the same size as the attention map. In this way, the spatial attention sub-network can generate a pair of contrast feature maps related to the human body and the background, respectively.

(3) Contrast channel attention

There are some problems when using contrast spatial attention alone: Since the mask information is a single channel and the features of the hidden layer of the convolutional neural network are usually multi-channel, it is necessary to copy the learned spatial attention map to make it into multi-channel data. Since the feature map of the middle layer of the network has passed through several neural network layers, the relationship between certain spatial features and the weights of different channels has been learned. When using such single-channel attention ignoring the channel relationship and the feature map of the middle layer of the network to perform spatial attention operations, although the feature map obtains better spatial feature weights, the learned channel weights are lost to a certain extent.

This section uses a simple example to illustrate the effect of single-channel spatial attention on channel weights. The average of a channel’s feature map is usually used to represent the weight of the channel, such as global average pooling. This article assumes a feature map with 2 channels and g size. The separate feature maps of the two channels are 1, 0, 2, 0 and 1, 0, 0, 0, and the weight ratio of the two channels is 3 to 1. Now, we assume that the spatial attention of a single channel is 1, 0, 0, 0, and after the spatial attention is element-multiplied with the feature maps of the two channels, the feature maps of the two channels become 1, 0, 0, 0 and 1, 0, 0, 0. At this time, the channel weight ratio is 1 : 1. In this case, the spatial attention of the single channel damages the channel weights that the network has learned. Therefore, this paper uses both channel attention and spatial attention in the design of the network. Moreover, channel attention can not only eliminate the influence of spatial attention on the channel weights of feature maps, but also enable feature maps to learn better channel weights.

Since spatial attention is a pair of contrasting attention, this article also designs it as a pair of contrasting attention in the design of channel attention. The mixed attention sub-network of the entire process of channel attention is compared. After two MSCANs, an intermediate feature map f_stage-2 of size 96 × 16 × 40 is obtained. After that, global average pooling is performed to obtain 96-dimensional feature vectors, namely: $z_{c} = \frac{1}{W \times H} \sum_{i = 1}^{W} \sum_{j = 1}^{H} pix (i, j)$ (6)

Among them, W and H respectively represent the length and width of the pixel level of the feature map, and pix (i, j) represents the feature value at position (i, j).

After that, z_c successively passes through two fully connected layers with 96 neurons to obtain 96-dimensional characteristic sales volume f_{z
_c}. Finally, f_{z
_c} gets the positive channel attention Ψ⁺ through the sigmoid activation function: $Ψ^{+} = σ (f_{z_{c}})$ (7)

Among them, σ (x) = 1/(1 + exp(- x)) is the sigmoid function. $Ψ^{+} (i) + Ψ^{-} (j) = 1$ (8)

The attention of the pair of contrast channels is adjusted to 96 × 16 × 40, and the adjusted channel contrast attention is used on the intermediate feature map f_stage-2: $f_{c - att}^{+} = f_{stage - 2} \otimes Ψ^{+}$ (9)

$f_{c - att}^{-} = f_{stage - 2} \otimes Ψ^{-}$ (10)

Among them, ⊗ means element multiplication. Since ⊗ is a multiplication of elements, both $f_{c - att}^{+}$ and $f_{c - att}^{-}$ are 96 × 16 × 40 in size.

(4) Mixed contrast attention

The comparison space attention feature map f_att and the comparison channel attention feature map f_c-att are obtained, and the subsequent work is to fuse the two. Since both spatial attention and channel attention are processed by the sigmoid function and both are attention operations performed on the feature map f_stage-2, the two are equivalent in magnitude. Therefore, we can directly use element addition to fuse two attention feature maps: $f_{F - att}^{+} = f_{att}^{+} \oplus f_{c - att}^{+}$ (11)

$f_{F - att}^{-} = f_{att}^{-} \oplus f_{c - att}^{- +}$ (12)

Among them, ⊕ means adding elements.

In the model, the overall branch remains unchanged and the intermediate feature f_stage-2 is still used. However, the human branch and the background branch use mixed contrast attention feature maps $f_{F - att}^{+}$ and $f_{F - att}^{-}$ . Meanwhile, The inputs f_full, $f_{F - att}^{+}$ and $f_{F - att}^{-}$ of the three branches are fed into the two MSCANs afterwards to generate the final 128-dimensional feature vectors h_full h_body and h_bkgd, and the ternary loss is applied to the three branches. Moreover, h_full and h_body are pulled closer and h_full and h_bkgd are pushed away. The last three feature vectors are sent to the last fully connected layer, and feature vectors h_full-idh_body-id and h_bkgd-id are generated for identity loss operations.

4 Objective function and algorithm flow

This section introduces the objective function of the model and the entire algorithm flow of training. After the attention operation, the feature maps $f_{full} f_{F - att}^{+}$ and $f_{F - att}^{-}$ of the three branches in the model are obtained. Then, these three feature maps are respectively fed into the two subsequent MSCANs to generate the final feature vectors h_fullh_body and h_bkgd. The purpose of this paper is to reduce the impact of the cluttered background on the effect of pedestrian re-recognition. Therefore, h_full is used as the anchor sample, h_body is used as the positive sample, and h_bkgd is used as the negative sample. At the same time, the overall branch feature vector h_full and the human branch feature vector h_body are pushed close, and the overall branch feature vector h_full and the background feature vector h_bkgd are pushed away. In order to achieve this purpose, the regional level ternary loss function is used, which can be expressed as: $L_{trip} = {∥ h_{full} - h_{body} ∥}_{2}^{2} + max {(m - {∥ h_{full} - h_{bkgd} ∥}_{2}^{2}), 0}$ (13)

Among them, m is the boundary parameter. By minimizing this loss function, in the feature space, the features from the global branch and the human branch will be drawn closer, but the features of the global branch and the background branch will be pushed away. Finally, the characteristics of the overall branch will be more concentrated on the human body area than the cluttered background, which can improve the effect of pedestrian re-identification.

While using the region-level ternary loss function for the three branches, a softmax operation is performed on the last layer of the three branches to predict the identity information of pedestrians. Furthermore, the loss function is denoted as L_id using the cross-entropy loss function. The overall loss function of FCANet is: $L_{FCANet} = L_{id} + α \cdot L_{trip} + β \cdot L_{att}$ (14)

Among them, α and β are hyperparameters.

After training FCANet, it is used as the basic model of the twin network to continue training to reduce the distance between features generated by the same pedestrian and increase the distance between features generated by different pedestrians. Among them, the weights of the two networks are shared. After the RGB-M inputs of pedestrians p and g are given and the pedestrian feature vectors generated by the overall branch are recorded as h (p) and h (g), respectively, the loss function of the twin network is defined as: $L_{sia} = {\begin{matrix} {∥ h (p) - h (g) ∥}_{2}^{2}, p = g \\ max {m - {∥ h (p) - h (g) ∥}_{2}^{2}, 0}, p \neq g \end{matrix}$ (15)

Among them, m is the boundary parameter. According to previous work, in order to improve the effect of pedestrian re-identification, the contrast loss function and the identity loss function are used simultaneously. Subsequently, FCANet’s regional-level loss function is added. At this time, the overall loss function becomes: $\begin{matrix} L_{all} = L_{id (p, g)} + λ \cdot L_{sia (p, g)} + \\ α \cdot L_{trip (p, g)} + β \cdot L_{att (p, g)} \end{matrix}$ (16)

Among them, λ is a hyperparameter.

5 Model training

(1) Parameter design

The expansion ratios of the three parallel expansion convolutions of each MSCAN are set to 1, 2, and 3, respectively, and the number of convolution kernels is 32. In the last two fully connected layers, the number of neurons in the first fully connected layer is 128, and the number of neurons in the second is the number of pedestrians in the training set. Moreover, the L2 regularization parameter is set to 0.005.

The three parameters in the loss function: λ α β are set to 0.01, 0.01, and 0.1, respectively. The boundary value m is set to 100. The initial learning rate of the mixed contrast attention network is set to 0.01. During the training process, it iterated about 75,000 times, and the learning rate is reduced every 15,000 times. In the training process of the twin network, the initial learning rate is set to 0.0001, and the learning rate is reduced every iteration 15000 times. During the training process of the twin network, the initial learning rate is set to 0.0001.

(2) Training process

The input data is preprocessed. For three-channel images: For the value pix (i, j) of each pixel (i, j) of each channel in the image, subtract it from the mean and divide by 255: ${pix}_{i, j}^{\land} = \frac{pix (i, j) - E [pix]}{255}$ (17)

For the mask image, the grayscale image is read, and then the same normalization operation as the three-channel image is performed. After that, the pre-processed three-channel image and mask image are spliced by channel as the input of the network.

First, FCANet is trained, and the training is stopped after 75,000 iterations. After that, training of the twin network was started, and the trained FCANet is used as the basic model of the twin network to continuously train the twin network until it converged. The feature vector f_full generated by the overall branch is used for pedestrian re-recognition. The entire process is shown in Fig. 2.

(3) Feature distance measurement method

Fig. 2

Twin network.

After the model training is completed, the pedestrian images in the probe set and the gallery set are sent to the model to obtain feature vectors. The distance or similarity between the probe image and the gallery image feature according to the feature vector is calculated, the gallery images are sorted according to the similarity, and the sorted gallery image and probe image are matched respectively. In this paper, the feature distance measurement method is Re-rank.

First, the distance between image feature vectors is calculated by using Euclidean distance, and the images in the gallery are sorted for different probe images. Re-rank is to reorder the sorted gallery image sequence $L (p, G) = {g_{1}^{0}, g_{2}^{0}, \dots, g_{N}^{0}}$ by recalculating the similarity between each image and the probe image. Training process diagram as show in Fig. 3.

Fig. 3

Training process diagram.

The top k samples with the highest similarity to the probe image p in the original sorted sequence $L (p, G) = {g_{1}^{0}, g_{2}^{0}, \dots, g_{N}^{0}}$ of the image are defined as: $N (p, k) = {g_{1}^{0}, g_{2}^{0}, \dots, g_{k}^{0}}$ (18)

N (p, k) sample is: |N (p, k) | = k. According to N (p, k), the interrelation set of probe image p and gallery image is defined as: $R (p, k) = {g_{i} | g_{i} \in N (p, k) \land (p \in N (g_{i}, k))}$ (19)

According to the above definition, it can be seen that the connection between the gallery image and the probe image in R (p, k) is larger than that in N (p, k).

However, there is a problem with this definition. Due to changes in lighting, pedestrian poses, and viewing angles, the positive samples in the gallery dataset may belong to N (p, k) but not R (p, k). In order to solve this problem, the robustness of R (p, k) is increased and extended to R * (p, k): $\begin{matrix} R * (p, k) \leftarrow R (p, k) \cup R (q, \frac{1}{2} k) \\ s . t . | R (p, k) \cup R (q, \frac{1}{2} k) | ⩾ \frac{2}{3} | R (q, \frac{1}{2} k) | \\ \forall q \in R (p, k) \end{matrix}$ (20)

After expansion, there are more positive samples in R * (p, k), and these newly added samples are more similar to the probe image in comparison with the original samples in R (p, k).

The coincidence degree of the mutual relationship set of the two images represents the similarity of the two images to a certain extent, that is, the more coincidence, the higher the similarity of the two images. Therefore, a new measure can be defined based on the relationship between the two images-the Jaccard distance: $d_{j} (p, q_{i}) = 1 - \frac{| R * (p, k) \cap R * (g_{i}, k) |}{| R * (p, k) \cap R * (g_{i}, k) |}$ (21)

According to the interaction set of the image, the feature vector of the image is re-encoded: $V_{p} = [V_{p, g_{1}}, V_{p, g_{2}}, \dots, V_{p, g_{N}}]$ (22)

Among them, $V_{p, g_{i}} = {\begin{matrix} 1, g_{i} \in R * (p, k) \\ 0, g_{i} \notin R * (p, k) \end{matrix}$ (23)

According to the encoding method of V_p, it can be seen that each sample in R * (p, k) has the same weight. However, defining weights based on the similarity to the image should result in better ranking results, that is, samples with higher similarity should obtain greater weights. Therefore, the Gaussian kernel improvement formula (22) according to a pair of sample distances is: $V_{p, g_{i}} = {\begin{matrix} e^{- d (p, g_{i})}, g_{i} \in R * (p, k) \\ 0, g_{i} \notin R * (p, k) \end{matrix}$ (24)

Among them, d (p, g_i) represents the distance between the image p and g_i measured by using the original measurement method, and Euclidean distance is used in this paper. Through this improvement, the hard 0 and 1 weights are improved to soft weights. In soft weights, the closer to the probe image, the higher the weight.

After the coding method of V_p is improved, the calculation method of Jaccard distance will also change accordingly. In Jaccard distance: $| R^{*} (p, k) \cap R^{*} (g_{i}, k) | = {∥ min (V_{p}, V_{g_{i}}) ∥}_{1}$ (25) $| R^{*} (p, k) \cap R^{*} (g_{i}, k) | = {∥ max (V_{p}, V_{g_{i}}) ∥}_{1}$ (26)

Among them, the minimization and maximization operations are performed on the elements in the vector, and ∥ · ∥ ₁ represents L₁ normalization. The final calculation method of Jaccard distance is: $d_{J} (p, g_{i}) = 1 - \frac{\sum_{j = 1}^{N} min (V_{p, g_{j}}, V_{g_{i}, g_{j}})}{\sum_{j = 1}^{N} max (V_{p, g_{j}}, V_{g_{i}, g_{j}})}$ (27)

Since the images of the same pedestrian will have similar characteristics, the final coding features of the extended image are: $V_{p} = \frac{1}{| N (p, k) |} \sum_{g_{i} \in N (p, k)} V_{g_{i}}$ (28)

Considering that the original distance measurement method also has a certain effect, the Jaccard distance and the original distance are mixed as the final measurement distance: $d * (p, g_{i}) = (1 - λ) d_{J} (p, g_{i}) + λ d (p, g_{i})$ (29)

Among them λ ∈ [0, 1].

6 Model performance analysis

This research model is subjected to performance analysis, and this research model is named ML-VR.

First, from the characteristics of the data set, it can be seen that the expression intensity of the image sequence in the CK+dataset is gradually increased from the natural expression to the strongest expression intensity, and the expression intensity of the image sequence in the Oulu-CASIA dataset is also gradually changed from the natural state to the strongest expression intensity. Therefore, based on the ensemble method of regression trees method, the three frames with the strongest expression strength are detected by Landmarks, and finally the face is aligned and normalized to 48×48 pixels. Then, ML-VR is used to extract facial expression features from the pictures. When the model proposed in this paper is used for learning, deep features with better discriminability and robustness can be obtained. Among them, the balance parameters λ and λ1 are set to 0.008, α1 and α2 are set to 0.1 and 1.0, respectively, and ξ is 2.0. During the test, only the ML-VR model was used to extract 512-dimensional depth features. Finally, the extracted features are input to the SoftMax classifier for classification. Because the number of facial expressions included in the CK+dataset and the Oulu-CASIA dataset is different, the SoftMax classifier is used in the CK+dataset to divide the extracted features one of the seven categories: angry, disgusting, happy, sad, surprised, afraid, and despising. However, the features extracted in the Oulu-CASIA dataset are divided into one of six categories: angry, disgusted, happy, sad, surprised, and afraid. In order to verify the effectiveness of the soft attention mechanism and the regularized center loss respectively, four sets of comparative experiments are carried out.

(1) The experiment only uses SoftMax classification loss as the objective function to optimize the multi-scale convolutional neural network without fused soft attention mechanism-MSCNN+Ls (2) The experiment uses only SoftMax classification loss as the objective function to optimize the multi-scale attention convolutional neural network with soft attention mechanism-ML-VR + Ls (3) The experiment uses the center loss and SoftMax classification loss as the objective function to optimize the multi-scale convolutional neural network with soft attention mechanism-ML-VR + Ls + Lc(4) The experiment uses regularized center loss and SoftMax classification loss as the objective function to optimize the multi-scale attention convolutional neural network with soft attention mechanism-ML-VR + Ls + LRC. First, the confusion matrix of the model is analyzed, and the results are shown in Tables 1, 2, Figs. 4 and 5:

Table 1
ML-VR recognition rate confusion matrix in CK+

AN CO DI FE HA SA SU

AN 1.1 0 0 0 0 0 0

CO 0 1.1 0 0 0 0 0

DI 0 0.04 1.04 0.02 0 0 0

FE 0 0 0 1.1 0 0 0

HA 0 0 0 0 1.1 0 0

SA 0 0 0 0 0 1.1 0

SU 0 0 0 0.0407 0 0 1.06

	AN	CO	DI	FE	HA	SA	SU
AN	1.1	0	0	0	0	0	0
CO	0	1.1	0	0	0	0	0
DI	0	0.04	1.04	0.02	0	0	0
FE	0	0	0	1.1	0	0	0
HA	0	0	0	0	1.1	0	0
SA	0	0	0	0	0	1.1	0
SU	0	0	0	0.0407	0	0	1.06

Table 2

ML-VR recognition rate confusion matrix in Oulu-CASIA

	AN	DI	FE	HA	SA	SU
AN	0.77	0.22	0.08	0.03	0	0
DI	0	0.77	0.33	0	0	0
FE	0	0	0.91	0.19	0	0
HA	0.04	0	0	1.06	0	0
SA	0.22	0.19	0	0	0.69	0
SU	0	0	0	0	0	1.1

Fig. 4

The statistical diagram of ML-VR recognition rate in CK+.

Fig. 5

The statistical diagram of ML-VR recognition rate in Oulu-CASIA.

From the confusion matrix in the table above, it can be seen that the three expressions of despising, disgusting and fear may have recognition errors. The reason is that when the objects in the database present these three expressions, the mouth area or the eye area may change similarly. In addition, people will open their mouths unconsciously when they are afraid and happy, so these two expressions will also be confused. Finally, the three expressions of sadness, anger and disgust will also make a judgment error. The reason for this is that the areas of mouth, eyes and eyebrows of the three expressions are similar and have similar changes. Compared with other comparative experiments, the multi-scale attention convolutional neural network model incorporating the soft attention mechanism has achieved relatively excellent recognition results in the test data set in the CK+database. In particular, its recognition accuracy for expressions such as anger, contempt, fear, happiness, sadness is high. However, the facial expression recognition algorithm proposed in this paper in the Oulu-CAISA database has relatively good recognition effect on expressions such as disgust, fear, sadness, happiness and surprise.

Finally, this research method is applied to the distance teaching classroom, and the research method is compared with the CNN model. In the model recognition, a total of 60 sets of experimental recognition are performed, and the results of the recognition accuracy are shown in Table 3 and Fig. 6.

Table 3

Statistical table of classroom performance identification accuracy (%)

	CNN	ML-VR		CNN	ML-VR
1	56.1	86.0	31	56.8	86.4
2	54.4	92.2	32	57.6	87.6
3	58.9	89.9	33	54.6	86.6
4	61.7	89.8	34	61.3	84.0
5	56.8	87.2	35	51.0	85.0
6	57.9	86.8	36	64.9	88.8
7	58.5	90.0	37	63.4	86.0
8	55.7	86.5	38	60.1	89.9
9	53.2	89.3	39	62.6	88.0
10	61.1	88.5	40	53.8	89.3
11	63.3	84.5	41	65.5	87.8
12	61.7	89.3	42	63.4	92.3
13	61.7	92.7	43	63.9	89.6
14	59.2	90.1	44	53.3	85.7
15	61.1	87.9	45	59.9	84.6
16	61.8	88.7	46	56.6	92.5
17	63.8	89.9	47	62.4	87.3
18	57.1	84.1	48	52.3	84.6
19	58.4	85.2	49	59.8	92.4
20	64.0	87.4	50	55.8	88.1
21	61.4	88.5	51	54.2	84.9
22	57.5	89.6	52	60.2	92.4
23	63.6	92.5	53	56.4	87.5
24	53.2	86.0	54	58.5	84.0
25	59.9	85.5	55	58.3	84.8
26	51.8	91.8	56	56.4	90.5
27	64.9	89.4	57	55.4	84.0
28	55.7	89.4	58	55.9	92.9
29	58.8	84.9	59	55.6	85.2

Fig. 6

Statistical diagram of model classroom performance identification accuracy.

It can be seen from Fig. 6 that the accuracy of this research model in class performance identification is close to 90%, and compared with the CNN model, this research model has a greater advantage.

7 Conclusion

The traditional re-recognition algorithm for student performance in the classroom requires manual design and extraction of features in the image. Moreover, the effect of student re-recognition depends heavily on the design and extraction of features, and algorithms often fail to achieve satisfactory results. As deep learning has achieved tremendous results in image classification, deep learning has been introduced into classroom re-recognition, and the effect of student feature re-recognition has been significantly improved. Since spatial attention is one-to-one contrasting attention, in the design of channel attention, this article also designs it as one-to-one contrasting attention. The mixed attention sub-network of the entire process of channel attention is compared. The mixed contrast attention in this paper can divide the image into two parts at the pixel level according to the importance of the features, and the human body branch and the background branch extract features from these two image parts, respectively. Therefore, in order to make full use of image features, this paper performs weighted fusion of the features extracted from different branches and uses the fused features to re-recognize student features. In addition, this study analyzes the results of this article through a controlled experiment. The research results show that the performance of the algorithm in this paper is good and in line with expectations.

References

Ding

and Tao

, Robust Face Recognition via Multimodal Deep Face Representation, IEEE Transactions on Multimedia 17(11) (2015), 2049–2058.

Hossain

M.S.

and Muhammad

, Cloud-Assisted Speech and Face Recognition Framework for Health Monitoring, Mobile Networks & Applications 20(3) (2015), 391–399.

Anjos

, Chakka

M.M.

and Marcel

, Motion-based counter-measures to photo attacks in face recognition, Biometrics, IET 3(3) (2014), 147–158.

Zhuang

, Chan

T.H.

, Yang

A.Y.

, et al., Single-Sample Face Recognition with Image Corruption and Misalignment via Sparse Illumination Transfer, Proceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 114(2-3) (2014), 272–287.

Wang

, Shi

, Shu

, et al., Embedded Manifold-Based Kernel Fisher Discriminant Analysis for Face Recognition, Neural processing letters 43(1) (2016), 1–16.

Chuk

, Chan

A.B.

and Hsiao

J.H.

, Understanding eye movements in face recognition using hidden Markov models, Journal of Vision 14(11) (2014), 8–8.

Bouchech

Hamdi

, Selection of optimal narrowband multispectral images for face recognition, Monthly Notices of the Royal Astronomical Society 402(4) (2015), 2140–2186.

Ramachandra

and Busch

, Presentation Attack Detection Methods for Face Recognition Systems: A Comprehensive Survey, ACM Computing Surveys 50(1) (2017), 8.1–8.37.

Huang

, Shan

, Wang

, et al., A Benchmark and Comparative Study of Video-Based Face Recognition on COX Face Database, IEEE Transactions on Image Processing 24(12) (2015), 5967–5981.

10.

Wei

C.P.

and Wang

Y.C.F.

, Undersampled Face Recognition via Robust Auxiliary Dictionary Learning, Image Processing IEEE Transactions on 24(6) (2015), 1722–1734.

11.

Ajit Krisshna

N.L.

, Deepak

V.K.

, Manikantan

, et al., Face recognition using transform domain feature extraction and PSO-based feature selection, Applied Soft Computing 22(Complete) (2014), 141–161.

12.

Weng

, Lu

and Tan

Y.P.

, Robust Point Set Matching for Partial Face Recognition, Image Processing IEEE Transactions on 25(3) (2016), 1163–1176.

13.

Cui

, Chang

, Shan

, et al., Joint sparse representation for video-based face recognition, Neurocomputing 135 (2014), 306–312.

14.

Phillips

P.J.

and Otoole

A.J.

, Comparison of human and computer performance across face recognition experiments, Image and Vision Computing 32(1) (2014), 74–85.

15.

Tang

, Zhu

, Yu

, et al., A novel sparse representation method based on virtual samples for face recognition, Neural Computing & Applications 24(3-4) (2014), 513–519.

16.

Mehta

, Yuan

and Egiazarian

, Face recognition using scale-adaptive directional and textural features, Pattern Recognition 47(5) (2014), 1846–1858.

17.

Raghavendra

, Raja

K.B.

and Busch

, Presentation Attack Detection for Face Recognition Using Light Field Camera, IEEE Transactions on Image Processing 24(3) (2015), 1060–1075.

18.

Yang

, Zhu

, Liu

, et al., Joint representation and pattern learning for robust face recognition, Neurocomputing 168(nov.30) (2015), 70–80.

19.

Yan

, Wang

and Suter

, Multi-subregion based correlation filter bank for robust face recognition, Pattern Recognition 47(11) (2014), 3487–3501.

20.

Meng

Zhenyu

, et al., A Parameter Adaptive Differential Evolution Based on Depth Information, Journal of Intelligent & Fuzzy Systems 38(5) (2020), 5661–5671.

21.

Fan

, Ni

, Zhu

, et al., Weighted sparse representation for face recognition, Neurocomputing 151(mar.3pt.1) (2015), 304–309.

22.

Vyshnav

M.T.

, et al., Random Fourier Feature Based Music-speech Classification, Journal of Intelligent & Fuzzy Systems 38(5), 6353–6363.

23.

Tang

, Feng

and Cai

, Weighted group sparse representation for undersampled face recognition, Neurocomputing 145 (2014), 402–415.

24.

Al-Arashi

W.H.

, Ibrahim

and Suandi

S.A.

, Optimizing principal component analysis performance for face recognition using genetic algorithm, Neurocomputing 128(mar.27) (2014), 415–420.

25.

Morelli Andrés

, Padovani

, Tepper

, et al., Face recognition on partially occluded images using compressed sensing, Pattern Recognition Letters 36(1) (2014), 235–242.

Performance analysis of distance teaching classroom based on machine learning and virtual reality

Abstract

Keywords

1 Introduction

2 Related work

3 Mixed attention subnetwork

Table 1 ML-VR recognition rate confusion matrix in CK+ AN CO DI FE HA SA SU AN 1.1 0 0 0 0 0 0 CO 0 1.1 0 0 0 0 0 DI 0 0.04 1.04 0.02 0 0 0 FE 0 0 0 1.1 0 0 0 HA 0 0 0 0 1.1 0 0 SA 0 0 0 0 0 1.1 0 SU 0 0 0 0.0407 0 0 1.06

References

Table 1
ML-VR recognition rate confusion matrix in CK+

AN CO DI FE HA SA SU

AN 1.1 0 0 0 0 0 0

CO 0 1.1 0 0 0 0 0

DI 0 0.04 1.04 0.02 0 0 0

FE 0 0 0 1.1 0 0 0

HA 0 0 0 0 1.1 0 0

SA 0 0 0 0 0 1.1 0

SU 0 0 0 0.0407 0 0 1.06