Short video emotion transmission based on 3D convolutional neural network and multi-head attention mechanism

Abstract

Generally, emotions in short videos are transmitted through characters behaviour and text content in the video. The mainstream recognition model is based on a convolutional neural network. However, its processing ability for video data is limited, its recognition accuracy is not high enough, and it is limited in processing complex text. To solve these problems, a new text recognition emotion classification model is designed by improving the text feature fusion module on the basis of the convolutional neural network model. Moreover, a new human behaviour recognition emotion classification model is designed by introducing multi-head attention mechanisms on the basis of a 3D convolutional neural network model. These results confirmed that the accuracy of the improved text recognition model was around 75%, while the original convolutional model's average accuracy was only about 67%. The 3D convolutional model under the multi-head attention mechanism had the highest recognition accuracy, with a recognition accuracy of 94.5% on the UCF101 database, which was 12 to 39 percentage points higher than the model under other attention mechanisms. The improved convolutional network model for short video text classification and behaviour recognition has more advantages than traditional models. These research results have certain value for classification models and can serve as technical references.

Keywords

three-dimensional convolutional neural network multi-head self-attention emotional communication behavior recognition text classification

1 Introduction

Computer vision technology has rapidly developed, and short videos have gradually become an indispensable part of people's lives. These make short video sentiment recognition and classification based on short video data analysis a current research hotspot. Usually, emotions in short videos are disseminated through the behaviour status of the characters in the video and the text content in the video.¹ As a method using convolutionl operations, the convolutional neural network (CNN) is applied widely in image recognition, object detection, face recognition, natural language processing, and other fields.² In terms of image recognition, CNN can classify, segment, and detect image information. In terms of text processing, CNN can recognize, associate, and classify text content.³ However, traditional CNN also has problems. When dealing with behavioural feature recognition, the use of an average sampling strategy to convert video data into image frames leads to information loss, resulting in insufficient recognition accuracy. Additionally, it is limited in processing complex text, making it difficult to process lengthy and complex text due to grammar constraints.⁴ In this context, this study attempts to make improvements based on traditional CNN, attempting to improve the feature fusion module in order to obtain a new text recognition and classification system. The study attempts to combine multi-head self-attention (MSA) with 3D CNN to obtain a new behaviour recognition classification model.

This study consists of four parts. Firstly, the current research status of convolutional models in text and behaviour recognition by experts was reviewed. Secondly, the methods to be used in this study were introduced. Then, based on the research method in the second part, experiments were conducted and the results were analyzed. Finally, a summary of the research results was provided.

2 Related works

With computer technology developing and people's increasing demand for information, the method of human behaviour analysis in videos has gradually shifted from manual analysis to deep machine learning analysis of algorithms. Dual flow networks are commonly used algorithms in the field of behaviour analysis. Tang J et al. believed that fusion methods were crucial for the performance of dual flow networks. Therefore, a new time fusion structure was proposed to fuse the information of dual flow joints for feature recognition and prediction of human motion. The structure included a time connection block and a reinforcement trajectory spatiotemporal block, which could increase the time continuity between the first predicted pose and the given pose, as well as between each predicted pose. The short-term and long-term predictions of H3.6 M, CMU-Mocap, and 3DPW benchmark datasets achieved better results than the improved model.⁵ For improving the feature fusion structure of the dual stream algorithm, A X Z et al. proposed an end-to-end refined dual stream fast region CNN algorithm. Based on the fast region CNN algorithm, RGB image features and depth image features were first extracted separately. Then, regions of interest were generated by mapping two types of image features. Subsequently, feature extraction and fusion were carried out using a feature fusion layer. The average recognition accuracy of this model was around 95%. Moreover, compared to relevant methods, its average accuracy was higher.⁶ Peng C et al. proposed a three-stream model using two different types of deep CNNs, using two optical flow estimation methods: an Epicflow (Edge preserving interpolation correspondence for optical flow) method that emphasizes motion boundaries, and a learning optical flow estimation method. Accurate optical flow was crucial in human motion recognition. If some measures of global behaviour of motion were included, generalization performance typically was improved by 1% −2%.⁷

Sentence semantic matching, as a core task of Natural-language understanding technology, widely exists in various interaction scenarios. Lu W et al. designed a novel human-machine language interaction method based on a 3D CNN module. Given a pair of sentences, the representations were connected together and fed into another 3D CNN module to capture the interaction features between them, thereby generating the final matching representation. Numerous experiments conducted on two real datasets showed that this model could achieve performance comparable to or even better than state-of-the-art competitive methods.⁸ Wang X et al. believed that the purpose of sentiment analysis was to identify the words’ emotional polarity. While existing research has predominantly employed CNN, issues such as truncation backpropagation and others frequently arise during training. Therefore, Wang X et al. put forward a model combining multiple attention mechanism (AM) for analyzing hierarchical sentiment. This experiment utilized multi-layer AM, including intra layer AM and inter layer AM, to get sentences’ hidden meanings. In inter layer AM, global attention was used to capture the interaction information between context and aspect words. Compared with the baseline model, this new method could provide optimal outcomes.⁹ Gan et al. posited that the inherent complexities of natural language, including its diverse semantics, multifaceted emotional polarities, and other intrinsic factors, continue to present significant obstacles to the advancement of emotional analysis methods. Therefore Gan C et al. put forward an extended joint architecture of scalable multi-channel CNN and bidirectional long short-term memory (CNN-BilSTM) model, and combined AM to analyze the emotional orientation. In addition, this algorithm introduced a loss function, which could avoid categories’ imbalance when training. This model's accuracy and Macro-F1 on one corpus were improved by more than 1.19% and 0.9%, respectively. Its accuracy and F1 on another corpus was improved by more than 1.7% and 1.214%, respectively.¹⁰

In summary, experts are constantly researching topics related to behaviour recognition and text classification. They proposed behaviour recognition and text recognition models based on CNN and dual flow networks and continuously improved them in the feature fusion module. However, CNN-based behaviour recognition models still have the problem of easily losing behaviour data, and current text recognition and classification models also have the problem of being difficult to handle complex and lengthy sentences. Therefore, it is worth researching how to design a model that can recognize complex text in short videos and accurately recognize human behaviour in videos.

3 Emotional communication in short video based on 3D CNN and MSA

In visual analysis of video data, the recognition of emotions in videos mainly includes human motion recognition within the video and emotion recognition of short video text content.¹¹ At present, the main CNN models used have limited processing capabilities for video data. When processing behavioural feature recognition, information loss may occur due to the use of an average sampling strategy to convert video data into image frames, resulting in insufficient recognition accuracy and limitations in processing complex text.¹² To address these issues, a new text recognition emotion classification model and a human behaviour recognition classification model are designed based on CNN models.

3.1 Design of a text emotion recognition model based on feature fusion strategy

To achieve higher accuracy in short video text relationship classification and extract important feature information, it needs to apply as many local features as possible in global relationship classification. CNN, as an application of deep learning, is essentially a process of finding the optimal weight value, which can be described by Equation (1).

θ * = \arg max_{θ} L (f (x, θ), y)

(1)

In Equation (1), L is the Loss function of CNN.

θ

contains all parameters in the parameter space of the text CNN model.

θ *

is the group with the best weight value among all parameters.

f (x, θ)

represents this model. y represents the entered data information. y is each sample's true label.¹³ After calculating the gradient of ownership weight values, the weight values will be updated, and the weight values updating for each iteration is represented by Equations (2) and (3).

υ_{i t e r} = β_{1} υ_{i t e r - 1} + (1 - β_{1}) g_{i t e r}

(2)

In Equation (3), the update direction at the

i t e r

iteration is represented by

υ_{i t e r}

β_{1}

is attenuation coefficient.

ω_{i t e r} = β_{2} ω_{i t e r - 1} + (1 - β_{2}) g_{i t e r} \cdot g_{i t e r}

(3)

In Equation (3), the momentum of the

i t e r

iteration is represented by

ω_{i t e r}

β_{2}

is attenuation coefficient.

When the model classifies text, it is first necessary to use Equation (4) to represent this sentence.

a_{1 : m} = a_{1} \oplus a_{2} \oplus \dots \oplus a_{m}

(4)

In Equation (4),

a_{m}

is used as a vector to represent a known vocabulary in the sentence. The entire sentence contains m such vectors for concatenation. Afterwards, each convolution kernel with a size of

h * d

can perform convolution operation on the matrix through Equation (5) to obtain new feature information

c_{i}

c_{i} = f (A \cdot a_{i : i + h - 1} + b)

(5)

In Equation (5), the nonlinear function is f, A is the matrix formed by the input vector,

a_{i : i + h - 1}

is the input vector, and b is the bias term. After being stimulated by the relu function, the feature map generated by each window is obtained, which is expressed by Equation (6).

c = [c_{1}, c_{2}, c_{3}, \dots, c_{n - h + 1}], c \in R^{n - h + 1}

(6)

In Equation (6),

R^{n - h + 1}

represents a set of all relationship types.

Considering that the text length is not fixed, the most important feature information of the corresponding feature layer will be captured through the maximum pooling operation. By pooling the global maximum value, two-dimensional data of feature layer is transformed into a one-dimensional vector. The pooled vector's length is equal to channels number in feature layer. This can achieve that each channel only retains the maximum value of data. This process is described by Equation (7).

α_{i}^{(0)} = max_{v} (\overset{\land}{s_{i}^{l}} (v))

(7)

In Equation (7),

α_{i}^{(0)}

is used as the vector.

\overset{\land}{s_{i}^{l}} (v)

is the v -th feature data of the i -th channel in the l -th layer. After obtaining this vector, it is input into the fully connected network, and the relu function activates it, expressed by Equation (8).

α_{j}^{(1)} = r e l u (\sum_{i} α_{i}^{(0)} w_{i j}^{(a_{1})} + b_{j}^{a_{1}})

(8)

In Equation (8),

α_{j}^{(1)}

is a hidden layer vector.

w_{i j}^{(a_{1})}

and

b_{j}^{a_{1}}

are the weight parameters for the update gate operation. After obtaining hidden layer, results are transmitted to output layer. In neural networks, both output and input are vectors with channel quantities equal to length. Activation function in output layer is sigmoid, and the output result will be a probability value in the interval (0,1), which is described by Equation (9).

α_{i}^{(2)} = s i g m o i d (\sum_{j} α_{j}^{(1)} w_{j i}^{(a_{2})} + b_{i}^{a_{2}})

(9)

In Equation (9),

w_{i j}^{(a_{2})}

and

b_{j}^{a_{2}}

are the weight parameters for the update gate operation. Finally, to suppress the worthless text data of some channels and retain important channel text data, Equation (10) is used for the resulting

α_{i}^{(2)}

s_{i}^{l} = α_{i}^{(2)} \overset{\land}{s_{i}^{l}}

(10)

In Equation (10),

\overset{\land}{s_{i}^{l}}

is the i -th channel's characteristic data in the l -th layer.

Although the emotional text feature information in sentences can be captured through maximum pooling operation, the feature information that can be extracted by a single convolutional layer is limited. To enhance the quantity of relationship extraction, it is essential to consider not only augmenting the volume of feature information following convolution but also incorporating the feature information of the training word vector into the convolutional network. This is because the training of word vectors is based on a text convolutional network model, which is rich in semantic information. Figure 1 shows the improved model structure.

Figure 1.

Improved text convolutional network model.

Compared with the traditional text sentiment classification convolutional network model, the new model improves the feature fusion module. Firstly, it is necessary to average the words that have already been represented as vectors in a sentence to obtain an average vector that can represent all words. Then, after activating the relu function, $O_{e}$ enters the fully connected layer. Combining the feature information that would have been obtained through convolution, both types of information are simultaneously used as the final feature output, providing two types of feature information for relationship classification. This information is more comprehensive than the emotional information that the original text convolutional neural model can handle.

When evaluating the modified model performance, indicator accuracy, recall rate, and F-value are introduced. Due to the fact that accuracy and recall rates are contradictory, it is necessary to introduce an F-value for comprehensive consideration in model detection, represented by Equation (11).¹⁴

F = (1 + β^{2}) \frac{p r e c i s i o n^{*} r e c a l l^{*} (1 + β^{2})}{(p r e c i s i o n + r e c a l l)}

(11)

In Equation (11),

β

represents the description of the importance of accuracy and recall. This study takes

β

as 1.

3.2 Design of a human motion recognition model based on 3D CNN and MSA

The principle of capturing human motion information in a three-dimensional convolutional model is to convolution the part formed by the superposition of multiple channels and a three-dimensional convolutional kernel. This enables feature mapping to connect from the convolutional layer to adjacent frames in the previous layer to obtain human motion status. 3D convolution process is described by Equation (12).¹⁵

v_{i j}^{x y z} = r e l u (b_{i j} + \sum_{m} \sum_{P = 0}^{P_{i} - 1} \sum_{q = 0}^{Q_{i} - 1} \sum_{r = 0}^{R_{i} - 1} k_{i j m}^{p q r} v_{(i - 1) m}^{(x + p) (y + q) (z + r)})

(12)

In Equation (12),

R_{i}

is the size of 3D convolutional kernel in the time dimension t.

P_{i}

is the size of 3D convolutional kernel in the spatial dimension x direction.

Q_{i}

is the size of 3D convolutional kernel in the spatial dimension y direction. p, q, r are positions in the three-dimensional x, y, and t directions. The value of the m-th convolutional kernel on the coordinate point

(p, q, r)

is represented by

k_{i j m}^{p q r}

b_{i j}

represents the current network bias. After operation, the convolution value

v_{i j}^{x y z}

of the j-th feature map of the i-th layer network at position

(x, y, z)

is obtained by activating the relu function. Figure 2 shows the structure of the human behaviour recognition model. In the short video preprocessing module, it is necessary to segment the input short video to obtain several small segments, and input them into the network layer in the format of optical flow frames or RGB original frames. The input video clip will be converted into image format through OpenCV encoding and decoding for storage. In the spatiotemporal feature extraction module, the network layer needs to extract spatial features of behaviour appearance and temporal features of behaviour in short videos. The optical flow or RGB sequences in short video segments are inputted into only 3DCNN frame by frame, and the image feature information is output after network layer operation. The AM module includes MSA structure and Long Short Term Memory (LSTM). After the feature extraction of the previous module, the video clip is first input into the MSA structure. Then LSTM is used to process and predict the parts with longer intervals and delays in the timing information.

Figure 2.

Structure of human behaviour recognition model.

This study considers that traditional AM structures can cause heavy computational burden when processing large images due to the excessive number of nodes in the image. MSA¹⁶ is used in this study to address this issue. The advantage of MSA is that it can improve model performance. By simultaneously focusing on multiple parts, the model can better understand the key information in the input data, thereby improving the accuracy and robustness of the model. MSA uses multiple query vectors $A = [a_{1}, a_{2}, \dots \dots, a_{m}]$ to simultaneously select multiple sets of information from the input data $(K, V) = [(k_{1}, v_{1}), (k_{2}, v_{2}), \dots \dots, (k_{m}, v_{m})]$ . During the selection process, the query vector $a_{i}$ captures different parts of the input information, which filters the feature information of the input data from different perspectives. The attention weight of the i -th query vector $q_{i}$ and the j -th input information $k_{j}$ is set to $w_{i j}$ . When MSA screening, the calculation process of attention weight is described by Equation (13).

w_{i j} = s o f t m a x (s (k_{j}, q_{i})) = \frac{\exp (s (k_{j}, q_{i}))}{\sum_{t = 1}^{n} \exp (s (k_{t}, q_{i}))} c o n t e x t_{i} = \sum_{j = 1}^{n} w_{i j} \cdot v_{j}

(13)

In Equation (13),

c o n t e x t_{i}

is the attention vector obtained through query vector. After obtaining the attention weight, all the selected query vectors are concatenated to obtain the final output result of MSA, which is described by Equation (14).

c o n t e x t = c o n t e x t_{1} \oplus c o n t e x t_{2} \oplus \dots \dots \oplus c o n t e x t_{m}

(14)

In Equation (14),

\oplus

describes the concatenation operation of vectors.

Figure 3 shows the structure diagram of the information fusion module. In the information fusion module, fusion operations are performed on spatiotemporal features and spatiotemporal scores. After passing through the spatiotemporal feature extraction module and MSA module, the short video segments generate feature vectors. The information fusion module performs score fusion on the feature vectors of different streams to predict classification. Then, the softmax classifier module is inputted to generate prediction scores. Finally, the global spatiotemporal prediction scores are fused to obtain the final prediction results of human motion state information.

Figure 3.

Structure of the information fusion module.

4 Experimental results and analysis

The word vectors required come from Word2vec. It is an open-source training word vector library from Google, which can convert a given word in a sentence into a vector to represent it through word vector technology.¹⁷ UCF101 database and HMDB51 database are used in the experiment, and the input video included Sc fragments, with each fragment containing Im frames.¹⁸ When a video with less than 90 frames appears, it is necessary to copy the last frame of the image until it reaches 90 frames. The AM time step parameter is 10. The evaluation indicators are evaluation accuracy, recall rate, and the harmonic average based on the F-Score principle. The accuracy of UCF101 (%) is implemented on the UCF101 database and HMDB51 (%) is implemented on the HMDB51 database.

4.1 Performance simulation results and analysis of short video text sentiment classification model

For a sentence x with a given m words, using word vector technology, the sentence can be represented as: each word is represented as a real value vector $x = {a_{1}, a_{2}, \dots, a_{m}}$ . This model is trained using the layered softmax method. Table 1 provides the key parameter settings for the training word vector.

Table 1.
Key parameter settings.

Parameter names Model selection Training method selection Word vector dimension Training window size

Take a value Skip-Gram Hierachical Softmax 60 8

Parameter names Sampling threshold How results are stored Learning rate Lowest frequency

Take a value le-3 0 0.025 8

Evaluation index Iteration time Rate of precision Rate of recall F-Score

Take a value / / / /

Parameter names	Model selection	Training method selection	Word vector dimension	Training window size
Take a value	Skip-Gram	Hierachical Softmax	60	8
Parameter names	Sampling threshold	How results are stored	Learning rate	Lowest frequency
Take a value	le-3	0	0.025	8
Evaluation index	Iteration time	Rate of precision	Rate of recall	F-Score
Take a value	/	/	/	/

Figure 4 shows the accuracy changes of the original text CNN and the improved text CNN in text extraction under the 60-word vector dimension. Accuracy is used as an evaluation indicator to represent the model's detection accuracy for samples that have already been detected, considering the influence of positional factors. From Figure 4, the accuracy of both the new and old models decreases as the detection results increase. The improved model has an accuracy of about 75%, while the average accuracy of the original model is only about 67%. After detecting 300 samples, the accuracy of the improved model remains at 73.2%, which is more than 10 percentage points higher than the original model.

Figure 4.

Trend of precision changes.

Figure 5 shows the trend of recall rate changes between new and old models during the experimental process. As the number of tested results increases, the overall recall rate of both new and old models is increasing. But when the detected results exceed 250, the recall rate of the original model decreases, below the peak of 76.3%. In comparison, the improved model remained stable in the experiment, with an average recall rate of 85.2%.

Figure 5.

Recall rate of old and new models.

Figure 6 shows the trade-off data of F-Score values between the new and old models under average accuracy and average recall. The average F-value of the improved model is about 6 percentage points higher than the original model, and more abundant short video text information is introduced by feature fusion strategies. The improved convolutional network model for short video text recognition has more advantages than traditional models.

Figure 6.

Comparison of F-values between new and old models under different sample sizes.

4.2 Experimental results and analysis of performance testing of emotion classification models for short video human motion recognition

The designed model is tested for human motion recognition emotion classification performance under different video clip sizes and image frame numbers. From the data listed in Table 2, the system has the lowest recognition accuracy on UCF101 and HMDB51 databases when video clip number is 4 and image frame number is 5, with 71.6% and 49.2%, respectively. When video clip number is 5 and image frame number is 20, the system has the highest recognition accuracy on the UCF101 database, at 94.8%. When video clip number is 6 and image frame number is 15, the system has the highest recognition accuracy on the HMDB51 database, at 71.5%.

Table 2.
Model recognition accuracy data for different values of Sc and Im.

Sc Im UCF101 (%) HMDB51 (%)

4 5 71.6 49.2

5 5 76.5 53.6

6 5 82.4 59.8

4 10 84.4 62.4

5 10 87.3 66.9

6 10 90.2 69.7

4 15 90.6 69.2

5 15 92.7 70.5

6 15 93.2 71.6

4 20 93.4 71.5

5 20 94.8 71.4

6 20 94.6 71.5

Sc	Im	UCF101 (%)	HMDB51 (%)
4	5	71.6	49.2
5	5	76.5	53.6
6	5	82.4	59.8
4	10	84.4	62.4
5	10	87.3	66.9
6	10	90.2	69.7
4	15	90.6	69.2
5	15	92.7	70.5
6	15	93.2	71.6
4	20	93.4	71.5
5	20	94.8	71.4
6	20	94.6	71.5

Figure 7 shows the trend of the recognition accuracy change of the model when Sc is taken as 4, 5, and 6 during the process of Im value changing from 5 to 20. From Figure 7, as image frame number increases, this model's overall recognition accuracy on both databases increases. When the image frame number is 5 or 10, the overall recognition accuracy of the model on the UCF101 database increases by about 10 percentage points as the video clip number increases, while the recognition accuracy on the HMDB51 database increases by 6 to 10 percentage points. When image frame number is high, ranging from 15 to 20, the increase in video clip number affects this model's recognizing accuracy lightly. Although the overall trend is on the rise, the increase is slightly small, with recognition accuracy increasing by 0.1 to 3 percentage points on both databases.

Figure 7.

Trend of recognition accuracy change of the model.

To compare the recognition accuracy of the model under different AMs, the recognition accuracy of the model is tested under the conditions of 6 video clips and 20 image frames. The recognition accuracy on UCF101 and HMDB51 databases is achieved in the experiment when SE block (Squeeze and Excitation), CBAM (Convolutional block attention module), MSA, and without AM are used. Figure 8 shows the changes in recognition accuracy of the model under different AMs. From Figure 8, compared to recognition models with other AMs, the model under MSA has the highest recognition accuracy, with a recognition accuracy of 94.5% on UCF101 database, which is 12 to 39 percentage points higher than models under other AMs. The recognition accuracy on HMDB51 database is 71.6%, which is 1 to 19 percentage points higher than the model under other AMs.

Figure 8.

Recognition accuracy of the model under different attention mechanisms.

Table 3.

Identification accuracy data of four algorithms.

UCF101 (%)
Algorithm image frames	5	10	15	20
3D + MSA	76.8	87.3	92.2	94.3
TSN	75.5	84.9	89.7	93.6
IDT	67.5	74.3	79.8	84.2
Two-Stream	68.9	71.3	78.6	88.1
HMDB51 (%)
Algorithm image frames	5	10	15	20
3D + MSA	54.2	66.3	70.4	71.5
TSN	63.6	65.2	67.8	68.5
IDT	50.1	52.3	54.6	57.2
Two-stream	52.6	55.3	57.1	59.4

To detect whether the model based on 3DCNN and incorporating AM has improved performance, the model designed in this study is compared with commonly used methods in behaviour recognition. They include IDT algorithm (improved dense trajectories, IDT), two stream convolutional networks, and 2D CNN (Temporary Segment Networks, TSN). The average recognition accuracy of different methods under different image frame numbers is compared in the experiment. Table 3 shows the recognition accuracy data of four algorithms. When the image frame number is 20, the average recognition accuracy of the model designed in this study is the highest at 94.3% on UCF101 database. On UCF101 database, the highest average recognition accuracy of this model is 71.5%.

Figure 9 shows the trend of recognition accuracy changes for the four algorithms. On UCF101 database, this designed model's overall recognizing accuracy is higher than other behaviour recognition algorithms. When image frames number is 20, its average recognition accuracy is 94.3%, which is 0.7 to 6 percentage points higher than other algorithms. On HMDB51 database, the recognition accuracy of the 2D convolutional network at low image frames is about 9 percentage points higher than the recognition accuracy of the model designed in this study, which is 63.6%. However, overall, the recognition accuracy of the model designed in this study increases as image frames number increases. When image frames number is 20, its recognition accuracy is 71.5%, which is 3 to 14 percentage points higher than other algorithms. Moreover, its average recognition accuracy is 65.6%, which is generally better than other algorithms.

Figure 9.

Trends in recognition accuracy of four algorithms.

5 Conclusion

In the text recognition and behaviour recognition tasks of short video sentiment analysis, traditional CNN is limited in processing complex text information, and processing behaviour data is prone to information loss. In this study, based on the CNN model, a new text recognition emotion classification model was designed by improving the text feature fusion module, and a new human behaviour recognition emotion classification model was designed by introducing MSA on the basis of the 3DCNN model. The results confirmed that the improved text recognition model's accuracy was about 75%, while the average accuracy of the original convolutional model was only about 67%. After detecting 300 samples, the accuracy of the improved model remained at 73.2%, which was more than 10 percentage points higher than the original model. The average F-value of the improved model was about 6 percentage points higher than that of the original model. Compared to recognition models with other AMs, the 3D convolutional model under MSA had the highest recognition accuracy, with a recognition accuracy of 94.5% on the UCF101 database, which was 12 to 39 percentage points higher than models under other AMs. The recognition accuracy on the HMDB51 database was 71.6%, which was 1 to 19 percentage points higher than the model under other AMs. On the UCF101 database, the overall recognition accuracy of the designed model in this study was higher than other behaviour recognition algorithms. The average recognition accuracy was 94.3% when the image frame number was 20, which was 0.7 to 6 percentage points higher than other algorithms. Overall, it is better than other algorithms. However, the selection of video databases in this study is not diverse enough, and it does not consider the existence of fuzzy videos in real life. This is also a direction that can be studied in the future.

Statements and declarations

Funding

The research is supported by: General research topics of preschool education in Shandong province “A study on parents’ park-choosing intention in Shandong province based on TPB theory” Periodic results, Project number: 2022XQJY047.

Conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Guo

Mustafaoglu

Koundal

. Spam detection using bidirectional transformers and machine learning classifier algorithms. J Comput Cognit Eng 2023; 2: 5–9.

Blaivas

. Are all deep learning architectures alike for point-of-care ultrasound?: Evidence from a cardiac image classification model suggests otherwise. J Ultrasound Med 2020; 39: 1187–1194.

Meng

Zhang

Chen

, et al. Triplet interactive attention network for cross-modality person re-identification. Pattern Recognit Lett 2021; 152: 202–209.

Lin

Wang

. Integrated image sensor and light convolutional neural network for image classification. Math Probl Eng 2021; 2021: 5573031.1–5573031.7.

Tang

Zhang

Yin

. Temporal consistency two-stream CNN for human motion prediction. Neurocomputing 2022; 468: 245–256.

Zhu

Chen

Zheng

, et al. Automatic recognition of lactating sow postures by refined two-stream RGB-D faster R-CNN – Science Direct. Biosyst Eng 2020; 189: 116–132.

Peng

Huang

Tsoi

, et al. Motion boundary emphasised optical flow method for human action recognition. IET Comput Vision 2020; 14: 378–390.

Wang

, et al. Sentence semantic matching based on 3D CNN for human–robot language interaction. ACM Trans Internet Technol 2021; 21: 1–24.

Wang

Tang

Yang

, et al. A novel network with multiple attention mechanisms for aspect-level sentiment analysis. Knowl-Based Syst 2021; 227: 107196.1–107196.12.

10.

Gan

Feng

Zhang

. Scalable multi-channel dilated CNN-BiLSTM model with attention mechanism for Chinese textual sentiment analysis. Future Gener Comput Syst 2021; 118: 297–309.

11.

Qin

Zhang

Liu

, et al. A visual place recognition approach using learnable feature map filtering and graph attention networks. Neurocomputing 2021; 457: 277–292.

12.

Chen

Huang

, et al. Neighbor enhanced graph convolutional networks for node classification and recommendation. Knowl-based Syst 2022; 246: 108594.1–108594.10.

13.

Wang

Yan

, et al. Neural graph personalized ranking for top-N recommendation. Knowl-Based Syst 2020; 213: 106426.1–106426.9.

14.

Mao

, et al. Improving convolutional neural network for text classification by recursive data pruning. Neurocomputing 2020; 414: 143–152.

15.

Jin

Zhou

, et al. Attention mechanism-based CNN for facial expression recognition. Neurocomputing 2020; 411: 340–350.

16.

Mishra

Sanyal

, et al. Real time human action recognition using triggered frame extraction and a typical CNN heuristic. Pattern Recognit Lett 2020; 135: 329–336.

17.

. Research on methods of English text detection and recognition based on neural network detection model. Sci Programming 2021; 2021: 6406856.1–6406856.11.

18.

Wei

Kamruzzaman

. Inter/intra-category discriminative features for aerial image classification: a quality-aware selection model. Future Gener Comput Syst. 2021; 119: 77–83.

Short video emotion transmission based on 3D convolutional neural network and multi-head attention mechanism

Abstract

Keywords

1 Introduction

2 Related works

3 Emotional communication in short video based on 3D CNN and MSA

3.1 Design of a text emotion recognition model based on feature fusion strategy

4.1 Performance simulation results and analysis of short video text sentiment classification model

Table 2. Model recognition accuracy data for different values of Sc and Im. Sc Im UCF101 (%) HMDB51 (%) 4 5 71.6 49.2 5 5 76.5 53.6 6 5 82.4 59.8 4 10 84.4 62.4 5 10 87.3 66.9 6 10 90.2 69.7 4 15 90.6 69.2 5 15 92.7 70.5 6 15 93.2 71.6 4 20 93.4 71.5 5 20 94.8 71.4 6 20 94.6 71.5

Statements and declarations

Funding

Conflicting interests

References

Table 2.
Model recognition accuracy data for different values of Sc and Im.

Sc Im UCF101 (%) HMDB51 (%)

4 5 71.6 49.2

5 5 76.5 53.6

6 5 82.4 59.8

4 10 84.4 62.4

5 10 87.3 66.9

6 10 90.2 69.7

4 15 90.6 69.2

5 15 92.7 70.5

6 15 93.2 71.6

4 20 93.4 71.5

5 20 94.8 71.4

6 20 94.6 71.5