The Syncretic Effect of Dual-Source Data on Affective Computing in Online Learning Contexts: A Perspective From Convolutional Neural Network With Attention Mechanism

Abstract

Affective computing (AC) has been regarded as a relevant approach to identifying online learners’ mental states and predicting their learning performance. Previous research mainly used one single-source data set, typically learners’ facial expression, to compute learners’ affection. However, a single facial expression may represent different affections in various head poses. This study proposed a dual-source data approach to solve the problem. Facial expression and head pose are two typical data sources that can be captured from online learning videos. The current study collected a dual-source data set of facial expressions and head poses from an online learning class in a middle school. A deep learning neural network using AlexNet with an attention mechanism was developed to verify the syncretic effect on affective computing of the proposed dual-source fusion strategy. The results show that the dual-source fusion approach significantly outperforms the single-source approach based on the AC recognition accuracy between the two approaches (dual-source approach using Attention-AlexNet model 80.96%; single-source approach, facial expression 76.65% and head pose 64.34%). This study contributes to the theoretical construction of the dual-source data fusion approach, and the empirical validation of the effect of the Attention-AlexNet neural network approach on affective computing in online learning contexts.

Keywords

affective computing multimodal dual-source data fusion method attention mechanism

Introduction

The COVID-19 global pandemic has witnessed online learning becoming an indispensable supplementary form in education. However, the wholly online learning model has impaired peers’ communications and teacher-student interactions, which results in inefficient learning performance and many psychological issues (B. Liu et al., 2021; Wei & Chou, 2020). This dilemma may be because of the lack of timely and effective responses to students’ affection, which may detrimentally impact interactions and learning information processing in online learning environments (Han et al., 2021; Z. Liu et al., 2019). Therefore, optimizing the AC approaches in online learning settings is of great theoretical and practical significance.

The cognitive psychological status of learners’ affection was regarded as a reliable predictor of their academic behaviours (Xu et al., 2014). It was also considered a critical indicator of assessing formative learning outcomes (Faria et al., 2017; Zhang et al., 2020). With the increasing progress in the construction and accumulation of the emotion data set, more and more deep learning and neural network models have been developed for affective computing using emotional data, which in return have facilitated the optimization of algorithms to improve the accuracy of affection computing (Kumar, 2021; Fouladgar et al., 2021). However, computing online learners’ affections have many barriers. Firstly, online learning requires lightweight technologies to meet the massive implementation. Prior research mainly employed heavyweight physiological feedback technology, such as EEG, and MRI, to understand learners’ behaviours and cranial nerve mechanism and recognize their emotions (H. Liu et al., 2021, 2021; Shu et al., 2018). However, heavyweight physiological technologies are typically utilized in small-scale experiments, which are challenging to be employed in a massive online learning context. Meanwhile, collecting physiological data was commonly invasive, distracting learners from a learning status and impair their affective expressions. For example, EEG data collection requires external EEG equipment to be in contact with the human head. Facial expression identification is a typical method in AC with many challenges. Although facial expression identification is a relatively lightweight AI technique for persons’ affection recognition (Barrett et al., 2019), sole facial expression data cannot accurately analyse learners’ affections. Given that varied instructive designs used online would generate distinct interactive behaviors between online learners and tutors, requiring multi-source data such as head movements poses to analyse learners’ affections collaboratively. Online learners are typically in an unsupervised learning context, and their facial expressions are vulnerable to the influence of head poses (Wang et al., 2020). Introducing complementary data sources for a single facial expression would improve the accuracy of AC to realize cross-validation and mutual compensation of data information (Wu et al., 2019).

Although deep learning algorithms, such as Convolutional Neural Network (CNN), have achieved good results in AC (Hassan et al., 2019), developing an optimized AC algorithm in online learning settings has two challenges. Firstly, some typical neural network algorithms are deficient in weights allocation, leading to the impairment of enhancing key features and weaken useless features. Wu et al. (2021) reported that CNN could not obtain useful feature information due to its small number of convolution layers, which challenge the accurate extraction of facial expression features and further undermine the facial expression recognition rate. Additionally, traditional CNN has the disadvantage of translation invariance which lead to the concealing of key information from the data (Hou et al., 2021; Sabour et al., 2017). Therefore, the compelling features of the input should be given importance, and computing resources should be allocated to the features. The second challenge is a lack of an affective computing data set to identify online learners’ affections. Students’ affection and expressions may differ from other working situations, which warrants developing a learner affection data set suitable for online learning. Especially in the post-pandemic era, massive online learning has become a new normal learning mode, and online learners’ affections may be affected by pressures from various dynamics in the learning settings (Lockee, 2021).

Tackling the above challenges, this study collected a dual-source affection data set in online learning settings in a middle school, including students’ facial expression images and head pose images. The facial expression images include emotions such as happy, confused, calm and boredom. The head pose images include happy, confused, calm and boredom, which consists of feature points of head contour, eyes, eyebrows, mouth, and nose. Based on AlexNet (Krizhevsky et al., 2012), this study employed the attention mechanism to construct the Attention-AlexNet model that can determine important regions in facial expressions and head poses image data, and allocate more weights on them. By constructing dual-source affection data and AlexNet with attention mechanism, the current study compares the effects of different affection computing models, obtains an optimal model for assessing students’ learning status in an online learning context, and verifies the effectiveness of affective computing a proposed dual-source fusion strategy.

Related Work

Affective Computing in Online Learning Environments

The COVID-19 pandemic has been significantly reforming learning from physical environments to online settings, enabling the effect of affective computing on online learners. However, online learning, primarily conducted in an unsupervised and monotonous context, may increase the risk of perceived stress and anxiety, even leading to learning exhaustion and career burnout (Mheidly et al., 2020). Some researchers found that the positive effect of students’ affection on the performance in the test, students with positive affection showed better performance (Kaur et al., 2021). Another study has certified students’ learning outcomes were related to their affections by investigating the relations between students’ negative affections, acceptance of online education and learning outcomes during the COVID-19 crisis (Tzafilkou et al., 2021). Because the positive affectivity improves students learning motivation and then generate influence in learning performance (Jiménez et al., 2018), while some negative affection, such as boredom lead to insufficient engaged concentration which further impair learning efficiency (Pardos et al., 2013). The above shreds of evidence suggest that affection is a crucial component of meaningful learning. Meanwhile, it is an essential measurement of learning performance. However, many students still fail to regulate their affections in an unsupervised context especially in the remoting learning settings, which appeal to teachers to to obtain students’ learning affections and make appropriate interventions efficiently.

Learners’ affections can be measured by varied data sources, such as facial expression, pose, and physiological parameters. Facial expression is an essential predictor to evaluate an individual’s affective status. Based on facial identification, Z. Liu et al. (2017) constructed a human-robot interaction system, by which human affections could be recognized and artificial facial expression could be generated. Besides, some researchers have found that head pose are another essential factor driving reliably for learners’ affection. Stephens-Fripp et al. (2017) carried out a research to identify learners’ affections and obtain a reliable recognition rate based on body gait and pose. Furthermore, some researchers employed physiological parameters to recognize students’ learning affections. Based on electrocardiograph signals, Hsu et al. (2017) presented an automatic recognition algorithm for exploring the recognition of students’ current affection. The above studies conclude that most AC studies are confined to single-source data and rarely explore multi-source data fusion.

Multi-Source Data Fusion in Affective Computing

Multi-source data fusion is an emerging subject of further affective computing research but poses many challenges. On the one hand, the data relevant to affection was diverse, including EEG, ECG, facial expression, pose and behavior. The syncretic effect of multi-source data can outperform the limitations posed by processing a single form of data and offer more references for AC, which improves the accuracy and robustness of general computing (Cimtay et al., 2020). On the other hand, multi-source data fusion’s AC ability shows an obvious advantage compared with single-source data. Poria et al. (2015) proposed a multimodal information extraction agent, which used three-modal (text, audio, and video) features to enhance the multimodal information extraction process, inferred and aggregated the affective information and achieved an accuracy 87.95%. Salama et al. (2021) compared three approaches (EEG, face, fusion-based) in AC and concluded that the fusion-based affective computing approach had the best accuracy.

Fusion of facial expression and head pose is an effective method to improve affection recognition accuracy. Head pose affects the accuracy of facial expression recognition, considering the facial expression with varied head poses may present different affections (Valstar et al., 2017). For example, a smile face may denote excited affection or helpless affection, while head pose may help to be an interpretive variable to certify the feature of smile. Additionally, most facial expression recognition methods were conducted through the picture captured by a frontal or near frontal head posture, while its accuracy is usually significantly reduced when using a non-frontal posture for testing (Vieriu et al., 2015). Moreover, head poses carry affective information that is complementary rather than redundant to the affection content in facial expressions (Adams et al., 2015), which presents a contextual cue for the interpretation of the affection by facial expressions (Hess et al., 2007)”. B. Abisado et al. (2020) constructed a multi-modal affection model to identify examinees’ five academic affections based on facial expressions and head pose data, and the accuracy is significantly increased to 92.66%. The head pose and facial expressions was also fused in the context of online learning, and it was certified as a reliable approach for measuring students’ engagement and affective states (Yu et al., 2021).

Current research mainly focused on two fusion approaches of multi-source: feature-level fusion and decision-level fusion (Jiang et al., 2020). Feature-level fusion connects features extracted from each modal to a new feature vector. Firstly, feature-level fusion needs to extract features’ representation by each modal data to obtain the commonness of different data sources. It syncretizes multiple independent data sets into a single feature vector and inputs into the machine or deep learning classifier to make decisions. Decision-level fusion syncretises the different data training results, which use the AC models to train affection data and then syncretize the models’ output. Different from feature-level fusion, decision layer fusion emphasizes the distinctions among additional features and can sharpen the discriminative ability. In the current studies, multi-source data fusion has been carried out to conduct AC. Bandara et al. (2016) combined ECG and EDA data to implement affection classification utilizing the decision-level fusion approach. Using EEG and facial expression data, Huang et al. (2017) recognized four affection states (happiness, neutrality, sadness, fear) by decision-level fusion methods. Syncretizeing electroencephalography (EEG) data and facial expression data, D. Li et al. (2019) employed a decision-level fusion framework for detecting affections continuously. Lately, using a feature-level fusion technique, Bairavel and Krishnamurthy (2020) proposed an audio-video-textual-based multimodal affection analysis approach to reach a satisfactory identification rate. Furthermore, based on speech and image data, Y. Li et al. (2018) compared different data fusion methods in AC and found that the decision-level fusion approach is superior to the feature-level fusion approach.

Nevertheless, feature-level fusion has some deficiencies. Firstly, representing the time synchronization between multi-source data features likely results in redundancy between different data (Wu et al., 2006). Secondly, highly correlated feature-level data likely result in a strong correlation between errors in multi-source data. Thus, decision-level fusion may be a more suitable approach for affective computing based on facial expression and head pose data.

CNN and Attention Mechanism

Compared with traditional machine learning methods, deep learning has become the core driving force in the era of artificial intelligence because of its strong learning ability, satisfactory adaptability, higher efficiency, and accuracy. As a feedforward neural network with depth structure including convolution calculation, CNN is one of the representative deep learning algorithms. The basic architecture of the CNN consists of the input, convolution, pooling, full-connection and output layers. According to the structural classification of the convolution layer and pooling layer, the convolution neural network model is divided into network structures with different performances, such as AlexNet, VggNet, ResNet, and GoogleNet.

The existing research outcomes on attention mechanism have been adopted for computer vision research. Several types of attention mechanisms are commonly used in CNNs, such as the Squeeze-and-Excitation Network and the Convolutional Block Attention Module. In CNN, visual attention was typically classified into spatial and channel attention. Spatial attention functions as a transformer of extracting the key information from the spatial information and commonly ignores the information in the channel, which confines the spatial transformation method to the stage of feature extraction of the original picture. Furthermore, to focus on the image’s task-related area effectively, channel attention enables the neural network to determine automatically which channel is important or unimportant and then assign the appropriate weights (Guo et al., 2022). However, the information was typically pooled into the channel’s attention globally, ignoring the local information in each channel. To compensate for the disadvantage of only one kind of attention, Woo et al. (2018) designed a network structure with an attention mechanism in the two dimensions of feature channel and feature space, called the Convolutional Block Attention Modul. It is a lightweight and generic module that can focus properly on target objects, and accurate attention and noise reduction of irrelevant clutters.

CNN and Attention Mechanism for Affective Computing

The Convolutional Neural Network has shown outstanding performance in affective computing (Aydoğdu, 2021). Many CNN structures are applied to AC, such as AlexNet, ResNet, and VggNet (Gan, 2018). AlexNet has more layers and stronger learning capability among various CNNs, and has advantages in online learners' affective computing (J. Li et al., 2021). On the one hand, containing five convolution layers and three fully connected layers, AlexNet has a simple network structure avoiding redundant network parameters, which improve the convergence rate and generalization ability (Krizhevsky et al., 2012). Khare and Bajaj (2020) evaluated four CNNs in affection recognition, and AlexNet offered quicker training and testing. Based on flexible strucures, AlexNet is more convenient to upgrade the structure and obtain a better effect for different online learning image data sets. On the other hand, AlexNet has good effects on facial expression recognition. Giannopoulos et al. (2018) used AlexNet to judge seven different emotions, which were superior to the accuracy of GoogLeNet given its shallow architecture. Demircan and Örnek (2020) used AlexNet architecture for affective computing from speech data and achieved more discrimination. Kartali et al. (2018) compared three deep-learning approaches (fine-tuned AlexNet CNN, Affdex CNN and FER-CNN) based on convolutional neural networks to achieve the real-time affective computing of four basic affections (happiness, sadness, anger, and fear) from facial images. The results show that fine-tuned AlexNet CNN has better generalization power and performance in real-time applications. CNN has significant strengths in AC, but still has exist some limitations. It barely ignores irrelevant information and focuses on crucial details. Thus, paying more attention to data’s significant features has become a bottleneck problem in neural network algorithms applied in AC.

In recent years, a neural network with an attention mechanism has become an essential principle in deep learning research, which is used to distinguish the critical differences of different features (Lin et al., 2020). The attention mechanism is a combinatorial function to highlight the impact of a critical input on the output by calculating the probability distribution of attention. One of the effective approaches to CNN structure optimization is employing an attention mechanism. Attention mechanism, enlightened from the human vision, focuses on allocating input weight. It is a data processing method widely used in classification tasks, such as natural language processing, image recognition and speech recognition through an attention map, to improve the receptive field of the underlying features and highlight the more favourable features for classification (Wang et al., 2017). The previous study has proposed three ways to use attention in CNN to model a pair of sentences, including ABCNN-1, ABCNN-2, and ABCNN-3. The result shows that attention-based CNNs had a better accuracy and recall rate performance than CNNs without attention mechanisms. ABCNNs are beneficial in tackling the selection and textual entailment tasks, with competitive performance in paraphrasing identification. ABCNN-3 generally surpasses ABCNN-1 and ABCNN-2 because it can consider the attention of finer-grained granularity in each convolution-pooling block (Yin et al., 2016). Furthermore, introducing the attention mechanism into CNN can make the resultant neural network pay more attention to valuable features and generate more reasonable output (J. Li et al., 2020). For example, CNN with an attention mechanism was employed to identify the occlusion regions of the face and shift the attention from the occluded patches to other related but unobstructed ones. The approach focuses on the most discriminative un-occluded areas, significantly improving the recognition accuracy (Y. Li et al., 2018). Thus, CNN with an attention mechanism could contribute to better AC outcomes.

Methodology

This section includes three parts. The first part establishes the architecture of the dual-source affective computing model. The second part introduces the convolutional neural network approach with the attention mechanism. The third part introduces the decision-based fusion approach for affective computing.

The Architecture of Dual-Source Affective Computing Model

As shown in Figure 1, the architecture of the dual-source affective computing model contains three sequential stages: transformation, recognition, and fusion.

Figure 1.

Architecture of dual-source affective computing model.

The input data are facial expression image and head pose image. In the stage of transformation, the image data of facial expression and head pose was converted into matrix pixel data that could be figured out in the recognition stage. In the recognition stage, AlexNet with an attention mechanism was employed to process facial expression and head pose data. Firstly, the convolutional layers with attention layers were employed to extract features of the facial expressions data and head poses data. Then, the fully connected layers and SoftMax activation function were employed to generate the prediction results. In the fusion stage, the decision-level fusion approach was employed for syncretizing the data accessed from prediction results in the recognition stage. By giving different weights to the results of the recognition model, we obtained the syncretic calculating results of facial expression recognition and head pose recognition.

Convolutional Neural Network with Attention Mechanism

AlexNet

This study employed the AlexNet network as the primary neural network structure for learning AC, which has eight layers of weights, including five convolution layers and three full connections (Figure 2). On the one hand, AlexNet, as a deep convolution neural network model, is superior at extracting features and image classification and has been widely deployed in image recognition (Yuan & Zhang, 2016). On the other hand, the AlexNet has been proven as an efficient model in terms of training time, for example, employing “dropout” method that reduces overfitting in the fully-connected layers (Krizhevsky et al., 2012). Furthermore, AlexNet offers a reliable compromise between network simplicity and high performance and shows a good recognition effect on facial expression datasets that are Cohn-Kanade (CK+) dataset and FER dataset (Stolar et al., 2017; Shaees et al., 2020). Thus, it is an effective and stable way to identify students’ learning affections through AlexNet neural network.

Figure 2.

AlexNet neural network model.

Attention mechanism in convolutional neural networks

This study employs Convolutional Block Attention Module that applies attention to spatial and channel dimensions to maximize the key features and minimize the chaos of inefficient features, (Woo et al., 2018). The computation process of each attention (Figure 3) includes the following stages:

1. The feature map is compressed into two one-dimensional vectors according to its length and width input into the channel attention module. The weight is outputted through the channel attention module.

2. The attention weight is a dot product with the feature map, inputted through the spatial attention module to obtain the output weight and multiplied with the feature map.

3. The result of point multiplication is taken as the output of the attention module.

Figure 3.

Computation process of each attention map. AlexNet with attention an mechanism (Attention-AlexNet).

From the perspective of space and channel, this study developed an “Attention-AlexNet” neural network fusing the attention mechanism to the AlexNet convolutional neural network. The spatial attention mechanism enhances the contrast between salient and potentially irrelevant regions of affection image data. The channel-wise weight mechanism emphasizes the informative affection features while suppressing fewer valuable features (B. Li et al., 2021). In the Attention-AlexNet neural network, the attention module localizes within the third layer of the AlexNet network. The data processing of the Attention-AlexNet neural network is shown in Figure 4.

Figure 4.

Data processing of the Attention-AlexNet.

Firstly, the convolution layer of the Attention-AlexNet is responsible for the convolution operation to extract the feature map. Secondly, the pooling layer is accountable for reducing the size of the feature map, and the attention module is responsible for distributing the weight. Finally, the full connection layer is responsible for making decisions and outputting the prediction results. Procedure is as follows:

1. Adjust the size of the original facial expression or head pose image to (224, 224, 3).

2. Employ a convolution operation, the output feature layer size is 96, and the output size is (55, 55, 96); Employ a maximum pool operation, and the output size is (27, 27, 96).

3. Employ a convolution operation, the output feature layer size is 256, and the output size is (27, 27, 256); Employ an attention module distributing the weight; Employ a maximum pool operation, and the output size is (13, 13, 256).

4. Employ three convolution operations, the output feature layer size is 384, and the output size is (6, 6, 256); Employ a maximum pool operation, and the output size is (28, 28, 256)

5. Input full connection layer, the output size is (1, 1, 4096), two times in total.

6. Input the full connection layer, and the output size is (1, 1, 4).

7. Output predictions for each affection.

Decision-Level Fusion for Affective Computing

Decision-Level Fusion Method

Weighted rule fusion, sum and product rules and highest confidence fusion were principally used in the decision level to determine the output results (Gokberk & Akarun, 2006). Considering that distinct data sources will bring different learning emotion recognition results, the data with higher correlation will significantly impact the results. This study implemented the method of weight rule fusion, which is a fusion method that shows the importance of a part in the whole and distributed different scale coefficients to the dual-source data output results. The output results of the facial expression recognition and head pose recognition are in a probability distribution matrix, which can be represented as follows

p_{i}^{f a c i a l} = {p_{0}^{f a c i a l} 、 p_{1}^{f a c i a l} 、 p_{2}^{f a c i a l} 、 p_{3}^{f a c i a l}}

(1)

p_{j}^{p o s e} = {p_{0}^{p o s e} 、 p_{1}^{p o s e} 、 p_{2}^{p o s e} 、 p_{3}^{p o s e}}

(2)

The output of the recognition results is given different weights, represented as

p = x_{m} \times p^{f a c i a l} + x_{n} \times p^{p o s e}, x_{m} + x_{n} = 1,0 < x_{m} < 1,0 < x_{n} < 1.

(3)

p_{i}^{f a c i a l}

represents the output of four affections in the facial expression recognition model.

p_{0}^{f a c i a l}

means happy,

p_{1}^{f a c i a l}

means confused,

p_{2}^{f a c i a l}

means calm and

p_{3}^{f a c i a l}

means boredom.

p_{j}^{p o s e}

represents the output of multiple affections in the head pose recognition model,

p_{0}^{p o s e}

means happy,

p_{1}^{p o s e}

means confused,

p_{2}^{p o s e}

means calm,

p_{3}^{p o s e}

means boredom and

x_{m}

representss the weight given to the facial expression recognition result.

x_{n}

represents the weight provided to the head pose recognition result.

p

means the recognition accuracy of the dual-source data fusion.

Decision Level Fusion Process

The decision-level fusion approach syncretizes learners’ facial expression and head pose data (Figure 5). Firstly, the facial expression and head pose data are inputted into the corresponding affective computing model. Secondly, through model training, the recognition output results of each single-source data of facial expression and head pose are obtained. Finally, the decision-fusion approach syncretizes the facial expression and head pose, and affective computing models’ results are considered the final output result.

Figure 5.

Decision level fusion process of dual-source data.

Experiment

Dataset

Datasets play an essential role in the application of AC algorithms, but they may suffer from apparent biases caused by different cultures and collection conditions (Li & Deng, 2022). At present, affection data sets about middle school students for online learning are lacking. Thus, we constructed middle school students’ dual-source affection data sets to support deep learning model training and AC research.

Participants

36 middle school students, including 19 males and 17 females, were recruited in this experiment in Hangzhou, the East city of China The middle school is an affiliated school to the University where one of the authors of this study working for. One of the research members in this study was assigned to take off-campus internship there during the pandemic period. The facial expression and head pose data set were collected while students were participating in online learning. All students recruited were aware of the research aim and experimental procedure; all participants signed the consent confirm before participating in this experiment.

Data Acquisition and Screening

The physiological experiments are typically classified into invasive experiments and non-invasive experiments. To avoid learners’ distraction in online learning, web camera was used to collect facial expressions and head poses data as sources rather than invasive approach. Data acquisition shown in Figure 6. was carried out in the classrooms and meeting rooms of the middle school. The selected videos ranging from 6 to 10 minutes are popular science videos of different topics, with text introduction, explanation and some background music. The sample videos selected conform to two criteria to maximize learners’ attention during participating the online courses: the content is in the range of learners’ comprehensive ability, and the presentation is diversified and interesting, which help to inspire students' varied affections in the learning process. The data acquisition process is as follows: The staff guided the subjects into the experimental site. Participants filled in the basic information questionnaire and informed consent form. The participants conducted learning tasks online in a nature and quite learning environment, watched online videos, and filled in emotional questionnaires, and the research assistants recorded them. Assistants imported the recorded video files of the issues watching the teaching video into Elan software to assist the subjects in self-assessment of learning emotion. Assistants collected questionnaires and saved video data and students' self-evaluation data.

Figure 6.

Data collection in a case.

Data screening aims at video and image data, including machine screening and manual screening, screening steps are shown in Table 1. After data screening, we retained 9 hours, 54 minutes, and 08 seconds of the video data and 6649 picture data.

Table 1.

Screening Steps.

Process	Details
Filtering video data	Screening the videos containing middle school students’ facial expression and head pose manually. Deleting the videos with unclear images and wrong recordings
Capturing the subject image in the video	Intercepting the original video image that includes the subjects' facial expression and head pose at a node every 120 frames
Obtaining dual-source data image	Using OpenCV and HAAR feature-based face recognition classifier to obtain the subject’s facial expression and face image for reducing the interference of the environment
Deleting unqualified images	Deleting the images with blocked face, incomplete face image, unclear face and face deviation

Affection Judgment

Specific affections can frequently be perceived from facial expressions and head poses. The face is an affect display system, while the pose shows the person’s adaptive efforts regarding affect (Ekman & Friesen, 1967). The affection judgment of facial expression in this study was based on Ekman’s FACS (Facial Action Coding System) describing the relationship between inner affection and facial expression in detail. The FACS can express the regularity of facial muscle movement when human beings have the same affection, and is not affected by gender, age, race, education level and other factors. Additionally, learners’ varied affections could be detected by different head posture characteristics. The emotional judgment of head posture features is also based on FACS, focusing on the orientation of the face and the position of the cheeks (Trad et al., 2012).

Data Annotation

Data annotation is a process of processing data with the help of marking tools. The procedure of the data annotation in this study is classified labelling. The corresponding labels are selected from the established labels for labelling. This study has four learning affection labels: happy, confused, calm, and boredom. Data annotation is based on head pose images and facial expression images captured from recorded videos.

The data annotation in this study includes two parts: subject annotation and experimenter annotation. Firstly, the participants marked andpassed Elan6.0 (Sloetjes & Seibert, 2016) to evaluate their expressions and keep the corresponding affection labels. The purpose of selecting subject labeling is to provide more direct and accurate affection judgment criteria for experimental labeling personnel. Secondly, three experimenters annotated the image. The data annotation rules are as follows:

1. The image was determined to be a valid data label and was marked if the labels determined by the three annotators were the same.

2. The image was determined to be the data label determined by two annotators and was marked if the labels determined by two annotators were the same and that determined by the other annotator was different.

3. The image was determined as an invalid label, if the three annotators’ definitions were different.

Dataset Division

The data volume of facial expression and head pose data sets include 3939 images. The two data sets contained training, verification and test sets. The training set accounts for 60% of the total data, the verification set accounts for 20%, and the test set accounts for 20%. The facial expression data set (local) and the head pose data set (local) are shown in Figure 7. Although the head pose data and facial expression data are in the same emotional categories, the differences between head pose data and facial expression data in the modeling process are reflected in the content and form of data presentation: Firstly, the features of the data are different. The facial data focused on the features of facial organs, while the head pose data focus on the learners’ head skeleton. Furthermore, single data source may not be a viable judgment of supporting certain affections. For example, if the head pose is askew when facial expression is smiling, it may present a confused affection rather than a happy affection.

Figure 7.

The facial expression data set (local) and the head pose data set (local).

Model Training

The data set utilized in model training contained 3939 RGB facial expression image data with a resolution of 244 × 244 and 3939 head pose feature point image data with a resolution of 244 × 244. Each contains a training set, verification set, and test set, as shown in Table 2. To improve the generalization ability of the AC model, the adjustment of the parameter set includes optimizer, loss function, batch size and epoch. The parameter settings are as follows: The learning rate was set to 0.0001. β-1 was set to 0.9. The binary cross entropy was assigned as the loss function. The batch size is set to 128. The epoch is set at 70 (Nielsen, 2015).

Table 2.

Number of Datasets.

Dataset name	Total	Happy	Confused	Calm	Boredom
Training set	2363	415	556	735	657
Verification set	788	134	166	290	198
Test set	788	115	198	272	203
Total	3939	664	920	1297	1058

Firstly, model training needs to import the training data set into the AlexNet and Attention-AlexNet models and compute the model’s optimal weight matrix and a bias value. Secondly, it achieves the accuracy of the two neural network models on the validation data set. Furthermore, the progress of model training is determined by the output value of the loss function. The more visible the loss function’s output value is, the closer the model is to the truth. The model training is completed when the output value of the loss functions is zero.

When completing each training iteration of the model, the model was tested by the verification set. The accuracy of the validation set is the proportion of the correct prediction results to the total number of validation sets. The accuracy of the verification set can be employed as the evaluation criterion of the model, which with the highest accuracy will be the best model. In the end, the test set data are imported into the Attention-AlexNet model, and the prediction accuracy of the four types of affections is obtained. The accuracy of the test set is the proportion of the correct prediction results against the total number of test sets.

Results and Discussion

Results of Different Learning Affective Computing Models by Single-Source Data Set

After model training, the loss functions of facial expression and head pose data in the AlexNet, and Attention-AlexNet models converge and have a stable value, as shown in Figure 8. The accuracy of these models on the training set is more than 95%. The training set prediction accuracy and the loss function’s convergence show that these models have good performance and optimal overall prediction effect.

Figure 8.

Convergence of training set loss function of AlexNet model and Attention-AlexNet model based on single-source data.

The recognition accuracy results of the AlexNet model and Attention-AlexNet model based on single-source data for different learning affections in the verification set are shown in Figure 9. The facial expression model based on Attention-AlexNet has the best recognition effect on happy (88.70%) and confused (76.77%). The facial expression model based on AlexNet has the best recognition effect on calm (81.26%) and boredom (80.79%).

Figure 9.

The recognition accuracy of AlexNet model and Attention-AlexNet model for different learning affections based on single-source data set.

The results show that affective computing is effective based on single-source data and deep neural networks. Firstly, affective computing based on deep learning can implement the automated extraction of emotional features, simplify the data processing steps, decrease the access threshold, and enhance the algorithm’s efficiency. Secondly, CNN in deep learning has advantages in image data extraction, prediction, and large-scale image data processing. The CNN model based on attention mechanism can effectively improve the accuracy of learning affection recognition and optimize the affective computing model. Lastly, learning affective computing can also be identified based on the single-source data of facial expression and head pose, and the effect of using facial expression data is the best.

Results of Different Learning Affective Computing Models by Dual-Source Data Sets

Based on the weight rules, this study concludes the dual-source data fusion model’s results, which are the fusion result after giving weight to the facial expression model and head pose model. The results of different weights given to the overall output of the model are shown in Table 3.

Table 3.

The Results of the Fusion Model that are Giving Different Weights.

Fusion model type	$x_{m}$ (Facial expression weight)	$x_{n}$ (Head pose weight)	Data fusion accuracy (%)
AlexNet	0.1	0.9	65.48
	0.2	0.8	65.48
	0.3	0.7	66.37
	0.4	0.6	68.78
	0.5	0.5	77.53
	0.6	0.4	79.94
	0.7	0.3	80.20
	0.8	0.2	79.44
	0.9	0.1	78.93
Attention-AlexNet	0.1	0.9	64.46
	0.2	0.8	65.10
	0.3	0.7	66.11
	0.4	0.6	67.38
	0.5	0.5	74.49
	0.6	0.4	80.96
	0.7	0.3	80.71
	0.8	0.2	80.58
	0.9	0.1	80.32

From Table 3, it is found that when $x_{m}$ = 0.7, $x_{n}$ = 0.3, the accuracy of the dual-source data fusion model based on AlexNet is the highest at 80.20%. When $x_{m}$ = 0.6, $x_{n}$ = 0.4, the accuracy of the dual-source data fusion model based on Attention-AlexNet is the highest at 80.96%.

The dual-source data fusion model showed a better learning AC effect than the single-source data model in this study. On the one hand, learning affective computing based on dual-source data is a practical recognition approach that can effectively increase the accuracy of online learning affective computing. On the other hand, with the increasing application scenarios of AC, the problems to be solved become more in-depth and detailed. The available open-source data sets supporting AC can no longer meet the needs of current education scenarios and normalized applications. Constructing multi-source data sets that satisfy specific educational approaches’ needs will become a meaningful way to solve the challenge of personalized AC.

Optimal Model and Test Set Results

As is shown in Table 4, the dual-source data fusion model based on Attention-AlexNet reached the highest accuracy rate of 80.96% among six comparative models. Thus, compared with AlexNet, the Attention-AlexNet model has a better recognition effect. The mutual confirmation of dual-source data surpasses the single verification of facial expression data.

Table 4.

The Accuracy of Different Models for Affection Recognition.

Model type	Data type	Accuracy (%)
AlexNet	Single source(facial expression)	76.65
Attention-AlexNet	Single source (facial expression)	78.05
AlexNet	Single source (head pose)	64.34
Attention-AlexNet	Single source (head pose)	63.83
Decision level fusion (AlexNet)	Dual source (facial expression and head pose)	80.20
Decision level fusion (Attention-AlexNet)	Dual source (facial expression and head pose)	80.96

Figure 10 is the different affections’ confusion matrix, which shows the recognition results of learning affections of the optimal model with the test set as the data set. The abscissa of the matrix represents the predicted classification results, and the ordinate of the matrix represents the accurate classification results. The coordinates include happy, calm, boredom and confused. From the confusion matrix diagram’s colour depth and value, the model correctly identifies the number of calm affections up to 218. The number of correctly identifying happy affections is 99. The number of correctly identified boredom affections is 166. The number of correctly identified confused affections is 152.

Figure 10.

The confusion matrix of different learning affections of the optimal model.

From the the optimal model’ confusion matrix, we can compute the prediction accuracy of the four types affections, as shown in Table 5. Table 5 shows that among the optimal learning AC models, the recognition effect of happy affection is the best, with an accuracy of 86.09%, followed by boredom, calm and confused affections. The accuracy was 81.77%, 80.15%, and 76.77%, respectively.

Table 5.

The Recognition Accuracy of the Four Types Affection From the Optimal Model.

Affection type	Happy	Boredom	Calm	Confused
Accuracy (%)	86.09	81.77	80.15	76.77

From the results of the optimal model, the recognition effect of happy and boredom learning affections was more satisfactory. The possible reason is that the happy expression features are more outstanding, representing curved eyebrows, raised cheeks and exposed teeth. The head pose features an elongated jaw and mouth, which are easy to recognize. Boredom is mostly eyelid closure, lip up or down, and facial features and mostly head tilt and offset, are easy to identify correctly. The recognition accuracy of calm and confusion is relatively low. The possible reason is that their facial expression and head pose features are not prominent.

The above results show that: the deep learning AC model of dual-source data fusion has higher accuracy than single-source data, and the effect of the dual-source data fusion model based on Attention-AlexNet is the best. The AlexNet model with an increased attention mechanism has a better recognition effect than the AlexNet model. The optimal AC model (Attention-AlexNet) has the most satisfactory recognition effect on happy, followed by boredom, calm and confusion affection. Based on these findings, educational researchers can optimize the neural network model from the perspective of the integration of diverse data and attention mechanisms to obtain better algorithm’s efficiency. Furthermore, the developed dual-source AC model may be conducive to providing affective care to students, bringing a decent flow state for them, and further decreasing the dropout rate of online learning (Semerci & Goularas, 2021).

Conclusion and Limitations

This study contributed to developing CNN with the dual-source data set and fusion model. Based on establishing a dual-source emotion data set in online learning settings, the current research constructed the neural network deep learning model combined with attention mechanism, which significantly increased the accuracy rate for online learners’ AC. However, some limitations still existed in this study. Firstly, the amount of data collected in this study is relatively small in deep learning research. In the future, we will expand the range of research objects, such as college students and learners in special education, by which we will establish a relatively holistic online learning affection data set. Secondly, the data fused in this study only includes facial expression and head pose; more source data generated from learners, for example, EEG, voice, and eye-tracking are expected to be captured. In addition, the limited number of sample size was a practical constraint due to COVID-19 pandemic, therefore the data set is from one school inone country which may lead to cultural bias of this research. However, we will expand our data set from different countries and regions to reduce cultural bias and further validate the applicability of this approach in the future.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the National Natural Science Foundation of China (62177042; 6217021982); Humanity and Social Science Funding of China’ Minister of Education (20YJC880118); The 2021 key lab funding of Beihang University of China (VRLAB2021D10; The 2021 key lab funding of Anhui Jianzhu University of China (IBES2020KF02).

Ethical Approval

Before the study commenced, the authors had applied for ethics approval and get the approved. The consent forms were sent to all participants. The data collection approaches were approved by all participants; participants' facial images are processed with mosaic to avoid infringement of portrait rights.

ORCID iDs

Nian-Shing Chen

Yonggu Wang

Author Biographies

Xuesong Zhai, PhD, is a distinguished researcher of Educational Technology in the College of Education at Zhejiang University. His research interests are included but not limited to the smart learning environments, educational information system, and artificial intelligence in education. He is the Area Editor of a EAI Transactions on e-Learning and Associate editor of International Journal of Global Lifelong Learning.

Jiaqi Xu is a doctoral candidate at College of Education, Zhejiang University. His research interests are artificial intelligence in education, affective computing and online learning.

Nian-Shing Chen, PhD, is a Chair Professor in the Institute for Research Excellence in Learning Sciences and Program of Learning Sciences at the National Taiwan Normal University. His current research interests include game-based learning, gesture-based learning and educational robotics. He has organised and chaired numerous international conferences and workshops in the area of advanced learning technologies. He is a golden core member of IEEE, ACM and the former Chair of the IEEE Technical Committee on Learning Technology.

Jun Shen, PhD, is a associated professor at the University of Wollongong in Wollongong, NSW Australia. His expertise is on e-earning and m-learning, Web services and semantic Web. He has been editor, associate editor, pc chair, guest editor, pc member for numerous journals and conferences published by IEEE, ACM, Elsevier, and Springer. He was a chair of Education Chapter of IEEE NSW section since 2007. He is a senior member of both the IEEE and ACM.

Yan Li, PhD, is a professor of Educational Technology in the College of Education at Zhejiang University. Her research interests are e-learning, distance education, ICT education, STEM education, media education and diffusion of educational innovation.

Yonggu Wang, PhD, is a professor of Educational Technology in the College of Educational Science and Technology at Zhejiang University of Technology. His research interests are learning science and technology, intelligent education, vocational education informatizavvvvtion.

Xiaoyan Chu is a master candidateat College of Education, Zhejiang University. Her research interests are online and blended learning, artificial intelligence in education, learning sciences and technology.

Yumeng Zhu is a doctoral candidate at College of Education, Zhejiang University. Her research interests are artificial intelligence in education, learning analytics and educational data mining, and assessment of education education informationization.

References

Abisado M.

De Los Trinos

M I

L. Rodriguez, R.

V Arias Jr

A. R.

(2020). An academic affect dataset: spontaneous facial expressions and head poses collected during online examination. In 2020 The 4th International Conference on E-Society, E-Education and E-Technology (pp. 83–87). IEEE. https://doi.org/10.1145/3421682.3421692

Adams

Mahmoud

Baltrušaitis

Robinson

(2015). Decoupling facial expressions and head motions in complex emotions. In International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 274–280). IEEE. https://doi.org/10.1109/ACII.2015.7344583

Aydoğdu

Ş.

(2021). A new student modeling technique with convolutional neural networks: LearnerPrints. Journal of Educational Computing Research, 59(4), 603–619. https://doi.org/10.1177/0735633120969216

Bairavel

Krishnamurthy

(2020). Novel OGBEE-based feature selection and feature-level fusion with MLP neural network for social media multimodal sentiment analysis. Soft Computing, 24(24), 18431–18445. https://doi.org/10.1007/s00500-020-05049-6

Bandara

Song

Hirshfield

Velipasalar

(2016). A more complete picture of emotion using electrocardiogram and electrodermal activity to complement cognitive data. In International Conference on Augmented Cognition (pp. 287–298). Springer. https://doi.org/10.1007/978-3-319-39955-3_27

Barrett

L. F.

Adolphs

Marsella

Martinez

A. M.

Pollak

S. D.

(2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest, 20(1), 1–68. https://doi.org/10.1177/1529100619832930

Cimtay

Ekmekcioglu

Caglar-Ozhan

(2020). Cross-subject multimodal emotion recognition based on hybrid fusion. IEEE Access, 8(2020), 168865–168878. https://doi.org/10.1109/ACCESS.2020.3023871

Demircan

Örnek

H. K.

(2020). Comparison of the effects of mel coefficients and spectrogram images via deep learning in emotion classification. Traitement du Signal, 37(1), 51–57. https://doi.org/10.18280/ts.370107

Ekman

Friesen

W. V.

(1967). Head and body cues in the judgment of emotion: A reformulation. Perceptual and Motor Skills, 24(3), 711–724. https://doi.org/10.2466/pms.1967.24.3.711

10.

Faria

A. R.

Almeida

Martins

Gonçalves

Martins

Branco

(2017). A global perspective on an emotional learning model proposal. Telematics and Informatics, 34(6), 824–837. https://doi.org/10.1016/j.tele.2016.08.007

11.

Fouladgar

Alirezaie

Främling

(2021). CN-Waterfall: A deep convolutional neural network for multimodal physiological affect detection. Neural Computing and Applications, 34(3), 1–20. https://doi.org/10.1007/s00521-021-06516-3

12.

Gan

(2018). Facial expression recognition using convolutional neural network. In Proceedings of the 2nd international conference on vision, image and signal processing (pp. 1–5). https://doi.org/10.1145/3271553.3271584

13.

Giannopoulos

Perikos

Hatzilygeroudis

(2018). Deep learning approaches for facial emotion recognition: A case study on FER-2013. In Advances in hybridization of intelligent methods (pp. 1–16). https://doi.org/10.1007/978-3-319-66790-4_1

14.

Gokberk

Akarun

(2006, August). Comparative analysis of decision-level fusion algorithms for 3D face recognition. In 18th International Conference on Pattern Recognition (ICPR’06) (Vol. 3, pp. 1018–1021). https://doi.org/10.1109/ICPR.2006.387

15.

Guo

M. H.

T. X.

Liu

J. J.

Liu

Z. N.

Jiang

P. T.

T. J.

Zhang

S.-H.

Martin

R. R.

Cheng

M.-M.

S. M.

(2022). Attention mechanisms in computer vision: A survey (pp. 1–38). Computational Visual Media. https://doi.org/10.1007/s41095-022-0271-y

16.

Han

Z. M.

Huang

C. Q.

J. H.

Tsai

C. C.

(2021). Identifying patterns of epistemic emotions with respect to interactions in massive online open courses using deep learning and social network analysis. Computers in Human Behavior, 122(2021), 106843. https://doi.org/10.1016/j.chb.2021.106843

17.

Hassan

M. M.

Alam

M. G. R.

Uddin

M. Z.

Huda

Almogren

Fortino

(2019). Human emotion recognition using deep belief network architecture. Information Fusion, 51(2019), 10–18. https://doi.org/10.1016/j.inffus.2018.10.009

18.

Hess

Adams

R. B.

Kleck

R. E.

(2007). Looking at you or looking elsewhere: The influence of head orientation on the signal value of emotional facial expressions. Motivation and Emotion, 31(2), 137–144. https://doi.org/10.1007/s11031-007-9057-x

19.

Hou

Wang

Chen

Abraham

(2021). Attention-based learning of self-media data for marketing intention detection. Engineering Applications of Artificial Intelligence, 98(2021), 104118. https://doi.org/10.1016/j.engappai.2020.104118

20.

Hsu

Y. L.

Wang

J. S.

Chiang

W. C.

Hung

C. H.

(2017). Automatic ECG-based emotion recognition in music listening. IEEE Transactions on Affective Computing, 11(1), 85–99. https://doi.org/10.1109/TAFFC.2017.2781732

21.

Huang

Yang

Liao

Pan

(2017). Fusion of facial expressions and EEG for multimodal emotion recognition. Computational Intelligence and Neuroscience, 2017(2), 1–8. https://doi.org/10.1155/2017/2107451

22.

Jiang

Hossain

M. S.

Chen

Alelaiwi

Al-Hammadi

(2020). A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition. Information Fusion, 53, 209–221. https://doi.org/10.1016/j.inffus.2019.06.019

23.

Jiménez

Juarez-Ramirez

Castillo

V. H.

Ramírez-Noriega

(2018). Integrating affective learning into intelligent tutoring systems. Universal Access in the Information Society, 17(4), 679–692. https://doi.org/10.1007/s10209-017-0524-1

24.

Kartali

Roglic

Barjaktarovic

Duric-Jovicic

Jankovic

M. M.

(2018). Real-time algorithms for facial emotion recognition: A comparison of different approaches. In 2018 14th Symposium on Neural Networks and Applications (NEUREL) (pp. 1–4). https://doi.org/10.1109/NEUREL.2018.8587011

25.

Kaur

Kumar

Kaushal

(2021). Affective state and learning environment based analysis of students’ performance in online assessment. International Journal of Cognitive Computing in Engineering, 2(2021), 12–20. https://doi.org/10.1016/j.ijcce.2020.12.00

26.

Khare

S. K.

Bajaj

(2020). Time–frequency representation and convolutional neural network-based emotion recognition. IEEE Transactions on Neural Networks and Learning Systems, 32(7), 2901–2909. https://doi.org/10.1109/TNNLS.2020.3008938

27.

Krizhevsky

Sutskever

Hinton

G. E.

(2012). Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25(6), 1097–1105. https://dx-doi-org.web.bisu.edu.cn/10.1145/3065386

28.

Kumar

(2021). Deep learning based affective computing. Journal of Enterprise Information Management, 34(5), 1551–1575. https://doi.org/10.1108/JEIM-12-2020-0536

29.

Ren

Jiang

Miao

Feng

Jin

(2021). SCEP—a new image dimensional emotion recognition model based on spatial and channel-wise attention mechanisms. IEEE Access, 9(2021), 25278–25290. https://doi.org/10.1109/ACCESS.2021.3057373

30.

Wang

Liu

Chi

Dong

Song

Gao

Qiang.

Song

(2019). The fusion of electroencephalography and facial expression for continuous emotion recognition. IEEE Access, 7(2019), 155724–155736. https://doi.org/10.1109/access.2019.2949707

31.

Chen

Zhao

(2021). Emotion recognition of Chinese paintings at the thirteenth national exhibition of fines arts in China based on advanced affective computing. Frontiers in Psychology, 12(2021), 741665. https://doi.org/10.3389/fpsyg.2021.741665

32.

Jin

Zhou

Kubota

(2020). Attention mechanism-based CNN for facial expression recognition. Neurocomputing, 411, 340–350. https://doi.org/10.1016/j.neucom.2020.06.014

33.

Zhao

Yao

(2018). Multi-modal emotion recognition based on speech and image. In Advances in Multimedia Information Processing – PCM 2017 (pp. 844–853). https://doi.org/10.1007/978-3-319-77380-3_81

34.

Zeng

Shan

Chen

(2018). Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE Transactions on Image Processing, 28(5), 2439–2450. https://doi.org/10.1109/TIP.2018.2886767

35.

Lin

Sun

Shen

Pritchard

Cui

Beydoun

Chen

(2020). Deep-cross-attention recommendation model for knowledge sharing micro learning service. In International conference on artificial intelligence in education (pp. 168–173). Springer. https://doi.org/10.1007/978-3-030-52239-1_31

36.

Liu

Xing

Zeng

(2021). Quantifying the influence of achievement emotions for student learning in MOOCs. Journal of Educational Computing Research, 59(3), 429–452. https://doi.org/10.1177/0735633120967318

37.

Liu

Zhang

Kong

(2021). Review on emotion recognition based on electroencephalography. Frontiers in Computational Neuroscience, 15(2021), 758212. https://doi.org/10.3389/fncom.2021.758212

38.

Liu

Cao

Chen

Zhang

Zhou

Mao

(2017). A facial expression emotion recognition based human-robot interaction system. IEEE/CAA Journal of Automatica Sinica, 4(4), 668–676. https://doi.org/10.1109/JAS.2017.7510622

39.

Liu

Yang

Rüdian

Liu

Zhao

Wang

(2019). Temporal emotion-aspect modeling for discovering what students are concerned about in online course forums. Interactive Learning Environments, 27(5-6), 598–627. https://doi.org/10.1080/10494820.2019.1610449

40.

Lockee

B. B.

(2021). Online education in the post-COVID era. Nature Electronics, 4(1), 5–6. https://doi.org/10.1038/s41928-020-00534-0

41.

Mheidly

Fares

M. Y.

Fares

(2020). Coping with stress and burnout associated with telecommunication and online learning. Front Public Health, 8(2020), 574969. https://doi.org/10.3389/fpubh.2020.574969

42.

Nielsen

M. A.

(2015) Neural networks and deep learning (Vol. 25). Determination Press.

43.

Pardos

Z. A.

Baker

R. S.

San Pedro

M. O.

Gowda

S. M.

Gowda

S. M.

(2013). Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In Proceedings of the third international conference on learning analytics and knowledge (pp. 117–124). https://doi.org/10.1145/2460296.2460320

44.

Poria

Cambria

Hussain

Huang

G. B.

(2015). Towards an intelligent framework for multimodal affective data analysis. Neural Networks, 63, 104–116. https://doi.org/10.1016/j.neunet.2014.10.005

45.

Sabour

Frosst

Hinton

G. E.

(2017). Dynamic routing between capsules. Advances in Neural Information Processing Systems, 30(2017), 3859–3869. https://doi.org/10.48550/arXiv.1710.09829

46.

Salama

E. S.

El-Khoribi

R. A.

Shoman

M. E.

Shalaby

M. A. W.

(2021). A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition. Egyptian Informatics Journal, 22(2), 167–176. https://doi.org/10.1016/j.eij.2020.07.005

47.

Semerci

Y. C.

Goularas

(2021). Evaluation of students' flow state in an E-learning environment through activity and performance using deep learning techniques. Journal of Educational Computing Research, 59(5), 960–987. https://doi.org/10.1177/0735633120979836

48.

Shaees

Naeem

Arslan

Naeem

M. R.

Ali

S. H.

Aldabbas

(2020). Facial emotion recognition using transfer learning. In 2020 International Conference on Computing and Information Technology (ICCIT-1441) (pp. 1–5). https://doi.org/10.1109/ICCIT-144147971.2020.9213757

49.

Shu

Xie

Yang

Liao

Yang

(2018). A review of emotion recognition using physiological signals. Sensors Electronic Resource, 18(7), 2074. https://doi.org/10.3390/s18072074

50.

Sloetjes

Seibert

(2016). Measuring by marking; the multimedia annotation tool ELAN. In A. Spink, G. Riedel, L. Zhou, L. Teekens, R. Albatal, & C. Gurrin (Eds.), Measuring Behavior 2016, 10th International Conference on Methods and Techniques in Behavioral Research 492–495.

51.

Stephens-Fripp

Naghdy

Stirling

Naghdy

(2017). Automatic affect perception based on body gait and posture: A survey. International Journal of Social Robotics, 9(5), 617–641. https://doi.org/10.1007/s12369-017-0427-6

52.

Stolar

M. N.

Lech

Bolia

R. S.

Skinner

(2017). Real time speech emotion recognition using RGB image classification and transfer learning. In 2017 11th International Conference on Signal Processing and Communication Systems (ICSPCS) (pp. 1–8). https://doi.org/10.1109/ICSPCS.2017.8270472

53.

Trad

Hajj

El-Hajj

Al-Jamil

(2012). Facial action unit and emotion recognition with head pose variations. In International Conference on Advanced Data Mining and Applications (pp. 383–394). https://doi.org/10.1007/978-3-642-35527-1_32

54.

Tzafilkou

Perifanou

Economides

A. A.

(2021). Negative emotions, cognitive load, acceptance, and self-perceived learning outcome in emergency remote education during COVID-19. Educ Inf Technol (Dordr), 26(6), 1–25. https://doi.org/10.1007/s10639-021-10604-1

55.

Valstar

M. F.

Sánchez-Lozano

Cohn

J. F.

Jeni

L. A.

Girard

J. M.

Zhang

Yin

Pantic

(2017). Addressing Head Pose in the Third Facial Expression Recognition and Analysis Challenge. Proceedings of the International Conference on Automatic Face and Gesture Recognition. IEEE International Conference on Automatic Face & Gesture Recognition 2017, 839–847 https://doi.org/10.1109/FG.2017.107

56.

Wang

Jiang

Qian

Yang

Zhang

Wang

Tang

(2017). Residual attention network for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3156–3164).

57.

Wang

Peng

Yang

Meng

Qiao

(2020). Region attention networks for pose and occlusion robust facial expression recognition. IEEE Transactions on Image Processing, 29(2020), 4057–4069. https://doi.org/10.1109/TIP.2019.2956143

58.

Wei

H. C.

Chou

(2020). Online learning performance and satisfaction: Do perceptions and readiness matter? Distance Education, 41(1), 48–69. https://doi.org/10.1080/01587919.2020.1724768

59.

Woo

Park

Lee

J. Y.

Kweon

I. S.

(2018). Cbam: Convolutional block attention module (pp. 3–19). Proceedings of the European conference on computer vision (ECCV).

60.

Y. Y.

Shi

Q. W.

Dai

(2021). Behavior recognition algorithm based on the fusion of SE-R3D and LSTM network. IEEE Access, 9(2021), 141002–141012. https://doi.org/10.1109/ACCESS.2021.3119609

61.

Huang

Wang

(2019). Continuous emotion recognition in videos by fusing facial expression, head pose and eye gaze. In 2019 International Conference on Multimodal Interaction (pp. 40–48). https://doi.org/10.1145/3340555.3353739

62.

Cai

Meng

(2006). Multi-level fusion of audio and visual features for speaker identification. In International Conference on Biometrics (pp. 493–499). Springer. https://doi.org/10.1007/11608288_66

63.

Fan

(2014). Emotion management in online groupwork reported by Chinese students. Educational Technology Research and Development, 62(6), 795–819. https://doi.org/10.1007/s11423-014-9359-0

64.

Yin

Schütze

Xiang

Zhou

(2016). Abcnn: Attention-based convolutional neural network for modeling sentence pairs. Transactions of the Association for Computational Linguistics, 4(2016), 259–272. https://doi.org/10.1162/tacl_a_00097

65.

Yuan

Z. W.

Zhang

(2016). Feature extraction and image retrieval based on AlexNet. In Eighth International Conference on Digital Image Processing (ICDIP 2016), 10033(2016), 65–69. SPIE. https://doi.org/10.1117/12.2243849

66.

Zhang

Liu

Cao

Liu

(2020). Data-driven online learning engagement detection via facial expression and mouse behavior recognition technology. Journal of Educational Computing Research, 58(1), 63–86. https://doi.org/10.1177/0735633119825575