Learning Analytics Based on Wearable Devices: A Systematic Literature Review From 2011 to 2021

What?

There are a variety of wearable devices available, such as smart watch, finger pulse oximeter, mind-wave headsets, smart glasses, and chest patch. Based on the part of the body where the wearable devices are applied, it can be divided into three main types: head-mounted, wrist-worn, and chest-strap. In the reviewed literature, wristband devices were the most frequently used type, accounting for 64% of the total. Moreover, the data collected was largely from wrist-worn devices. These devices are lightweight, portable, and integrated perfectly with students’ gears. Generally, they are also continuously accessible to users, enabling students and teachers to access information and receive notifications promptly, unobtrusively, whenever, and wherever they go. Approximately 34% of the studies used head-mounted devices. Head-mounted devices are relatively inflexible and less convenient, but they are essential for collecting EEG data and also enhance the student learning experience from a visual perspective. In addition, about 10% of the papers used data from chest-strap wearable devices. In general, heart rate monitoring results from chest-strap devices are more reliable as they are closer to the heart. However, they are less used because they are not as convenient and comfortable as wristband devices. Note that some of the reviewed articles used multiple types of wearable devices, resulting in a final distribution sum greater than 100%, with the same situation occurring later.

The following images are examples of the three types of wearable devices. The wristband (Figure 3(a)) collects data from the skin surface, which contains a photo plethysmography (PPG) sensor to measure blood volume pulses. NeuroSky’s brainwave sensing headset (Figure 3(b)) effectively measures electrical activities released during firing of brain neurons using EEG sensors. Based on the shift of neuron firing rate, the student’s mental states underlying the EEG data can be explored and studied in depth. As illustrated in Figure 3 (c), BITalino Kit with various pluggable sensors (ECG、EMG、EEG、EDA and ACC) can be used in various locations on the body, including chest and collarbone.OPEN IN VIEWER

Adequate data is an important prerequisite for learning analytics. However, without data information related to learning status, students cannot receive personalized instruction. Wearable devices embedded with various types of sensors can capture a variety of information (e.g., physical activity information and physiological signal information) during the learning process, broadening the scope of educational data. Specifically, sensors can be mainly divided into three types such as activity sensors, biosensors, and environmental sensors (Hendrik et al., 2012). In order to obtain amounts of quality data in the actual teaching process, it is very important to adopt non-invasive and low-cost sensors for learning analytics (see Table 1). Wearable activity sensor is a tiny component placed in various parts of the human body. It can accurately capture and transmit information about the wearer’s motion, such as acceleration, angular velocity, etc. The main function of the motion sensor is to facilitate human activity recognition, which is widely used in capturing exercise status in sports, learning status in outdoor learning activities, and student behaviors in classrooms. Biosensors allow non-invasive measurements of physiological characteristics of the human body, such as ECG, heart rate, respiration rate, and EEG. It is mostly used to monitor students’ mental health and analyze their learning status. Environmental sensors enable capturing scene information about the learning environment and can help monitor changes in the surroundings to adaptively improve learning conditions. The categories and functions of sensors are shown in Table 1.

OPEN IN VIEWER

Table 1.

Categories and Functions of Sensors.

Sensor Type	Specific Sensors	Measurement Data	Used for
Activity sensor	Accelerometer/G-sensor	Orientation of the device, acceleration, and velocity	Logging learning activities, behaviors and pacing etc.; VR applications; Gamification; Pressure seats
	Gyroscope	Angular velocity, rotation of the device
	Magnetometer/M-sensor	Magnetic declination: The change of earth’s magnetic field
	Load gauge	Strength level
	Pedometer	Step counter
Bio-sensor	Galvanic skin response (GSR) sensors	GSR data	Application in various wearable devices; real-time monitoring of students’ physiological signal data.; monitoring and prediction of learning states, such as emotions, class participation, and attention etc.
	Body temperature sensor	SKT: Skin temperature
	Photoplethysmography (PPG) sensor	PPG data
	Biological pulse sensor	Heart rate (HR) data
	Electrocardiogram (ECG) sensor	ECG data
	Electromyography (EMG) sensor	EMG data
	Electroencephalogram (EEG) sensor	EEG data
	Electrodermal activity (EDA) sensor	EDA data
	Sweat sensor	PH data
	Oxygen saturation sensor	SpO2 data
	Blood volume pulses sensor	BVP data
	Eye-tracking sensor	Blinking and pupil focusing data
Environmental sensor	Location GPS	Determine the exact location	Identify student location and physical activity behavior
	Barometer	The air pressure being exerted on device

The abundance of built-in sensors offers great potentials for ubiquitous learning analytics through wearable devices. There are two main categories of wearable data: physiological and movement data. The analysis and training of physiological data, like HR, ECG, and EEG, allows for real-time monitoring of students’ physiological status (Antoniou et al., 2020; Ahonen et al., 2018; Aritzeta et al., 2017; Lew & Tang, 2017; Lin & Kao, 2018). On the other hand, the exploration and study of movement data, such as gyroscope and tri-axial accelerometer data, can help physical education teachers to accurately grasp students’ movement levels and provide appropriate guidance (Liang et al., 2021; Li et al., 2019; Lee et al., 2016; Frömel et al., 2021; Esakia & Kotut, 2020). Figure 4 represents the classification of the reviewing articles depending on the types of wearable data. On many occasions, multiple data are used in combination to achieve the research objectives. Movement data were often used together, so no detailed distinction is made here.OPEN IN VIEWER

Heart rate (HR) and heart rate variability (HRV) are the most widely utilized data in the reviewed literature (approx. 36%) (Cowley et al., 2013; Millings et al., 2015; Yun et al., 2017; Lin & Kao, 2018). Heart rate indicates the number of heart beats per minute, which may vary according to the needs of the body as well as the state of the physiology. Heart rate variability refers to the temporal variation of the heartbeat cycle, which can be calculated from heart rate signals. Notably, HR and HRV are often used in conjunction with other physiological data to monitor physiological states or predict learning performance. Electrodermal activity (EDA), aka galvanic skin response (GSR), ranks second place (31%) (Khan et al., 2019; Moukayed et al., 2018; Thammasan et al., 2020; Hernandez et al., 2014). Electrodermal activity (EDA) describes the subtle variations in skin electrical activity. Based on traditional EDA theory, the skin resistance changes in response to sweat gland status. At the same time, sweating is under the control of sympathetic nervous system. In this way, skin conductance can serve as an indicator of emotions and sympathetic responses (i.e., psychological or physiological arousal). EDA is now also increasingly used to measure other metrics. For example, Zhang et al. (2018) used wristbands to record physiological data from middle-school students in the classroom and showed a significant correlation between academic performance (finals grade) and EDA response. An additional regression analysis demonstrated that transient EDA response was a significant predictor of academic performance.

EEG data has been used in 20% of the studies. An electroencephalogram (EEG) is a recording of brain activity. The EEG signal is usually divided into five frequency bands in the frequency domain, and the different frequency bands can reflect the different activity states of the brain (Mostow et al., 2011; Chaouachi et al., 2011; Gal´an & Beal, 2012; Chang et al., 2019). ECG is a simple test that uses a sensor attached to one’ s skin for detecting the electrical signals generated when the heart beats, which is the data used in less than 10% of the researches. In addition to the monitoring of ECG data, heart rate and other characteristics can also be calculated from the ECG for further study (Ahonen et al., 2018; Kanna et al., 2018; Pourmohammadi & Maleki, 2020; Sanchez et al., 2020). Blood volume pulse (BVP) is another method of measuring heart rate, based on the amount of blood passing through the local area tissue with each beat (pulse) of the heart (Gao et al., 2020; Pijeira-Díaz et al., 2016).

Movement data is another common type of data (about 30% of the reviewed papers), mainly from physical education curriculum and classroom activities. Wearable devices can be used in physical education to monitor students’ basic physical functions, movement speed, intensity, and movements, enabling targeted instruction. In addition, skin temperature (SKT) and others (e.g., calorie, SpO2, and EMG) are often used in conjunction with other physiological data to improve the reliability of studies, especially in the context of academic anxiety detection (Calderón, 2016; Zhou et al., 2019; Klamma et al., 2019; Lai et al., 2019; Li & Sano, 2020).

In addition to the wearable data discussed above, there are several other types of data. For example, survey and interview (7.9%) data were used to obtain learners’ demographic characteristics, behavioral motivations, and attitudes toward wearable devices (Zhou et al., 2020; Mangaroska et al., 2021; Coskun & Cagiltay, 2021). These data can also help teachers to understand the role of wearables in curriculum instruction and pedagogical innovation, to study the pedagogical effects as well as the drawbacks of using wearables, and to make some further suggestions for improvement. Moreover, student self-reported data (e.g., academic emotions and self-efficacy, anxiety and isolation in the experiment, cognitive load, etc.) and academic performance data were typically used together to explore their mapping relations to student-generated physiological signal data to assist in multimodal learning analytics (Di Mitri et al., 2017; Ciolacu et al., 2019; Giannakos et al., 2020).

Who?

Regarding the stakeholders who directly benefited from the studies reviewed, Figure 5 shows the result. There are four main categories such as students, instructional staff, experts, and system designers. Of course, all the researchers involved are the most direct beneficiaries. A large portion of the research was conducted with students (about 65%), which may help other stakeholders as well (Muldner & Burleson, 2015; Egilmez et al., 2017; Nomura et al., 2019; Bolinski et al., 2021). By using wearable devices as an auxiliary tool, on the one hand, students can deepen their learning experience, enhance their interaction with learning materials, and increase their motivation. On the other hand, wearable devices provide timely access to students’ physiological data, which enables real-time monitoring and analyze of students’ learning status and helps to make timely adjustments and interventions for learning. In particular, small portions of these studies focused on students with special needs. Wearable technology bridges the communication gap between special students and normal students, and also helps them to integrate into society and participate in various activities in life and learning, providing them with a more inclusive educational environment (Díaz et al., 2016; Mehmood & Lee, 2017; Kalist et al., 2020; Nuguri et al., 2021). Instructional staff, including instructors, course designers, or teaching assistants, came in the second (35%) place to assist them in gaining insights about course design and instructional practices (Edwards et al., 2017; Khakurel et al., 2019; Dong & Li, 2021; Acevedo et al., 2021). Teachers can get a clearer picture of student performance and engagement in classroom activities; instructors can adjust teaching strategies based on student learning feedback; and wearable devices can also promote visualized content delivery for training purposes through offering immersive experiences. There are also approximately 17.5% that serve educational experts, policymakers, and institutions through investigations into how useful wearables can be in facilitating multimodal learning and smart classrooms (Arroyo et al., 2017; Ciolacu et al., 2019; Al-Emran, 2021). With the increasing integration of wearable devices and computer-aided education systems, system designers (including IoT experts and learning analytics experts) have emerged as another important stakeholder group. About 21% of the articles were designed to inform them of the current and potential system design patterns and elements that can help designers construct more humane and intelligent multimodal learning platforms (Ciolacu et al., 2018; Liang et al., 2019; Zhou et al., 2019; Vishkaie, 2020; López Camacho et al., 2020).OPEN IN VIEWER

Why?

Research into learning analytics based on wearable devices has different purposes, with an ultimate goal of facilitating student learning, faculty teaching, or technological advancement. The primary targets are either exploring and understanding the impact of wearable devices on learning, or developing a model which can automatically identify, monitor and predict learning status, and provide timely intervention. Thus, with such a model, it is possible to use wearable devices to maximize the benefits of learning analytics. About half of the literature reviewed were devoted to experimental exploration and the other half to modeling student learning data for diverse research goals. While some papers aspire both, for example, researchers have constructed different models to explore and discuss current mainstream technologies (Hernandez et al., 2014; Chen et al., 2017; Giannakos et al., 2020). A few papers (6%) reviewed other papers or presented a framework that laid the groundwork for data analysis (Henrie et al., 2015; Demir & Demir, 2018; Motti, 2019; de Arriba-Pérez et al., 2019). Figure 6 shows the reviewed literatures categorized according to their research objectives.OPEN IN VIEWER

Most of the research focused on exploration and understanding related to wearables and learning analytics. Some of these studies aim to explore the impact of using wearables on learning interest, student interaction, classroom motivation, and self-regulated learning (Di Mitri et al., 2017; Garcia et al., 2018; Liang et al., 2019; Darnell & Krieg, 2019). Also, a considerable number of review articles have explored the potential of wearables to personalize instruction, which could be of great help to teachers in improving course scheduling and promoting teaching effectiveness (Prieto et al., 2016, 2017; Calderón, 2016; Holstein et al., 2018). Certainly, there are scholars who have focused on student and teacher attitudes, needs, and recommendations for wearables to explore how to maximize the benefits of wearables (Kuo et al., 2017; Khakurel et al., 2019; Sato et al., 2021). In addition, researchers have explored the relationship between students’ physiological signals and research goals, as well as the importance of different physiological features (Harley et al., 2015; Edwards et al., 2017; Zhang et al., 2018; Oweis et al., 2018). Overall, exploratory research can provide researchers with a holistic view of wearable device learning analysis, which includes a range of components such as device selection, data processing and analysis, curriculum design, and discussion of results.

Monitoring refers to the continuous detection of students’ physiological data and movement data through wearable devices and the timely provision of notifications, alerts, and content delivery. Such type of research is commonly enhanced by visualization. Combining monitoring and modeling analysis allows for further predictions of student learning status, such as emotion and cognitive engagement (Chen & Wang, 2018; Darnell & Krieg, 2019; KONDO et al., 2020; Zhong & Li, 2020). Approximately 27% of the publications reviewed fell into this category. Prediction refers to the analysis of collected data to predict current classroom activities or learning status of students. Examples include building models that can identify students’ practical activities (active or inactive) by using heart rates and calories, with an excellent prediction accuracy of 95% (Zhou et al., 2019), predict positive or negative emotions to better assist distance learning based on academic emotions (Awais et al., 2020). These models help learners to be aware of their learning status in time for self-regulation. However, most predictive models are geared toward teachers to help them keep track of classroom realities, prioritize interventions, and increase instructional effectiveness.

Real-time monitoring and prediction of students serves as the prerequisite for precision interventions and personalized learning, which was the goal of fewer than 13% of the reviewed articles (Millings et al., 2015; Lu et al., 2017; Pijeira-Díaz et al., 2018; Wang et al., 2020). For instance, intervention strategies can be made once a student’s academic stress is detected. Recommendations are an automated form of intervention, which draw learners into content that may spark their interest or better match their current knowledge and skills. In addition, personalized settings also help tailor the learning experience specifically to each student based on their unique skills, abilities, status and experience to best match the learner’s preferences. For instance, learning content can be presented according to the participant’s current knowledge skills and learning status, which can reduce cognitive load and improve learning efficiency.

How?

There are four main methods used in the reviewed literature, namely, signal processing, statistical analysis, data mining, and visualization. Noting that some studies involve multiple methods at the same time.

Signal processing techniques are essential in physiological signal-based learning analytics tasks. First, the original physiological signal needs to be pre-processed to improve the quality of the data for an accurate modeling of learning status. Preprocessing generally includes down sampling, filtering, artifact removal, and feature extraction. Common physiological signal analysis methods include Independent Component Analysis (ICA), Power Spectral Density (PSD), Wavelet Analysis (WA), etc. From the perspective of traditional signal processing, time domain, frequency domain features, and non-linear features can be extracted from physiological signals, which can also be extracted by discrete wavelet transform. Rodríguez-Arce et al. (2020) used physiological features in constructing an anxiety and stress identification system. They used a mental arithmetic task to induce stress of students in an academic environment, with features extracted from five physiological data (including HR, SKT, GSR, BR, and SpO2), and finally achieved 95% accuracy in stress identification using an SVM classifier.

Statistical analysis is a common method of data analysis, which was applied in 44% of the reviewed papers. Descriptive statistics can dig out the statistical features of the data, such as percentage, mean, skewness and kurtosis etc. These features can give us an initial understanding of the data, while some of them can be used for subsequent prediction or classification. Statistics was also utilized to find relationships and closeness between variables, mainly including correlation analysis and regression analysis. In this category, the relationship between physiological data (e.g., HRV, ECG, and EEG) and measurement targets (e.g., academic performance, academic stress, and cognitive load) has been frequently explored. Furthermore, statistical analysis such as independent t-tests and analysis of variance (ANOVA) were applied in testing the proposed research hypotheses. For example, Chen and Wu (2015) utilized HRV data in exploring the effect of varied types of video lectures on individual learning emotions through two-way ANOVA with Scheffe test.

Data mining is the most popular method (about 49% of the reviewed literature). Data mining aims to discover hidden patterns and other valuable information in large data sets. Prediction and classification based on machine learning techniques are the most extensively applied methods in the category of data mining. Commonly used machine learning techniques include Naive Bayesian, regression models, support vector machines (SVM), k-Nearest Neighbor (KNN), and neural networks. Deep learning techniques like long short-term memory networks (LSTM) and convolutional neural networks (CNN) can also be used for classification and prediction, and another important function of it is the automatic extraction of data features. Fuzzy Logic, which has the ability to identify, represent, manipulate, interpret and exploit data and information that are vague and lack certainty, was applied to detecting academic emotions (Syaiful et al., 2019). Clustering is another technique that aggregates similar examples together to distinguish different sets. For example, Lehikoinen et al. (2019) clustered together adolescents with similar achievement emotions or psychophysiological reactions.

As a form of visual feedback, visualization techniques are relatively underused, accounting for only about 6%. It aims to display the process or outcome information to the target stakeholders in a visual and effective way using static or dynamic charts and graphics. Some researchers designed visual tools, such as dashboards or warning beacons, to maximize the efficiency of instructors in monitoring students’ classroom performance and learning status. In addition, some researchers explored the effectiveness of different visualizations of learning materials on motivating learner engagement and attention. For instance, Kim (2018) constructed a color fuzzy model that represents students’ participation in class through saturation. Depending on the students’ immersion in the classroom, the app on the teacher’s phone changed in real-time like a chameleon. It helps teacher monitor classroom conditions while providing timely feedback to students.

Cognitive domain

Cognition (Greeno et al., 1996) is “the mental action or process of acquiring knowledge and understanding through thought, experience, and the senses.” It covers numerous parts of the learning process, like engagement, attention, knowledge judgment, and memory. The cognitive process allows students to apply available knowledge and seek out additional knowledge, which is an essential part in the learning process. Thus, cognitive state detection is an important branch of learning analytics. The perception and assessment of learners’ cognitive state is a complex process. With the assistance of wearable technologies, learners’ cognitive state such as attention, classroom engagement, cognitive load, and creativity, can be measured and assessed objectively and effectively.

Attention refers to the behavior and cognitive process of selective attention to certain pieces of information (Anderson, 2005). In a previous study (Wu & Cheng, 2019), it turned out that students’ attention has a remarkable influence on their learning efficiency and academic outcomes. Therefore, it is beneficial for both students and teachers to understand the attention level of students in the classroom. Traditional methods used to measure the attention level of students are direct observation, laboratory tests, and questionnaires. While such approaches are appropriate in some cases, they are highly subjective, have a limited amount of data, and require human intervention. Among wearable technologies, the data obtained by traditional methods are generally used as an aid to measure attentional state. Chen and Wang (2018) developed a novel mechanism for attention detection and alerts based on EEG signals, which can help online instructors enhance the continuous attention level of inattentive students while performing online synchronous teaching. An experimental control with 83 and 65 seventh grade students verified the effectiveness of the mechanism. It was also found that sustained attention and attention alert frequency were strong predictors of academic performance. In addition to MindSet headsets, the wristband also has a major impact on attention monitoring. Spann et al. (2017) examined how heart rate variability (HRV) relates to attention and self-regulation, which were conducted in real-life settings. They studied 52 participants of multiple ages and learning stages who came to the museum. Their HRV data, attention, and self-regulation status were monitored during the task via Empatica E4 Wristband. Combining HRV and self-reported data for statistical analysis, and the preliminary results suggest a higher HRV is linked to objective measurements of adolescents’ attention and self-regulation. Poor internal consistency of the questionnaire and the small sample size limit any robust conclusions at that point.

Engagement is an important measure of attention, which indicates the level of mental effort students put into the learning process, as well as their willingness, need, and urge to learn in order to achieve academic success (Saeed & Zyngier, 2012). The most common method to measure engagement includes student self-report, particularly quantitative scales (Henrie et al., 2015). Some researchers have also explored student engagement by observing subtle learning behavior, such as note-taking or questioning. They stated that the degree of student engagement in learning tasks can be inferred from students’ self-presentation and behavior in public activities (Chi & Wylie, 2014). Wearable technology allows for the automatic external representation of such subjective feelings through bodily movements and physiological information. Camacho et al. (2020) tracked the learning trajectory of secondary school students with the help of wearable devices to promote their learning engagement. Lu et al. (2019) developed a non-intrusive system based on a bracelet. They used a decision tree classification model to classify 2 weeks of classroom activity data (including stillness, hand-raising, writings, head-scratching and other actions) collected from 71 secondary school students. The level of engagement (including low, normal and high) was then inferred from the students’ behaviors based on the ICAP framework (basing on students’ overt behaviors, the framework classifies cognitive engagement activities.).

Cognitive load theory posits that the number and types of cognitive loads during instruction can either facilitate or hinder learning outcomes. In addition, cognitive load theory states there is a finite amount of working memory, and overly demanding learning tasks require too much working memory, which can slow down the rate of learning (Paas et al., 2003). Thus, there is a strong link between cognitive load and learning effectiveness. There are three main cognitive load measurement techniques: self-report, subjective evaluation scale, performance measurement, and psychophysiological measures (Cain, 2007). Wearable sensing technology allows for greater accuracy and easier manipulation of psychophysiological measurement methods. Its main advantages are objectivity of measurement, sensitivity to different cognitive processes and the implicit and continuous measurement of processes. Örün and Akbulut (2019) explored the effects of physical environment and multitasking on cognitive load with the help of an EEG headset. The study found an increase in perceived mental effort for those performing multitasking and an effect of different physical environments on cognitive load. It also mentioned that students’ perceived mental effort was related to beta frequency of F7 (F7 is one of the signal acquisition channels of the EEG device and is located in the frontal lobe).

Affective domain

Desautels (2016) noted that: “Emotions drive our attention and perception.” In particular, academic emotion is a major element that affects learning performance. Just as established in previous work, learning emotions, such as confidence, boredom, and confusion, are important indicators of academic achievement and predictors of academic performance (MacCann et al., 2011; Pekrun et al., 2014). Thus, academic emotion recognition is a vital branch in learning analytics. Researchers have developed numerous emotion classification models and even affective recommender systems based on physiological signals, body movements, facial expressions, verbal information, online text, interaction logs, and questionnaire scales, with high recognition rates for general emotions (Salazar et al., 2021). Sensing technology is an important means of emotion recognition, using devices such as wristbands, pulse meters, and respiratory sensors to capture physiological signals, wearable motion sensors to capture information about the learner’s movements, combined with changes in facial expressions recorded by optical and depth cameras to improve the accuracy of emotion recognition, some examples are shown in Table 2.

So far, emotion recognition techniques are well developed. According to Russell’s circumplex model of affect, positive affect (surprise, excitement, pleasure, satisfaction, relaxation) and negative emotions (fear, tension, disappointment, grief, sadness, frustration, and boredom) cover almost all human emotional expressions (Russell, 1980). Although emotions are diverse and complex in structure, teachers, and intelligent learning systems only need to understand the most basic and common emotions of learners to make intelligent decisions. Shen et al. (2009) used an e-Learning platform integrated with three wearable devices to collect four kinds of physiological data: blood volume pressure (BVP), skin conductance (SC), heart rate (HR), and electrocardiogram (EEG). According to Russell’s two-dimensional affective model, the physiological data collected were classified into four distinct affects: confusion, engagement, hopefulness, and boredom. The final classification accuracy of 86.3% was achieved using SVM, which made it possible for teachers to grasp the emotion changes of the distance students. There are also a number of researchers who have conducted similar emotion detection with students at different stages of learning in different learning environments (Geršak et al., 2020; Harley et al., 2015; Pijeira-Díaz et al., 2018).

Isen et al. (1987) pointed out that positive emotions have a facilitating effect on the learning process, especially in the context of creative problem solving. In contrast, negative emotions may hinder student, disrupt the learning process and reduce educational effectiveness. Izard et al. (1984) has investigated the extent to which cognitive activity is affected by negative emotions. Chen and Sun (2012) also pointed out that student who was anxious, angry and depressed had difficulties in studying. Therefore, when the system detects that a student is in a negative mood or has a persistently low emotional experience, it is vital to provide feedback and moderation accordingly. For example, the perceived emotional information is integrated into the online video lectures, tailoring the learning materials to the learner’s emotional state and enhancing the learning experience (Choi et al., 2019); teachers regulate the teaching process according to learners' emotional responses to improve learning efficiency (Di Lascio et al., 2017). In addition, wearable devices have been used to aid emotional communication to enhance emotional perception, self-expression and interaction for children with special needs (Mehmood & Lee, 2017; Xiao et al., 2020)

Behavioral Domain

In the field of learning analytics, identifying learners’ behavioral movements, usually both learning-related and learning-irrelevant, is regarded as an important research direction. Distinguishing and describing body movements precisely will directly assist in building a deeper and more powerful insight into the learner, leading to a holistic and comprehensive analysis of their learning process. Furthermore, the behavioral action information can be an important cue for inferring skill levels and learning status, such as students’ mastery of physical education skill and engagement in classroom activity (Lu et al., 2017).

The widespread use of wearable devices and the rapid development of activity recognition technologies based on sensory data greatly facilitate researches on learners’ behavioral movements. Physical movement recognition is divided into two main types: vision-based sensing and inertia-based sensing. Visual sensing uses mainly optical sensors and depth sensors such as Microsoft Kinect and Google Glass to record body movements. Vision-based human behavior recognition is easy to operate and captures a wealth of information, but its performance is affected by the location of the device, the subject’s range of motion, and the surrounding environment. Motion recognition based on inertial sensing uses wearable sensing devices such as magnetometer, gyroscope, and accelerometer to record information about the wearer’s movements. This method is not constrained to time or space, and allows for continuous recording and interaction for a long time, but due to its own limitations, the long-term use may lead to the wear and tear of sensing devices and a decrease in measurement accuracy.

The majority of current researches have concentrated on low-level recognition of basic movements, like standing, sitting, walking, running, and looking up (Almusawi et al., 2021; Zhong & Li, 2020; Zhong, 2021), or recognition of embodied learning behaviors, such as writing, discussing, raising hands, listening to teacher, and mind-wandering (Okur et al., 2017; Yazawa et al., 2018). Simple external behaviors play a supportive role in learning analytics, but there is growing interest in the relationship between learners' movements and higher psychological mechanisms such as cognition and emotion (Bowman et al., 2017; Camacho et al., 2020), representing their cognitive states and emotional experiences through their different movement behaviors. In addition, sensor data-based movement recognition can be used to improve students’ movement skills in sport. By identifying the key movements of students, the accuracy, strength, and stability of their movements can be determined so that personalized instructions can be customized (Almusawi et al., 2021). Meanwhile, wearable devices such as smart glasses are also frequently used by medical students to improve their operation skills (Guze, 2015; Mill et al., 2021).