Measuring posture change to detect emotional conditions for workers: A scoping review

Abstract

BACKGROUND:

The emotional management of workers can not only increase the efficiency of work, but also contribute to the improvement of the productivity of a company.

OBJECTIVE:

This scoping review surveyed the literature to identify the relationship between postural expression and emotion during sedentary tasks.

METHODS:

We searched relevant literature published up to December 1, 2019 using seven electronic databases (PubMed, CINAHL, Embase, Web of Science, PsycINFO, IEEE Xplore, and MEDLINE Complete).

RESULTS:

A total of 14 publications were included in this scoping review. It was found that the application of pressure sensor and camera-based measurement equipment was effective. Additionally, it was proposed to predict the emotional state of the worker by using forward and backward movements as the main variable as opposed to left and right movements. The information-based analysis technique was able to further increase the accuracy of workers’ emotion prediction.

CONCLUSIONS:

The emotion prediction of workers based on sitting posture could be confirmed for certain movements, and the information-based technical method could further increase the accuracy of prediction. Expansion of information-based technical research will further increase the possibility of predicting the emotions of workers based on posture, and this will in turn promote safer and more efficient work performance.

Keywords

Affective state body position working environment scoping review

1 Introduction

Emotions are reactions to subjective experiences and thoughts and are associated with physiological and behavioral changes in humans [1, 2]. Generally, people experience various feelings when performing purposeful tasks or engaging in certain occupations. In recent years, researchers have expressed a growing interest in detecting the emotions individuals experience while performing activities in work environments. Furthermore, it has been shown that stress from excessive workloads can be related to work efficiency [3 –5].

Workers experience a positive flow of emotions and have a clear sense of purpose when difficulties and abilities are balanced. This natural flow of emotions evokes pleasant emotions during work participation [6]. Engagement during work is the behavior wherein workers are fully invested in completing a task and focused in achieving a result [7, 8]. Positive emotions during work not only increase the concentration of workers, but also improve their performance, thereby giving them a sense of accomplishment. This has been shown to have a positive effect on the job retention rate of workers; the management efforts of employers improve the emotional state of workers, which in turn contribute to the improvement of productivity. Therefore, it is important to develop a method that can reduce the burden on workers and predict their emotional state during specific tasks [9].

Emotional problems can often be identified earlier than cognitive problems and can therefore be important predictors of cognitive problems [10 –15]. Certain approaches to the detection of emotional states have been suggested, but most of these have not been standardized or generalized [16]. It is possible for facial and postural expressions to be analyzed automatically in real-time. It is therefore important to understand the interaction between emotions and facial and postural expressions. However, relatively little is known about the interaction between emotions and postural expression [17]. In addition, there has been a growing interest in user discomfort regarding wearable devices such as smart glasses and smart watches, with researchers finding that users felt uncomfortable with continuous use of these devices. For this reason, researchers have suggested real-time assessment through embedded sensors placed on objects used frequently in daily life.

In white-collar or blue-collar jobs, many workers spend a majority of their time in a sitting position. Although physical activity is one of the basic functions of human beings, the amount of sedentary activity time is increasing worldwide [18, 19]. Common examples of sedentary behavior are computer or other screen-based use and viewing and desk-based tasks. In all of these activities, a majority of time is spent in a sitting position. Clearly, the sedentary work of blue-collar workers compared to white-collar workers is not often mentioned. Blue collar work can be described as non-clear occupational roles within an industry (e.g. driver, electrician or factory worker) [20].

With the increasing use of technology, sedentary work is taking up a majority of workers’ time. In a previous study, the maximum sitting time for young adults per day was estimated to be 9.2 hours [21]. Thus, the sitting position can be a favorable condition for real-time access to physical and emotional information that may affect an individual’s ability to perform their work. The sitting position enables easy and convenient detection of information that can explain one’s emotional state. This scoping review surveyed the literature to identify the relationship between postural expression and emotion during sedentary tasks.

2 Methods

We searched seven electronic databases for relevant literature published as of 1^st December 2019. We used the framework proposed by Arksey and O’Malley, which has a five-stage review process [22]. First, we reviewed relevant research and selected three research questions. These are the initial exploratory research questions: (1) Can postural expression while performing sedentary tasks be considered as a predictive variable of human emotions? (2) What are the tools and measurement variables that can predict emotions through changes in sitting posture? (3) What are the future research directions and key considerations?

The following eligibility criteria were used: (1) journal article (2) published from 2008 to 1^st December 2019, (3) written in English, (4) aimed at confirming the relationship between seated posture and human emotion through a study. Exclusion criteria were: (1) non-human studies, (2) studies involving mental problems, (3) not using a chair in the study, (4) treatment studies of musculoskeletal problems, and (5) training studies for emotional regulation. We used the terms mentioned in a list of sitting posture-related (sitting posture OR on the chair OR on the table) and emotion-related (emotion OR motivation OR affective OR expression OR mood) items as search terms in conjunction with “sensor, system, device, detection, and recognition” to identify relevant research articles from seven electronic databases (PubMed, CINAHL, Embase, Web of Science, PsycINFO, IEEE Xplore, and MEDLINE Complete). Discrepancies in the extracted data were discussed between the two reviewers until an agreement was reached or, if necessary, were resolved through arbitration by a third-party reviewer. A flow diagram shows the process of study selection (Fig. 1).

Fig. 1

Flowchart illustrating the inclusion process.

3 Results

3.1 Characteristics of sedentary tasks in the studies

To compensate for the absence of previous studies, studies that can be explained in a similar working environment were selected for this review, and the results were synthesized. Sedentary tasks in these studies were classified into three types. First, cognitively demanding tasks were applied in seven studies. Three articles used settings based on game environments [14 , 24] and four articles utilized challenging tasks that required a logical reasoning process [25 –28]. Second, the researchers defined emotion-related activities. Four studies predicted emotion-relevant actions set by researchers using measurement equipment [2 , 29–31]. This was not a process of discovering a change of posture according to workers’ natural emotional change, but an experimental situation to define and distinguish posture based on emotion. Third, a media-related task was applied in two studies [32]. This study used videos that induced emotions (Table 1).

Table 1
Summary of study characteristics and key results from 14 studies

Article citation Sensatory work Evaluation tools* Combined physiological tools Measured posture expression variables Posture expression pattern analysis

Balzarotti et al. (2014) [23] – Web exploration
– Boring, enemy, and quiz games – High-resolution web cameras (Observer 7.0 NOLDUS)* ECG, EDA – Face muscle movement
– Gaze direction
– Posture and head movement
– Vocal behavior N/A

Grafsgaard et al. (2012) [25] – Computational thinking – Kinect depth sensors – HeadDepth
– MidTorsoDepth
– LowerTorsoDepth N/A

Gravina and Li (2019) [29] – Emotion-relevant actions – Force sensing resistor 406
– Shimmer motion sensor – Pressure distribution
– Wrist, arm, leg, and ankle motion WEKA data-mining toolkit

Gavrilescu (2015) [32] – Watch 6 emotion inducing videos – Vision based tracking system – Hand gesture
– Body gesture GER system

Kumar et al. (2016) [16] – 6 emotion based activities – RFduino sensor platform (four pressure sensors)* Breathing rate – Pressure distribution 13 machine learning classifiers

Li et al. (2018) [2] – Emotion-relevant activities – FSR pressure sensors
– Shimmer motion sensor – Pressure distribution
– Arm shaking and movement HMM-based algorithm

Michishita et al. (2013) [17] – 24 Emotion-relevant actions – Pressure sensor (seat and foot)
– Accelerometer – Head angles
– Waist positions
– Trunk angles N/A

Oosterwijk et al. (2009) [26] – Word generation task – Headset
– Distal camera – Posture height
– Horizontal movement N/A

Qiu and Helbig (2012) [27] – Simple reaction time
– Compensatory visual tracking
– Tower of London
– Two-column addition – Camcorders
– Cartesian coordinate system – Distance between the left eyepoint and the display’s center point
– Distance between the left shoulder and the hip
– Sagittal plane angles N/A

Riemer et al. (2017) [14] – Platform game – Microsoft Kinect for Windows v2(off-the-shelf depth-image sensor) – Position of upper body joint
– Rotation of the head N/A

Saneiro et al. (2014) [28] – Mathematical problems – Webcam
– Kinect Studio – Facial expressions
– Body movement N/A

Shibata and Kijima (2012) [30] – 24 Emotion-relevant actions – Pressure sensor
– Accelerometer – Neck movement
– Trunk movement
– Arm movement N/A

Shibata et al. (2013) [31] – 24 Emotion-relevant actions – Pressure sensor (seat and foot)
– Accelerometer – Head angles
– Waist positions
– Trunk angles N/A

van den Hoogen et al. (2008) [24] – Shooter game – Accelerometer
– Force-sensitive sensors built into the legs of the chair
– Pressure sensitive mouse – Upper body movement
– Number of shifts in position
– Pressure changes of chair and mouse N/A

Article citation	Sensatory work	Evaluation tools* Combined physiological tools	Measured posture expression variables	Posture expression pattern analysis
Balzarotti et al. (2014) [23]	– Web exploration – Boring, enemy, and quiz games	– High-resolution web cameras (Observer 7.0 NOLDUS)* ECG, EDA	– Face muscle movement – Gaze direction – Posture and head movement – Vocal behavior	N/A
Grafsgaard et al. (2012) [25]	– Computational thinking	– Kinect depth sensors	– HeadDepth – MidTorsoDepth – LowerTorsoDepth	N/A
Gravina and Li (2019) [29]	– Emotion-relevant actions	– Force sensing resistor 406 – Shimmer motion sensor	– Pressure distribution – Wrist, arm, leg, and ankle motion	WEKA data-mining toolkit
Gavrilescu (2015) [32]	– Watch 6 emotion inducing videos	– Vision based tracking system	– Hand gesture – Body gesture	GER system
Kumar et al. (2016) [16]	– 6 emotion based activities	– RFduino sensor platform (four pressure sensors)* Breathing rate	– Pressure distribution	13 machine learning classifiers
Li et al. (2018) [2]	– Emotion-relevant activities	– FSR pressure sensors – Shimmer motion sensor	– Pressure distribution – Arm shaking and movement	HMM-based algorithm
Michishita et al. (2013) [17]	– 24 Emotion-relevant actions	– Pressure sensor (seat and foot) – Accelerometer	– Head angles – Waist positions – Trunk angles	N/A
Oosterwijk et al. (2009) [26]	– Word generation task	– Headset – Distal camera	– Posture height – Horizontal movement	N/A
Qiu and Helbig (2012) [27]	– Simple reaction time – Compensatory visual tracking – Tower of London – Two-column addition	– Camcorders – Cartesian coordinate system	– Distance between the left eyepoint and the display’s center point – Distance between the left shoulder and the hip – Sagittal plane angles	N/A
Riemer et al. (2017) [14]	– Platform game	– Microsoft Kinect for Windows v2(off-the-shelf depth-image sensor)	– Position of upper body joint – Rotation of the head	N/A
Saneiro et al. (2014) [28]	– Mathematical problems	– Webcam – Kinect Studio	– Facial expressions – Body movement	N/A
Shibata and Kijima (2012) [30]	– 24 Emotion-relevant actions	– Pressure sensor – Accelerometer	– Neck movement – Trunk movement – Arm movement	N/A
Shibata et al. (2013) [31]	– 24 Emotion-relevant actions	– Pressure sensor (seat and foot) – Accelerometer	– Head angles – Waist positions – Trunk angles	N/A
van den Hoogen et al. (2008) [24]	– Shooter game	– Accelerometer – Force-sensitive sensors built into the legs of the chair – Pressure sensitive mouse	– Upper body movement – Number of shifts in position – Pressure changes of chair and mouse	N/A

ECG: Electrocardiogram, EDA: Electrodermal activity, GER: Gesture Emotion Recognition, HMM: Hidden Markov Model.

3.2 Sedentary tasks and emotions

Despite the development of technology and automation, it is still a labor-intensive industry. In the case of office work, even in the absence of physical stress, mental stress causes various emotional states in workers (e.g., joy, anxiety, excitement, relaxation). In addition, these emotional states can have an effect on workers’ (1) cognitive states such as attention and motivation, (2) decision-making ability to solve problem situations, (3) sleep disorders and headaches, and so on. It can also be linked to problems such as mental and physical health conditions [33, 34]. These impacts in-turn affect job performance in areas such as safety, health, quality, and productivity [35]. Generally, measured emotional variations represent emotional dimensions such as arousal, valence, and dominance. Valence can be explained as the way in which an individual judges a stimulus (unpleasant vs pleasant), and arousal can be conceptualized as ranging from calm to excited, varying based on the degree of stimulation an individual feels about a stimulus. Finally, the term dominance/control refers to an individual’s sense of control over a given stimulus [36]. Our review confirmed that most studies considered high-arousal negative-valence (e.g., frustration), low-arousal negative-valence (e.g., boredom), and high-arousal positive-valence (e.g., joy) (Table 2).

Table 2
Summary of study characteristics and key results from 14 studies

Article citation Participants Postural expression Emotion variables Main finding

Balzarotti et al. (2014) [23] N = 24 (12 females), 23.1 years Forward and backward –Boredom –Frustration –Amusement –Surprise –Amusement, frustration and surprising: the highest number of postural behavior units (F(3,66) = 19.34, p < .001)

Grafsgaard et al. (2012) [25] N = 33 (undergraduate students) Forward –Frustration –Frustration↑: farther from the interface(r = –.536, p = .0013), shifted position (r = .527, p = .0016)

Gravina and Li (2019) [29] N = 8 (4 females), 26 years Forward and backward –Interest –Sadness –Frustration –Happiness –Interest and happiness related activity: less marked than the frustration and sadness related activities–Sensor fusion can achieve high recognition accuracy (91.8%)

Gavrilescu (2015) [32] N = 64 (32 females), 18 35 years Forward and backward + Left and right –Fear –Anger–Surprise–Happiness–Sadness–Disgust –Postural expressions are not specifically reported–FER+GER (recognition rate): Fear(89.7%)>Happiness(89.2%)>Surprise(89%)>Anger(87.8%)>Sadness(84.4%)>Disgust(76.7%)

Kumar et al. (2016) [16] N = 5 Forward and backward + Left and right –Fear –Anger–Boring–Happiness–Sadness –Postural expressions are not specifically reported–Extremely Randomized Trees classifier (86.46%)

Li et al. (2018) [2] N = 8 (4 females), 26.0 years Forward and backward + Left and right –Shame –Fear–Joy –Fused feature set (J48 + NB+RF): Shame(100%), Fear(99.8%), Joy(97.8%)

Michishita et al. (2013) [17] N = 28 (2 females) early 20s Forward and backward –Arousal factor–Valence factor–Dominance factor –High arousal: vertically straight–Positive valence: leans backward–High dominance: leans forward–Low dominance: leans backward–Average correct rates: worried (71%), angry (63%), sad (62%) are easy to recognize from the sitting postures

Oosterwijk et al. (2009) [26] N = 57 (52 females), undergraduate students Vertical + Forward and backward (horizontal) –Pride –Disappointment –Generation of disappointment words: decrease in posture height along the vertical axis (F(3,140)¼ = 16.89, p < .0001)–Generation of pride words: decrease was absent (t(56)¼ = 2.76, p < .004)–Not find horizontal changes

Qiu and Helbig (2012) [27] N = 22 (8 females), 24.14 years Forward and backward –Positive affect–Negative affect –Higher over work load: closer to the displays (χ² (1) = 16.632, p < .001)(distance between shoulder and hip decreased)–Emotions of participants were not significantly changed.

Riemer et al. (2017) [14] N = 70 (46 females), 22.29 years Vertical + Forward and backward + Left and right –Enjoyment–Boredom–Frustration –Frustration: more closely to the screen(χ² (1) = 4.10, P = 0.043)

Saneiro et al. (2014) [28] N = 75 Vertical + Forward and backward + Left and right –Arousal factor–Valence factor–Dominance factor –Emotional reactions: influenced by the duration and difficulty of the task–Valence and arousal levels: impact on the body movements

Shibata and Kijima (2012) [30] N = 10 (2 females), early 20s Forward and backward –Arousal factor–Valence factor–Dominance factor –High arousal: vertically straight–Positive valence: leans backward

Shibata et al. (2013) [31] N = 28 (2 females), Early 20s Forward and backward –Arousal factor–Valence factor–Dominance factor –Hight arousal: vertically straight–Positive valence: leans backward–Low dominance: leans forward–High dominance (offensive to others): leans backward

van den Hoogen et al. (2008) [24] N = 32 (5 females), 22.42 years Vertical + Forward and backward + Left and right –Arousal factor–Valence factor–Dominance factor –Pressure changes on the chair and mouse to differ significantly between the levels (F(2,29) = 5.52, p = .006/F(2,29) = 11.72, p < .001)–Pressure changes on a chair related to the level of difficulty (r = .320, p < .001)

Article citation	Participants	Postural expression	Emotion variables	Main finding
Balzarotti et al. (2014) [23]	N = 24 (12 females), 23.1 years	Forward and backward	–Boredom –Frustration –Amusement –Surprise	–Amusement, frustration and surprising: the highest number of postural behavior units (F(3,66) = 19.34, p < .001)
Grafsgaard et al. (2012) [25]	N = 33 (undergraduate students)	Forward	–Frustration	–Frustration↑: farther from the interface(r = –.536, p = .0013), shifted position (r = .527, p = .0016)
Gravina and Li (2019) [29]	N = 8 (4 females), 26 years	Forward and backward	–Interest –Sadness –Frustration –Happiness	–Interest and happiness related activity: less marked than the frustration and sadness related activities–Sensor fusion can achieve high recognition accuracy (91.8%)
Gavrilescu (2015) [32]	N = 64 (32 females), 18 35 years	Forward and backward + Left and right	–Fear –Anger–Surprise–Happiness–Sadness–Disgust	–Postural expressions are not specifically reported–FER+GER (recognition rate): Fear(89.7%)>Happiness(89.2%)>Surprise(89%)>Anger(87.8%)>Sadness(84.4%)>Disgust(76.7%)
Kumar et al. (2016) [16]	N = 5	Forward and backward + Left and right	–Fear –Anger–Boring–Happiness–Sadness	–Postural expressions are not specifically reported–Extremely Randomized Trees classifier (86.46%)
Li et al. (2018) [2]	N = 8 (4 females), 26.0 years	Forward and backward + Left and right	–Shame –Fear–Joy	–Fused feature set (J48 + NB+RF): Shame(100%), Fear(99.8%), Joy(97.8%)
Michishita et al. (2013) [17]	N = 28 (2 females) early 20s	Forward and backward	–Arousal factor–Valence factor–Dominance factor	–High arousal: vertically straight–Positive valence: leans backward–High dominance: leans forward–Low dominance: leans backward–Average correct rates: worried (71%), angry (63%), sad (62%) are easy to recognize from the sitting postures
Oosterwijk et al. (2009) [26]	N = 57 (52 females), undergraduate students	Vertical + Forward and backward (horizontal)	–Pride –Disappointment	–Generation of disappointment words: decrease in posture height along the vertical axis (F(3,140)¼ = 16.89, p < .0001)–Generation of pride words: decrease was absent (t(56)¼ = 2.76, p < .004)–Not find horizontal changes
Qiu and Helbig (2012) [27]	N = 22 (8 females), 24.14 years	Forward and backward	–Positive affect–Negative affect	–Higher over work load: closer to the displays (χ² (1) = 16.632, p < .001)(distance between shoulder and hip decreased)–Emotions of participants were not significantly changed.
Riemer et al. (2017) [14]	N = 70 (46 females), 22.29 years	Vertical + Forward and backward + Left and right	–Enjoyment–Boredom–Frustration	–Frustration: more closely to the screen(χ² (1) = 4.10, P = 0.043)
Saneiro et al. (2014) [28]	N = 75	Vertical + Forward and backward + Left and right	–Arousal factor–Valence factor–Dominance factor	–Emotional reactions: influenced by the duration and difficulty of the task–Valence and arousal levels: impact on the body movements
Shibata and Kijima (2012) [30]	N = 10 (2 females), early 20s	Forward and backward	–Arousal factor–Valence factor–Dominance factor	–High arousal: vertically straight–Positive valence: leans backward
Shibata et al. (2013) [31]	N = 28 (2 females), Early 20s	Forward and backward	–Arousal factor–Valence factor–Dominance factor	–Hight arousal: vertically straight–Positive valence: leans backward–Low dominance: leans forward–High dominance (offensive to others): leans backward
van den Hoogen et al. (2008) [24]	N = 32 (5 females), 22.42 years	Vertical + Forward and backward + Left and right	–Arousal factor–Valence factor–Dominance factor	–Pressure changes on the chair and mouse to differ significantly between the levels (F(2,29) = 5.52, p = .006/F(2,29) = 11.72, p < .001)–Pressure changes on a chair related to the level of difficulty (r = .320, p < .001)

FER: Facial Emotion Recognition, GER: Gesture Emotion Recognition, J48: J48 decision tree, NB: Naive Bayes classifier, RF: Random Forest.

3.3 Posture expression associated with emotion

In most studies, we confirmed that the tendency to tilt backward indicates a high-arousal positive emotional state, and to tilt forward indicates a high-arousal negative emotional state. However, there were no significant differences noted between left and right movements. We analyzed the relationships between the arousal and valence aspects and postural expression during sedentary tasks in each study [17 , 31]. High-arousal was associated with the vertical posture. Arousal was influenced by the difficulty level of a task and affected body movements when involving a cognitive process [28]. There is evidence of an increase in arousal when frustration or challenge is experienced. Additionally, the number of movements was related to high-arousal states [24]. Positive valence was associated with the body leaning backward on the chair. The number of movements is also influenced by the difficulty level of a task. Frustration and boredom lead participants to adopt a posture defined as a forward tilt or decrease in posture height (slumped posture) [17 , 31]. In addition, high-arousal negative-valence (e.g., fear) showed more marked relationships with posture changes than low-arousal negative-valence (e.g., shame) and high-arousal positive-valence (e.g., joy) [2]. Some studies have introduced a learning machine system that can analyze data; these studies suggested a new approach that could increase the recognition rate through a building system that combines the recorded data of facial expressions, body postures, and hand gestures [2 , 32] (Table 2).

3.4 Emotion detection tools in sedentary tasks

The emotion detection tools used to assess the change of posture expression in studies were motion tracking, depth-image sensor, pressure sensor, and accelerometer. Most of these tools evaluated pressure or movement on the chair during sedentary tasks. Six studies used pressure sensors and accelerometers to detect emotions while sitting and working [2 , 29–31]. One study was conducted using only a pressure sensor without an accelerometer [16]. The variables measured through the equipment were analyzed by defining the main variables such as facial muscle movement, expression change, gaze change, and head, arm, wrist, trunk, leg, and ankle movement. In most of these studies, head, torso, arm, and wrist movements were measured and analyzed as major variables. Some studies also used high-resolution cameras or camcorders to detect emotions while participants performed sedentary tasks. Among the selected studies, five studies [23 , 32] were used, and two cases of depth sensors using body type characteristics were used [14, 27] (Table 2). Interestingly, in two studies, measurements using physiological variables such as ECG (electrocardiogram), EDA (electrodermal activity), and breathing rates were simultaneously applied. These devices confirmed heart rate and parameters of skin electrical activity. In addition, in four studies, motion patterns were analyzed using a machine learning algorithm-based analysis method to predict the emotions of workers through motion pattern analysis and to predict patterned motions (Table 1).

4 Discussion

In this scoping review, the characteristics of postural expression based on emotional states were compared, and the applicability of techniques to detect workers’ emotional states was explored. Workplace environmental factors can promote success or, if unresolved, reduce work efficiency and increase job-related stress. The latter can lead to reduced job performance and reduced retention of employees [37]. Most postural expressions found in this review were analyzed based on the characteristics of participants’ postures when sitting on a chair, and the differences between various tasks were compared.

This scoping review was conducted to address three main research questions. First, “Can postural expression while performing sedentary tasks be considered as a predictive variable of human emotions?” In particular, the emotional variables of frustration, shame, and pleasure were more clearly distinguished emotional variables and were highly related with significant changes in the body [2 , 23–32]. Higher levels of arousal in experimental work allow for higher levels of detection through posture-related variables [17 , 30]. Back-and-forth movements from a sitting position, as evident in the literature, were relatively related with positive and negative emotional responses to tasks. It is interesting to note that some studies have pre-defined emotion-based behavioral traits and then studied the predictability of defined movements [2 , 29–31]. This study focuses on the possibility of identifying emotional state based on the movement of workers by defining human movement characteristics in advance; therefore, it can be assumed that these studies are more advanced than the discovery of the movement characteristics.

Negative emotional states in the work environment affect individual productivity, but few studies have investigated how to detect and manage these emotional changes [38 –41]. Clearly, confirmation of the feasibility of these measures would be beneficial in clearly distinguishing the characteristics of postural expression for each task and providing appropriate task-context adjustments.

To answer the second research question, “What are the tools and measurement variables that can predict emotions through changes in sitting posture?”, the techniques used to measure emotional changes were motion tracking, pressure sensors, depth image sensors, and accelerometers. In particular, the method of evaluating the pressure or movement exerted on the chair during the activity was most often used. Most motion detection systems can be classified into three categories: wearable sensor-based, ambient sensor-based, and computer vision-based methods. Wearable sensor-based methods generally rely on accelerometer sensors attached to the user’s body [42]. The ambient sensor-based motion detection system uses external sensors built into the environment, including pressure sensors, acoustic sensors, and EMG sensors [43 –45]. Pressure sensors are commonly used because they are low-cost and unobtrusive. However, a major drawback of these sensors is the low detection accuracy (less than 90%) [46]. Our review found that the use of pressure sensors was high. The problem of precision was countered by combining it with other equipment and utilizing data mining algorithms to increase the accuracy of the distinction [14, 27]. Increasing precision by combining pressure sensors with other measurement elements should be considered in future research plans to identify and measure the emotional changes of workers. Additionally, attention should be paid to a specifically important method: a camera that provides a depth map in real-time and is a detection and tracking method that can reliably extract the trajectory of the body, which has proven useful for precise motion detection [47]. In general, omnidirectional cameras are useful if a wide field of view is required. In addition, thermal imaging video cameras that detect the amount of heat radiation emitted/reflected from objects in the scene may provide valuable information for reliable motion detection [48]. However, the disadvantage of the existing equipment is its high cost, leading to its limited application in general environments. Recently, however, Kinect’s sensor was proposed to detect human motion [49]. The depth map provided by the Kinect sensor is sufficient to extract people from the background [49, 50]. In addition, people can be detected at any time by estimating a dense depth map based on the spot pattern of the infrared laser [50].

In the future, the application of a technology that can extract camera-based measurement will be an effective method to detect the emotional state of workers. In addition, data mining and machine learning methods can be applied to increase the accuracy of distinguishing emotional states based on movement. In everyday life, people develop a variety of postures and movements. Pattern recognition and classification is an important detection method to differentiate normal activities from changes in movement due to changes in emotion [51]. Feature extraction and selection is the process of identifying relevant features or attributes in collected data [52]. Decision trees (DT) is one of the oldest algorithms used in motion pattern classification problems [53]. This review showed that extremely randomized trees spent less time training the model compared to random forest (98.66 seconds) and gradient boosting trees (2291.03 seconds). In addition, postures according to emotions could be classified through J48, SVM (Support Vector Machine), and Naive Bayes classifier (NB) to confirm the detected movement. Among these, J48 was the most effective classification system [14, 27]. Recently, these methods have been used in diagnostic and prognostic models for neuromuscular and musculoskeletal pathology and fall prediction, focused on activity perception-clinic patient monitoring [54]. As such, machine learning methods are expanding in motor biomechanics and were attempted in two of the studies included in this review. Predictive modeling can be used to map input data to a given output and predict future events with confidence. This is one of the significant methods and trends in recent research. Prediction is necessary and important in human life. Recently, machine learning has been widely used as a method that can easily perform such predictions. In general, applying a machine learning algorithm can generate a predictive model for a target variable. Machine learning methods will increase the availability of interpretation and prediction of external information [55].

The third question guiding this review was, “What are the future research directions and key considerations?” There are some points in this review that warrant caution when interpreting the results. The studies selected for this review did not use an actual working situation as an experimental situation. Additionally, since the applied task is not selected as the actual work content, there may be limitations in generalization of the results. However, we can generalize these findings to a certain extent since most work is cognitive-based, and negative emotional changes such as job stress appear prior to physical changes. Therefore, the work of this review will be the result of having the possibility of generalization of the work environment of physical and mental workers. These measurement systems of workers’ emotions can be an important safeguard, especially for white-collar workers. However, most job characteristics involve unpredictable situations, and it can therefore be difficult and impractical to clearly understand a person’s emotions while sitting. However, predicting the emotional state of workers by applying the analysis method through the accumulation of long-term behavioral variable data can be an applicable system [56 –58].

Another important topic covered in this review is applied skills for measurement. Several documents have emphasized the possibility of applying additional technologies to the working environment such as prediction algorithms and data mining or machine learning classifiers. The realization and use of methods that can predict the emotional state of a worker is a new field of research that can enhance work experiences [56 –59]. These benefits can be extended to workers in various occupations. However, more research is needed to investigate whether these methods are an alternative to identifying the emotional state of the worker. It will be important to investigate how to securely apply such methods to identify the reactions and adjustments of workers to conditions of the work environment [60 –63].

Continuously measuring the emotional state of workers in the workplace can provide important information to reveal how workers’ emotions change in the workplace. Data-based analyses have recently begun to be used in various fields such as e-health, e-learning, and recommendation systems. A system that analyzes and interprets information based on data accumulated over a long period of time has great potential in various areas, including mental health monitoring. Given the importance of mental health in modern society, researchers will now have to find a way to accurately recognize human emotions and develop realistic applications and efficient intervention plans to maintain mental health. For example, a workplace system with an emotion recognition module will be able to monitor the psychological state of workers in real time and accordingly provide the appropriate response and treatment when required [64, 65].

A limitation of this review is that there is no study on actual workers, and generalizability of the working environment is limited. Thus, detailed information regarding job types and individual characteristics of workers cannot be provided. In addition, the majority of experimental participants were 20 to 40 years of age, while the overall age range of actual workers has been found to mainly fall in the range of 30 to 50 years.

5 Conclusion

Back-and-forth movements from a seated position were relatively associated with positive and negative emotional responses to tasks. Measuring equipment based on pressure sensor and cameras were effective in understanding worker movements in response to their emotions. Their emotions could be predicted from their sitting posture when associated with movements; using a data-based technical method here could further increase the accuracy of the predictions. An expanding use of such data-based technologies in predicting workers’ emotions to ensure their safety while achieving more efficiency and productivity will result in further advances in workers’ emotion prediction technology.

Conflict of interest

None to report.

References

Shon

, Im

, Park

, Lim

, Jang

, Kim

. Emotional stress state detection using genetic algorithm-based feature selection on EEG signals, International Journal of Environmental Research and Public Health 2018;15(11):2461.

, Gravina

, Fortino

. Posture and gesture analysis supporting emotional activity recognition. Paper presented at the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2018;2742-7.

Arpaia

, Moccaldi

, Prevete

, Sannino

, Tedesco

. A wearable EEG instrument for real-time frontal asymmetry monitoring in worker stress analysis, IEEE Transactions on Instrumentation and Measurement 2020;69(10):8335–43.

Jebelli

, Choi

, Lee

. Application of wearable biosensors to construction sites, I: Assessing workers’ stress. Journal of Construction Engineering and Management 2019;145(12):04019079.

Jebelli

, Hwang

, Lee

. EEG-based workers’ stress recognition at construction sites, Automation in Construction 2018;93:315–24.

Bakker

, Schaufeli

, Leiter

, Taris

. Work engagement: An emerging concept in occupational health psychology, Work Stress 2008;22:187–200.

Kajiwara

, Yonekura

, Kimura

. Prediction of future mood using majority vote based on certainty factor, Sensors and Materials 2018;30:1473–86.

Ferreira

, Ferreira

, Kim

, Siirtola

, Röning

, et al. Assessing real-time cognitive load based on psychophysiological measures for younger and older adults. 2014 IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB): IEEE. 2014; 39-48.

Choi

, Ahmed

, Gutierrez-Osuna

. Development and evaluation of an ambulatory stress monitor based on wearable sensors, IEEE Transactions on Information Technology in Biomedicine 2011;16(2):279–86.

10.

Shih

, Chien

, Chiang

. Elucidating the relationship between work attention performance and emotions arising from listening to music, Work 2016;55(2):489–94.

11.

Wahab

, Kamaruddin

, Palaniappan

, Li

, Khosrowabadi

. EEG signals for emotion recognition, Journal of Computational Methods in Sciences and Engineering.-S 2010;10(S1):S1–S11.

12.

Kalpana

, Jude

. Human emotion recognition using intelligent approaches: A review, Intelligent Decision Technolgies 2019;13(4):417–33.

13.

Julia

, Zhang

, Dilawar Khan

. Emotional intelligence could forge self-efficacy, work effort and career satisfaction, Human Systems Management 2017;36(2):141–9.

14.

Riemer

, Frommel

, Layher

, Neumann

, Schrader

. Identifyingfeatures of bodily expression as indicators of emotional experienceduring multimedia learning, Frontiers in Psychology 2017;27(8):1303.

15.

Sarabia-Cobo

, García-Rodríguez

, Navas

, Ellgring

. Emotional processing in patients with mild cognitive impairment: Theinfluence of the valence and intensity of emotional stimuli: Thevalence and intensity of emotional stimuli influence emotionalprocessing in patients with mild cognitive impairment, Journal ofthe Neurological Sciences 2015;357(1-2):222–8.

16.

Kumar

, Bayliff

, De

, Evans

, Das

, Makos

. Care-chair:Sedentary activities and behavior assessment with smart sensing onchair backrest, Paper presented at the IEEE InternationalConference on Smart Computing (SMARTCOMP) 2016)1–8.

17.

Michishita

, Naito

, Shibata

. Analysis and modelling ofaffective japanese sitting postures, Paper presented at the International Conference on Signal-Image Technology &Internet-Based Systems 2013)765–70.

18.

Church

, Thomas

, Tudor-Locke

, Katzmarzyk

, Earnest

, Rodarte

, et

. Trends over 5 decades in US occupation-related physical activity and their associations with obesity, PloS one.e 2011;6(5):19657.

19.

McVeigh

, Winkler

, Howie

, Tremblay

, Smith

, Abbott

, et al. Objectively measured patterns of sedentary time and physical activity in young adults of the Raine study cohort, International Journal of Behavioral Nutrition and Physical Activity 2016;13(1):1–12.

20.

Gilson

, Pavey

, Vandelanotte

, Duncan

, Gomersall

, Trost

, et al. Chronic disease risks and use of a smartphone application during a physical activity and dietary intervention in Australian truck drivers, Australian and New Zealand Journal of Public Health 2016;40(1):91–3.

21.

McVeigh

, Winkler

, Howie

, Tremblay

, Smith

, Abbott

22.

Arksey

, O’Malley

. Scoping studies: Towards a methodological framework, International Journal of Social Research Methodology 2005;8(1):19–32.

23.

Balzarotti

, Piccini

, Andreoni

, Ciceri

. “I know that youknow how I feel”: Behavioral and physiological signals demonstrateemotional attunement while interacting with a computer simulatingemotional intelligence, Journal of Nonverbal Behavior 2014;38:283–99.

24.

van den Hoogen , Wouter

, IJsselsteijn

, de Kort

. Exploring behavioral expressions of player experience in digital games. Paper presented at the Proceedings of the Workshop on Facial and Bodily Expression for Control and Adaptation of Games ECAG 2008. 2008;11-9.

25.

Grafsgaard

, Boyer

, Wiebe

, Lester

. Analyzing posture and affect in task-oriented tutoring. Paper presented at the Twenty-Fifth International FLAIRS Conference. 2012.

26.

Oosterwijk

, Rotteveel

, Fischer

, Hess

. Embodied emotionconcepts: How generating words about pride and disappointmentinfluences posture, European Journal of Social Psychology 2009;39(3):457–66.

27.

Qiu

, Helbig

. Body posture as an indicator of workload in mentalwork, Human Factors 2012;54(4):626–35.

28.

Saneiro

, Santos

, Salmeron-Majadas

, Boticario

. Towards emotion detection in educational scenarios from facial expressions and body movements through multimodal approaches. The Scientific World Journal. 2014; 484873.

29.

Gravina

, Li

. Emotion-relevant activity recognition based onsmart cushion using multi-sensor fusion, Inform Fusion 2019;48:1–10.

30.

Shibata

, Kijima

. Emotion recognition modeling of sitting postures by using pressure sensors and accelerometers. Paper presented at the Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). 2012;1124-7..

31.

Shibata

, Michishita

, Bianchi-Berthouze

. Analysis and modelling of affective japanese sitting postures by japanese and british observers. Paper presented at the 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. 2013;91-6.

32.

Gavrilescu

. Recognizing emotions from videos by studying facial expressions, body postures and hand gestures. Paper presented at the 2015 23rd Telecommunications Forum Telfor (TELFOR). 2015;720-3.

33.

Stress management in the construction industry. The IOSH (Institute of Occupational Health and Safety) Hong Kong Annual Safety Conference: Hong Kong; 2015.

34.

Tixier

, Hallowell

, Albert

, van Boven

, Kleiner

. Psychological antecedents of risk-taking behavior in construction, Journal of Construction Engineering and Management 2014;140(11):04014052.

35.

Leung

, Liang

, Chan

. Development of astressors–stress–performance–outcome model forexpatriate construction professionals, Journal of ConstructionEngineering and Management 2017;143(5):04016121.

36.

Söderholm

, Häyry

, Laine

, Karrasch

. Valence and arousal ratings for 420 Finnish nouns by age and gender, PloS One 2013;8(8):e72859.

37.

Fehr

, Fulmer

, Awtrey

, Miller

. The grateful workplace: A multilevel model of gratitude in organizations, Academy of Management Review 2017;42(2):361–81.

38.

Gander

, Proyer

, Ruch

. Positive psychology interventions addressing pleasure, engagement, meaning, positive relationships, and accomplishment increase well-being and ameliorate depressive symptoms: A randomized, placebo-controlled online study, Frontiers Psychology 2016;20(7):686.

39.

Hodgins

, Dadich

. Positive emotion in knowledge creation, Journal of Health Organization and Management 2017;31(2):162–74.

40.

Pradhan

, Jena

, Singh

. Examining the role of emotional intelligence between organizational learning and adaptive performance in indian manufacturing industries, Journal of Workplace Learning 2017;29(3):235–47.

41.

Methot

, Melwani

, Rothman

. The space between us: A social-functional emotions view of ambivalent and indifferent workplace relationships, Journal of Management 2017;43(4):1789–819.

42.

Shen

VRL

, Lai

. The implementation of a smartphone-based fall detection system using a high-level fuzzy petri net, Applied Soft Computing 2015;26:390–400.

43.

Cheng

, Chen

, Shen

. A framework for daily activity monitoring and fall detection based on surface electromyography and accelerometer signals, IEEE Journal of Biomedical and Health Informatics 2013;17(1):38–45.

44.

Nashwa

, Tan

, Pivot

, Anthony

. Fall detection and prevention for the elderly: A review of trends and challenges, International Journal on Smart Sensing and Intelligent Systems 2013;6(3):1230–66.

45.

Feng

, Liu

, Zhu

. Fall detection for elderly person care in a vision-based home surveillance environment using a monocular camera, Signal, Image and Video Processing 2014;8:1129–38.

46.

Delahoz

, Labrador

. Survey on fall detection and fall prevention using wearable and external sensors, Sensors 2014;14:19806–42.

47.

Rougier

, Meunier

, St-Arnaud

, Rousseau

. 3D head tracking for fall detection using a single calibrated camera, Image and Vision Computing 2013;31(3):246–54.

48.

Sokolova

, Fernandez-Caballero

. Fuzzy decision making model for human fall detection and inactivity monitoring, Intelligent Decision Technologies 2013;24(2):215–28.

49.

Webster

, Celik

. Systematic review of Kinect applications in elderly care and stroke rehabilitation, Journal of Neuroengineering and Rehabilitation 2014;11:108.

50.

Kwolek

, Kepski

. Improving fall detection by the use of depth sensor and accelerometer, Neurocomputing 2015;30:637–45.

51.

Polu

. Human activity recognition on smartphones using machine learning algorithms, International Journal for Innovative Research in Science & Technology 2018;5(6):31–7.

52.

Haselman

, Hauck

. The future of integrated circuits: A survey of nanoelectronics Proceedings of the IEEE 2010;98(1):11–38.

53.

Bian

, Hou

, Chau

, Magnenat-Thalmann

, Fall detection based on body part tracking using a depth camera, IEEE Journal of Biomedical and Health Informatics 2015;19(2):430–9.

54.

Zago

, Kleiner

AFR

, Federolf

. Editorial: Machine learning approaches to human movement analysis, Frontier in Bioengineering and Biotechnology 2021;22:638793.

55.

Zhou

, Fischer

, Tunca

, Brahms

, Ersoy

, Granacher

, et al. How we found our imu: Guidelines to IMU selection and a comparison of seven IMUs for pervasive healthcare applications, Sensors 2020;20:1–28.

56.

Michalak

, Mischnat

, Teismann

. Sitting posture makes a difference-embodiment effects on depressive memory bias, Clinical Psychology & Psychotherpy 2014;21:519–24.

57.

Schiopu

. Workplace emotions and job satisfaction, International Journal of Economic Theory 2015;5:277–82.

58.

Schwartz

, Kapellusch

, Schrempf

, Probst

, Haller

, Baca

. Effect of alternating postures on cognitive performance for healthy people performing sedentary work, Ergonomics 2018;61(6):778–95.

59.

Bailenson

, Pontikakis

, Mauss

, Gross

, Jabon

, Hutcherson

CAC

, et al. Real-time classification of evoked emotions using facial feature tracking and physiological responses, International Journal of Human-Computer Studies 2008;66(5):303–17.

60.

Morgan

, D’Mello

. The effect of positive vs. negative emotion on multitasking. Paper presented at the Proceed- ings of theHumanFactors and Ergonomics Society Annual Meeting. 2013;848-52.

61.

Pinheiro

, Ramos

, Donizete

, Picanco

, De Oliveira

. Workplace emotion Monitoring-An emotionoriented system hidden behind a receptionist robot. Mechatronics and Robotics Engineering for Advanced and Intelligent Manufacturing. 2017;407-20.

62.

Wall

, Russell

, Moore

. Positive emotion in workplace impact, Journal of Work-Applied Management 2017;9(2):129–46.

63.

Laura

, Lixin

. Buring out? Watch your won incivility and the emotions you spread, Work 2019;64(4):671–83.

64.

Mutsumi

, Makoto

. Structural relationships among occupational dysfunction, stress coping, and occupational participation for healthcare workers, Work 2019;64(4):833–41.

65.

Zhang

, Yin

, Chen

, Nichele

. Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review, Information Fusion 2020;59:103–26.