Abstract
Advances in robotics have contributed to the prevalence of human-robot collaboration (HRC). Working and interacting with collaborative robots in close proximity can be psychologically stressful. Therefore, it is important to understand the impacts of human-robot interaction (HRI) on mental stress to promote psychological well-being at the workplace. To this end, this study investigated how the HRI presence, complexity, and modality affect psychological stress in humans and discussed possible HRI design criteria during HRC. An experimental setup was implemented in which human operators worked with a collaborative robot on a Lego assembly task, using different interaction paradigms involving pressing buttons, showing hand gestures, and giving verbal commands. The NASA-Task Load Index, as a subjective measure, and the physiological galvanic skin conductance response, as an objective measure, were used to assess the levels of mental stress. The results revealed that the introduction of interactions during HRC helped reduce mental stress and that complex interactions resulted in higher mental stress than simple interactions. Meanwhile, the use of certain interaction modalities, such as verbal commands or hand gestures, led to significantly higher mental stress than pressing buttons, while no significant difference on mental stress was found between showing hand gestures and giving verbal commands.
Introduction
In recent years, human-robot collaboration (HRC) has been growing rapidly in the context of smart manufacturing and Industry 4.0. Collaborative robots (co-robots) are employed to assist human operators in achieving unprecedented flexibility, where human cognitive skills and dexterity are mutually reinforced with the physical capabilities of the co-robots. Typically, co-robots are used for repetitive and physically demanding tasks, while human operators are responsible for advanced decision-making and fine-tuning tasks (Cherubini et al., 2016). In HRC, human operators and co-robots perform tasks concurrently or jointly within a collaborative workspace. For example, in an HRC assembly task, the co-robot delivers and holds a part with pre-drill screw holes and the human operator turns screws into it. As the level of collaboration continues to increase, workspaces are shared more intensively, leading to a symbiotic HRC (Wang et al., 2019).
Close collaboration with robots raises concerns related to workplace health and safety. Physical safety issues such as collisions may occur as isolating workers from co-robots is not an option in HRC tasks. One way to ensure safe coexistence between humans and robots is through proactive retraction of the co-robot end effector depending on the location of workers. This can be accomplished using motion-tracking devices, such as inertial sensors (Meziane et al., 2014), depth sensors (Mohammed et al., 2017), or RGB cameras (Xie et al., 2022). In addition to physical collision, co-robots can lead to mental stress for human operators, which can negatively affect interaction and collaboration performance (Gervasi et al., 2022). Therefore, it is equally important to study and understand human mental stress in HRC to optimize the process and achieve workplace wellness.
Mental stress can be assessed through subjective and objective measures. Subjective ratings, such as self-report questionnaires, have been commonly used to estimate levels of mental stress in humans (Aigrain et al., 2018). Participants are asked to answer a variety of questions about their experiences in the experiment. The main limitation of self-reporting is that participants cannot report in real time and may not express their true feelings (Bethel et al., 2007). The use of objective measures, such as physiological signals, is an important complement to subjective measures and provides insight into the subconscious and psychobiological phenomena involved (Rubagotti et al., 2022). Galvanic skin response (GSR), also known as electrodermal activity (EDA), measures skin conductivity and effectively reflects people’s emotional states. GSR readings significantly increase when the stress level increases (Shi et al., 2007). Many researchers have used GSR as an objective measure to assess mental stress. For example, Healey (2005) analyzed the GSR data collected during realworld driving tasks to determine the driver’s relative stress level. Giakoumis (2012) developed an automatic stress detection algorithm by using GSR data as an important basis. Yin (2022) evaluated the impacts of in-vehicle warnings on mental stress and safety performance through GSR signals.
Previous studies have investigated relevant robot factors, such as robot attributes and motion characteristics, to explore their correlation with human mental status. The experimental study conducted by Rahimi and Karwowski (1990) indicated that robot sizes and initial speeds were the significant main effects on the perception of safe robot speed. This finding was later verified by Duffy (2006) in a virtual reality environment. Arai (2010) concluded that operators experience high mental stress when robots move toward them at high speed and recommended that additional notice should be provided before a robot moves. Another study by Lu (2022) examined human psychological stress during HRC handover tasks, noting that the end effector approaching within the worker’s field of view at a low speed and with a restricted trajectory can cause significantly less mental stress.
Moreover, in complex HRC scenarios, collaboration is facilitated by human-robot interaction (HRI), which can occur through different sensory channels, including touch, sight, and hearing (Bonarini, 2020). Several studies have shown that interaction between human and robot can influence user experiences and emotions. For example, an experimental study conducted by Nomura (2008) found that negative attitudes towards a communication robot were associated with specific behaviors, such as the duration of conversation with the robot. A similar study by de Graaf (2013) found that interactions with social robots influenced people’s emotions and increased their anxiety about talking to the robot. In a more recent study, Gervasi (2022) found that introducing interactions via a button during collaborative assembly tasks significantly reduced perceived discomfort compared to scenarios without interaction.
To further contribute to the field of HRC, we conducted an experimental study to investigate how different HRI paradigms affect the psychological stress of human operators during HRC. A collaborative Lego assembly task was implemented and physiological GSR data were applied to measure mental stress along with self-report questionnaires. We aim to evaluate how the presence of interactions during HRC affects human psychological stress and the differences across interaction complexity and modalities, including pressing buttons (touch channel), showing hand gestures (sight channel), and giving verbal commands (hearing channel).
Method
Participants
A total of ten healthy participants (4 females and 6 males, average age of 27 years (SD = 3.6 years)) with no acute or chronic musculoskeletal disorders were recruited for this experimental study. All participants had no or little previous experience in HRC but were familiar with Lego assembly tasks. The experimental protocol was approved by the university institutional review board. All participants signed a consent form prior to participation in the study.
Experiment Setup
In this experiment, a Sawyer robot arm (Rethink Robotics) was used to complete the collaborative Lego assembly task, as shown in Figure 1. The robot control was realized through the Robot Operating System (ROS noetic) framework in Linux Ubuntu 20.04 environment.
The collaborative assembly task adopted in the experiment.
The assembly task can be briefly described as follows: First, the co-robot picks up a large Lego block from Zone 1 and delivers it to the participant sitting in Zone 2. Then, the participant takes the large Lego block from the co-robot and secures small irregular-shaped Lego pieces on the large Lego block. The participant then needs to enter a serial number shown on the Lego block using an iPad. This procedure is repeated five times in each collaboration session.
To investigate the effects of HRI paradigms on human mental stress, three relevant factors involved in HRC were examined, namely (i) the interaction presence, (ii) the interaction complexity, and (iii) the interaction modality. In terms of interaction presence, no-interaction sessions were designed. In no-interaction sessions, the co-robot repetitively delivered blocks at a specific time interval (16 seconds) and automatically opened the gripper in 2 seconds once it reached the operator’s position. In terms of interaction complexity, two levels were considered: simple interaction and complex interaction. Specifically, in simple interaction, the human operator communicates with the robot only once when the operator needs the co-robot to open the gripper; this action of gripper opening is considered risk critical because improper opening times may result in block falling, thus disrupting the collaborative task. In complex interaction, in addition to requesting the robot to open the gripper, the human operator needs to communicate with the robot for initializing the delivery of the large Lego block; this delivery action is considered non-risk critical since it does not directly affect whether the takeover task is successful.
For interaction modality, three levels were adopted, including pressing buttons, showing hand gestures, and giving verbal commands. These three interaction modalities were achieved with off-the-shelf hardware, i.e., a numeric mini keyboard, a webcam, and a wireless microphone, as shown in Figure 2. For pressing buttons, the numeric programmable keyboard was placed on the table and adjusted to the most comfortable position according to each participant’s preference. It was connected to the same workstation that controls the corobot, with key “1” programmed to deliver the next Lego block and key “4” programmed to open the gripper. In terms of gestures, an RGB webcam (Logitech BRIO) was also placed on the table for capturing the hand movements and gestures. Twenty-one landmarks of the hand were first extracted using Google MediaPipe Hands and then fed into a self-trained gesture recognition model. The “OK” gesture was used to deliver the next Lego block, and the “Open” gesture was used to open the gripper, as is illustrated in Figure 3. Furthermore, a wireless microphone (RODE) was clipped to the collar of the participant’s clothes and connected to the workstation via a remote receiver for recording verbal commands. Google Speech API was adopted for speech recognition. The operator must utter a statement containing “open”, such as “please open the gripper” to open the gripper, and a statement containing “next”, such as “please deliver the next”, to pass the next Lego block. All the algorithms for realizing different HRI paradigms were implemented in Python (3.8).
Devices used to interact with the co-robot (from left to right): extended keyboard, webcam, and wireless microphone.
Hand gestures: “OK” sign (left) and “Open” sign (right).
Each experiment consisted of eight sessions, including two no-interaction sessions (one at the beginning and one at the end of the experiment) and six interaction combinations (2 interaction complexity × 3 interaction modality). The reason for conducting two sessions to determine the no-interaction mental stress level was to eliminate the time order effect as much as possible. Also, the sequence of the interaction combinations was randomized. As speaking potentially affects stress levels, a split-plot randomization process was utilized to minimize the communication between the researcher and participant during session-to-session transitions. First, the sequence of interaction complexity factor was randomized, i.e., each participant was randomly assigned to the complexity of interacting once or twice. Then, within each complexity, all three communication modalities were again completely randomized.
Before the experiment, a training session was given to each participant. The researcher first introduced the participants to the experimental procedures and explained how to use the devices to interact with the co-robot. Afterward, participants practiced with Lego blocks until they became familiar with the assembly tasks. Between each session, there was a 4-minute period for participants to take a break and answer the self-report questionnaire.
Data Acquisition and Analysis
Galvanic Skin Response (GSR)
In studies involving emotional arousal, skin conductance is a commonly used physiological measure that refers to the varying electrical properties of the skin in response to sweat secretion by sweat glands. In particular, eccrine sweat glands are mostly involved in emotional responses (Dawson et al., 2007). It is recommended that GSR should be recorded in areas with a high density of eccrine sweat glands, such as the palms and soles (600 to 700 glands/cm2) (Saga, 2002). Also, the findings of van Dooren (2012) indicated that the feet, fingers, and shoulders were the most responsive recording locations. Therefore, we chose foot as the recording location since both hands were occupied in the collaborative assembly tasks. A non-invasive Shimmer3 GSR+ device was used in this experiment, and two electrodes were placed on the medial side of foot sole, as shown in Figure 4.
Locations of GSR electrodes.
Skin conductance is composed of skin conductance level (SCL) and skin conductance response (SCR). SCL is the tonic level which refers to the absolute conductance level in the absence of a measurable stimulus. SCR is the phasic increases superimposed on SCL, reflecting the response to internal or external stimuli (Dawson et al., 2007). Therefore, the phasic component of the GSR data was extracted to evaluate the participant’s mental stress level.
GSR data processing was performed by using Ledalab, a MATLAB-based software. The original signals recorded at 256 Hz were first down-sampled to 64 Hz and then decomposed into tonic and phasic components using continuous decomposition analysis (Benedek & Kaernbach, 2010). The mean value of the phasic component of each interaction session was used for statistical analysis. For no interaction, the response was calculated as the average of the means of the two sessions. As SCR is susceptible to individual differences, the obtained data were normalized within each individual, i.e., divided by the maximum mean value of all sessions, which has been adopted in other studies (Arai et al., 2010; Lu et al., 2022).
NASA-Task Load Index (NASA-TLX)
Participants were also asked to fill out the NASA-TLX questionnaires after completing each HRC session. NASA-TLX performs a multidimensional assessment of the overall mental workload based on six subscales, including mental demand, physical demand, temporal demand, performance, effort, and frustration. For each dimension, the response scale is essentially a bipolar description (e.g., Low/High), with a line of 21 marks. Values were rounded up if a participant marked between two tick marks. The average of the six subscales was used for statistical analysis (Zakeri et al., 2021).
Results
Statistical analyses, including analysis of variance (ANOVA) tests and Tukey HSD post hoc tests, were conducted to analyze the effects of HRI paradigms on mental stress. Besides the factors we intended to study, participants were considered blocking factors, and time was considered a covariate. The statistical significance level was set at 0.05.
One-way ANOVA was performed to analyze the effects of HRI presence, i.e., comparing the no-interaction condition to all other HRC sessions that involved interactions. The results are presented in Figure 5, where the left bar chart represents the objective assessment using normalized physiological GSR data, and the right bar chart represents the subjective assessment using the NASA-TLX questionnaire. Both assessments showed a significant effect of the introduction of HRI on human mental stress (for the objective assessment, F(6,59) = 10.1097, p<0.001; for the subjective assessment, F(6,59) = 6.2544, p<0.001). The outcomes indicated that the interactions between human operators and the co-robot contributed to the mitigation of mental stress in collaborative assembly tasks, reducing the GSR levels from 0.95 to 0.54 and the NASA-TLX index from 10.8 to 6.6. Tukey HSD further confirmed that people experienced significantly higher mental stress in no-interaction sessions compared to all other six collaboration sessions that involved interactions.
ANOVA results of the presence of interactions.
Furthermore, two-way repeated measures ANOVA was performed to analyze the effects of interaction complexity and interaction modality, as shown in Figure 6 and Figure 7, respectively. In terms of interaction complexity, the results revealed that the effects on mental stress were statistically significant (for objective assessment, F(1,44) = 6.2035, p=0.0166; for subjective assessment, F(1,44) = 30.84, p<0.001). Compared to simple interaction, multiple and complex interactions had negative effects and induced relatively higher mental stress, resulting in a 0.12 increase in GSR levels and a 2.24 increase in the NASA-TLX index.
ANOVA results of interaction complexity (*: statistically significant with p<0.05; **: statistically significant with p<0.01).
ANOVA results of interaction modality: button, gesture, and voice (*: statistically significant with p<0.05).
Likewise, the results indicated significant effects of interaction modality on mental stress (for objective assessment, F(2,44) = 3.6899, p=0.033; for subjective assessment, F(2,44) = 3.3528, p = 0.0441). However, Tukey HSD post-hoc tests showed different results regarding objective and subjective assessment. For physiological GSR data, Tukey HSD confirmed that giving verbal commands elicited significantly higher stress than pressing buttons. In contrast, for NASA-TLX, Tukey HSD confirmed that showing hand gestures induces significantly higher stress than pressing buttons. Meanwhile, no significant differences were found between showing hand gestures and giving verbal commands, and the interaction effect of complexity and modality was also not significant.
Discussion
The physiological GSR results indicated that people experienced significantly higher mental stress when working with co-robots without interaction. This finding was also confirmed by self-reporting using the NASA-TLX index. In the absence of interaction, human operators must adapt to the pace of the co-robot and take over the Lego blocks in time to prevent them from falling, which imposes more mental demands on operators. In contrast, in the presence of interaction, the human operator can communicate with the co-robot when they need the co-robot to deliver the next Lego block or open its gripper. Thereby, human operators can adjust the pace of the HRC process according to their needs, which helps reduce mental stress during the collaboration. This finding is consistent with the results of Gervasi’s (2022) study that the introduction of HRI in collaborative tasks reduced perceived discomfort compared to scenarios without interaction.
In terms of interaction complexity, objective and subjective assessment came to the same conclusion. Although the introduction of interactions in HRC can help reduce mental stress, multiple and complex interactions can lead to relatively higher levels of mental stress compared to simple interactions. Nomura (2008) also found similar results that the anxiety increased after repeated interaction with a robot. Interactions with collaborative robots can also be a source of stress for human operators, so the level and extent of interactions in human-robot collaboration need to be carefully designed. For example, one should identify risk-critical subtasks in HRC, such as opening the gripper, and only introduce interactions to these subtasks for alleviating workers’ mental stress.
The results also showed statistically significant differences among different interaction modalities. Compared to the communication modalities of using hand gestures and speaking, the traditional communication style of button pressing elicited significantly less mental stress, as confirmed by both objective and subjective assessments. Similarly, in a study recently conducted by Xuan (2019), it was noted that using gestures to control a TV in a home environment setting caused more mental stress than using a remote button control panel. In addition, there was no significant difference found between using hand gestures and verbal commands. However, the subjective assessment showed a different trend between using gestures and speaking compared to the objective assessment, i.e., in the subjective assessment, verbal commands caused more mental stress than showing gestures, whereas in the objective assessment, using gestures caused higher mental stress. A possible reason could be the relatively small sample size, as only ten participants took part in this study.
Conclusion
Collaborative robots are increasingly being deployed in a variety of industrial settings, sharing a common workplace with human operators. Research focusing on aspects of human psychological effects is critical to achieving workplace wellness for HRC. This study investigated the impact of human-robot interactions on mental stress during collaborative Lego assembly tasks with different levels of interaction presence, interaction complexity, and interaction modality. Subjective measures using the NASA-TLX index and objective measures using the physiological GSR data were applied to assess the level of mental stress. The findings provided practical insights for optimizing the HRC process regarding reducing mental stress on human teammates. It is concluded that interactions between human operators and the co-robot can effectively reduce operators’ mental stress. In addition, it is recommended that interactions should be integrated into subtasks that are risk-critical, and redundant interactions should be avoided. Last but not least, novel HRI modalities, such as showing hand gestures and giving verbal commands, can lead to higher mental stress than pressing buttons, which should be taken into consideration when designing HRI paradigms.
