Empirical study for human engagement in collaborative robot programming

3.1 Repertory grid

To fully understand a smart cobot programming system, we need to collect a proper set of knowledge from cobot experts, while considering that human experts often have personal bias. Repertory Grid (RG) (Repertory Grids, n.d.) is used to understand the smart cobot programming system and to collect knowledge for requirements and specifications. RG is a cognitive interviewing technique devised by George Kelly and based on Personal Construct Psychology in the context of psychotherapy. RG is not restricted to clinical psychology and has been applied to education, market research, entertainment computing (Mol et al., 2021), and socio-technical systems (Dey & Lee, 2017). RG allows the interviewees (cobot experts) to interpret his or her cobot programming experience in a less biased way. Specifically, the RG analysis is used to identify cobot programming characteristics (as constructs) explained in Session 3.1.1 and programming approaches (as elements) explained in Session 3.1.2, and to cluster the constructs and elements, explained in Session 3.1.3, to obtain requirements for smart cobot programming system.

The RG analysis requires four main components: topic selection; element selection; construct selection; and rating and analysis. Here, the topic of the RG process is “analyzing cobot programming approaches”, which is aligned with our purpose. Section 3.1.1 lists different programming approaches with a description as elements. Section 3.1.2 defines the constructs, which are used to differentiate the programming approaches (elements). The constructs are identified by conducting a bipolar comparison to distinguish the pairs of elements. Section 3.1.3 illustrates the RG results showing the similarity of constructs, related construct clusters, and insights obtained.

3.1.3 RG analysis results and observations

After RG rating, the RG table was analyzed with R software (R: The R Project for Statistical Computing, n.d.) and OpenRepGrid package (Burr et al., 2020). The analysis results are presented as a Bertin matrix (Clustering of Constructs and Elements, n.d.). There are two distance trees for elements and constructs, each at the bottom and right sides of the matrix table (Fig. 1). The default cluster analysis method, Euclidean, is used. Euclidean measures a squared distance between two vectors, which is from the respondent ratings. The shorter Euclidean distance between the constructors, the more similar they are (http://docu.openrepgrid.org/clustering.html). Four clusters show strong connections based on the similarity method mentioned above, and meaningful constructors for the study are marked in red boxes Fig. 1. RG clustered Bertin matrix table with related constructs cluster marked. Some noticeable observations are summarized and interpreted below for further discussion.

OPEN IN VIEWER

Fig. 1

RG clustered Bertin matrix table with related constructs cluster marked.

The precise positioning of the cobot and the time for program validation are interrelated.

The development cost for the system user interface and a situational awareness system are pertinent, and they may be related to reliable programming codes.

The cobot programming time is connected to the cobot idle time during manual program updating, and they may also be connected to the shape intricacy level.

The difficulty of programming adjustment and the programmer’s emotional workload are connected.

Reviewing the four clusters indicated in the Bertin matrix, the RG study provided us with additional insights related to the smart cobot programming system. For example, as more human-focused cobot systems dominate the market, the human-related constructs should be investigated further to achieve a human-friendly workspace (Cluster #2 and Cluster #4). These constructs represent some essential requirements, such as a cobot programming user interface, monitoring cameras, recognition algorithm, decision making, and deliverable programming codes in a physical smart cobot programming system.

For Cluster #4, a programmer with a high emotional workload may have more complexity and difficulty to perform program adjustments while interacting with the cobot. For Cluster #2, the low development cost for a system user interface is related to a non-situational awareness system and a non-reliable programming code. Converting these two cluster results into a physical system, a user interface should provide easy programming adjustment as it relates to operator mental concerns, and sensor technology should be included to collect operator status and provide feedback. These observations are considered to identify the requirements of the smart cobot programming systems, discussed in more detail in the next section. Although the RG analysis shows some similarity between some constructs, specific cause and effect relationships and conflicting relationships may not be well captured. These aspects will be studied further and reported as a separate article.

To conceptualize and visualize the smart cobot programming system requirements obtained from the RG analysis, we first build a concept map for the groups of related insightful information and the requirements identified, Fig. 2. The concepts are named based on observations from the RG analysis and related literature study and their relationships are indicated in a simple map. As an example, the “cobot” in the concept map’s first layer subject has several sub-subjects: “programming method”, “Programming user interface” and “Programming codes,” which are extracted from our RG analysis table. As mentioned in the previous section, the difficulty of cobot programming involves a substantial relationship with the programmer’s emotional workload (indicated from Cluster #2), and the concept map includes human operator sensing. Based on the aforementioned information, we concluded a smart cobot programming system requires several aspects, including human operators, machine hardware, and data collection tools in the concept map, Fig. 2.

OPEN IN VIEWER

Fig. 2

A concept map of smart cobot programming system with potential components.

Here, we briefly explain the process of the concept map construction. In the concept map, a smart human-robot collaboration system is the central topic (i.e., a starting point). Obviously, the system should contain humans and cobots. Data, for the humans and cobots, and data analysis is included in the concept map. To gather data, advanced sensor technologies are necessary. In a human-cobot system, the recognition target is important to decide at the beginning of the process of development. After these concepts are included in the map, more detailed information is included to enrich the concept map as shown in Fig. 2. According to the constructed concept map, we can detail the requirements and functionality specifications of a smart cobot programming system. Some examples are shown below.

A smart cobot programming system should be able to perceive the environmental events with respect to time or space in a data form.

4.1 System architecture

The smart cobot programming system integrates sensor data acquisition and data analysis process. Essentially, the system collects and utilizes information from humans and cobots to assist programming tasks in a work cell, as indicated in the requirements. In Fig. 3, the system structure is divided into three layers: physical, transformation, and analytics. The physical layer corresponds to the physical assets in the programming work environment. The programmer is the system end-user, cobots are the task actuators, and the task corresponds to sequence planning, workspace design, and process parameters. The transformation layer represents the translation of the respective physical asset metrics into valuable metrics. The last layer, analytics, incorporates asset modeling and performance classification.

OPEN IN VIEWER

Fig. 3

Human engagement integrated smart human-cobot training system.

The system is designed for users who program and assist cobots in complex human-robot tasks. To improve cobot involvement in the task, the system aims to optimize user programming efficiency and task performance by raising their programming knowledge. A better understanding of the cobot-task interaction can support cobots to achieve more complex tasks and decrease human effort at the same time. Consequently, it is critical that the system tracks human engagement status to understand the human programming conditions. According to the RG discussion mentioned in the previous sections, the system requires detection of whether a human is becoming distracted or disengaged and whether the task brings mental and emotional stress to the human. The system enables programmers to know if the current procedural task is appropriate for their status and capacity at the time, or if they should enhance their focus.

Cobots cooperate with human workers to complete repetitive tasks according to pre-set programs. However, traditional cobot systems do not consider the physical and mental condition of human workers. If humans are unable to perceive such conditions, human-cobot collaboration cannot be more efficient. The system in this paper, on the other hand, is designed to eliminate this gap through continuous human performance insight feedback to programmers and a self-adjusting process based on human engagement. In addition to human information, cobot information is collected to complete the cobot programming insight feedback loop. Cobot-related information is obtained through built-in and external sensors that monitor cobot components and process status, such as trajectory, current position, and speed. This information is critical to analyzing programmer performance, including programming time, process cycle time, and script quality.

The proposed system also considers the workspace and human model when evaluating cobot data in the process performance analysis. Workspace defines task burdens and environment details into the system, such as descriptive tasks and handled object position. Certain tasks require more programmer attention due to the high accuracy level requirements (e.g., manipulating a cobot’s tool in a low process tolerance) or complex sequence planning (e.g., assembling large numbers of components). Workspace data, then, is critical to systematically define subtask boundaries. A human model includes programming skill level, programming preference, and engagement behavior. The first two factors of the human analysis are a qualitative evaluation that can be performed through interviews, while the last one is obtained using human sensors. The proposed system uses web cameras as human sensors to capture real-time video of human-cobot interactions. During a task, facial recognition algorithms are used to provide and measure real-time human engagement metrics.

In this study, a customized OpenFace (Baltrusaitis et al., 2018) process and the algorithm based on work in Huynh et al. are employed to estimate human engagement (Thong Huynh et al., 2019). The gaze direction, eye landmarks, and head pose are calculated for every frame in the video with OpenFace and sent to a python module in real-time. Over a specific time, pre-defined (e.g., five seconds), the python module estimates the engagement score for that interval by dividing the received sequence into multiple overlap segments with equal length. Then, statistical features are computed for each segment and fed to two streams of a deep learning model as illustrated in Fig. 4. Each stream contains two long short-term memory (LSTM) layers for learning the temporal relationship between segments and two fully connected (FC) layers used for estimating scores for each segment, then the results are averaged to calculate the engagement score. The goal of the first FC layer is to transfer the feature set into a new representation that is learnable and improves system performance.

OPEN IN VIEWER

Fig. 4

An overview of deep learning model for engagement estimation (Thong Huynh et al., 2019).

5.1 Cobot programming empirical study scenarios and experimentation setup

An empirical study has been developed and analyzed to demonstrate the smart cobot programming system. A flowchart of the programming evaluation process is shown in Fig. 5. Initially, two experiments were designed by cobot experts. Each of them required the 11 participants to program a cobot for a disassembly task. Each of the experiments had its own difficulty and goal. Once a disassembly task was selected, participants with different cobot programming experiences executed the disassembly programming and evaluation. Two cameras were used to record their engagement. In addition, cobot performance metrics were taken during and after programming. Then, participant engagement and cobot data were used to evaluate the smart cobot programming framework.

OPEN IN VIEWER

Fig. 5

Human-cobot programming empirical study scenario.

Experimental data in this study was collected from ten participants composed of beginners and advanced programmers. Each participant was given process goals to be completed using a Universal Robot (UR) 5E cobot, but they were not instructed on how to complete each process. As a minimum requirement, cobot movement (controller and hand-guidance movements) and tool commands were taught to participants before the cobot programming evaluation. Before the participants began, the participants were allowed to ask questions to the experiment supervisor. The supervisor was responsible for setting up the testbed, including connecting the cobot and camera to the computer and ensuring that experimental data was being recorded.

After setup, the supervisor was instructed to leave the work area and let the participants complete the task without further assistance. During the experiment, participant engagement scores were obtained with OpenFace while cobot-related data was gathered through cobot supporting software, UR Log Viewer (Universal Robots - Log Viewer V1.2.1.0, n.d.), and teach pendant. When the task was completed, the participants notified the supervisor to stop the data collection regardless of whether they were successful or not.

Two disassembly tasks were designed based on cobot command and sequence complexity. The first task is shown in Fig. 6, where Fig. 6a represents the initial state of the process and Fig. 6b represents the final object location. As the first level of complexity, this task required the programmer to use basic cobot commands to pick an object and place it in a different location. It tested the participant’s ability to understand cobot movement and gripping commands. The user had to take a ball bearing from the shaft of a partially disassembled alternator and place it in a specific location to successfully complete the task, as displayed in Fig. 6b. The bearing was moved using the cobot gripper, and it could not collide with the alternator or perform a movement that would result in a cobot stop during the process. The cobot started from a pre-determined home position and the task was completed when the cobot was returned to the home position.

OPEN IN VIEWER

Fig. 6

First cobot disassembly programming task. (a) initial step; (b) second state.

The second, more complex, disassembly task is shown in Fig. 7a-d. This task required more advanced cobot programming knowledge and operation experience. It maintained the same programming abilities; however, with more constrained movements. The cobot had to grip four pins located at the corners of an object and place them into a box, Fig. 7b. The pinhole diameter was 1 millimeter (0.04 inches) larger than the pin body, which required the user to precisely retrieve the pin without the gripper dropping the pin by colliding with the main frame. The second step required participants to remove the frame using their own hand and place it in the designated area (Fig. 7c). The third step was to remove four more pins on the base plate and place them into the box (Fig. 7d). Similar to the first task, the participants created a cobot program script starting from and returning to the home position.

OPEN IN VIEWER

Fig. 7

Second cobot disassembly programming task. (a) components description in the object partially disassembled; (b) initial assembly state; (c) object state after disassembling four external pins; (d) object state after disassembling the frame component.

Figure 8 shows the workspace designed for the cobot programming empirical study. It consists of a cobot UR5e, a work object, two cameras, and a laptop. The laptop had OpenFace and UR Log Viewer installed for human and cobot monitoring. The two cameras were connected to the computer through USB cables and each camera had independent facial recognition algorithms. The cobot was connected via controller-laptop through ethernet cables and established a connection with the UR Log Viewer. The UR5e end-effector was a Robotic Hand-E gripper. Participants were required to use only the teach pendant or hand-guidance movements for programming. Position, speed, acceleration, joint torque, and other data were captured through the UR Log Viewer during the programming phase.

OPEN IN VIEWER

Fig. 8

Empirical study workspace setup.

OpenFace software measured the participant’s engagement during cobot programming based on facial features. As displayed in Fig. 8, one camera was placed on the teach pendant and the other camera had a stand placed beside the cobot. The camera layout aimed to capture the programmer’s facial features when focusing on specific process aspects, either the programming interface or the cobot-object interaction.

Plastic strips and blue lines were used to limit the workspace and keep participants within the camera’s field of view. The plastic strips emitted a sound when the participant stepped on them as a reminder to return to the boundary. It was used to avoid losing track of users’ faces while not dramatically deviating participant attention from the cobot programming task. The laptop was placed outside of the camera view so the experiment supervisor could observe the user’s behavior.

Based on cluster results and the system requirements described in Section 3, there was a need to receive human inputs to capture emotional workload via task engagement and to evaluate the program accuracy and complexity via cobot program development and execution measures. Therefore, four parameters were selected in this study to reflect the state of the programmer when programming a cobot, Table 3. Programming time was measured using the video recording and the movement of the cobot from and to the home position. This parameter was used as a measure of the difficulty of the cobot programming task for the participants (i.e., the more challenging the task, the longer the programming time). Two engagement scores, each from one camera, were also collected to reflect the human-task engagement. Decision time recorded the length of time participants spent in understanding task requirements, thinking about solutions, and planning the cobot trajectory. It was measured by summing the time that the cobot remained static during the programming time.

Three cobot parameters were used to evaluate the cobot operation during the experiments: process cycle time, task success, and script score, Table 4. The process cycle time was defined as the amount of time the cobot took to complete the task. It was extracted from the final cobot program script execution. Cobot programs should avoid longer paths, leading to a shorter process cycle time. Task success referred to whether the cobot could complete the task according to the program script written by the participant. It was a binary parameter, where 0 indicated failure of task and 1 success. Failure conditions included collisions, parts falling, and parts being placed at incorrect locations. The script (cobot program) score was graded based on the number of lines and commands utilized. The script score ranged between 0 and 1. Since a shorter script is preferred, a higher score represents a more concise program.

Different behaviors among the programmers with distinct levels of experience were observed from the engagement data. The typical engagement scores of an advanced and a beginner programmer are displayed in Figs. 9 and 10, object camera and controller camera, respectively. These figures exemplify the behavior of having the average value of controller engagement higher than the object engagement for advanced programmers and having the opposite be observed for beginner programmers. The cause of this behavior is hypothesized to be due to the user focus being directed towards the less confident areas of the workspace, which increases the respective camera metric. In the experiments, the beginners dedicated more time to looking at the commands and deciding which action to take while the experts dedicated more time looking at the object and the cobot position during the tasks. The second observable difference was that beginner programmer had more frequent oscillation in controller engagement score. The oscillation was related to the redirection of the face from the controller to the cobot position, and vice versa, while moving the cobot using the teach pendant. This pattern was caused by their inexperience in how the cobot should be positioned in a certain location causing a rerouting of the cobot trajectory.

OPEN IN VIEWER

Fig. 9

Engagement scores of object camera.

OPEN IN VIEWER

Fig. 10

Engagement scores of controller camera.

Similar engagement patterns were also observed across users with different experience levels. During the programming task, participant behavior can be classified into three actions: programming, teach pendant movement, and hand-guidance movement. The first action was defined by looking for and selecting the commands at the teach pendant. It combined both the time for selecting a command and the decision-making time for choosing the next action. The second and third differentiated the way the cobot was moved, defined by selecting the directions on the teach pendant or moving the cobot by using the programmer’s hands, respectively. According to the nature of the action, the user’s attention was focused on the teach pendant, object, or both. Consequently, it was observed that the object engagement score was classified as low, medium, and high for programming, teaching pendant, and hand-guidance movement respectively. Furthermore, this pattern was observed independent of the programmer’s level of expertise.

There were observable points in time during the collection of the engagement data that the camera could not retrieve a score. The inability to produce a score was due to the requirement that the camera track movement during the entire collection interval period, set as three seconds in this experiment. If the camera lost face tracking for more than the collection interval, the engagement score would be a zero value. This occurred in two situations: (1) face tracking obstruction and (2) loss of attention. The first occurred when the user positioned their face close to the object location, blocking the camera view with the cobot arm. The second situation, mostly on the controller camera, happened when the user was distracted due to an external disturbance, such as a conversation or environmental noise. Both situations were considered a cause of task disengagement and they were not ignored during the programming assessment.

Results collected from both tasks are displayed in Fig. 11. The data obtained from the experiments were also aggregated by the programming experience due to the similarities in the engagement behavior. The expert group was represented by two individuals. The beginner group had seven participants with none to very little programming experience with any type of robot. Programming time, decision time, and process cycle time parameters were transformed using max-min normalization among all the data collected to allow comparison between the two classes. There were experiments that took sixteen minutes and others that were accomplished in less than two minutes, the normalization provided improved visualization of the difference between the programmer classes. The average, minimum, and maximum points from each engagement collection were also determined and plotted. Due to the unique analysis of the engagement behavior in this case scenario, the time series analysis was not covered by this work. Instead, a basic statistical analysis was completed, which provided an initial understanding of the relationship between the programmer class and the engagement behavior.

OPEN IN VIEWER

Fig. 11

Human and cobot data collected during experiments.

Furthermore, the advanced programmer class was used to benchmark the optimal human and cobot performance to incorporate human programming insight in the smart cobot programming system. Ideally, any user would reach the same values as the advanced programmer with proper training. Consequently, a beginner programmer should work on or be assisted with program aspects that have the most disparity between the user and the benchmark class. However, there were some parameters with a wider gap between beginner and benchmark users, such as process cycle time, average controller engagement, and minimum controller engagement. The narrower the parameter’s deviation, the less variance a beginner is allowed and the higher the effort required to be reached the advanced mean value.

Of the non-engagement parameters, the script score and decision time require knowledge and confidence in the process. They can be related to the maximum controller engagement score when associating the feeling of success of using human training memory to solve a programming problem. Expanding this association, they presented a lower deviation compared to other parameters, which could be a key evaluation for identifying an expertise level of a chosen programmer. The programming time and process cycle time, on the other hand, are the quantitative performance classifiers. Although they have high variation among the experiments, there are distinct data regions for beginner and expert programmers. Better performance by experts is hypothesized to be due to a better understanding of how to position the cobot efficiently and where to move it during the entire task. This object focus insight is visualized on the average object engagement, which also had different regions for both classes.