Operator Engagement Detection for Robot Behavior Adaptation

4.1 Cognitive Tasks

The aim of the tasks was to invoke in the participants the following five affective states: engagement, anxiety, boredom, frustration and anger. The tasks chosen were solving anagrams and playing Pong. The anagram-solving task has been previously employed to explore relationships between both electrodermal and cardiovascular activity with mental anxiety (Pecchinenda, A., & Smith, C. A., 1996). Emotional responses were manipulated in this task by presenting the participant with anagrams of varying difficulty levels, as established through pilot work. A long series of trivially easy anagrams caused boredom, an optimal mix of solvable and difficult anagrams caused engagement, unsolvable or extremely difficult anagrams elicited frustration and giving time deadlines generated anxiety. All these conditions were well tested during the development stage of the task design and piloting.

The Pong task consisted of a series of trials each lasting up to four minutes, in which the participant played a variant of the early, classic video game “Pong”. This game has been used in the past by researchers to study anxiety, performance, and gender differences (Brown, R. M. et al, 1997). Various parameters of the game were manipulated to elicit the required affective responses. These included: ball speed and size, paddle speed and size, sluggish or over-responsive keyboard and random keyboard response. Low speeds and large sizes of ball and paddle made games boring after a while, whereas high speed ball and paddle along with smaller sizes of the two made the game engaging. Very high speeds caused anxiety at times. Sluggish or over-responsive keyboard induced frustration and anger. The relative difficulties of various trial configurations were established through pilot work.

During the experiment, participants were presented with cognitive computer tasks that elicited a variety of affective responses. Six participants (four women and two men) took part in the experiment. Their age range was from 24 to 45 years. After initial briefing regarding the computer tasks, sensors were attached to the participant's body. Each participant took part in six sessions of the above two tasks – three one hour sessions of solving anagrams and three one hour sessions of playing Pong. These tasks spanned a period of one month. In each session, before starting the actual tasks baseline recording was done which was used later to offset day-variability. Each session consisted of 3 minute epochs followed by a questionnaire for self-reporting. During the tasks, the participant's physiology was monitored with the help of wearable biofeedback sensors and Biopac data acquisition system (www.biopac.com). The Biopac sensors are small, easy to wear and unobtrusive during the tasks. The digitally sampled sensor information was sent serially to the computer using an Ethernet cable. The signals monitored consisted of electrocardiogram, bio-impedance, electromyogram (from the corrugator, zygomaticus and upper trapezius muscles), galvanic skin response, peripheral temperature, blood volume pulse, and heart sound.

During the tasks, the participants periodically reported their perceived subjective emotional states. This information was collected using a battery of five self-report questions (regarding their perceived anxiety, engagement, anger, frustration and boredom) rated on a nine-point Likert scale. Self-reports were used as reference points to link the objective physiological data to participants' subjective affective state. Each task sequence was subdivided into a series of discrete epochs that were bounded by the self-reported affective state assessments. These assessments occurred every three minutes for the anagram task and every 2–4 minutes for the Pong task. The participants reported their affective state on a scale of 1–9 where 1 indicated the lowest level and 9 indicated the maximum level.

4.2 Physiological Basis and Signal Measurement

There is good evidence that the physiological activity associated with affective state can be differentiated and systematically organized (Bradley, M. M., 2000). The transition from one emotional state to another is generally accompanied by dynamic shifts in indicators of Autonomic Nervous System (ANS) activity. The physiological signals we examined were: various features of cardiovascular activity, including interbeat interval, relative pulse volume, pulse transit time, heart sound, and pre-ejection period; electrodermal activity (tonic and phasic response from skin conductance) and electromyogram (EMG) activity (from corrugator supercilii, zygomaticus, and upper trapezius muscles). These signals were selected because they were likely to demonstrate variability as a function of our targeted affective states as well as they could be measured non-invasively and were relatively resistant to movement artifact. A detailed description of the physiological signals and features can be found in our previous work (Rani, P. et al, 2004). Multiple features as shown in Table 1 were derived for each physiological measure. “Sym” is the power associated with the sympathetic nervous system activity of the heart (in the frequency band 0.04–0.15 Hz.). “Para” is the power associated with the heart's parasympathetic nervous system activity (in the frequency band 0.15–0.4 Hz.). InterBeat Interval (IBI) is the time interval in milliseconds between two “R” waves in the ECG waveform in millisecond. IBI ECGmean and IBI ECGstd are the mean and standard deviation of the IBI. Photoplethysmograph signal (PPG) measures changes in the volume of blood in the fingertip associated with the pulse cycle, and it provides an index of the relative constriction versus dilation of the blood vessels in the periphery. Pulse transit time (PTT) is the time it takes for the pulse pressure wave to travel from the heart to the periphery, and it is estimated by computing the time between systole at the heart (as indicated by the R-wave of the ECG) and the peak of the pulse wave reaching the peripheral site where PPG is being measured. Heart Sound signal measures sounds generated during each heartbeat. These sounds are produced by blood turbulence primarily due to the closing of the valves within the heart. The features extracted from the heart sound signal consisted of the mean and standard deviation of the 3rd, 4th, and 5th level coefficients of the Daubechies wavelet transform. Bioelectrical impedance analysis (BIA) measures the impedance or opposition to the flow of an electric current through the body fluids contained mainly in the lean and fat tissue. A common variable in recent psychophysiology research, pre-ejection period (PEP) derived from impedance cardiogram (ICG) and ECG measures the latency between the onset of electromechanical systole, and the onset of left-ventricular ejection and is most heavily influenced by sympathetic innervation of the heart. Electrodermal activity consists of two main components – Tonic response and Phasic skin conductance response. Tonic skin conductance refers to the ongoing or the baseline level of skin conductance in the absence of any particular discrete environmental events. Phasic skin conductance refers to the event related changes that occur, caused by a momentary increase in skin conductance (resembling a peak). The EMG signal from Corrugator Supercilii muscle (eyebrow) captures a person's frown and detects the tension in that region. It is also a valuable source of blink information and helps us determine the blink rate. The EMG signal from the Zygomaticus Major muscle captures the muscle movements while smiling. Upper Trapezius muscle activity measures the tension in the shoulders, one of the most common sites in the body for developing stress. The useful features derived from EMG activity were: mean, slope, standard deviation, mean frequency and median frequency. Blink movement could be detected from the Corrugator Supercilii activity. Mean amplitude of blink activity and mean interblink interval were also calculated from Corrugator EMG.

Various signal processing techniques such as Fourier transform, wavelet transform, thresholding, and peak detection, were used to derive relevant features from the physiological signals. All these features mentioned above are powerful indicators of the underlying affective state of the person showing this response. We have exploited this dependence of a person's physiological response on affect to detect and identify engagement in real-time

Some of the above-mentioned physiological signals and a few others have been used by researchers in the past to study attention/engagement in people (Fredriskon, M. et al 1986, Pope, A. T. et al, 1995). Spontaneous skin conductance activity has been shown to be associated with task engagement (Pecchinenda, A., & Smith, C. A., 1996). It was also shown in a later work by Kirby and Smith that increased effort and increased memory load caused an increase in skin conductance responses (SCRs) within a trial; on the other hand, increased vigilance was associated with fewer SCRs (Kirby, L. D., & Smith, C. A., 1997). Fairclough and Venables examined a range of psychophysiological responses – electroencephalogram (EEG), electrocardiogram (ECG), Skin Conductance (SC), electro-oculogram (EOG), and respiratory rate and found that psychophysiology predicted a significant proportion of the variance for task engagement (Fairclough, S. H. & Venables L., 2005). Iani et al. found a significant difference in blood pulse volume amplitude between rest and task periods, suggesting that the measure reflected changes in sympathetic activity due to task engagement (Iani, C. et al, 2004).

Their results also indicated that reduced pulse wave amplitude, signaling vasoconstriction, resulted when participants spent effort. Many of these studies provide a basis for our investigation. However, unlike our work, none of the above-mentioned studies attempts to infer the underlying psychological state in real-time. Furthermore, none of these works is in the field of human-robot interaction.

Various methods of extracting physiological features exist but efforts to identify the exact markers related to emotions, such as anger, fear, or sadness have not been successful chiefly due to person-stereotypy and situation-stereotypy

(Lacey, J. L. & Lacey, B. C., 1958). That is, within a given context, different individuals express the same emotion with different characteristic response patterns (person-stereotypy). In a similar manner, across contexts the same individual may express the same emotion differentially, with different contexts causing characteristic responses (situation-stereotypy). The novelty of the presented affect-recognition system is that it is both individual- and context-specific in order to accommodate the differences encountered in emotion expression. It is expected that in the future with enough data and understanding, affect recognizers for a class of people can be developed.

The participants periodically reported their perceived subjective emotional states. This self-report was collected using a battery of fourteen questions rated on nine-point Likert scales. These questions asked them to report their emotions such as anxiety, challenge, and engagement as related to the task that they just performed. An engagement index was determined from the rating that the participants provided regarding their engagement during the task.

4.3. Engagement Prediction based on Regression Tree

In the previous research works in emotion recognition, change in emotion has been considered either along a continuous dimension (e.g., valence or arousal) or among discrete states. Various machine learning and pattern recognition methods have been applied for determining the underlying affective state from cues such as facial expressions, vocal intonations, and physiology. Fuzzy logic has been employed for emotion recognition from facial expression (Moriyama, T. et al 1999). Fuzzy logic has also been used to detect anxiety from physiological signals by our research group 0 and by Hudlicka et al. in (Hudlicka, E., & McNeese, M. D., 2002). There are several works on emotion detection from speech based on k-nearest neighbors algorithm (Petrushin, V. A., 2000), linear and nonlinear regression analysis (Rani, P. et al 2003). Discriminant analysis has also been used to detect discrete emotional states from physiological measures (Ark, W. et al 1999). A combination of Sequential Floating Forward Search and Fisher Projection methods was presented in (Vyzas, E., & Picard, R. W., 1998) to analyze affective psychological states. Neural networks have been extensively used in detecting facial expression (Zhao, J., & Kearney, G., 1996), facial expression and voice quality (Fellenz, W., A. et al, 2000). The Bayesian approach to emotion detection is another important analysis tool that has been used successfully. In (Qi, Y., & Picard, R., 2002) a Bayesian classification method was employed to predict the frustration level of computer users based on pressure signals from mouse sensors. A Naïve Bayes classifier was used to predict emotions based on facial expressions (Sebe, N. et al, 2002. A Hidden Markov Model based emotion detection technique was investigated for emotion recognition (Cohen, I. et al, 2000).

In this paper we have used regression trees (Breiman, L., 1993) (also known as decision trees) to determine a person's affective state from a set of features derived from physiological signals. The choice of regression tree method emerges from our previous comparative study of four machine learning methods- K-Nearest Neighbor, Regression Tree, Bayesian Network and Support Vector Machine as applied to the domain of affect recognition (Liu, C. et al 2005). The results showed that regression tree technique gave the second best classification accuracy − 83.5% (after Support Vector Machines that showed 85.8% accuracy) and was most space and time efficient. Regression tree method has not been employed before for physiology-based affect detection and recognition. Person-specific regression trees were created to handle the occurrence of person-stereotypy.

Regression tree learning, a frequently used inductive inference method, approximates discrete valued functions that adapt well to noisy data and are capable of learning disjunctive expressions. For the regression tree-based affect recognizer that we built, the input consisted of the physiological feature set and the target function consisted of the affect levels (participant's self-reports that represented the participant's assessment of his/her own affective state). The main challenge was the complex nature of the input physiological data sets. This complexity was primarily due to the (i) high dimensionality of the input feature space (there are currently forty-six features and this will increase as the number of affect detection modalities increases.), (ii) mixture of data types, and (iii) nonstandard data structures. Additionally, a few physiological data sets were noisy where the biofeedback sensors had picked up some movement artifacts. These data sets had to be discarded, resulting in the missing attributes.

In this work, we used regression tree to determine a participant's level of engagement fro his/her physiological response. The objective was to train a regression tree such that it could accept as input an array of feature vectors derived from physiological signals and provide as input the engagement level associated with it. The output was a discrete number in the range 0–9 where 0 signified lowest level and 9 signified highest level of engagement. The steps involved in building a regression tree are shown in Fig. 1. Physiological signals recorded from the participant engaged in PC-based task were processed to extract the input feature set. The participant's self-report at the end of each epoch regarding his/her affective states provided the target variable or the output. While creating the tree, two primary issues were: (i) Choosing the best attribute to split the examples at each stage, and (ii) Avoiding data over fitting. Many different criteria could be defined for selecting the best split at each node. In this work, Gini Index (Breiman, T., 1993) function was used to evaluate the goodness of all the possible split points along all the attributes. For a dataset D consisting of n records, each belonging to one of the m classes, the Gini Index can be defined as:

G i n i ( D ) = 1 − ∑ i = 1 m p i 2

(1)

where pi is the relative frequency of class i in D. If D is partitioned into two subsets D1 and D2 based on a particular useful attribute, the index of the partitioned data Gini(D,C) can be obtained by:

G i n i ( D , C ) = n 1 n G i n i ( D 1 ) + n 2 n G i n i ( D 2 )

(2)

where n1 and n2 are the number of examples of D1 and D2, respectively, and C is the splitting criterion. Here the attribute with the minimum Gini Index was chosen as the best attribute to split. We used the Statistics Toolbox of Matlab for regression tree functions such as generating, pruning and evaluating regression trees (www.mathworks.com). Trees were pruned based on an optimal pruning scheme that first pruned branches that gave the least improvement in error cost. Pruning was performed to remove the redundant nodes as bigger, overfitted trees have higher misclassification rates. Thus, based on the input set of physiological features described earlier, the affect recognizer provided a quantitative understanding of the person's affective states.

OPEN IN VIEWER

Fig. 1.

Creating a Regression Tree

5.1 Control Architecture for Implicit Human-Robot Interaction

For a human-robot interaction to mimic similar human-human interaction, it is essential that both the robot and the human have implicit as well as explicit communication with each other. This requires a systematic information flow between the human, robot and the environment. A generalized model of human-machine system developed by Riley (Riley, V., 1989) represents such an information flow that can be systematically modified according to any rule-base to represent a particular level of automation in human-machine interaction. It is a powerful front-end analysis method that can be employed to identify human-machine protocols as well as the automation concerns that accompany the design of such systems. The general model represents the most complex level of automation embedded in the most complicated form of human-machine interaction. We modified this model to represent the specific system developed for human-Oracle interaction in a teleoperation experiment. This system mimicked a teleoperation task where an operator teleoperated a mobile robot in a given workspace. In this case Oracle is expected to behave as an intelligent partner to the operator. This requires Oracle to respond appropriately to the engagement levels of the operator while not undermining the importance of its own safety and goals. The reduced architecture is shown in Figure 1.

As seen in Figure 2, in the top-left “robot input” quadrant, Oracle receives information from both the world and the operator through various sensors. The world information is obtained through the infrared sensors, touch sensors, light sensors etc. Oracle receives this information, infers the world state and forms a model of the world for itself. Operator information that is received by Oracle is of two types – control input and affective input. The control input contains teleoperation information and is provided to Oracle through a joystick. The affective information is obtained through biofeedback sensors that measure physiological signals of the operator, for instance- cardiac activity, electrodermal activity, muscle activity, temperature etc. This information is used to interpret the affective state of the operator, for example to judge whether the operator is getting inattentive, fatigued, or stressed (implicit communication). Oracle also receives explicit commands from the operator through the control sensors (explicit communication). Instead of blindly following every teleoperation command of the operator, Oracle continuously evaluates the engagement level of the operator and modifies its behavior accordingly. On detecting low levels of engagement, Oracle alerts the operator and in some cases can even stop itself.

OPEN IN VIEWER

Fig. 2.

Human-Robot Interaction Framework (Modified From Riley's Original Information Flow Model)

The “robot output” quadrant contains nodes that determine the behavior of Oracle. Oracle uses its representation of the world, knowledge of its own goals and the urgency level of the situation to determine the best course of action. Oracle is assisted by a set of rules to determine its behavior as shown in Figure 3. In our previous works 0,0 we have employed a hybrid subsumption architecture model to determine Oracle's behavior. Oracle's decision results either in an action or initiation of speech on its part. In the “human input” quadrant the human receives information both from the world and Oracle. The human can perceive the dialogues initiated by Oracle and observe its behavior. He/she then exploits this knowledge to infer Oracle's state as well as predict what it might do next. This inference, the world representation that the human forms, and his/her own ultimate goals are all employed by the human to determine the next action. The resulting human's action might be simply to monitor Oracle's actions or issue a command to it. Hence, in each cycle of the loop, there is a methodical information flow in between the world, Oracle and the human. At the very fundamental level, this is a sense-infer-plan-act loop wherein, Oracle and the human utilize the available information to interact with each other and to take actions that influence each other and the world.

OPEN IN VIEWER

Fig. 3.

Behavior Types for Mobile Robot

The objective of the experiment was to develop and implement real-time, affect-sensitive human-robot coordination during teleoperation. The robot was expected to recognize and respond to the varying levels of engagement of the human operator.

However, due to the limitations explained in Section III, the experiment was performed in two parts. First, we designed a regression tree based affect-recognizer that could identify or “understand” the physiological responses of a person. Second, we implemented a human-robot co-operation task consisting of Oracle being teleoperated by an operator/user whose (apparent) physiological state was being continuously monitored. The first part of the experiment is described in Section IV. The second part of the experiment consisted of implementing a real-time human-robot interaction during teleoperation. Oracle was supposed to recognize the operator's level of engagement through continuous physiological sensing, and act accordingly to address the situation. Since closed-loop experiment could not be performed, the physiological signals collected in the first part were streamed in at real-time speed to Oracle as if it were coming from the person controlling the robot. This allowed us to demonstrate that if such physiological signals are generated during a human-robot task, the robot could detect the engagement level of the operator and subsequently modify its behaviors. Controlling the Oracle was done by sending low-level commands from a desktop PC using a standard RS-232 serial port. Commands were wirelessly sent to Oracle by the operator that allowed control over the drive motors, heading motion, gripper, sensors, etc. The teleoperation task consisted of picking and depositing of objects (coke cans) by Oracle in a given workspace. Oracle was teleoperated by an operator via a joystick in accomplishing this task. While the teleoperation task continued, the physiological data collected in the first part of the experiment was sent to the robot as if coming from the operator in real time. The behavior modifications of Oracle depended upon the level of operator engagement detected. The continuous values obtained from regression tree were discretized to obtain three levels of engagement – high, medium and low. As shown in Figure 3 Oracle could switch between three types of behavior depending upon the level of operator engagement. When it detected high levels of engagement, it followed the operator's commands being sent to it. As the level of engagement decreased to medium level, it alerted the operator by speaking out its current movement, for instance, “I am turning left now.” When the level of engagement dipped lower, Oracle lowers its speed to a minimum after giving a warning to the operator and eventually stopped.

6.1 Engagement Detection

Forty nine features/indices were derived from the following measured physiological signals – ECG, EMG (corrugator, zygomaticus, and upper trapezius), peripheral temperature, galvanic skin response, blood pulse volume, heart sounds, and. bio-impedance. On determining the correlation between the indices derived from the physiological signals and the self reported level of engagement, it was found that certain physiological indices were well correlated with the state of engagement. A correlation of 0.3 and above was considered a significant value and was supported by previous research. Fig. 4 shows the correlation values for indices that were highly correlated with engagement for Participant 5. The useful indices for this participant were IBI - dz/dt (Interbeat Interval from the difference signal of bio-impedance), IBI- ECG (R-R interval from ECG), Tonic Mean (mean level of tonic skin conductance), Phasic Rate, Phasic Mean, Phasic Max (rate, mean amplitude and maximum amplitude of the phasic response of skin conductance), Temp std (standard deviation of the peripheral skin temperature), blink amp (mean amplitude of the eye blink response), and Zyg Med Freq (Median frequency of zygomaticus EMG).

OPEN IN VIEWER

Fig. 4.

Correlation of Physiological Indices with Self-Reported Engagement for Participant #1

For each of the six participants there were distinct physiological indices that showed significant correlation with their state of engagement. However, these indices were not identical for all participants due to the well-known phenomenon of subject-stereotypy (Within a given context, different individuals express the same emotion with different characteristic response patterns). Fig. 5 shows the correlations for Participant #1. Person-stereotypy can be seen on comparing the two plots.

OPEN IN VIEWER

Fig. 5.

Correlation of Physiological Indices with Self-Reported Engagement for Participant #5

Not only were the useful indices different for the two participants, the extent of correlation for the same index is different for each participant. It was observed that phasic rate was well correlated for all the participants. The other important indices were mean pulse transit time, mean blink amplitude, mean tonic level, and standard deviation of zygomaticus EMG. Such individual differences can be accounted for by the phenomena of person-stereotypy. We used regression tree methodology to compute a participant's level of engagement from a set of features derived from his/her physiological response. Our approach was to determine the specific pattern for each person and automatically create a regression tree according to individual responses.

It was also observed that based on the indices, it was possible to distinguish between different affective states, for instance between engagement and anxiety and between engagement and boredom. Due to space limitation, we only present the results of selected participants, even though the traits observed were generic. Fig. 5 shows that the indices that indicate engagement can be differentiated well from the state of boredom while Fig. 6 shows that engagement is also well differentiated from the state of anxiety for Participant #5. All the six participants showed clear distinction between the five measured affective states based on physiological indices.

OPEN IN VIEWER

Fig. 6.

Comparison between Correlation of Self-Reported Boredom and Self-Reported Engagement with Physiological Indices for P #5

During the teleoperation experiment, an operator would control Oracle via a joystick to perform a pick and place task. Oracle was supposed to pick specific objects (empty coke cans) from the workspace and place them in a specified area. At the same time, Oracle also received physiological feedback as if it were coming in real-time from the operator. Oracle modified its behaviors depending upon the perceived engagement level of the operator (Fig. 3). During the experiment, an engagement index inferred from the various physiological indices described before was computed every 30 seconds. The engagement index was computed using the regression tree that was previously created from the data collected during the PC based cognitive tasks. Self-reports of participants were available every three minutes to validate the correctness of the real-time algorithm used for engagement detection. There were 36 teleoperation trials done (six for each participant). Each session lasted one hour. Fig. 8 shows a teleoperation trial in progress.

OPEN IN VIEWER

Fig. 7.

Comparison between Correlation of Self-Reported Anxiety and Self-Reported Engagement with Physiological Indices for P #5

OPEN IN VIEWER

Fig. 8.

Teleoperation Experiment

Fig. 9 shows the continuous evaluation of the engagement level in real-time along with the periodic self-reporting from Participant #1.

OPEN IN VIEWER

Fig. 9.

Continuous Evaluation of Engagement Index

The solid line indicates the predicted value of engagement (calculated every 30 seconds) and the stars indicate the periodic self-reporting (every 3 minutes) of the participant regarding his/her perceived engagement. It can be seen that during the one-hour trial, there were two missed alarms and one false alarm.

For Participant #3, a confusion matrix (Table 2) shows the number of false alarms and missed alarms during the real time detection of engagement from the physiological data during all of the six sessions (training data set size is 50). The engagement values have been divided into three different groups – low, medium and high according to the rule based system that determines the behavior changes. It can be seen that the detection is accurate (17+47+20)/97=86.60 % times, alarms are missed 7/97=7.22 % and false alarms occur 6/97=6.19 % times.

OPEN IN VIEWER

Table 2

Confusion Matrix for P #3

	Predicted Value
		Low	Medium	High
Actual Value	Low	17	4	1
	Medium	1	47	2
	High	0	5	20

Fig. 10 shows the variation in the percentage accuracy of the engagement detection system with change in the size of the training set across all participants. The training datasets were formed by selecting a subset of the entire training dataset while keeping the testing data set constant. The accuracies presented in Fig. 10 were computed over the invariant testing dataset.

OPEN IN VIEWER

Fig. 10.

Mean % Accuracy with Change in Size of Training Data Set For All Six Participants

It can be seen that as the size of training data set was increased the overall accuracy became higher. We expect that in the future we can train our engagement detection system on larger training datasets and thereby achieve higher accuracy in detecting engagement.