Human Adaptive Mechatronics and Human-System Modelling

Abstract

Several topics in projects for mechatronics studies, which are 'Human Adaptive Mechatronics (HAM)' and 'Human-System Modelling (HSM)', are presented in this paper. The main research theme of the HAM project is a design strategy for a new intelligent mechatronics system, which enhances operators' skills during machine operation. Skill analyses and control system design have been addressed. In the HSM project, human modelling based on hierarchical classification of skills was studied, including the following five types of skills: social, planning, cognitive, motion and sensory-motor skills. This paper includes digests of these research topics and the outcomes concerning each type of skill. Relationships with other research activities, knowledge and information that will be helpful for readers who are trying to study assistive human-mechatronics systems are also mentioned.

Keywords

Mechatronics Skill Analysis Assistive Control

1. Introduction

There are a growing number of disciplines for human-friendly mechatronics design such as Human-factors engineering, Ergonomics design and Human centred design. These technologies have been utilized for the design of mechatronics systems and gadgets in our daily life. No matter how well mechatronics are designed, however, there are always situations where the user has to adapt to a mechanical system on one level or another. The reason is that usually the machine itself is not designed so as to adapt to each human user, although human users can adapt to each mechatronics system. Therefore, to resolve such a seemingly contradictory situation, the concept of Human Adaptive Mechatronics was launched in 2004 [1]. A HAM-project was started as a 21st century Centre of Excellence (COE) programme supported by MEXT (Japanese Ministry of Education, Culture, Sports, Science and Technology) during 2004-2008 [2,3]. Required functions for HAM are the ability to adapt to an operator's skill and assist them. Therefore, human-skill evaluation methods and assistance control methods (for instance, a tiny force assist control on an haptic system [4], an adaptive impedance control [5, 6], an assistant control system recovering against inadequate human operation [7] and safe manual control [8]) were studied and several applications, such as a surgery support robot system [9], were developed.

The conceptual meaning of “skill” has been, however, dealt with ambiguously. In other words, systematic approaches considering skill type have not been discussed sufficiently in the HAM project. Hence, as the next challenge after the HAM project, new research into ‘human-system modelling’ was carried out during 2008-2010 for the systematization of skill evaluation and assistance control [10]. The main purposes of this are ‘the establishment of systematic human system modelling’ and ‘the accumulation of assistance methods and skill evaluation methods’. For the first purpose, a hierarchical classification of skills was proposed, as shown in Fig. 1. This classification was based on existing human models (such as a Norman's seven stage model, a model human processor and Rasmussen's skill/rule/knowledge model) and outcomes obtained from the HAM project [11].

The hierarchy consists of the following skills: social- (S1), planning- (S2), cognitive- (S3), motion- (S4) and sensory-motor (S5) skills. The S5, a sensory-motor skill, is the lowest and relates mainly to voluntary motion. This skill concerns cooperation between the neural system and the musculo-skeletal system inside the human body. S4 (motion skill) relates to the continuous execution of each body part motion. For example, the sequential execution of finger, hand and arm motions when using a hand tool. S3 is cognitive skill. This mainly concerns recognition of circumstance and includes a wide range of cognitive issues, such as the understanding of meanings. S2 (planning skill) relates to understanding a whole task process and recognising sub-tasks within the whole task. This skill also relates to the task management of sub-tasks, which requires consideration of events' causality and time scheduling. S1 is social skill, which relates to communication, conversation, negotiation and the estimation of other intentions.

Of course this hierarchical classification is not perfect; however, such classification will be helpful for research concerning human skills and for the design of human assistive systems. For instance, when a job involves assistance system design, the system designer will be able to find a better approach of how to deal with each skill type after clarifying the priority of these skill types. By referring to similar methods categorized as each skill type, concrete solutions for the design of an assistive system can be obtained. Therefore, this paper introduces research activities, knowledge and information, which will be useful for readers who are trying to study assistive mechatronics systems. In particular:

Digests of the authors' research activities and outcomes concerning each skill type are shown and

Items and factors that should be considered for each skill are explained.

The organization of this paper is as follows. Section II introduces experimental tasks, which were designed or used in the authors' study. These tasks are explained in advance in order to make the following topics easier to understand for readers. In Section III, analyses and outcomes relating to each skill type are explained with an introduction to the background and other existing studies. Section IV introduces a comprehensive analysis that addresses the social skills involving other levels. Finally, Section V is a conclusion of the research.

2. Experimental tasks to investigate individual skill levels

Research on the human-system modelling project has been carried out basically through the following process: 1) design of experimental tasks concerning the hierarchical classification of skills, 2) collection of data of human behaviour measured using these experimental tasks, 3) proposal of a skill evaluation method by analysis of the collected data, 4) suggestion of how to design an assistance system by utilizing the obtained knowledge and 5) verification of the presented assistance system by using the experimental task again. In order to help the reader understand the topics described in the latter sections, each experimental task is explained in this section.

A. Stick stabilization task (ss-task)

Stick balancing is a good example of a method to study a voluntary motion mechanism, since the participant has to recognize the status of the stick (that is a controlled object) and control his/her own hand according to the status. Many studies on stick balancing or manual control of a pendulum have been reported [12 -19]. The following facts about a skilled stick balancer are known:

The distribution of changes in hand velocity is a truncated Levy distribution [15].

The power spectrum of fluctuations in the stick's height shows two scaling regions with two different power laws [14, 18, 20].

About 98% of corrective movements occur faster when compared to the time delay of human perception [16].

Figure 1.

Hierarchical classification of skills

Since knowledge of the skill properties required for stick balancing is well known, a stick balancing task was adopted for the authors' study and investigation of the sensory-motor skill. Since our primary research concern is human assistance in a machine operation, an experimental system to stabilize a virtual computer-graphics (CG) stick by manipulation of a machine was developed, as shown in Fig. 2(a). The experimental system consists of two units: a real-time computer graphics generation of a virtual pendulum and a haptic interface device. The participant manipulated a grip fixed to the slider of the interface device and the force exerted by the participant was measured by a sensor embedded in the slider. The motion of the virtual pendulum was computed in real time using the measured force. A process and principle for acquiring the sensory-motor skill were investigated by analysing the relationship between the force and status of the virtual pendulum by the system identification method along with a brain monitoring method

An optical topography system, which is a non-invasive measurement of changes in the concentration of oxy- and deoxy-haemoglobin using near-infrared spectroscopy (NIRS), was used as a brain monitoring method, as shown in Fig. 2(b). Since “the hemodynamic response is partly related to neuronal activity [21]” the activation strength in each local brain area can be estimated by detecting changes in the concentration. NIRS is robust against electrical noise because it utilizes optical measurement, which allows it to measure the brain activation of natural behaviour in a non-restrictive environment, such as speaking [22], reading [23] and language recognition [24]. Because of the admissibility to participant's natural body motion and the robustness against electrical noise from other machines manipulated by him/her, NIRS was adopted for our study.

B. Remote radio control task (rr-task)

In this task, a participant executes tele-operation of radio-controlled constructing machines [25]. This task was devised to analyse the hand and eye motion of the operator in order to investigate cognitive and planning skills. The purpose of the operation was basic soil excavation work, as shown in Fig. 3(a).

The field included three drilling sites, one unloading site, a motorable road and restricted areas. The operator manipulated both an excavator and a truck and used console switches to move the machines to the drilling site, collect sample pieces with the excavator, load the pieces on the truck bed and carry them to the unload site by the truck. He/she was required to optimize the task scheduling by considering shortening the task time. Wireless cameras on the excavator and the truck captured video images and displayed the images on monitors for the operator, as shown in Fig. 3(b). To measure the positions of the machines in the field, infrared LED markers were placed on the work area and were attached to the machines and were observed by a camera attached to the ceiling. Their positions and directions were computed by the Labview system with an image-processing module. The operation of switches on the console was also recorded by the A/D and DIO interface in the Labview system. Eye motion of the operator was also measured and utilized for skill analysis.

C. Dynamic driving task (dd-task)

This task is a kind of driving simulator that imitates the dynamics of a hovercraft on a giant slalom course. A hovercraft was adopted since its manipulation was difficult for almost all participants in the beginning. Fig. 4 shows the whole course. The course consists of side walls, a start-line, a goal line and 15 check-flags. Participants were required to propel the hovercraft to the goal passing through all 15 flags. At that time, he/she was requested to consider the following two points: whether to be as fast as possible and as close to the flags as possible. The operator controls the hovercraft's thruster forces by manipulating a joystick, as shown in Fig. 5(a). Logs of the joystick manipulation, hovercraft status and gaze position were measured and the human control characteristics in training were analysed.

Figure 2.

Experimental setup of stick balancing task (a) and the head probe of the NIRS system (b).

Figure 3.

Radio controlled construction equipment in a work area (a) and the operation console (b).

Figure 4.

Course layout for the dynamic driving task

D. Cooperative carrying task (cc-task)

This task was designed to collect data of human behaviour in order to analyse social and planning skills. Conversation among participants and their eye motions were recorded and brain activity, brain waves and the robots' manipulation data were measured during the task. Fig. 6 shows the experimental scene and the recorded image for gaze analysis.

In the cooperation task, three participants worked in the same virtual space. Each participant sat in front of each monitor, used a joystick to manipulate their own virtual mobile robot, cooperated with the other operators and conveyed three boxes to target places. Fig. 7 shows an overhead view of the virtual space. The virtual space layout was intentionally designed so as to require cooperation, such as simultaneous action to rotate the box, to induce self-motivated operation among participants.

3. Analysis and its outcome

Results obtained by the experimental tasks, which were mentioned in Section II, are explained in this section with respect to each skill type.

A. Sensory-motor Skill (S5)

Sensory-motor control is an essential human processing/motion system, which involves haptic and visual perception, information processing in the brain and an activation of muscles in the arm and leg. In the manipulation of most mechatronics systems, visual information is mainly utilized. Specifically for skilled manipulation of a machine, adequate ability of a visuomotor control is required for the operator to control their own arm and hand in order to adjust motion of an interface device such as a handle or a lever depending on a change of scene. As studies of visuomotor control, numerous studies such as a linear servo control model [26], a PID-based time-variant model having randomness [27], and an optimal control model [28] are known, and they have been utilized for a design of human-machine systems. Although the focuses of those models differ according to its objective, the main points of the discussion about visuomotor control are (a) time-delay compensation, (b) learning and (c) estimation. Concerning point (a), delays lie between about 30ms for a spinal reflex and up to 200-300ms for a visually guided response [29]. Since a delay decreases the stability of the whole system from a control engineering viewpoint, the visuomotor control cannot work well without some compensation for delay. The Smith-predictor is often used as a human model to compensate for time-delays in the nerve system [30] when a target system controlled is stable. However, if the controlled system is unstable, the Smith-predictor is not adequate for the human control model since an estimation error in the Smith-predictor does not decrease theoretically [31]. In that case, it is said that a human model behaves like a forward model in a sensory pre-processing loop. In short, there is more than one way to explain the human visuomotor process. Regarding the discussion of point (b), a feedback-error-learning model is widely accepted [32]. The concept of the model is that a forward model-based controller gives the most suitable explanation for the visuomotor control after learning [33]. The forward model is considered to be a basic mechanism for estimating the next state of the body motion [31]. This corresponds to point (c).

The mechanisms of the visuomotor control are explained as mentioned above. However, few models can explain the stages of the learning process clearly. Therefore, in our study the changes in brain activities and control characteristics of the human operators were investigated from the beginner-level till the expert-level by using the ss-task. First, participants were trained in the ss-task and hand motion/force was measured. At the training stage, at least ten trials a day (over more than five days) were imposed for each participant. Written consent and ethical approval were obtained before the examinations (all experiments were done after consent and ethical approval.) Second, in order to investigate the distribution of changes in the hand velocity of participants, the skill levels of the participants were classified into high-performance (HP), moderate-performance (MP) and low-performance (LP) by referring to Cabrera's studies [14]. Finally, using the data of the HP participants, the control characteristics of the skilled stick balancers were identified. In particular, if a human is assumed to be a controller that perceives an inclination angle of the pendulum θ and outputs the hand force f_h, the following control law in consideration of the time shift can be defined:

f_{h} (t) = k_{θ} \cdot θ (t - Δ_{θ}) + k_{\dot{θ}} \cdot \dot{θ} (t - Δ_{\dot{θ}}) + k_{d},

(1)

where k_θ and $k_{\dot{θ}}$ are constant gains to be estimated and t is time. Here, Δ_θ and $Δ_{\dot{θ}}$ are time shifts and k_d is the drift term. To search for the best values of the time shifts, the error index e, which was computed by averaging the identification error of | f_h(t) – f̄_h |, was checked. Here, f̄_h is the estimated force that was computed using the identified parameters k̄_θ, ${\bar{k}}_{\dot{θ}}$ and k̄_d. The best combination of Δ_θ and $Δ_{\dot{θ}}$ was determined by changing them to Δ_θ: −1 → +1 and $Δ_{\dot{θ}} : - 1 \to + 1$ . Fig. 8 shows the strength distribution of the identified coefficients k̄_θ, ${\bar{k}}_{\dot{θ}}$ , k̄_d and e. The vertical and horizontal axes on each graph are Δ_θ and $Δ_{\dot{θ}}$ , respectively. The white (black) area in each graph indicates the large (small) value.

Figure 5.

An experimental scene of the dynamic driving task (a) and a camera image captured by the gaze measurement system (b).

Figure 6.

An experimental scene of the cooperative carrying task (a) and a sample of recorded image (b).

Figure 7.

Overhead view of the virtual space for cooperation-carrying task

As the drift term k̄_d is sufficiently smaller than any of |k̄_θ| or $| {\bar{k}}_{\dot{θ}} |$ , Eq. (1) appears to be appropriate as the form of the control law of the expert. Moreover, the magnitude of $| {\bar{k}}_{\dot{θ}} \cdot \dot{θ} |$ was larger than that of the other terms; hence, it transpired that the expert was paying attention to the change in velocity more than posture information. Furthermore, Fig. 8 shows that the error index in the vertical axis is smallest when Δ_θ = 0.18 and that the error index in the horizontal axis is smallest when Δ_θ = −0.04. This negative sign of Δ_θ is interpreted as the prediction concerning the measured velocity information. In conclusion, the expert recognized the posture of the controlled object about 0.18 seconds after observation and predicted the velocity about 0.04 seconds before by using one's internal model. It can be surmised that the prediction based on an internal model of the controlled object is key to the skilled visuomotor control.

Next, the measured NIRS data were analysed to find the brain property of the skilled balancers. Generally, the primary motor cortex (MsI) and the primary somatosensory cortex (SmI) are important for voluntary motion and the movements of most muscles in the body are controlled by local regions in these cortices. Furthermore, the supplementary motor area (SMA) and the premotor cortex (PMC) relate to generating command of the voluntary motion from the generated intentions. Locations of these brain areas and the measurement position of the NIRS headset are illustrated in Fig. 9(a) and 9(b), respectively. The International 10-20 measurement system, which is an application method that uses the electrodes of an electroencephalogram and is based on the distance between the nasion and the inion of the scalp, was used to determine the position of the probe. Changes in the concentration of total haemoglobin (sum of oxy- and deoxy-heamoglobin) were measured using an ETG-4000 system (Hitachi Medical Corporation, Tokyo, Japan). Reflections of lasers in the near infrared were measured 10 times during each sampling interval and the measured data was output every 100ms by averaging these reflections to attenuate noise effect.

The 48ch data of total haemoglobin were analysed using principal component analysis (PCA). As a result, a topography map in the brain showed the difference between the HP, MP and LP participants, as shown in Fig. 10. When we investigated activation of the HP participant's brain, activation of the right and left regions corresponding to the eyeball in the primary motor cortex (MsI) could be recognized (the corresponding area is labelled using (a3) on images (a) in Fig. 10). Further activation in the right arm and hand regions (that are located at MsI in the left hemisphere) was not strong. On the other hand, strong activations for the MP and LP participants were found in the narrow areas of the torso and hip, as shown by (b1) and (c1) in Fig. 10. From these facts, it would be appropriate to think that activation of the ocular motor area is a characteristic of the HP participant. This means a skilled person concentrates on observing more than controlling.

Figure 8.

Strength distribution of the identified coefficient: (a) k̄_θ, (b) k̄_θ, (c) k̄_d and (d) e.

Figure 9.

Positional relation of side view of the brain (a) and the corresponding position of NIRS headset (b).

According to a study by Cabrera, a skilled stick balancer performs the on-off intermittency control (the drift-and-act control). That is, an expert at stick balancing moves the hand faster than the time delay of human perception using a feed forward control and he/she changes this into feedback control when the pendulum link tilts largely. Existence of the on-off intermittency control means that a human is not a simple continuous-time controller but a complex of controllers that are switched depending on the circumstances. However, from the above-mentioned identification results, it was shown that stick balancing control can be described well by one linear model involving two types of time shifts for the position and velocity, as defined by Eq. (1). This indicates that the on-off intermittency control can be approximated by one control law involving two types of time shifts. This formula will be effective as one for human controller models to design human-assistive mechatronics [34].

B. Motion Skill (S4)

The general motion of a person consists of a sequence of simple motions including voluntary motion. Therefore, the performance of the action decreases if transitions between motions are not sufficient. To investigate the skill of such general motion, characteristics of the transition and the motion control in operators were analysed using the dd-task, which is more general than the ss-task from the point of view of a general machine operation. Using gaze measurement during machine operation, all the operation sequences were segmented into piece-wise linear intervals by checking the saccadic eye movement of the operator's change in gaze position from one target to another.

An illustration that explains the spatial relation in the dd-task virtual space is shown in Fig. 11, where ζ is a relative angle of the hovercraft's travelling direction to the closest flag (=first flag) and d is the relative distance between the hovercraft and the first flag. Here, θ is a change of the elevation angle against the fore object and is known as optic flow, which is an effective index for analysing driving behaviour [35]. Note that d and ζ are geometrical variables in a virtual 3D space and the θ is a variable against the 2D monitor screen. This index is based on the concept that a human utilizes a change of elevation angle to estimate a change of perspective.

The control law of the operator was identified using the following piece-wise linear model with three inputs (ζ, θ, d) and one output (a rotation command to the hovercraft, γ).

\begin{array}{c} γ [t^{'}] = a_{1} ξ [t^{'}] + a_{2} ξ [t^{'} - 1] + a_{3} ξ [t^{'} - 2] \\ + b_{1} θ [t^{'}] + b_{2} θ [t^{'} - 1] + b_{3} θ [t^{'} - 2] \\ + c_{1} d [t^{'}] + c_{2} d [t^{'} - 1] + c_{3} d [t^{'} - 2] \end{array}

(2)

where t' is the time when the operator saw the second flag first and a_i,b_i and c_i (i=1,2,3) are gain parameters to be identified. After identification, the transition of the ratios in these gain parameters, which were computed as follows, were investigated to check the learning process:

ρ_{ξ} : = 100 \cdot a v e (\tilde{a} / (\tilde{a} + \tilde{b} + \tilde{c}))

(3)

ρ_{θ} : = 100 \cdot a v e (\tilde{b} / (\tilde{a} + \tilde{b} + \tilde{c}))

(4)

ρ_{d} : = 100 \cdot a v e (\tilde{c} / (\tilde{a} + \tilde{b} + \tilde{c}))

(5)

Figure 10.

Topographical images of stick balancers in the three skill levels

Figure 11.

Conceptual illustration of spatial relation around a virtual hovercraft of the dd-task.

where $\tilde{a} : = \sum_{i = 1}^{3} | a_{i} | / 3$ , $\tilde{b} : = \sum_{i = 1}^{3} | b_{i} | / 3$ , $\tilde{c} : = \sum_{i = 1}^{3} | c_{i} | / 3$ and ave (*) means an average operation.

Five trials a day during five days were carried out by five novice participants and the obtained data were analysed by the aforementioned procedure. Table 1 summarizes the correlation factors between each ratio ρ and the manipulation performance (sum of minimum approach distances to flags, E_f) for each participant.

Table 1.

Correlation coefficients of gain ratio against a manipulation performance

Participant	r(E_f,ρ_ζ)	r(E_f,ρ_θ)	r(E_f,ρ_d)
A	-0.18	0.55	-0.37
B	-0.71	0.81	-0.15
C	-0.32	0.44	-0.08
D	-0.16	0.77	-0.75
E	-0.36	0.72	-0.44

Since the table shows that the correlations of E_f and ρ_θ are all positive, the gain ratio of θ decreases as the participant becomes more skilled, because E_f decreases as the trial increases. In other words, a weakening of the utilization of the elevation angle information is related to an increase in skill. Similarly, other correlation factors of (E_f vs ρ_ζ) and (E_f vs ρ_d) are all negative. Hence, the participants increased the gains to the relative angle ζ and the distance d. These variables are not observable directly to the operator's eye since ζ and d are variables in the virtual 3D space. Therefore, it is interpreted that a skilled operator can recognize the virtual 3D spacial relation from the 2D image information displayed in the monitor. This fact is useful for evaluating skill in tele-operations.

C. Cognitive Skill (S3)

For the operation of mechatronics, especially for vehicle machine operation, an ability to understand the environment around the operated machine, that is cognitive skill, is required. Although research regarding cognitive skill essentially belongs to the psychological field, how to utilize it for a mechatronics system should be considered for mechatronics engineers. Therefore, our study concerning cognitive skill was promoted to find a better methodology for a design of a mechatronic assistance system by utilizing the rr-task.

First, we focused on the gaze from which the cognitive activity might be estimated from outside. The cognitive skill for machine operation was investigated through analysis of the eye motion. As a result, video analysis of the gaze pattern in the training process showed the following facts: at the early stage, the operator tends to watch more than one monitor on the console desk in order to watch the target object; at the skilled stage, he/she watches a space and the ground through one main monitor and does not see the target directly [36]. These facts are interpreted as follows. The operator acquires proficiency in environmental cognition and does not pay attention to objects directly, then uses visual perception mainly to check positional relation inside the workspace. Such characteristic behaviour can be utilized to estimate cognitive skill level from eye motion.

Second, a machine function to evaluate an operator's cognitive skill level qualitatively was studied. One powerful computational tool for this is ontology, which is a network model to describe a knowledge structure. Originally, ontology is a term used in philosophy and is the study of the nature of being, existence or reality in general. In more recent computer science, this concept is used as the basic category of being and relations that can be shared by both a human and a computer. Building up ontology consists of two steps: (1) extraction of relations of ‘concept classes’ as a ‘semantic link’ and (2) repetition of segmentation of each concept class into small classes. In our research, structures of the knowledge required to perform the cc-task were built by the Action Fast Method [37] and the methodology to apply it to a design of human-machine systems was studied. For instance, a part of the cc-task ontology is shown in Fig. 12. Since factors and causality that relate to present statuses can be estimated by referring the link network of the concept classes in the obtained ontology, the network structure of ontology can be utilized to guess an operator's (future) intention and to evaluate their cognitive skill. An applied example is mentioned in Section 3.D2.

D. Planning Skill (S2)

D1. Evaluation of skill from the reaching action

After mastering both voluntary motion control (S5) and the combination of their motions (S4), a planning skill to manage sub-actions is required. Since planning is a reiteration of judgement and the execution of actions, superiority and inferiority in the connections of these two processes appears as a difference in the level of the planning skill. Conversely, the detection of unnatural motions may be effective in evaluating the level. In fact, such unnatural motion is known as microslip, which is a jerky movement in a hand reaching action. By Reed's definition, there are four types of microslip: trajectory change, hesitation, change of the hand shape and touching objects [38]. It was reported that microslips appear more frequently under high cognitive loads and complex circumstances [39]. In our previous study, the relationship between task performance and microslips was investigated through a coffee-making test and the strong correlation was confirmed statistically [40]. This result indicates that a human's planning skill can be estimated by observing the hand motion.

Figure 12.

A part of the cc-task ontology

Therefore, using the rr-task, the planning skill of a machine operation was investigated by focusing on the operators' hand motion. To be more precise, the discrete movement of hand was investigated using Fitts' law [41]. Fitts' law is an experimental formula concerning the time and distance of a voluntary hand motion. This law is valid for a reaching action of other body parts and of non-straight line courses [42]. Fitts' law has been used for the evaluation of computer interface designs, such as a GUI [43,44] or a stylus [45]. The present authors also guessed that the level of planning skill might be detected by checking the error against Fitts' law when inadequate action like a microslip occurred. Therefore, the travelling time/distance of the discrete hand motions in a machine manipulation on the console desk were investigated. Fig. 13 shows the layout of the switches on the control desk. Each planar coordinate value (x, y) of the switch position was registered as a table and the distance D was computed as a Euclid norm between the starting point (x, y)s and the ending point (x, y)e of each hand reaching motion. The target width W is the size of lever grips and was also obtained from a pre-registered table.

The following formula was used for the analysis by referring the Fitts' law.

^{R} T = k_{1} + k_{2} x + e

(6)

x : = \log_{2} (2 D / W),

(7)

where ^RT is the travelling time of one reaching action and k₁ and k₂ are constant coefficients. x is an index of difficulty and it depends only on geometric data of the distance and width. e is a fitting error to Fitts' law. e was newly introduced in our study and was assumed to vary by cognitive load on the operator. The coefficients k₁ and k₂ were identified by a least square method with Eq. (6) using the multiple paired values {^RT,x} from all the reaching actions data. Describing the identified coefficients as k̂₁ and k̂₂, the fitting error e[i] of the ith reaching action was computed as

e [i] =^{R} T [i] - ({\hat{k}}_{1} + {\hat{k}}_{2} x [i]) .

(8)

Fig. 14(a) and 14(b) show the mean value and covariance of the fitting error to Fitts' law, respectively. Each dotted line denotes the data of each participant and the bold solid line is the average for all eight participants. It was confirmed that the covariance of the fitting error decreased (Fig. 14(b)) as the number of trials increased, although the mean of e was almost flat (Fig. 14(a)). This fact indicates that covariance of the fitting error to Fitts' law is usable for evaluating the planning skill [46].

D2. Estimator of operator's intention

Since planning is a decision of operational intentions, a function to guess the operator's intention is also required for an intelligent assistive machine, which enhances human operation. Estimation of a user's intention is quite useful for various applications, such as in assistance software [47], prediction of users' requests on the Internet [48] and marketing [49]. Estimation of intentions for a general human-machine system is, however, more difficult than the above-mentioned successful examples since it is also difficult to identify the information types which are utilized for the operator's decision making.

To address this problem, the present authors proposed an algorithm to estimate an operator's intention using the clustering technique with the multivariate time-sequence data of the machine operation [50]. In the algorithm, the intentions are assumed to be an internal status of the human model. Therefore, the problem can be considered as a design of an observer that estimates the operator's internal status in the human-machine system, which consists of a human (operator), a machine (to be manipulated), an environment and the work task, as shown in Fig. 15.

Figure 13.

Layout of console switches and coordinate system

Figure 14.

Transition of the fitting error to Fitts' law: (a) mean, (b) covariance

Figure 15.

System architecture of human�]machine system

Describing these three types of statuses as the machine status (M-status, qm), the environment status (E-status, qe) and the task status (T-status, qt), a human during machine operation can be defined as an information-processing system, ρ = H(z) · s, where s := [q^t_m,q^t_e,q^t_t]^t is the information to be recognized by the human, ρ is the output of operation commands to the machine and z is the intention of the human. Furthermore, based on the concept of ‘spotlight of selective attention’ in Global Workspace Theory [51], mathematical expression of z is defined as a vector $z \in {[0, 1]}^{n_{z}}$ of which element corresponds to one intention strength of one operation action (n_z is a size of vector z). Then, the probabilistic distribution bel(z), that is a belief of the intention z, is estimated by the Bayes filtering technique. The basic algorithm for estimation of bel(z) is given as follows:

Algorithm: Bayes filter

\bar{bel} (z_{t}) = \int p (z_{t} | s_{t}, z_{t - 1}) \cdot bel (z_{t - 1}) d z_{t - 1}

(9)

bel (z_{t}) = η \cdot p (ρ_{t} | z_{t}) \cdot \bar{bel} (z_{t}),

(10)

where subscript t and t-1 mean the time counters. This algorithm is defined with iterative equations that are computed from time index t=1 to the final time T. p(z_t|s_t,z_t-1corresponds to a probabilistic distribution of a transition of intention from z_t-1 to z_t given input s_t. p(ρ_t|z_t)is the conditional probabilistic distribution of judgment that outputs ρ _t if the intention z_t happens to be true. η is a so-called Bayes normalization constant. Eq. (9) is a prediction to obtain a belief $\bar{bel} (z_{t})$ at the time of t. Eq. (10) is called a measurement update and adjusts the prediction $\bar{bel} (z_{t})$ by considering the probabilityp(ρ_t | z_t). Via this update, a new belief bel(z_t) at time t is obtained. The estimation of the intention, say ẑ_t, is computed as a median of the distribution of bel(z_t).

In our approach, the state transition relation of an operator's intentions was formed using Self-Organizing Map (SOM), which was trained using the measured data of the operation and environmental variables with the reference intention sequence. SOM is an artificial neural network for discriminating multi-dimensional data and is adequate for data processing of large dimensional data since the SOM technique can compress multidimensional information into a lower (two) dimensional map by keeping original topological information. In other words, similar types of data gather together and different types get away from each other on the clustering map. Such Bayesian property embedded in the SOM is expected to be utilized to predict a transition of status and we utilized it for the Bayes filtering algorithm to compute the probability of state transition. Refer to [50] for the detail.

Applying the proposed algorithm to the rr-task, the estimator's ability was verified. First, types of remote operation of the construction equipment and the definitions of elements ⁽¹⁾ z, … ⁽¹⁵⁾z in intention vectors z were defined, as summarized in Table 2.

Table 2.

Classified operational intention modes, variables and labels

Variable	Label	Meanings

(1)_z … (3)_z	T/A-*	Truck's approach to the drilling site-*
(4)_z … (6)_z	E/A-*	Excavator's approach to the drilling site-*
(7)_z … (9)_z	T/P-*	Truck's positioning around the drilling site-*
(10)_z … (12)_z	E/P-*	Excavator's positioning around the drilling site-*
(13)_z	E/D	Excavator's digging
(14)_z	E/L	Excavator's loading of the payload
(15)_z	T/TU	Transport of the payload by Track and unloading

(=a,b,c: Identifier of the drilling site)

Second, the SOM was trained using time-sequence data including the machine's status and command of operation. The obtained SOM is shown in Fig. 16. Labels described in Table 2 are attached to each cluster of the corresponding mode in the figure. It can be confirmed that obvious clusters are formed.

Third, using the above-mentioned Bayes filtering algorithm with the obtained SOM, say the SOM-Bayes estimator, intentions of other operators were estimated as a validation. Fig. 17 shows the transitions of the estimated intention by comparing with the human-discerned intentions. Lines show the transitions of intention modes from T/A-a (⁽¹⁾ z) to T/TU (⁽¹⁵⁾ z). Red lines were identified by the SOM-Bayes estimator. Blue lines were discerned by a human analyst. The red lines by the estimator overlap with the blue lines by the human analyst well in the case of almost all intention modes.

Figure 16.

Operation modes map generated by SOM

Additional analysis confirmed that the estimator could identify the intention modes at 44-94% concordance ratios against normal intention modes whose periods can be found by about 70% of human analysts [50]. Moreover, it was shown that distances between clusters on the SOM have a strong relation to the operator's skill level in task performance [52]. As shown above, it was demonstrated that not only estimation of intentions but also evaluation of skill is available by utilizing SOM. In addition, the SOM-Bayes intention estimator can be enhanced by combination with a Petri net based on ontology, which was shown in the previous Section III.3. Refer to [53] for the details.

4. Comprehensive analysis of high-level skill

Skills from the sensor-motor skill (S5) to the planning skill (S2) concern a relationship between one operator and a machine(s). Social skill (S1) differs from those skills in terms of an existence of more than one person; hence, the social skill is conceptually highly independent compared with the other skills. Social skill is, however, inseparably connected to the other skills since an enhancement of social skill comes after mastering those other skills. For instance, the planning skill is often taught to other people via social communication.

The authors studied social skill by using the cc-task investigating other skills. The cc-task is a type of the vehicle tele-operation task, which is similar to dd-task. In this experiment, the participants joined the cc-task after they had mastered the manipulation of a virtual vehicle so as to eliminate an effect of a learning process of the sensory-motor skill. Several studies about social skill using the cc-task are introduced below.

A. Role in a collaborative work

Individual characteristics of the operators' behaviour in the collaborative work were investigated through analysis of log data, including the machine operation and vehicle motion of all participants. The individual characteristics were classified into three types: observer, leader and transporter. Through analysis, it was confirmed that adequate and obvious role allotment according to circumstance increased the efficiency of the whole work [54]. Fig. 18 shows that the distribution between the ratio of conveying and the ratio of observation had spread as the cooperative skill increased. This study could present one of indexes to evaluate social skill.

B. Analysis of spoken language

From the point of view of communication, which is a representative property of social skill, corpuses of utterance in conversation among participants were analysed. It was found that corpuses used among them were commonalized as the social skill increased and that common ground was formed [55]. This fact indicates that sharing of a common communication scheme specialized for a given task is significant for a skilled team.

C. Analysis of eye motion

Eye motion is important not only for visuomotor control and perception to recognize circumstance but also for nonverbal communication. Eye motion occurs in order to guess another person's intention by watching the eyes, or is utilized as a cue for mutual confirmation with other people. A well-known example is a joint attention [56]. Direction of gaze is also utilized for nonverbal communication by combination with pragmatics [57]. These follow that eye motion may reflect social skill. Therefore, in this subsection, an attempt to derive an equation to estimate team skill from the measurement of the eye motion of an individual is shown.

Basic eye-movements are saccade, fixation, smooth pursuit and nystagmus. In our study, saccade and fixation were investigated in common with existing studies [58, 59]. Specifically, the following factors were investigated: (1) response time of eye-movement against the target object's motion, (2) gaze duration, (3) frequency of saccade motion, (4) velocity of saccade and (5) movement distance of saccade. Moreover, it is known that the driver of the vehicle tends not to notice events captured in the peripheral vision area since he/she concentrates on the direction of travel of the vehicle [60]. It is also known that inessential eye movement of a skilled driver decreases [61]. Taking into consideration these facts, two cases of low and high speed driving were considered against the above-mentioned five factors and total ten characteristics factors were investigated in our study.

Figure 17.

Transitions of intentions (Red lines: by SOM-Bayes estimator, Blue lines: by human analyst)

Figure 18.

Change of ratio of action and observation in the cooperative work.

Since the aim of the analysis was to induce an equation, which evaluates a canonical team-skill, the induction was processed eliminating outliers from the measured data. Specifically, data including strong individual differences were eliminated by the Smirnov-Grubbs' outlier test (p<0.1) and the normality of the remaining data was checked by Lilliefors test (p<0.1). Then, correlation of averages of each factor among participants was investigated. It was confirmed that three factors, which are saccade velocity of high-speed driving, saccade distance of high-speed driving and saccade velocity of low-speed driving, have a high correlation with the number of trials. Next, these three eye-movement factors were decomposed by PCA. Finally, an equation to estimate a team skill was derived by using the weighting coefficients corresponding to the first dominant component vector on the PCA. Transition of the skill values predicted by the deduced equation is shown in Fig. 19. Investigating those predicted values against another skill index, a high correlation (0.56-0.84) was confirmed. Hence, the effectiveness of the deduced skill equation was confirmed [62]. This result can be utilized to evaluate team skill in collaboration work using the machine manipulation.

5. Conclusion

This paper introduced a hierarchical classification of skills, which is helpful for designing assistive systems for machine operators. After the experimental tasks for investigations into the five types of skills in the classification were explained, research applications and the outcomes, which are the skill evaluation methods and assistive system designs, were presented. Although any task is related to various types of skill level, utilization of the hierarchical classification of skills will make what is important for designing an adequate assistive system for the given task clear.

Figure 19.

Transition of predicted skill level (Seven participants' and its average data)

Footnotes

6. Acknowledgments

This research was supported by a Grant-in-Aid for the 21st Century COE (Centre of Excellence) Programme and a Grant-in-Aid for Scientific Research (A) from the Ministry of Education, Culture, Sports, Science and Technology, Japan. The authors wish to thank the Ministry for their generous financial assistance and the many participants who cooperated for our experiments.

References

Furuta

, “Control of Pendulum: From Super Mechano-System to Human Adaptive Mechatronics” Plenary, Proceedings of the 42nd IEEE Conference on Decision and Control, Hawaii, USA, 2003: 1498–1507.

Harashima

, “Human Adaptive Mechatronics” Keynote speech, Proceedings of the IEEE Industrial Electronics Society, Raleigh, USA, 2005.

Harashima

Suzuki

, “Intelligent Mechatronics and Robotics” keynote speech, Proceedings of the 2008 IEEE International Conference on Emerging Technologies and Factory Automation (ETFA2008), Hamburg, Germany, Sept. 2008.

Suzuki

Kurihara

Furuta

Harashima

, “Assistance Control on a Haptic System for Human Adaptive Mechatronics” Journal of Advanced Robotics, 2006; 20(3): 323–348.

Suzuki

Kurihara

Furuta

Harashima

Pan

, “Variable Dynamic Assist Control on Haptic Systems for Human Adaptive Mechatronics” Proceedings of the 44th IEEE Conference on Decision and Control and European Control Conference (CDC-ECC'05), Seville, Spain, December, 2005: 4596–4601.

Suzuki

Furuta

, “Adaptive impedance control to enhance human skill on a haptic interface system” in the special issue of Adaptive Control Theory and Applications, Journal of Control Science and Engineering, 2012, Article ID 365067.

Furuta

Kado

Shiratori

Suzuki

, “Assisting control for pendulum-like juggling in human adaptive mechatronics” Journal of Systems and Control Engineering, Special Issues, IMechE, 2011; 225(6): 709–720.

Åkesson

Åström

K. J.

, “Safe manual control of the Furuta pendulum” Proceedings of IEEE International Conference Control Applications, Mexico City, NM, 2001: 890–895.

Miyawaki

Masamune

Suzuki

Yoshimitsu

Vain

, “Scrub nurse robot system – Intraoperative motion analysis of a scrub nurse and timed-autobata-based model for surgery” IEEE Transaction Industrial Electronics, 2005; 52(5): 1227–1235.

10.

Harashima

Suzuki

, “Human Adaptive Mechatronics and Human-System Modeling” Plenary talk, Proceedings of the 12th IEEE International Workshop on Advanced Motion Control (AMC), Sarajevo, Bosnia and Herzegovina, 2012.

11.

Suzuki

, “Human Adaptive Mecathorincs” IEEE Industrial Electronics Magazine, 2010; 4(2): 28–35.

12.

Foo

Kelso

J. A. S.

de Guzman

G. C.

, “Functional stabilization of fixed points: Human pole balancing using time to balance information” Journal of Experimental Psychology: Human Perception and Performance, 2000; 26: 1281–1297.

13.

Åkesson

Åström

K. J.

, “Manual Control and Stabilization of an Inverted Pendulum” Proceedings of 16th IFAC World Congress, Prague, Czech Republic, 2005.

14.

Cabrera

J. L.

Milton

J. G.

, “Stick balancing: On-off intermittency and survival times” Nonlinear Studies, 2003; 10(2): 1–13.

15.

Cabrera

J. L.

Milton

J. G.

, “Human stick balancing: Tuning Levy flights to improve balancing control” Chaos, 2004; 14(3): 691–698.

16.

Cabrera

J. L.

Milton

J. G.

, “On-Off Intermittency in a Human Balancing Task” Physical Review Letters, 2002; 89(15): 158702/1–4.

17.

Priplata

Niemi

Salen

Harry

Lipsitz

L. A.

Collins

J. J.

, “Noise-enhanced human balance control” Physical Review Letters, 2002; 89(23): 238101:1–4.

18.

Cabrera

J. L.

Bormann

Eurich

Ohira

Milton

, “State-dependent noise and human balance control” Fluctuations and Noise Letters, 2004; 4: L107–L118.

19.

Milton

J. G.

Ohira

Cabrera

J. L.

Fraiser

R. M.

Gyorffy

J. B.

Ruiz

F. K.

Strauss

M. A.

Balch

E. C.

Marin

P. J.

Alexander

J. L.

, “Balancing with Vibration: A Prelude for ‘Drift and Act’ Balance Control” PLoS ONE, 2009; 4(10): e7427.

20.

Venkataramani

S. C.

Antonsen

T. M.

Jr. Ott

Sommerer

J. C.

, “On-off intermittency: Power spectrum and fractal properties of time series” Physica D, 1996; 96: 66–99.

21.

Daunizeau

Grova

Marrelec

Mattout

Jbabdi

Peegrini-Issac

Lina

J. M.

Benali

, “Symmetrical event-related EEG/fMRI information fusion in a variational Bayesian framework” Neuroimage, 2007; 36(1): 69–87.

22.

Herrmann

M. J.

Ehlis

A.-C.

Fallgatter

A.J.

, “Frontal activation during a verbal-fluency task as measured by near-infrared spectroscopy” Brain Research Bulletin, 2003; 61: 51–56.

23.

Noguchi

Takeuchi

Sakai

K. L.

, “Lateralized activation in the inferior frontal cortex during syntactic processing: An event-related optical topography study” Human Brain Mapping, 2002; 17: 89–99.

24.

Pena

Maki

Kovačiċ

Dehaene-Lambertz

Koizumi

Bouquet

Mehler

, “Sounds and silence: An optical topography study of language recognition at birth” Proceedings of the National Academy of Science of the USA, 2003; 100(20): 11702–11705.

25.

Suzuki

Harashima

, “Analysis of machine operation skills using hand discrete movement” Proceedings of the 2008 IEEE International Conference on Emerging Technologies and Factory Automation (ETFA2008), Hamburg, Germany, 2008: 156–163.

26.

Tustin

, “The Nature of the Operator's Response in Manual Control and its Implications for Controller Design” Journal of Inst. of Electrical Engineering, 1947; 94(IIA): 190–202.

27.

Ragazzini

J. R.

, “Engineering Aspects of the Human Being as a Servo-Mechanism” Meeting of the American Psychological Association, 1948.

28.

Baron

Kleinman

D. L.

Levison

W. H.

, “An Optimal Control Model of Human Response Part II: Prediction of Human Performance in a Complex Task” Automatica, 1970; 6: 371–383.

29.

Miall

R. C.

Wolpert

D. M.

, “Forward Models for Physiological Motor Control” Neural Networks, 1996; 9(8): 1265–1279.

30.

Miall

R. C.

Weir

D. J.

Wolpert

D. M.

Stein

J. F.

, “Is the Cerebellum a Smith Predictor?” Journal of Motor Behavior, 1993; 25(3): 203–216.

31.

Mehata

Schaal

, “Forward Models in Visuomotor Control” Journal of Neurophysiology, 2001; 88: 942–953.

32.

Wolpert

D. M.

Miall

R. C.

Kawato

, “Internal models in the cerebellum” Trends in Cognitive Sciences, 1998; 2(9): 338–347.

33.

Bhushan

Shadmehr

, “Computational nature of human adaptive control during learning of reaching movements in force fields” Biological Cybernetics, 1999; 81: 39–60.

34.

Suzuki

Harashima

Furuta

, “Human Control Law and Brain Activity of Voluntary Motion by Utilizing a Balancing Task with an Inverted Pendulum” Advances in Human-Computer Interaction, 2010; Vol. 2010, Article ID 215825; doi:10.1155/2010/215825.

35.

Brackstone

M. A.

Sultan

McDonald

, “Findings on the approach process between Vehicles” Transportation Research Record, 2000; Vol.1724, paper No.00-0831: 21–28.

36.

Kobayashi

, “Relations between eye gaze and cognitive factors in a transportation task using remote control” Proceedings of 18th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN09), 2009, Toyama, Japan.

37.

Mizoguchi

Ikeda

Seta

Vanwelkenhuysen

, “Ontology for Modeling the World from Problem Solving Perspectives” Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI-95) Workshop on Basic Ontological Issues in Knowledge Sharing, 1995: 1–12.

38.

Reed

E. S.

Schoenherr

, “The neuropsychology of everyday life: On the nature and significance of micro-slips in everyday activities” Unpublished manuscript, 1992.

39.

Suzuki

Mishima

Sasaki

, “Affordance and Variability of Action -Environment, Action and Microslips-” Journal of Japan Society for Fuzzy Theory and Systems, 1997; 9(6): 826–837.

40.

Kobayashi

Yasuda

Suzuki

, “The relations between development of human's manipulative skills and physiological signals of brain and hand” Proceedings of the International Journal of Assistive Robotics and Mechatronics, 2006; 7(1): 49–61.

41.

Fitts

P. M.

, “The information capacity of the human motor system in controlling the amplitude of movement” Journal of Experimental Psychology, 1954; 47(6): 381–391.

42.

Accot

Zhai

, “Beyond Fitts' Law: Models for Trajectory-Based HCI Tasks” in Proceedings of the SIGCHI conference on Human factors in computing systems (CHI'97), New York, USA, 1997: 295–302.

43.

McGuffin

M. J.

Balakrishnan

, “Fitts' law and expanding targets: Experimental studies and designs for user interfaces” Transactions on Computer-Human Interaction, 2005; 12(4): 388–422.

44.

Zhai

Conversy

Beaudouin-Lafon

Guiard

, “Human on-line response to target expansion” in Proceedings of the SIGCHI conference on Human factors in computing systems (CHI'03), 2003; 5(1): 177–184.

45.

Ren

Mizobuchi

, “Investigating the Usability of the Stylus Pen on Handheld Devices” in Proceedings of The Fourth Annual Workshop on HCI Research in MIS, Las Vegas, USA, 2005: 30–34.

46.

Suzuki

Harashima

, “Skill evaluation from observation of discrete hand movements during console operation” Journal of Robotics, 2010; Vol. 2010, Article ID 967379: Doi:10.1155/2010/967379.

47.

Horvitz

Breese

Heckerman

Hovel

Rommelse

, “The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users” Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, Madison, WI, USA, 1998: 256–265.

48.

Zukerman

Albrecht

D. W.

Nicholson

A. E.

, “Predicting Users' Requests on the WWW” Proceedings of the Seventh International Conference on User Modeling, Banff, Canada, 1999: 275–284.

49.

Gutman

, “A means-end chain model based on consumer categorization process” Journal of Marketing, 1982; 46: 60–72.

50.

Suzuki

Harashima

, “Estimation algorithm of machine operational intention by Bayes filtering with Self-Organizing Map” Advances in Human-Computer Interaction, 2012; Vol.2011, Article ID 724587: Doi:10.1155/2012/724587.

51.

Baars

B. J.

, A cognitive theory of consciousness, Cambridge University Press, New York, 1988.

52.

Suzuki

Harashima

, “Estimation of operational intentions utilizing Self-Organizing Map with Bayes filtering” Proceedings of the 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2010), Taipei, Taiwan, 2010: 2249–2255.

53.

Suzuki

Harashima

, “Context-aware Bayesian intention estimator using Self-Organizing Map and Petri net” Proceedings of 20th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN2011), Atlanta, GA, USA, 2011: 314–320.

54.

Igarashi

Suzuki

Kobayashi

Harashima

, “Role Sharing Analysis on Multi-operator Cooperative Work” Proceedings of 18th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN09), Toyama, Japan, 2009: 1060–1065.

55.

Nakata

Kobayashi

Kumata

Suzuki

, “Human Speech Ontology Changes in Virtual Collaborative Work” Proceedings of the 4th International Conference on Human System Interaction (HSI 2011), Yokohama, Japan, 2011: 363–368.

56.

Yasuda

Kobayashi

, “Influence of 10 Seconds Interval in Pragmatic Interpretation” Proceedings of 19th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN10), Viareggio, Italy, 2010.

57.

Yasuda

Kobayashi

, “Timing of adults' utterances and interpretation of word meanings in a discrepant labeling situation” Proceedings of the 30th Annual Meeting of the Cognitive Science Society, 2008: 675–680.

58.

Ito

Matsubara

Grimbergen

, “Cognitive Science Approach to Shogi Playing Processes (1) - Some Results on Memory Experiments” IPSJ; 43(10): 2998–3011.

59.

Sakita

Ogawara

Murakami

Kawamura

Ikeuchi

, “Flexible Cooperation between Human and Robot by interpreting Human Intention from Gaze Information” Proceedings of IEEE/RSJ International Conference on Intelligent Robotics and Systems (IROS2004), Sendai, Japan, 2004: 846–851.

60.

Miura

Shinohara

, “Safety Problems of the Car Navigation from the Viewpoint of Psychology of Attention” IATSS Review, 2001; 26(4): 33–44.

61.

Ohno

Ogasawara

, “Information Acquisition Model of Highly Interactive Tasks” Proceedings of ICCS/JCSS'99, Tokyo, Japan, 1999: 288–293.

62.

Kuronuma

Suzuki

Igarashi

, “Estimation method of awareness level using an iconic model for support robots” Proceedings of IFToMM Asian Conference on Mechanism and Machine Science, Taipei, Taiwan, 2010.