Cognitive Human-Machine Interface Applied in Remote Support for Industrial Robot Systems

2.1 Naturally Acquired Data

When controlling an industrial robot system locally, one acquires data both by means of sensory devices, which present measured data via HMI, and by one's human senses, simply by direct system observation. In the remote control, however, one is physically separated from the system, and hence the data acquired by direct system observation, called Naturally Acquired Data (NAD), must be acquired and presented by an additional system (Fig. 1) or will be missed. The data are referred to as Natural, since similarly to Natural Language Processing [18], NAD refers to natural human capabilities:

Definition 1.

Naturally Acquired Data (NAD) are the data that are acquired by human senses without use of artificial aids.

Definition 2.

Natural Acquisition of Data is the process in which human acquires data by human senses without use of artificial aids.

Remark 1.

The concept of NAD is neither restricted to industrial robots, nor to remote operation. It can be applied for any data that are sensed by a human without use of artificial aids. For example, a driver (Fig. 2) acquires data through both direct system observation (NAD) and indirect system observation aided by, e.g., speed and fuel indicators (non-NAD).

Remark 2.

NAD cannot be easily expressed by means of numbers, words, or any other arrangements of ordered abstractive symbols, due to its complexity, strong context-dependency, and the frequent uniqueness of data entities. For example, one is neither able to easily express a complex traffic situation on a street only by means of numbers and words (Fig. 2), nor do thus with a state of an industrial robot cell (Fig. 1).

Since NAD is inherently symbolic, it shall be considered as symbolic data, which require different treatment than commonly encountered classical data. The difference between symbolic and classical data, the conversion between the types, and the principles of Symbolic Data Analysis (SDA), have all been explained by Billard and Diday in [19]. While appropriate tools to process NAD exist, the architecture of additional system acquiring and presenting NAD for remote operation of industrial robots still remains a challenge. In order to effectively comprehend system state and operate it, the remote operator shall be delivered with NAD providing sufficient degree of telepresence [20].

OPEN IN VIEWER

Figure 1.

NAD in remote control of an industrial robot. The camera-like symbol expresses additional system used to provide NAD.

OPEN IN VIEWER

Figure 2.

NAD in car driving.

2.2 Cognitive Human-Machine Interface

The majority of present HMI systems communicate with human through miscellaneous displays expressing data in a form directly inherited from books – by a mixture of structured text, numbers, and supporting illustrations, which has been adopted rather as a proven solution, than a result of multidisciplinary investigation involving engineers and psychologists. Though the mixture is accurate and unambiguous, it requires human brain to perform such mental operations as reading and counting, which have been developed in relatively recent human history due to the needs of civilization [21], [22]. Before civilization, survival in the natural environment caused other abilities to evolve, such as the ability to adaptively adjust sensing range [23], categorize perceived phenomena and objects [24], ignore insignificant and focus on significant ones [25], learn new categories [26], and solve encountered problems [27]. For instance when directly observing an industrial robot system, one can easily track several moving objects, estimate their motion parameters, and moreover envisage possible collisions and their consequences, by utilizing the abilities. On the contrary, the same data cannot be as efficiently provided by means of standard HMI, which is excellent for providing classical and fairly useless for symbolic data. There is a need, therefore, to develop a new type of HMI.

The most promising avenue appears to be development of multimodal HMI involving Cognitive Info-communication, since data acquisition and presentation in the communication are based upon the mentioned human abilities [16] – multiple senses are redundantly involved in parallel, and symbolic data is provided in a form of stimuli/action patterns corresponding to known psychological concepts:

Definition 3.

Cognitive Human-Machine Interface (CogHMI or Cognitive HMI) is the HMI that allows humans to communicate with a machine by means of symbolic data.

Remark 3.

In the fact, every HMI is to some extent a Cognitive HMI, since after a few uses even “a numerical displayer” becomes a unique concept of “the numerical displayer” that provides an unknown value of a known parameter. Cognitive HMI is used, therefore, to specify the HMI that at least partially is design to provide symbolic data. A trivial example of an everyday-use device which has Cognitive HMI is a TV set. Though it does not allow for real interaction as it is for Teleoperation [28] and Virtual Reality [29], it provides symbolic data in the most primitive way, by direct reproduction of realty. For instance, when watching a speaker on the television news, one easily recognizes his or her emotions by facial expression and voice intonation.

One can expect that the application of Cognitive HMI in the system will raise the efficiency of remote operation, since it is capable of providing NAD in an easily comprehended form. The use of Cognitive HMI will also allow for parallel data presentation, enhance human attention management [30], and make interaction less tiring, perhaps even enjoyable.

2.3 Non-Engineering Concepts in the Investigation

Orienting reflex is an involuntary body movement that orients an organism's receptors towards the part of environment in which sudden changes of stimuli occur [31]. For instance, when one suddenly hears the noise of a truck approaching from behind, one immediately turns one's head around to evaluate the danger by one's eyes.

Sensory adaptation occurs when a stimuli does not change and is at a safe range for the sensing organism [23]. For instance, when one is concentrated on reading an article, one does not feel the pressure exerted on one's foot in a slipper, since it is at a safe range. One will become, however, immediately aware of the pressure, when a child playing nearby steps on one's foot.

A Concept represents an entire class of objects (e.g., an apple), states (e.g., being warm), phenomenon (e.g., shining), or abstractions (e.g., truth, and the number four), that have common properties. One has an implicit knowledge of how concepts relate to each other, e.g., “dogs” are members of the larger concept “mammal”, which includes also “humans” [32].

Categorization is a process of assigning an object to a concept. When categorizing an object, one treats it according to its properties associated with the concept, including the properties that were not perceived directly [24].

Localization and recognition are major functions of human perceptual system. The function of localization is used to spatially localize perceived objects, whereas by means of recognition, perceived objects are categorized into concepts [33].

Selective attention (e.g., selective listening) is the cognitive process in which one remains selectively focused on one aspect of the environment, while filtering or ignoring others. For example, when selectively listening, one is focused on only one sound [25].

Sensory substitution is the phenomenon in which perceptions characteristic of one sensory modality are made when another sensory modality is stimulated. For example, blind people can not only accurately recognize and localize objects by means of audition, but also perceive images comparable with visual. Sensory substitution is a result of a certain combination of needs and training [34].

3.1 Structure of the System

The system consists of local and remote rooms that are physically separated (Fig. 3). The industrial robot system is situated in the local room, whereas the operator, who exercises control over the system, is situated in the remote room. Communication between the rooms is executed by local and remote PCs implementing LabView software. The local PC is interfaced with the industrial robot system and additional sensory devices installed for the purposes of the system, whereas the remote PC is interfaced with the remote operator via HMI devices.

3.2 Sensory System in the Local Room

The robot controller (NACHI AX10CE-5111) is connected with local PC via NACHI Communication Software (RS232) and Real-Time Communication Unit (Ethernet); thus system data and movement of the robot manipulator (NACHI SH133-03) can be controlled in a real time. In addition, load cell force sensor (LORD 125/600 F/T), which is also interfaced with the PC (RS232) and is positioned on the end of the manipulator arm, allows the measurement of contact forces. The whole system can be observed remotely by means of moveable Ethernet camera (Panasonic BB-HCM515), whose pitch, yaw, and zoom are adjustable.

OPEN IN VIEWER

Figure 3.

Structure of the experimental system.

OPEN IN VIEWER

Table 1.

Data acquisition and presentation.

No	Local Room				Remote Room
	Source of Data	Acquired Data	Human Senses Involved in Sensing NAD		HMI Devices Providing the NAD	Human Senses Involved by the HMI	Data Expression
1		the robot's axis movement	Vision/Audition		Headphones	Audition	rising/falling cello melody for positive/negative axis movement; spoken messages
2		the robot in singularity area	Vision		Headphones	Audition	disturbing sound with volume rising when immerging into the area; spoken messages
3		perpendicularity of the robot's tool to a surface	Vision		Headphones	Audition	consonance sound with volume rising when approaching the perpendicularity; spoken messages
4		close proximity to the robot's software endstop	Vision		Headphones	Audition	rhythmical beats with tempo rising when approaching the endstop; spoken messages
5		the robot's motors turned on/off	Vision/Audition		Headphones	Audition	for motors turned on the data no. 1–4 can be perceived; spoken messages
6		the robot's emergency stop pressed	Vision/Audition		Headphones	Audition	sound of alert, when pressed; spoken message
7		occurrence of collision between the robot and environment	Vision/Audition		Headphones	Audition	when occur, sound of crashing window; spoken message
8a		magnitude of contact forces developed on the robot's end effector	usually cannot be directly sensed		Headphones	Audition	triangle waveforms whose amplitude, distortion and panning are varying with regard to input data
8v					LCD Monitor	Vision	2D/3D graph of the forces (FX,FY,Fz) that can be widely adjusted with regard to color and shape
9	Ethernet Camera	camera's vision	Vision		LCD Monitor	Vision	camera's vision

3.3 Human-Machine Interface in the Remote Room

When designing the experimental system, it was decided to incorporate only audio-visual HMI devices, since the devices are commonly available on the market and can be inexpensively applied. Another reason is that one can expect the dependences observed for human vision and auditory sense to be widely universal for other senses as well.

The HMI consists of two LCD monitors, headphone set, 3D mouse (3D Connexion Space Navigator) which is used to control the manipulator's movement remotely, and a standard PC mouse and keyboard.

3.4 Data Acquisition and Presentation

In Table 1, the data dealt with by means of Cognitive Info-communication methods is presented. In the part entitled “Local Room”, source and type of acquired data are determined, as well as human senses involved in sensing the data locally. In the part entitled “Remote Room”, one can see how the data are presented to the remote operator, and which of the human senses are utilized in the presentation. What is particularly worthy to notice in the table is that some of the data provided remotely are presented to different senses than they would be for direct system observation in local control (sensory substitution).

The data enumerated as 1–8 in Table 1 are provided also in a form of conventional numbers. They are, however, not included in the table, since they are insignificant for the following considerations.

4.1 Visual Presentation

The qualitative difference between the lack and presence of forces (Table 1, no. 8v) is immediately perceived when one is focused on the projection (Fig. 4). Conversely, when one is not focused, the difference cannot be perceived. Without selective (visual) attention, such projections therefore appear to be useless.

By means of 2D projections, one can easily evaluate quantitative differences between the forces, since the difference in length is clearly perceived (Fig. 4a). The projections, however, do not provide any information about spatial relations between the forces. On the contrary, 3D projections (Fig. 4b-c), when spatially adjusted to the orientation of the camera view, allow one to comprehend and associate the relations with observed manipulator's movements by means of localization function.

OPEN IN VIEWER

Figure 4.

Camera's vision with visualization of the contact forces.

Shape behavior of bars, which expresses contact forces, can be beneficially used to indicate their magnitude, only when it is associated with known concepts. For instance, bowed bar (Fig. 4c) can be easily associated with bowed ruler – one immediately associates its deformed shape with tactile sensations developed when bending the object. Similarly, a bar with moving strips (Fig. 4b) can be associated with liquid flowing into an elastic container – changes in a force are similar to changes in container's volume, or pressure developed in the container.

In contrast to shapes, one's abilities to recognize changes in colors, and to associate them with corresponding contact forces, appears to be strongly limited, since colors play minor role in recognition function [35]. Such clearly distinguishable colors as red and blue, however, can be effectively used to indicate opposite states. For example, negative and positive direction of forces (Fig. 4) can be associated with concepts “cold” and “hot”.

4.2 Sounds Corresponding to Concepts

It was observed that the most efficient means of symbolic data presentation are sounds directly corresponding to known concepts (Table 1, no. 6 and 7). For instance, the sound of crashing window (no. 6) is immediately categorized into a concept of “collision”, since in everyday life the sound is likely associated with a potential danger of injury. Likewise, the sound of alert (no. 7) is categorized into a concept of “emergency state”, since it is similar to the sound of a fire fighters' or ambulance's siren, which is usually heard in emergency situations.

Though some of the sounds do not correspond directly to known concepts (Table 1, no. 1–3), when the sounds are clearly distinguishable and provide symbolic data, one can easily develop new concepts for them. For instance, one learns to associate a cello melody with the concept of “the robot's axis movement” (no. 1), since it is unlikely to hear any other cello when controlling the system. Likewise, the disturbing sound (no. 2) is associated with the concept of “singularity area”, as it is equally undesired to hearing the sound. Deeply focusing consonance sound of organs (no. 3) is associated with the concept of “perpendicularity”, which is equally desired to hearing the sound.

New concepts can be also developed by analogy to existing ones (Table 1, no. 4 and 8a). The density of rhythmical beats (no. 4), which rises when approaching the endstop, seems to be analogical to tactically perceived pressure – similarly to fast beating, high tactile pressure cannot be ignored; thus the closer is the endstop, the more one is forced to be focused on the sound (selective listening). Another such example is the sound of triangle waveforms (no. 8a). Above a certain threshold, the higher a contact force, the more corresponding waveform is distorted – the level of waveform distortion appears to be analogical to the level of roughness perceived tactically [36].

In general, it was observed that the sounds corresponding to concepts tend to be able to be used only to provide symbolic data.

4.3 Musical Sounds

The cognition of musical sounds (Table 1, no. 1–4 and 8a) is strongly dependent on individual musical skills. It has observed that well trained individuals can successfully acquire classical data with a limited resolution, while average individuals are merely able to recognize qualitative differences such as the difference between rising and falling melody (no. 1). Since the sounds are perceived rather as sounds corresponding to different concepts than musical sounds, one can develop new concepts for them to provide symbolic data. It was also observed that one quickly accustomed to consonant sounds (no. 3), and without conscious intension to follow them, one forgets about their presence. On the contrary dissonance sounds (no. 2) cannot be neglected; hence they are suitable to indicate undesired situations, e.g., robot approaches singularity. The overall drawback of musical sounds is that they require selective attention, and thus cannot be effectively comprehended in parallel.

OPEN IN VIEWER

Figure 5.

A typical robot operation performed remotely - adjustment of contact forces. The plot demonstrates quantitatively changes in human focus indicated by the filled parts, and the relationships between provided to human data and action.

4.4 Spoken Messages

The spoken messages appear (Table 1, no. 1–7) to be a beneficial addition to the sounds, since they can provide symbolic data, similarly as it is for two individuals exchanging the data in a conversation. One also rapidly learns to recognize the voice of speaker; thus a new concept is developed – “a spoken message in the system”. Since the content of the messages is commonly relevant for the control, the occurrence of the voice results in selective listening. The listening is, however, also a drawback of the method, since one neglects other sounds in a great range.

4.5 Mixed Audio-Visual Presentation

When sounds corresponding to known concepts are played in parallel, one does not encounter any significant difficulties in data comprehension. On the contrary, spoken messages and sounds utilizing the musical ear cannot be comprehended in parallel, since they require selective listening.

It is a remarkable fact that, without the sounds, when not observing the displays, one is not aware of occurring contact forces (Table 1, no. 8). However, when the sounds corresponding to contact forces (no. 8a) occur, one involuntarily turns one's head/eyes into the display due to orienting reflex. In general, one usually at first realizes the occurrence of the forces by means of sound (no. 8a), and then verifies their magnitude via visualization (no. 8v), which for an average human provides more accurate data than the sound.

On Fig. 5 one can see an orienting reflex occurring in a typical robot operation – contact force adjustment. The appearance of contact force sounds (step 4), causes one to move focus from camera vision to visualization of contact forces (step 5). The operator's state and the changes in auditory focus were subjectively reported by the operator, while the changes in visual focus and human actions were registered objectively by the camera recording gaze movement and the LabView software.

5.1 Data Presentation Methods

Basing on the observations, three data presentation methods can be distinguished (q-DPM, Q-DPM and S-DPM: Table 2) in which, depending on the method and circumstances, data is comprehended in 1–3 steps (Table 3) - obviously, the fewer the steps in data comprehension, the shorter the comprehension time.

OPEN IN VIEWER

Table 2.

Qualitative comparison of data presentation methods. The cells with the desired properties were filled to clarify the table.

	Data Presentation Methods
	Quantitative			Qualitative	Symbolic
Abbreviation	q-DPM	Q-DPM			S-DPM
Data Wrapping	Numbers	Text	Speech	Musical Sounds	Symbols
Conceptuality	no	yes	yes	yes	yes
Accuracy	unlimited	limited	limited	limited	strongly limited
Comprehension Time	long	moderate	moderate	moderate	negligible
Involuntary Comprehension	no	usually no	usually no	usually no	yes
Parallelization	no	no	no	no	yes
Multimodality	no	no	no	no	yes

OPEN IN VIEWER

Table 3.

Data Comprehension for different data presentation methods. The cells are filled to clarify the table.

Data Comprehension		Data Presentation Methods
	Process	q-DPM	Q-DPM	S-DPM
1	Pattern recognition	yes	depends	No
2	Concept recognition	yes	yes	depends
3	Context recognition	yes	yes	yes

OPEN IN VIEWER

Figure 6.

DPMs and typical features for exemplary data.

Since the mechanisms of human cognition are still not very well understood, the division is in a certain range contractual, and some exceptions can be found for it. For instance, while numbers undoubtedly convey quantitative data, and symbols undoubtedly convey symbolic data, one can argue that text, speech and musical sounds should not be classified as qualitative, since they can also convey symbolic data. In fact, at the final stage of each DPM, data are transformed into symbolical form as concepts, since humans usually use concepts for thinking.

5.2 Data Reduction in Data-Rich Environments

Nowadays, data acquisition is no longer a challenge [39]. The great number of easily available, inexpensive sensors and the high bandwidth of current communication standards enable the development of complex Distributed Sensor Networks (DSN) which automatically acquire large amounts of data. The present challenges are, instead, efficient information and knowledge extraction, and efficient data presentation for human.

In biological systems, such as with humans, a small number of senses provides large amounts of data, which at the early stage of processing are reduced into the data necessary at that point in order to operate. The data is subsequently generalized into concepts among which again only a few reach consciousness. Despite that reduction, one is still capable of effectively perceiving complex states and dependencies of the surrounding environment. If data in data-rich environments, such as the remote operation of an industrial robot system, is not treated analogically, data comprehension becomes ineffective. Therefore, the two main reasons why the environments are usually difficult to operate in are information overloading (too much data at once without generalization) and no information filtering (presenting all the data, including irrelevant data).

Though one is able to measure the everyday environment with almost unlimited accuracy, usually a general understanding of the environment is sufficient to effectively operate in it. First of all, the HMI shall provide only generalized data that are easy to comprehend and sufficient to understand the system state. Then, it can provide more details, either if the need for them is inferred from the current system state and user's performance, or if it is actively requested by the user through the HMI. The investigation has shown that there are three DPMs that can be synergically used in the approach – system state shall be presented mainly on the level of S-DPM, whereas the levels of Q-DPM and q-DPM shall be entered only if details are necessary (Fig. 7). In addition, such phenomena as the observed orienting reflex can be used to support attention management and switching between DPMs.

OPEN IN VIEWER

Figure 7.

DPMs for data-reduction in data-rich environments.

5.3 Structure of Cognitive Human-Machine Interface

The investigation has exposed the limitations of the existing HMI, which cannot be overcome by simple improvements. Basing on the obtained results and the multidisciplinary discussion presented previously, we have proposed a structure of Cognitive Human-Machine Interface (CogHMI, Fig. 8) which allows reducing or even completely removing these limitations.

Opposite to traditional HMI, CogHMI adjusts its performance to current human needs, acting rather like an active mediator between the human and the data system, than a passive data displayer operated by a control panel. CogHMI is capable of guessing human needs, adaptively selecting data (Information Filtering and Importance Evaluation), and providing the data in the most convenient form for the human (Cognitive Info Encoding). Similarly, as it is for two natural cognitive systems, e.g., two humans, the data between CogHMI and the human flow in two feedback loops – conscious and unconscious – and the performance of the CogHMI subcomponents is adjusted with regard to estimation of the human and system states. While the system state is evaluated by means of a distributed sensor network and a system model, e.g., a model of an industrial robot cell, the human state is evaluated by means of a customizable human cognitive model and multimodal input devices. The devices provide both conscious input (Human Requests), such as keyboard typing, and unconscious input (Human Needs), such as eye movements.

Though named differently, certain CogHMI functionalities have already been developed, and some of them can even be found in commercially available systems. CogHMI's functionalities can be found, for instance, in the systems which monitor driver fatigue, lack of attention, and sleepiness [40], and in the systems detecting drunk driver such as the commercially available Driver Alcohol Detection System for Safety (DADSS) [41]. Both of them evaluate the human state without conscious use of the systems to support human performance. Another more sophisticated example is Intelligent Space (iSpace) [42], which is an environment that attempts to evaluate human needs by means of a distributed sensor network, and to satisfy them with the aid of service robots.

OPEN IN VIEWER

Figure 8.

Structure of CogHMI - the HMI with conscious and unconscious feedback loops. The stages of data processing, which were discussed in detail previously, are emphasized using colours.