Human models in human-in-the-loop control systems

Abstract

Understanding the collaboration between physical systems and human is an essential task in man-machine systems. This area of research has been significantly explored in the recent years with the focus on the machine side. Much less attention has been directed to the other side in the man-machine systems, which is the human. The aim of this review is to discuses the major directions and challenges in man-machine systems from control theory perspective with a focus on human modeling and human attributes in the man-machine system. Four directions has been identified; these are: 1) Understanding the attributes and the limitation of the human operator; 2) Categorizing the human-in-the-loop applications and derive generic bounders for each category; 3) Building a realistic application-based model for the human behavior; and 4) Integrating the human models into a formal control synthesis methodology. We surveyed the contribution in each of the four challenges and pointed out the limitation in the proposed ideas. Particular attention is given to the model identification of the human behavior.

Keywords

Human-in-the-loop human behavior cyber-physical systems

1 Introduction

Nowadays, humans are interacting with machines in every aspect of their lives, either actively controlling the machine or passively getting served by them. Human-in-the-loop is the term that is often used in the control theory literature to describe the involvement of the human with physical systems as [1 –13]. The human-in-the-loop particularly refers to a situation where a system or a machine is controlled, fully or partially by a human. The control is refereed to be active if it was through some control manipulators. Human-in-the-loop can also mean that, the human is monitored or even controlled by a machine, which we refer as passive. In the active human-in-the-loop setup, the human observe the output of the system through, for instance, a display in which he or she can view all the required information to update their control actions or their decisions. This is typical feedback interconnection control of a dynamic system (the plant) and a controller (the human) see Fig. 1. Therefore, the human-in-the-loop can be modeled as an input-output system, similarly as any dynamical system. This has lead to the development of several dynamical models that mimics the human behavior in human-in-the-loop applications.

Fig.1

Human in the loop control system.

The term human-in-the-loop appears also with a different perspective in the field of human-computer interaction (HCI). HCI focuses on the interfaces between users and computer technology.Understanding of human behavior and observing the way in which humans interact with different computer technology platforms and design technologies that allow humans to interact with computers in different ways [14 –16].

One important direction of human-in-the-loop research is the study of skill transfer between the skills of human operators and autonomous systems [17 –21], where autonomous systems such as robots have the opportunity to learn from the operator. Also, shared control between robots and humans is another human-in-the-loop application, where a rich research questions such as co-adaptation between the human and the robot are raised. Furthermore, the behavioral and social change of humans while working and cooperating with different levels of intelligent systems is another important area where the concept of man-machine systems is presented [22 –31].

The term behavior is defined in [15] as a set of temporal changes in neuromuscular and physiological activity that can last from a few milliseconds (a blink) to minutes (talking) or hours (sitting). Examples of this behavioral signals include attitude such as fear, inattention, joy or stress, Also, it can be manipulation such as the actions used to move an objects in the surrounding environment. Understanding the human behavior is done by translating their behavioral signals in particular context into a description of the given behavior.

One way to understand an operator behavior in a cyber-physical system is to analyze his input and output from a control theory and data mining perspective. Finding unknown repeated behaviors or patterns can lead to better understanding of operator’s action in particular situations. Also, mining the repeated behaviors of skilled operators can be used as a benchmark for teaching either other human operators or autonomous systems.

The man-machine systems are strongly connected to automation and therefore, control theory was the primary contributor to this field [32, 33]. Control theory scientists started in the early 60th or even before to build models for human behavior using dynamical systems and control tools. Manual control was the known name at that time, where the human works as an operator that operate the system. Since then, the development of the human models continued to use the automatic control tools. Manual control still exists in highly automatic and intelligent systems such as aircraft and ground vehicles, and a skilled human operator is able to make the process performs at the desired efficiency level.

The term human-in-the-loop, in control theory perspective, simply means a plant or a system controlled fully or partially by a human through some control manipulators such as joystick or control panel. The human can observe the output of the system through a display in which he or she can view all the required information to update their actions. This setting is a typical feedback interconnection of a dynamic system (the plant or the system under control) and a controller (the human operator).

Most of the existing models for the human operator, as we will see later in the paper, treats the human as a physical controller either linear or nonlinear. However, any physical controller is composed of several mechanical components and softwares. These components are able to repeat the same actions under the similar conditions for the system under control for any number of times. In contrast, the human, as an operator or a controller, cannot repeat or reproduce the same exact actions under the same stimulus. This is due to several reasons such as fatigue and stress. In addition, the physical controller can work continuously for long time without any issue, however, a human operator cannot function for more than one regular working shift [34].

Cyber-physical systems, Internet of things, human-robot interaction are modern names that accommodate human-in-the-loop applications. In [35], three main challenges in feedback control with humans in the loop in cyber-physical systems are defined. They are: (i) the need for a comprehensive understanding of the complete spectrum of the types of human-in-the-loop controls, (ii) the need for extensions to system identification or other techniques to derive models of human behaviors, and (iii) determining how to incorporate human behavior models into the formal methodology of feedback control. In the recent survey [36], an overview of the human-in-the-loop in cyber-physical systems is presented with a taxonomic exercise focused on the general roles of the human component together with a requirement analysis.

The aim of this paper is to discuses the major directions and challenges in man-machine systems from automatic control theory perspective. Four directions have been identified:

Understanding the attributes and the limitation of the human operator;

Categorizing the human-in-the-loop applications and derive generic bounders for each category;

Building a realistic application-based model for the human behavior, and

Integrating the human models into a formal control synthesis methodology

In the following sections, a detailed description of each of the above direction with highlights on the limitation in the proposed ideas is discussed. Particular attention is given to the model identification of the human behavior [35, 36].

2 Human physical limitations and behavioral attributes

The first step towards real man-machine system understanding is to understand the human physical limitations and behavioral attributes while dealing with physical dynamical systems. This involves many aspects such as the way in which humans process information, the limitation of humans sensory system and humans decision-making process. As it becomes clear that humans behavior and response in control loop inside cyber-physical systems are fundamentally different from the physical controllers. This is because of the difference between the biological nature of humans and the artificial and cyber-physical nature of machines. This difference makes several significant challenges understanding man-machine systems.

In this section, we will discuss our review by highlighting some of the physical limitation and behavioral attributes that distinguish humans from dynamical cyber-physical systems. The term behavior is defined as a set of temporal changes in neuromuscular and physiological activity that can last from a few milliseconds (a blink) to minutes (talking) or hours (sitting) [15]. In other words, the set of actions or reactions of a person in response to external or internal stimulus situation. Behavior also can be observed, recorded and measured. This allows better understand diffidence between humans and physical dynamical systems. Equally understanding this difference will help us to map the engineering theories and modeling tools in order to model human subjects in human in the loop system.

There are certain basic properties of humans that are routinely observed when studying human-machine interactions. The following list highlights some of the factors that define the physical limitation and behavioral attributes of the human operator in human-machine interactions.

Response Delay:

Humans, in a most simple form, do not behave linearly; they exhibit time delays in their reaction to stimuli [37]. For instance, the response delay in vision is at least 40 msec [38].

Threshold Limits:

We, humans have limited sensing and actuating capabilities. There are threshold limits that we can not operate beyond them. For instance, our vision sensory is limited by what it is known as flicker fusion threshold. Equally, our audible frequency is bounded by 20 to 20, 000 Hz. The Same limitation applies to our actuation capabilities. These limitations make humans very limited compared with the dynamicalsystems.

Short-Term Memory and Working Memory:

As described in [39], short term memory is defined as the ability for holding a small amount of information in our minds in an active and readily available state for a very short time. This duration of time is in the order of few seconds. On the other hand, working memory is a cognitive system with a limited ability that is responsible for the transient holding, processing, and manipulation of information. Working memory is an important process for reasoning and the guidance of the way that we makes decisions [39].

Anticipation and Prediction:

Prediction can be defined as our belief that some event will happen in particular circumstances. However, anticipation is when we are being ready for what might possibly take place and prepare a plan for how to deal with it once occurred. The main diffidence here is that prediction does not require action, while anticipation normally does and affect our current decision. Therefore, anticipatory behavior is a process that depend on a compilation of our belief about the future, the experience from the past, and what we are going through in our present. We, humans, adopt the anticipatory behavior in any decisions-making. This makes us compliantly different for the way that dynamical systems work.

Learning:

Humans are capable of learning new knowledge in many different ways, which is a unique characteristic of human beings. This knowledge allows human to increase their capacity and experience. The concept of learning in humans is different than the learning in machines, which is in the end of the day statistical learning.

Adaptation:

Humans are very adaptable systems; they are able to cope with the environment changes around them. The learning capability and the anticipatory behavior of humans gives them very high level of adaptation. Of course, there are dynamical systems that can be described as an adaptive systems; for instance, adaptive control systems. However, the notion of adaptation in human is more deeper and involved. For instance, human is the ability to adapt to not only different controlled dynamics but to altered operating conditions as well.

Multi-Inputs Behavior:

Human relay on several inputs and integrate them to make a decision. For instance, human rely on vision, vestibular and kinesthetic, and auditory in order to drive a car.

Creating New Strategies; Not Everything We Know We Learned:

As indicated in [40], it is not only anticipation that control the human activity. Several other factors have as strong effects such as guessing, expectation, prediction, forecast, and planning. These factors some times comes against what we anticipate. In other words, anticipation and reaction can be considered only in their unity [40].

Highly Nonlinear or Perhaps Chaotic Behavior:

Scientific research of brain activity in varies events have shown that only chaotic behaviour are exists when we come to human mind. As concluded in [41], We have found that brain function cannot be explained in terms of features of neurons taken individually or as part of a local network, nor is it adequately characterized as a passive reaction to stimuli. The nature of the internal feedback in the brain system along with the incremental uncertainties, which amplified with time suggests that the brain is a chaotic system, where a long-term predictions of a person behaviour isimpossible [41].

Causality vs Non-causality:

The causality is a fundamental property in physical systems. A system is causal if its output depends only on present and past inputs. In other words, the output of the causal system do not depend on the future inputs. This property makes these physical systems realizable and can be implemented in real life. In contrast, a system which present output or response depends on future inputs is called as a non-causal system.

Mathematically speaking, if we defined h (t) be the response of a linear time invariant system at t = 0 . Let u (t) and y (t) denote the input and output of the system. Suppose that u (t) =0 for t < 0. Then we have the following relation

$y (t) = \int_{0}^{\infty} h (t - τ) u (τ) d τ = \int_{- \infty}^{0} h (τ) u (t - τ) d τ .$

In the case of linear time invariant system is causal then there should be no response before the input is applied, i.e. h (t) =0 for t < 0. Therefore, the causal system can be written as follows $y (t) = \int_{0}^{t} h (τ) u (t - τ) d τ$ t ≥ 0 .

This shows that for a causal system, h (t) gives zero weight to inputs that occur in the future. Systems that do not have this property are non-causal.

3 Taxonomy for human-in-the-loop applications

Generic taxonomies and classifications are required to allow for comparison of different human-in-the-loop applications and approaches. It is obvious that human behavior will depend on the type of the application they are involved in. Understanding the different types of applications and categorize them will help in identifying patterns of human behavior that emerge in different applications. A very limited effort has been given to this direction. Only the recent survey [36] attempt to present a taxonomy of the applications that involves human. The taxonomy is based on the human roles in a givenapplication.

Taxonomy based applications should involve several key features that distinguish different application and the human role in this application. Here we present some of these key features that we think are essential in such taxonomy.

3.1 Intelligence level of the system under control

The behavioral aspect of the operator will depend on the intelligence level of the system under control and the ability of the system to execute decisions autonomously. Grouping the applications with the same level of intelligence will help to identify similar behavioral patterns of the human operators. Defining the characteristics and the boundaries of their levels is not an easy task, however, it is an essential key feature that defines the human model behavior in controlling task.

For instance, one could consider level zero correspondence to complete the manual operation, where the system has no apprehension of its current state and all sensing, control and actuation is done externally, for example, position status indicator, a locally flashing fault indicator light. Whereas the highest level of intelligence correspondence to complete autonomous system that can execute complex decisions and has the ability to predict and adapt to all situations. Between there two levels, there should be several other levels that stare from basic automation process to advanced control strategies.

3.2 System’s controllability

Engineering systems have different controllability degree based, for instance, on the nonholonomic nature of the system. For example, nonholonomic mechanical systems such as wheeled mobile robots, cars, autonomous underwater vehicles, unmanned aerial vehicles, under-actuated robots, cannot move in an arbitrary direction in its configuration space. The degree of controllability often defined as the minimum input energy to change the system’s states [42].

3.3 Cost of failure to operate the system

Failure is something we try to avoid at all costs, since the cost of failure is expensive. Our decision-making process and our behavior, while operating a particular system, depends strongly on the cost of failure of that system. In other words, systems with high failure cost will force us to behave more carefully than the systems with less failure cost.

3.4 The system resilience and robustness

A resilient system is a system that adapt to challenges and uncertainty by changing its method of operations while continuing to function. However, a robust system is the system that continue functioning in the presence of bounded uncertainty without any changes to the original system. The degree of robustness and resilience of the controlled system affect the human behavior while controlling the system. Therefore, it is logical to classify the systems using its degree of robustness and resilience.

3.5 Skills needed to operate the system

Human behavior, as an operator or a controller in a particular scenario, will depend on the time of contact between the human and the controlled system.

4 Modeling and identification of human behaviors

Model identification for human behaviors is, so far the most visited research area in the human-in-the-loop. Control theory was the main founder for most of the human models. Since the 1960s, there has been strong progress in finding a proper model that mimic the human behavior in manual control. Most of this models were driven from an application such as airspace to model the human as a pilot.

The first recognized control theory orientated models for the human operator was early in the 60s. A human operator was modeled as servomechanisms with sensors and actuators acting as the sole-in-the-loop control element, Fig. 1. The independents of the operator model and the plant or the machine was assumed as an initial hypothesis, i.e., the human operator behavior model could be combined with any control process or any machine he is controlling. The work by McRuer and his colleagues in [32 , 43] was very successful, in particular, the crossover model. In this section, we will review briefly some of this model and their developments. In Table 1, a summary of the of models considered in this paper is presented.

Table 1
A summary of the models discussed above

Human Models

Control theory-based models Bio-inspired models Cognitive models

Quasi-linear model Drift Diffusion Model Cognitive models consider the low-level details of human behavior

Optimal control model Coupled drift diffusion model and typically focuses on prediction and generality.

Internal model hypothesis Collective decision making model

Preview control model

Intermittent control

Hidden Markov models

Human Models
Quasi-linear model	Drift Diffusion Model	Cognitive models consider the low-level details of human behavior
Optimal control model	Coupled drift diffusion model	and typically focuses on prediction and generality.
Internal model hypothesis	Collective decision making model
Preview control model
Intermittent control
Hidden Markov models

4.1 Control theory-based models for human

Quasi-linear model:

There have been several models for human as an operator or a controller that operate a dynamical system. Perhaps, the quasi-linear model is the first to be known. An early review on the development of that model is given in [43] by the model founders.

The quasi-linear model developed from the fact that most of the nonlinear systems have similar responses to specific inputs comparing to the responses of equivalent linear systems to the same inputs. For a given input-nonlinear-system combination, the response of the nonlinear system can be divided into two parts; one component which corresponds to the response of an equivalent linear element driven by that input and an additional quantity, called the remnant, which represents the difference between the response of the actual and the equivalent linear element [43].

$H (s) = K \frac{T_{L} s + 1}{T_{I} s + 1} \frac{e^{- s τ}}{T_{N} s + 1}$ (1) Here, K is the human gain, τ is the time delay due to the human reaction time, T_L is the lead time constant, T_I is the lag time constant, and T_N is the neuromotor dynamics constant.

This model is also known as crossover model since the performance of the human based on this model depends on the crossover frequency ω_c. At this frequency, the open loop transfer function satisfies

$| H (j ω_{c}) S (j ω_{c}) | = 1,$ (2) where S (s) is the transfer function of the controlled plant. This implies that ω_c is acting as a performance boundary of the closed-loop between the controlled element and the human. For more about this model, [34, 43].

Optimal control model:

The advances in modern control theory by Kalman and others led to more sophisticated models for the human operator. Kleinman and others introduced an optimal control-based model (OCM) for the human operator in [44, 45]. The main hypothesis behind the optimal control-based model is that a well-trained, well-motivated human operator behaves in an optimal manner, subject to his inherent limitations and to the requirements of the control task [44]. In this model, the human operator is working as a minimizer of the error between deviations of the actual process trajectory and the desired trajectory.

Based on the assumption that human behaves optimally in some sense, [44] proposed a quantitative model for the response characteristics of the human operator. The model consists three main elements; time-delay, a representation of neuromotor dynamics, and controller remnant as limitations; see Fig. 2.

Internal model hypothesis:

The internal model in control theory refer to the process that simulates the response of a given system in order to estimate the output of the system disturbance and was first intruded in [46]. In the recent work in [2 , 47–50], reversal experimental setup was conducted to show that human operator can be modeled according to the internal model hypothesis using system identification. They argue that human operator model can be divided into two components, one of which is the inverse dynamics of the controlled system in a feedforward and the other component is to be identified from the system identification; see Fig. 3.

Preview control model:

In many control systems, it is required that the outputs should track the reference signals in the presence of exogenous disturbances, see Fig. 4. If the future information of the reference signal or the disturbances is available, then we can greatly improve the performance of transient responses.

In the preview control method, the most important thing is that the known future demand output or disturbance signal affects the operation of the control system. As a typical method, the augmented error system plays a vital role in the design of the controller in discrete-time systems. Since the relationship between the system and the future signal was established by the difference operator, discrete-time systems have become very popular in many research fields.

Intermittent control:

Gawthrop and others developed a model for the human operator based on intermittent control theory [51 –59]. The intermittent control model for a human operator are building on the optimal control model developed by [44]. The model assumes the following hypothesis general theory of human control systems must include continuous as well as intermittent processes which incorporate discrete switching [53].

Hidden Markov models:

Human behavior model based on Hidden Markov model was proposed in [60]. In this model, the human is considered as a device with many numbers of internal mental states. Each mental state has its own particular control model that describe a particular behavior and interstate transition probabilities. An example of such models would be a set of standard linear systems (e.g., Kalman filters with a simple control law), with different dynamics and measurements. These linear models are sequenced with a Markov network of probabilistic transitions. The short-term behavior is described by the linear model dynamics, and long-term behavior is described by the transition between these linear models.

Fig.2

Optimal control model for the human.

Fig.3

Optimal control model for the human.

Fig.4

Preview control.

4.2 Bio-inspired models

Bio-inspired models for human behavior focus on the collective decision-making of group of humans share some interest. These models are mathematical models inspired by the biological evolution of animals birds, insects, or humans, that describe how a group of individuals can make a collective decision. It combines approaches in the social and natural sciences. Bio-inspired models are based on the fact that group decisions in animals pose many similar questions to those in humans [61]. In [61], they suggested a categorization of group decisions into two classes; aggregate/consensus decisions, and interactive/combined decisions. Aggregate/consensus decisions when a group makes a single collective decision, e.g. between multiple options, that is ‘binding’ in some way for all members. Interactive/combined decisions in which there need not be a single collectively binding decision, but in which individuals decide interdependently with oneanother.

Drift Diffusion Model

This model considers decision-making scenario in which two alternative choice task is given to each individual and has to decide between the two alternatives [62]. The evidence aggregation is modeled by the drift-diffusion process defined by $dx (t) = β d t + σ d W (t), x (0) = x_{0},$ (3) where β∈ and σ ∈ _>0 are, respectively, the drift rate and the diffusion rate, W (t) is the standard one-dimensional Weiner process, x (t) is the aggregate evidence at time t, and x₀ is the initial evidence.

Coupled drift diffusion model

This model considers a set of n decision-makers performing a two alternative choice task and let their interaction topology be modeled by a connected undirected graph G with Laplacian matrix L ∈ ^n×n. The model is mathematically described by the following coupled drift diffusion model: $dx (t) = (β 1_{n} - L x (t)) d t + σ I_{n} d W_{n} (t),$ (4)

Collective decision making model:

In [63], they consider a set of N agents. Let $A \in ℝ^{N \times N}$ be the agent network adjacency matrix, with a_ij ≥ 0 and a_ii = 0 for all i, j = 1, …, N and j ≠ i. Also, $D \in ℝ^{N \times N}$ be a diagonal matrix with $D_{ii} = d_{i} : = \sum_{j = 1}^{N} a_{ij}$ and let L = D - A be the network Laplacian matrix, the standing assumption that the interconnection graph is strongly connected and balanced, that is, rankL = N - 1 and $1_{N}^{T} L = L 1_{N} = 0$ , where 1_N is the N-column-vector with all unitary entries is also made.

The distributed dynamics as a candidate for the realization of bio-inspired collective decision-making behavior is given as follows; ${\dot{x}}_{i} = - d_{i} x_{i} + \sum_{j = 1}^{N} u a_{ij} S (x_{j}), i = 1, \dots, N$ (5) where $S : ℝ \to (- 1, 1)$ is a smooth sigmoidal function.

Where, the term uS (x_j) in (5) is the opinion of agent j as perceived by a generic agent i. The parameter u controls this smaller value and models the social effort: higher social effort leads to a broader range of opinions being assessed correctly. With this interpretation, the uninformed opinion dynamics (5) is the continuous time version of the process in which each agent at each time updates her opinion to a convex combination of her opinion with the perceived opinions of her neighbors.

4.3 Cognitive models for human behavior

Cognitive models consider the low-level details of human behavior and typically focuses on prediction and generality [64]. For instance, how eye movement on a screen, the characteristics of human memory, the level of reasoning required to act on a particular problem or make a choice and the time required to press a button. Psychological research is the leading buddy of building cognitive models, which constraints of the human cognitive system [64]. Several attempts to combine computational and psychological theories to expand the human behavior models [65 –67]. These computational models can be represented as a computer program that combines deductive and experimental methods in one computational approach. In this way, experiments can be designed with the model, and the dynamic properties of the model can be examined by studying its actual behavior and allow us to observe and measure their behavior [68]. These models are different from statistical and mathematical models, which describe a particular phenomenon or behavior but do not have the capability to generate similar behavior.

5 Integrating human model into the control synthesis methods

Incorporating the human behavior as an element of control synthesis methodology is the most important research challenge from our point of view. Yet, there is limited work in the lecture in this direction. Possibly because the human model is not fully utilized yet.

As indicated in [35], integrating human model to be part of the system itself will have several advantages as it will allow analyzing the property of the entire system using feedback control strategies. Also, this will allow running optimization methods across the whole system including people, building, or even city to optimize for diffident parameters to maximize generalized utilities, e.g., health improvement or saving energy. Some of these immediate advantages of integrating the human model into the control synthesis methods are presented in following subsections.

5.1 Optimal allocation of humans within the loop

The notion of optimality is an essential notion in control theory. Optimal allocation of sensors and actuators.

Table 2
A generalized Fitts List presented in [69]

Advantages of machine Advantages of human

The ability of data and information visualization Owns the ability of flexibility and adaptability to the environment

Strong abilities in managing and storing data Has creativity in solving problems

Memory of large quantities of data, knowledge and rules Has a good visual perception ability

The logical reasoning ability based on the rules Has the emotion and can consider the abstract concept

Simple and repeat decision-making ability Owns strong ability of accumulating experience and learning

The ability of complex mathematical operations for a long time Has the ability of predicting and processing accidents

The ability to deal with several variable combinatorial problems Communicates complex information in a variety of ways

The ability to work for a long time Owns the ability of conceptualization and have advantages in dealing with

fuzzy and inaccurate information

Parallel processing capacity of knowledge Has good symbol or spatial reasoning ability

Fast computing speed Greatly shortens the time of making decision because of the experience and

intuition of instinct

Highly accurate calculation Good at describing and classifying observed things

Higher predictability because of the procedure set before running Predicts the development of situation according to common sense knowledge

The ability to treat benign structural problems Can well control error and uncertain events

In the long run, owns high cost-effectiveness ratio Effectively narrows the search space of the optimal decisions

The ability to work in harsh environment Uses more wide strategic sets to solve problems

Advantages of machine	Advantages of human
The ability of data and information visualization	Owns the ability of flexibility and adaptability to the environment
Strong abilities in managing and storing data	Has creativity in solving problems
Memory of large quantities of data, knowledge and rules	Has a good visual perception ability
The logical reasoning ability based on the rules	Has the emotion and can consider the abstract concept
Simple and repeat decision-making ability	Owns strong ability of accumulating experience and learning
The ability of complex mathematical operations for a long time	Has the ability of predicting and processing accidents
The ability to deal with several variable combinatorial problems	Communicates complex information in a variety of ways
The ability to work for a long time	Owns the ability of conceptualization and have advantages in dealing with
	fuzzy and inaccurate information
Parallel processing capacity of knowledge	Has good symbol or spatial reasoning ability
Fast computing speed	Greatly shortens the time of making decision because of the experience and
	intuition of instinct
Highly accurate calculation	Good at describing and classifying observed things
Higher predictability because of the procedure set before running	Predicts the development of situation according to common sense knowledge
The ability to treat benign structural problems	Can well control error and uncertain events
In the long run, owns high cost-effectiveness ratio	Effectively narrows the search space of the optimal decisions
The ability to work in harsh environment	Uses more wide strategic sets to solve problems

Human allocation and their functionality within the control loop is a necessary step in man-machine system design. In the system design stage, a decision whether a particular function will be allocated to a human, technology or mix of human and technology is to be made. This process is called function allocation process [69].

The difference between human and machine capabilities has a long history in man-machine systems and human factors. Perhaps, Fitts list [70] is the earliest to be considered that made a clear distinction between human and machine capabilities. A Generalized Fitts list presented in [69] shown in Table 2.

Optimal allocation of these functions will lead to the system to be more effective and reliable. The notion of optimality is an essential notion in control system design. However, optimal control theory focuses on the design of optimal system components only. The challenge here is to be able to formulate and quantify concepts such as error rates, costs, hazards, human values, technological feasibility, ethical issues and the desire of people to consider the optimal function allocation of humans within theloop.

5.2 Building personalized control system

Closing the loop between humans and machines will require deep understanding to both the machines and the humans to be able to build a reliable interface. These interfaces are backed up with a control system that allows the operator to interact with the controlled system. The design of such controllers is optimized for a wide range of operators with a different set of skills and capabilities to allow for wider range of users. A more reliable control system is, the control system that learns and adapt to the particular operator, i.e., a control system that has the notion of personalization.

The personalized control system is similar in concept to adaptive control is the control method that adapt to a controlled system with parameters which vary with time, however, in the personalized control system, the controller adapted to the operator skills and behavior. Equally, personalized control system "learn" the operator behavior with time and build a model for that particular operator.

Building personalized control system depend on learning the operator model and understanding the set of skills that he/she applies to control or interact with the given system. Machine learning with the computational power available these days provide powerful tools to generate and analyze rich data online in quick and efficient manner. Utilizing such tools in our control system design for human in the loop application will facilitate the concept of the personalized control system. For instance, if the operator is allowed to operate or control the system for some time, the controller uses some machine learning techniques to build a model for that operator and adapts its parameters to the identified model.

For instance, one way to understand an operator behavior in a cyber-physical system is to analyze his input and output from a control theory and data mining perspective to mine for the repeated habits. Finding unknown repeated behaviors or patterns can lead to better understanding of operator’s action in particular situations. More so, mining the repeated behaviors of skilled operators can be used as a benchmark for teaching either other human operators or autonomous systems.

In [71], we introduced a method for pattern (repeated behavior or habits) detection in trajectory data generated in the human in the loop applications. The proposed method is suitable for pattern detection in position trajectories of moving objects in 3-dimensional space or 2-dimensional plane. The method is a geometrical shape-based pattern detection, which operates in two phases. The first phase consists of trajectory segmentation, creating segment fingerprint, clustering and symbolic representations. Then, the second phase of the proposed method is pattern extraction. The second phase is inherited from the motif finding algorithms in time series data and DNA sequences.

6 Concluding remarks

The purpose of this research is to find out the collaboration between physical systems and humans in a man-machine systems. The research suggested that, modeling human behaviours, is still at an early stage of investigation. The available models are either very general or very specific for a particular scenario. As it is obvious that the human model involves highly nonlinear complex dynamics. One way of achieving reasonable model is to build a library of generic mental models in different situations. Here mental models are internal representations of the external environment that humans use to interact with the world. The parameters of these generic mental models can be configured for every individual.

The review raise several possible research questions, which can be summarized as follow:

How to integrate long decision-making models into human models?

How to build generic classification and taxonomy for human in the loop applications?

How to equip autonomous systems with cognitive and personalization capabilities in order to increase the cooperation with humans in human in the loop systems?

How to optimally place human subjects into automated systems?

How to create, quantify, and test the cooperative behavior between humans and autonomous systems?

In our opinion, in order to answer some of these questions, human modeling problem should be formulated as of building a tunable generic model (or set of models) that describe the human behavior in controlled environment from both micro-muscle and decision-making perspective taking into consideration uncertainty of the controlled system.

References

M.A.

Schumann ,

Drusinsky ,

J.B.

Michael and

Wijesekera , Modeling human-in-the-loop security analysis and decision-making processes, IEEE Transactions on Software Engineering (2014). ISSN 00985589. doi: 10.1109/TSE.2014.2302433.

Zhang ,

D.V.

Kalashnikov and

Mehrotra , Context-assisted face clustering framework with human-inthe-loop, International Journal of Multimedia Information Retrieval (2014). ISSN 2192-6611. doi: 10.1007/s13735-014-0052-1

D.L.

Atkins ,

Klapaukh ,

W.N.

Browne and

Zhang , Evolution of aesthetically pleasing images without human-in-the-loop, In 2010 IEEE World Congress on Computational Intelligence, WCCI 2010 - 2010 IEEE Congress on Evolutionary Computation, CEC 2010, 2010. ISBN 9781424469109. doi: 10.1109/CEC.2010.5586283.

Williams ,

Gharbi ,

Ulusan ,

Ergun ,

Xiaofeng ,

Zhang and

Harteveld , Toward Human in the Loop Optimization Through Game-Based Experiments, In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts - CHI PLAY Companion ’16, 2016. ISBN 9781450344586. doi: 10.1145/2968120.2987733.

Driggs-Campbell ,

Shia and

Bajcsy , Improved driver modeling for human-in-the-loop vehicular control, In IEEE International Conference on Robotics and Automation, 2015. ISBN VO -. doi: 10.1109/ICRA.2015.7139410.

Lorrie Faith Cranor. A Framework for Reasoning About the Human in the Loop. 2008.

S.B.

Bortolami ,

K.R.

Duda and

N.K.

Borer , Markov analysis of human-in-the-loop system performance, In IEEE Aerospace Conference Proceedings, 2010. ISBN 9781424438884. doi: 10.1109/AERO.2010.5446860.

Tang ,

Zhao and

Yang , Stability control for a walking-chair robot with human in the loop, International Journal of Advanced Robotic Systems (2009). ISSN 17298806. doi: 10.5772-6769.

Li ,

A.H.

Miller ,

Chopra ,

M.A.

Ranzato and

Weston , Dilaogue Learning With Human-In-The-Loop, International Conference on Learning Representations, 2017.

10.

Han ,

Hamm and

Sim , Personalized video summarization with human in the loop, In IEEE Workshop on Applications of Computer Vision, WACV, 2011. ISBN 9781424494965. doi: 10.1109/WACV.2011.5711483.

11.

Schirner ,

Erdogmus ,

Chowdhury and

Padir , The future of human-in-the-loop cyberphysical systems, Computer (2013). ISSN 00189162. doi: 10.1109/MC.2013.31.

12.

Herman and

Iglesias , Human-in-the-loop issues for demining, Proceedings of the Society for Optics and Photonics, 1999, ISSN 0277786X.

13.

Li ,

Sadigh ,

S.S.

Sastry and

S.A.

Seshia , Synthesis for human-in-the-loop control systems, In Lecture Notes in Computer Science, 2014. ISBN9783642548611.

14.

I.S.

MacKenzie , Fitts’ law as a research and design tool in human-computer interaction, Human-Computer Interaction 7(1) (1992), 91–139.

15.

Pantic ,

Pentland ,

Nijholt and

T.S.

Huang , Human computing and machine understanding of human behavior: A survey, In Artifical Intelligence for Human Computing, Springer, 2007, pp. 47–71.

16.

Alan , Human-Computer Interaction, Springer, 2009.

17.

D.G.

Robinson and

J.C.

Robinson , Breaking barriers to skill transfer, 1985.

18.

J.Z.

Self ,

R.K.

Vinayagam ,

J.T.

Fry and

North , Bridging the gap between user intention and model parameters for human-in-the-loop data analytics, In Proceedings of the Workshop on Human-In-the-Loop Data Analytics - HILDA ’16, 2016. ISBN 9781450342070. doi: 10.1145/2939502.2939505.

19.

M.A.

Zamani and

Oztop , Simultaneous human-robot adaptation for effective skill transfer, In Proceedings of the 17th International Conference on Advanced Robotics, ICAR, 2015. ISBN 9781467375092. doi: 10.1109/ICAR.2015.7251437

20.

A.E.

Leeper ,

Hsiao ,

Ciocarlie ,

Takayama and

Gossow , Strategies for human-in-theloop robotic grasping, In Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction - HRI ’12, 2012. ISBN 9781450310635. doi: 10.1145/2157689.2157691

21.

A.C.

Koenig and

Riener , The human in the loop, In Neurorehabilitation Technology, Second Edition, 2016, ISBN9783319286037.

22.

Hirota , Casual communication between robots and humans using fuzzy interruption technology on internet, In 5th International Symposium on Computational Intelligence and Intelligent Informatics (ISCIII), 2011. ISBN 978-1-4577-1860-1. doi: 10.1109/ISCIII.2011.6069743

23.

Ogata and

Sugano , Communication between behaviorbased robots with emotion model and humans...., Man, and Cybernetics, 1998. ISSN 1062-922X. doi: 10.1109/ICSMC.1998.727840

24.

Ogata ,

Matsuyama ,

Komiya ,

Ida ,

Noda and

Sugano , Development of emotional communication robot: WAMOEBA-2R-experimental evaluation of the emotional communication between robots and humans, In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2000. ISBN VO - 1. doi: 10.1109/IROS.2000.894601

25.

Nikolaidis and

Weinberg , Playing with the masters: A model for improvisatory musical interaction between robots and humans, In Proceedings - IEEE International Workshop on Robot and Human Interactive Communication, 2010. ISBN 9781424479917. doi: 10.1109/ROMAN.2010.5598621

26.

H.A.

Samani and

A.D.

Cheok , Probability of love between robots and humans, In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, 2010. ISBN 9781424466757. doi: 10.1109/IROS.2010.5650886

27.

Nomura and

Kawakami , Relationships between robots’ self-disclosures and humans’ anxiety toward robots, In Proceedings - 2011 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Workshops, WI-IAT 2011, 2011. ISBN 9780769545134. doi: 10.1109/WI-IAT.2011.17

28.

Mistry and

Schaal , Representation and Control of the Task Space in Humans and Humanoid Robots, In Representation and Control of the Task Space in Humans and Humoid Robots, 2015, ISBN9781420093667.

29.

J.V.

Nickerson , Robots and humans reconvening. 2004. ISBN 0780385667. doi: 10.1109/ICSMC.2004.1400757.

30.

Dragone ,

B.R.

Duffy and

G.M.P.

O’Hare , Social interaction between robots, avatars & humans, In Proceedings - IEEE InternationalWorkshop on Robot and Human Interactive Communication, 2005. ISBN 0780392752. doi: 10.1109/ROMAN.2005.1513751

31.

D.A.

Abbink ,

Carlson and

Mulder , Special Session Call for Papers SMC 2012 Special Session on Shared Control, In Special Session Call for Papers on SMC Special Session on Shared Control, 2012.

32.

D.T.

McRuer and

E.S.

Krendel , The human operator as a servo system element, Journal of the Franklin Institute 267(5) (1959), 381–403.

33.

D.T.

McRuer and

E.S.

Krendel , Mathematical models of human pilot behavior. Technical report, DTIC Document, 1974.

34.

C.W.D.

Silva ,

Khoshnoud ,

Li and

Saman , Halgamuge, Mechatronics: Fundamentals and Applications, CRC Press, 2015.

35.

Munir ,

J.A.

Stankovic ,

C.-J.M.

Liang and

Lin , Cyber physical system challenges for human-inthe-loop control, In Presented as part of the 8th International Workshop on Feedback Computing, Berkeley, CA, 2013. USENIX.

36.

D.S.

Nunes ,

Zhang and

J.S.

Silva , A survey on human-in-the-loop applications towards an internet of all, IEEE Communications Surveys & Tutorials 17(2) (2015), 944–965.

37.

C.C.

Macadam , Understanding and modeling the human driver, Vehicle System Dynamics 40(1-3) (2003), 101–134.

38.

D.A.

Wardle , The time delay in human vision, The Physics Teacher 36(7) (1998), 442–444.

39.

Diamond , Executive functions, Annual Review of Psychology 64 (2013), 135.

40.

Nadin , Not everything we know we learned, In Anticipatory Behavior in Adaptive Learning Systems, Springer, 2003, pp. 23–43.

41.

C.A.

Skarda and

W.J.

Freeman , Chaos and the new science of the brain, Concepts in Neuroscience 1(2) (1990), 275–285.

42.

C.N.

Viswanathan ,

R.W.

Longman and

P.W.

Likins , A degree of controllability definition-fundamental concepts and application to modal systems, Journal of Guidance, Control, and Dynamics 7(2) (1984), 222–230.

43.

D.T.

McRuer and

H.R.

Jex , A review of quasi-linear pilot models, IEEE Transactions on Human Factors in Electronics (3) (1967), 231–249.

44.

D.L.

Kleinman ,

Baron and

W.H.

Levison , An optimal control model of human response part i: Theory and validation, Automatica 6(3) (1970), 357–369. ISSN 0005-1098.

45.

Baron ,

D.L.

Kleinman and

W.H.

Levison , An optimal control model of human response part ii: prediction of human performance in a complex task, Automatica 6(3) (1970), 371–383.

46.

B.A.

Francis and

W.M.

Wonham , The internal model principle of control theory, Automatica 12(5) (1976), 457–465.

47.

Zhang and

J.B.

Hoagg , Subsystem identification of multivariable feedback and feedforward systems, Automatica 72 (2016), 131–137.

48.

Zhang and

J.B.

Hoagg , Frequency-domain subsystem identification with application to modeling human control behavior, Systems & Control Letters 87 (2016), 36–46.

49.

Zhang ,

Wang ,

J.B.

Hoagg and

T.M.

Seigler , The roles of feedback and feedforward as humans learn to control unknown dynamic systems, IEEE Transactions on Cybernetics (2017).

50.

S.A.S.

Mousavi ,

Zhang ,

T.M.

Seigler and

J.B.

Hoagg , Characteristics that make dynamic systems difficult for a human to control, In American Control Conference (ACC), IEEE, 2016, pp. 4391–4396.

51.

Gawthrop ,

Gollee and

Loram , Intermittent control in man and machine, Event-Based Control and Signal Processing, 2015.

52.

Gawthrop ,

K.-Y.

Lee ,

Halaki and

O?Dwyer , Human stick balancing: An intermittent control explanation, Biological Cybernetics 107(6) (2013), 637–652.

53.

Gawthrop ,

Loram ,

Gollee and

Lakie , Intermittent control models of human standing: similarities and differences, Biological Cybernetics 108(2) (2014), 159–168.

54.

Gawthrop ,

Loram ,

Lakie and

Gollee , Intermittent control: A computational theory of human control, Biological Cybernetics 104(1) (2011), 31–51.

55.

Gollee ,

Mamma ,

I.D.

Loram and

P.J.

Gawthrop , Frequency-domain identification of the human controller, Biological Cybernetics, 2012, pp. 1–14.

56.

Loram ,

Lakie ,

Gollee and

Gawthrop , Human control of an inverted pendulum: Is intermittent control effective? In Proceedings of The Physiological Society. The Physiological Society, 2010.

57.

I.D.

Loram ,

Gollee ,

Lakie and

P.J.

Gawthrop , Human control of an inverted pendulum: is continuous control necessary? is intermittent control effective? is intermittent control physiological? The Journal of Physiology 589(2) (2011), 307–324.

58.

I.D.

Loram ,

C.V.D.

Kamp ,

Gollee and

P.J.

Gawthrop , Identification of intermittent control in man and machine, Journal of The Royal Society Interface (2012), rsif20120142.

59.

Mamma ,

Gollee ,

P.J.

Gawthrop and

I.D.

Loram , Intermittent control explains human motor remnant without additive noise. In Control & Automation (MED), 2011 19th Mediterranean Conference on, IEEE, 2011, pp. 558–563.

60.

Pentland and

Liu , Modeling and prediction of human behavior, Neural Computation 11(1) (1999), 229–242.

61.

Conradt and

List , Group decisions in humans and animals: A survey, Philosophical Transactions of the Royal Society of London B: Biological Sciences 364(1518) (2009), 719–742.

62.

Ratcliff and

McKoon , The diffusion decision model: theory and data for two-choice decision tasks, Neural Computation 20(4) (2008), 873–922.

63.

Franci ,

Srivastava and

N.E.

Leonard , A realization theory for bio-inspired collective decision-making. arXiv preprint arXiv:1503.08526, 2015.

64.

R.E.

Wray and

R.S.

Chong , Comparing cognitive models and human behavior models: Two computational tools for expressing human behavior, Journal of Aerospace Computing, Information, and Communication 4(5) (2007), 836–852.

65.

Frawley , Vygotsky and Cognitive Science: Language and the Unification of the Social and Computational Mind, ERIC, 1997.

66.

Fum ,

F.D.

Missier and

Stocco , The cognitive modeling of human behavior: Why a model is (sometimes) better than 10,000 words, Cognitive Systems Research 8(3) (2007), 135–142.

67.

Bechtel and

Abrahamsen , Dynamic mechanistic explanation: Computational modeling of circadian rhythms as an exemplar for cognitive science, Studies in History and Philosophy of Science Part A 41(3) (2010), 321–333.

68.

Goldberg , The Executive Brain: Frontal Lobes and the Civilized Mind, Oxford University Press, USA, 2002.

69.

Zhang ,

Zhili and

Zhang , Man-machine function allocation based on uncertain linguistic multiple attribute decision making, Chinese Journal of Aeronautics 24(6) (2011), 816–822.

70.

J.C.F.

de Winter and

Dodou , Why the fitts list has persisted throughout the history of function allocation, Cognition, Technology & Work 16(1) (2014), 1–11. ISSN 1435-5566. doi: 10.1007/s10111-011-0188-1. URL https://dx-doi-org.web.bisu.edu.cn/10.1007/s10111-011-0188-1.

71.

M.A.

Mabrok and

Abdel-Aty , Pattern detection for time series trajectories in human in the loop applications, Intelligent & Fuzzy Systems (JIFS) (2019).

Human Models
Control theory-based models	Bio-inspired models	Cognitive models
Quasi-linear model	Drift Diffusion Model	Cognitive models consider the low-level details of human behavior
Optimal control model	Coupled drift diffusion model	and typically focuses on prediction and generality.
Internal model hypothesis	Collective decision making model
Preview control model
Intermittent control
Hidden Markov models