Telepresence Mechatronic Robot (TEBoT): Towards the design and control of socially interactive bio-inspired system

2.1 Interactive robots

In last decades, a number of commercial and non-commercial interactive systems/robots have been developed. These robots can be broadly categorized into i) Non-Anthropomorphic Robots (N-APR) and ii) Anthropomorphic Robots (APR). The characteristics and appearance of N-APRs are not similar to real human characteristics and appearance, but they still possess some socially interactive skills. The most common example of N-APRs are mobile robotics telepresence (MRP) systems. The basic construction of almost all MRPs consist of a mobile robotic base, an LCD screen, a camera, a microphone and some non-verbal gestures, like hand gestures, etc. A comprehensive review on MRPs can be found in [9]. The other example of N-APR robots includes Mebot [10], ESP [11], Jibo [12], Keepon [13], etc. On the other hand, the characteristics and appearance of APRs are similar to real human characteristics and appearance; for example, HRP-4C [14], Actroid DER3[15], Geminoid [16], Furhat [17], etc. are the APRs which look very similar to real human and it can be hard to decide whether it is a robot or a human. There also exists a set of APRs called android which have characteristics similar to humans but their faces are not similar to humans such as ASIMO [8], Telenoid [18], iCub [19], etc. The detail review on socially interactive N-APRs and APRs can be found in [20].

When it comes to video teleconferencing application, there are limitations in almost all previously developed N-APRs and APRs. The N-APRs do not present an accurate nonverbal gestures and APRs are complex, expensive and just represent one person. Furthermore, these systems are explicitly controlled by the mouse, keyboard and other hand-held devices. By considering these limitations, we have designed and built a novel socially interactive system - named - TEBoT, which can present an accurate head gesture along with audio-video communication. The research in psychology shows that among all non-verbal modalities during human conversation, facial expressions and head movements are the most important for the flow of information [21]. TEBoT has the capabilities and appearance of APR and simplicity of N-APR. TEBoT is portable, cheap, easily controllable and present exact non-verbal head gestures. Furthermore, a novel single camera-based human-in-the-loop control strategy is devised for the TEBoT control.

2.2 Human motion capturing approaches

To actuate a socially interactive bio-inspired robot, one of the best possible way is to input it with a real human motion [22]. Approaches to capture these human-body motions can be categorized into two sub-categories:

2.3 Contribution

The key contribution is in the design and control of a socially interactive head/neck robot, where real head motion dynamics are considered in the design of mechanical assembly. For motion control, we have proposed a novel vision based geometric head pose estimation technique to capture the human head movements and actuate the robotic platform in real time.

3.1 Procedure

For this work, we have recruited seven participants ranging in the age from 22 to 45 years. The participants are asked to conduct recordings of the angular motion [ψ (t), θ (t), φ (t)] and angular velocities [ψ ̇ ( t ), θ ̇ ( t ), φ ̇ ( t )] by using a head-mounted IMU. The participants are asked to perform five tasks with their head movements, where each task are performed more than ten times. These five tasks represent the actions of saying Yes, No, May-be and also two motion patterns for performing a Circle and triangle.

Head Nod (Yes): Move head up and down, exciting mainly the pitch coordinate.

3.2 Frequency analysis

The frequency analysis measures the operating frequency-band of the human-head movements for all the five tasks. The frequency spectrum for each task is estimated by using the measurements of angular velocities. This can be done by applying fast fourier transform (FFT) and/or power spectral density (PSD). The results of the frequency analysis for each task is presented in Table 1. The table shows the minimum, maximum and the mean frequencies of human head movements. Two out of five trajectories can be visualized in Fig. 2, where we show the angular positions [ψ (t), θ (t), φ (t)], angular velocities [ψ ̇(t), θ ̇(t), φ ̇(t)] and the frequency spectrum for head nod and head circle movements. These frequency-analysis results are used in the design of a robot controller.

3.3 Velocity analysis and threshold calculation

The collected raw data is further post-processed to calculate other important attributes such as, kinematic constraints. These quantities give an information about a range of the head movement and constraints in velocities. These results are shown in Table 2. This information is used to design our robot architecture. Additionally, the velocity constraints indicate the required specifications for the actuators.

Table 2 shows that the head nod, head shake and head roll movements also involve components of yaw, pitch and roll angles. Ideally, these movements should present only pitch movement for head nod, yaw movement for head shake and roll movement for head roll. These extraneous rotational components define our software threshold, which removes an undesired movement around zero velocities both in position estimation and control.

4.1 Biological system: Human neck

Human neck has a complex anatomical structure formed by seven cervical vertebrae (see Fig. 3a) and around twenty main muscles (see Fig. 3b). The main role of the neck is to support human head in balance while performing different head movements. Human neck muscles are responsible for performing head movements and the combination of cervical vertebrae and muscles are used to hold the head in upright position. There are three essential head motions, i.e. yaw, pitch and roll (see Fig. 4) and all other movements are combinations and varying percentages of these three head movements. With reference to cervical vertebrae, the neck muscles are divided into left and right side of the neck muscles; to present a parallel configuration acting at both sides of the shoulders. Working in pair, the left and right sides of the neck muscles control the pitch movement. Working individually, these muscles control the yaw and roll movements (see [38] for more details).

4.2 Biological inspired design

To mimic the movements of a human neck, a system should satisfy not only the mobility properties of a human neck, but also static and dynamic characteristics. The CAD model of our mechatronic system is shown in Fig. 5(a) and a working prototype in Fig. 5(c). Our design uses two active limbs (label 7 in Fig. 5(a)) and one passive limb (label 8 in Fig. 5(a)). The passive limb is a central rod connected between a base (label 3 in Fig. 5(a)) and a mounting assembly (label 4 in Fig. 5(a)) via a universal joint (label 6 in Fig. 5(a)). Whereas, the active limbs are the connecting rods connected between the motors (label 2 in Fig. 5(a)) and the mounting assembly. The function of the passive limb is similar to the function of a cervical vertebrae in human neck and the function of the active limbs resembles the function of human neck muscles. The passive limb in combination with two active limbs are used to support the mounting assembly of a tablet PC (label 1 in Fig. 5(a)). The active limbs control the pitch and roll movements; the motion of the yaw is performed by a motor inside the base (see Fig. 5(b) (label D)). Similar to the human neck anatomy, our system is a 3DOF parallel kinematic system actuated by three servo motors assembled on the base.

The results of head motion analysis are considered in the mechanical design of our robot. From Table II, the maximum of average peak angular values for yaw (ψ), pitch (θ) and roll (φ) movements are 1.43 rad, 0.642 rad and 0.65 rad, respectively. Considering these results our mechanical design undergoes ±1.45 rad for yaw movement, ±0.76 rad for pitch and roll movements.

4.3 Motor selection

In upright equilibrium, the weight of the mounting assembly (label 4 in Fig. 5(a)) and tablet PC (label 1 in Fig. 5(a)) is supported by the passive limb (label 8 in Fig. 5(a)). The whole upper assembly is attached to the base through ball bearings (see Fig. 5(b)). Two parameters are considered important in the selection of motors. The torque in kg-cm (or N-m) and the speed in revolution per minute RPM (or rad/sec).

4.4 Controller selection

The servo motors function using PWM signals. To generate these PWM signals, a controller is required that can handle at least three PWM signals in parallel. Based on this requirement we have selected Arduino Uno, which operates at 5 V with a clock frequency of 16 MHz. It has 14 digital input/output pins, of which 6 can be used as a PWM output.

5.1 From CAD to TEBoT simulation

In order to use a CAD model for MBD, the properties of the physical model have to be transformed into its corresponding set of differential equations describing the system dynamics and the equations of motion. Using the state-of-the-art control engineering design software this can be done by using certain add-ons in e.g. Solid Works. One example of this procedure is by the use of the Simmechanics [40], a product of Mathworks, that allows users to convert a CAD design model into a set of differential equations that represent the dynamics of a robotic motion. These set of differential equations can be simulated and further used for model based control design by applying other tools within the mathworks products.

5.2 Robot motion control

The task is to construct a controller for our TEBoT that meets the desired behavior of a closed loop control system. The dynamics of yaw movement are simple and linear so the yaw controller is simple proportional controller. On the other hand, the pitch and roll dynamics of TEBoT are nonlinear and are given by:

x ̇ ( t ) = f ( x ( t ) , u ( t ) ) y ( t ) = h ( x ( t ) , u ( t ) ) ,(1)

To simplify the controller design for pitch and roll movement, we apply a linearization of the model using Taylor series [41].

x ̇ ( t ) = A x ( t ) + B u ( t ) y ( t ) = C x ( t ) + D u ( t ) .(2) The basic goal of the controller is to specify the torques for two servo motors for the desired pitch and roll angles. The multiple-input multiple-output (MIMO) closed loop diagram of a system is shown in Fig. 6. The multiple inputs are the desired pitch and roll angles denoted by θ _d and φ _d , respectively. The multiple outputs are the required torques for the left and right servo denoted by τ _l and τ _r , respectively. The G is the plant, which is imported from the solidworks. There are two controllers used in our system; one main controller denoted by C _m and the low-level controller denoted by C _s . The low-level controller (C _s ) is the PID controller for each servo motor to follow the given trajectories as given by:

τ = - K p ( q - q ref ) - K d ( q ̇ - q ̇ ref ) - K I ∫ ( q - q ref ) ,(3)

Where K _p denotes the values of proportional gains, K _d the values of derivative gains, and K _I the values of integral gains. The gains of the multiple PID controllers are tuned automatically in simulink by using the tune tab of the PID controller.

The main controller C _m maps the desired trajectories of the TEBoT to the corresponding servo trajectories using inverse kinematics. It starts with the desired rotational angles (θ _d and φ _d ) of the TEBoT and calculates the required servo rotations (q _l and q _r ) to achieve this. The gains of the C _m for the given crossover frequencies between 1 and 11 rad/sec (taken from Table 1) are tuned with the help of optimization methods described in [42] using MATLAB with minimal loop interaction and adequate MIMO stability margins.

6.1 Face detection

The input to our geometric head pose estimation algorithm is the video frames containing, i) the face of a person and ii) the undesired background. We first detect a human face in a cluttered video stream. Towards this end, number of algorithms have been proposed [43, 44]. However, we have employed Haar-feature-based cascade classifiers proposed by Paul Viola and Michael Jones for human face detection [45]. This algorithm is a two-step process which consists of a training step and a testing step. In training step, the algorithm learns to differentiate between a face image and background. In testing step, the algorithm uses the training information to detect a face in a video stream (see Fig. 7(a)).

6.2 Facial features detection

The second step is to find facial feature points in a detected face. Facial features include eyes, nose, lips, mouth, eye-brows and facial boundary. Numerous methods exist for detecting human facial features as presented in [46], In this work, we have employed the well-known Constrained Local Model (CLM) approach [47]. The CLM is also a two-step process which contains a training session and a testing session. In training session, the shape and texture models are built from a training set of large number of labeled face images. The shape model includes a face and facial feature points and the texture model includes the intensity values of the face and facial features. In testing session, our algorithm iteratively estimates the facial feature location of an unseen image by using the combined information of the shape and texture models. The result of CLM is shown in Fig. 7(b).

6.3 Estimating human neck’s reference frame

Human head has three independent movements (yaw, pitch and roll) around Y, X and Z axes as shown in Fig. 8. The video is formed by a sequence of consecutive images, and images from standard camera contains only 2D information, i.e. X and Y coordinates. This information can be used to compute a roll angle of a human head. However, it requires 3D information for computing yaw and pitch angles. Hence, this step consists of estimation of the Z coordinates to define a reference frame of a human neck. This reference frame is assumed to be at C2 of the spinal column as shown in Fig. 9, and it is found by using the location of the eyes and the facial boundaries [48]. These features allow us to compute the width w, height h, and the distance between the center of the eyes d from the detected face, see Fig. 9. Given these quantities, the neck reference frame is given by: O n = ( X n , Y n , Z n ) = ( w 2 , h 2 , - λ d ) .(4)

This 3D reference point is used to estimate the yaw and pitch angles of the human head.

In the case of the roll angle φ, it is sufficient to know the positions of the center of each eye, i.e. E _l  = (X _l , Y _l ) for the left eye and E _r  = (X _r , Y _r ) for the right eye. Therefore, φ is computed by right angle triangle shown in Fig. 10(b): φ = tan - 1 ( Y r - Y l X r - X l ) .(5)

For estimating the yaw ψ and pitch θ angles, we define the vector v ∈ ℝ 3 from O _n to O _e , where O _e  = (X _e , Y _e , Z _e ) is middle point of the center of the eyes, see Fig. 10(a).

The projections of v onto the planes XZ and YZ are given by

a = Proj XZ ( v ) , b = Proj YZ ( v ) ,(6)

The angles between the projections and the Z-axes give us the yaw ψ and pitch θ angles of the human head as computed below: ψ = tan - 1 ( X e - X n Z e - Z n ) ,(7)θ = tan - 1 ( Y e - Y n Z e - Z n ) ,(8)