Uncalibrated Neuro-Visual Servoing Control for Multiple Robot Arms

Abstract

Diverse image-based tracking schemes for a robot moving in free motion have been proposed and experimentally validated. However, few visual servoing schemes have addressed the tracking of the desired trajectory and the contact forces for multiple robot arms. The main difficulty stems from the fact that camera information cannot be used to drive force trajectories. Recognizing this fact, a unique error manifold that includes position-velocity and force errors in orthogonal complements is proposed. A synergistic scheme that fuses camera, encoder and force sensor signals into a unique error manifold allows proposing a control system which guarantees exponential tracking errors under parametric uncertainty. Additionally a small neural network driven by a second order sliding mode surface is derived to compensate robot dynamics. Residual errors that arise because of the finite size of the neural network are compensated via an orthogonalized second order sliding mode. The performance of the proposed scheme, in two significant applications of the multiple robot arms, is validated through numerical simulations.

Keywords

Multiple Robot Arms Constrained Visual Servoing Neural Networks Second Order Sliding Mode Control Jacobian Uncertainty

1. Introduction

To increase the executing tasks of the robot manipulators in several practical applications, the use of multiple robot arms working together in the same workspace and cooperatively has been proposed. Working abilities, loading capabilities and object manipulation are some advantages of multiple robot arms with respect to a single robot. In order to resolve the complex dynamic interactions in these robotic systems, many studies have focused attention on designing robust and adaptive control schemes to minimize the hardware and software.

Although several approaches for cooperative schemes [9, 13, 15], fast control [5, 8, 14, 26], load distribution and path planning [19] have been reported in the literature, a difficultly in synthesizing these approaches in practice is the inexact knowledge of almost all system parameters.

Additionally, to avoid the computational load and the ill-posed nature on the inverse kinematics in the joint robot control, controllers in operational coordinates or the use of visual information called visual servoing can be addressed as an alternative solution.

Despite the fact that several pieces of research have addressed vision/force control [1, 4, 7, 27] for a single robot, few of them show robustness to uncertainties in robot parameters or camera parameters. In [27] a controller based on the multisensor fusion is proposed where the signals from sensors are used to correct uncalibrated parameters. But the force/position controller and the planner require an exact knowledge of the robot manipulator. An adaptive algorithm is used in [21, 22] to estimate the visual Jacobian matrix in free motion. This approach is based on the assumption that the visual Jacobian matrix can be decomposed as a product of two matrices. The first matrix requires kinematic parameters of the robot manipulator, while the second matrix is assumed unknown. That is, the rotation matrix between the robot base frame with respect to the vision frame is unknown. On the other hand, a neurovisual servoing control of a planar robot manipulator is presented in [23] where it is assumed that the link lengths of the robot manipulator are not exactly known. The neural network is used to avoid the drift in the parameter estimated and some possible overshoot in the estimated of the gravitational vector. That is, it is not necessary to know the gravity vector and boundedness of the Jacobian matrix against a perturbation. At the same time, there is a dearth of research that extends or combines these visual approaches to multiple robot arms under parametric uncertainty.

This paper aims to design a theoretical neuro-visual servoing controller that guarantees analytically image-based tracking of constrained multiple robot arms subject to parametric uncertainty, that is, i-th robot and the camera parameters are unknown. The neuro-visual servoing control is based on a second order sliding surface for visual position and contact force trajectories respectively. To approximate the i-th robot dynamics and the parameter variations, a small neural network driven by the sliding mode is used. The residual error that arises from this bounded approximation is eliminated with the sliding mode into a unique error manifold. In contrast with other approaches, in this paper exponential convergence of image-based position error and contact forces are guaranteed with smooth control and chattering free. Finally, the parametric uncertainty on the i-th robot manipulator and camera are parameterized in terms of the regressor time as a parameter vector. Then, to get parametric uncertainty, this vector is multiplied by a factor with respect to the nominal value.

The viability of this approach is demonstrated through two typical applications of the multiple robot manipulators in a horizontal plane. In the first simulation, a pair of robot manipulators grasps an object firmly applying the desired force through the contact point, while the end-effector of the i-th robot manipulator follows the desired visual trajectory. In the second simulation a more complex task is presented, one that involves the changing dynamics and an impact regime to manipulate an object by two robotic fingers. Strictly speaking, transition tasks involve free motion, impact and constrained motion regimes. Impulsive and unilateral constraints arise in the impact regime, which makes it very difficult to design a control system. The simple approach would be to avoid impact and to commute consistently between a free and constrained controller. That is, given that algebraic constraints arise in the constrained regime, a virtual constraint will be proposed to model a DAE1 system for all regimes. Thus, the control scheme proposed is based on the avoiding impact regime, through zero transition velocity from free to constrained motion, which means that impulsive dynamics does not appear. Differently then [18, 24, 25], this approach is one of fast constrained object manoeuvring for non-redundant multiple robot arms. Furthermore, a decentralized cooperative tracking is obtained without using the model of the robot manipulator, nor exact knowledge of the inverse Jacobian.

2. Constrained dynamics of a multiple arms robot

Dynamical Equations. Consider a robotic system composed of l rigid robot manipulators that is holding an object, the differential algebraic dynamic model of the i-th manipulator, i ∊ {1,‥‥,l}, constrained by the object's geometry can be expressed by

Η_{i} (q_{i}) {\ddot{q}}_{i} + C_{i} (q_{i,} {\dot{q}}_{i}) {\dot{q}}_{i} + g_{i} (q_{i}) = u_{i} + J_{φ_{i +}}^{T} (q_{i}) λ_{i}

(1)

φ_{i} (q_{i}) = 0

(2)

where $q_{i}, {\dot{q}}_{i}, {\ddot{q}}_{i} ∊ ℜ^{n_{i}}$ are angular positions, velocities and accelerations of the revolute joints of the i-th manipulator, which has n_i degrees of freedom, $H_{i} (q_{i}) ∊ ℜ^{n_{i} \times n_{i}}$ is a positive-definite inertial matrix, $C_{i} (q_{i}, {\dot{q}}_{i}) ∊ ℜ^{n_{i} \times n_{i}}$ is a Coriolis matrix which models centripetal and centrifugal forces, $g_{i} (q_{i}) ∊ ℜ^{n_{i}}$ is a vector of gravitational torques, and $u_{i} ∊ ℜ^{n_{i}}$ is a vector of input-controlled joint torques. If the end-effector of the i-th robot manipulator is in touch with the frictionless and infinitely rigid surface of the object, which is modelled by the geometric function $φ_{i} (q_{i}) : ℜ^{n_{i}} \to ℜ^{m_{i}}$ with m_i > 0 representing the number of independent contact points of the i-th manipulator, then

\begin{array}{l} J_{φ}_{_{i}} (q_{i}) = \nabla φ_{i} \\ J_{φ}_{_{i +}} (q_{i}) = J_{φ_{i}}^{T} (q_{i}) {(J_{φ}_{_{i}} (q_{i}) J_{φ_{i}}^{T} (q_{i}))}^{- 1} \end{array}

where $J_{φ}_{_{i}} (q_{i}) ∊ ℜ^{m_{i}}^{\times n_{i}}$ is the constraint Jacobian, $J_{φ}_{i +} (q_{i}) ∊ ℜ^{m_{i}}^{\times n_{i}}$ is an orthonormal matrix, and $λ_{i} ∊ ℜ^{m_{i}}$ a Lagrange multiplier that represents the magnitude of the contact force.

Manipulation Space. When the robotic system manipulates an object, the position of the i-th end effector is geometrically constrained on a p_i = n_i – m_i dimensional manifold O_ri, where

O_{r}_{i} = {(q_{i}, {\dot{q}}_{i}) : φ_{i} (q_{i}) = 0, {\dot{φ}}_{i} (q_{i}) = 0}

From the standpoint of path planning, equation (2) stands for the trajectory to be followed which runs on the surface of the object. Thus, the constraint imposed by equation (2) should be satisfied in all time which means that the manipulator does not invade the object's space.

Even though the formalism presented through the paper can be extended for a i-th rigid manipulator, this work will focus in a robotic system with two manipulators (i = 2), each of them having a single contact point (m_i = 1, for i ∊ {1,2}). This is a planar robotic system, thus the manipulators move in a horizontal plane.

Open Loop Error Equation. According to the properties of equation (1), which can be parametrized linearly [10], in terms of a nominal reference ${({\dot{q}}_{r i}, {\ddot{q}}_{r i})}^{T} ∊ ℜ^{2 n_{i}}$ as follows

H_{i} (q_{i}) {\ddot{q}}_{r i} + C_{i} (q_{i}, {\dot{q}}_{i}) {\dot{q}}_{r i} + g_{i} (q_{i}) = Y_{r i} m_{i}

(3)

where the regressor

Y_{r i} = Y_{r i} (q_{i}, {\dot{q}}_{i,} {\dot{q}}_{r i,} {\ddot{q}}_{r i}) ∊ ℜ^{n_{i} \times n_{i}}

is composed of known non-linear functions, and m_i ∊ ℜ^pi represents unknown but constant parameters. If we add and subtract equation (3) to equation (1), then the following open loop error equation arises

H_{i} (q_{i}) {\dot{s}}_{i} + C_{i} (q_{i}, {\dot{q}}_{i}) s_{i} = u_{i} + ​ J_{φ_{i +}}^{T} (q_{i}) λ_{i} - Y_{r i} m_{i}

(4)

where the joint error manifold is defined by

s_{i} = {\dot{q}}_{i} - {\dot{q}}_{r i}

(5)

Constrained Positions and Velocities. Once the manipulators jointly grasp an object and hold it securely, they keep pulling and pushing. Thus, in order to work in a cooperative manner and to rigidly manipulate an object, we need to model these effects. This is achieved by using constrained position and constrained velocity variables [14]. Considering the total derivative of the holonomic constraints imposed by equation (2), the constrained position and constrained velocity are defined as

p_{i} = \sum_{j = 1}^{l} \int_{t o}^{t f} J_{φ_{j}} (q_{j}) {\dot{q}}_{j} d t, {\dot{p}}_{i} = \sum_{j = 1}^{l} J_{φ_{j}} (q_{j}) {\dot{q}}_{j}

(6)

respectively. Consequently, to propose an appropriate error coordinate system, equation (6) should be included in the error manifold through ${\dot{q}}_{r i} .$ .

In order to design an image-based decentralized controller without resorting to $H_{i} (q_{i}), C_{i} (q_{i}, {\dot{q}}_{i})$ and g_i(q_i), we need to define a joint error manifold that uses the visual position of the CCD camera sensor. Firstly, let us establish a mapping between joint velocities and end-effector visual velocities.

Visual Kinematics. Let the direct and differential kinematics for the i-th robot manipulator be defined by

\begin{array}{l} x_{b}_{_{i}} = f_{i} (q_{i}) \\ {\dot{x}}_{b_{i}} = J_{i} (q_{i}) {\dot{q}}_{i} \end{array}

(7)

where f_i:ℜⁿ → ℜⁿ is the generally non-linear forward kinematics, $x_{b_{i}} ∊ ℜ^{n}$ represents the end-effector position of the i-th robot manipulator in the operational workspace, and J_i(q_i) its Jacobian matrix. Given that a planar robot is used in this work, the operational space has two dimensions and n = 2. In this paper we use a widely accepted fixed (static) camera configuration [6], which has a thin lens without aberration, whose mathematical description consists of a composition of four transformations that take into account: joint to Cartesian transformation, Cartesian to camera transformation, camera to CCD transformation, and CCD to image transformation. Therefore, the 2D visual position $x_{s_{i}} ∊ ℜ^{2}$ of the CCD camera sensor, which maps the i-th end-effector position into the image space (screen), is given by

x_{s_{i}} = α R x_{b i} + b

(8)

with

\begin{array}{l} α = α_{0} \frac{λ_{f}}{λ_{f} - z}, R = [\begin{matrix} \cos (θ) & - \sin (θ) \\ \sin (θ) & \cos (θ) \end{matrix}] \\ a n d b = α_{0} \frac{λ_{f}}{λ_{f} - z} [\begin{array}{l} ^{v} O_{b 1} \\ ^{v} O_{b 2} \end{array}] + [\begin{array}{l} u_{c} \\ v_{c} \end{array}] \end{array}

(9)

where λ_f is the focal distance along optical axis, z stands for the depth of field, ${[^{v} O_{b 1},^{v} O_{b 2}]}^{T}$ defines the distance between optical axis and the i-th robot base, and ${[u_{c}, v_{c}]}^{T}$ defines the translation of the camera centre to the image centre. Taking the time derivative of equation (8), the differential camera model is

{\dot{x}}_{s_{i}} = α R {\dot{x}}_{b i}

(10)

Notice that the constant transformation αR maps the static robot Cartesian velocities into visual Cartesian velocities, and that the visual flow ${\dot{x}}_{s i}$ does not contain any independent input. Now, substituting equation (7) into equation (10), the relation between image velocities with joint velocities is defined as

{\dot{x}}_{s_{i}} = α R J_{i} (q_{i}) {\dot{q}}_{i,}

(11)

where ${\dot{x}}_{s i} ∊ ℜ^{2}$ determines the visual velocity of the i-th robot end-effector, and αRJ_i(q_i) maps the i-th joint velocities to visual velocities.

Taking into account the equation (11), the inverse differential visual kinematics expression for the i-th robot manipulator is obtained

{\dot{q}}_{i} = J_{R i n v_{i}} {\dot{x}}_{s_{i}}

(12)

where $J_{R i n v_{i}} = J {(q_{i})}^{- 1} R^{- 1} α^{- 1} \equiv J {(q_{i})}^{- 1} R_{α}^{- 1}$ , with $J_{R i n v_{i}} ∊ ℜ^{n_{i} \times n_{i}}$ is a non-degenerate matrix. It is important to notice that the entries of $J_{R i n v_{i}}$ are a function of the robot manipulators and the camera parameters. That is, equation (12) says that the joint velocities of the i-th robot manipulator appear as a function of the visual velocities.

Global Joint Space Decomposition. Consider a partition of the joint space coordinate q_i as follows

q_{i} = [\begin{array}{l} q_{i 1} \\ q_{i 2} \end{array}]

(13)

where $q_{i 1} ∊ ℜ^{m_{i}}$ and $q_{i 2} ∊ ℜ^{n_{i} - m_{i}}$ . Due to the kinematic constraint $φ_{i} (q_{i}) ∊ ℜ^{m_{i}}$ there are m_i dependent coordinates which define q_i1. According to the implicit function theorem [16], a relationship between q_i1 and q_i2 always exists, i.e, q_i1 = Ω(q_i2), at least locally. Taking the derivative of ϕ_i(q_i) = 0, with its corresponding partition given by equation (13), we obtain

J_{φ_{i}} (q_{i}) {\dot{q}}_{i} = [J_{φ_{i 1}} (q_{i}) {\dot{q}}_{i 1} + J_{φ_{i 2}} (q_{i}) {\dot{q}}_{i 2}] = 0

(14)

with $J_{φ_{i 1}} (q_{1}) ∊ ℜ^{m_{i} \times m_{i}}$ and $J_{φ_{i 2}} (q_{i}) ∊ ℜ^{m_{i} \times (n_{i} - m_{i})}$ . Taking into account the inverse differential visual kinematics defined in equation (12), it is possible to determine an m_i × m_i nonsingular matrix for q_i1 in visual space. Then, the velocity of the joints of the i-th robot can be expressed as

{\dot{q}}_{i} = [\begin{array}{l} {\dot{q}}_{i 1} \\ {\dot{q}}_{i 2} \end{array}] = J_{R i n v}_{_{i}} {\dot{x}}_{s_{i}} = [\begin{array}{l} {\bar{J}}_{R i n v_{i 1}} \\ {\bar{J}}_{R i n v_{i}_{2}} \end{array}] {\dot{x}}_{s_{i}} = [\begin{array}{l} {\bar{J}}_{R i n v_{i}_{1}} {\dot{x}}_{s_{i}^{}} \\ {\bar{J}}_{R i n v_{i}_{2}} {\dot{x}}_{s_{i}} \end{array}]

(15)

with ${\bar{J}}_{R i n v_{i 1}} ∊ ℜ^{m_{i} \times n_{i}}$ and ${\bar{J}}_{R i n v_{i 2}} ∊ ℜ^{(n_{i} - m_{i}) \times n_{i}}$ . Substituting ${\dot{q}}_{i 2}$ of equation (15) into (14), one obtains

J_{φ_{i 1}} (q_{i}) {\dot{q}}_{i 1} + J_{φ_{i 2}} (q_{i}) {\bar{J}}_{R i n v_{i}_{2}} {\dot{x}}_{s_{i}^{}} = 0

(16)

Solving equation (16) for q_i1 yields

{\dot{q}}_{i 1} = - J_{φ_{i 1}} {(q_{i})}^{- 1} J_{φ_{i 2}} (q_{i}) {\bar{J}}_{R i n v_{i 2}} {\dot{x}}_{s_{i}} = Ψ {\bar{J}}_{R i n v_{i 2}} {\dot{x}}_{s_{i}}

(17)

with

Ψ = - J_{φ_{i 1}}^{^{- 1}} (q_{i}) J_{φ_{i 2}} (q_{i})

(18)

where $J_{φ i 1}^{- 1} (q_{i})$ always exists on Ω because rank[J_φ1 (q_i)] = m_i, i.e., there are m_i independent contact points, and $Ψ : ℜ^{n_{i} - m_{i}} \to ℜ^{m_{i}}$ is well posed. That is, equation (17) is well posed because $J_{φ 1}^{- 1} (q_{i})$ exists in the finite workspace imposed by the holonomic constraint defined in equation (2). Taking into account the partitions expressed by equations (13) and (17), the velocity of the generalized coordinates can be written as

{\dot{q}}_{i} = Q_{i} J_{R i n v_{2}} {\dot{x}}_{s_{i}}

(19)

where $Q_{i} ∊ ℜ^{n_{i} \times (n_{i} - m_{i})}$ has a full rank (n_i – m_i), and in general will have the following form

Q_{i} = (\begin{array}{l} Ψ \\ I_{n_{i}}_{- m_{i}} \end{array})

Since J_φ1 (q_i) Q_i = 0, the image of Q_i lies in the null space of J_φ1 (q_i), i.e., the n-state space is decomposed into two orthogonal subspaces such that it can be written as direct sum ℜ_n range {J_φi (q_i)}⊗ range {Q}.

Then, independent joint generalized coordinates are formed by q_i2 and its derivative.

3. Visual system

Visual Error Manifolds. Given that Q_i and J_φi+ (q_i) are orthogonal complements, and ${\dot{q}}_{i}$ lies in the tangent plane at the contact point, a natural decomposition of ${\dot{q}}_{i}$ that arises at the contact point can be defined as

{\dot{q}}_{i} = Q_{i} J_{R i n v}_{_{2}} {\dot{x}}_{s_{i}} + J_{φ_{i +}} (q_{i}) {\dot{p}}_{i}

(20)

where

Q_{i} = I_{n_{i}} - J_{φ_{i +}} (q_{i}) J_{φ i} (q_{i})

In order to assure the position and force tracking, the nominal visual position/force reference ${\dot{q}}_{r i}$ can be designed as

\begin{array}{l} {\dot{q}}_{r_{i}} = \underset{P o s i t i o n}{\underset{︸}{Q_{i} J_{R i n v_{2}} {\dot{x}}_{r_{i}}}} + \underset{C o n s t r a i n e d v e l o c i t y}{\underset{︸}{J_{φ_{i +}}^{T} (q_{i}) ({\dot{p}}_{d_{i}} - E_{2_{i}} Δ p_{i})}} + \\ + \underset{F o r c e}{\underset{︸}{b_{i} J_{φ_{i +}}^{T} (q_{i}) (s_{v}_{f_{i}} + g_{f_{i}} \int_{t 0}^{t} s g n (s_{v f_{i}} (ζ)) d ζ))}} \end{array}

(21)

where ${\dot{x}}_{r i}$ stands for a visual nominal reference which parameterizes the system in terms of visual coordinates, E_2i is a positive definite diagonal matrix, $\nabla p_{i} = p_{i} - p_{d_{i}} = 0,$ with $p_{d_{i}}$ the desired constraint position, and the subscript d_i denotes desired reference, b_i and $g_{f}_{_{i}}$ are positive gain scalars, $s_{v f_{i}} = s_{f_{i}} - s_{d f_{i}}, s_{f_{i}} = Δ f_{i}, Δ f_{i} = \int_{t_{0}}^{t} (λ_{i} - λ_{d_{i}}) (ζ) d ζ^{2}, λ_{d_{i}}$ the desired force, $s_{d f_{i}} = s_{f_{i}} (t_{0}) e^{- k_{f_{i}}^{t_{0}}},$ and the function sgn(y) is the discontinuous signum function of vector y.

Uncalibrated Joint Error Surface. Assuming that the parameters of the i-th manipulator are uncertainties and the parameters α, θ and b are not well known because the camera is not calibrated, this implies that the nominal reference defined previously in equation (21) cannot be used. Since we are interested in simultaneously controlling the visual position-force of the i-th manipulator, the noncalibrated nominal reference ${\dot{\overset{⌢}{q}}}_{r_{i}}$ at the velocity level is

\begin{array}{l} {\dot{\overset{⌢}{q}}}_{r_{i}} = Q_{i} {\overset{⌢}{J}}_{R i n v_{2}} {\dot{x}}_{r_{i}} + J_{φ_{i +}} (q_{i}) ({\dot{p}}_{d_{i}} - E_{2_{i}} Δ p_{i}) + \\ + b_{i} J_{φ_{i +}}^{T} (q_{i}) (s_{v}_{f_{i}} + g_{f_{i}} \int_{t 0}^{t} s g n (s_{v f_{i}})) \end{array}

(22)

where ${\overset{⌢}{J}}_{R i n v_{2}}$ stands as an estimate of $J_{R i n v_{2}}$ so that the rank of ${\overset{⌢}{J}}_{R i n v_{2}}$ is n_i – m_i, for all t and for all q_i ∊ Ω. In addition, $E_{2 i}, g_{f_{i}}$ and $s_{v}_{f_{i}}, b_{i}$ are defined as in equation (21). Now, substituting (20) and (22) into equation (5), the uncertain visual error manifold becomes

{\overset{⌢}{s}}_{i} = Q_{i} (J_{R i n v_{2}} {\dot{x}}_{s_{i}} - {\overset{⌢}{J}}_{R i n v_{2}} {\dot{x}}_{r_{i}}) - J_{φ_{i +}}^{T} (q_{i}) (Δ {\dot{p}}_{i} + ξ_{2_{i}} Δ p_{i} + b_{i} s_{v x}_{f_{i}})

(23)

where $s_{v x}_{f_{i}} = s_{v}_{f_{i}} + g_{f i} \int_{t_{0}}^{t} s g n (s_{v}_{f_{i}})$ is called the extended orthogonalized force manifold. In order to guarantee the tracking of the visual errors, a second order sliding mode is proposed through the nominal visual reference ${\dot{x}}_{r_{i}}$ given as

{\dot{x}}_{r_{i}} = {\dot{x}}_{s d_{i}} - A_{i} Δ x_{s}_{_{i}} + s_{d p_{i}} - G_{p_{i}} \int_{t 0}^{t} s g n (s_{x_{i}})

(24)

where $A_{i}, G_{p_{i}},$ are positive-definite diagonal matrices, $Δ x_{s}_{_{i}} = x_{s}_{_{i}} - x_{s d_{i}},$ and $s_{d p}_{_{i}} = s_{x_{i}} (t_{0}) e x p^{- k_{p i}}^{(t - t 0)}$ with $s_{x_{i}} = Δ x_{s}_{_{i}} + A_{i} Δ x_{s_{i}} .$ Given that the sliding mode condition is satisfied for $s_{x_{i}}$ , and not for ${\overset{⌢}{s}}_{i},$ we have that the switching over the Cartesian manifold will be invariant to parametric uncertainties thanks to ${\dot{x}}_{r_{i}} .$

Open Loop Error Dynamics. The parametrization of the i-th robot manipulator (see equation (3)), requires the derivative of equation (22), which is discontinuous. Given that we will use neural networks to handle the entire system, we will simplify the equation (3) in order to avoid discontinuous signals. One way of doing this is to decompose ${\ddot{q}}_{r_{i}}$ into continuous and discontinuous terms as follows

{\ddot{\overset{⌢}{q}}}_{r_{i}} = {\ddot{\overset{⌢}{q}}}_{c o n t_{i}} + {\ddot{\overset{⌢}{q}}}_{d i s c_{i}}

(25)

with

\begin{array}{l} {\ddot{\overset{⌢}{q}}}_{con t_{i}} = {\dot{Q}}_{i} {\overset{⌢}{J}}_{{Rinv}_{i 2}} {\dot{x}}_{r_{i}} + Q_{i} {\dot{\overset{⌢}{J}}}_{{Rinv}_{i 2}} {\dot{x}}_{r_{i}} + Q_{i} J_{{Rinv}_{i 2}} (x_{{sd}_{i}} - A_{i} Δ {\dot{x}}_{s_{i}} + {\dot{s}}_{{sp}_{i}} - G_{p_{i}} \tanh (λ_{p_{i}} s_{x_{i}})) \\ + b_{i} {\dot{J}}_{φ_{i}}^{T} (q_{i}) s_{{vxf}_{i}} + b_{i} J_{φ_{i}}^{T} (q_{i}) (Δ f_{i} - {\dot{s}}_{{df}_{i}} + g_{f_{i}} \tanh (λ_{f_{i}} s_{{vf}_{i}})) \\ {\ddot{\overset{⌢}{q}}}_{dis c_{i}} = Q_{i} {\overset{⌢}{J}}_{{Rinv}_{i 2}} z_{p_{i}} - b_{i} J_{φ_{i}}^{T} (q_{i}) z_{f_{i}} \end{array}

with $z_{p i} = \tanh (λ_{p_{i}} s_{x_{i}}) - s g n (s_{x_{i}})$ , $z_{f_{i}} = \tanh (λ_{f_{i}} s_{v f_{i}}) - s g n (s_{v f_{i}}),$ and $λ_{p_{i}}, λ_{f_{i}} ∊ ℜ^{n \times n} > 0.$ . The continuous function tanh(*) is used as approximation of discontinuous function sgn(*), such that tanh(0) = 0 and $\tanh (λ_{p_{i}} s_{x_{i}}) \to s g n (s_{x_{i}}), \tanh (λ_{f_{i}} s_{v f_{i}}) \to s g n (s_{v f_{i}})$ when $λ_{i} \to \infty .$ . Notice that $z_{p_{i}}, z_{f_{i}}$ have the following properties: $z_{x} \geq - 1, z_{x} \leq + 1.$ Finally, the parametrization of (3) using the equations (22) and (25) gives rise to

H_{i} (q_{i}) {\ddot{\overset{⌢}{q}}}_{r_{i}} + C_{i} (q_{i}, {\dot{q}}_{i}) {\dot{\overset{⌢}{q}}}_{r_{i}} + g_{i} (q_{i}) = Y_{c o n t_{i}} {\overset{⌢}{m}}_{i} + u_{d_{i}}

(26)

where $Y_{c o n t_{i}} = Y_{b_{i}} (q_{i}, {\dot{q}}_{i}, {\ddot{\overset{⌢}{q}}}_{r_{i}}, {\ddot{\overset{⌢}{q}}}_{c o n t_{i}})$ is continuous due to $({\ddot{\overset{⌢}{q}}}_{r_{i}}, {\ddot{\overset{⌢}{q}}}_{c o n t_{i}}) ∊ C^{1}$ and $u_{d_{i}} = H_{i} ({\dot{q}}_{i}) {\ddot{q}}_{d i s c_{i}}$ is considered as an endogenous bounded disturbance subject to matching conditions. It is important to notice that $u_{d_{i}}$ cannot be compensated by the neural network, because it is discontinuous. However, $u_{d_{i}}$ is bounded due to $z_{p_{i}}$ and $z_{f_{i}}$ being bounded. Adding and subtracting equation (26) into equation (1), the open loop error equation arises as

H_{i} (q_{i}) {\dot{\overset{⌢}{s}}}_{i} + C_{i} (q_{i}, {\dot{q}}_{i}) {\overset{⌢}{s}}_{i} = u_{i} - Y_{c o n t_{i}} {\overset{⌢}{m}}_{i} + J_{φ_{i +}}^{T} (q_{i}) λ_{i} + u_{d_{i}}

(27)

It remains now to design the controller u_i that satisfies the problem raised. Our controller will compensate $Y_{c o n t_{i}} {\overset{⌢}{m}}_{i}$ with a low dimensional neural network, while the approximation error of the i-th neural network and $z_{p_{i}}$ , $z_{f_{i}}$ in $u_{d_{i}}$ will be compensated with an orthogonalized second order sliding mode inner loop. At this stage, before designing u_i, we discuss the design of the neural network architecture as an associator to approximated $Y_{c o n t_{i}} {\overset{⌢}{m}}_{i}$ .

4. Uncertain neuro-visual based controller

4.1 Neural Network Architecture

A tree network structure that satisfies the Stone-Weierstrass theorem [2] is used to approximate continuous regressor $Y_{c o n t_{i}},$ i.e., many neurons on one layer feed a single neuron on the next layer. The input-output relationship for this generic architecture is given as $y_{i} = φ (\sum_{k = 1}^{m} x_{k} w_{k}),$ where x_k is the k-th input to the neural network, w_k is the k-th weight, and ϕ the activation function. Notice that tree structure could have one or more hidden layers, where the linear activation function is used as the last stage of a multilayer neural network. Considering the property that neural networks can approximate any smooth function f(x), where φ belongs to a compact set s of Rⁿ [3], [11], we can use a sufficiently large neural network to approximate some function f(x) such that

\min_{{\tilde{w}}_{i}} | f (x) - \overset{⌢}{f} (x, \tilde{w}) |

with $\overset{⌢}{f} (x, \tilde{w}) \equiv φ (x \cdot \tilde{w}),$ and $‖ \tilde{w} ‖ \leq w_{m a x},$ is minimized.

Given that the regressor $Y_{c o n t_{i}}$ is continuous, it will be approximated by an ADAptive LINear Element (ADALINE) [20], which consists of a single neuron of the McCulloch-Pitts type. Using a matrix representation, the approximator will be designed such that

\min_{{\tilde{w}}_{i}} ‖ Y_{c o n t_{i}} {\overset{⌢}{m}}_{i} - {\overset{⌢}{f}}_{i} (X_{i}, {\tilde{w}}_{i}) ‖

with ${\overset{⌢}{f}}_{i} (X_{i}, {\tilde{w}}_{i}) \equiv X_{i}^{T} {\tilde{w}}_{i}, X_{i}^{T} = [q_{i}, {\dot{q}}_{i}, {\ddot{\overset{⌢}{q}}}_{r_{i}}, {\ddot{\overset{⌢}{q}}}_{c o n t_{i}}] ∊ ℜ^{n_{i} \times p_{i}}, {\tilde{w}}_{i} ∊ ℜ^{p_{i}},$ and $‖ {\tilde{w}}_{i} ‖ \leq w_{m a x},$ is minimized. Thus, the low dimensional neural network is used to approximate the unknown function $f (x) = Y_{c o n t_{i}} {\overset{⌢}{m}}_{i}$ by

\overset{⌢}{f} (x) = X_{i}^{T} {\tilde{w}}_{i} + e_{i} (X_{i})

(28)

where $X_{T}^{i}$ are the inputs to the neural network, ${\tilde{w}}_{i}$ are the estimated neural network weights for the i-th manipulator and $‖ e_{i} (X_{i}) ‖ \leq ∊_{N}$ is a functional reconstruction error with ∊_N > 0.

Given that the size of the neural network is defined by taking into account only the regressor elements, its approximation capacity is severely limited. However, instead of increasing its number of neurons in order to reduce the residual error, the error will be eliminated by means of a training procedure based on an orthogonalized second order sliding mode.

4.2 Neural-Visual Controller Design

Substituting equation (28) into equation (27), we have

H_{i} (q_{i}) {\dot{\overset{⌢}{s}}}_{i} + C_{i} (q_{i}, {\dot{q}}_{i}) {\overset{⌢}{s}}_{i} = u_{i} - X_{i} {\overset{⌢}{w}}_{i} + J_{φ_{i} +}^{T} (q_{i}) λ_{i} + {\bar{u}}_{d}

(29)

where ${\bar{u}}_{d} = u_{d_{i}} - e_{i} (X_{i}) .$

Now we are ready to present the main result.

Theorem 4.1. Assume that all initial conditions and desired trajectories of the i-th manipulator belong to Ω. Consider the dynamical model of the i-th manipulator in the closed-loop with the control law and neuro-adaptive law defined as

\begin{array}{l} u_{i} = - K_{d_{i}} {\overset{⌢}{s}}_{i} + x_{i}^{T} {\overset{⌢}{w}}_{i} + J_{φ_{i} +}^{T} (q_{i}) (- λ_{d_{i}} + η_{i} Δ f_{i}) \\ + γ_{F_{i}} J_{φ_{i}}^{T} (q_{i}) (\tanh (μ_{i} s_{v f_{i}}) + η_{i} \int_{t_{0}}^{t} s g n (s_{v f_{i}})) \end{array}

(30)

{\dot{\overset{⌢}{w}}}_{i} = - Γ_{i} X_{i} {\overset{⌢}{s}}_{i}

(31)

where $K_{d_{i}} ∊ ℜ^{n_{i} \times n_{i}}, {\overset{⌢}{w}}_{i} ∊ ℜ^{p_{i}}, Γ_{i} ∊ ℜ^{p_{i} \times p_{i}}$ and $λ_{d_{i}}$ is the i-th desired contact force. Then, for small initial conditions and large enough feedback gain matrix $K_{d_{i}},$ there is exponential convergence of the visual position error and force tracking error despite the existence of parametric uncertainties in ${\overset{⌢}{J}}_{R i n v_{i 2}} .$ .

Substituting equation (30) into equation (29), the closed-loop equation becomes

\begin{array}{l} H_{i} (q_{i}) {\dot{\overset{⌢}{s}}}_{i} + C_{i} (q_{i}, {\dot{q}}_{i}) {\overset{⌢}{s}}_{i} = \\ = - K_{d_{i}} {\overset{⌢}{s}}_{i} + x_{i}^{T} Δ w_{i} + J_{φ_{i} +}^{T} (q_{i}) ({\dot{s}}_{v f_{i}} + η_{i} s_{v f_{i}}) + {\bar{u}}_{d} \end{array}

(32)

where $Δ w_{i} = {\overset{⌢}{w}}_{i} - {\tilde{w}}_{i} .$

Proof: The proof is divided in three parts:

Part I. [Boundedness of Closed-loop Trajectories] Consider the time derivative of the following Lyapunov candidate function

V = \frac{1}{2} \sum_{i = 1}^{l} ({\overset{⌢}{s}}_{i}^{T} H_{i} (q_{i}) {\overset{⌢}{s}}_{i} + b_{i} s_{v f_{i}}^{T} s_{v f_{i}}^{} + Δ w_{i}^{T} Γ_{i}^{- 1} Δ w_{i})

(33)

along the solutions of (32), then it yields

\dot{V} \leq \sum_{i = 1}^{l} (- {\overset{⌢}{s}}_{i}^{T} K_{d_{i}} {\overset{⌢}{s}}_{i} - η_{i} b_{i} s_{v f_{_{i}}}^{T} s_{v f_{_{i}}}^{} + ‖ {\overset{⌢}{s}}_{r_{i}} ‖ {\bar{u}}_{d})

(34)

where ${\overset{⌢}{s}}_{i}^{T} {\bar{u}}_{d}$ is radially unbounded only when ${\overset{⌢}{s}}_{i} \to \infty$ and for bounded signals ${\overset{⌢}{s}}_{i}^{T} {\bar{u}}_{d}$ is zero only at ${\overset{⌢}{s}}_{i}^{T} = 0.$ For the small initial errors belonging to a neighbourhood ∊₁ and ∊₂ centred in the equilibriums ${\overset{⌢}{s}}_{i} = 0$ and ${\overset{⌢}{s}}_{v f_{i}} = 0,$ there exists a large enough feedback gain matrix $K_{d_{i}}$ such that each of the components of ${\overset{⌢}{s}}_{i}$ converges to ∊₁ > 0 as t → ∞, and ${\overset{⌢}{s}}_{v f_{i}}$ also converge to ∊₂ > 0 when t → ∞. This result stands for local stability of ${\overset{⌢}{s}}_{i}$ provided that the state is near the desired trajectories for any initial conditions. The boundedness in the L∞ sense leads to the existence of the constant ∊₃ > 0 such that

‖ {\overset{⌢}{s}}_{i} ‖ \leq ∊_{3}

Since desired trajectories are C² and feedback gains are bounded, we have that $({\dot{q}}_{r i}, {\ddot{q}}_{c o n t_{i}}) ∊ L \infty,$ which implies that $Y_{c o n t_{i}} ∊ L \infty, X_{i} ∊ L \infty$ and ${\dot{\overset{⌢}{w}}}_{i} ∊ L \infty .$ Then, the output of the neural network is also bounded. According the previous arguments and the boundedness of the i-th robot dynamics, its Coriolis matrix and gravitational term, and given that inertia matrix is positive-definite and upper bounded, the right hand side of equation (32) is bounded, such that ${\dot{\overset{⌢}{s}}}_{i} ∊ L \infty .$ Then, there exists a bounded scalar ∊₄ > 0 such that

‖ {\dot{\overset{⌢}{s}}}_{i} ‖ \leq ∊_{4}

So far, we can conclude the boundedness of all closed-loop error signals. To prove convergence of tracking errors, the sliding modes condition must be verified in the force subspace $J_{φ_{i +}}^{T}$ and in the position subspace Q_i.

Part II.a [Sliding modes for the velocity subspace]. Adding and subtracting $Q_{i} J_{R i n v_{2}} {\dot{x}}_{r_{i}}$ to (23), we obtain

\begin{array}{l} {\overset{⌢}{s}}_{i} = Q_{i} {J_{R i n v_{2}} s_{v x p_{i}} - Δ J_{R i n v_{2}} {\dot{x}}_{r_{i}}} - \\ - J_{φ_{i} +}^{T} (q_{i}) {Δ {\dot{p}}_{i} + ξ_{2_{i}} Δ p_{i}} - b_{i} J_{φ_{i} +}^{T} (q_{i}) s_{v x f_{i}} \end{array}

(35)

where $s_{v x p_{i}} = s_{v p_{i}} + γ_{p_{i}} \int_{t_{0}}^{t} s g n (s_{v p_{i}})$ is called extended orthogonalized position manifold and $Δ J_{R i n v_{2}} = {\overset{⌢}{J}}_{R i n v_{2}} - J_{R i n v_{2}} < σ,$ for σ > 0 under the assumption that $J_{R i n v_{2}}$ is not degenerated in Ω, and ${\overset{⌢}{J}}_{R i n v_{2}}$ is composed by trigonometric functions. Now, since ${\overset{⌢}{s}}_{i} ∊ L \infty, φ (q)$ is smooth and lies in the reachable robot space Ω, and $J_{R i n v_{2}}$ and Q_i are bounded, we have that $\frac{d}{d t} Q_{i} J_{{Rinv}_{2}} s_{{v x p}_{i}}$ and $\frac{d}{d t}$ $J_{φ_{i} +}^{T} s_{v x f_{i}}$ are bounded (this is possible because ${\dot{J}}_{φ_{i}}^{T}$ and Q_i are bounded). Then, there exist bounded constants ∊₅ > 0 and ∊₆ > 0 such that

| {\dot{s}}_{v x p_{i}} | < ∊_{5}, | {\dot{s}}_{v x f_{i}} | < ∊_{6}

Multiplying (35) by $Q_{T}^{i}$ and rearranging terms we obtain

\begin{array}{l} Q_{i}^{T} {\overset{⌢}{s}}_{i} = Q_{i}^{T} Q_{i} {J_{R i n v_{2}} s_{v x p_{i}} - Δ J_{R i n v_{2}} {\dot{x}}_{r_{i}}} - \\ - \underset{= 0}{\underset{︸}{Q_{i}^{T} J_{φ_{i} +}^{T} (q_{i}) {Δ {\dot{p}}_{i} + ξ_{2_{i}} Δ p_{i}}}} - \underset{= 0}{\underset{︸}{b_{i} Q_{i}^{T} J_{φ_{i} +}^{T} (q_{i}) s_{v x f_{i}}}} \end{array}

(36)

and

\begin{array}{l} J_{R i n v_{2}} s_{v x p_{i}} = Q_{i}^{Δ} {\overset{⌢}{s}}_{i} + Δ J_{R i n v_{2}} {\dot{x}}_{r_{i}} \\ {\bar{s}}_{v x p_{i}} = Q_{i}^{Δ} {\overset{⌢}{s}}_{i} + Δ J_{R i n v_{2}} {\dot{x}}_{r_{i}} \end{array}

(37)

where $Q_{i}^{Δ} = {(Q_{i}^{T} Q_{i})}^{- 1} Q_{i}^{T}$ and ${\bar{s}}_{v p x_{i}}$ is the local projection of $s_{v p x_{i}}$ onto $i m ({\bar{J}}_{2}) .$ Given that there are n_i – m_i independent coordinates, and m_i dependent coordinates can be expressed as a function of independent coordinates, then the holonomic constraint is well posed. That is, once a solution (given by a initial condition) is chosen, this solution cannot jump to another solution of $J_{R i n v_{1}}$ since we have ruled out crossing singularities, so it stays always in the domain of this solution. Then this local domain in fact becomes the global domain of this solution. So, it is reasonable to assume a bijection in this domain, between ${\bar{s}}_{v p x_{i}}$ and $s_{v p x_{i}}$ . Hence, without loss of generality we can analyse either ${\bar{s}}_{v p x_{i}}$ or $s_{v p x_{i}}$ . Taking into account the definition of $s_{v p x_{i}}$ , (37) becomes

s_{v p_{i}} = - γ_{p i} \int_{t_{0}}^{t} s g n (s_{v p_{i}}) + \underset{a}{\underset{︸}{Q_{i}^{Δ} {\overset{⌢}{s}}_{i} + Δ J_{R i n v_{2}} {\dot{x}}_{r_{i}}}}

(38)

Multiplying the time derivative of the equation (38) by $s_{v p_{i}}^{T}$ , we obtain that

\begin{array}{l} s_{v p_{i}}^{T} {\dot{s}}_{v p_{i}} = s_{v p_{i}}^{T} (- γ_{p i} s g n (s_{v p_{i}}) + \frac{d}{d t} a) \\ \leq - γ_{p i} | s_{v p_{i}} | + | s_{v p_{i}} | | \frac{d}{d t} a | \\ \leq - γ_{p i} | s_{v p_{i}} | + ∊_{7} | s_{v p} | \\ \leq - (γ_{p} - ∊_{7}) | s_{v p_{i}} | \\ \leq - μ_{2} | s_{v p_{i}} | \end{array}

(39)

where μ₂ = γ_pi – ∊₇ and $| \frac{d}{d t} a | \leq ∊_{7} .$ Therefore, we guarantee the existence of a sliding mode at $s_{v p_{i}} = 0$ if

γ_{p i} > ∊_{7}

at time $t_{p i} = \frac{| s_{v p i} (t_{0}) |}{μ_{2}} .$ Then by definition of $s_{v p_{i}}$ and given that $s_{v p_{i}} (t_{0}) = 0$

s_{v p_{i}} (t) = 0 \forall t

Then, a second order sliding mode is enforced, for all time, in the subspace of position.

Part II.b [Sliding modes for the force subspace] If we multiply equation (35) for $J_{φ_{i}}$ , we obtain

\begin{array}{l} J_{φ_{i}} {\overset{⌢}{s}}_{i} = \underset{= 0}{\underset{︸}{J_{φ_{i}} Q_{i} {J_{R i n v_{2}} s_{v x p_{i}} - Δ J_{R i n v_{2}} {\dot{x}}_{i}}}} - \\ - \underset{= 0}{\underset{︸}{J_{φ_{i}} J_{φ_{i} +}^{T} (q_{i}) {Δ {\dot{p}}_{i} + ξ_{2_{i}} Δ p_{i}}}} - b_{i} J_{φ_{i} +}^{T} (q_{i}) s_{v x f_{i}}, \\ b_{i}^{- 1} J_{φ_{i}}^{#} {\overset{⌢}{s}}_{i} = s_{v f_{i}} + g_{f_{i}} \int s g n (s_{v f_{i}}) \end{array}

where $J_{φ_{i}}^{#} (q_{i}) = {(J_{φ_{i}} J_{φ_{i} +}^{T} (q_{i}))}^{- 1} J_{φ_{i}} .$ The time derivative of the equation (40), multiplied by $s_{v f_{i}}^{T}$ , becomes

\begin{array}{l} s_{v f_{i}}^{T} {\dot{s}}_{v f_{i}}^{} = - g_{f_{i}} | s_{v f_{i}}^{} | + s_{v f_{i}}^{T} \frac{d}{d t} (b_{i}^{- 1} J_{φ_{i}}^{#} (q_{i}) {\overset{⌢}{s}}_{i}) \\ \leq - g_{f_{i}} | s_{v f_{i}}^{} | + | s_{v f_{i}}^{} | | \frac{d}{d t} (b_{i}^{- 1} J_{φ_{i}}^{#} (q_{i}) {\overset{⌢}{s}}_{i}) | \\ \leq - μ_{f_{i}} | s_{v f_{i}}^{} | \end{array}

(41)

where $μ_{f_{i}} = g_{f i} - ∊_{8},$ and $| \frac{d}{d t} [b_{i}^{- 1} J_{φ_{i}}^{#} (q_{i}) {\overset{⌢}{s}}_{i}] | \leq ∊_{8} .$ If g_fi > ∊₈, then a sliding mode at $s_{v f_{i}} (t) = 0$ is induced for all time, because $t_{f} \leq \frac{| s_{v f_{i}} (t_{0}) |}{μ_{f_{i}}} = 0$ since $s_{v f_{i}} (t_{0}) = 0$ .

Part III.a [Visual tracking errors] Since a sliding mode exists for all time at $s_{v p_{i}} (t) = 0,$ then we have

s_{v x p_{i}}^{} = s_{v p_{i}}^{} \forall t \to Δ x_{s i}^{\cdot} = - A_{i} Δ x_{s i}^{} + s_{d p_{i}}^{} (t_{0}) e^{- κ_{p_{i}} (t - t_{0})}

implying that position tracking errors globally and exponentially tends to zero. This is x_si → x_sdi and ${\dot{x}}_{s i}^{} \to {\dot{x}}_{s d i}^{} .$

Part III.b [Force tracking errors] Since a sliding mode $s_{v f_{i}} (t) = 0$ is induced for all time, this means that $Δ f_{i} = Δ f_{i} (t_{0}) e^{- κ_{f_{i}} (t - t_{0})} \to f_{i} = 0$ exponentially fast, thus it implies that the convergence of force tracking errors arises, $λ_{i} \to λ_{d_{i}}$ exponentially fast. Then $s_{r_{i}} = 0$ implies that $\lim_{t \to \infty} Δ {\dot{p}}_{i} = 0, \lim_{t \to \infty} Δ p_{i} = 0$ .

5. Numerical simulation results

In order to demonstrate the usefulness of our decentralized controller, we present digital simulations implemented on the stiff numerical solver on Matlab 5.3, under a 1 [ms] sampling time. In these simulations the motion of the object is confined to the horizontal plane. Additionally, each experiment was executed under different initial conditions, zero initial velocity and parametric uncertainties on the i-th manipulator and the camera. As mentioned in the section 4, the neural network has only one, layer and four weights per degree of freedom, where w_rm–ij (0) = 0 with m = 1, 2, i = 1, 2 and j = 1, 2, 3, 4.

The i-th robot parameters are defined as: masses $m_{1_{i}} = 0.0103589 [K g]$ , $m_{2_{i}} = 0.09 141[K g]$ , lengths $l_{1_{i}} = 0.08 [m]$ , $l_{2_{i}} = 0.06 [m]$ , masses centre, $l_{c 1_{i}} = 0.03647 [m]$ , $l_{c 2_{i}} = 0.03035 [m]$ , inertia $I_{1_{i}} = 0.000347 [K g m^{2}]$ , $I_{2_{i}} = 0.0020887 [K g m^{2}]$ for i-th robot finger. The Sony DFWVL500 CCD camera parameters are defined as: clock-wise rotation angle $θ = \frac{π}{2},$ scale factor α_v = 77772 [pixeles / m], depth field of view z = 1.5 [m], camera offset $^{v} O_{b} = {[- 0.2 - 0.1]}^{T} [m],$ offset ∊_IO_I = [0.00050.0003]^T [m] and focal length λ_f = 0.008 [m]. Finally, it is assumed that the image and joint coordinates are available in these simulations. As a reference, each figure presented has four graphics where the joints of robot j_r and the robot number r_m are defined as follows $R_{j_{r}, r_{m}}$ for j_r, r_m = 1, 2.

As a first case we consider a pair of robotic manipulators grasping a rigid object, i.e. the contact between object and manipulators is a frictionless contact point. The desired task consists of the end-effector of the i-th robot manipulator following a desired visual trajectories while a desired force is applied in the normal direction through the contact point.

Exponential visual tracking errors for position and velocity are presented in Figures 1 and 2, respectively. The system trajectories converge to an error that can be considered as zero (less than 0.1 pixel error). After a transient response in a few milliseconds in the first joints of the robotic manipulators, the control effort applied to each joint is smooth and chattering free, Fig. 3. Finally, the performance of the contact force and force tracking errors are presented in Fig. 4. As is expected, the simulation results show exponential convergence without any knowledge of i-th robot dynamics and the camera, i.e., the Jacobian ${\overset{⌢}{J}}_{R i n v_{2}}$ is parameterized in terms of regressor times as parameter vector ${\overset{⌢}{m}}_{i} .$ Then, the vector is multiplied by a factor to get X% of uncertainty with respect to the nominal value. In these simulations, the desired image-based desired trajectories are defined as follows: $x_{s d_{i}} = 0, 6 [p i x e l s], y_{s d_{i}} = 0.05 - 0.03 \sin (0.5 t) [p i x e l s]$ and $λ_{d_{i}} (t) = 0.2 + 0.4 \sin (5 t) [N]$ , respectively. In addition, the feedback gains used in this experiment are: $\begin{array}{l} K_{d_{i}} = d i a g [10], A_{i} = d i a g [3.8], ξ_{i} = d i a g [10], \\ b_{i} = d i a g [0.29], γ_{f i} = d i a g [0.0001], \\ γ_{p i} = d i a g [0.0001], k = 20 \end{array}$ .

Figure 1.

Visual position tracking errors for two robot fingers grasping an object.

Figure 2.

Visual velocity tracking errors for two robot fingers grasping an object.

Figure 3.

Smooth control applied to each joint.

Figure 4.

Contact force and force errors applied to the object by each finger.

Finally, we assumed that 90% of parametric uncertainties on ${\overset{⌢}{J}}_{{Rinv}_{2}}$ exists.

As a second case, a general scheme for free motion and constrained motion through a smooth transition is presented. A smooth transition allows us to avoid an impact regime between the robotic finger and the object, that is, a zero transition velocity from the free motion to constrained motion can be guaranteed. Therefore, impulsive dynamics does not appear and position controller commute stably to position/force controller and vice versa. To ensure that each robot finger approaches on the object surface to a velocity near to zero, it is necessary to change the feedback gain A_i defined in (24) by $A_{i} (t) = α_{0_{i}} \frac{ξ_{i}}{1 - ξ_{i} + δ_{i}}$ where $α_{0_{i}} = 1 + \in$ , for small positive scalar ∊ such that α₀ is close to 1 and 0 < δ ≪ 1. The time base generator ξ_i = ξ_i(t) ∊ C² must be provided by the user so as ξ_i goes smoothly from 0 to 1 in finite time $t_{i} = t_{b_{i}}$ and ξ_i = ξ_i(t) is a bell-shaped derivative of ξ_i such that $ξ_{i} (t_{0}) = ξ_{i} (t_{b_{i}}) \equiv 0$ [17]. Once a finite time convergence of position and velocity trajectories is assured in free motion, a stable switching of the closed-loop system to constrained motion is guaranteed.

Under the framework presented in the previous sections, the i-th geometric function in these three phases is defined as

\begin{array}{l} φ_{i} (q_{i}) > 0, 0 < t \leq t_{1} \Rightarrow ({\overset{⌢}{s}}_{i}, t) ∊ U \\ φ_{i} (q_{i}) \leq 0, t_{1} < t = (t + T_{c}) \Rightarrow ({\overset{⌢}{s}}_{i}, t) ∊ I \\ φ_{i} (q_{i}) \leq 0, t \geq (t_{1} + T_{c}) \Rightarrow ({\overset{⌢}{s}}_{i}, t) ∊ C \end{array}

where U is the unconstrained space (Phase 1), I is the transition space (Phase 2), C is the constrained space (Phase 3), t₁ is the contact time which is defined by the user through t_bi and T_c is the contact time which is a very small time period. For free and constrained tasks, Jφ_i+ q_i and Q_i are defined as follows

\begin{array}{l} J_{φ_{i} +} (q_{i}) = {\begin{cases} 0, ∊ U \\ \nabla φ_{i} / ‖ φ_{i} ‖, ∊ I \cup C . \end{cases}, & Q_{i} = {\begin{cases} I, ∊ U \\ I_{n_{i}} - J_{φ_{i} +} (q_{i}) J_{φ_{i}} (q_{i}), ∊ I \cup C . \end{cases} \end{array}

In addition, the noncalibrated nominal reference ${\dot{\overset{⌢}{q}}}_{r_{i}}$ becomes:

{\dot{\overset{⌢}{q}}}_{r_{i}} = {\begin{cases} Q_{i} J_{R i n v_{2}}^{- 1} {\dot{x}}_{r_{i}}, ∊ U \\ Q_{i} {\overset{⌢}{J}}_{R i n v_{2}}^{} {\dot{x}}_{r_{i}} + J_{φ_{i} +} (q_{i}) {{\dot{p}}_{d_{i}} - E_{2_{i}} Δ p_{i}} + b_{i} J_{φ_{i} +}^{T} (q_{i}) {s_{v f_{i}} + g_{f_{i}} \int_{t_{0}}^{t} sgn (s_{v f_{i}})}, ∊ I \cup C . \end{cases}

Therefore, the uncertain visual error manifold is defined as

{\overset{⌢}{s}}_{i} = {\begin{cases} Q_{i} (J_{R i n v_{2}}^{} {\dot{x}}_{s_{i}} - {\overset{⌢}{J}}_{R i n v_{2}}^{} {\dot{x}}_{r_{i}}), ∊ U \\ Q_{i} (J_{R i n v_{2}}^{} {\dot{x}}_{s_{i}} - {\overset{⌢}{J}}_{R i n v_{2}}^{} {\dot{x}}_{r_{i}}) - J_{φ_{i} +}^{T} (q_{i}) {Δ {\dot{p}}_{i} + ξ_{2_{i}} Δ p_{i} + b_{i} s_{v x f_{i}}}, ∊ I \cup C . \end{cases}

Then, using (42) the open loop error equation for free and constrained tasks is given as in equation (27), i.e.,

H_{i} (q_{i}) {\dot{\overset{⌢}{s}}}_{i} + C_{i} (q_{i}, {\dot{q}}_{i}) {\overset{⌢}{s}}_{i} = u_{i} - Y_{c o n t_{i}} {\overset{⌢}{m}}_{i} + J_{φ_{i} +}^{T} (q_{i}) λ_{i} + u_{d_{i}}

(43)

where

J_{φ_{i} +}^{T} (q_{i}) λ_{i} = {\begin{cases} 0, ∊ U \\ J_{φ_{i} +}^{T} (q_{i}) λ_{c o n t_{i}}, ∊ I \cup C . \end{cases}

with $λ_{c o n t_{i}}$ is now the contact force. Finally, the neuro-adaptive law is given as

u_{i} = {\begin{cases} - K_{d_{i}} {\overset{⌢}{s}}_{i} + x_{i}^{T} {\overset{⌢}{w}}_{i}, ∊ U \\ - K_{d_{i}} {\overset{⌢}{s}}_{i} + x_{i}^{T} {\overset{⌢}{w}}_{i} + J_{φ_{i} +}^{T} (q_{i}) [- λ_{d_{i}} + η_{i} Δ f_{i}] \\ + γ_{F_{i}} J_{φ_{i} +}^{T} (q_{i}) [\tanh (μ_{i} s_{v f_{i}}) + η_{i} \int_{t_{0}}^{t} sgn (s_{v f_{i}})], ∊ I \cup C . \end{cases}

and

{\dot{\overset{⌢}{w}}}_{i} = {\begin{cases} - Γ_{i} X_{i}^{T} {\overset{⌢}{s}}_{i}, ∊ U \\ - Γ_{i} X_{i}^{T} {\overset{⌢}{s}}_{i}, ∊ I \cup C . \end{cases}

where $K_{d_{i}}, γ_{F_{i}}, η_{i}, μ_{i}, λ_{d_{i}}, Γ_{i}$ have the same dimensions as defined in equations (30) and (31) respectively.

From the previous definitions for transition tasks, we proceed to describe the task. Consider two rigid planar robots as fingers which are not grasping the object initially. Then, from any initial condition the end-effector of the i-th robot finger will follow a desired trajectory (cubic polynomial) in finite time t_bi (free control) under 75% of parametric uncertainty on ${\overset{⌢}{J}}_{R i n v_{2}}$ . Once stable contact is achieved, the position controller commutes to a force/position controller (constrained motion). Then, $λ_{d i} = 2 \sin (5 t) [N], x_{s d_{i}} = 0.3482 [p i x e l s]$ and $y_{s d_{i}} = 0.4203 - 0.1 s e n (0.2 t) [p i x e l s]$ represent the force and desired visual trajectories in constrained motion under 75% of parametric uncertainty on ${\overset{⌢}{J}}_{R i n v_{2}} .$ Visual tracking errors for free and constrained motion are shown in Fig. 5. The impulse response between position and force controller commutation is avoided, as shown in Fig. 6. Instead, the performance of the control signal is smooth and chattering free. Additionally, in Fig.8 we can observe that the change in the velocity before and after collision, is the same, i.e. $\dot{q} (t_{b}) = \dot{q} (t_{b} + h),$ with h > 0 so that acceleration is analytic. On the other hand, Fig. 7 shows that before t ≤ 1.5[s], the contact force is zero, however, for t>1.5[s] the force converges to a desired value after a few seconds, i.e., the robot grasps an object rigidly. The feedback gains used in this experiment are:

Figure 5.

Visual position tracking errors for transition tasks.

Figure 6.

Smooth control input applied to each joint.

Figure 7.

Contact force and force error applied to the object by each finger.

Figure 8.

Visual velocity tracking errors for transition tasks.

\begin{array}{l} K_{d_{i}} = d i a g [10], A_{i} = d i a g [3.8], ξ_{i} = d i a g [10], \\ b_{i} = d i a g [0.29], γ_{f i} = d i a g [0.0001], \\ γ_{p i} = d i a g [0.0001], k = 20. \end{array}

Finally, we have not obtained a systematic procedure to tune the control gains basically because of the nonlinear nature of the closed-loop system. Thus, the feedback gains are tuned on a trial-and-error-basis, according to the interplay of each gain in the closed-loop system.

6. Conclusions

A neuro-visual servoing controller which guarantees image-based tracking of constrained multiple robot arms subject to parametric uncertainty is proposed. Exponential convergence in the visual force/position subspaces is guaranteed, even when neither robot parameters nor camera parameters are known. The main feature of our scheme is the ability to fuse the image coordinates/visual velocities and contact forces in orthogonal complements. The neural network control loop compensates for DAE dynamics, while an inner piecewise continuous sliding mode control loop adds the missing effort to induce sliding modes. Numerical simulations allow us to validate our proposed scheme through two significant applications: two robotic manipulators grasping a rigid object, and two robotic fingers executing tasks that involve changing dynamics and impact regime.

Footnotes

7. Acknowledgment

This work was partially supported by the FAI project ICI-002-11 of the Univ. de Los Andes, Chile.

1

Differential Algebraic Equation.

References

Baeten

Verdonck

Bruyninckx

De Schutter

(2000), Combining Force Control and Visual Servoing for Planar Contour Following, Machine Intelligence and Robotic Control, vol.2, no.2, pp.3–9.

Cotter

N. E.

(1990), The Stone-Weierstrass Theorem and Its Application to Neural Network, IEEE Trans. on Neural Networks, vol.1, no.4, pp.290–295.

Cybenko

(1989), Approximation by Superpositions of a Sigmoidal Function, Mathematics of Control, Signals, and Systems, vol. 2, pp. 303–314.

Dixon

W.E.

Zergeroglu

Fang

Dawson

D.M.

(2002), Object Tracking by a Robot Manipulator: A Robust Cooperative Visual Servoing Approach, Proc. IEEE International Conference on Robotics and Automation, pp.211–216.

SS.

Huang

Lee

T.H.

(2001), Model-based and Neural-network-based Adaptive Control of Two Robotic Arms Manipulating an Object with Relative Motion, International Journal of Systems Science, vol.32, no.1, pp.9–23.

Hutchinson

Hager

(1996), A Tutorial on Visual Servo Control, IEEE Trans. on Robotics and Automation, vol.12, no.5, pp.651–670.

Ishikawa

Sakane

Sato

Tsukune

(1996), Estimation of Contact Position between a Grasped Object and the Environment based on Sensor Fusion of Vision and Force, IEEE/SICE/RSJ Int. Conference Multi-sensor Fusion and Integration for Intelligent Systems, pp.116–123.

Kawasaki

Bin

Ramli R.

Ueki

(2006), Decentralized Adaptive Coordinated Control of Multiple Robot Arms for Constrained Tasks, Journal of Robotics and Mechatronics, vol.8, no.5, pp.580–588.

Kim

Hori

(1997), Analysis and Control of Grasping Motion by Two Cooperative Robots, Proceedings of the Power Conversion Conference, pp.513–518.

10.

Lewis

F.L.

Abdallaah

C.T.

(1994), Control of Robot Manipulators, New York: Macmillan.

11.

Lewis

F.L.

Yessildirek

Liu

(1996), Multilayer Neural-Net Robot Controller with Guaranteed Tracking Performance, IEEE Trans. Neural Networks, vol.7, no.2, pp.388–399.

12.

Kwan

C.M.

Yessildirek

Lewis

F.L.

(1999), Robust Force/Motion Control of Constrained Robots Using Neural Network, Journal of Robotics Systems, vol.16, no.12, pp.697–714.

13.

Liu

Y.H.

Arimoto

(1996), Distributively Controlling Two Robots Handling an Object in the Task Space without any Communication, IEEE Trans. on Automatic Control, vol.41, no.8, pp.11931198.

14.

Liu

Y.H.

Arimoto

Kitagaki

Parra-Vega

(1997), Model-based Adaptive and Distributed Controller for Holonomic Cooperations of Multiple Manipulators, Int. Journal of Control, vol.67, no.5, pp.649–673.

15.

Lui

Karim

(2000), Robust Control of Planar Dual-arm Cooperative Manipulators, Robotics and Computer-Integrated Manufacturing, vol.16, no.2–3, pp.109–119.

16.

McClamroch

N.H.

Wang

(1998), Feedback Stabilization and Tracking of Constrained Robots, IEEE Trans. on Automatic Control, vol.33, pp.419–426.

17.

Parra-Vega

Hirzinger

(2000), Finite-time Tracking for Robot Manipulators with Singularity-Free Continuous Control: A Passivity-based Approach, Proc. of the 39th IEEE Conference on Decision and Control, pp.5085–5090.

18.

Sarkar

Yun

Ellis

(1998), Live-Constraint-Based Control for Contact Transitions, IEEE Trans. on Robotics and Automation, vol.14, no.5, pp.743–754.

19.

Szewczyk

Plumet

Bidaud

(2002), Planning and Controlling Cooperating Robots through Distributed Impedance, Journal of Robotic Systems, vol.19, no.6, pp.283–297.

20.

Widrow

(1960), Adaptive Switching Circuits, Institute of Radio Engineering IRE WESCON Convention Record, Part 4, NY, pp. 96–104, 1960.

21.

Shen

Sun

Liu

Y.H.

(2003), Asymptotic Trajectory Tracking of Manipulators using Uncalibrated Visual Feedback, IEEE/ASME Trans. on Mechatronics, vol.8, no.1, pp.87–98.

22.

Liu

Y.H.

Hesheng

Dongxiang

(2006), Dynamic Tracking of Manipulators using Visual Feedback from an Uncalibrated Fixed Camera, Proc. IEEE International Conference on Robotics and Automation, pp.4124–4129.

23.

Loreto

Garrido

(2006), Stable Neurovisual Servoing for Robot Manipulators, IEEE Trans. on Neural Networks, vol. 17, no. 4, pp. 953–965.

24.

Pagilla

P.R.

(2004), An Experimental Study of Planar Impact of a Robot Manipulator, IEEE/ASME Transactions on Mechatronics, vol.9, no.1, pp.123–128.

25.

Shoji

Inaba

Fukuda

(1991), Impact Control of Grasping, IEEE Trans. on Industrial Electronics, vol.38, no.3, pp.187–194.

26.

Uzmay

Burkan

Satihaya

(2004), Application of Robust and Adaptive Control Techniques to Cooperative Manipulation, Control Engineering Practice, vol.12, no.2, pp.139–148.

27.

Xiao

(2000), Sensor-hybrid Position/Force Control of a Robot Manipulator in an Uncalibrated Environment, IEEE Trans. Control System Technology, vol.8, no.4, pp. 635–645.