Design of robot visual servo controller based on neural network

Abstract

Robot visual servo control research makes the robot more intelligent, adaptive and efficient. In this paper, the eye in the hand and image-based robotic visual servo system is built according to the analysis of the structure and control model of the robot visual servo system. The robot Jacobian matrix is gained based on SDOF manipulator Robot kinematics analysis and simulation. The coordinate transformation of the visual servo system and the image Jacobi matrix are analyzed and deduced. Finally, the visual servo controller is designed based on the neural network intelligent control algorithm. Simulation results show that this design not only simplifies the complexity of image Jacobi matrix and hand eye calibration, but also has good real-time performance and generalization ability.

Keywords

Image-based visual servo Jacobian matrix neural network simulation

1. Introduction

Robot visual servo control is the use of camera-captured image as a methodology to acquire image information on the target’s relative posture to robot-end effector, and form a closed-loop robot position control system by comparing the feedback value obtained from image processing with the preset expected value. It is an organic integration of both robot control technology and robot visual technology. The concept of visual servo was first brought up by Hill and Park in 1979 [1]. Although, so far, robot visual control has made great developments in China and at aboard, there are still many technical difficulties remaining unsolved. The main factors impeding visual servo system from extensive use include real-time performance issues due to low image collecting speed and long image processing time to the system, as well as the increase of calculation volume of the system upon the use of visual information [2].

1.1 The visual servo control system of image identification

A typical robot visual servo system consists of five farts: Lighting, camera, image gathering card, visual processor and the robot itself. For a robot visual servo system to run in good condition, fast processing capability and the design of identification controller are the keys. Its working process is as follow: Convert light signal first into electric signal, and then into digital signal via image gathering card, making it possible for visual processor to carry out direct process; use visual processor to calculate the error between the above-mentioned image and the expected image, the result will be mapped into control signal, which will drive the joint robot to move (see Fig. 1).

Figure 1.

Robot identification control system.

Image processing technology is used in robot tracking and positioning, continuous image sequence analysis and moving target detection to identify the specific location of each image. Then by connecting the locations of the moving target in continuous images together, a complete moving track of the target is obtained.

1.2 The classification of the robot identification control system

1.2.1 The classification based on position of the camera

Based on the relative location of camera to robot, visual servo system can be divided into 2 types: Eye-in-Hand and Eye-to-Hand [3].

For Eye-in-Hand System, camera is fixed to a certain location in robot space (i.e. the top right or external surface) via visual location device. In such way, fixed image resolution can be gained, as well as the image information and working environment of the robot arm. Such way of installation is easy for the control system to integrate, but the downside is the poor flexibility, the shadow phenomena of image feature that disables the camera from making detailed description on the task probably is the reason. While in Eye-to-Hand System, the camera is placed on robot’s arm, providing extensive view to avoid image shadow problems. Meanwhile, it can also improve the image resolution and measure accuracy by adjusting the arm’s position for the camera to approach the object observed. Its demerit is the weight of the camera loaded on robot’s arm. Because of its location, the camera is easy to be damaged. More importantly, the camera will loose the target from its vision scope as approaching. Eye-to-Hand System will be explored in this paper.

1.2.2 The classification based on control model

According to different control models, there are position-based visual servo system and image-based visual servo system. In line with the acquired target image, position-based visual servo system uses camera model and image information to estimate target’s relative posture to robot-end effector, and calculate the error between the effector’s current posture and the posture of the estimated target, and then complete the robotic visual servo task with visual servo controller’s feedback [4]. See Fig. 2 for its system structure. Through its 3D Cartesian Coordinates-based description on the expected robotic posture and trajectory, this servo model obtains definite physical meaning and higher feasibility, but it requires accurate marking of the camera’s intrinsic and extrinsic parameters. As its control accuracy is largely affected by camera calibration error and the error of robot arm motion model, the system is mainly used for 2D-space positioning and tracking.

Figure 2.

Position-based robotic visual servo system.

Image-based visual servo system uses image feature information to acquire image feature error, and controls robot’s joint motion via visual servo controller. This kind of visual servo system usually requires real-time nonlinear conversion information from image space to robot space, namely real-time Jacobian matrix. See Fig. 3 for its system structure [4].

Figure 3.

Image-based robotic visual servo system.

Such servo system, instead of reconstructuring a 3D model, is capable of calculating image feature error directly in 2D space. It has high control accuracy and is free from the influence of calibration error and special model error. However, the servo progress requires real-time Jacobian matrix and heavy calculation. It is difficult for both system stability analysis and controller design, let alone the easiness for it to enter into image Jacobian matrix singularity during servo stage.

This paper explores the robotic visual servo control by image-based visual servo control model, and the use of neural network controlling algorithm to simplify model mathematics calculation.

2. Jacobian matrix

2.1 Image Jacobian matrix

Image Jacobi matrix is used to describe the characteristic space of the system to represent the mapping relationship between the robot’s hands and eyes in the robot Visual Servo Control. The image Jacobi matrix describes the differential relationship between the image feature space and the robot end pose space. But the relationship between the change rate of the image feature and the angular velocity of the robot’s joint is determined by the compound Jacobi matrix.

The coordinates of robot in the task space is $q=[q_{1},q_{2},\ldots,q_{m}]^{T}$ . The corresponding position characteristics of robot in the image space is $f=[f_{1},f_{2},\ldots,f_{n}]^{T}$ . Then we can get the image Jacobi matrix to be defined as:

$\displaystyle\dot{f}=J_{\text{image}}\cdot\dot{q}$

and

$\displaystyle J_{\text{image}}=\left[\begin{array}[]{cccc}\frac{\partial f_{1}% }{\partial q_{1}}&\frac{\partial f_{1}}{\partial q_{2}}&\ldots&\frac{\partial f% _{1}}{\partial q_{m}}\\ \frac{\partial f_{2}}{\partial q_{1}}&\frac{\partial f_{2}}{\partial q_{2}}&% \ldots&\frac{\partial f_{2}}{\partial q_{m}}\\ \ldots&\ldots&\ldots&\ldots\\ \frac{\partial f_{n}}{\partial q_{1}}&\frac{\partial f_{n}}{\partial q_{2}}&% \ldots&\frac{\partial f_{n}}{\partial q_{m}}\end{array}\right]_{n\times m}$

In this expression, $\dot{f}\in R_{n}$ is system image feature parameters, $\dot{q}\in R_{m}$ is joint velocity parameters for $m$ free degree robot, $J_{\text{image}}$ is image Jacobian matrix.

2.2 Robot kinematics simulation

In this paper, a six DOF serial robot is put to use. According to the D-H method, the Descartes coordinate system of the robot is built. The MATLAB robot toolbox is applied to build the robot model and calculate the forward and reverse solution of the robot. According to the robot parameters, the Robotics Toolbox is used to build the simulation model of the robot in MATLAB software (Fig. 4). The robot terminal executor is set to move from the reference point A to the reference point B, and its running track is shown in Fig. 5.

The jtraj function is used to plan the joint space: $[q,q_{d},q_{dd}]=\text{jtraj}(q_{0},q_{1},N)$ . It gets the running trace of the robot terminal from A point to B point. The curves of the angle, the angular velocity and the angular acceleration of each joint of the robot is shown in Fig. 6.

2.3 Robot Jacobi matrix

The robot Jacobi matrix describes the linear relationship between the speed of the operating space of the terminal and the speed of the joint space. The motion equation of the robot is set to $r=r(q)$ . The equation expresses the relationship between the end operating space and the robot joint space. If both sides of the equation is took the derivative to time, the relationship between the angular velocity of a robot’s joint $\dot{q}$ and the speed of the robot’s end $\dot{r}$ can be get:

$\displaystyle\dot{r}=J_{\text{robot}}\dot{q}$

In this expression, $\dot{r}$ is the speed of the robot’s terminal executor in the operating space. It includes the speed of translation and the speed of rotation. $\dot{q}$ is the velocity of robot’s joint. $J_{\text{robot}}$ is the robot Jacobi matrix. It is a partial derivative matrix of $p\times m$ for a robot with $M$ joints.

$\displaystyle J_{\text{robot}}=\left[\begin{array}[]{ccc}\frac{\partial r_{1}}% {\partial q_{1}}&\ldots&\frac{\partial r_{1}}{\partial q_{m}}\\ \vdots&\ddots&\vdots\\ \frac{\partial r_{p}}{\partial q_{1}}&\ldots&\frac{\partial r_{p}}{\partial q_% {m}}\end{array}\right]$

Figure 4.

SDOF robot model.

Figure 5.

Robot motion curve.

Figure 6.

Robot motion simulation.

2.4 Compound Jacobi matrix

Based on the transformation of visual projection mapping, a differential equation between the motion of a robot’s joint and the change of the corresponding image features is built. It can be expressed as:

$\displaystyle\dot{f}=J_{i}\cdot\dot{r}=J_{i}J_{r}\cdot\dot{q}$

Therefore, the mapping relationship between the target image feature space and the robot joint space is expressed as:

$\displaystyle\dot{f}=J\cdot\dot{q}$

In this expression, $J_{i}$ is the image Jacobi matrix. $J=J_{i}J_{r}$ is the compound Jacobi matrix of system. It describes the proportional relationship between the joint movements of a manipulator and the change of its corresponding image features in a real time state. It effectively characterizes the real time characteristic relationship between the robot space and the image space. But the process of solving the real time image Jacobi matrix is time-consuming.

3. Design of visual servo control system based on neural network

3.1 Design of visual servo control system

According to the input and output parameters and the characteristics of the system, three layers of BP neural network are adopted in this paper. The network weights are acquired offline training and then used for online control. The structure of the system is shown in Fig. 7.

Figure 7.

Visual servo control system structure.

The role of neural network is to replace the solution of the compound Jacobi matrix of the robot. Then the deviation signal is get to map to the Cartesian space of the robot. Next the increment of the robot’s joint is get to pose to the robot servo controller [5]. Finally, the robot servo controller controls the joint action of the robot to grasp the object.

3.2 Design of servo controller

The key problem here is to estimate the relationship between the changes of the characteristic parameters of the image and the changes of pose of the robot. The control amount (e) is calculated by the difference between the image feature extracted by the camera and the difference value of the desired image feature. The BP neural network model is used to transform it into the motion space of the robot to drive the motion of each joint of the manipulator ( $\dot{q})$ .

Assuming $\xi_{d}$ is the Desired image feature. $\xi$ is the feedback image feature in real time. Then the difference between the two can be expressed as:

$\displaystyle e=\xi_{d}-\xi$

The visual servo controller is:

$\displaystyle\dot{q}=k_{p}J^{+}e$

In this expression, $\dot{q}$ is expressed as the angular velocity of the joint of the robot. $k_{p}$ is expressed as control gain. $J^{+}$ is expressed as the pseudo inverse matrix for the compound Jacobi matrix [6]. When the image features in the image apace are the same as the given image features, the visual servo task is finished.

4. Stimulation of neural network controller

The neural network training sample and test sample in this paper all come from robot motion and camera image identification. Network input is the differential between the current image feature and the expected one, and the output is the differential between the current joint angle and the expected one. In study stage, record the change of corresponding image feature according to the change of corresponding joint angle recorded from random move of robot-end effector in order to gain training sample for off-line network training. In executive stage, use neural network input to obtain the change of the expected joint angle.

Select four vertexes on the workpiece as its feature points, corresponding one-to-one to the feature points on the image. Select 8 network input layer nodes as the location differentials between the four image feature points and expected image feature points, and 6 output layer nodes representing the changing values of the robot’s six joint angles. Use pinhole-modeled camera, focal length 0.008 m, target imaging depth 0.4 m. Controller’s proportional coefficient (kp) is 1.5.

Figure 8.

Network fitting result error.

Figure 9.

Network performance curve.

As can be seen from the network performance curve (Fig. 9), network error reaches 10–2 at epoch 80 and levels out, which basically meets the network error requirement. Figure 8 shows its fitting result. Network correlation coefficient (R) equals to 0.98215, thus the network output generally meets the targets. Network output performance function (mse) equals to 3.8591e-005, indicating good performance.

5. Conclusion

In this paper, a neural network controller is designed based on the analysis of the robot Jacobi matrix and the image Jacobi matrix. The process of computing the image Jacobi matrix and the hand eye calibration is simplified based on neural network. The control algorithm is streamlined and optimized. From the simulation results and the output data of neural network, we can see that the controller designed in this paper can effectively approximate the nonlinear relationship between image difference and joint angle change. It not only simplifies the complexity of the computational process to a large extent, but also has good real-time and generalization ability. Therefore, it can be well applied to the robot visual servo control.

References

Yang

Y.X.

, Intelligent Robot Visual Servoing System Based on Image, Ph.D. Xi’an University of Technology, 2003.

Y.X.

, Research on Robot Uncalibrated Visual Servoing System Based on Multi-Intelligent Control Methods, Ph.D. South China University of Technology, 2009.

Wang

, Design of the Visual Servo System of 5-DOF Manipulaitor RM-501, Ph.D. Harbin Institute of Technology, 2011.

Tan

and Li

, Visual Measurement and Control for Robots, National Defence Industry Press, BeiJing, 2011, pp. 214–256.

Zhang

J.J.

and Wu

H.T.

, Simulation of an uncalibrated robotic visual servo system based on neural network, Applied Science and Technology 38(9) (2014), 1–5.

Liu

, Research on Uncalibrated Visual Servo Control System, Ph.D. Yanshan University, 2013.