A novel flexible two-step method for eye-to-hand calibration for robot assembly system

Abstract

In the visual robot system, the calibration of the hand-eye system is very important, which has a great influence on the positioning accuracy of the robot. Traditional methods are either complicated or require advanced external equipment. This paper proposes a new flexible method for hand-eye calibration, which is simple and flexible. Firstly, the robot carries the target to perform two transformation motions to solve the rotation relationship, and then the robot tool coordinate system performs several rotation motions to solve the transformation relationship. The paper provides empirical insights about how the robot hand-eye system is calibrated by controlling the robot to perform the specified motion without expensive and complicated 3D measurement equipment. The experiment and analysis indicate that the developed hand-eye calibration has high precision in 6-DOF industrial robot assembly application.

Keywords

Hand-eye calibration robot automatic assembly transformation matrix control optimization

Introduction

Machine vision has the ability to sense and measure the environment, and can guide robots to perform tasks such as grasping and assembly. Ahmad et al.¹ introduced a ToF sensor based information collection and intelligent decision methodology in order to localize the un-known, un-programmed obstacles and propose a safe peg-in-hole automated assembly process. Zeng et al.² presented an approach that leverages multi-view RGB-D data and self-supervised, data-driven learning to overcome the difficulties to realize reliably recognizes and locates objects amid cluttered environments, self-occlusions, sensor noise, and a large variety of objects. Wada et al.³ proposed a practical method to compose a highly reliable picking system with verification-based approach to reduce the rate of wrong picking and raise the reliability of picking ordered objects. In order to convert the position information in the vision system to the robot coordinate system, hand-eye calibration is an important step among them, and its accuracy affects the accuracy of robot positioning.

The traditional hand-eye calibration method needs to solve the equation AX = XB. The problem of solving equation AX = XB is very important in multiple fields such as robotics, computer vision or machine vision. In the problem, A and B are parameterizations of rigid body motions, while X is an unknown transformation. Condurache et al.⁴ proposed a new approach for solving the AX = XB sensor calibration problem by mapping the SE₃ classic formulation into an orthogonal dual tensors set SO₃ representation. For the Eye-in-Hand form, the camera is mounted on the end of the robot, and the camera carries a fixed calibration plate image at dozens of positions at the end. For the Eye-to-Hand form, the robot is used to carry the calibration plate for dozens of positions, and the camera is fixed to capture the calibration plate image. Jiang et al.⁵ proposed a calibration strategy for vision-guide robot assembly system, in which the eye-to-hand calibration method is developed with the uniaxial rotation of the robot joints and the pose tracking from the stereovision with large field of view. Jiang et al.⁶ also proposed eye-in-hand calibration method based on the homograph and the transformation of coordinates in the vision-guided robot grasping system. The effectiveness is proved by the great performance of the grasping results. The rotation axes between the two movements of the robot cannot be parallel to each other. The solution method can be divided into separable^7–9 or simultaneous.^10–12

There are also some other methods. Ma et al.¹³ proposed a technique for calibrating the head-eye geometry and the camera intrinsic parameters, which requires no reference object and directly uses the images of the environment. Yang et al.¹⁴ established a model that includes both the robot parameter error and the hand-eye relationship error. The robotic end-loaded structured light sensor is used to measure the fixed coordinates of the spherical center in the space. After transforming a few poses, the initial solution of the hand-eye relationship is obtained, and then the initial solution is substituted into the error model, and the robot error and the hand-eye error are simultaneously solved by the spherical center constraint. In order to solve the problem that the middle eye can not see the hand under the Eye-in-Hand situation, Zhang et al.¹⁵ let the robot end suction cup carry the calibration plate, the camera obtains the image of the calibration plate in the plane mirror, and solves the camera to the end target by changing the posture of the plane mirror three times. The pose is then measured by the auxiliary camera to measure the distance from the end target to the suction cup, and finally the hand-eye relationship of the camera to the suction cup is obtained. The robot does not move during the whole process, and the parameter error caused by the robot movement is avoided.

In recent years, with the improvement of GPU computing capabilities, some hand-eye relationship calibration methods based on deep learning have also appeared.^16–19 The basic principle is that after collecting a large number of scratch samples, a large convolutional neural network is trained so that the robot can learn the spatial relationship between the gripper and the object. Finally, a single image is used to predict the success of the gripper’s movement in the task space. The possibility of grasping does not require camera calibration and the current pose of the robot. However, it requires hundreds of thousands of sets of training data, and the trained neural network is configured for a specific robot, which will cause problems when converted to other targets.

However, it is usually required to achieve hand-eye calibration quickly and accurately in practical engineering applications. The above methods can not solve the problems that the poor absolute positioning accuracy of commercial robots, which results in the 1∼3 mm absolute positioning error in the hand-eye calibration data. At the same time, whether solving the equation AX = XB or based on the deep learning, the above methods need to shoot the calibration plate and record the robot’s position and pose for many times, which can not satisfies the application requirements of fast calibration. In order to solve these problems, this paper designs a new flexible two-step hand-eye calibration method for the Eye-to-Hand form. The rotation matrix of the calibration parameters is acquired with twice of robot transformation. The transformation matrix parameters are solved with the robot rotation operation without additional complicated and expensive measurement equipment for example CMM, laser tracker, and so on. Experiments show that the proposed method has stronger robustness and faster calibration process than the traditional method in practical application. The rest of the paper is as follows. We introduce the application scenarios of this paper and explain the principle of the hand-eye calibration method proposed in this paper in Section 2. We also introduce the system structure of this paper and carry out experiments to verify the calibration accuracy in Section 3. The results show that the method proposed in this paper has high accuracy.

Method

Measuring system configuration

The industrial robot is widely used in aerospace manufacturing field. It can be arranged to work precisely and repeatability. The robot gripping assembly is common task in industrial application, the measurement form is shown in Figure 1. The position accuracy of the workpiece should be within 2 mm in the region $1000 \times 1000$ mm² in this practical application. The gripper is fixed on the robot flange which is called end effector. The coordinate of the flange can be read with the robot system, the relationship of position and pose between the end effector and the flange is unknown. When the end effector is installed on the robot for work, the coordinate of grip point should be calibrated precisely. The gripper of the robot carries the workpiece from position 1 to the fixed position 2 above the monocular vision system, and the monocular vision system measures the workpiece. In order to convert the coordinates in the camera measurement coordinate system to the robot base calibration system, a hand-eye calibration method suitable for this layout must be designed. It is also necessary to use a camera to measure the captured part for capturing the part precisely because of the previous calibration and the robot gripping error.

Figure 1.

Measuring system configuration: (a) the station composition, (b) the actual measurement environment.

The novel two-step hand-eye calibration method

The coordinate system of the Eye-to-Hand structure which is needed to calibrated is shown in the Figure 2. The tool coordinate system is $O_{t} - X_{t} Y_{t} Z_{t}$ , the camera coordinate system is $O_{c} - X_{c} Y_{c} Z_{c}$ , and the robot base coordinate system is $O_{b} - X_{b} Y_{b} Z_{b}$ . Each time the robot grips the workpiece to take a picture at a fixed point above the monocular vision system, the relative positional relationship ${}_{c}^{t}T$ of the tool coordinate system established on the robot end-effectors relative to the camera coordinate system is ensured each time the robot is repeatedly positioned. It is fixed, so the relative position of the tool coordinate system and the camera coordinate system which is used to measure the position is calibrated, and the conversion relationship ${}_{t}^{b}T$ between the tool coordinate system and the robot base coordinate system can be read by the teach pendant, and the base is calculated. The relationship ${}_{c}^{b}T$ between the coordinate system and the camera coordinate system is also calculated.

Figure 2.

The coordinate system in calibration.

Several typical calibration methods are discussed in the introduction, some of which require the relative positional relationship between the camera and the robot. In the case of Eye-to-Hand, the positional relationship between the camera and the robot is very variable. Some methods are not suitable for use in all layout situations, and the calibration method of the robot carrying a calibration plate at the end of the robot can be applied to most occasions, so we will use the camera to calibrate the calibration plate on the end of the robot as a new two-step hand-eye calibration method.

Before the hand-eye calibration, the tool coordinate system had been calibrated by the four-point method described above, and the camera had also obtained the internal parameters and distortion coefficients after a single target. In order to obtain the relationship between the camera and the tool coordinate system, the camera must measure the point coordinates on the tool coordinate system. Since there is no cooperation target on the end-effectors, the code point calibration plate shown in Figure 3 is designed for the end-effectors. The coordinate system of the code point calibration plate is marked as $O_{d} - X_{d} Y_{d} Z_{d}$ , the X and Y axis directions are marked in the figure, and the Z axis is determined by the right-hand rule. Specific process is as follows:

Figure 3.

Code point calibration board design.

Due to the use of coded landmarks, canny edge detection is performed on the image first, and noise and non-coded landmarks are filtered through a series of constraints. Then, the least squares ellipse fitting is used to locate the coded mark points, and the contours of the coded mark points are segmented and filled based on the ellipse fitting error. Finally, the coding flag point is decoded. The corresponding code point and number is shown in Table 1. The decode numbers stands for the serial number of the code in the code library. No.0 is the origin of the coordinate $O_{d} - X_{d} Y_{d} Z_{d}$ . No.0 and No.1 form the X axis. No.0 and No.2 form the Y axis. According to X axis and Y axis, the Z axis is determined by the right-hand rule.

Table 1.

Code point decoding and corresponding serial numbers.

Decode numbers	51	61	277	287
No.	0	1	2	3

The distance between code points is known as 40mm. There are two reasons for using code points: First, there are more than one calibration plate in the camera field of view, there may be other circular targets, and the calibration plate is not fixed in the image. The code point can be effectively distinguished from other circular targets by decoding and concentric circle constraints, thereby quickly extracting the center of the circle. Secondly, by decoding, the four code points can be directly separated to form a coordinate system without additional design sorting algorithm.

In the calibration process, the monocular vision system takes a picture of the code point calibration plate to obtain the image coordinates of the center of the code point. Since the relative positional relationship between the code points is known, and the image coordinates of the center of the code point are obtained, we can obtain the coordinates of the code point in the camera coordinate system by solving the P4P problem combined with the camera internal parameters.

The hand-eye calibration method is divided into two steps. First, the rotation relationship between the tool coordinate system and the camera coordinate system is calibrated, and then the transformation relationship is calibrated.

Calibration of the rotation relationship

The rotation matrix of the tool coordinate system to the camera coordinate system is ${}_{t}^{c}R$ . Here we need to analyze the conversion relationship between the two coordinate systems. Let the two coordinate systems be $O_{A} - X_{A} Y_{A} Z_{A}$ and $O_{B} - X_{B} Y_{B} Z_{B}$ , and the transformation relationship between the fixed points in space between the two coordinate systems can be expressed as

[\begin{matrix} x_{A} \\ y_{A} \\ z_{A} \\ 1 \end{matrix}] = [\begin{matrix} {\hat{X}}_{B} • {\hat{X}}_{A} & {\hat{Y}}_{B} • {\hat{X}}_{A} & {\hat{Z}}_{B} • {\hat{X}}_{A} & t_{x} \\ {\hat{X}}_{B} • {\hat{Y}}_{A} & {\hat{Y}}_{B} • {\hat{Y}}_{A} & {\hat{Z}}_{B} • {\hat{Y}}_{A} & t_{y} \\ {\hat{X}}_{B} • {\hat{Z}}_{A} & {\hat{Y}}_{B} • {\hat{Z}}_{A} & {\hat{Z}}_{B} • {\hat{Z}}_{A} & t_{z} \\ 0 & 0 & 0 & 1 \end{matrix}] [\begin{matrix} x_{B} \\ y_{B} \\ z_{B} \\ 1 \end{matrix}]

(1)

The first column of the transformation matrix can be regarded as the projection of the unit vector on the X axis of $O_{B} - X_{B} Y_{B} Z_{B}$ onto the X, Y, and Z axes of the coordinate system $O_{A} - X_{A} Y_{A} Z_{A}$ . The second and third columns are the same. The fourth column is the coordinates of the origin of $O_{B} - X_{B} Y_{B} Z_{B}$ in $O_{A} - X_{A} Y_{A} Z_{A}$ . So each column of ${}_{t}^{c}R$ can be viewed as a projection of the unit vector on the principal axis of the tool coordinate system on each axis of the camera coordinate system.

Set the initial value of the rotation matrix as ${}_{t}^{c}R'$ , the calibration steps are as follows:

Calculation of ${}^{c}{\hat{X}}_{t}$ . Control the end of the robot to move a distance along the X-axis direction of the tool coordinate system. The moving distance does not need to be recorded. At the starting point and the end point, respectively, take images of the calibration plate to obtain the coordinates of the origin of the calibration plate coordinate system $O_{d}$ in the camera coordinate system. The coordinates calculated by the positive direction of the X-axis of the coordinate system are ${{}^{c}p}_{1}$ and ${{}^{c}p}_{2}$ respectively. we obtain the first column ${}^{c}{\hat{X}}_{t}$ of ${}_{t}^{c}R'$ by unitization after ${{}^{c}p}_{2} - {{}^{c}p}_{1}$ ;

Calculation of ${}^{c}{\hat{Y}}_{t}$ . Control the end of the robot to move a distance along the Y-axis direction of the tool coordinate system, and the subsequent steps are the same as (i), obtaining the second column ${}^{c}{\hat{Y}}_{t}$ of ${}_{t}^{c}R'$ ;

${}^{c}{\hat{X}}_{t}$ and ${}^{c}{\hat{Y}}_{t}$ cross-multiply to obtain the third column ${}^{c}{\hat{Z}}_{t}$ of ${}_{t}^{c}R'$ ;

Since ${}_{t}^{c}R'$ is not a strict rotation matrix, SVD decomposition is performed on ${}_{t}^{c}R'$ to find the rotation matrix ${}_{t}^{c}R = U V^{T}$ closest to ${}_{t}^{c}R'$ .

After the calculation of ${}_{t}^{c}R$ , the rotation matrix of the camera coordinate system to the tool coordinate system is ${}_{c}^{t}R = {}_{t}^{c}{R^{- 1}}$ .

Calibration of transformation relationship

The transformation matrix of the camera coordinate system to the tool coordinate system is ${}_{c}^{t}t$ . The transformation relationship can be solved by the end-effectors clamping the calibration plate to transform multiple poses, and the coordinate system involved is more, and the conversion relationship is shown in Figure 4.

Figure 4.

Alignment of the relationship between the coordinate systems involved in the transformation relationship.

In the fixed photographing position, the tool coordinate system is $O_{t} - X_{t} Y_{t} Z_{t}$ . At this time, the coordinate system of the calibration plate clamped on the end-effectors is $O_{d} - X_{d} Y_{d} Z_{d}$ , the conversion relationship between the tool coordinate system and the camera coordinate system is ${}_{t}^{c}T$ , and the tool coordinate system changes the pose, the new one. The tool coordinate system is $O_{t 1} - X_{t 1} Y_{t 1} Z_{t 1}$ , the corresponding calibration plate coordinate system is $O_{d 1} - X_{d 1} Y_{d 1} Z_{d 1}$ , and the conversion relationship between the tool coordinate system and the camera coordinate system is ${}_{t 1}^{c}T$ . Note that the origin of the calibration plate coordinate system is the center of gravity of the four code points, and the direction is the same as the direction of the tool coordinate system. Therefore, the calibration plate coordinate system has only a transformation relationship with the tool coordinate system, and because of the clamping and calibration during the calibration process. The board does not move relatively, so the conversion relationship between the calibration board and the tool coordinate system is

[\begin{matrix} I_{3 \times 3} & {}_{d}^{t}t \\ 0 & 1 \end{matrix}] = {}_{d}^{t}T = {}_{d 1}^{t 1}T

(2)

${}_{d}^{t}t$ is the coordinate of the origin of the calibration plate coordinate system in the tool coordinate system. Let the coordinates of the code point p in $O_{f} - X_{f} Y_{f} Z_{f}$ be ${{}^{f}p}_{i} (i = 0, 1, 2, 3)$ , the coordinates in $O_{c} - X_{c} Y_{c} Z_{c}$ be ${{}^{c}p}_{i}$ , the coordinates in $O_{t} - X_{t} Y_{t} Z_{t}$ be ${{}^{t}p}_{i}$ , and the rotation and transformation relations of $O_{f} - X_{f} Y_{f} Z_{f}$ to $O_{c} - X_{c} Y_{c} Z_{c}$ be ${{}^{f}R}_{c}$ and ${}_{f}^{c}t$ , respectively, which are measured by the monocular vision system. The rotation and transformation relations of $O_{c} - X_{c} Y_{c} Z_{c}$ to $O_{t} - X_{t} Y_{t} Z_{t}$ are ${}_{c}^{t}R$ and ${}_{c}^{t}t$ respectively. We assume ${}_{c}^{t}t = 0$ , so the conversion of ${{}^{f}p}_{i}$ to ${{}^{t}p}_{i}$ is

{\begin{matrix} {{}^{c}p}_{i} = {{}^{f}R}_{c}^{f} • p_{i} + {}_{f}^{c}t \\ {{}^{t}p}_{i} = {{}_{c}^{t}R}^{c} • p_{i} + {}_{c}^{t}t \end{matrix} (i = 0, 1, 2, 3)

(3)

According to the previous definition of $O_{d} - X_{d} Y_{d} Z_{d}$ , it is known

{{}^{t}O}_{d} = \sum_{i = 0}^{3} {{}^{t}p}_{i} / 4

(4)

However, the ${{}^{t}O}_{d}$ calculated here is different from the real value by a ${}_{c}^{t}t$ . If we can get that the true ${{}^{t}O}_{d}$ is also ${}_{d}^{t}t$ , the ${}_{c}^{t}t$ can be calculated, so the problem is now converted to solve ${}_{d}^{t}t$ .

Let the point p of the calibration plate be ${}^{d 1}p$ in $O_{d 1} - X_{d 1} Y_{d 1} Z_{d 1}$ . According to Figure 2, the coordinate ${}^{c}p$ of the point under $O_{c} - X_{c} Y_{c} Z_{c}$ can be expressed as

[\begin{matrix} {}^{c}p \\ 1 \end{matrix}] = [\begin{matrix} {}_{t 1}^{c}R & {}_{t 1}^{c}t \\ 0 & 1 \end{matrix}] [\begin{matrix} {}_{d 1}^{t 1}R & {}_{d 1}^{t 1}t \\ 0 & 1 \end{matrix}] [\begin{matrix} {}^{d 1}p \\ 1 \end{matrix}]

(5)

it can also be expressed as

[\begin{matrix} {}^{c}p \\ 1 \end{matrix}] = [\begin{matrix} {}_{t}^{c}R & {}_{t}^{c}t \\ 0 & 1 \end{matrix}] [\begin{matrix} {}_{d}^{t}R & {}_{d}^{t}t \\ 0 & 1 \end{matrix}] [\begin{matrix} {}_{d 1}^{d}R & {}_{d 1}^{d}t \\ 0 & 1 \end{matrix}] [\begin{matrix} {}^{d 1}p \\ 1 \end{matrix}]

(6)

With the equations (4) and (5), So there is

\begin{matrix} [\begin{matrix} {}_{t 1}^{c}R & {}_{t 1}^{c}t \\ 0 & 1 \end{matrix}] [\begin{matrix} {}_{d 1}^{t 1}R & {}_{d 1}^{t 1}t \\ 0 & 1 \end{matrix}] [\begin{matrix} {}^{d 1}p \\ 1 \end{matrix}] = [\begin{matrix} {}_{t}^{c}R & {}_{t}^{c}t \\ 0 & 1 \end{matrix}] [\begin{matrix} {}_{d}^{t}R & {}_{d}^{t}t \\ 0 & 1 \end{matrix}] \\ [\begin{matrix} {}_{d 1}^{d}R & {}_{d 1}^{d}t \\ 0 & 1 \end{matrix}] [\begin{matrix} {}^{d 1}p \\ 1 \end{matrix}] \end{matrix}

(7)

Combine equation (1), expand and simplify equation (6)

{\begin{matrix} {}_{t 1}^{c}R {}_{d 1}^{t 1}R = {}_{t}^{c}R {}_{d 1}^{d}R \\ ({}_{t 1}^{c}R - {}_{t}^{c}R) {}_{d}^{t}t = {}_{t}^{c}R {}_{d 1}^{d}t + {}_{t}^{c}t - {}_{t 1}^{c}t \end{matrix}

(8)

If the tool coordinate system is fixed and the rotation transformation is only performs around the TCP point, there is no transformation, then ${}_{t}^{c}t = {}_{t 1}^{c}t$ , the lower formula of formula (7) can be further simplified to

({}_{t 1}^{c}R - {}_{t}^{c}R) {}_{d}^{t}t = {}_{t}^{c}R {}_{d 1}^{d}t

(9)

In the above formula, ${}_{d 1}^{d}t$ is an unknown quantity. ${}_{d 1}^{d}t$ can be regarded as the coordinates of $O_{d 1}$ under $O_{d} - X_{d} Y_{d} Z_{d}$ . The coordinates of $O_{d}$ and $O_{d 1}$ under $O_{t} - X_{t} Y_{t} Z_{t}$ are ${{}^{t}O}_{d}$ and ${{}^{t}O}_{d 1}$ . Since $O_{d} - X_{d} Y_{d} Z_{d}$ and $O_{t} - X_{t} Y_{t} Z_{t}$ are in the same direction, so ${}_{d 1}^{d}t = {{}^{t}O}_{d 1} - {{}^{t}O}_{d}$ . According to formula (2) and formula (3), finally

{}_{d 1}^{d}t = \frac{1}{4} \sum_{i = 0}^{3} ({{}_{c}^{t}R}_{f 1}^{c} {{}^{f}p}_{i} + {}_{c}^{t}R_{f 1}^{c} t - {}_{c}^{t}R {{}^{f}R}_{c}^{f} p_{i} - {}_{c}^{t}R {}_{f}^{c}t)

(10)

In the above formula, ${}_{f 1}^{c}R$ and ${}_{f 1}^{c}t$ are the rotation and transformation relations of $O_{f} - X_{f} Y_{f} Z_{f}$ to $O_{c} - X_{c} Y_{c} Z_{c}$ after the tool coordinate system transforms the pose, and ${}_{c}^{t}t$ has been eliminated at the time of subtraction. Therefore, assuming that ${}_{c}^{t}t$ is any value which does not affect the calculation of ${}_{d 1}^{d}t$ .

Equation (8) can be regarded as the form of $Ax = b$ . It should be noted that both ${}_{t 1}^{c}R$ and ${}_{t}^{c}R$ are rotation matrices, and the matrix determinant obtained by the difference is zero. If the tool coordinate system only rotates a new position, only one equation of the formula (8) can be obtained and ${}_{d}^{t}t$ cannot be solved. Therefore, the tool coordinate system needs to be rotated multiple positions to solve the least squares solution of ${}_{d}^{t}t$ .

After calculating ${}_{d}^{t}t$ , according to the previous analysis, there is ${}_{c}^{t}t = {}_{d}^{t}t - {{}^{t}O}_{d}$ , so we can calibrate the transformation matrix of the camera coordinate system to the tool coordinate system.

Experiment results

The experiment processes include two parts. In the first part, the method of numerical simulation is used to realize the eye-hand calibration with different degrees of Gaussian error to the origin data of simulation, and analyze the difference with traditional method. In the second part, the proposed method is implemented and the error is analyzed.

For comparison, we simulate the traditional hand-eye calibration method for solving the AX = XB equation, and then calculate the error as the basis for measuring the accuracy of these methods. In the simulation, a Gaussian error with a mean of zero and a sigma of 0.1 to 4.0 is set. After calculating the hand-eye matrix X by different methods, $‖ A_{i} X - X B_{i} ‖ (i = 1, 2, \dots, n)$ is used as the error. The error data of experiments is shown in the Table 2. Methods 1∼5 corresponds to references 7∼11. The average accuracy of hand-eye calibration is about 1.8mm with the popular methods. The calibration accuracy decreases gradually with the increase of sigma.

Table. 2.

The error of hand-eye calibration by different method.

(a) sigma of 0.1.
MethodNumber	1	2	3	4	5
1	3.49595	3.49749	3.49741	3.5129	3.51505
2	0.823361	0.824579	0.823879	0.832076	0.850045
3	0.589875	0.589468	0.589721	0.60068	0.603205
4	2.86114	2.86146	2.86152	2.86657	2.86666
5	1.72122	1.72171	1.72154	1.6885	1.68963
Average error	1.898309	1.898941	1.898814	1.900145	1.904918
(b) sigma of 0.5.
MethodNumber	1	2	3	4	5
1	3.2117	3.21194	3.21217	3.21141	3.21276
2	0.997723	0.997902	0.997829	1.00923	1.01277
3	2.67168	2.67346	2.67254	2.63895	2.60215
4	3.54679	3.54611	3.54610	3.54391	3.54253
5	0.890628	0.889942	0.890381	0.889021	0.887122
Average error	2.263704	2.263871	2.263804	2.258504	2.251466
(c) sigma of 1.0.
MethodNumber	1	2	3	4	5
1	2.53159	2.53129	2.53155	2.50305	2.50293
2	2.75175	2.7534	2.75382	2.752	2.75947
3	2.51809	2.5177	2.51813	2.50441	2.50855
4	1.88348	1.88418	1.88385	1.86476	1.86237
5	3.44858	3.44913	3.44885	3.45234	3.43788
Average error	2.6267	2.62714	2.62724	2.61531	2.61424
(d) sigma of 2.0.
MethodNumber	1	2	3	4	5
1	2.85789	2.85822	2.85819	2.87022	2.87953
2	4.76858	4.76781	4.76763	4.79787	4.79399
3	0.317142	0.317135	0.317141	0.323173	0.324216
4	1.68715	1.68712	1.68722	1.67697	1.68197
5	2.54927	2.54931	2.54911	2.5596	2.55815
Average error	2.436006	2.435919	2.435858	2.445567	2.447571
(e) sigma of 4.0.
MethodNumber	1	2	3	4	5
1	5.77108	5.77205	5.77213	5.75327	5.74574
2	1.51552	1.51645	1.51667	1.52278	1.52406
3	0.713459	0.714153	0.713504	0.703665	0.695929
4	1.43723	1.43718	1.43706	1.4389	1.43818
5	4.2107	4.21085	4.21167	4.19131	4.20125
Average error	2.729598	2.730137	2.730207	2.721985	2.721032

Our experimental hardware includes calibration boards, industrial robots, industrial cameras, and lenses. The 6-DOF industrial robot used in this experiment is KUKA KR60HA. KUKA KR60HA is a 6-axis industrial robot with a rated load of 60 kg, it has a 6 degrees of freedom, l, repeat positioning accuracy of ± 0.05 mm, track repeatability of ±0.16 mm, a working space of 27.24 m³, and a maximum horizontal and vertical working distance of 2033 mm and 3003 mm. The specific dimension information of the robot is shown in Figure 5(a). The corresponding controller model is KR C4. The camera uses The Imaging Source’s DMK 33G274, the lens is Computar M0814-MP2, and the focal length of lens is 8 mm. The experimental hardware information is shown in the Figure 5(b).

Figure 5.

The experimental condition: (a) the dimensions of the robot, (b) the experimental hardware information.

For the new two-step calibration proposed in this paper, first, the rotation relationship between the camera and the tool coordinate system is calibrated as described in Section 2. The image of the code point calibration plate acquired during the calibration process is shown in the Figure 6.

Figure 6.

Calibration of rotation relationship in two-step method: (a) position 1 and code point detection, (b) position 2 (c) Position 3.

It can be seen that the background environment of the calibration plate is more complicated, and the center of the circle can be well extracted due to the coding point. Finally, we calculate the rotation relationship ${}_{c}^{t}R$ . ${}_{c}^{t}R$ is the rotation matrix of the tool coordinate system to the camera coordinate system.

{}_{c}^{t}R = [\begin{matrix} 0.01135 & 0.9989 & 0.04505 \\ 0.9999 & - 0.01184 & 0.01065 \\ 0.01117 & 0.04492 & - 0.9989 \end{matrix}]

(11)

Then control the tool coordinate system to rotate different angles, collect the calibration plate image under each attitude and record the reading of the tool coordinate system in the robot base coordinate system, and substitute the equation (8) to form a linear equation system to obtain ${}_{d}^{t}t = (10.444, 2.8617, - 4.8581)^{T}$ . The ${{}^{t}O}_{d} = (21.658, 2.751, - 267.633)^{T}$ and finally calculate ${}_{c}^{t}t = (- 11.214, 0.1112, 262.78)^{T}$ . Combine the rotation transformation matrix to obtain the homogeneous transformation matrix ${}_{c}^{t}T$ of the monocular camera coordinate system to the tool coordinate system. ${}_{t}^{b}T$ can be obtained by reading the corresponding parameters by the teach pendant.

{}_{t}^{b}T = [\begin{matrix} 1 & - 6.98 e - 3 & 2.44 e - 3 & 995.39 \\ - 6.98 e - 3 & - 1 & - 8.9 e - 4 & - 341.41 \\ 2.44 e - 3 & 8.73 e - 4 & - 1 & 752.34 \\ 0 & 0 & 0 & 1 \end{matrix}]

(12)

Finally calculate the hand-eye relationship

\begin{matrix} {}_{c}^{b}T = {}_{t}^{b}T {}_{c}^{t}T = \\ [\begin{matrix} 4.39 e - 3 & 0.9991 & 0.04254 & 984.82 \\ - 0.9999 & 4.83 e - 3 & - 0.01007 & - 341.68 \\ - 0.01027 & - 0.04249 & 0.999 & 489.54 \\ 0 & 0 & 0 & 1 \end{matrix}] \end{matrix}

(13)

The entire calibration process requires only a few translations and rotations of the robot, which is very fast. As a proof of accuracy, ${}_{c}^{b}T$ is substituted into the AX=XB equation, taking the error $er r_{i} = ‖ A_{i} X - X B_{i} ‖ (i = 1, 2, \dots, n)$ . A and B are calculated by the robot carrying the calibration plate to move multiple positions. The final translation error component (x, y, z) and angular error component (a, b, c) of X are shown in the Table 3 with 5 sets of experiments. The error components are substituted into the error equation and the $er r_{i} = ‖ A_{i} X - X B_{i} ‖ (i = 1, 2, \dots, n)$ is calculated as the calibration accuracy. There are 10 times measurement experiments, and the final error calculation is shown in the Figure 7(a). The average error is 1.815 mm which is better than other traditional methods.

Table 3.

Error decomposition and corresponding times.

No.	x	y	z	a	b	c
1	1.2055	1.0217	0.2667	0.0437	0.2121	0.2057
2	1.2383	2.6438	0.8533	0.0420	0.5903	0.3984
3	0.6228	2.1036	0.2667	0.0827	0.5439	0.0865
4	0.3097	0.3134	0.3819	0.0092	0.1807	0.0620
5	1.5032	0.6463	0.0065	0.0282	0.0714	0.2610

Figure 7

The analysis of grasping experiment: (a) error of hand-eye calibration, (b) the result of grasping workpiece.

From the Table 3, we can find that maximum error of translation part of hand-eye calibration result is 2.64 mm and minimum error of translation part of hand-eye calibration result is 0.0065 mm. At the same time, the maximum error of rotate part of hand-eye calibration result is 0.5903 degrees and the minimum error of rotate part of hand-eye calibration result is 0.0092 degrees. The experiments show that the above error is acceptable and the calibration accuracy satisfies the needs to grasp the workpiece in practical engineering application. The result of experiment is showed as Figure 7(b).

It can be seen that the calibration method proposed in this paper is flexible and has the accuracy which is satisfied the grasping precision for robot assembly, the accuracy of this method is better than the direct product method. However, we can still find the error from the Figure 7(b). The reasons which results in the errors may be as follows: Firstly, the absolute positioning error of the industrial robot is too large and our method does not completely eliminate this part of the error. Secondly, the calibration of tool coordinate system lead into some errors. Finally, the precision of the code point calibration board used in the experiment is limited, which can also result in the error.

Conclusion

In this paper, a calibration method of the robot hand-eye system by controlling the robot to perform the specified motion is proposed, and the transformation matrix between the hand-eye systems is obtained. Compared with a series of methods of solving equation AX = XB, the developed method greatly reduces the times of camera shooting and robot motion in the calibration process, and simplifies the calibration process. At the same time, it overcomes the low calibration accuracy caused by the large absolute positioning error of the commercial robot, so that the calibration can be completed without the help of other measuring equipment in the calibration process. The experimental results show that the proposed method can satisfies the accuracy requirements of hand-eye calibration of 6-DOF robot, and the proposed method can quickly satisfy the speed requirements of practical application.

In general, some other kinds of robots which can operate the end operator rotation and transformation motion can be calibrated with our developed method. In the future, we will study the scalability of our method on other kinds of robots besides 6-DOF robots. The limitation of the method is that the calibration process is a fusion with multi-step rotation and transformation motion, the high and stable calibration accuracy can be acquired with suitable optimization algorithm. In addition, the calibration process requires a large amount of work, and there is still a lot of research space for optimization of calibration results.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work is supported by the Fundamental Research Funds for the Central Universities (CN)(NS2020030), the Jiangsu Province Nature Science Fund (BK20191280) and National Key Research and Development Project of China (2019YFB2006100 and2019YFB1707500).

ORCID iD

Haihua Cui

References

Ahmad

Plapper

Safe and automated assembly process using vision assisted robot manipulator. Procedia CIRP 2016; 41: 771–776.

Zeng

Song

, et al. Multi-view self-supervised deep learning for 6D pose estimation in the Amazon picking challenge. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017, pp.1699–1706. New York: IEEE.

Wada

Sugiura

Yanokura

, et al. Pick-and-verify: verification-based highly reliable picking system for various target objects in clutter. Adv Robot 2017; 31(6): 311–321.

Condurache

Burlacu

Orthogonal dual tensor method for solving the AX= XB sensor calibration problem. Mech Mach Theory 2016; 104: 382–404.

Jiang

Cui

Cheng

A calibration strategy for visually guided robot assembly system of large cabin. Measurement 2020; 163: 107991

Jiang

Cheng

Cui

, et al. Dual-camera-based method for identification and location of scattered self-plugging rivets for robot grasping. Measurement 2019; 134: 688–697.

Tsai

Lenz

RK.

A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. IEEE Trans Rob Autom 2002; 5(3): 345–358.

Park

Martin

BJ.

Robot sensor calibration: solving AX= XB on the Euclidean group. IEEE Trans Rob Autom 1994; 10(5): 717–721.

Horaud

Dornaika

Hand-eye calibration. Int J Rob Res 1995; 14(3): 195–210.

10.

Andreff

Horaud

Espiau

On-line hand-eye calibration. In: Proceedings of the second international conference on 3-D digital imaging and modeling (Cat No PR00062), F, Ottawa, ON, Canada, 8 October 1999. New York: IEEE.

11.

Daniilidis

Hand-eye calibration using dual quaternions. Int J Rob Res 1999; 18(3): 286–298.

12.

Strobl

Hirzinger

Optimal hand-eye calibration. In: Proceedings of the 2006 IEEE/RSJ international conference on intelligent robots and system, Piscataway, NJ, USA, 2006, pp.4647–4653. New York: IEEE.

13.

SD.

A self-calibration technique for active vision systems. IEEE Trans Rob Autom 1996; 12(1): 114–120.

14.

Yang

Yin

Ren

, et al. Improvement of calibration method for robotic flexible visual measurement systems. Opt Precis Eng 2014; 22(12): 3239–3246.

15.

Zhang

Jia

Zhang

Eye-on-hand calibration method based on specular reflection. China Mech Eng 2018; 29(10): 1153–1158+1165.

16.

Kappler

Bohg

Schaal

. Leveraging big data for grasp planning. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), Washington State Convention Center, Seattle, WA, USA, 25–30 May 2015, pp.4304–4311. New York: IEEE.

17.

Levine

Finn

Darrell

, et al. End-to-end training of deep visuomotor policies. J Mach Learn Res. 2016; 17(1): 1334–1373.

18.

Pinto

Gupta

Supersizing self-supervision: learning to grasp from 50K tries and 700 robot hours. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, 16–21 May 2016. New York: IEEE.

19.

Levine

Pastor

Krizhevsky

, et al. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Rob Res 2018; 37(4–5): 421–436.