Hand-eye calibration method with a three-dimensional-vision sensor considering the rotation parameters of the robot pose

Abstract

Hand-eye calibration is a fundamental step for a robot equipped with vision systems. However, this problem usually interacts with robot calibration because robot geometric parameters are not very precise. In this article, a new calibration method considering the rotation parameters of the robot pose is proposed. First, a constraint least square model is established assuming that each spherical center measurement of standard ball is equal in the robot base frame, which provides an initial solution. To further improve the solution accuracy, a nonlinear calibration model in the sensor frame is established. Since it can reduce one error accumulation process, a more accurate reference point can be used for optimization. Then, the rotation parameters of the robot pose whose slight errors cause large disturbance to the solution are selected by analyzing the coefficient matrices of the error items. Finally, the hand-eye transformation parameters are refined together with the rotation parameters in the nonlinear optimization solution. Some comparative simulations are performed between the modified least square method, constrained least square method, and the proposed method. The experiments are conducted on a 5-axis hybrid robot named TriMule to demonstrate the superior accuracy of the proposed method.

Keywords

Machine vision hand-eye calibration 3D-vision sensor machining robot

Introduction

The use of industrial robots is widespread in diverse machining operations, such as drilling, milling, and grinding. Compared with machine tools, machining robots have many advantages of large workspace, high degree of freedom (DOF), cost-effectiveness, and high flexibility. Hence, there have been attempts to use industrial robots for machining processes instead of machine tools. Recently, machining robots are expected to perform more complex tasks, such as low-volume automated production in unstructured environments. It requires the machining robots not only to have good machining performance but should be equipped with relevant measuring system to assist the robots to locate the machining position of workpiece.^1

–4 The commonly used measuring system in industry can be divided into contact measurement and noncontact measurement. Due to the complex and irregular surface profile of the workpiece, the contact measuring system is difficult to meet the needs of industry. The visual measuring system, as one of the widely applied noncontact measuring systems, is characterized by low cost, compact structure, easy integration, and installation and is capable of realizing multiple fields of view measurement when it is integrated into the robotic system and forms robotic visual measuring system. And the problems such as occlusion, shadow, and inefficient measurement data can be overcome.⁵ There are many visual measuring systems, such as monocular vision, stereo vision, structured light, and fringe projection. Fringe projection measuring system has been widely used in surface and mold measurement, rapid prototyping, reverse engineering, industrial detection, and other fields due to its characteristics of fast measurement speed, high precision, strong system reliability, and large amount of acquired point cloud. To obtain three-dimensional (3D) point clouds of the workpiece accurately and fast to determine the machining information, such as the position of the workpiece relative to the robot end effector and the machining allowance, a 3D-vision sensor based on fringe projection is adopted into the machining robot.

Two kinds of calibration need to be performed before the robotic visual measuring system is used. One is visual system calibration^6
–8 that is used to determine the internal parameters of the camera (called camera calibration) and external parameters between two cameras (called stereo calibration) and describes the mapping relationship between 3D space coordinates and two-dimensional (2D) image coordinates. Usually, the visual measuring system has been calibrated in advance by producer. The other is hand-eye calibration, which is used to determine the transformation between vision sensor and robot end effector. In this article, hand-eye calibration aims to establish cooperation between hand and eye to assist machining robotic positioning. A typical process of measurement-assisted robotic machining is shown in Figure 1. In the measurement process (No. 1), a 3D-vision sensor mounted onto the end effector of a machining robot measures the workpiece to obtain the machining information in the vision sensor frame. Through hand-eye transformation (No. 2), the machining information under the vision sensor frame is transformed into the robot end-effector frame. Finally, according to the current position of the robot end effector and the machining information, the robot end effector is driven to the machining position to complete machining (No. 3). It can be seen that hand-eye calibration is a key step that accurately determine the position and pose relationship between vision sensor and robot end effector to ensure the machining accuracy of the robot.

Figure 1.

Typical process of measurement-assisted robotic machining.

Essentially, hand-eye calibration is to solve the homogeneous transformation matrix between the vision sensor frame and the robot end-effector frame, also known as the hand-eye transformation matrix. There are many typical algorithms for solving the matrix, including screw theory,⁹ quaternions,¹⁰ dual quaternions,¹¹ Kronecker product,^12,13 and so on. According to the calculating process, hand-eye calibration methods can be divided into three categories. The first belongs to separable closed-form solutions,^14
–16 which is to solve rotation and translation separately. The rotation can be solved firstly by linear equations and then the translation is solved with the solved rotation. This kind of method is simple in derivation and convenient in solution, but the error of estimated rotation will propagate into the solution of the translation. It is sensitive to the noise in the collected data. The second belongs to simultaneous solutions,^17
–19 which is to solve rotation and translation simultaneously. Obviously, there is no error accumulation problem in this kind of method, but the rotation estimated cannot guarantee the orthogonality,²⁰ such as Kronecker product method. The rotation can be further optimized to satisfy the orthogonality, but the translation cannot meet the accuracy requirement. The third belongs to iterative solutions,^21
–23 which is to solve rotation and translation simultaneously by optimizing iteratively the objective function established. This kind of method usually has high accurate solution but is complex and time-consuming. In addition, the accuracy of solution depends on the initial value for iteration. Choosing wisely, linear method is generally adopted for estimating initial value and followed by iterative optimization for accurate solution.

In summary, various hand-eye calibration methods have been developed and tremendous progress has been made in solving hand-eye transformation matrix. However, most researchers only focus on the solution of the hand-eye matrix by different mathematical tools or how to avoid error accumulation during the solution. The effects of the robot pose error on the solution accuracy are not fully considered in their methods which are applied only in the case of low accuracy requirement. Some researchers have proposed to calibrate the hand-eye and the robot simultaneously,^24,25 which leads to complicated derivation and may arise linear correlation problems in parameter estimation. Especially for parallel or hybrid robots, the error modeling is actually a complicated problem.^26,27

This article aims to guide the positioning of machining robots using a 3D-vision sensor, which is integrated at the end effector of machining robots. An accurate hand-eye calibration method only considering the rotation parameters of the robot pose is proposed. The hand-eye calibration model is simple and easy to establish. In addition, the linear correlation problems are avoided through the parameter analysis. The main contributions of our work are summarized as follows:

A constrained least square solution is introduced to provide good initial values for the next step of optimization.

A nonlinear optimization solution that further improve the solution accuracy is presented simultaneously to optimize the rotation parameters that have a significant impact on the solution accuracy.

Simulations and experiments are conducted to compare the three different methods and demonstrate the superiority of the proposed method (PM).

The article is structured as follow. The second section introduces robotic visual measuring system, coordinate system and coordinate transformation, and conventional calibration method. In the third section, the PM of hand-eye calibration is presented and parameter analysis is also performed to discuss the effects of different parameters on the solution accuracy. Some simulation and experimental results are detailed in the fourth section, and the article ends with concluding remarks in the fifth section.

Background knowledge

Robotic visual measuring system

As shown in Figure 1, robotic visual measuring system is mainly composed of a 5-axis hybrid robot and a 3D-vision sensor. The 3D-vision sensor is mounted onto the A/C type wrist of the 5-axis hybrid robot, which can be considered as being fixed in spatial position with the end effector of the robot. In the working process of the system, the 5-axis hybrid robot cooperates with the 3D-vision sensor. After the robot moves to the designated position, the sensor performs the measurement function. After the measurement, the robot moves to the next designated position, and the sensor continues the measurement, and so on, until the whole measurement task is completed. Finally, the measured data of multiple robot poses are unified in the same coordinate frame, which generally is the workpiece coordinate frame, and the complex surface profile is reconstructed or the end effector of the robot is adjusted to the specified position for machining according to the measurement information. TCP/IP protocol is adopted to construct data communication between the robot and the vision sensor, which can implement real-time control.

The 3D-vision sensor consists of three sensors, which are two industrial charge coupled device (CCD) cameras and a digital light processing projector, as shown in Figure 2. The measurement principle is based on fringe projection profilometry. First, multifrequency fringes are projected onto the object, which is deformed by the depth change of the object and modulates the depth information into the fringes. At the same time, two CCD cameras on the left and right capture the deformed fringe images and transmit the images to data processing system. And then through phase analysis and phase unwrapping, the same imaging points of object in the left and right camera planes can be matched. Finally, 3D information of the object is recovered according to the system configuration parameters.

Figure 2.

Schematic diagram of 3D-vision sensor. 3D: three dimensional.

Coordinate system and coordinate transformation

As shown in Figure 3, hand-eye calibration mainly involves three coordinate frames: 3D-vision sensor frame (called sensor frame for short [SF]), robot end-effector frame (called end-effector frame for short [EF]), and robot base frame (called base frame for short [BF]). The origins of these coordinate frames are S, E, and B, respectively. $T_{E}^{S}$ is hand-eye transformation matrix from the sensor frame to the end-effector frame and is an unknown quantity to be solved. $T_{B}^{E}$ is the homogeneous transformation matrix from the end-effector frame to the base frame and is a known quantity. For any homogeneous transformation matrix T , it includes a rotation matrix R and a translation vector t , and it can be expressed in the form of homogeneous matrix as follows

T = [\begin{matrix} R & t \\ 0^{T} & 1 \end{matrix}]

where the rotation matrix R is an orthogonal matrix with nine elements, consisting of three orthogonal column vectors. It can be expressed as

R = [\begin{matrix} r_{1} & r_{2} & r_{3} \end{matrix}] = [\begin{matrix} r_{11} & r_{12} & r_{13} \\ r_{21} & r_{22} & r_{23} \\ r_{31} & r_{32} & r_{33} \end{matrix}]

Figure 3.

Schematic diagram of hand-eye calibration.

A standard ball is placed in the robot workspace for calibration, the position of which is guaranteed to be fixed. P is the spherical center.

After the robot moves to a certain position, the sensor scans the standard ball and obtains the coordinate of spherical center. The relationship that the coordinate of the spherical center is transformed from the sensor frame ${SF}$ to the base frame ${BF}$ is established

p^{B} = T_{B}^{E} T_{E}^{S} p^{S}

where $p^{B}$ represents the homogeneous coordinate of the spherical center P in the base frame, $p^{S}$ represents the homogeneous coordinate of the spherical center P in the sensor frame.

Conventional calibration method

When the robot moves to different positions and the sensor scans the standard sphere simultaneous, we can obtain the following equations²⁴

{\begin{cases} p^{B} = {}_{1}T ​_{B}^{E} T_{E}^{S} p_{1}^{S} \\ p^{B} = {}_{2}T ​_{B}^{E} T_{E}^{S} p_{2}^{S} \\ ⋮ \\ p^{B} = {}_{n}T ​_{B}^{E} T_{E}^{S} p_{n}^{S} \end{cases}

Since the position of the standard ball relative to the robot base remains invariable, without loss of generality, the first measurement of the spherical center in the base frame is regarded as the reference point, and the following constraint equations can be established

{}_{1}T ​_{B}^{E} T_{E}^{S} p_{1}^{S} = {}_{i}T ​_{B}^{E} T_{E}^{S} p_{i}^{S} i = 2, 3... n

By handling equation (5), we can obtain

A_{i} x = b_{i}

where $A_{i} = [\begin{matrix} p_{s 1}^{T} \otimes R_{1} - p_{s i}^{T} \otimes R_{i} & R_{1} - R_{i} \end{matrix}] i = 2, 3... n$ , $x = (\begin{matrix} vec (R_{h}) \\ t_{h} \end{matrix})$ , $b_{i} = t_{i} - t_{1}$ , $p_{s 1}$ , and $p_{s i}$ are the corresponding nonhomogeneous coordinates of $p_{1}^{S}$ and $p_{i}^{S}$ , respectively. $R_{1}$ and $R_{i}$ are the corresponding rotation matrices of ${}_{1}T _{B}^{E}$ and ${}_{i}T _{B}^{E}$ , respectively. $t_{1}$ and $t_{i}$ are the corresponding translation vectors of ${}_{1}T _{B}^{E}$ and ${}_{i}T _{B}^{E}$ , respectively. $R_{h}$ and $t_{h}$ are, respectively, the rotation matrix and translation vector of the hand-eye transformation matrix $T_{E}^{S}$ . $vec ()$ represents vectorization operation. And $vec (R_{h}) = {(\begin{matrix} r_{h 1}^{T} & r_{h 2}^{T} & r_{h 3}^{T} \end{matrix})}^{T}$ , $R_{h} = [\begin{matrix} r_{h 1} & r_{h 2} & r_{h 3} \end{matrix}] . \otimes$ denotes Kronecker operation. Superscript T is transpose symbol. Assuming $C 1$ is a $n_{1} \times n_{2}$ matrix, $C 2$ is a $n_{3} \times n_{4}$ matrix, then $C 1 \otimes C 2$ is a $n_{1} n_{3} \times n_{2} n_{4}$ matrix, and the following equation can be obtained

C 1 \otimes C 2 = [\begin{matrix} c_{11} C 2 & \dots & c_{1 n_{2}} C 2 \\ ⋮ & ⋱ & ⋮ \\ c_{n_{1} 1} C 2 & \dots & c_{n_{1} n_{2}} C 2 \end{matrix}]

According to equation (4), multiple equations similar to equation (6) can be obtained. The least square solution of hand-eye transformation parameters obtained is as follows

{\tilde{x}}_{LS} = {(\sum_{i = 2}^{n} A_{i}^{T} A_{i})}^{- 1} (\sum_{i = 2}^{n} A_{i}^{T} b_{i})

Since no constraint is imposed on the rotation of the hand-eye transformation matrix for solution, the rotation estimated is not orthogonal. The direct method is to rearrange the rotation ${\tilde{x}}_{LS} (1 : 9)$ of the least square solution ${\tilde{x}}_{LS}$ into the form of a rotation matrix

{\tilde{R}}_{LS} = matrix ({\tilde{x}}_{LS} (1 : 9), 3, 3)

where $matrix (x (1 : 9), 3, 3)$ is matrixing operation and is inverse operation of $vec (R_{h})$ . Assuming c is a $m_{1} m_{2}$ vector, then $matrix (c, m_{1}, m_{2})$ means to rearrange c as a $m_{1} \times m_{2}$ matrix. ${\tilde{x}}_{LS} (1 : 9)$ represents the first nine elements in the vector ${\tilde{x}}_{LS}$ . The singular value decomposition of the rotation matrix obtained ${\tilde{R}}_{L S}$ is computed in the form

{\tilde{R}}_{LS} = UΛ V^{T}

The ${\tilde{R}}_{LS}$ is corrected as follows

{\tilde{R}}_{MLS} = U V^{T}

The results with corrected rotation matrix are referred as modified least square solution ${\tilde{x}}_{MLS}$ as follows:

{\tilde{x}}_{MLS} = (\begin{matrix} vec ({\tilde{R}}_{MLS}) \\ {\tilde{x}}_{LS} (10 : 12) \end{matrix})

Proposed method

Constraint least square solution

The translation ${\tilde{x}}_{LS} (10 : 12)$ of the modified least square solution is not corrected synchronously with the rotation matrix ${\tilde{R}}_{L S}$ . And ${\tilde{x}}_{M L S}$ is no longer optimal in the sense of least squares. Considering the orthogonality of the rotation matrix, a constraint least square model is established

\begin{array}{l} min \sum_{i = 2}^{n} {‖ A_{i} x - b_{i} ‖}^{2} \\ s . t . {\begin{cases} r_{h 1}^{T} \cdot r_{h 1} - 1 = 0 \\ r_{h 2}^{T} \cdot r_{h 2} - 1 = 0 \\ r_{h 3}^{T} \cdot r_{h 3} - 1 = 0 \\ r_{h 1}^{T} \cdot r_{h 2} = 0 \\ r_{h 2}^{T} \cdot r_{h 3} = 0 \\ r_{h 1}^{T} \cdot r_{h 3} = 0 \end{cases} \end{array}

where ‖‖ denotes the Frobenius norm. This model is solved by introducing Lagrange multipliers²⁸ and the constraint least square solution is given as

({\tilde{x}}_{CLS}, {\tilde{λ}}_{j}) = \underset{x, λ_{j}}{arg min} \sum_{i = 2}^{n} {‖ A_{i} x - b_{i} ‖}^{2} + \sum_{j = 1}^{3} λ_{j} (x^{T} G_{j} x - 1) + \sum_{j = 4}^{6} λ_{j} (x^{T} G_{j} x)

where $λ_{i}$ is a Lagrange multiplier. G_i is a coefficient matrix corresponding to the constraint in equation (13). The obtained rotation in equation (14) is considered as being orthogonal.

Nonlinear solution considering the rotation parameters of the robot pose

According to equation (5), the accuracy of the reference point in the base frame is affected by both the measurement error of the sensor and the robot pose error. To obtain a more accurate reference point, the first measurement of the spherical center in the sensor frame can be referred as the reference point. As the accuracy of the reference point is only affected by the measurement error and one error accumulation process is reduced, the accuracy of the reference point in the sensor frame is higher than that in the base frame, which provides the possibility to make a better estimation for the hand-eye transformation parameters. However, the measurement by the sensor transformed from the sensor frame under other robot poses to the sensor frame where the reference point is located suffers from more error propagation and error accumulation. Therefore, additional parameters are introduced to reduce the effects of the errors on the solution accuracy. In addition, excessive additional parameters will lead to linear correlation between the parameters and reduce the accuracy of the solution. It is necessary to analyze the effects of all the candidate parameters on the solution accuracy. Through the parameter analysis in the “Parameter analysis” section, we observed that slight errors of the rotation parameters can cause large amount of error. Hence, the rotation parameters of the robot pose are regarded as the additional parameters and are optimized in a nonlinear calibration model together with the hand-eye transformation parameters. Based on the reference point in the sensor frame, the nonlinear calibration model is established as follows

f_{i} (R_{i}, R_{1}, R_{h}, t_{h}) = p_{s 1 i} - p_{s 1}, i = 2, ..., n

p_{s 1 i} = {(R_{1} R_{h})}^{T} R_{i} R_{h} p_{s i} + (R_{h}^{T} R_{1}^{T} R_{i} - R_{h}^{T}) t_{h} + {(R_{1} R_{h})}^{T} (t_{i} - t_{1})

where $f_{i} (R_{i}, R_{1}, R_{h}, t_{h})$ means the ith measurement error in the sensor frame when the first measurement of the spherical center is as the reference point. $p_{s 1 i}$ represents the coordinates of the spherical center transformed from the sensor frame with the ith robot pose to the sensor frame with the first robot pose. In addition, to ensure the orthogonality of the rotation matrix calculated in the optimization process, the rotation matrices $R_{i}, R_{h}, R_{1}$ are parameterized by 3D vectors $ω_{i}, ω_{h}, ω_{1}$ , respectively. The parameters of the nonlinear model include $R_{i}, R_{h}, R_{1}, t_{h}$ . Estimating the parameters of the nonlinear model in equation (15) is equivalent to minimizing the following objective function

\sum_{i = 2}^{n} f_{i} {(R_{i}, R_{1}, R_{h}, t_{h})}^{T} f_{i} (R_{i}, R_{1}, R_{h}, t_{h})

This is a nonlinear optimization problem, which is solved using Levenberg-Marquardt (LM) algorithm.²⁹ Nonlinear optimization requires a good initial value to ensure the accuracy of the solution. Here, the constraint least square solution in equation (14) is used as the initial values. We can obtain

({\tilde{R}}_{i}, {\tilde{R}}_{1}, {\tilde{R}}_{h, PM}, {\tilde{t}}_{h, PM}) \underset{R_{i}, R_{1}, R_{h}, t_{h}}{= argmin} \sum_{i = 2}^{n} f_{i} {(R_{i}, R_{1}, {\tilde{R}}_{h, CLS}, {\tilde{t}}_{h,CLS})}^{T} f_{i} (R_{i}, R_{1}, {\tilde{R}}_{h, CLS}, {\tilde{t}}_{h, CLS})

where ${\tilde{R}}_{h, CLS}, {\tilde{t}}_{h, CLS}$ represents the hand-eye transformation parameters solved by equation (14). The central step of LM algorithm is the solution of the normal equations that is given as follows

(J^{T} J + β E_{3 n + 6}) Δ u= - J^{T} f

where

J = {[\begin{matrix} J_{2}^{T} & J_{3}^{T} & \dots & J_{n}^{T} \end{matrix}]}^{T}, J_{i} = [\begin{matrix} \frac{\partial f_{i}}{\partial ω_{2}} & \dots & \frac{\partial f_{i}}{\partial ω_{n}} & \frac{\partial f_{i}}{\partial ω_{1}} & \frac{\partial f_{i}}{\partial ω_{h}} & \frac{\partial f_{i}}{\partial t_{h}} \end{matrix}], i = 2, .., n

f = {[\begin{matrix} f_{2}^{T} & f_{3}^{T} & \dots & f_{n}^{T} \end{matrix}]}^{T}

Δ u= {[\begin{matrix} Δ ω_{2}^{T} & \dots & Δ ω_{n}^{T} & Δ ω_{1}^{T} & Δ ω_{h}^{T} & Δ t_{h}^{T} \end{matrix}]}^{T}

J is the Jacobian matrix of f . The parameter β varies from iteration to iteration.²⁹ $E_{3 n + 6}$ is the $(3 n + 6) \times (3 n + 6)$ identity matrix. The iterative value $Δ u$ can be obtained by solving the normal equations.

Δ u= - {(J^{T} J + β E_{3 n + 6})}^{- 1} J^{T} f

The obtained $Δ u$ is subsequently used to update $R_{i}, R_{1}, R_{h}, t_{h}$ by

{\begin{cases} R_{i, new} = R_{i, old} exp ({[Δ ω_{i}]}^{\land}), i = 1, 2, ..., n, h \\ t_{h, new} = t_{h, old} + Δ t_{h} \end{cases}

The iterative process does not terminate until the variation of $‖ f ‖$ falls below a set threshold, which indicates that the parameters converge to the optimal value.

The PM is concluded in Algorithm 1:

Algorithm 1.

Nonlinear optimization method considering the rotation parameters

Parameter analysis

In this section, parameter analysis is carried on to select the key parameters that have a great impact on the solution accuracy of hand-eye transformation parameters. The key parameters will be as additional parameters and optimized together with the hand-eye transformation parameters to improve the solution accuracy.

It has been proved³⁰ that the solution accuracy is related to the condition number of the Jacobian matrix J . Matrices with small condition number are said to be well-conditioned. In other words, the smaller the condition number of J , the better the solution. To understand the impact of data errors on the solution, the total differential of function $f_{i} (R_{i}, R_{1}, R_{h}, t_{h})$ is given as follows

Δ f_{i} = \frac{\partial f_{i}}{\partial p_{s i}} Δ p_{s i} + \frac{\partial f_{i}}{\partial t_{i}} Δ t_{i} - \frac{\partial f_{i}}{\partial t_{1}} Δ t_{1} + \frac{\partial f_{i}}{\partial ω_{i}} Δ ω_{i} + \frac{\partial f_{i}}{\partial ω_{1}} Δ ω_{1} + \frac{\partial f_{i}}{\partial ω_{h}} Δ ω_{h} + \frac{\partial f_{i}}{\partial t_{h}} Δ t_{h}

where $f_{i}$ is short for $f_{i} (R_{i}, R_{1}, R_{h}, t_{h})$ . And $Δ p_{s i}, Δ t_{i}, Δ t_{1}, Δ ω_{i}, Δ ω_{1}$ denote the data error items, the coefficient matrices of which need to be analyzed. The derivatives of $f_{i}$ with respect to $p_{s i}, t_{i}, t_{1}, ω_{i}, ω_{1}$ are given as follows

{\begin{cases} \frac{\partial f_{i}}{\partial p_{s i}} = {(R_{1} R_{h})}^{T} R_{i} R_{h} \\ \frac{\partial f_{i}}{\partial t_{i}} = {(R_{1} R_{h})}^{T} \\ \frac{\partial f_{i}}{\partial t_{1}} = {(R_{1} R_{h})}^{T} \\ \frac{\partial f_{i}}{\partial ω_{i}} = - {(R_{1} R_{h})}^{T} R_{i} {[R_{h} p_{s i} + t_{h}]}^{\land} \\ \frac{\partial f_{i}}{\partial ω_{1}} = R_{h}^{T} {[R_{1}^{T} (R_{i} R_{h} p_{s i} + R_{i} t_{h} + t_{i} - t_{1})]}^{\land} \end{cases}

In addition, $Δ ω_{h}, Δ t_{h}$ denote the parameter items that need to be estimated. The derivatives of $f_{i}$ with respect to $ω_{h}, t_{h}$ are given as follows:

{\begin{cases} \frac{\partial f_{i}}{\partial ω_{h}} = {[{(R_{1} R_{h})}^{T} (R_{i} R_{h} p_{s i} + (R_{i} - R_{1}) t_{h} + t_{i} - t_{1})]}^{\land} - {(R_{1} R_{h})}^{T} R_{i} R_{h} {[p_{s i}]}^{\land} \\ \frac{\partial f_{i}}{\partial t_{h}} = {(R_{1} R_{h})}^{T} R_{i} - R_{h}^{T} \end{cases}

where ${[]}^{\land}$ denotes Lie bracket.³¹ It can be seen that $\frac{\partial f_{i}}{\partial t_{i}}, i = 2, ..., n$ remains unchanged and is equal to $\frac{\partial f_{i}}{\partial t_{1}}$ . If $t_{i}, t_{1}$ are optimized as additional parameters, it will cause the linear correlation of the columns of J , which will seriously increase the condition number of J . Note that $\frac{\partial f_{i}}{\partial p_{s i}}$ is an orthogonal matrix and does not change the norm of the error $Δ p_{s i}$ , and the measurement accuracy of the sensor is relatively high (The measurement accuracy is given in the “Experiments” section), the effect of the measurement error caused by the sensor is ignored.

Observe the constitution of $\frac{\partial f_{i}}{\partial ω_{i}}$ and $\frac{\partial f_{i}}{\partial ω_{1}}$ , we can find that $\frac{\partial f_{i}}{\partial ω_{i}}$ and $\frac{\partial f_{i}}{\partial ω_{1}}$ can amplify the norm of $Δ ω_{i}$ and $Δ ω_{1}$ , respectively, due to these items $p_{s i}, t_{h}, t_{i} - t_{1}$ . This means that slight errors $Δ ω_{i}$ and $Δ ω_{1}$ can cause large error items $\frac{\partial f_{i}}{\partial ω_{i}} Δ ω_{i}$ and $\frac{\partial f_{i}}{\partial ω_{1}} Δ ω_{1}$ . As these errors are shared by the hand-eye transformation parameters $R_{h}, t_{h}$ , the solution will suffer from a large disturbance. The rotation parameters of the robot pose should be used as the additional parameters that significantly impact the solution accuracy and are optimized together with the hand-eye transformation parameters.

Simulations and experiments

Some simulations and experiments are carried out to verify the effectiveness of the PM. The PM is compared with the modified least square method (MLS) and the constraint least square method (CLS) under different aspects.

Simulations

The simulations are performed to compare the performance of the three methods described in this article from noise level and data bulk. The true values of the hand-eye transformation parameters ${\bar{ω}}_{h}, {\bar{t}}_{h}$ are given. The TriMule robot³² is used for the simulations. m(m = 10, 20…100) robot poses ${{\bar{ω}}_{i}, {\bar{t}}_{i}}_{i = 1, ..., m}$ are randomly generated in the robot workspace, and one measurement data $p_{s i}$ by the sensor under some robot pose is randomly generated simultaneously. According to equation (5), the remaining m − 1 measurement data ${p_{s i}}_{i = 1, ..., m - 1}$ can be calculated. Different levels of noise are added to the generated data as follows: $p_{s i} \leftarrow {\bar{p}}_{s i} + Δ p_{s i}, ω_{i} \leftarrow {\bar{ω}}_{i} + Δ ω_{i}, t_{i} \leftarrow {\bar{t}}_{i} + Δ t_{i}$

{\begin{cases} Δ p_{s i} \sim N (0, κ diag (σ_{p x}^{2}, σ_{p y}^{2}, σ_{p z}^{2})) \\ Δ ω_{i} \sim N (0, κ diag (σ_{ω x}^{2}, σ_{ω y}^{2}, σ_{ω z}^{2})) \\ Δ t_{i} \sim N (0, κ diag (σ_{t x}^{2}, σ_{t y}^{2}, σ_{t z}^{2})) \end{cases}

where $Δ p_{s i}, Δ ω_{i}, Δ t_{i}$ are normally distributed noise. $κ \in [1, 10]$ is noise level. $σ_{i}^{2}, i = p x, ..., t z$ is variance. The rotation error $e r_{ro}$ and translation error $e r_{tr}$ are defined in the following equations

e r_{ro} = \frac{1}{N} \sum_{i = 1}^{N} ‖ {\tilde{ω}}_{h, i} - {\bar{ω}}_{h} ‖

e r_{tr} = \frac{1}{N} \sum_{i = 1}^{N} ‖ {\tilde{t}}_{h, i} - {\bar{t}}_{h} ‖

where $\tilde{□}$ and $\bar{□}$ strand for the estimated value and the true value, respectively. The rotation matrices of the hand-eye transformation matrix ${\tilde{R}}_{h}, {\bar{R}}_{h}$ are parameterized by 3D vectors ${\tilde{ω}}_{h}, {\bar{ω}}_{h}$ . The rotation matrix of the robot pose ${\bar{R}}_{i}$ generated is also parameterized by 3D vector ${\bar{ω}}_{i}$ . N is total number of trials.

(a) Noise level

We set 10 noise levels and randomly select 20 sets of data to estimate the hand-eye transformation matrix with the three methods described above under each noise level; 500 trials are carried out under each noise level to get the average error of rotation and translation. The simulation results are shown in Figure 4. It shows the rotation error and translation error of the three methods at different noise levels.

Figure 4.

Rotation error and translation error at different noise levels.

It can be seen that with the increase of noise level, the rotation error and the translation error of the three methods increase. In terms of solving the rotation matrix, the accuracy difference of the three methods is not obvious, while in solving the translation vector, the accuracy of the PM is significantly better than the other two methods.

(b) Data bulk

We select different amounts of data (m = 10, 20…100) to estimate the hand-eye transformation matrix using the three methods under low, medium, and high noise levels. Similarly, 500 trials are carried out under each noise level and the same amount of data to get the average error of rotation and translation. The simulation results are shown in Figure 5. Figure 5(a) to (c) shows the rotation error and translation error of the three methods with different amounts of data under low, medium, and high noise levels, respectively.

Figure 5.

Rotation error and translation error with different amounts of data: (a) low noise level, (b) medium noise level, and (c) high noise level.

The simulation results indicate that the rotation error and the translation error of the three methods decrease with the amount of data increasing under each noise level. The accuracy improvement of the three methods is the most obvious when the amount of data becomes from 10 to 20. When about 40 sets of data, the accuracy improvement rate slows down significantly.

In terms of solving the rotation matrix, the PM is slightly better in accuracy than the other two methods before 20 sets of data. After 30 sets of data, MLS becomes the best in accuracy, and the PM is almost the same as CLS in accuracy.

In terms of solving the translation vector, the PM shows excellent advantages in accuracy. The accuracy difference between CLS and MLS gradually becomes smaller.

According to the simulation results of noise level and data bulk, the PM outperforms in the accuracy of the hand-eye transformation matrix than MLS and CLS. Balancing accuracy and efficiency in practical applications, it is recommended to select about 20 sets of data to estimate hand-eye transformation matrix.

Experiments

(a) Experimental setup

The experimental setup mainly consists of a 5-axis hybrid robot called TriMule (averaged repeatability 0.02 mm), a 3D-vision sensor developed by the research team of Tianjin University (field of view 170 mm × 170 mm), a designed artifact with known dimensions and data processing system, as shown in Figure 6. The artifact has two standard balls, ball A with the diameter of 19.9886 mm, ball B with the diameter of 19.9878 mm, whose center distance between the two balls is 59.8573 mm. The data processing system monitors the measurement state in real time and guarantees the measurement quality of the 3D-vision sensor. It also provides images and 3D-point cloud display.

Figure 6.

Experimental setup.

(b) 3D-vision sensor accuracy test

Before verifying the calibration method, the measurement accuracy of the sensor should be tested. The artifact is measured by the sensor from different positions, and the 3D-point cloud of the artifact is obtained. The diameter of ball A and ball B and the center distance between the two balls are calculated by fitting the point cloud measured. By comparing the calculated value with the standard value of the artifact, we can evaluate the measurement accuracy of the sensor. The measurement experiment of the artifact is shown Figure 7. The measurement results are listed in Table 1. Mean absolute error (MAE) represents the deviation between the measured value of the sensor and the truth value. Standard deviation (SD) represents the dispersion of the measurement data. MAE of ball A diameter, ball B diameter and the center distance is 0.0150 mm, 0.0101 mm, and 0.0058 mm, and SD is 0.0041 mm, 0.0094 mm, and 0.0045 mm, respectively, indicating that the accuracy of the sensor is enough high for the hand-eye calibration.

Figure 7.

Measurement experiment: (a) measurement of the artifact and (b) 3D-point cloud of the artifact. 3D: three dimensional.

Table 1.

Measurement results of the artifact.

Position	Ball A diameter (mm)	Ball B diameter (mm)	Center distance (mm)	Ball A diameter error (mm)	Ball B diameter error (mm)	Center distance error (mm)
1	20.0075	19.9738	59.8696	0.0189	−0.0140	0.0123
2	19.9954	19.9737	59.8506	0.0068	−0.0141	−0.0067
3	20.0036	20.0001	59.8626	0.0150	0.0123	0.0053
4	20.0063	19.9895	59.8628	0.0177	0.0017	0.0055
5	20.0025	19.9790	59.8609	0.0139	−0.0088	0.0036
6	20.0057	19.9730	59.8610	0.0171	−0.0148	0.0037
7	20.0080	19.9903	59.8637	0.0194	−0.0148	0.0037
8	20.0082	19.9863	59.8638	0.0196	−0.0015	0.0065
9	19.9986	19.9706	59.8621	0.0010	−0.0172	0.0048
10	20.0007	19.9733	59.8608	0.0121	−0.0145	0.0035
MAE (mm)				0.0150	0.0101	0.0058
SD (mm)				0.0041	0.0094	0.0045

SD: standard deviation; MAE: mean absolute error.

To compare the accuracy of the three methods, the hand-eye calibration experiment is carried out with the ball A of the artifact, and the hand-eye transformation matrix is calculated using the three methods. An important application of hand-eye calibration is to assist robotic positioning with vision sensors. Under the same conditions, the accuracy of auxiliary robotic positioning reflects the accuracy of hand-eye calibration. Hence, an analog experiment is designed to simulate the robotic positioning process of assisted by the 3D-vision sensor. By comparing the positioning accuracy, the accuracy of the three methods in solving the hand-eye transformation matrix can be indirectly evaluated.

The artifact is placed at a position in the robot workspace. The ball A of the artifact is measured by the sensor from one robot pose, and the ball B is measured from the other robot pose. The spherical center coordinates of ball A and ball B are transformed from the sensor frame to the base frame, respectively. The spherical center coordinates of ball A in the base frame are regarded as the starting point, and the center coordinates of ball B are regarded as the target point for positioning. The positioning error $e r_{po}$ is defined as follows

e r_{po} = | d_{est} - d_{s} |

d_{est} = ‖ {}_{1}T ​_{B}^{E} {\tilde{T}}_{E}^{S} p_{a}^{S} - {}_{2}T ​_{B}^{E} {\tilde{T}}_{E}^{S} p_{b}^{S} ‖

where $d_{est}, d_{s}$ represent the estimated value and standard value of the ball center distance. $p_{a}^{S}, p_{b}^{S}$ represent the homogeneous coordinates of the ball A’s center in the sensor frame at one robot pose ${}_{1}T _{B}^{E}$ and ball B’s center at the other robot pose ${}_{2}T _{B}^{E}$ , respectively.

The artifact is placed at 20 different positions in the robot workspace. The positioning errors are shown in Figure 8. The maximum/mean positioning errors are 0.1133/0.0367 mm (MLS), 0.0961/0.0278 mm (CLS), and 0.0703/0.0244 mm (PM), respectively. It can be seen that the positioning errors with the PM get lower than that with the other two methods.

Figure 8.

Positioning error at different positions of the artifact.

(d) Point cloud registration

Another important application of hand-eye calibration is to associate point clouds of the workpiece measured from different robot poses into a common coordinate frame to reconstruct the 3D model of the workpiece. Similarly, the accuracy of the three methods can be indirectly compared through the accuracy of the 3D model reconstructed.

The artifact is placed at a position in the robot workspace and measured by the sensor from two different robot poses. Both sets of point clouds are transformed from the sensor frame to the base frame and are stitched together to get the stitched model. The diameter of ball A and ball B of the stitched model can be calculated and be compared with the standard value of the artifact to get the diameter errors. The point cloud registration process is shown in Figure 9.

Figure 9.

Point cloud registration process.

The artifact is placed at 20 different positions in the robot workspace. The diameter errors of ball A and ball B are shown in Figure 10. The experimental results of point cloud registration are summarized in Table 2 and further demonstrate the superiority of the PM.

Figure 10.

Diameter error of stitched point cloud at different positions of the artifact: (a) ball A and (b) ball B.

Table 2.

Experimental results of point cloud registration.

Method	Maximum diameter error of ball A (mm)	Mean diameter error of ball A (mm)	Maximum diameter error of ball B (mm)	Mean diameter error of ball B (mm)
MLS	0.1565	0.1078	0.1433	0.0932
CLS	0.1258	0.0793	0.1105	0.0771
PM	0.0899	0.0528	0.0788	0.0480

Conclusions

To guide the positioning of the machining robots, a 3D-vision sensor is integrated at the end effector of machining robots and form the robotic visual measuring system for positioning and measurement. We have proposed an accurate hand-eye calibration method to determine the transformation between the sensor and the robot end effector. A hand-eye calibration model is established in the sensor frame and is capable of providing the more accurate reference point for the solution of hand-eye transformation parameters. A nonlinear optimization algorithm has been proposed. It is implemented with the consideration of the effects of the robot pose error, where it should be necessary to consider in high precision robotic machining. To avoid the linear correlation problems, parameter analysis is performed and the rotation parameters of the robot pose having a significant impact on the solution accuracy are selected as additional parameters to be optimized. A constraint least square solution has been given and provides good initial values for optimization. The simulation results on noise level and data bulk demonstrate that the PM has higher accuracy in solving hand-eye transformation matrix than MLS and CLS. Positioning and measurement experiments are designed and are carried out on a 5-axis hybrid robot named TriMule. The maximum/mean positioning errors have been reduced from 0.1133/0.0367 mm (MLS), 0.0961/0.0278 mm (CLS) to 0.0703/0.0244 mm (PM) in the positioning experiment, and the maximum diameter errors of ball A and ball B have been reduced from 0.1565/0.1433 mm (MLS), 0.1258/0.1105 mm (CLS) to 0.0899/0.0788 mm (PM) in the point cloud registration experiment. The experimental results further prove the superiority of the PM over MLS and CLS in terms of positioning accuracy and point cloud stitching accuracy. Additionally, the measurement accuracy of the 3D-vision sensor is also tested.

In the future work, we will fix the machining robot on a mobile platform and develop a mobile robotic machining system for large complex components. The calibration method will be investigated to ensure global positioning of the mobile robot.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work is supported by National Key Research and Development Program (Grant No. 2017YFE0111300), National Natural Science Foundation of China (Grant No. 51721003 and No. 51775376), and EU H2020 project (Grant No. 734272).

ORCID iD

Jinsheng Fu

References

Kim

Nam

, et al. Robotic machining: a review of recent progress. Int J Precis Eng Manuf 2019; 20: 1629–1642.

Savio

De Chiffre

Carmignato

, et al. Economic benefits of metrology in manufacturing. CIRP Ann-Manuf Techn 2016; 65(1): 495–498.

Moeller

Schmidt

Koch

, et al. Real time pose control of an industrial robotic system for machining of large scale components in aerospace industry using laser tracker system. SAE Int J Aerosp 2017; 10(2): 100–108.

Verl

Valente

Melkote

, et al. Robots in machining. CIRP Ann-Manuf Techn 2019; 68: 799–822.

Zhu

Guo

, et al. Calibration of a portable laser 3-D scanner used by a robot and its use in measurement. Opt Eng 2008; 47(1): 017202.

Liu

Yin

Guo

, et al. Rapid global calibration technology for hybrid visual inspection system. Sensors 2017; 17(6): 1440.

Zhang

. A flexible new technique for camera calibration. IEEE T Pattern Anal 2000; 22(11): 1330–1334.

Wang

Zhao

Yang

. Camera calibration method based on Pascal’s theorem. Int J Adv Robot Syst 2019; 16(3): 1–10.

Zhao

Liu

. A hand–eye calibration algorithm based on screw motions. Robotica 2009; 27: 217–223.

10.

Horaud

Dornaika

. Hand-eye calibration. Int J Robot Res 1995; 14(3): 195–210.

11.

Daniilidis

. Hand-eye calibration using dual quaternions. Int J Robot Res 1999; 18(3): 286–298.

12.

Shah

. Solving the robot-world/hand-eye calibration problem using the Kronecker product. J Mech Robot 2013; 5(3): 031007.

13.

Zhang

Gao

. Hand-eye calibration and grasping pose calculation with motion error compensation and vertical-component correction for 4-R (2-SS) parallel robot. Int J Adv Robot Syst 2020; 17(2): 1–14.

14.

Ren

Yin

Zhu

. Calibration technology in application of robot-laser scanning system. Opt Eng 2012; 51(11): 114204.

15.

Chen

Xiong

, et al. A noise-tolerant algorithm for robot-sensor calibration using a planar disk of arbitrary 3-D orientation. IEEE T Autom Sci Eng 2018; 15(1): 251–263.

16.

Tsai

Lenz

R K

. A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. IEEE T Robot Autom 1989; 5(3): 345–358.

17.

Wang

. Simultaneous robot-world and hand-eye calibration using dual-quaternions and Kronecker product. Int J Phys Sci 2010; 5(10): 1530–1536.

18.

Ulrich

Heider

Steger

Hand-eye calibration of SCARA robots. In: 9th Open German-Russian workshop on pattern recognition & image understanding, Koblenz, Germany, 1–5 December 2014.

19.

Bayrocorrochano

Daniilidis

Sommer

, et al. Motor algebra for 3D kinematics: the case of the hand-eye calibration. J Math Imaging Vis 2000; 13(2): 79–100.

20.

Shah

Eastman

Hong

. An overview of robotsensor calibration methods for evaluation of perception systems. In: Proceedings of the Workshop on Performance Metrics for Intelligent Systems (ed Madhavan

), PerMIS’12, College Park, MD USA, 20 March 2012, pp. 15–20. Maryland: ACM.

21.

Ulrich

Steger

. Hand-eye calibration of SCARA robots using dual quaternions. Pattern Recog Image Anal 2016; 26(1): 231–239.

22.

Zuang

Shiu

. A noise-tolerant algorithm for robotic hand-eye calibration with or without sensor orientation measurement. IEEE T Syst Man Cy 1993; 23(4): 1168–1175.

23.

Tabb

Yousef

. Solving the robot-world hand-eye(s) calibration problem with iterative methods. Mach Vision Appl 2017; 28(5-6): 569–590.

24.

Yin

Ren

Guo

, et al. Development and calibration of an integrated 3D scanning system for high-accuracy large-scale metrology. Measurement 2014; 54: 65–76.

25.

Xie

Zhang

, et al. Hand-eye calibration in visually-guided robot grinding. IEEE Trans Cybern 2016; 46(11): 2634–2642.

26.

Wang

You

. An overview of dynamic parameter identification of robots. Robot Cim-Int Manuf 2010; 26(5): 414–419.

27.

Gao

, et al. Mechatronics modeling and vibration analysis of a 2-DOF parallel manipulator in a 5-DOF hybrid machine tool. Mech Mach Theory 2018; 121: 430–445.

28.

Quo

. A quadratic constraint solution method for TDOA and FDOA localization. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (ed Tomer

), Prague, Czech Republic, 22–27 May 2011, pp. 2588–2591. IEEE.

29.

. The Levenberg-Marquardt algorithm, implementation and theory. In: Watson

(ed) Numerical analysis, lecture notes in mathematics, Berlin: Springer-Verlag. 1977.

30.

Golub

Loan

CFV

. Matrix computations. Baltimore: Johns Hopkins University Press, 1983.

31.

Murray

Sastry

. A mathematical introduction to robotic manipulation. Boca Raton: CRC press, 1994.

32.

Liu

Huang

. Inverse kinematics of a 5-axis hybrid robot with non-singular tool path generation. Robot Cim-Int Manuf, 2019; 56: 140–148.