Research on the principle and improvement method of hand-eye calibration based on limited-memory with bound constraints

Abstract

Technology of hand-eye calibration based on monocular vision is used widely in the field of industrial robots. For the eye-to-hand calibration, an improved nine-point method is proposed, in which a suction cup tool is used instead of a calibration plate to avoid the experimental error caused by the displacement of the calibration plate and end of robotic arm during the calibration process, so that the calibration can be completed quickly while the calibration accuracy is guaranteed, and the error can be controlled within ±0.04 mm. For the eye-in-hand calibration mode, an improved non-contact calibration method is proposed, which obtains the size and range of the initial values through the installation mode of the camera and the robotic arm, establishes the conversion equation from the pixel coordinates to the base coordinates of the robotic arm through these initial values, and then performs the hand-eye matrix solving through the L-BFGS-B (Limited-memory Broyden–Fletcher–Goldfarb–Shanno with Bound constraints) algorithm. After many experimental evidence, this method can control the calibration error within ±0.1 mm, and the calibration consumes time within 5 min, which effectively improves the calibration efficiency and calibration accuracy.

Keywords

label attachment hand-eye calibration limited-memory industrial robots

Introduction

Recently, the robot of industry based on monocular vision has been widely used in many fields, and together with monocular vision, industrial robots are able to quickly recognize and accurately grasp targets. However, before use, it is necessary to calibrate between industrial camera and industrial robot to establish the transformation relationship under the frame of axes of the industrial camera to the frame of axes of the industrial robot, and this calibration technique is hand-eye calibration.^1,2 Since the hand-eye calibration problem was proposed by Tscai and Shiu et al. in 1989, scholars at home and abroad have conducted a lot of research on it. In the current application field, according to the different calibration methods, hand-eye calibration can be categorized into eye-in-hand and eye-to-hand.³ And according to the different solution methods, it can be categorized into two-step method and one-step method. Among them, the one-step method usually provides higher accuracy, but the computational process is also more complicated and time-consuming. The two-step method is less accurate in some cases, but it is simpler and less computationally intensive and is better for industrial robots with known kinematic models.⁴

The nine-point calibration method is commonly used in industry,⁵ which requires shooting nine points, obtaining the pixel coordinates of the centers of the nine points, then controlling the end of the industrial robot to touch the centers of these nine points in turn, and recording the corresponding position of the end of the industrial robot, so as to establish the conversion matrix of the pixel coordinates and the coordinates of the industrial robot, and to complete the hand-eye calibration. However, this calibration method⁶ takes a long time, the calibration accuracy is too dependent on the operator’s experience, and during the calibration process, since the end of the industrial robot needs to be controlled to touch the centers of the nine points sequentially, this contact calibration will easily lead to the movement of the calibration plate due to improper operation, which will lead to the introduction of errors.

To solve above problems, the thesis proposes a simple and fast non-contact calibration method that utilizes the mounting constraints of the industrial camera and the industrial robot as well as the shooting height, which can be measured to obtain initial values and a given range of initial values. These initial values include the three Euler angles of the camera system of coordinate relative to frame of axes of the industrial robot and the translations of the X-axis and the Y-axis, the height from the end of the industrial robot to the calibration plate, the height from the optical centers of the camera to the calibration plate, and the conversion matrix from the pixel to the base system of coordinate is established, which is solved to obtain the final value and the conversion matrix by iterating using the L-BFGS-B⁷ algorithm. Experiments have proved that this method has the advantages of simple operation steps, short calibration time, high accuracy, and robustness of the algorithm, etc. Through the obtained hand-eye calibration matrix, the industrial robot successfully grasps the target, so this method can be applied in practical operation.

Experimental procedure

The equipment used in this experiment is as follows: Mitsubishi’s RV-4FRL series of six degrees of freedom industrial robots, Hikvision’s 1200w pixel industrial camera, HALCON’s HC-75 specification of the circular asymmetric calibration plate, a white light source, and a host computer, as shown in Figure 1.

Figure 1.

Setting up the experimental environment.

This experiment adopts the eye-in-hand mode, and the experimental process is shown in Figure 2; first of all, the construction of the experimental environment, the industrial camera is installed in the end of the industrial robot, the light source is installed in the lens of the industrial camera, adjust the focal length of the industrial camera until it can take a clear photo of the calibration plate, and fix the camera aperture and the focal length to remain unchanged. When using the industrial camera, it is necessary to calibrate the industrial camera, the host computer connected to the industrial camera through the network port, shoot 9-16 calibration plate photos,^8–10 through the HALCON software to calibrate the camera, and get the camera internal reference and distortion coefficient. Then the hand-eye calibration is carried out, the host computer is connected to the industrial robot and the camera through the network port, the industrial robot is controlled to move and take photos of the calibration plate, one photo corresponds to a set of robotic arm end position, and finally the rotational translation matrix is obtained after the calculation, and the accuracy of this experiment and the time required for the experiment are verified.

Figure 2.

Experimental procedure.

Hand-eye calibration

Camera calibration model

Before hand eye calibration, the camera needs calibration, and we need to obtain the camera’s internal parameter and distortion coefficient. The camera’s internal parameter is an intrinsic attribute of the camera, which is a parameter needed for subsequent calculations. Obtaining the camera’s aberration coefficient helps us to correct the aberration of the image, which facilitates the subsequent acquisition of more accurate pixel information.

For camera calibration, we first establish four systems of coordinates, It is the pixel coordinate, image coordinate, camera coordinate, and world coordinate, as shown in Figure 3.

Figure 3.

Camera model.

Assuming that the coordinates of the same point under the pixel system¹¹ of coordinate are $p (u, v)$ and the coordinates under the worldsystem of coordinate are $P_{W} (X_{W}, Y_{W}, Z_{W})$ , the conversion from pixel coordinates to world coordinates is as shown in (1):

s = [\begin{array}{l} u \\ v \\ 1 \end{array}] = [\begin{array}{l} f_{x} & 0 & u_{0} \\ 0 & f_{y} & v_{0} \\ 0 & 0 & 1 \end{array}] [\begin{array}{l} R & t \\ 0 & 1 \end{array}] [\begin{array}{l} X_{W} \\ Y_{W} \\ Z_{W} \\ 1 \end{array}] = M_{1} M_{2} [\begin{array}{l} X_{W} \\ Y_{W} \\ Z_{W} \\ 1 \end{array}]

(1)

where s is the scaling factor, which can be obtained from the depth information, M1 is the inner parameter, where

f x = f / d x

and

f y = f / d y

, f is the focal length of the camera, denote the pixel size of a single pixel in the X- axis and Y-axis directions, respectively, in the pixel system of coordinate, u₀ and v₀ are the coordinates of the center point of the image system of coordinate under the pixel system of coordinate, and M₂ is the outer parameter, which consists of R and t, where R is the a 3x3 rotation matrix and t is a 3x1 translation matrix.

Aberrations occur during camera imaging. Aberrations are categorized into radial and tangential aberrations. Radial aberrations are caused by the shape of the lens itself, while tangential aberrations are caused by deviations in the mounting position of the lens and the photoreceptor, the CCD/CMOS,¹² during the assembly process of camera production. For radial aberration, it can be expressed by the Taylor series expansion, where x and yare the actual coordinates after aberration, x₁ and x₂ are the corrected coordinates, and r is the distance of the point from the imaging center, as in (2) and (3):

x_{1} = x (1 + k_{1} r^{2} + k_{2} r^{4} + k_{3} r^{6})

(2)

y_{1} = y (1 + k_{1} r^{2} + k_{2} r^{4} + k_{3} r^{6})

(3)

For industrial cameras, the distortion is very small, so the tangential distortion can be ignored, and the radial distortion coefficient k₃ can also be ignored; we only need to calculate the k₁ and k₂ parameters for the subsequent distortion correction.

In this thesis, the Zhang’s calibration method is used for the calibration of the camera: the camera is kept motionless, the checkerboard calibration plate is used for the calibration, and the calibration plate is used to take 16–25 photographs in different positions under the field of view of the camera. The calibration plate takes up one-third of the shooting field of view in a single shot, and the overall calibration plate photo^13,14 covers the entire shooting field of view of the camera. By establishing the correspondence of the corner points in the shot of the calibration plate photo, the parameter of camera and the coefficient distortion are calculated.

Eye-in-hand calibration model

Hand-eye calibrations can be divided into eye-in-hand and eye-on-hand calibrations, which differ in the camera’s mounting position relative to the industrial robot. Eye-in-hand means that the camera is installed on the end of the robot of industrial, and the calibration is the matrix of transformation from the camera system of coordinate to the system of coordinate of the end of the industrial robot. Eyes-to-hand means that the camera is stationary, the focal length of its field of view is fixed, the imaging of a clear plane is the working plane of the industrial robot, and this time the calibration is the camera and the industrial robot base system of coordinate conversion matrix.

This experiment is carried out in an eye-in-hand manner, as shown in Figure 4.

Figure 4.

Eye-in-hand model.

Where the calibration plate pixel system of coordinate to the camera system of coordinate is converted to ${}_{C}^{O}T$ , the camerasystem of coordinate to the industrial robot endsystem of coordinate is converted to ${}_{E}^{C}T$ , the industrial robot end to the basesystem of coordinate is converted to ${}_{B}^{E}T$ , the calibration platesystem of coordinate to the base system of coordinate is converted to ${}_{B}^{O}T$ , and the hand-eye calibration is to obtain the conversion matrix of a process.

For a point on the calibration plate to the base system of coordinate, get the transformation relation, as in (4):

{}_{B}^{O}T = ({}_{C}^{O}T {}_{E}^{C}T {}_{B}^{E}T)

(4)

Control of industrial robots to drive the camera to move two times, you can take three pictures, which because the calibration process calibration plate is fixed relative to the base and the camera is fixed relative to the end of the fixed, so the ${}_{B}^{O}T$ and ${}_{E}^{C}T$ remain unchanged, the first time to move to get the conversion relation, as in (5):

{}_{B}^{O}T = ({}_{C 1}^{O}T {}_{E}^{C}T {}_{B}^{E 1}T)

(5)

Conversion relations obtained by the second move, as in (6):

{}_{B}^{O}T = ({}_{C 2}^{O}T {}_{E}^{C}T {}_{B}^{E 2}T)

(6)

From (5) and (6), we have (7):

({}_{C 1}^{O}T {}_{E}^{C}T {}_{B}^{E 1}T) = ({}_{C}^{O}T {}_{E}^{C}T {}_{B}^{E}T)

(7)

After transformation, the correspondence can be obtained as shown in (8):

{({}_{C 2}^{O}T)}^{- 1} ({}_{C 1}^{O}T {}_{E}^{C}T) = ({}_{E}^{C}T {}_{B}^{E 2}T) {({}_{B}^{E 1}T)}^{- 1}

(8)

where

A = {({}_{C 2}^{O}T)}^{- 1} {}_{C 1}^{O}T

B = {}_{B}^{E 2}T {({}_{B}^{E 1}T)}^{- 1}

X = {}_{E}^{C}T

, this is where the form of the solution

A X = X B

comes from, when n photographs are taken we get n-1 of these correspondence equations used to solve the X matrix.

In this thesis, a simple hand-eye calibration method is proposed; first of all, let the Z-axis of the end of the industrial robot be perpendicular to the horizontal work plane, the industrial camera is installed on the end of the industrial robot, the shooting angle is perpendicular to the work plane, keep the height Z_E of the end of the industrial robot unchanged, and then the height Z_C of the optical center of the industrial camera to the work plane is also unchanged. Ideally, of the three Euler angles of the industrial robot end system of coordinate and the camera system of coordinate, only the rotation angle $θ {}_{z}$ of the Z-axis is different, while the rotation angle $θ_{x} = 180 °$ of the X-axis and $θ_{y} = 0 °$ of the Y-axis. The horizontal X-axis and Y-axis translations from the camera to the end of the robotic arm are x and y, respectively, and the industrial camera has been calibrated at the time of use.

Then the high-precision calibration plate is placed on the calibration plane, the thickness of the calibration plate needs to be taken into account, try to keep the calibration plane and the horizontal working plane at the same height, and the positional relationship between the calibration plate and the base of the industrial robot stays unchanged during the calibration process.

Finally, the end of the industrial robot is controlled to drive the camera to take photos in several different positions by translating and rotating only around the Z-axis and recording the position of the end of the industrial robot corresponding to each photo, with a minimum of nine photos taken. The approximate initial values and initial value ranges can be obtained in advance by utilizing the constraints of the installation of the industrial camera and the industrial robot, the shooting height, and the height of the end to the calibration plate. These initial values are the three rotation angles of the industrial camera and the industrial robot end, the X-axis and Y-axis offsets of the industrial camera and the end, the height of the end to the calibration plate, and the height of the camera’s optical center to the calibration plate.

The principle of the L-BFGS-B algorithm is as follows.

Step 1: Initialization: choose the initial variable vector x as the starting point of the optimization problem and also set the initial step size. Calculate the gradient value of the objective function $\nabla f (x)$ .

Step 2: During each iteration, first construct an approximate inverse of the Hessian matrix using the history information, which is estimated by storing the last few iteration points and the gradient difference.

Step 3: Direction search: based on the constructed inverse of the approximate Hessian matrix and the gradient information, the search direction $d_{k}$ is determined. This can be obtained by multiplying the inverse of the current approximate Hessian matrix with the negative gradient, that is, $d_{k} = - B_{k} \nabla f (x)$ , where Bk is the inverse of the approximate Hessian matrix.

Step 4: Step control: the update step $α_{k}$ is determined by line search or other step control techniques to ensure that sufficient progress can be made in the search direction $d_{k}$ .

Step 5: Update the variables $x_{k} + 1 = x_{k} + α_{k} d_{k}$ according to the step size and search direction. Compute the gradient $\nabla f (x_{k + 1})$ of the objective function at the new point.

Step 6: Termination criterion: check the termination criterion such as gradient convergence and change in objective function value. If the termination criterion is satisfied, stop the iteration; otherwise return to step 2.

The L-BFGS-B algorithm handles bounded constraint problems by restricting the boundaries of the variables, which can be simple upper and lower boundaries, and the algorithm is corrected accordingly for violations of the constraints to ensure that the variables are within the constraints.

Experimental results and accuracy verification

The camera internal reference M₁ and the rotational translation matrix X that can be obtained by camera calibration and hand-eye calibration, respectively, M₁ are as shown in (9):

M_{1} = [\begin{array}{l} 4786.74 & 0 & 2041.21 \\ 0 & 4786.61 & 1518.04 \\ 0 & 0 & 1 \end{array}]

(9)

and X as shown in (10)

X = [\begin{array}{l} - 0.5566765 & - 0.8307294 & 0.720027 \\ 0.8372936 & - 0.5566765 & 0.402426 \end{array}]

(10)

Once the rotational translation matrix and initial values are obtained, the pixel coordinates can be converted to the industrial robot base system of coordinate using the previously established coordinate transformation relationship. Ideally, the position in the base system of coordinate obtained at this point is the end position of the industrial robot when the end moves to that point.

The nine-point calibration method is commonly used in industry to calibrate the industrial robot so that the pixel points on the two-dimensional plane correspond to the position of the industrial robot at the time of acquisition. This method depends on the operator’s experience in the operation process; after taking a photo containing nine dots, we need to take the pixel coordinates of the centers of the nine dots and control the end of the industrial robot to touch the centers of these nine dots in order to obtain the corresponding position of the end of the industrial robot, which is an exceptionally time-consuming step, and there are errors in the human eye, operational errors, etc., and the solution is the relationship between pixel dots and the position of the industrial robot, which is an affine transformation between pixel dots and robot position.

The principle of this experiment is to establish and solve the transformation relationship between the coordinates, first from the pixel coordinates to the camera system of coordinate, then from the camera system of coordinate to the industrial robot end system of coordinate, and finally from the industrial robot end system of coordinate to the industrial robot base system of coordinate. This experiment is simple, in the case of known industrial camera internal reference and distortion coefficients, only need to control the end of the industrial robot to drive the camera, at the same height, with different X-axis and Y-axis translation positions and different Z-axis rotational positions, to take at least nine photos, to the same point in these photos to establish the pixel system of coordinate to the base of the industrial robot system of coordinate conversion relationship. The missing Z-axis deep information can be obtained from the end of the robot to the horizontal work plane demonstration, bring in the initial parameters, and solve to get the final required rotational translation matrix.

To verify the accuracy of hand-eye calibration, the method is to compare the position of the same pixel point calculated by the obtained rotational translation matrix with the actual position of the end of the industrial robot touching the point, the difference between the calculated position and the actual position in the X-axis and Y-axis, respectively, for comparison, and the smaller the absolute value of the difference is, the higher the accuracy of the calibration, and the results of the accuracy verification are relatively intuitive. As shown in Table 1, the actual and calculated coordinates under the base system of coordinate correspond to the 10 pixel coordinates.

Table 1.

Actual and calculated coordinates corresponding to pixel coordinates.

	Pixel coordinates	Actual coordinates (mm)	Calculate coordinates (mm)
1	1411.34, 1817.47	432.89, −81.12	432.80, −81.17
2	1738.65, 1599.13	441.51, −96.18	441.47, −96.21
3	1978.57, 1780.16	432.69, −106.23	432.75, −106.30
4	1765.09, 1992.64	424.35, −96.28	424.39, −96.34
5	1489.48, 2108.23	419.86, −84.51	419.78, −84.47
6	1008.10, 2429.45	406.72, −62.16	406.69, −61.10
7	2582.81, 2325.33	407.15, −131.37	407.08, −131.45
8	2169.62, 2062.79	420.45, −113.38	419.43, −113.44
9	1952.48, 1387.30	450.36, −106.07	450.41, −106.15
10	1157.26, 1441.73	449.78, −71.05	449.74, −71.09

From the absolute value of the error between the calculated coordinates and the actual coordinates in Table 1, we can get the content of Table 2, from which we can see that the maximum error in the X-axis direction is 0.09 mm and the average error is 0.052 mm. The maximum error in the Y-axis direction is 0.08 mm, and the average error is 0.059 mm. The Z-axis error is not shown here because the depth information in the Z-axis direction can be obtained by the industrial robot demonstration teaching. Through several sets of experiments, the maximum error of each experiment is no more than 0.1 mm, and the average and maximum errors are stable, which proves that the calibration method is effective and feasible.

Table 2.

Average and maximum error of X-axis and Y-axis.

	Maximum error (mm)	Average error (mm)
X-axis	0.09	0.052
Y-axis	0.08	0.059

As shown in Figure 5, the comparison of absolute value of error in X-axis and absolute value of error in Y-axis of the data obtained from the nine-point method and the present experimental method, respectively, where x₁ and y₁ denotes the present experimental method and x₂ and y₂ denotes the nine-point method. The maximum error of the nine-point calibration method in the X-axis is 0.14 mm and the maximum error in the Y-axis is 0.12 mm. After many experiments, the average error of the nine-point calibration method is more than 0.1 mm, which shows that the experimental method has a significant improvement on the calibration accuracy. Nine-point calibration method in the calibration needs to control the end of the industrial robot to touch the center of the circle of nine points in turn and in this process will introduce the human eye error, a little inadvertently touching the calibration board caused by the calibration board to move, and then need to restart the operation. Under normal operation, after many experiments, the nine-point calibration method takes less than 5–10 minutes to collect photos and obtain the position of the end of the industrial robot corresponding to the nine points, whereas the experimental method only requires the control of the robotic arm to simply move to take photos and record the corresponding position, and the time required is less than 5 minutes, which is a significant improvement in terms of time.

Figure 5.

Comparison of absolute X-axis error and Y-axis.

The nine-point calibration method is an improved non-contact calibration method, which only requires controlling the robotic arm to drive the camera to take photos of the calibration board in a specific pose, without the need to control the end of the robotic arm to touch the calibration board. After multiple experimental verifications, the calibration method proposed in this article can control the calibration error within ±0.1 mm and the calibration time within 5 minutes. Compared with traditional calibration methods, this method is simpler to operate, significantly reduces calibration time, and has higher calibration accuracy.

Conclusion

In this thesis, a simple calibration method is proposed, comparing with the nine-point calibration method commonly used in the industrial field, the method is simple in operation, the required calibration time is short, the accuracy is high, all the parameters in the calibration model can be obtained at the same time, and the disadvantage is that at present it is only adapted to the planar range and more accurate depth information can be obtained using binocular cameras in the future. By building the experimental environment and system, the results of the experiment are analyzed, and the accuracy of the calibration results is verified, the calibration error is within 0.1 mm, the calibration accuracy is significantly improved, the calibration data acquisition time is within 5 min, and the calibration time required is significantly improved, which meets the requirements of the task of industrial robots to carry out the operation on the industrial site, and it has certain research significance.

Statements and declarations

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was support by Key Projects of Overseas Visiting for Outstanding Young Backbone Talents in Colleges and Universities with no. (gxfxzd2016223) and support by Anhui major science and technology special projects (no. 103016385021) and by Anhui Province Science and Technology Innovation Challenge Project (no. 202407180313).

ORCID iD

Yuhao Cao

References

Ding

Huang

, et al. Hand-eye calibration method based on three-dimensional visual measurement in robotic high-precision machining. Int J Adv Manuf Technol 2022; 119(5-6): 3845–3856.

Fangxun

Zerui

Wei

, et al. Hand-eye calibration of surgical instrument for robotic surgery using interactive manipulation. IEEE Rob Autom Lett 2020; 5(2): 1540–1547.

Boby

. Kinematic identification of industrial robot using end-effector mounted monocular camera bypassing measurement of 3-D pose. IEEE ASME Trans Mechatron 2022; 27(1): 383–394.

Zhang

J-Z

Jiang

, et al. Error correctable hand–eye calibration for stripe-laser vision-guided robotics. IEEE Trans Instrum Meas 2020; 69(10): 8314–8327.

Sarabandi

Porta

J-M

Thomas

. Hand-eye calibration made easy through a closed-form two-stage method. IEEE Robot Autom Lett, 2022; 7(2): 3679–3686.

Wang

Fan

Jing

, et al. An efficient calibration method of line structured light vision sensor in robotic eye-in-hand system. IEEE Sens J 2020; 20(11): 6200–6208.

Qin

BWT

. A novel hand-eye calibration method for industrial robot and line laser vision sensor. Sens Rev 2023; 43(4): 259–265.

Zhongming

Chen

Liqun

, et al. A regularization-patching dual quaternion optimization method for solving the hand-eye calibration problem. J Optim Theor Appl 2024; 200(3): 1193–1215.

Haihua

Ruichao

Zhou

, et al. A novel flexible two-step method for eye-to-hand calibration for robot assembly system. Measurement and Control 2020; 53(9-10): 2020–2029.

10.

Peters

Knoll

. Robot self-calibration using actuated 3D sensors. J Field Robot 2023; 41(2): 327–346.

11.

Sun

Wang

, et al. Hand-eye calibration: 4-D procrustes analysis approach. IEEE Trans Instrum Meas 2020; 69(6): 2966–2981.

12.

Pan

Chen

, et al. A dual quaternion-based approach for coordinate calibration of dual robots in collaborative motion. IEEE Robot Autom Lett 2020; 5(3): 4086–4093.

13.

Yan

, et al. An efficient extended target detection method based on region growing and contour tracking algorithm. Proc Inst Mech Eng G J Aerosp Eng 2018; 232(5): 825–836.

14.

Bengtson

Bak

Andreasen Struijk

LNS

, et al. A review of computer vision for semi-autonomous control of assistive robotic manipulators (ARMs). Disabil Rehabil Assist Technol 2020; 15(7): 731–745.