Hand–eye calibration and grasping pose calculation with motion error compensation and vertical-component correction for 4-R(2-SS) parallel robot

Abstract

Due to motion constraint of 4-R(2-SS) parallel robot, it is difficult to calculate the translation component of hand–eye calibration based on the existing model solving method accurately. Additionally, the camera calibration error, robot motion error, and invalid calibration motion poses make it difficult to achieve fast and accurate online hand–eye calibration. Therefore, we propose a hand–eye calibration method with motion error compensation and vertical-component correction for 4-R(2-SS) parallel robot by improving the existing eye-to-hand model and solving method. Firstly, the eye-to-hand model of single camera is improved and the robot motion error in the improved model is compensated to reduce the influence of camera calibration error and robot motion error on model accuracy. Secondly, the vertical-component of hand–eye calibration is corrected based on vertical constraint between calibration plate and end effector in parallel robot to calculate the pose and motion error in calibration of 4-R(2-SS) parallel robot accurately. Thirdly, the nontrivial solution constraint of eye-to-hand model is constructed and adopted to remove invalid calibration motion poses and plan calibration motion. Finally, the proposed method was verified by experiments with a fruit sorting system based on 4-R(2-SS) parallel robot. Compared with random motion, the existing model, and solving method, the average time of online calibration based on planned motion decreases by 29.773 s and the average error of calibration based on the improved model and solving method decreases by 151.293. The proposed method can improve the accuracy and efficiency of hand–eye calibration of 4-R(2-SS) parallel robot effectively and further realize accurate and fast grasping.

Keywords

Eye-to-hand hand–eye calibration motion planning parallel robot stereovision grasping

Introduction

The automatic sorting of fruits based on robot is of great significance to the automated, large-scale, and accuracy development of agricultural production and agriculture product processing.^1
–3 During the automatic sorting of fruits, the accurate and reliable calculation of grasping pose is the precondition to realize the accurate, fast, and nondestructive grasping control of robot.⁴ Currently, the main methods of grasping pose calculation include infrared image analysis, spectrum analysis, and machine vision. Compared with other methods, the machine vision has the advantages of noncontact, good adaptability, cost-effectiveness, and so on. It is more suitable for grasping pose calculation of fruits.⁵ However, there is a challenging problem in grasping pose calculation based on machine vision, which is online hand–eye calibration with high accuracy and efficiency.^6
–8 In addition, the parallel robot, which features strong rigidity, stable structure, and high precision, places greater demands on accuracy and efficiency of online hand–eye calibration.^9,10 Currently, the main hand–eye systems include eye-to-hand and eye-in-hand according to the pose relationship between camera and end effector of robot. The camera of eye-to-hand system is fixed outside the robot.¹¹ The camera pose in the world coordinate system remains unchanged. The pose relationship between camera and end effector changes with the robot motion.¹² The camera of eye-in-hand system is fixed on the end effector and moves with the robot.^13,14 The pose relationship between camera and end effector remains unchanged. The eye-in-hand system can move the camera close to the object to acquire a clear image, but it is difficult to ensure that the object appears in the field of view of camera. In addition, the shake of camera while moving and the image smear caused by acceleration while acquiring image would affect the accuracy of calibration. The eye-to-hand system with stationary camera has the advantages of high detecting accuracy and good stability, which is more suitable for the fruit sorting system of parallel robot with limited work space.¹⁵

The main methods of hand–eye calibration include measuring method, active motion calibration method, self-calibration method, and calibration method based on calibration object. Because the measuring method needs the aid of measuring equipment with high accuracy, such as an optical coordinate measurement machine,¹⁶ the cost is high and the process is complicated. The active motion calibration method has strict requirement for the motion accuracy of vision system. It is not suitable for eye-to-hand system with stationary camera.¹⁷ In addition, it is difficult to realize the active motion calibration because the robot motion control and trajectory calculation are difficult to achieve accurately and efficiently.^18,19 The self-calibration method only uses the camera parameters and the constraint of robot to realize hand–eye calibration, which makes the calculation complex and the accuracy difficult to improve.²⁰ The calibration method based on calibration object establishes the relationship between camera and robot by calibration object to reduce the difficulty of pose calculation. It can improve the accuracy and speed of hand–eye calibration effectively.^21,22 So this article uses the calibration method based on calibration object to calibrate the eye-to-hand system.

Currently, the solving methods for hand–eye model mainly include homogeneous transformation method,^23,24 matrix rearrangement method,²⁵ method based on matrix vectorization and Kronecker product,²⁶ dual quaternion method,²⁷ and Lie group method.²⁸ The rotation matrix is expressed by rotating an angle around an axis in homogeneous transformation method, which is obtained by calculating the unit vector and rotation angle of rotation axis. In the matrix rearrangement method, the rotation matrix is obtained by rearranging matrix elements to transform implicit equation of rotation into explicit equation in hand–eye model. However, the obtained rotation matrix is not orthogonal matrix. Solving the translation vector based on nonorthogonal rotation matrix would lead to greater translation error. The dual quaternion method uses the dual quaternion to represent the motion in space. The Lie group method adopts the Lie group theory to transform the minimization problem into the least square fit problem in solving equations based on multiple sets of observations for obtaining a simple and clear solution. The method based on matrix vectorization and Kronecker product transforms the hand–eye model equations into explicit equations that are easy to solve. Compared with other methods, the method based on matrix vectorization and Kronecker product can calculate the rotation matrix and translation vector in hand–eye model simultaneously, which can reduce the transfer error. It has the advantages of good robustness, high accuracy, and calculation speed. It is more suitable for the online hand–eye calibration and grasping pose calculation of 4-R(2-SS) parallel robot based on machine vision.

The existing solving methods for hand–eye model require the robot to perform multidirectional rotation and large-scale translation during calibration. For robots with degree of freedom (DOF) constraint, it is difficult to meet this requirement. Therefore, some scholars proposed a calibration method for the robots with DOF constraint by limiting the rotation angle of camera to nonzero during calibration. However, for the robots with rotational constraint, it is difficult to calculate the Z-direction translation component accurately.²⁹ Some scholars used the three orthogonal translations of robot to linearize rotation component for reaching the solution conditions, but the translation error would also affect the accuracy of model solving.³⁰ Some scholars constructed constraint matrices to achieve hand–eye calibration of 4-DOF and 6-DOF robots based on rotational and translational constraint information, but the Z-direction translation component has not yet been taken into account.³¹ Then some scholars researched the robots with rotational constraint, such as Cartesian robot and Selective Compliance Assembly Robot Arm robot. The hand–eye calibration model of robot is solved by setting the camera as an orthogonal projection pose and reducing the number of unknown parameters of hand–eye calibration matrix based on a two-dimensional hand–eye calibration model. However, the calculation of Z-direction translation component needs to be corrected based on additional reference object. The accuracy would be affected by the thickness and machining error of reference object.³²

Therefore, for the 4-R(2-SS) parallel robot with 4-DOF of rotational constraint, it is difficult to calculate eye-to-hand model parameters based on existing hand–eye calibration methods accurately. Additionally, the motion error of robot, calibration error of camera, and invalid calibration motion poses of robot make it difficult to achieve fast and accurate online hand–eye calibration and grasping pose calculation of 4-R(2-SS) parallel robot.³³ In this article, an online hand–eye calibration method with motion error compensation and vertical-component correction for 4-R(2-SS) parallel robot is proposed and applied to grasping pose calculation. An improved eye-to-hand model of stereovision with motion error compensation is proposed to reduce the influence of camera calibration error and robot motion error on the accuracy of eye-to-hand model. The existing model solving method based on matrix vectorization and Kronecker product is improved to calculate the pose and motion error in hand–eye calibration of 4-R(2-SS) parallel robot with rotational constraint accurately. A calibration motion planning method based on nontrivial solution constraint of eye-to-hand model is proposed for improving the accuracy and efficiency of online hand–eye calibration and further realizing accurate and fast grasping of parallel robot.

Materials and methods

Due to motion constraint of 4-R(2-SS) parallel robot, it is difficult to calculate translation component of hand–eye calibration based on the existing model solving method accurately. Additionally, the camera calibration error, robot motion error, and invalid calibration motion poses make it difficult to achieve fast and accurate online hand–eye calibration. Therefore, we propose a hand–eye calibration method with motion error compensation and vertical-component correction for 4-R(2-SS) parallel robot by improving the existing eye-to-hand model and solving method.

Model solving method based on matrix vectorization and Kronecker product and limitation analysis

The model solving method based on matrix vectorization and Kronecker product transforms the hand–eye model equations into explicit equations. For calculating the rotation matrix and translation vector in hand–eye model, it is necessary for the robot to move with some specific requirements.

Model solving method based on matrix vectorization and Kronecker product

To reduce the influence of measuring error of calibration plate pose ${}^{g}H_{b}$ in end-effector coordinate system on the solving accuracy of hand–eye calibration model, the hand–eye calibration model $AX = XB$ is constructed by eliminating ${}^{g}H_{b}$ . $A = {}^{d}H_{b (i + 1)} {}^{d}H_{b i}^{- 1}$ , $B = {}^{w}H_{g (i + 1)} {}^{w}H_{g i}^{- 1}$ , and $X = {}^{d}H_{w}$ . The ${}^{d}H_{b}$ represents the pose matrix of calibration plate in the basic coordinate system of camera. The ${}^{w}H_{g}$ represents the pose matrix of end effector in the basic coordinate system of robot. The ${}^{d}H_{w}$ represents the pose matrix of the basic coordinate system of robot in the basic coordinate system of camera. Therefore, the matrices A, B , and X can be decoupled into the rotation matrix and translation vector as follows:

A = [\begin{array}{l} R_{A} t_{A} \\ 0^{T} 1 \end{array}], B = [\begin{array}{l} R_{B} t_{B} \\ 0^{T} 1 \end{array}], and X = [\begin{array}{l} R t \\ 0^{T} 1 \end{array}]

The sizes of rotation matrices R , $R_{A}$ , and $R_{B}$ are $3 \times 3$ and the sizes of translation vectors $t_{A}$ , $t_{B}$ , and t are $3 \times 1$ . Taking X as an example, the rotation matrix and the translation vector are, respectively:

R = [\begin{array}{l} R_{11} R_{12} R_{13} \\ R_{21} R_{22} R_{23} \\ R_{31} R_{32} R_{33} \end{array}] a n d t = [\begin{array}{l} t_{x} \\ t_{y} \\ t_{z} \end{array}]

Therefore, the hand–eye model $AX = XB$ can be expressed as follows:

[\begin{array}{l} R_{A} t_{A} \\ 0^{T} 1 \end{array}] [\begin{array}{l} R t \\ 0^{T} 1 \end{array}] = [\begin{array}{l} R t \\ 0^{T} 1 \end{array}] [\begin{array}{l} R_{B} t_{B} \\ 0^{T} 1 \end{array}]

Then:

{\begin{cases} R_{A} R = R R_{B} \\ R_{A} t + t_{A} = R t_{B} + t \end{cases}

The rotation and translation parts of equation (4) are transformed respectively. The rotation matrices R , $R_{A}$ , and $R_{B}$ are the unit orthogonal matrix, so:

(I_{9} - R_{A} \otimes R_{B}) vec (R) = 0_{9}

(I_{3} \otimes t_{B}^{T}) vec (R) + (I_{3} - R_{A}) t = t_{A}

The $I_{9}$ and $I_{3}$ are ninth-order and third-order unit matrices, respectively. The $0_{9}$ is zero matrix with $9 \times 1$ . Based on the two motions of end effector of parallel robot during hand–eye calibration, we can obtain as follows:

[\begin{array}{l} I_{9} - R_{A 1} \otimes R_{B 1} \\ I_{9} - R_{A 2} \otimes R_{B 2} \end{array}] vec (R) = 0_{18}

[\begin{array}{l} I_{3} \otimes t_{B 1}^{T} \\ I_{3} \otimes t_{B 2}^{T} \end{array}] vec (R) + [\begin{array}{l} I_{3} - R_{A 1} \\ I_{3} - R_{A 2} \end{array}] t = [\begin{array}{l} t_{A 1} \\ t_{A 2} \end{array}]

The $R_{A 1}$ , $R_{B 1}$ , $R_{A 2}$ , and $R_{B 2}$ are rotation matrices of two motions, respectively. The $t_{A 1}$ , $t_{A 2}$ , $t_{B 1}$ , and $t_{B 2}$ are translation vectors of two motions, respectively. The $0_{18}$ is zero matrix with $18 \times 1$ . The equation (7) is transformed into $Q \cdot vec (R) = 0$ and solved by singular value decomposition. Then the results are introduced into equation (8) to calculate the rotation matrix R and translation vector t based on the least square method. Finally, based on equation (1), the pose matrix X of hand–eye model is solved.

Limitation analysis of model solving method based on matrix vectorization and Kronecker product for 4-R(2-SS) parallel robot

If the end effector does not rotate, in which $R_{A} = R_{B} = I$ , the translation part of the decoupled eye-to-hand model equation (4) may be transformed into:

t_{A} = R t_{B}

Then:

[\begin{array}{l} t_{A x} \\ t_{A y} \\ t_{A z} \end{array}] = [\begin{array}{l} R_{11} R_{12} R_{13} \\ R_{21} R_{22} R_{23} \\ R_{31} R_{32} R_{33} \end{array}] [\begin{array}{l} t_{B x} \\ t_{B y} \\ t_{B z} \end{array}]

Therefore, three equations can be obtained at each motion moment of end effector. For the nine unknowns of the rotation matrix R , it is necessary for the three translations without rotation of end effector, which are not in the same plane. Then the rotation matrix R of eye-to-hand model can be obtained. The calibration plate is fixed on the top of the end effector. The Z-axes of calibration plate coordinate system and end-effector coordinate system are collinear. The end effector of 4-R(2-SS) parallel robot with 4-DOF only can rotate around the Z-axis, so the calibration plate only can rotate around the Z-axis. Based on the motion constraint, the rotation matrix $R_{A}$ in the pose transformation matrix A is expressed as follows:

R_{A} = {Rot}_{A} (Z, θ_{A})

The $θ_{A}$ is angle rotating around the Z-axis. Equation (11) and obtained translation vectors $t_{A}$ and $t_{B}$ are introduced into equation (4):

{Rot}_{A} (Z, θ_{A}) t + t_{A} = R t_{B} + t

The $Rot (k, θ)$ represents a rotation around arbitrary axis $k$ at angle $θ$ . An arbitrary rotation matrix can be expressed by $Rot (k, θ)$ as follows:

\begin{array}{c} Rot (k, θ) = & [\begin{array}{l} k_{x} k_{x} vers θ + cos θ k_{y} k_{x} vers θ - k_{z} sin θ k_{z} k_{x} vers θ + k_{y} sin θ \\ k_{x} k_{y} vers θ + k_{z} sin θ k_{y} k_{y} vers θ + cos θ k_{z} k_{y} vers θ - k_{x} sin θ \\ k_{x} k_{z} vers θ - k_{y} sin θ k_{y} k_{z} vers θ + k_{x} sin θ k_{z} k_{z} vers θ + cos θ \end{array}] \end{array}

where $vers θ = (1 - cos θ)$ , $k = (k_{x}, k_{y}, k_{z})$ . Equation (12) is transformed based on equation (13):

[\begin{array}{c} cos θ_{A} - 1 & - sin θ_{A} & 0 \\ sin θ_{A} & cos θ_{A} - 1 & 0 \\ 0 & 0 & 0 \end{array}] [\begin{array}{l} t_{x} \\ t_{y} \\ t_{z} \end{array}] = [\begin{array}{l} R_{11} t_{B x} + R_{12} t_{B y} + R_{13} t_{B z} - t_{A x} \\ R_{21} t_{B x} + R_{22} t_{B y} + R_{23} t_{B z} - t_{A y} \\ R_{31} t_{B x} + R_{32} t_{B y} + R_{33} t_{B z} - t_{A z} \end{array}]

To make equation (14) have nontrivial solution, the matrix $R o t_{A} (Z, θ_{A}) - I$ would be not a zero matrix, so $θ_{A} \neq 2 k π$ . From equation (14), we can see that it is difficult to calculate the Z-direction translation component t_z based on the rotation around Z-axis only. So to obtain all translation components, it is necessary for the end effector to rotate at least twice around two nonparallel axes. However, the 4-R(2-SS) parallel robot with 4-DOF cannot meet the motion requirement. Therefore, in this article, the existing model solving method based on matrix vectorization and Kronecker product is improved to propose an improved solving method with vertical-component correction for the eye-to-hand model. Additionally, an improved eye-to-hand model of stereovision with motion error compensation is proposed. The calibration motion of parallel robot is planned based on the nontrivial solution constraint of eye-to-hand model to improve the accuracy and efficiency of hand–eye calibration of 4-R(2-SS) parallel robot.

Improved eye-to-hand model and model solving method

The eye-to-hand model of single camera is improved and the robot motion error in improved model is compensated to reduce the influence of camera calibration error and robot motion error on model accuracy. The vertical-component of hand–eye calibration is corrected based on vertical constraint between calibration plate and end effector in parallel robot to calculate the pose and the motion error in calibration of 4-R(2-SS) parallel robot accurately.

Improved eye-to-hand model of stereovision with motion error compensation

(1) Eye-to-hand model of stereovision. The existing eye-to-hand model is usually based on the single camera in the basic coordinate system of camera. It is difficult to improve the calibration accuracy. Therefore, this article constructs the hand–eye models of all cameras according to the pose relationship between cameras in stereovision. Based on the hand–eye model, the two cameras are modeled, respectively:

{\begin{cases} A_{d} X_{d} = X_{d} B \\ A_{c} X_{c} = X_{c} B \end{cases}

where $A_{d} = {}^{d}H_{b (i + 1)} {}^{d}H_{b i}^{- 1}$ and $A_{c} = {}^{c}H_{b (i + 1)} {}^{c}H_{b i}^{- 1}$ are the pose transformation matrices of calibration plate in the two camera coordinate systems, respectively. The $X_{d} = {}^{d}H_{w}$ and $X_{c} = {}^{c}H_{w}$ are the pose matrices of the basic coordinate system of robot in the two camera coordinate systems, respectively. The eye-to-hand model of stereovision can be obtained by transforming equation (15) based on pose relationship between cameras, as shown in equation (16).

{\begin{cases} A_{d} X_{d} = X_{d} B \\ A_{c} ({}^{c}H_{d} X_{d}) = ({}^{c}H_{d} X_{d}) B \end{cases}

(2) Eye-to-hand model with motion error compensation. The hand–eye calibration error mainly comes from the pose error of end effector in the basic coordinate system of robot and the pose error of calibration plate in the basic coordinate system of camera, which is derived from the motion error of robot. The motion error of robot can be considered to be caused by differential transformation of robot coordinate system, which can be calculated by differential motion model of robot. The differential motion of robot can be considered to consist of differential rotation $R (δ_{x}, δ_{y}, δ_{z})$ and differential translation $T (d x, d y, d z)$ . Taking into account the differential motion $d H$ in the motion of end effector, the new pose matrix $H + d H$ can be obtained, where the $d H$ can be calculated based on equation (17).

d H = Δ \cdot H = [\begin{array}{l} 0 - δ_{z} δ_{y} d x \\ δ_{z} 0 - δ_{x} d y \\ - δ_{y} δ_{x} 0 d z \\ 0 0 0 1 \end{array}] \cdot H

Although the calibration plate is fixed on the top of the end effector and the motion of calibration plate is consistent with that of the end effector in the eye-to-hand system of parallel robot, the pose matrix and differential motion matrix of calibration plate and end effector are different in the different coordinate systems. Therefore, the hand–eye model $AX = XB$ can be improved by motion error compensation based on differential motion, as shown in equation (18). The $Δ A$ is the differential motion of calibration plate in the camera coordinate system. The $Δ B$ is the differential motion of end effector in the basic coordinate system of robot.

{\begin{cases} (A_{d} + d A_{d}) X_{d} = X_{d} (B + d B) \\ (A_{c} + d A_{c}) ({}^{c}H_{d} X_{d}) = ({}^{c}H_{d} X_{d}) (B + d B) \end{cases}

Improved solving method with vertical-component correction for the eye-to-hand model

Because of the motion constraint of the 4-R(2-SS) parallel robot with 4-DOF, it is difficult to calculate the Z-direction translation component t_z of pose matrix X according to the existing model solving method based on matrix vectorization and Kronecker product. Some scholars used $t_{z} = 0$ to calculate. Although it can meet the calibration requirement, the low calibration accuracy makes it difficult to calculate grasping pose based on the hand–eye calibration result accurately. Therefore, considering the structural stability of the 4-R(2-SS) parallel robot with 4-DOF, this article proposes a method of vertical-component correction to improve the existing model solving method based on matrix vectorization and Kronecker product. It can solve the eye-to-hand model of 4-R(2-SS) parallel robot with 4-DOF accurately.

Due to the structural stability and motion constraint of the 4-R(2-SS) parallel robot with 4-DOF, the Z-axes of the end-effector coordinate system $O_{g} - X_{g} Y_{g} Z_{g}$ and the basic coordinate system of parallel robot $O_{w} - X_{w} Y_{w} Z_{w}$ can be kept parallel, as shown in Figure 1. Because the calibration plate is fixed on the top of the end effector, the Z-axes of the $O_{g} - X_{g} Y_{g} Z_{g}$ and the calibration plate coordinate system $O_{b} - X_{b} Y_{b} Z_{b}$ can be kept collinear. The Z-direction translation component t_z of pose matrix X can be calculated by transformation relationship between other coordinate systems. The pose matrix ${}^{d}H_{w}$ of the $O_{w} - X_{w} Y_{w} Z_{w}$ in the basic coordinate system of camera $O_{d} - X_{d} Y_{d} Z_{d}$ obtained by the existing model solving method based on matrix vectorization and Kronecker product is:

Figure 1.

Coordinate diagram in eye-to-hand system of 4-R(2-SS) parallel robot.

X = {}^{d}H_{w} = [\begin{array}{l} R t \\ 0^{T} 1 \end{array}] = [\begin{array}{l} R_{11} R_{12} R_{13} t_{x} \\ R_{21} R_{22} R_{23} t_{y} \\ R_{31} R_{32} R_{33} t_{z} \\ 0 0 0 1 \end{array}]

According to the orthogonality of rotation matrix in the pose matrix ${}^{d}H_{w}$ , the inverse matrix ${}^{w}H_{d}$ of ${}^{d}H_{w}$ can be obtained as follows:

{}^{w}H_{d} = [\begin{array}{l} R^{T} - R^{T} t \\ 0^{T} 1 \end{array}] = [\begin{array}{l} R_{11} R_{21} R_{31} - R_{11} t_{x} - R_{21} t_{y} - R_{31} t_{z} \\ R_{12} R_{22} R_{32} - R_{12} t_{x} - R_{22} t_{y} - R_{32} t_{z} \\ R_{13} R_{23} R_{33} - R_{13} t_{x} - R_{23} t_{y} - R_{33} t_{z} \\ 0 0 0 1 \end{array}]

The Z-coordinate ${}^{w}z_{d}$ of O_d in the $O_{w} - X_{w} Y_{w} Z_{w}$ can be calculated according to equation (21).

{}^{w}z_{d} = {}^{w}z_{g} - {}^{b}z_{d} - {}^{b}z_{g}

The vertical component ${}^{w}z_{g}$ of end-effector pose in the $O_{w} - X_{w} Y_{w} Z_{w}$ can be calculated according to the pose matrix ${}^{w}H_{g}$ obtained by forward kinematics of 4-R(2-SS) parallel robot. The vertical component ${}^{b}z_{g}$ of end-effector pose in the $O_{b} - X_{b} Y_{b} Z_{b}$ can be obtained by measuring. The vertical component ${}^{b}z_{d}$ of the infrared camera pose in the $O_{b} - X_{b} Y_{b} Z_{b}$ can be calculated according to the ${}^{d}H_{b}$ obtained by vision calibration as follows:

\begin{array}{l} {}^{b}H_{d} = [\begin{array}{l} {}^{d}R_{b}^{T} − {}^{d}R_{b}^{T} {}^{d}t_{b} \\ 0^{T} 1 \end{array}] \\ = [\begin{array}{l} {}^{d}R_{b 11} {}^{d}R_{b 21} {}^{d}R_{b 31} - {}^{d}R_{b 11} {}^{d}t_{b x} - {}^{d}R_{b 21} {}^{d}t_{b y} - {}^{d}R_{b 31} {}^{d}t_{b z} \\ {}^{d}R_{b 12} {}^{d}R_{b 22} {}^{d}R_{b 32} - {}^{d}R_{b 12} {}^{d}t_{b x} - {}^{d}R_{b 22} {}^{d}t_{b y} - {}^{d}R_{b 32} {}^{d}t_{b z} \\ {}^{d}R_{b 13} {}^{d}R_{b 23} {}^{d}R_{b 33} - {}^{d}R_{b 13} {}^{d}t_{b x} - {}^{d}R_{b 23} {}^{d}t_{b y} - {}^{d}R_{b 33} {}^{d}t_{b z} \\ 0 0 0 1 \end{array}] \end{array}

{}^{b}z_{d} = - {}^{d}R_{b 13} {}^{d}t_{b x} - {}^{d}R_{b 23} {}^{d}t_{b y} - {}^{d}R_{b 33} {}^{d}t_{b z}

Then, according to equations (20) and (21), the Z-direction translation component t_z of pose matrix X can be obtained as follows:

t_{z} = \frac{- R_{13} t_{x} - R_{23} t_{y} - {}^{w}z_{g} + {}^{b}z_{d} + {}^{b}z_{g}}{R_{33}}

Thus, the vertical component of pose matrix X obtained by the existing model solving method based on matrix vectorization and Kronecker product is corrected by calculated t_z to realize the accurate solving of eye-to-hand model for 4-R(2-SS) parallel robot with 4-DOF.

Hand–eye calibration and grasping pose calculation based on improved eye-to-hand model and model solving method for 4-R(2-SS) parallel robot

The nontrivial solution constraint of eye-to-hand model is constructed and adopted to remove invalid calibration motion poses and plan calibration motion. The pose transformation matrix of end effector from current pose to optimal grasping pose in $O_{w} - X_{w} Y_{w} Z_{w}$ is calculated. The improved eye-to-hand model and model solving method for 4-R(2-SS) parallel robot are used for hand–eye calibration and grasping pose calculation.

Fruit sorting system based on 4-R(2-SS) parallel robot

The 4-R(2-SS) parallel robot with 4-DOF for automatic sorting of fruits is shown in Figure 2. The parallel robot body includes parallel mechanism consisting of 4-R(2-SS) (R represents the revolute joint and S represents the spherical joint) side chains with the same kinematic structure and end effector which is a clamping mechanism. It can realize three-dimensional translation and one-dimensional rotation around Z-axis. According to the requirement of fruit sorting, the eye-to-hand system is adopted, in which the camera is fixed outside the robot and does not move with the end effector. The end -effector of robot moves in the field of view of camera. To obtain the three-dimensional pose of fruit, the Kinect camera based on time of flight measuring method is used for acquiring images.³⁴ The Kinect camera is composed of a color camera and an infrared camera. To achieve online hand–eye calibration, the calibration plate is fixed on the top of the end effector. The parallel robot grasps the fruit based on the fruit pose obtained from the image during fruit sorting. Therefore, it is necessary to calibrate the cameras and the parallel robot to obtain the relationship between the basic coordinate system of camera and the basic coordinate system of parallel robot before sorting, so as to realize the transformation from the pose of fruit in camera coordinate system to grasping pose of end effector in parallel robot coordinate system.

Figure 2.

4-R(2-SS) parallel robot with 4-DOF for automatic sorting of fruits. 1: Parallel mechanism; 2: calibration plate; 3: clamping mechanism; 4: kinect camera; and 5: fruit. DOF: degree of freedom.

Motion planning and hand–eye calibration of 4-R(2-SS) parallel robot based on nontrivial solution constraint of eye-to-hand model

(1) Constructing nontrivial solution constraint of eye-to-hand model. To calculate the pose of the basic coordinate system of parallel robot in the basic coordinate system of camera, the model equations constructed by calibration data obtained from the calibration motions of parallel robot need to have nontrivial solutions. Therefore, the nontrivial solution constraint of eye-to-hand model is constructed. It is adopted to remove the invalid calibration motion poses and plan hand–eye calibration motion of end effector of parallel robot for improving the accuracy and efficiency of online hand–eye calibration. The specific constraints are as follows:

(a) The pose transformation matrix B between two calibration motions of end effector needs to satisfy $B \neq I$ .

(b) The rotation angle $θ_{A}$ between two calibration motions of end effector or calibration plate needs to satisfy $θ_{A} \neq 2 k π$ .

(c) The pose transformation matrix A between two calibration motions of calibration plate and the pose transformation matrix B between two calibration motions of end effector have common characteristic roots.

For the constraint (a), if $B = I$ , there is no change between two calibration motions of end effector and the model cannot be solved. For the constraint (b), if $θ_{A} = 2 k π$ , the translation component of pose matrix X cannot be calculated based on equation (14). For the constraint (c), the necessary and sufficient condition for $AX = XB$ to have nontrivial solution is that A and B have common characteristic roots, which are proved as follows:

If J and L are the Jordan normal forms of A and B , respectively, and $A = TJ T^{- 1}$ and $B = HL H^{- 1}$ , then $AX = XB \Leftrightarrow JZ = ZL$ , where $Z = T^{- 1} XH$ . If characteristic roots of J and L are $λ_{i} (i = 1, 2, ..., k)$ and $η_{j} (j = 1, 2, ..., k)$ , respectively, then the elements on the principal diagonal of k ²th order square matrix obtained by $JZ = ZL$ based on the Jordan normal form are $λ_{i} - η_{j}$ $(i = 1, 2, ..., k; j = 1, 2, ..., k)$ .

Necessity: If $λ_{i} \neq η_{j} (i = 1, 2, ..., k; j = 1, 2, ..., k)$ , then for the homogeneous equation $QY = 0$ transformed from $JZ = ZL$ , a k ²th order unit matrix can be obtained by elementary row operations of Q where $IY = 0$ . Therefore, the $IY = 0$ has only trivial solution. Then the $JZ = ZL$ and $AX = XB$ have only trivial solution.

Sufficiency: If the $JZ = ZL$ has only nontrivial solution, then the Q is a nonsingular matrix. So the elements on the principal diagonal of Q satisfy $λ_{i} - η_{j} \neq 0 (i = 1, 2, ..., k; j = 1, 2, ..., k)$ , that is $λ_{i} \neq η_{j}$ . QED

(2) Hand–eye calibration based on planned motion. The translational motion path of end effector of 4-R(2-SS) parallel robot is planned firstly. The cylindrical work space for fruit sorting in the 4-R(2-SS) parallel robot is divided into three parts averagely. The positions with the same interval in each part are taken as the ideal handeye calibration positions which are labeled from bottom to top $i = 1, 2, 3, ..., 15$ , that is – $C 1, ... C 5, B 1, ..., B 5, A 1, ..., A 5$ .

Then the end effector does small-scale rotation and large-scale translation randomly near the ideal hand–eye calibration positions. The random motion poses of end effector are chosen according to the proposed three constraints in this article. As shown in Figure 3, the motion poses of end effector, which satisfy the nontrivial solution constraint, are adopted to construct the improved eye-to-hand model of stereovision with motion error compensation. The improved solving method with vertical-component correction is adopted to solve the eye-to-hand model. Then the accurate and fast hand–eye calibration for 4-R(2-SS) parallel robot with 4-DOF is realized based on the motion planning.

Figure 3.

Hand–eye calibration for 4-R(2-SS) parallel robot with 4-DOF based on the motion planning. DOF: degree of freedom.

Grasping pose calculation of 4-R(2-SS) parallel robot with 4-DOF

To achieve accurate and stable grasping of fruit, the end effector of parallel robot needs to move to the position of fruit and grasp the fruit with an optimal grasping pose in the fruit sorting system based on 4-R(2-SS) parallel robot with 4-DOF. Assuming that the optimal grasping pose is $H_{p}$ , to make the end-effector change from the current pose $H_{g}$ to the optimal grasping pose $H_{p}$ accurately, the $H_{p}$ needs to be transformed and represented as ${}^{w}H_{p}$ in the $O_{w} - X_{w} Y_{w} Z_{w}$ , as shown in equation (25).

{}^{g}H_{p} = {}^{w}H_{g}^{- 1} \cdot {}^{w}H_{p} = {}^{w}H_{g}^{- 1} \cdot {}^{w}H_{d} \cdot {}^{d}H_{p}

The ${}^{g}H_{p}$ and ${}^{d}H_{p}$ are the optimal grasping poses in $O_{g} - X_{g} Y_{g} Z_{g}$ and $O_{d} - X_{d} Y_{d} Z_{d}$ , respectively. The ${}^{w}H_{g}$ is the current pose of end effector in the $O_{w} - X_{w} Y_{w} Z_{w}$ . The grasping model equation (25) of 4-R(2-SS) parallel robot is also improved by motion error obtained from hand–eye calibration based on differential motion as follows:

{}^{g}H_{p} = ({}^{w}H_{g} + d B)^{- 1} \cdot {}^{w}H_{d} \cdot {}^{d}H_{p}

The current pose ${}^{w}H_{g}$ of end effector in the $O_{w} - X_{w} Y_{w} Z_{w}$ can be obtained by forward kinematics of 4-R(2-SS) parallel robot. The kinematics equation of the 4-R(2-SS) parallel robot with 4-DOF in this article is as follows:

\begin{array}{l} x^{2} + y^{2} - 2 (e + l_{1} cos θ_{i}) (x cos γ_{i} + y sin γ_{i}) \\ + {(e + l_{1} cos θ_{i})}^{2} + {(z + ε_{i} s + l_{1} sin θ_{i})}^{2} - l_{2}^{2} = 0 \end{array}

where $i = 1, 2, 3, 4$ . As shown in Figure 4, the $(x, y, z)$ is the coordinate of point P ₂ in the $O_{w} - X_{w} Y_{w} Z_{w}$ . The norm of vector $e_{i} = e {(cos γ_{i}, sin γ_{i}, 0)}^{T}$ , which is the vector from O to A_i , represents the incircle radius difference between the moving and stationary platform. The $γ_{i} = (i - 1) π / 2$ represents the structural angle of stationary platform. The l ₁ and l ₂ represent the length of master arm and slave arm in side chain i, respectively. The $θ_{i}$ represents the angle of the master arm i. The s is the Z-direction translation of auxiliary platform relative to moving platform. The $s = p (θ / 2 π)$ , where p is the lead screw pitch and $θ$ is lead screw angle. Then:

ε_{i} = {\begin{cases} 0 i = 1, 3 \\ 1 i = 2, 4 \end{cases}

Figure 4.

Structure of 4-R(2-SS) parallel robot with 4-DOF. DOF: degree of freedom.

Then, according to equation (27), the pose $(x_{g}, y_{g}, z_{g}, θ_{g})$ of end effector can be calculated as follows:

{\begin{cases} x_{g} = x \\ y_{g} = y \\ z_{g} = z - s - c - g \\ θ_{g} = 2 π s / p \end{cases}

where the c is the distance between P ₁ and P. The g is the distance between P and O_g . Based on equations (1), (2), and (13), the obtained pose parameters $(x_{g}, y_{g}, z_{g}, θ_{g})$ can be transformed into a pose matrix ${}^{w}H_{g}$ . Then, the pose matrix X obtained from hand–eye calibration is inverted to calculate the pose matrix ${}^{w}H_{d}$ of the infrared camera in the $O_{w} - X_{w} Y_{w} Z_{w}$ . The cameras are calibrated by the camera calibration method based on single plane checkerboard to calculate the optimal grasping pose ${}^{d}H_{p}$ . Finally, according to the grasping model of 4-R(2-SS) parallel robot in equation (26), the pose relationship matrix ${}^{g}H_{p}$ between the current pose and the optimal grasping pose of end effector can be calculated to realize the accurate and stable grasping of fruit.

Results and discussion

The proposed method was verified by experiments with a self-developed novel fruit sorting system based on 4-R(2-SS) parallel robot with 4-DOF. The hardware of experiment included the calibration plate with $7 \times 7$ circles with a diameter of 3.5 mm and spacing of 7 mm, and Intel(R)Core(TM)i5-6600 CPU with 8 GB memory.

Comparison of online hand–eye calibration according to random motion and planned motion based on nontrivial solution constraint of eye-to-hand model

The end effector fixed with calibration plate does random motion and planned motion based on nontrivial solution constraint of eye-to-hand model in the field of view of Kinect camera, respectively. At each calibration position, the calibration plate images are acquired by color camera and infrared camera in Kinect camera, respectively. According to equation (27), the pose of end effector at each calibration position is calculated and transformed into the pose matrix ${}^{w}H_{g}$ . The online hand–eye calibration process based on random motion is as follows:

The end effector randomly moves to 15 calibration positions for calibration data acquisition.

The hand–eye model equations are constructed to solve the pose matrix X based on the 15 calibration motion poses.

If there is no solution to the model equations, the end effector removes and processes (a) and (b) are repeated. Otherwise, the online hand–eye calibration is completed.

The planned motion based on nontrivial solution constraint of eye-to-hand model is shown in Figure 5. The orange line represents the motion path of end effector. The blue dots represent the ideal calibration positions. The boxes represent the actual calibration positions after the random motion of end effector near the ideal calibration positions. The cylinder represents the work space for fruit sorting in 4-R(2-SS) parallel robot.

Figure 5.

Planned motion based on nontrivial solution constraint of eye-to-hand model.

The improved eye-to-hand model of stereovision with motion error compensation and the improved solving method with vertical-component correction for the eye-to-hand model are applied to the experiments of online hand-eye calibration based on random motion and planned motion. The average error $\bar{Err}$ , average calibration time $\bar{t}$ , and orthogonality of rotation matrix R in multigroup experiments are presented in Table 1. The average calibration time $\bar{t}$ includes motion time of end effector, time of calibration data acquisition, and time of model solving. The average error $\bar{Err}$ can be calculated based on equations (30) and (31), where the ${Err}_{q}$ is an error in the qth experiment, $i = 1, 2, ..., 15$ , $q = 1, 2, ..., Q$ , and Q is the number of experiments.

\begin{array}{l} {Err}_{q} = ({‖ (A_{q d i} + d A_{q d}) X_{q} - X_{q} (B_{q i} + d B_{q}) ‖}_{2} \\ + {‖ (A_{q c i} + d A_{q c}) ({}^{c}H_{d} X_{q}) - ({}^{c}H_{d} X_{q}) (B_{q i} + d B_{q}) ‖}_{2}) / 2 \end{array}

\bar{Err} = \frac{\sum_{q = 1}^{Q} {Err}_{q}}{Q}

Table 1.

Experimental results of online hand–eye calibration.

Motion	Eye-to-hand model	Model solving	$\bar{Err}$	${\bar{O R}}_{1}$ (×10⁻⁵)	${\bar{O R}}_{2}$ (×10⁻⁵)	${\bar{O R}}_{3}$ (×10⁻⁵)	$\bar{t}$ (s)
Random motion	Improved eye-to-hand model	Improved solving method	1.124	8.611	6.489	6.109	58.947
Planned motion	Improved eye-to-hand model	Improved solving method	1.028	−0.945	−1.355	−1.967	29.174

The obtained rotation matrix R is expressed by $R = [R_{1}, R_{2}, R_{3}]$ . Then the orthogonality of rotation matrix R can be analyzed based on equations (32). The ${\bar{O R}}_{1}$ , ${\bar{O R}}_{2}$ , and ${\bar{O R}}_{3}$ are average dot products of column vectors of R .

{\bar{O R}}_{1} = \frac{\sum_{q = 1}^{Q} {(R_{1}^{T} R_{2})}_{q}}{Q}, {\bar{O R}}_{2} = \frac{\sum_{q = 1}^{Q} {(R_{1}^{T} R_{3})}_{q}}{Q}, and {\bar{O R}}_{3} = \frac{\sum_{q = 1}^{Q} {(R_{2}^{T} R_{3})}_{q}}{Q}

From Table 1, we can see that compared with the random motion, the average error and average time of online hand–eye calibration based on planned motion decrease by 0.096 and 29.773 s, respectively. The planned motion based on nontrivial solution constraint of eye-to-hand model has the advantages that the motion distribution in the work space is uniform, and the invalid calibration motion poses can be removed in advance. Therefore, the planned motion can improve the calibration accuracy and reduce the calibration time, so as to lay the foundation for accurate and fast grasping of parallel robot.

Comparison of hand–eye calibration based on existing and improved methods

The end effector fixed with calibration plate does planned motion based on nontrivial solution constraint of eye-to-hand model in the field of view of Kinect camera. At each calibration position, the calibration plate images are acquired by color camera and infrared camera in Kinect camera, respectively, as shown in Figure 6. According to equation (27), the pose of end effector at each calibration position is calculated and transformed into pose matrix ${}^{w}H_{g}$ . The matrices ${}^{w}H_{g}$ of some valid calibration motion poses after choosing are presented in Table 2, where the unit of rotation is ° and the unit of translation is mm. At each ideal calibration position, multigroup calibration data corresponding to different poses of end effector are acquired and saved for testing hand–eye models and model solving methods.

Figure 6.

Calibration plate images. (a) Color images and (b) infrared images.

Table 2.

Matrix elements of some valid calibration motion poses.

${}^{w}H_{g}$	R ₁₁	R ₁₂	R ₂₁	R ₂₂	R ₃₃	t_x	t_y	t_z
1	0.515	0.857	−0.857	0.515	1	38.595	6.847	410.187
2	−0.804	0.595	−0.595	−0.804	1	32.002	159.598	382.191
3	−0.532	−0.847	0.847	−0.532	1	18.354	−121.150	403.068
4	0.978	−0.209	0.209	0.978	1	5.995	25.119	504.322
5	0.700	−0.714	0.714	0.700	1	184.220	4.310	477.784
6	−0.935	−0.353	0.353	−0.935	1	−131.994	25.572	470.555
7	−0.273	0.962	−0.962	−0.273	1	14.369	−141.359	477.741
8	0.999	−0.041	0.041	0.999	1	158.239	15.943	554.729
9	−0.145	−0.989	0.989	−0.145	1	−144.884	23.327	575.677
10	0.975	0.222	−0.222	0.975	1	26.219	−144.793	578.128

Based on the acquired multigroup calibration data, the existing eye-to-hand model, the improved eye-to-hand model of stereovision with motion error compensation, the model solving method based on matrix vectorization and Kronecker product, and the improved solving method with vertical-component correction are applied to hand-eye calibration experiments of 4-R(2-SS) parallel robot with 4-DOF, respectively. The average error $\bar{Err}$ , average time $\bar{t}$ of model solving, and orthogonality of rotation matrix R are presented in Table 3, which can be calculated according to equations (30)–(32).

Table 3.

Experimental results of hand-eye calibration.

Motion	Eye-to-hand model	Model solving	$\bar{Err}$	${\bar{O R}}_{1}$ (×10⁻⁵)	${\bar{O R}}_{2}$ (×10⁻⁵)	${\bar{O R}}_{3}$ (×10⁻⁵)
Planned motion	Existing eye-to-hand model	Model solving method based on matrix vectorization and Kronecker product	152.324	55.50	50.14	59.34
	Existing eye-to-hand model	Improved solving method with vertical-component correction	1.137	54.12	41.36	51.10
	Improved eye-to-hand model	Model solving method based on matrix vectorization and Kronecker product	152.322	−0.910	−1.969	−1.834
	Improved eye-to-hand model	Improved solving method with vertical-component correction	1.029	−0.894	−1.410	−1.887

From Table 3, we can see that compared with the existing eye-to-hand model, the average errors of hand–eye calibration according to the improved eye-to-hand model combined with existing and improved solving methods decrease by 0.002 and 0.108, respectively. The orthogonality of the obtained rotation matrix $R$ is also better. Compared with the model solving method based on matrix vectorization and Kronecker product, the improved solving method with vertical-component correction for the eye-to-hand model can accurately calculate the Z-direction translation component. The average error decreases by 151.293. It can improve the orthogonality of the obtained rotation matrix $R$ and the accuracy of hand–eye calibration.

Comparison of grasping pose calculation according to existing and improved hand–eye calibration methods

To verify the advantage of the hand–eye calibration method with motion error compensation and vertical-component correction for 4-R(2-SS) parallel robot when it is applied to grasping pose calculation, the existing and improved hand–eye calibration methods are applied to the experiments of grasping cluster fruit, respectively. The images of cluster fruit are acquired by Kinect camera firstly, as shown in Figure 7. The optimal grasping pose ${}^{d}H_{p}$ in the $O_{d} - X_{d} Y_{d} Z_{d}$ is calculated.

Figure 7.

Cluster fruit regions in color images.

Then, the hand–eye calibration results according to the existing eye-to-hand model, the improved eye-to-hand model of stereovision with motion error compensation, the model solving method based on matrix vectorization and Kronecker product, and the improved solving method with vertical-component correction are applied to grasping pose calculation, respectively. According to the obtained ${}^{w}H_{g}$ and ${}^{w}H_{d}$ , the ${}^{g}H_{p}$ is calculated based on the grasping model of 4-R(2-SS) parallel robot in equation (26). Finally, the pose of end effector is transformed from the current pose to the optimal grasping pose based on the ${}^{g}H_{p}$ to grasp the cluster fruit. Then the transformed pose of end effector is compared with the optimal grasping pose measured by the Leica absolute tracker system (Leica AT402) of Leica Geosystems AG and the digital compass (HMR3100) of Honeywell. As shown in Figure 8, the errors of pose parameters $(x, y, z, α, β, γ)$ are analyzed, respectively.

Figure 8.

The errors of grasping pose calculation based on hand–eye calibration results. (a) x-errors, (b) y-errors, (c) z-errors, (d) α-errors, (e) β-errors, and (f) γ-errors. p: The existing eye-to-hand model and the model solving method based on matrix vectorization and Kronecker product. q: The existing eye-to-hand model and the improved solving method with vertical-component correction. s: The improved eye-to-hand model of stereovision with motion error compensation and the model solving method based on matrix vectorization and Kronecker product. t: The improved eye-to-hand model of stereovision with motion error compensation and the improved solving method with vertical-component correction.

From Figure 8, we can see that compared with existing eye-to-hand model and model solving methods, the average errors of x, y, z, α, β, and γ according to the improved eye-to-hand model and model solving method decrease by 0.626 mm, 0.712 mm, 152.182 mm, 0.724°, 0.656°, and 0.689°, respectively. The average translation error of grasping pose decreases by 51.173 mm, and the average rotation error of grasping pose decreases by 0.690°. Because the Z-direction translation component of pose matrix $X$ is set to zero in the model solving method based on matrix vectorization and Kronecker product, the error of z is affected by the Z-direction distance between the $O_{d} - X_{d} Y_{d} Z_{d}$ and the $O_{w} - X_{w} Y_{w} Z_{w}$ . Therefore, the average z-error of grasping pose calculation according to the hand–eye calibration results of the model solving method based on matrix vectorization and Kronecker product is larger. The improved eye-to-hand model and model solving method proposed in this article can reduce the pose errors and improve the accuracy and efficiency of grasping pose calculation effectively.

Conclusions

Because of the 4-DOF motion constraint of robot, the motion error of robot, the calibration error of camera, and the invalid calibration motion poses of robot, it is difficult to improve the accuracy and efficiency of online hand–eye calibration of 4-R(2-SS) parallel robot. Therefore, an online hand–eye calibration method with motion error compensation and vertical-component correction for 4-R(2-SS) parallel robot is proposed and applied to grasping pose calculation in this article. The following conclusions can be drawn:

An improved eye-to-hand model of stereovision with motion error compensation is proposed to reduce the influence of camera calibration error and robot motion error on accuracy of eye-to-hand model of 4-R(2-SS) parallel robot.

The vertical component of eye-to-hand model parameters is corrected based on the vertical constraint between calibration plate and end effector. The existing model solving method based on matrix vectorization and Kronecker product is improved to solve the problem that it is difficult to calculate eye-to-hand model parameters due to the rotational constraint of 4-R(2-SS) parallel robot with 4-DOF accurately.

The nontrivial solution constraint of eye-to-hand model is constructed. A calibration motion planning method based on nontrivial solution constraint of eye-to-hand model is proposed for improving the accuracy and efficiency of online hand–eye calibration.

The proposed method was verified by experiments with a self-developed novel fruit sorting system based on 4-R(2-SS) parallel robot with 4-DOF. Experimental results demonstrate that the improved eye-to-hand model and solving method and the proposed calibration motion planning method can improve the accuracy and efficiency of hand–eye calibration of 4-R(2-SS) parallel robot effectively and further realize accurate and fast grasping of parallel robot.

In the future work, we will explore how to further improve the accuracy and efficiency of online hand–eye calibration of 4-R(2-SS) parallel robot, so that it can be applied to more complex environment. Additionally, we will also study how to improve the reliability of online hand–eye calibration to lay the foundation for accurate and fast grasping based on machine vision and parallel robot.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China (grant no. 51375210); the Postgraduate Research and Practice Innovation Program of Jiangsu Province (grant no. KYCX17_1780); the Zhenjiang Municipal Key Research and Development Program (grant no. GZ2018004); and the Priority Academic Program Development of Jiangsu Higher Education Institutions.

ORCID iD

Guo-Qin Gao

References

Blasco

Aleixos

Molto

. Machine vision system for automatic quality grading of fruit. Biosyst Eng 2003; 85(4): 415–423.

Glossas

Aspragathos

. Fuzzy logic grasp control using tactile sensors. Mechatronics 2001; 11(7): 899–920.

Wang

Tang

Xiao

. Grape size detection and online gradation based on machine vision. Int J Agr Biol Eng 2017; 10(1): 226–233.

Sun

Nozaki

Murakami

, et al. Grasping point estimation based on stored motion and depth data in motion reproduction system. In: IEEE international conference on mechatronics (ICM), Ilmenau, Germany, 18–20 March 2019, pp. 471–476.

Bezak

Bozek

Nikitin

Advanced robotic grasping system using deep learning. In: 6th Conference on modelling of mechanical and mechatronic systems (MMaMS), Vysoke Tatry, Slovakia, 25–27 November 2014, pp. 10–20.

Levine

Pastor

Krizhevsky

, et al. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Robot Res 2016; 1(10): 173–184

Ranjan

Kumar

Laxmi

, et al. Identification and control of NAO humanoid robot to grasp an object using monocular vision. In: Second international conference on electrical, computer and communication technologies, Coimbatore, India, 22–24 February 2017, pp.1–5.

Lin

Gonzalez

Cheng

, et al. Vision based object grasping of industrial manipulator. In: International conference on advanced robotics and intelligent systems, Taipei, Taiwan, 31 August–2 September 2016.

Zhang

, et al. Simultaneous path planning and trajectory optimization for high-speed sorting system. Int J Adv Robot Syst 2018; 15(5): 1–13.

10.

Tho

Thinh

. Using ANFIS to predict picking position of the fruits sorting system. In: International conference on system science & engineering, Ho Chi Minh, Vietnam, 21–23 July 2017, pp. 297–304.

11.

Cheng

Chang

. Dynamic performance improvement of direct image-based visual servoing in contour following. Int J Adv Robot Syst 2018; 15(1): 1–12.

12.

Taryudi Wang

. Eye to hand calibration using ANFIS for stereo vision-based object manipulation system. Microsyst Technol 2017; 24(1): 1–13.

13.

Chang

. Robotic assembly of smartphone back shells with eye-in-hand visual servoing. Robot Cim-Int Manuf 2018; 50: 102–113.

14.

Feng

, et al. Active 6-D position-pose estimation of a spatial circle using monocular eye-in-hand system. Int J Adv Robot Syst 2018; 15(1): 1–13.

15.

Carlson

Johansson

Robertsson

. Six DOF eye-to-hand calibration from 2D measurements using planar constraints. In: IEEE/RSJ international conference on intelligent robots & systems, Hamburg, Germany, 28 September–2 October 2015.

16.

Zeng

Xie

, et al. Relative posture-based kinematic calibration of a 6-RSS parallel robot by optical coordinate measurement machine. Int J Adv Robot Syst 2018; 15(2): 1–14.

17.

Malm

Heyden

. A new approach to hand-eye calibration. In: International conference on pattern recognition, Barcelona, Spain, 3–7 September 2000, pp. 525–529.

18.

Bozek

Pokorny

Svetlik

, et al. The calculations of Jordan curves trajectory of the robot movement. Int J Adv Robot Syst 2016; 13(5): 1–7.

19.

Pivarciova

Bozek

Turygin

, et al. Analysis of control and correction options of mobile robot trajectory by an inertial navigation system. Int J Adv Robot Syst 2018; 15(1): 1–15.

20.

Andreff

Horaud

Esaiau

. Robot hand-eye calibration using structure-from-motion. Int J Robot Res 2001; 20(3): 228–248.

21.

Heller

Henrion

Pajdla

. Hand-eye and robot-world calibration by global polynomial optimization. In: IEEE international conference on robotics & automation, Hong Kong, China, 31 May–7 June 2014, pp. 3157–3164.

22.

Kruger

Globally optimal hand-eye calibration. In: IEEE conference on computer vision & pattern recognition, Providence, RI, USA, 16–21 June 2012, pp. 1035–1042.

23.

Shiu

Ahmad

. Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX=XB. IEEE Transact Robot Automat 1989; 5(1): 16–29.

24.

Ren

. Finding the kinematic base frame of a robot by hand-eye calibration using 3D position data. IEEE Transact Automat Sci Eng 2017; (99): 1–11.

25.

Lam

Eye-in-hand visual servoing to improve accuracy in pool robotics. Master Thesis, Queen’s University, Canada, 2007.

26.

Andreft

Horaud

Espiau

. Robot hand-eye calibration using structure-from-motion. Int J Robot Res 2000; 20(3): 228–248.

27.

Konstantinos

. Hand-eye calibration using dual quaternions. Int J Robot Res 1998; 18(3): 286–298.

28.

Park

Martin

. Robot sensor calibration: solving AX=XB on the Euclidean group. IEEE Transact Robot Automat 2002; 10(5): 717–721.

29.

Zhuang

. Hand/eye calibration for electronic assembly robots. Robot Automat 1998; 14(4): 612–616.

30.

. A self-calibration technique for active vision systems. Robot Automat 1996; 12(1): 114–120.

31.

Zhang

Zheng

, et al. Hand-eye calibration method fusing rotational and translational constraint information. Chi J Scient Inst 2015; 36(11): 2443–2450.

32.

Xie

, et al. Vision-based calibration/compensation technique for automatic stiffener bonder. P I Mech Eng B-J Eng 2015; 231(1): 167–180.

33.

Shao

Guan

, et al. Dynamic performance analysis of the X4 high-speed pick-and-place parallel robot. Robot Cim-Int Manuf 2017; 46: 48–57.

34.

Bobovsky

Krys

Mostyn

. Kinect v2 infrared images correction. Int J Adv Robot Syst 2018; 15(1): 1–8.