Hand-eye calibration method and machine vision research based on sensor network

Abstract

With the promotion of Industry 4.0 reform, the trend of intelligent and precise production in the production workshop is gradually highlighted. This directly leads to higher requirements for robot hand eye coordination accuracy in automated workshops. In order to achieve more precise robot hand eye coordination control, this study designed a new mean calculation method based on the probability density theory, and designed a new mean robot hand eye calibration algorithm based on this. After the test, it is found that the translation error and rotation error calculated by the new mean algorithm are 0.26 and 0.92 respectively, which are significantly lower than other comparison algorithms when using all test samples of normal distribution. And the calculation time of the algorithm when using all the test samples is 2115 ms, which is also significantly lower than the comparison algorithm. The simulation results show that the new mean hand eye calibration method designed in this study can achieve more accurate hand eye coordination control of robots, and has certain application potential in high-precision industrial production scenarios.

Keywords

Hand-eye calibration machine vision sensor robotic arm probability density theory mean value calculation

1. Introduction

The physical grasping and other operations of the robot or the robot arm are similar to the corresponding actions of human beings. The former needs to sense its own coordinates and attitude information and external environment information through sensors. The more accurate the execution of this process, the more flexible and accurate the robot’s control behavior to external objects. The calibration algorithm is designed for this process [1]. In general, the “hand” in the calibration algorithm means the end of the execution part of the object, and the “eye” means the sensor that obtains environmental information, such as a camera, a camera, etc. [2, 3]. With the increasing demand for precision machining in the industrial field, the calibration accuracy of the traditional robot hand-eye calibration algorithm gradually cannot meet the requirements [4]. In addition, the traditional robot hand-eye calibration algorithm is prone to problems such as different sensor sampling frequencies, data incompatibility, data repetition, and time stamp misalignment during the process of acquiring calibration data. Therefore, this research innovatively tries to use the probability density theory to design a new mean value calculation method from the two perspectives of the rotation part and the translation part, and designs a new mean value robot hand eye calibration algorithm with this as the core. This is the main research method used in this research, and then carries out test research in the design to verify the impact of the design algorithm on the quality of the robot arm hand eye calibration work. Finally, we summarize the test results and analyze the advantages and disadvantages of the algorithm.

2. Related works

Accurate robot hand eye calibration (RHEC) is very important to improve the control accuracy and work efficiency of intelligent production machinery. Therefore, experts and scholars in the industry have conducted a lot of academic research on RHEC problems and machine vision performance improvement. Ali et al. [5] designed two improved algorithms for RHEC by studying the way of setting the objective function as a specific attitude error or re projection difference minimization problem. At the same time, the research team has also designed a wrong robot hand eye calibration algorithm, which is used together with the previous method to build a realistic response. Finally, the simulation test was carried out using simulation data, and it was found that compared with several calibration methods commonly used in the industry, the calibration method designed in this study has significantly improved the control accuracy and calculation time. Li et al. [6] analyzed the influence of line laser sensor RHEC error on the error of space point cloud reconstruction, and then designed a line laser RHEC algorithm based on three-dimensional repair technology considering the demand of some production workshops for rapid production of robots. In this algorithm, the newly defined number of error words is used to more accurately and subjectively reflect the calibration results. To solve the problem of low calibration efficiency, particle swarm optimization and Gaussian process are used to optimize the algorithm training method. According to the simulation results, the root mean square error of the new robot line laser hand eye calibration algorithm designed in this study is 0.12 mm, which is significantly better than the common robot hand eye calibration algorithm. For RHEC problem, Zhao [7] constructed an improved RHEC algorithm by using alternating linear programming to construct a semi convex objective function, and tested the algorithm with simulation data and real experimental data. The results show that the robustness of the improved RHEC algorithm is significantly improved compared with the traditional algorithm. Zou and Chen [8] proposed a RHEC algorithm for arc welding robots equipped with laser vision sensors. After the test, it is found that the new RHEC algorithm designed in this study can improve the RHEC accuracy of the arc welding robot in the use environment with small data volume. In order to alleviate the hand eye calibration problem of the light sensor of the welding robot, Zheng et al. [9] designed an algorithm integrating spatial straight line correction. When the robot randomly controls the end effector, the light plane projected by the linear structured light sensor can intersect with the calibration template. The algorithm can use the feature point information to construct the intersection of the outgoing structured light and the straight line. Moreover, the penalty function and the improved Powell algorithm are used to solve the position and attitude parameters of the robot simultaneously. After the test, the analysis found that the hand eye calibration algorithm designed in this study has good robustness and greatly improved the calculation accuracy compared with the traditional algorithm. During the experiment, the calibration accuracy is less than 0.2 mm.

Tan et al. [10] analyzed several literatures on the calibration of soft manipulators. The calibration problem is defined as a matrix exchange problem with some known conditions. On the premise of analyzing the identifiability of model parameters, a calibration method based on nonlinear rule optimization and intelligent evolutionary computation is designed. After extensive simulation tests, it is found that if the experiment is carried out on a real soft robot hand device. The extrapolation error of the soft manipulator hand eye calibration algorithm designed in this study is significantly lower than that of many comparison algorithms. Sharifzadeh et al. [11] found that the hand eye calibration of planar laser line sensor is a difficult process because only two-dimensional data are collected. Therefore, this study proposed a relatively simple RHEC algorithm, which requires significantly less user interaction and uses a single plane to calibrate artifacts. The significant advantage of this strategy is that it uses low-cost, simple and easy to manufacture artifacts. However, lower complexity may result in lower calibration data variations. Moreover, in order to realize robust RHEC using this artifact, the algorithm considers the influence of the robot positioning strategy to keep the change. After the test, it was found that this algorithm can significantly reduce the RHEC time of the planar laser line sensor. Yu and Xi [12] proposed a new method of robot arm position calibration to calibrate and compensate the hand eye posture of the robot detection system. Compared with the traditional hand eye calibration method, this calibration method has several unique characteristics. First, since the optical sensor is installed on the end part of the object, the two can be regarded as a whole, so the hand eye calibration can be carried out without external measuring equipment to achieve online calibration. Secondly, it can simultaneously calibrate all motion parameters of the robot in the same step to avoid error propagation, instead of calibrating the relationship between the end effector and the information sensor, the relationship between the machine itself and the environment. Finally, experiments were carried out in the detection system based on the 6-DOF serial robot, and the results show that the method has excellent hand eye calibration and position calibration performance.

To sum up, in order to optimize the calibration accuracy of RHEC algorithm, experts and scholars in the industry have made a large number of improvements to a variety of traditional calibration algorithms and have made a variety of research results. However, there are few changes in the calculation process of the mean value of the calibration data set that affects the calibration effect of the robot hand eye, which is the starting point of this study.

3. RHEC algorithm design based on sensor network and new mean definition

3.1 Mathematical model building of RHEC problem

RHEC algorithm is the basis for the robot to sense the external environment through sensors, and the research on this algorithm can obtain the position and attitude relationship of the external object relative to its own coordinate system [13], so as to improve the motion accuracy of the manipulator in scientific and industrial application scenarios. Among them, the hand refers to the end effector within the robot system, and the eye refers to the sensor or camera [14, 15]. At present, the research of RHEC can be divided into three aspects, namely, the calibration of the position and attitude relationship matrix of the calibration plate and the hand eye relationship matrix, and the hand eye calibration of the multi machine system. The research takes the robot hand eye relationship matrix as the research content for analysis. The problem can be divided into two cases: the eye outside the hand and the eye on the hand according to the geometric relationship of the machine system and the geometric position of the sensor, as shown in Fig. 1. Eye on hand means that sensors such as cameras are connected to the end effector of the robot, while the relative position between the calibration plate and the ground remains unchanged [16]. In this case, the relative positions of the sensors such as the camera and the robot end do not change before and after the movement. For the case where the eye is out of the hand, the robot arm end is separated from the sensor such as the camera, which is connected to the robot arm end [17]. The position of the calibration plate and the posture of the robot end before and after the movement are fixed. The research will analyze RHEC problems in the form of eyes on hands.

Figure 1.

Two forms of RHEC.

The RHEC algorithm is closely related to the accuracy of the probability calibration algorithm itself. At the same time, in order to improve the flexibility and application scope of the calibration algorithm, it is necessary to avoid over-reliance on the calibration object. Therefore, the mathematical model constructed by the study is shown in Fig. 2.

Figure 2.

Mathematical model of hand eye calibration.

Figure 2 contains a total of four coordinate systems, namely the calibration board coordinate system $O_{\textit{mar}}$ , the robot’s eye-camera and other sensor coordinate systems $O_{\textit{eye}}$ , the robot’s hand-end effector coordinate system $O_{\textit{hand}}$ , and the robot base coordinate system $O_{\textit{base}}$ . There are two sets of closed lines in the figure to show the calibration process, and the correlation satisfies Eq. (1).

$\displaystyle A_{i}X=XB_{i}$ (1)

In Eq. (1), the second transformation matrix relative to the sensor such as the camera at the end of the robot is $X$ , $A_{i}$ and $B_{i}$ are homogeneous transformation matrices describing the geometric relationship of the manipulator, the lower calibration plate and $O_{\textit{eye}}$ , $O_{\textit{base}}$ the lower robot end at the $i$ second and the $k$ third relative pose homogeneous transformation matrix are denoted by $A_{i}$ and $A_{i}$ respectively. The data set used in the robot hand-eye calibration process can be formed into a data pair in the hand-eye calibration matrix equation, which is called the calibration data set and can be $\left({A_{i},B_{i}}\right)$ represented. $\left({A_{i},B_{i}}\right)$ The essence is that there is a corresponding relationship in the data set, which can be calculated by Eq. (2).

$\displaystyle\left\{{\begin{array}[]{l}A_{i}=\bar{A}_{i}^{-1}\cdot\bar{A}_{k}% \\ B_{i}=\bar{B}_{i}\cdot\bar{B}_{k}^{-1}\\ \end{array}}\right.$ (2)

In Eq. (2), the pose data actually collected during the calibration process is $\bar{A}_{i},\bar{A}_{k},\bar{B}_{i},\bar{B}_{k}$ the homogeneous transformation matrix of the first $\bar{A}_{i}$ and second poses at the $k$ lower end of the robot are denoted by $O_{\textit{base}}$ , $i$ and $\bar{A}_{k}$ respectively. $O_{\textit{eye}}$ The homogeneous transformation matrices of the $\bar{B}_{i}$ first and second poses of the $k$ lower calibration board are denoted by $i$ and $\bar{B}_{k}$ respectively. In the process of solving the matrix equation, it is $\left({A_{i},B_{i}}\right)$ almost impossible to obtain accurate data to analyze the hand-eye relationship. When two pairs of mutually independent and accurate data sums are provided $\left({A_{1},B_{1}}\right)$ , $\left({A_{2},B_{2}}\right)$ and corresponding restriction requirements are given, a unique and definite solution can be obtained with the help of Eq. (1). Therefore, during the experiment, at least 3 pairs of absolute homogeneous transformation matrix data should be obtained to calculate the $\left({A_{1},B_{1}}\right)$ sum $\left({A_{2},B_{2}}\right)$ , and finally the hand-eye relationship can be obtained. However, in the actual measurement process, the sensor usually has noise, which $X$ hinders the acquisition of accurate measurements [18, 19]. Given this, the analysis of the hand-eye calibration problem can be translated into finding $\left({A_{i},B_{i}}\right)$ the minimum error for a calibration dataset containing some noisy data. Generally speaking, the hand-eye calibration problem can use two methods to obtain the values of the translation and rotation parts that cannot be determined, namely the partial calculation method and the unified calculation method [20]. For the part calculation method, Eq. (3) can be obtained by rotating and translating the two parts, and then the solution of the translation and rotation can be obtained.

$\displaystyle\left\{{\begin{array}[]{l}R_{A}R_{X}=R_{X}R_{B}\\ R_{A}t_{X}+t_{A}=R_{X}t_{B}+t_{X}\\ \end{array}}\right.$ (3)

In Eq. (3), the translation and transfer parts of the calibration problem are denoted by $t_{X},t_{A},t_{B}$ and $R_{A},R_{X},R_{B}$ respectively, and the expression is 3 $\times$ 1, 3 $\times$ 3.

The solution process of this method requires two pairs of high-precision data pairs $\left({A_{i},B_{i}}\right)$ , which is also the most common analytical method. However, in the non-degenerate case, the solution obtained by Eq. (3) does not have unique independence, which makes it impossible to obtain a unique solution during the solution process involving two degrees of freedom. For the unified calculation method, the translation and rotation can be obtained at the same time with the help of the cost function, or the matrix can be represented by theoretical forms such as dual four elements. The common solution method of this method is the cost optimization method, and the calculation equation is Eq. (4).

$\displaystyle C\left(X\right)=\sum\limits_{i=1}^{n}{w_{i}d^{2}\left({A_{i}X,XB% _{i}}\right)}$ (4)

In Eq. (4), the number of calibration data pairs, the value range is $>$ 2. A function based on a certain “distance” between variables in a special Euclidean group (Specic Euclidean Gromp; SE (3)) is defined as $d\left({X,Y}\right)$ , a series of weight parameters $w_{i}$ in the objective function is, and the objective cost function in the hand-eye calibration problem is $C\left(X\right)$ . In this method, the error existing in the sensor itself will cause the corresponding data pair to not be obtained.

Both methods need to be combined with probability density theory to obtain the final solution. The first method corresponds to the Batch series calibration algorithm, which obtains the final solution through the singular solution decomposition form and the probability density expression form on SE (3) [21, 22]. The second method corresponds to the application of information theory. It needs to set the objective function of relative entropy under probability density to express the hand eye calibration equation and get the final solution. Because the probability density of the research object is unknown in this study, it is difficult to set the objective function, so it is more reasonable to choose the first method to solve it in combination with the density probability theory.

3.2 Design of robot probabilistic hand-eye calibration algorithm based on new mean definition

In view of the shortcomings of the Batch series of calibration algorithms, such as complicated calculation process, low solution accuracy, and strict requirements for the selection of data sets, and there are great restrictions on the calibration of common scenarios, a new mean value definition method is studied and applied to the RHEC algorithm to solve the problem. Batch series calibration algorithm problem [23]. Firstly, the solution method of the new mean definition is given, and theoretically derived from translation and rotation, and then the Batch algorithm is optimized through this definition, and finally the flow chart of the whole algorithm is obtained [24]. Taking the calibration data set A as an example, $M_{A}$ the calculation expression of the mean value of the calibration data set is Eq. (5).

$\displaystyle M_{A}=\left[{\begin{array}[]{ll}R_{M_{A}}&t_{M_{A}}\\ 0^{T}&1\\ \end{array}}\right]$ (5)

In Eq. (5), $T$ represents the transpose transformation of the matrix, the same below, $t_{M_{A}}$ and $R_{M_{A}}$ refer to the translation and rotation parts of the mean respectively. The solution method of calibration dataset B is the same as that of dataset A. For the calculation of the mean value of the rotated part, the mean value is obtained by SE (3) mapping of the Batch method. For real data sets with low continuity, the mean can be set by weighted summation. Since the weighted summation of the rotation matrix directly does not satisfy the meaning of the SE (3) group elements, the research uses the class mean $\bar{R}_{M_{A}}$ to represent it, and the calculation equation is Eq. (6).

$\displaystyle\bar{R}_{M_{A}}=\frac{1}{n}\sum\limits_{i=1}^{n}{R_{A_{i}}\notin% \textit{SE}}$ (6)

In Eq. (6), $R_{A_{i}}$ is $A_{i}$ the rotational component of. The research will $\bar{R}_{M_{A}}$ use the singular value decomposition method to project to SE (3) to meet the fundamental meaning of the rotation matrix, and $\bar{R}_{M_{A}}$ the singular value decomposition result is Eq. (7).

$\displaystyle\bar{R}_{M_{A}}=U\sum{V^{T}}$ (7)

In Eq. (7), $U,V,\sum$ both are in the form of 3 $\times$ 3 matrix. With reference to the definition of rotation matrix, $\bar{R}_{M_{A}}$ can be mapped to SE (3) by Eq. (8) to meet the requirements.

$\displaystyle R_{M_{A}}=UV^{T}$ (8)

After the singular value decomposition is obtained, the rotated partial mean can be obtained. Therefore, the solution process of the rotation matrix mean can be summarized into three steps. First, the calibration data set is processed by weighted summation to obtain the class mean, then the class mean is decomposed by the singular value decomposition method, and finally, the mean value of the rotation matrix is obtained by multiplying the decomposed matrix.

For the calculation part of the mean value of the translation part $t_{M_{A}}$ , the research takes it as the main innovation point and focus. The conventional Batch method gives two assumptions in the calculation of translation, but the assumptions add constraints to the selection of calibration data sets, which will lead to a decrease in translation accuracy. The study calculates the mean of the translational part by deriving directly from the definition of SE (3). Taking data set A as an example, since the calibration dataset is usually in a discrete state, the translational mean value can be expressed by Eq. (9).

$\displaystyle\sum\limits_{i=1}^{n}{\log\left({M_{A}^{-1}A_{i}}\right)=0}$ (9)

Figure 3.

Flow chart of robot probabilistic hand eye calibration algorithm under the new mean definition.

Equation (10) can be derived due to the approximation given by the Batch method $\log\left({M_{A}^{-1}A_{i}}\right)$ .

$\displaystyle\sum\limits_{i=1}^{n}{\left[{\begin{array}[]{ll}\phi_{i}&p_{i}\\ 0^{T}&0\\ \end{array}}\right]}=0$ (10)

In Eq. (10), $\phi_{i}\in R^{3\times 1}$ is the rotation vector of the left part of Eq. (9). $p_{i}$ is an intermediate variable, which satisfies Eq. (11).

$\displaystyle J_{i}p_{i}=R_{M_{A}}^{T}\left({t_{A_{i}}-t_{M_{i}}}\right)$ (11)

In Eq. (11), $J_{i}$ is the Jacobian matrix, which $t_{A_{i}}$ is the translation component.

The core Eq. (12) of the new mean definition can be derived when only the translational component is considered.

$\displaystyle\sum\limits_{i=1}^{n}{J_{i}^{-1}}R_{M_{A}}^{T}\left({t_{A_{i}}-t_% {M_{A}}}\right)=0$ (12)

It can be seen from Eq. (12) that the solution of the mean value of the translation part requires the mean value of the rotating part. Equation (13) can be obtained by simplifying the equation on the basis of Eq. (12)

$\displaystyle t_{M_{A}}=R_{M_{A}}\left({\sum\limits_{i=1}^{n}{J_{i}^{-1}}}% \right)^{-1}\left({\sum\limits_{i=1}^{n}{J_{i}^{-1}}R_{M_{A}}^{T}t_{A_{i}}}\right)$ (13)

After obtaining the calculation and expression equation of the new mean definition, the research can obtain the flow chart of the robot probabilistic hand-eye calibration algorithm under the new mean definition, as shown in Fig. 3. The whole can be divided into three steps, namely, calculating the mean value of the rotating part, solving the singular value decomposition value of the rotating part, and calculating the mean value of the translation part. The output of the algorithm is the hand-eye calibration data set, and the output data is the solution of the hand-eye relationship. The research is based on the Batch series of hand-eye calibration algorithms, so as to reduce the error caused by probabilistic hand-eye calibration, avoid the limitation of the algorithm on the data set, and facilitate the hand-eye calibration and positioning of different probability distributions.

The common evaluation methods of robot probabilistic hand eye calibration algorithms are real calibration experiments and simulation calibration experiments. However, the real value of the calibration results cannot be obtained in the experiments. The research usually uses the simulation calibration experiments for evaluation, mainly the re projection method and the error matrix method [25, 26]. The error matrix method separates the translation and rotation parts. The calculation equation uses matrix operation to obtain the error between the real value and the calculated value, but the true value is unknown. The hand eye calibration error can also be obtained through the calibration data set. The re projection method obtains the hand eye relationship matrix, uses coordinate transformation to give the positioning of the calibration plate in the base coordinate system of the mechanical object, and compares the estimated value with the actual value to obtain the final result. Figure 4 is a diagram showing the relationship of the re projection method.

Figure 4.

Re projection diagram.

Points on $O_{\textit{base}}$ the calibration plate $P$ satisfy Eq. (14).

$\displaystyle P^{b}=\left[{\begin{array}[]{l}x_{t}\\ y_{t}\\ z_{t}\\ \end{array}}\right]$ (14)

In Eq. (14), $P^{b}$ it refers to the coordinates $O_{\textit{base}}$ of the $P$ points below, and the true values of different axes of the coordinate system are $t$ . If $O_{\textit{mar}}$ the coordinates in the $i$ middle are known, the $P$ second position relationship can be estimated according to the hand-eye relationship.

$\displaystyle\left[{\begin{array}[]{c}P^{b}\\ 1\\ \end{array}}\right]=\bar{A}_{i}\cdot X\cdot\bar{B}_{i}\left[{\begin{array}[]{c% }P^{m}\\ 1\\ \end{array}}\right]$ (15)

In Eq. (15), the coordinate of the $P$ point below is, and $O_{\textit{mar}}$ , $P^{m}$ the estimated value of the $P^{b}$ point pose is $P$ . Then compare the actual value and the estimated value to evaluate the hand-eye relationship matrix.

4. Improve the performance analysis of the robot hand-eye calibration algorithm in the visual task simulation test

4.1 Experimental scheme design

Probabilistic hand-eye calibration method defined by the new mean value through numerical simulation experiments. First, the simulation method is used to obtain the hand-eye calibration data set that should actually be collected by the sensor network system (because the direct use of the sensor network to collect data is costly and inefficient), and then different calibration methods are used to process the calibration data set to obtain the calculation solution, and then compare. The actual value and the calculated value, and finally the result of the calibration algorithm is evaluated by the error. The simulation experiment adopts the control variable method, and the main variables involved are the probability distribution of the calibration data set and the hand-eye calibration algorithm. Figure 5 is a flow chart of the simulation calibration experiment.

Figure 5.

Flow chart of simulation calibration experiment.

In the research, the current common methods of robot hand-eye calibration: Batch algorithm and Kronecker direct product algorithm are used as the performance comparison methods of the new mean algorithm designed in this research. Each robot hand-eye calibration algorithm is implemented through the Python programming language. After the complete data set is formed, 10% $\sim$ 100% of the samples are randomly selected according to different experimental schemes to verify the performance of each robot hand-eye calibration algorithm. Three indicators of translation error, rotation error and calculation time are used to measure the performance of each algorithm, and normal distribution and uniform distribution, which are more common in practical applications, are selected as the distribution methods of data set samples. In the study, 10 parallel experiments were carried out for each scheme.

4.2 Analysis of experimental results of performance test of improved hand eye calibration algorithm

The translation errors of the normal distribution datasets of each comparison algorithm are counted, as shown in Fig. 6. In Fig. 6, the horizontal axis is used to describe the percentage of the number of samples used in the test to the total number of samples, and the unit is %. The vertical axis is the translation error of the calibration results of each robot hand eye calibration algorithm on the normal distribution data set. Different colors and icons represent different calibration algorithms. According to the analysis of Fig. 6, with the increase of the proportion of the number of training samples, the translation error of the output results of each algorithm shows a law of fluctuating and decreasing, but the average change of the output results of each calibration algorithm is different. In addition to the new mean algorithm, the output results of other algorithms fluctuate disorderly with the increase of the proportion of the number of test samples, but the output values of the new mean algorithm show an overall decline law.

Figure 6.

Translation error statistics of normal distribution data.

Finally, when the number of test samples accounts for 100%, that is, all generated samples are used for algorithm performance test, the translation errors of batch algorithm, Kronecker direct product algorithm and new mean algorithm on the normal distribution data set are 5.61, 0.73 and 0.26 respectively. Then analyze the translation error of each calibration algorithm in the uniformly distributed data set, as shown in Fig. 7.

Figure 7.

Translation error statistics of uniformly distributed data.

The significance of the horizontal axis, vertical axis and graph in Fig. 7 is consistent with that in Fig. 6, but the data sets used in all experimental schemes shown in Fig. 7 are uniformly distributed. Through the analysis of Fig. 7, it is found that the change trend of translation error of each RHEC algorithm with the increase of the proportion of the number of test samples is consistent with the results under the condition of normal distribution data set. Only the mean amplitude of translation error of the new mean algorithm is smaller than the corresponding value under the condition of normal distribution data set. Next, the rotation error of each calibration algorithm on the test sample is analyzed. Take the normal distribution data set as an example, and the statistical results are shown in Fig. 8.

Figure 8.

Rotation error statistics of normal distribution data.

In Fig. 8, the horizontal axis is still the proportion of the number of test samples to the total samples, and the vertical axis is the rotation error of the calibration results of each RHEC algorithm on the normal distribution data set. Different colors and icons represent different calibration algorithms. Since the rotation error output by each algorithm has a small range of change, it is necessary to add lines for auxiliary analysis. It can be seen from the analysis of Fig. 8 that the rotation error of batch algorithm, Kronecker direct product algorithm and new mean algorithm on the normal distribution data set has no significant change trend. When the number of test samples accounts for 100%, the average rotation error is 10.17, 1.48 and 0.92 respectively. Finally, the rotation error of each calibration algorithm on uniformly distributed test samples is analyzed, and the calculated data is sorted out as shown in Fig. 9.

Figure 9.

Rotation error statistics of uniformly distributed data.

The meanings of the horizontal axis, vertical axis and graph in Fig. 9 are consistent with those in Fig. 8. From Fig. 9, we can see that the calculation rotation error of each algorithm for a uniformly distributed data set shows an overall decrease first and then tends to be stable with the increase of the proportion of test samples. change rules. And with the increase of the proportion of test samples, the standard deviation of the rotation errors of the parallel experiments of each algorithm gradually decreased. At this point, the practical performance analysis of each algorithm is over. With reference to the requirements of the algorithm applied to the robot hand-eye calibration task for the calculation speed of the algorithm itself, the following analysis will be conducted from the perspective of the algorithm’s operation time consumption, as shown in Fig. 10.

Figure 10.

The calculation time-consuming statistics of each hand-eye calibration algorithm.

The vertical axis in Fig. 10 is used to describe the average calculation time of the algorithm, and the unit is milliseconds. The gray dotted line represents the polynomial fitting curve of the calculation time curve of each algorithm. According to the analysis of the curve in Fig. 10, as a whole, with the increase of the number of test samples, the calculation time of each hand eye calibration algorithm shows a law of first accelerating, then decelerating, and finally increasing steadily. When the number of test samples is small, for example, when the proportion of test samples is 1%, the calculation time of the new mean algorithm designed in this study is the highest, which is 275 ms, which is 154 ms and 156 ms higher than the batch algorithm and Kronecker direct product algorithm respectively. When the test samples are all samples, the calculation time of the new mean algorithm is 2115 ms, which is 1947 ms and 143 ms less than the batch algorithm and Kronecker direct product algorithm respectively.

4.3 Feasibility and application verification of the improved hand eye calibration algorithm on the mechanical arm

Finally, the feasibility of the proposed algorithm is demonstrated. Contact a domestic manipulator manufacturer and cooperate with him to write the designed algorithm into the control program of an industrial manipulator of the company in C language. After the layout is completed, the shape of the manipulator is shown in Fig. 11. The physical test of hand eye calibration is carried out for this manipulator. The experimental results show that there is no obstacle in the control and movement of the manipulator after the algorithm is deployed, and the quality of hand eye annotation is significantly improved, which shows that the research method proposed in this study is feasible enough.

Figure 11.

Deployment feasibility verification diagram of the designed algorithm on the mechanical arm.

5. Conclusion

In order to improve the accuracy of RHEC, a new method for solving the mean value of calibration data set is proposed in this study, and a new RHEC algorithm is designed based on this method. After the test, it was found that the hand eye calibration algorithm, batch algorithm and Kronecker direct product algorithm designed in this study based on the sensor network and the new mean value had translation errors of 0.26, 5.61 and 0.73 respectively when the number of normal distribution test samples accounted for 100%. The rotation errors of the new mean algorithm, batch algorithm and Kronecker direct product algorithm are 0.92, 10.17 and 1.48 respectively when the number of normal distribution test samples accounts for 100%. The translation error and rotation error of the new mean algorithm are also significantly lower than the other two comparative hand eye calibration algorithms when the number of test samples with uniform error is 100%. From the perspective of algorithm calculation time-consuming, with the increase of the number of test samples, the calculation time-consuming of each hand eye calibration algorithm shows a law of first accelerating, then decelerating, and finally increasing steadily. When the number of test samples is small, for example, when the proportion of test samples is 1%, the calculation time of the new mean algorithm designed in this study is the highest, which is 275 ms, which is 154 ms and 156 ms higher than the batch algorithm and Kronecker direct product algorithm respectively. When the test samples are all samples, the calculation time of the new mean algorithm is 2115 ms, which is 1947 ms and 143 ms less than the batch algorithm and Kronecker direct product algorithm respectively. The experimental data show that the coordination accuracy of RHEC based on the new mean value proposed in this study is significantly higher than the common hand eye calibration algorithm, and the calculation speed is faster in large data application scenarios. However, due to limited research conditions, the design method was not deployed to industrial manipulator products for performance testing this time, which is also a research area that needs to be concerned in the future.

Footnotes

Funding

The research is supported by: Research on some key technologies of precision inspection based on machine learning, No.: 51765007; Research on active scheduling method of processing job based on big data, No.: 51675186; Research on coordinated motion control of mobile lower limbs exoskeleton rehabilitation robot based on multi-sensor information fusion, No.: 81960332.

References

Jiang

Gao

. An improved robot calibration method using the modified adjoint error model based on POE. Adv Robotics. 2020; 34(19): 1-10.

Idrobo-Pizo

Motta

Sampaio

. A calibration method for a laser triangulation scanner mounted on a robot arm for surface mapping. Sensors. 2019; 19(8): 1783-1802.

Wang

Jiang

Zhu

Ding

. Simultaneous calibration of multicoordinates for a dual-robot system by solving the AXB = YCZ problem. IEEE T Robot. 2021; 37(4): 1172-1185.

Wang

Jia

Chen

Liu

. Complete relative pose error model for robot calibration. Ind Robot. 2019; 46(3): 622-630.

Ali

Suominen

Gotchev

Morales

. Methods for simultaneous robot-world-hand-eye calibration: a comparative study. Sensors. 2019; 19(12): 2837-2852.

Dong

Gao

. A robot hand-eye calibration method of line laser sensor based on 3D reconstruction. Robot CIM-INT Manuf. 2021; 71: 102136-102137.

Zhao

. Simultaneous robot-world and hand-eye calibration by the alternative linear programming. Pattern Recogn Lett. 2019; 127(11): 174-180.

Zou

Chen

. Hand-eye calibration of arc welding robot and laser vision sensor through semidefinite programming. Ind Robot. 2019; 45(5): 597-610.

Zheng

Zhang

Luo

Wang

. Hand-eye calibration of welding robot based on the constraint of spatial line. Trans China Weld Inst. 2018; 39(8): 108-113.

10.

Tan

Ren

. Simultaneous robot-world, sensor-tip, and kinematics calibration of an underactuated robotic hand with soft fingers. IEEE Access. 2018; 6(12): 22705-22715.

11.

Sharifzadeh

Biro

Kinnell

. Robust hand-eye calibration of 2D laser sensors using a single-plane calibration artefact. Robot CIM-INT Manuf. 2020; 61: 1-10.

12.

. Simultaneous and on-line calibration of a robot-based inspecting system. Robot CIM-INT Manuf. 2018; 49: 349-360.

13.

Wang

Tao

Gong

Yin

. A mobile robotic measurement system for large-scale complex components based on optical scanning and visual tracking. Robot CIM-INT Manuf. 2021; 67(2): 102010-102011.

14.

Yam

Wang

Zhang

. N-doped ZnO nanosheets: towards high performance two dimensional catalysts. Nanotechnology. 2018; 29(10): 1-3.

15.

Zhu

Song

Lim

Ren

. Safety-enhanced model-free visual servoing for continuum tubular robots through singularity avoidance in confined environments. IEEE Access. 2019; 7(1): 21539-21558.

16.

Sentenac

Bugarin

Ducarouge

Devy

. Automated thermal 3D reconstruction based on a robot equipped with uncalibrated infrared stereovision cameras. Adv Eng Inform. 2018; 38: 203-215.

17.

Guo

Xiao

Zhang

Hao

. A tool centre point calibration method of a dual-robot NDT system for semi-enclosed workpiece testing. Ind Robot. 2019; 46(2): 202-210.

18.

Liu

Zhu

Dong

. An improved kinematic model for serial robot calibration based on local POE formula using position measurement. Ind Robot. 2018; 45(5): 573-584.

19.

Zhang

Qiu

Zhang

. Calibration method for hand-eye system with rotation and translation couplings. Appl Optics. 2019; 58(20): 5375-5387.

20.

Liu

. Robust and accurate hand-eye calibration method based on schur matric decomposition. Sensors. 2019; 19(20): 4490.

21.

Majeed

Karkee

Zhang

. Development and performance evaluation of a machine vision system and an integrated prototype for automated green shoot thinning in vineyards. J Field Robot. 2021; 38(6): 898-916.

22.

Eshkevari

Rezaee

Zarinbal

. Automatic dimensional defect detection for glass vials based on machine vision: A heuristic segmentation method. J Manuf Processes. 2021; 68(8): 973-989.

23.

Ali

Atia

. A lead through approach for programming a welding arm robot using machine vision. Robotica. 2021; 40(3): 464-474.

24.

Lin

Wang

. Intelligent detection of edge inconsistency for mechanical workpiece by machine vision with deep learning and variable geometry model. Appl Intell. 2020; 50(7): 2105-2119.

25.

Kim

Noh

Park

. Efficient neural network compression via transfer learning for machine vision inspection. Neurocomputing. 2020; 413(11): 294-304.

26.

Huang

Zhang

. A weak supervision machine vision detection method based on artificial defect simulation. Knowl-Based Syst. 2020; 208(4): 106466.1-106466.10.