Methods of detection by single camera for target pose in visual sorting

Abstract

In the process of visual sorting, in order to solve the problem that the irregular contour target position is difficult to locate quickly and accurately, and it is difficult to realize the self-defined grab point, a method of target pose detection based on the contour center of gravity edge distance vector and local polar coordinates is proposed. Firstly, the target contour is determined by adaptive edge detection and the comparison of contour moments, and the larger distance direction from the contour center of gravity to the smallest outer rectangle is defined as the direction from the contour center of gravity to the edge distance vector. The deflection angle of the target is determined by the vector direction and the position of the outer rectangle to detect the deflection angle of the irregular workpiece. On this basis, judge whether the designated point is the relative center of gravity in the local polar coordinate system, and realize the intelligent positioning of the set point on the target by detecting the coordinates of the designated point in the center of gravity positioning. The experimental results show that the method is suitable for complex contour targets, and the positioning results are accurate and the positioning time is short, so it has high engineering application value.

Keywords

Edge distance vector declination detection attitude detection relative center contour recognition fixed point positioning

1 Introduction

Intelligent visual sorting used the camera to realize the function of the human eye, it determined the information of the target position by image processing method, and then controlled the robot to complete the specified operation process [6]. Intelligent visual sorting has enabled humans to get rid of the tedious work of high repetition, improve the accuracy of sorting and improve productivity, so it has been widely used in the industry about quality inspection of electronics, selection of parts, sorting and packaging of food, etc. [11 –13]. The confirmation of the posed information about the target in image processing plays an important role in the stability of the intelligent visual sorting system, which determines the speed and precision of the sorting system.

The posed information of the target includes the two meanings of position and deflection angle, and the existing method of image processing can be generally divided into two categories: three-dimensional spatial posed location and two-dimensional spatial posed location.

Regarding posed location of the target in three-dimensional space, [10] created a machine learning algorithm of regression forest by using HOG features on a single image to achieve three-dimensional spatial location of the target position. [16] used artificial neural network to train a large number of samples to complete the spatial positioning of the workpiece feeding process; Huang [4] used a three-coordinate measuring machine and a monocular camera to achieve stereoscopic part measurement by matching the same name point of the edge image centroid offset. [9] proposed the method of combining five laser beams with a CCD camera to achieve the spatial posed location of free-form surface through light point cross-curve information. After analysis, the research of the existing target 3D spatial positioning is either the algorithm is complex or needs to be completed with other sensors, the real-time is poor and the cost is high. In addition, there are many products that need to be sorted on the current production line. The thickness of the products is thin and the specifications are the same, such as the inspection of PCB boards and the sorting and packing of food packaging bags. Therefore, sorting can ignore the height of the target, only to obtain two-dimensional spatial position information has been able to meet the sorting needs. [1] extracted the welding stud feature information by the monocular vision image, and realized the measurement of parameters about the stud position through the calculation model; [17] used the vector of linear features and shape features to identify and locate the workpiece; [7] used the combination of accelerated robust features and target data to determine the robust center point to achieve target localization. [14] used the method of Hu moment and SVM machine learning to identify simple contour targets and calculate the size parameters. When the above-mentioned algorithm for the measurement of flat space position is applied to the sorting system, it is only suitable for simple-looking workpieces (the outline consists of a straight line or circle) and cannot be artificially specified for special grip points, which is inflexible.

In order to expand the applicability of the visual sorting system to different products, and solve the problem of quickly and accurately locating irregular target position, an algorithm is proposed to calculate the contour center of gravity by applying contour moment in the single-view image, and to determine inspection of deflected angle about the target by analyzing the contour center of gravity to the length of the outer rectangular margin. Considering that the pick-up point of the target is set manually under special circumstances, the method of local polar coordinates is used to detect the deflection angle of irregular workpiece. On this basis, if the specified point is the relative center of gravity in the local polar coordinate system, the intelligent positioning of the set point on the target can be realized by detecting the coordinates of the specified point of the center of gravity positioning. Based on the conveyor belt as the background, the attitude determination method is verified to have high accuracy, robustness and engineering practicability by repeating the attitude tests of several key points under different attitudes.

2 The process of pose detection about two-dimensional space targets

The process for posed detection of target based on the vector of the contour center-of-gravity edge distance and local polar coordinates is shown in Fig. 1.

Fig. 1

Process of target pose detection method.

First of all, the RGB color image obtained by the camera is processed and analyzed, and the external outline, which have a good robustness and embody the target feature, is taken as the matching template, and the function of recognition is realized by comparing the Hu moment of the target profile with the template profile. After the target is confirmed, the center of gravity about contour is calculated by using the contour moment, the vector from the center of gravity to the margin is confirmed by comparing distance about the center of gravity and two short edges of the smallest outer rectangle about the outline. Finally, the spatial relative position of the specified point and the center of gravity about contour is confirmed in the template, coordinates about the center of gravity are found in process of the target recognition, and the position of the specified point on the target is confirmed by the geometry of the bureau pole coordinates, so as to complete the detected process of the entire target position.

3 Pre-processing image and extraction and matching of contour

3.1 Grayscale noise reduction

The detection of feature contours is based on grayscale images, so the RGB images captured by the camera are used by formula (1) for grayscale processing. $g = 0.299 \times R + 0.587 \times G + 0.114 \times B$ (1)

The surface of the conveyor belt is affected by factors such as floating dust and texture reflection. The collected images will show some bright spots like impulse noise, and these bright spots will affect the quality of edge detection. In response to this phenomenon, the grayscale image is denoised to suppress the impulse noise by Median filtering. This experiment uses a 1040×780 resolution image. It can be seen from the Fig. 2 that the template size is ideal for 7×7 filtering.

Fig. 2

Grayscale image and oise reduction of image by median filtering.

3.2 Adaptive edge detection

The edge detection of the image uses the method of adaptive edge detection about CoG (Canny of a gamma) [5]. Applying gamma correction to the median filter image for contrast enhancement, the formula for correction is: $y = (x + esp)^{γ}$ (2)

In formula (2), the choice of the value about γ enhances the target contrast and refines the details of the target edge. In order to make the belt in the gray scale into a low gray scale, the workpiece (key) appears as a high gray scale, and γ = 2 is determined experimentally (when γ > 1, the high gray area of the image is strengthened, and the low gray area is weakened).

Edge detection is performed on the workpiece by using a double threshold Canny operator to obtain a gamma corrected image. Considering that there may be a slight break in the key contour detected by the CoG method, a template of 7×7 closing operation is performed after this method to ensure that the key contour is continuous. The effect of adaptive edge detection by CoG and processing effect of closed operation is shown in Fig. 3. It can be seen from Fig. 3 that the combination of CoG edge detection and closed operation keeps the outer contour of the target continuous and complete, and the features of the outer contour have a good robustness and recognition of feature.

Fig. 3

Adaptive edge detection by CoG and processing effect of closed operation.

3.3 Extraction and matching of feature contour

In the template image, the area of the outer contour about the key must be the largest, and the outer contour feature with stability can be extracted by comparing the size of each edge envelope. However, because of the complexity of the background environment in the process of actual target location, the largest contour is not necessarily the target of acquisition. In this case, the template matching algorithm needs to be introduced to obtain the appropriate target object. In this paper, the method of Hu moment matching with rotation and scaling invariance [8] is used to match and analyze the target contour and the template contour for the task of target recognition. The extracted outer contour about key is shown in Fig. 4.

Fig. 4

The outer outline used to match the feature.

4 The deflected angle of the target by vector of the center-of-gravity edge distance

4.1 Get the contour center of gravity and the minimum external rectangle

The center of gravity of the contour can be calculated from the contour moment, and the (p,q) moment of the contour is: $mp, q = \sum_{x, y} I (x, y) x^{p} y^{q}$ (3)

p qare the moments in the x, y dimension, and the coordinates of the center of gravity about the contour for G (x_G, y_G) are obtained: $[\begin{matrix} x_{G} \\ y_{G} \end{matrix}] = [\begin{matrix} m_{10} / m_{00} \\ m_{01} / m_{00} \end{matrix}]$ (4)

In visual sorting, if the outline of the grab target is irregular, the judgment of the corner directly is not conducive to quantitative description. Refer to [2 , 18] for the minimum contour of the target contour, the angle of contour is determined by combining the pose of the circumscribed rectangle with the position of the center of gravity G of the contour, and the minimum circumscribed rectangle is shown in Fig. 5.

Fig. 5

Determine the outline minimum external rectangle.

4.2 Determine the direction about the vector of the contour center-of-gravity edge distance

In this experiment, the minimum circumscribed rectangle is made for the extracted key contour, and a Cartesian coordinate system as shown in Fig. 6 is established (the coordinate system origin o is the upper left corner of the image).The corner point near the y-axis is set to p0, and the other three corner points are p1, p2, p3, respectively. If the line segment p0p1 is perpendicular to the x-axis, the upper corner point is p0.When the circumscribed rectangle is determined, the lengths of p0p1 and p1p2 are h and w, respectively, the centroid point of the rectangle is A (x_A, y_A), the center of gravity of the contour is G (x_G, y_G), and the angle between the rectangle and the x-axis is β. Since the irregular contour is asymmetrical with the mid-perpendicular line of the line segment p1p2, the distance between the center of gravity G and two short sides of the rectangle is not equal (S_G01 ≠ S_G23), Let one side of Max (S_G01, S_G23) be the positive direction of the key contour, that is the direction of $\vec{GF}$ (the vector of the contour center-of-gravity edge distance). The direction about the vector of the contour center-of-gravity edge distance for $\vec{GF}$ can be determined to calculate the angle of the contour relative to the x axis.

Fig. 6

Coordinates determined by the target profile vector of the contour center-of-gravity edge distance.

The vector of the contour center-of-gravity edge distance for $\vec{GF}$ is solved as follows:

– Determine the short side of the rectangle and compare the lengths of p0p1 and p1p2. The p0 coordinate can be obtained from the geometric relationship of Fig. 6. $x_{p 0} = x_{A} - \frac{1}{2} (h sin β + w cos β)$ (5) $y_{p 0} = y_{A} + \frac{1}{2} (w sin β + h cos β)$ (6)

The coordinates of p1: $x_{p 1} = x_{p 0} + h sin β$ (7) $y_{p 1} = y_{p 0} + h cos β$ (8)

The coordinates of p2: $x_{p 2} = x_{p 1} + w cos β$ (9) $y_{p 2} = y_{p 1} + w sin β$ (10)

The length of p0p1: $L_{01} = \sqrt{(x_{p 1} - x_{p 0})^{2} + (y_{p 1} - y_{p 0})^{2}}$ (11)

The length of p1p2: $L_{12} = \sqrt{(x_{p 2} - x_{p 1})^{2} + (y_{p 2} - y_{p 1})^{2}}$ (12)

Compare the size of L01 and L12 and determine the two short side positions of the circumscribed rectangle.

– Judging the direction of the larger distance for the center of gravity to the short side, that is, comparing the distance between the two short sides and the point G for S_G01 and S_G23.

Let the short side of the rectangle be p0p1, and the slope of the line p0p1 as follows. $k_{01} = \frac{y_{p 1} - y_{p 0}}{x_{p 1} - x_{p 0}}$ (13)

Intercept of line p0p1: $b_{01} = y_{p 0} - k_{01} x_{p 0}$ (14)

General formula of the equation about the line p0p1: $k_{01} x - y_{01} + b_{01} = 0$ (15)

The distance of the line p0p1 and the point of the center of gravity for G (x_G, y_G): $S_{G 01} = | \frac{k_{01} x_{G} - y_{G} + b_{01}}{\sqrt{k_{01}^{2} + 1}} |$ (16)

Because the rectangle is an axisymmetric figure, if $S_{G 01} < \frac{1}{2} p 1 p 2$ , the positive direction for $\vec{GF}$ is shown in Fig. 6. If not, the opposite direction is $\vec{GF}$ .

In addition, if p0p1 is the long side of the rectangle, similarly, the distance from G (x_G, y_G) to p1p2 can be obtained. $S_{G 12} = | \frac{k_{12} x_{G} - y_{G} + b_{12}}{\sqrt{k_{12}^{2} + 1}} |$ (17)

4.3 Determine the deflected angle of the target

The direction of $\vec{GF}$ , the position of the rectangle and the center of gravity point G can be shown as the four cases shown in Fig. 7.

Fig. 7

Contour deflected angle in four different cases.

The corner of the profile relative to the x-axis:

When L₀₁ ⩽ L₁₂, $S_{G 01} ⩽ \frac{1}{2} L_{12}$ , α = β.

When L₀₁ ⩽ L₁₂, $S_{G 01} ⩾ \frac{1}{2} L_{12}$ , α = β-π.

When L₀₁ ⩾ L₁₂, $S_{G 12} ⩽ \frac{1}{2} L_{01}$ , $α = β + \frac{π}{2}$ .

When L₀₁ ⩾ L₁₂, $S_{G 12} ⩾ \frac{1}{2} L_{01}$ , $α = β - \frac{π}{2}$ .

When x_p0 = x_p1, the slope of line p0p1 does not exist, and S_G01 or S_G12 is the x coordinate difference or the y coordinate difference between the point G and the point p1.

According to the above calculation process, the rotation angle of the template contour with respect to the x-axis for α_model and the rotation angle of the target contour with respect to the x-axis for α_objective can be calculated in real time, and then the difference operation can be used to obtain the rotation angle of the target relative to the template in the actual demand, as shown in the formula (18): $α_{O - M} = α_{objective} - α_{mod el}$ (18)

5 Determine specified point coordinates by method of polar vector

5.1 Determine the location of the specified point in the local polar coordinates

For the target with uniform mass distribution, in order to ensure the balance of the object during the movement, the grab point is set to the center of gravity of the contour, but in special cases, the grab point can be set artificially.

When the template object position is shown in Fig. 8, the local right-angle coordinates MGN (GM parallel with ox) and local polar coordinates for $\vec{GF}$ are respectively established with the center of gravity G as the coordinate origin. Let $C (x_{C}^{'}, y_{C}^{'})$ be the artificial selection point in the template, calculate the length of $\vec{GC}$ and the angle of $\vec{GC}$ and $\vec{GF}$ in polar coordinates $\vec{GF}$ . Therefore, in the target recognition, the position of the specified point C in the target image can be accurately located according to the direction about the vector of center-of-gravity edge distance for $\vec{GF}$ and the relative position of the C point in the polar coordinates $\vec{GF}$ .

The center of gravity about the contour for $G (x_{G}^{'}, y_{G}^{'})$ is calculated by the contour moment, and the angleα of the vector of center-of-gravity edge distance for $\vec{GF}$ relative to the x-axis is also a known amount, the length of $\vec{GC}$ in the coordinate system xoy by the geometry of the plane in Fig. 8 can be expressed as:

Fig. 8

Determine the location of the specified point in the local polar coordinates.

$∥ \vec{GC} ∥ = \sqrt{(y_{C} - y_{G})^{2} + (x_{C} - x_{G})^{2}}$ (19)

In the coordinate system MGN, if point C is in one or four quadrants, the angle of $\vec{GC}$ and GM $α + θ = arctan (\frac{y_{C} - y_{G}}{x_{C} - x_{G}})$ (20)

If point C is in the two and three quadrant, the angle of $\vec{GC}$ and GM: $α + θ = arctan (\frac{y_{C} - y_{G}}{x_{C} - x_{G}}) + π$ (21) so the angle θ of $\vec{GC}$ and $\vec{GF}$ : $θ = arctan (\frac{y_{C} - y_{G}}{x_{C} - x_{G}}) - α$ (22) Or $θ = arctan (\frac{y_{C} - y_{G}}{x_{C} - x_{G}}) + π - α$ (23)

5.2 Determine the location of the specified point by the local polar coordinates

If the direction of the center of gravity about target contour for G (x_G, y_G) and the vector of center-of-gravity edge distance for $\vec{GF}$ has been determined in the target image, the coordinates of the specified point C (x_C, y_C): $x_{C} = x_{G} + \sqrt{(y_{C}^{'} - y_{G}^{'})^{2} + (x_{C}^{'} - x_{G}^{'})^{2}} cos (θ + α)$ (24) $y_{C} = y_{G} + \sqrt{(y_{C}^{'} - y_{G}^{'})^{2} + (x_{C}^{'} - x_{G}^{'})^{2}} sin (θ + α)$ (25)

6 Experimental results and analysis

In order to verify the effectiveness of the proposed “Methods of detection by single camera for target pose in visual sorting”, the following experiments are designed.

First of all, a large number of images for cluttering keys are collected under the environment of a conveyor belt. In the process, the visual software used was designed to process data in a personal desktop computer with 8 G memory and a main frequency of 3.6 GHz, which was designed in combination with the Emgu CV library and C # language. The resolution of the CCD camera is 1040×780, millimeters: pixels = 1 : 5.

Table 1 is the experimental data of images randomly selected during the experiment.

Table 1
Algorithm experiment and performance evaluation

Test items Similar key 1 Similar key 2 Similar key 3 Other key

The actual coordinates of the specified point X(mm) 109.7 60.3 160.3 90.1

Detection coordinates of specified points X(mm) 109.3 60.6 159.8 –

Detection coordinates of specified points Y(mm) 72.0 96.4 63.2 23.6

Detection coordinates of specified points Y(mm) 71.5 96.8 63.4 –

The actual angle of the goal and the template (Radian) 0.163 –2.317 0.963 0.924

The detection angle of the target and the template (Radian) 0.155 –2.323 0.945 –

Processing time (ms) 8 8 8 8

Similarity 0.017 0.012 0.020 0.12

Test items	Similar key 1	Similar key 2	Similar key 3	Other key
The actual coordinates of the specified point X(mm)	109.7	60.3	160.3	90.1
Detection coordinates of specified points X(mm)	109.3	60.6	159.8	–
Detection coordinates of specified points Y(mm)	72.0	96.4	63.2	23.6
Detection coordinates of specified points Y(mm)	71.5	96.8	63.4	–
The actual angle of the goal and the template (Radian)	0.163	–2.317	0.963	0.924
The detection angle of the target and the template (Radian)	0.155	–2.323	0.945	–
Processing time (ms)	8	8	8	8
Similarity	0.017	0.012	0.020	0.12

The experimental results show that the proposed target position detection method has good adaptability to the impurities and reflection phenomena existing on the conveyor belt, and the recognition accuracy is as high as 98.6%. And the average positioning deviation of the method for positioning detection is 0.5 mm, the average angular deviation is 0.02 arc, and the repeat positioning accuracy is 0.005 arc, which meets the accuracy requirements of ordinary sorting products. Observing the time-consuming situation of the positioning detection process, it can be found that the average time-consuming of this method is 8 ms, which has strong real-time performance.

Figure 9 shows several typical pose detection results in the experiment. It can be seen that the method has a good detected effect on the grab point and deflected angle of the target (Fig. 9(a)), and the grip point can be flexibly set. (Fig. 9(b)). It still has good detected ability under strong illumination (Fig. 9(c)) and interference target (Fig. 9(d)). It can adapt to the target of complex contours in engineering applications and has high robustness.

Fig. 9

Several typical pose detection results.

7 Conclusion

In this paper, the method of detecting the target pose based on the combination of the vector of the contour center-of-gravity edge distance and the local polar coordinate is proposed, and the fast and accurate pose detection of the irregular target is realized. The direction about the vector of the center-of-gravity edge distance is proposed to determine the deflected angle of the target, which solves the problem of the inspection of deflected angle about irregular work pieces. The detection of real-time positioning about custom points is solved by using the local polar coordinate system with margin vector as the positive direction to refer to the center of gravity about the contour. In the process of target angle detection, the algorithm uses simple geometric operation, so the processing speed is improved well.

After analysis of a large number of experimental results, it is known that this method still has some shortcomings in the detection of overlapping targets, such as the difficulty of automatic segmentation recognition. Therefore, in the next research stage, the method in this paper will be further optimized from the perspective of automatic segmentation.

Footnotes

Acknowledgments

The study was supported by “Science Research Project of Liaoning Education Department, China (No. JDL2016026)”.

References

and Zhang

, Positon and attitude parameter measurement of stud welding based on monocular vision, Journal of Optoelectronics·Laser25(10) (2014), 1931–1935.

, Wang

, Mi

, Huang

, Zhang

, Yang

, Jiang

and Octavian

, Research on regional clustering and two-stage svm method for container truck recognition, Discrete and Continuous Dynamical Systems Series S12(4–5) (2019), 1117–1133.

Zhang

F.Q.

, Wang

G.F.

and Zeng

Q.N.

, New algorithm for minimum enclosing rectangle of the object in the image region based on center-of-gravity principle, Infrared and Laser Engineering42(5) (2013), 1382–1387.

Huang

F.S.

, Liu

E.F.

, Fang

Y.X.

and Han

, Single camera stereo vision recognization for Parts’ pose based on intelligent three coordinate measuring machine, Optics and Precision Engineering21(5) (2013), 1326–1332.

Fei

, Xie

, Li

, Wang

and Chen

, Adaptive license plate correction and extraction based on character feature constraint, Chinese Journal of Scientific Instrument37(3) (2016), 632–639.

Guo

, Luo

and Zhang

, Machine vision and applications, Electronic Science and Technology34(7) (2014), 185–188.

Song

K.T.

and Chang

C.H.

, Object pose estimation for grasping based on robust center point detection, Proceedings of 2011 8th Asian Control Conference (ASCC)5 (2011), 305–310.

M.K.

, Visual pattern recognition by moment invariants, IRE Transactions on Information Theory,8(2) (1962), 179–187.

Lee

R.T.

and Shiou

F.J.

, Multi-beam laser probe for measuring position and orientation of freeform surface, Measurement44(1) (2011), 1–10.

10.

Shimizu

, Koyasu

, Kobayashi

and Kuno

, Object pose estimation using category information from a single image, Workshop on Frontiers of Computer Vision4(4) (2015), 598–607.

11.

Sudhakar

, Francis

and Balaji

, Odd mean labeling for two star graph, Applied Mathematics & Nonlinear Sciences2(1) (2017), 195–200.

12.

S.M.

and Huang

X.H.

, Review of development and application of industrial robot technology, Journal ofMechanical & Electrical Engineering32(1) (2015), 1–13.

13.

Lokesha

, Shruti

and Deepika

, Reckoning of the dissimilar topological indices of human liver, Applied Mathematics & Nonlinear Sciences3(1) (2018), 265–276.

14.

Chen

and Fan

L.Q.

, Research and application of workpiece detection and recognition based on computer vision technology, Mechanical & Electrical Integration2 (2016), 8–11, 30.

15.

Gao

and Wang

W.F.

, The fifth geometric-arithmetic index of bridge graph and carbon nanocones, Journal of Difference Equations and Applications23 (2017), 100–109.

16.

Murray

W.R.

and Billingsley

, Neural-network-based pose recognition in flexible parts feeders: preliminary results for reduced training data, IEEE/ASME International Conference on Advanced Intelligent Mechatronics (8) 1999, 404

17.

Tang

, Wu

Q.X.

and Zhu

, Methods research of plane recognition and localization based on regional division, Machinery Design & Manufacture10 (2015), 172–175.

18.

Chen

Z.N.

, Zhang

, Peng

Z.R.

and Zhong

K.Y.

, Workpiece location and recognition based on machine vision, Electronic Science and Technology29(4) (2016), 99–103.