Bolting Cabin Assistance System Using a Sensor Network

Abstract

The bolting cabin assistance system prevents operators from facing dangerous situations. This system consists of a bolting robot control system and a top view supervisory system. In order to control the bolting robot, circular Hough transforms and fuzzy reasoning are used. First, the circular Hough transform roughly estimates the location of the bolt hole. After that, errors of estimation are compensated for using fuzzy reasoning. In order to track a bolt hole, a region of interest (ROI) is used. By setting the region in which to search for a bolt hole, the algorithm tracks the location of the bolt hole. In order to choose an ROI, a template-based matching algorithm is used. In order to make the top view supervisory system, four cameras are installed at the left, right, front and back of the robot. The four individual images from the various cameras are combined to make the top view image after correcting for distortion.

Keywords

Visual Servo Fuzzy Reasoning Circular Hough Transform Template-Based Matching Distortion Correction

1. Introduction

In the construction field, workers can encounter many dangerous situations. The bolting robot helps workers work safely and can reduce the prevalence of such hazardous situations. This paper presents the concept of the bolting cabin and proposes the bolting robot control algorithm using circular Hough transforms and template matching to estimate the location of the bolt hole, an error compensation algorithm using fuzzy reasoning and a top view supervisory system that can observe the cabin from multiple angles. The bolting robot control system helps workers in assembling bolts safely. In vision-based bolt hole tracking, “bolt hole detection” is the key technique to estimate the location of the bolt holes. Circular Hough transforms and template-based matching are used to identify circles in the images. The circular Hough transform belongs to the set of feature-based matching algorithms. Feature-based matching uses the edges of shapes and intensity of particles to recognize an object. However, these features are not recognized by a camera in low illumination conditions. In contrast to feature-based matching, even though template-based matching is not affected by illumination conditions, it cannot perform accurate recognition. To overcome such shortcomings, this paper suggests an illumination-robust detection algorithm using the mixed matching algorithm. The mixed matching algorithm has merits compared to feature-based matching or template-based matching. As the raw image is affected by radial distortion, it is difficult to extract the bolt hole from the raw image and to estimate the location of the bolt hole. To deal with these problems, fuzzy reasoning is used.

The top view supervisory system provides video imaging that can observe the cabin from various angles. As previously mentioned, the raw image is distorted. In this paper, a radial distortion correction algorithm, a perspective distortion correction algorithm and an image combination method are proposed.

2. Related work

Various robots have been developed for construction, such as a fireproofing spray robot, robot crane and a tool-developing robot [1 -3]. An approach to modularize house-building and an assembly robot have been proposed, and a large-scale unmanned construction project was tested in Japan [4,5]. Since the development of high-tension bolts, the bolting method has been preferred to construct high-rise steel structures. However, it is difficult to automate the bolting process. Consequently, human workers still climb the vertical steel beams on top of high-rise steel structures at construction sites, as shown in Figure 1. The manual process is time-consuming, inefficient and extremely dangerous. To resolve this problem, bolting cabins and bolting cabin assistance systems are proposed in this research which can provide an automated bolting process.

Figure 1.

Bolting work in high places 3. Concept of the bolting cabin

3. Concept of the bolting cabin

The bolting cabin system consists of a bolting robot and cabin. The cabin is a mobile robot that moves on a construction site and a bolting robot is a tool that performs bolting work. To perform bolting work, the bolting robot control system was developed, and to guarantee the safety of workers in the cabin, the workers can observe the area around the cabin using a top view supervisory system. Figure 2 shows the bolting cabin system.

Figure 2.

Bolting cabin system

4. Bolting robot control system

4.1 Organization of the bolting robot

The bolting robot consists of a camera, laser range finder, gantry robot and bolting tool. Figure 3 shows the bolting tool system. Using the camera, the bolt hole is extracted from the image. Using the extracted bolt hole location and the laser range finder, the gantry robot can calculate the appropriate position for fitting the bolt. Once the gantry robot is in position, the bolting tool performs the bolt-related operations.

Figure 3.

Organization of the bolting robot

Figure 4.

Template-based matching

Figure 5.

Estimation of the bolt hole radius

4.2 Template-based matching

Template-based matching is one of several pattern recognition techniques used to find a desired object that is similar in size and shape to a given template. For the similarity measurement, the sum of absolute differences (SAD) and the sum of squared differences (SSD) are widely used in various applications [6,7]. In addition, the normalized cross correlation (NCC) measurement is also popular for similar measurements. NCC is more robust than SAD and SSD changes under illumination [8]. In this paper, an NCC algorithm is used.

4.3 Circular Hough Transform

The Hough transform is a feature extraction technique used in image analysis, computer vision and digital image processing [8]. The purpose of this technique is to find imperfect instances of objects within a certain class of shapes by using a voting procedure. This voting procedure is carried out in a parameter space, from which object candidates are obtained as local maxima in a so-called accumulator space, which is explicitly constructed by the algorithm for the computation of the Hough transform. The classical Hough transform was concerned with the identification of lines in images, but later, the Hough transform was extended to locate the positions of arbitrary shapes, most commonly circles or ellipses. The circular Hough transform has been widely used to extract circles and ellipses. Its main disadvantage is that the computational complexity of the algorithm increases according to the dimensions of a shape. If the radii of the circles are known, the complexity of the algorithm can be reduced [9]. Using a laser range finder, we can calculate the radii of the circles.

4.4 Estimation of the radius of a bolt hole

R is the radius of a circle in an image plane. We should estimate R. Let f, d and r denote the focal length (f), the camera centre to frame distance (d) and the radius (r) of the hole in the steel frame. We can obtain reliable data for each parameter using the laser range finder. We can derive a proportional Eq. (1) with given parameters f, d and r.

R = r • \frac{f}{d}

(1)

4.5 Compensation of lens distortion

Images captured by real cameras suffer from lens distortion, which is a blend of perspective and radial (barrel) distortion [10]. Distortion is inherent in all images taken through lens systems [11]. These kinds of distortions cause errors when images are processed. In this paper, radial distortion is taken into consideration.

To solve this problem, image stretching is generally used. Figure 6 shows a distorted and corrected image.

Figure 6.

Distorted and distortion-corrected image

However, when using a corrected image, an additional calculation is needed, thereby slowing down the total processing speed. Moreover, it decreases the image resolution and contrast of the raw image.

In order to compensate for lens distortion, this paper suggests fuzzy reasoning based on Hartley's radial distortion model [12].

\hat{x} = x_{c} + L (r) (x - x_{c})

(2)

\hat{y} = y_{c} + L (r) (y - y_{c})

(3)

\begin{matrix} (x, y) : m e a s u r e d p o i n t \\ (\hat{x}, \hat{y}) : c o r r e c t e d p o i n t \\ (x_{c}, y_{c}) : c e n t r e o f b a r r e l d i s t o r t i o n \\ r = \sqrt{{(x - x_{c})}^{2} + {(y - y_{c})}^{2}} \end{matrix}

In Eqs. (2) and (3), distortion coefficient L(r) is determined by r(r >= 0, L(0) = 1). The distortion coefficient L(r) is approximated by the Taylor expansion [13].

L (r) = 1 + k_{1} r + k_{2} r^{2} \dots

(4)

Distortion coefficient L(r) decreases with the distance from the centre.

Figure 7 shows circle distortion caused by barrel distortion. In the centre of the image, the circle is not distorted, however, circle distortion increases with the distance from the centre. This results in an inaccurate distance calculation that can lead to inaccurate input data for the robot.

Figure 7.

Circle distortion caused by barrel distortion

The best way to compensate for circle distortion is to estimate the error at each pixel. Error means the difference between the distance calculated using Eq. (1) and real distance from the image centre to the centre of the circle. However, this approach is too time- and effort-intensive to be practical.

Thus, another approach is suggested, using fuzzy reasoning [14 -16].

First, calculate L(r) using previous research results [17]. Second, divide the image appropriately by L(r) and estimate the distance error within each divided section. Figure 8 shows an image divided by L(r) and Tables 1 and 2 show L(r) and the estimation error at each line.

Table 1.

Left side vertical line

Location (Pixel)	0 (long)	163 (mid)	208 (short)	270 (near)	320 (centre)
L(r)	0.78	0.86	0.91	0.93	1
Error of X (Pixel)	17	12	7	5	0

Table 2.

Upper side horizontal line

Location (Pixel)	0 (long)	111 (mid)	153 (short)	217 (near)	240 (centre)
L(r)	0.84	0.89	0.93	0.95	1
Error of Y (Pixel)	15	9	5	3	0

Figure 8.

Image (640 × 480) divided by L(r)

L(r) and the error of each line are symmetric with respect to the origin. The error of a pixel refers to the difference between the location of a line in the distorted image and the location in the real image. Using these results, the error of every pixel is estimated using fuzzy reasoning. Figure 9 and Table 3 show the use of the fuzzy membership function and the fuzzy rule (left upper side). The estimated distance of a point to its origin is used as a fuzzy set.

Table 3.

Fuzzy rules for pixel compensation

Vertical Horizontal	Long	Mid	Short	Near	Centre
Long	(17,15)	(12,15)	(7,15)	(5,15)	(0,15)
Mid	(17,9)	(12,9)	(7,9)	(5,9)	(0,9)
Short	(17,5)	(12,5)	(7,5)	(5,5)	(0,5)
Near	(17,3)	(12,3)	(7,3)	(5,3)	(0,3)
Centre	(17,0)	(12,0)	(7,0)	(5,0)	(0,0)

Figure 9.

Fuzzy membership function

4.6 Experiment and result

Figure 10 is a block diagram that shows the complete bolt hole tracking process.

Figure 10.

Block diagram of the bolt hole tracking process

Experiments were done using Template Matching (TM), circular Hough transform (CHT), compensated circular Hough transform (CCHT) and mixed matching (MM) algorithm. Figure 11 shows the extracted shapes of the bolt hole circles and their relative distances.

Table 4 contains the difference between the real distance and the estimated distance between the centres of the two circles.

Table 4.

Estimated and real distances

	TM	CHT	CCHT	MM
Estimation (mm)	69.312	67.821	67.141	67.201
Real (mm)	67	67	67	67
Error (mm)	2.321	0.821	0.141	0.201

Table 5 shows the difference between the real diameter and the estimated diameter of each circle. These tables show the average error of each algorithm. Template matching showed the worst results. The circular Hough transform and the compensated circular Hough transform show better results.

Table 5.

Estimated diameter and real diameter

	TM	CHT	CCHT	MM
Estimation (mm)	27.319	26.216	26.132	26.112
Real (mm)	26	26	26	26
Error (mm)	1.319	0.216	0.132	0.112

To compare the compensated circular Hough transform and the mixed matching algorithms, an experiment was performed 30 min after sunset. Figure 12 shows the results of bolt hole extraction after sunset using the mixed matching algorithm and Figure 13 shows the extracted shapes from Figure 12.

Figure 11.

Arrange symbols of circles and compute distances

Figure 12.

Result of bolt hole extraction using the mixed matching algorithm

Figure 13.

Extracted shapes from Figure 11

Tables 6 and 7 show the estimated distances between each circle using the compensated circular Hough transform and the mixed matching algorithm.

Table 6.

Distance between each circle using the compensated circular Hough transform

	Estimation (mm)	Real (mm)	Error (mm)
N1	64.9134	60	4.9134
N2	204.4767	206	2.5233
N3	60.9297	60	0.9297
N4	63.0031	60	3.0031
N5	62.7421	60	2.7421

Table 7.

Distance between each circle using the mixed matching algorithm

	Estimation (mm)	Real (mm)	Error (mm)
N1	60.7832	60	0.8732
N2	60.9124	206	0.9124
N3	60.9041	60	0.9041
N4	60.8824	60	0.8824
N5	60.8467	60	0.8467

When comparing Table 6 to Table 7, the results of Table 6 show huge errors and variance. This means that the compensated circular Hough transform is unsuitable and the mixed matching algorithm proved to be much superior to other algorithms tested.

5. Top view supervisory system

The top-view supervisory system helps the operator in observing the bolting robot from various angles. Using this system, the operator can prevent accidents and the ensuing stress regarding the unseen area can be significantly reduced. In this study, four wide-angle cameras are installed at the front, back, left and right sides of the bolting robot. Each image from the respective cameras is used to make the combined top-view image after distortion (barrel, perspective) correction. Figure 14 shows a block diagram of the top view supervisory system.

Figure 14.

Block diagram of the image processing steps

5.1 Organization of the top view supervisory system

To make the top view image of the robot, a wide-angle camera is mounted at the top of the robot. The captured images from the four cameras at various angles are combined into one image via 4 by 1 MUX. The combined image is input into a vision PC by a grabber. The input image is processed by the vision PC. Figure 15 shows the organization of the system.

Figure 15.

Organization of the system

The captured images from this process are shown in Figure 16. The top-left part is the front, the top-right is the back, the bottom-left is the left and the bottom-right is the right view of the robot. This image has both radial and perspective distortion. Using these raw images, a top view supervisory image is made.

Figure 16.

Captured raw images

5.2 Radial distortion correction of captured images

The radial distortion correction algorithm is handled in Section 2.5

Figure 17 shows the radial distortion correction image. In contrast to the raw image, the curves of the image have been transformed to straight lines.

Figure 17.

Radial distortion correction image

5.3 Perspective distortion correction of the image

It is impossible to accurately depict a 3D reality on a 2D plane. Perspective distortion is determined by the relative distance at which the image is captured and viewed, and is due to the angle of view of the image being either wider or narrower than the angle of view at which the image is viewed, hence the apparent relative distances differ from what is expected. Figure 18 shows a perspective distortion situation wherein parallel rails converge at a vanishing point.

Figure 18.

Convergence of parallel rails at a vanishing point

To correct perspective distortion, the image is stretched with respect to the vanishing point. A formula for distortion correction is given as follows:

To correct line l that is leaning toward the vanishing point, l is transformed to l'. (x₁, y₁) is a rotation axis of line l. Point x́ is a point on the line l, í is the correction line of line l, point x is an arbitrary point in the image and point b is the corrected point.

The equation of line l is described as Eq. (4) and point x́ on line l is described as Eq. (5).

y = d (x - x_{1}) + y_{1}

(5)

\overset{́}{x} = \frac{y - y_{1}}{d} + x_{1}

(6)

Corrected point b is derived from Eq. (6).

180 - x^{'} : 180 - x_{1} = 180 - b : 180 - x

(7)

b = 180 (x - x_{1}) + \frac{x^{'} (180 - x)}{180 - x_{1}}

(8)

Using the previous results, Figure 20 shows the distortion corrected images.

Figure 19.

Perspective distortion correction

Figure 20.

Distorted and distortion corrected image

5.4 Combination of the distortion corrected image

To properly perceive the multi-perspective image of the robot, the distortion corrected images are appropriately combined. Figure 21 shows the selected regions displayed in the distortion corrected image.

Figure 21.

Selected regions combined into the corrected image

Using the selected images, the top view supervisory image is stitched together and Figure 22 shows the complete top view supervisory image. In the case of the front view image, distortion is not corrected because the image is hidden by the bolting tool. For that reason, perspective distortion correction is difficult.

Figure 22.

Top view supervisory system

6. Conclusion

In this paper, a new concept of the bolting cabin system is introduced. The bolting cabin assistance system consists of the bolting robot control system and the top view supervisory system. The bolting robot control system is a remote control system that operates the gantry robot and bolting tool, and the top view supervisory system provides the stitched-together image depicting the cabin from various angles. The bolting cabin control system can prevent workers from encountering dangerous situations. Moreover, the bolting robot can be modified to use other tools and the cabin can perform other construction tasks. Using this system, an application was developed and it is being used in the real construction field. Figure 23 shows the developed application.

Figure 23.

Developed bolting robot assistance application

Footnotes

7. Acknowledgments

This work was supported by a Korea University Grant. The authors would like to thank the Ministry of Land, Transport and Maritime Affairs, Korea. This work was also supported in part by the High-Tech Fusion Construction Technology Development Program.

References

Yoshida

(2006) A short history of construction robots research & development in a Japanese company Proc. of the International Symposium on Automation and Robotics in Construction 188–193.

Lytle

A.M.

Saidi

K.S.

Stone

W.C.

Gross

J.L.

(2003) Towards an Intelligent Job Site: Status of the NIST Automated Steel Construction Test Bed Proc. of the International Symposium on Automation and Robotics in Construction 203–208. [3] Seward D.W, Zied K, Rhiel J (2000) The development of a robotic system for tool deployment in hazardous environments Proc. of the International Symposium on Automation and Robotics in Construction 1–6.

Balaguer

(2000) A new wave of construction automation and robotics in Japan Proc. of the International Symposium on Automation and Robotics in Construction 1–7.

Ban

(2002) Unmanned construction system: Present status and challenges Proc. of the International Symposium on Automation and Robotics in Construction 241–246.

Wei

Lai

, (2008) Fast Template Matching Based on Normalized Cross Correlation With Adaptive Multilevel Winner Update IEEE Transactions on Image Processing 17: 2227–2235.

Lai

Wang

Huang

Ren

Shi

(2009) Fast Template Matching Based on Local Entropy Difference Image and Signal Processing 1–5.

Forsyth

D.A.

Ponce

(2002) Computer Vision: A Modern Approach. Prentice Hall. New Jersey.

Choe

Y.G.

Lee

H.C.

Lim

M.T.

(2009) Vision-based Estimation of Location Using Circular Hough Transform International conferences on control, automation and systems4821–4826.

Liu

(2009) A Curve-Fitting Method for Correcting Radial Distortion International Congress on Image and Signal Processing 1–4.

10.

de Villiers

J.P.

Leuschner

F.W.

(2008) Centi-pixel Accurate Real-time Inverse Distortion Correction International Symposium on Optomechatronic Technologies 1–8.

11.

Richard

Andrew

(2003) Multiple View Geometry. Cambridge University Press. England.

12.

Y.H.

Woo

B.W.

Lim

M.T.

(2010) Bolting Robot Assistance System Using Image Processing, International Conference on Control, Automation and Systems 2342–2345.

13.

Castillo

Martinez-Marroquin

Melin

Valdez

Soria

(2012) Comparative study of bio-inspired algorithms applied to the optimization of type-1 and type-2 fuzzy controllers for an autonomous mobile robot Information Sciences 192:19–38.

14.

Cazarez-Castro

N.R.

Aguilar

L.T.

Castillo

(2010) Fuzzy logic control with genetic membership function parameters optimization for the output regulation of a servomechanism with nonlinear backlash Expert Systems with Applications 37:4368–4378.

15.

Martinez-Soto

Castillo

Aguilar

L.T.

(2009) Optimization of interval type-2 fuzzy logic controllers for a perturbed autonomous wheeled mobile robot using genetic algorithms Information Sciences 179:2158–2174.

16.

Woo

B.W.

Kim

Y.J.

Lim

M.T.

(2009) Barrel Distortion Correction of Camera Lens Using Hough Transform Korea Automatic Control Conference.

17.

Bin

(2009) Line Feature Recognition Based on Differential Projection Transform Information Engineering and Computer Science 1–4.

18.

Chia

A.Y.S.

Leung

M.K.H.

Eng

H.L.

Rahardja

(2007) Ellipse Detection with Hough Transform in One Dimensional Parametric Space IEEE International Conference on San Antonio vol. 5.

19.

Illingworth

Kittler

(1988) Survey: A Survey of the Hough Transform Computer Vision, Graphics and Image Processing 87–116.

20.

Umbaugh (1993) Computer Vision Graphics and Image Processing: Image Understanding. Prentice Hall. New Jersey