Visual prediction model for weld seam offset based on deep learning

Abstract

To test the weld penetration situation, a new method based on deep learning has been researched. The welding technique in paper is TIG weld. The weld testing and experiment system has been firstly setup. Weld experiments can be performed. A CCD sensor is used to snap images during the welding procession. A neural filter and a narrow band filer are chosen to setup a composite system, which can filter weld arc light. The experiment system can capture several groups of weld pool images in real time. The pool images are processed by median filter and gray transformation operations. On this basis, the CNN(convolution neural network) is built up. The input layer contains 215×215 neurodes. The character extraction network is a single convolution layer composed by 20 convolutional filters, which are 9×9. The active function of convolution layer is the ReLU function. The 2×2 average pooling method is used for the pool layer. And the classifier network contains one hidden and a output layer. There are 100 neurodes in hidden layer, and the activated function of it is the ReLU function. The neurodes number in output layer is three, which stand for the unfused, fused and overfused condition of weldments. The Soft max function is determined to be activated function for output layer. Then 300 pool images are used to be sample data, which are captured by welding experiments. And the set up neural network can be trained by the data. So a visual CNN model can be setup for weld penetration predicting. In the end of the paper, experiment is performed to test the accuracy. Another 100 pool images are imported to the setup CNN model. The precision is up to 92%, which is showed that the setup network has certain accuracy rate.

Keywords

weld pool image weld penetration prediction deep learning convolution neural network

Introduction

Weld is a important technique, which can be used for the connecting of metal. It can be used in many social fields such as the mechanic manufacture, military, aerospace, shipbuilding industry, and so on. And about 40% steel structural parts are connected by weld technique.¹ The welding quality at connection position determines the stability of finished production, even the safety of human productive or living activity. The flaw detection operation is performed for mental parts quality control when welding is finished. The nondestructive testing and X-ray detection at the weld region are the common methods.² The detection after welding can help to find the quality flaw of metal parts. However, if the weld flaw can be found or predicted during the welding procession, the weld efficiency would be greatly improved.

In fact, skilled weld worker can observe the change of weld pool formation, change weld gun attitude and weld velocity to control weld penetration status during the welding procession. So the penetration of metal connection position can be made sure and an excellent weld seam can be obtained. It can be found that there exists abundant information on the pool of weld position, which contain heat transfer condition, the stability of welding and penetration situation. Therefore, the status of weld penetration can be inferred by formation feature on the weld pool indirectly. In order to obtained the weld pool information, many wonderful research has been performed. The visual sensor method, arc sound method, pool vibration method, infrared sensor method and the optics sensor method have been researched for weld pool measurement.^3–6

And the visual detection method is very promising, and the method is widely used in the field of weld pool depth control, because it has the merits of untouched and large information obtainment.⁷ Zhan Aiwen determined the K-TIG weld to be the welding technology, and research a weld penetration detection method for thick plate weld.⁸ A high-dynamic CCD camera is used to snap images of weld pool and keyhole entrance. Then several image operations are performed to extract the edge of weld pool and keyhole entrance. Characteristic parameters on the pool, keyhole entrance, weld current and weld speed are chosen to be input parameters. Weld penetration status are determined to be the output. The BP neural network is chosen to infer the weld penetration status recognition model. Ji Shengpeng applied the VGG deep learning neural network to built up the weld pool back side image penetration prediction model.⁹ The weld parts status can be detected, which are the unfused, fused and over fused status. Duan Mingrui obtained the weld pool edge information by image filtering, Hough line detection, Gamma transformation and edge detection operations, which are based the Open CV. Then the convolution neural network is applied to set up the penetration prediction model.¹⁰ Xu Zihui applied a fog remove algorithm to perform the image pretreatment, which is based on the morphology. The quality of weld pool image can be improve. Then the Alexnet-LSTM deep learning neural network is setup to the model for weld penetration prediction.¹¹ Xie Xiaotong set up a laser vision system to capture weld pool images, and several image operations are performed to extract the pool edge, which is based on the Alexnet-LSTM deep convolution neural network.¹² The multiple linear regression, support vector regression and XGboost algorithm are applied for the prediction of weld pool width, which is back of weld pool. And the status for weld part penetration can be determined. Jiang Zheng use the aerospace aluminum alloy robot welding to be the application example.¹³ Visual sensor is applied to capture weld image in high-dynamic. The generative adversarial network is used to filter the arc disturbance for weld pool. Then the weld pool edge can be extracted based on cascade regression tree and the weld width at the front of pool can be obtained. The weld current, the wire feed rate, the assemble information (gap and misalignment), weld width at the front side are determined to be input. The Light GBM algorithm is applied to predict the weld width, which is back of weld parts. Then the status of weld penetration is predicted by the weld width at back of weld parts.

Learn from the existing method, a new weld depth controlling method has been deeply studied in the paper. The new method is based by weld pool images. TIG is determined to be the weld technique. A weld experiment system has been firstly setup. Several groups of weld images can be snapped. Then median filtering and gray transform operators are caught out to the images, which are made up to be training and testing sample data. Then a deep learning convolution neural network is constructed. The captured weld pool images are used to be sample data and used to train the setup convolution neural network. And the prediction model for weld parts penetration is deduced. The setup model has supplied a novel method for quality control during the welding procession.

Setup of experiment platform

In the paper, the technique for weld experiment is determine to be the tungsten inert gas arc (TIG) weld. The welding experiment system contains a four-axis mechanical structure, a visual sensor, industry control computer, welding image capture system, motor control system, and so on. The welding experiment system is shown in Figure 1. The mechanical structure is composed by a gantry frame and a weld table, which are actuated by stepping motors. There are welding fixtures on the welding table. They can be used for weld parts clamping.

Figure 1.

Weld experimental system.

The welding control system contains industry computer, PMAC (Programmable Multi-Axis Controller) motion control card, image snapping card. The motion control and image snapping card are based on PCI (Peripheral Component Interconnect) bus, which are plug and play. The cards supply dynamic link libraries for motion control and image capture, which are very convenient for the secondary development of users. Based on the Windows operation system, a welding experiment software system has been developed, which is shown in Figure 2. The system can realize several functions, such as axis speed control, position control, image capture, weld teach, data procession, and so on.

Figure 2.

Weld control software system.

There are travel switches installed along the axis on the gantry frame and weld working table. They can be used to prevent from moving over range. During the welding procession, the Y axis is the direction for feeding. CCD (Charge-Coupled Device) sensor captured weld images. The images are decoded and A/D transferred by image captured card. Then the digitized images are sent to control computer. The images are then saved into the hard disk. To filter the disturbance produced by weld arc light, there are light filters for vision sensor, which are installed in front of the CCD (Charge-Coupled Device) camera. Several groups of weld pool images can be snapped.

Obtainment and pre-treament of pool image

Obtainment of pool image

Obtainment of clear weld images is very significant to control weld penetration for the new method based on the weld images. Weld arc light is very strong during the actual welding procession. Many useful information has been flooded by the hard weld arc light. Therefore, it should try to filter the light disturbance. Many researcher has tried different methods. Some one has applied a narrow band filter for light filtration. But the weld arc can’t be filtered entirely, which is that the arc light is still very glare. The weld pool can’t be distinguished clearly. However, some one has tried a common neutral dimming method, which will lower the contrast of pool image with the reduce of arc light. Some detail of pool image will be disappeared. In the paper, a composite filtering system has been researched. The filtering system contains a narrow band filter, which is 650 nm central wavelength, and a 7# neutral dimming filter. So several wonderful pool images can be obtained. The snapped pool image filtered by narrow band filter or neutral filter is shown in Figure 3.

Figure 3.

Pool Images filtered by narrow band filter or neutral filter. (a) Image after narrow band filter and (b) image after neutral filter.

A composite filter system is setup in the paper. The filtering system contains a narrow band filter, which is 650 nm central wavelength, and a 7# neutral dimming filter. Then more wonderful pool image is obtained, which is shown in Figure 4.

Figure 4.

Image after composite filter.

Pretreatment of pool image

For reducing the arc light disturbance,a median filter is firstly performed in the paper, which is the 3×3 template median filter. Then a gray transfer operator has been caught out to the pool image. The original image is shown in Figure 4. The pool image processed by median filter and gray enhancement operators are shown in Figures 5(a) and 5(b).

Figure 5.

Pretreatment of weld image. (a) Pool image after median filter and (b) pool image after median filter and gray transfer operation.

Definition of weld penetration status

There are three weld penetration statuses for steel plate, which are the unfused, fused and over fused status. Under the situation that the weld parts aren’t destroyed, the penetration condition can be speculated by some characteristic parameters of weld procession indirectly. In some research paper, the pool width is chosen to be the characteristic parameter for measuring penetration status, which is at back of weld part.¹³ During a welding procession, the weld penetration degree is determined by the heat conduction under the torch. When the heat condition changes, the gray degree and form of weld pool and weld arc would become different,too. Therefore, it is possible that the penetration condition of weld parts can be speculated indirectly by some character parameter of weld images, which is shown in Figure 6.

Figure 6.

Different penetration status weld images. (a) Unfused weld image, (b) fused weld image, and (c) over fused weld image.

In the research, the weld seam width at the back of weld part is used to determined the penetration status, which can be applied to be training and testing data set for the setup of weld penetration model later. In this paper , the weld seam width at back of weld part less than 1.5 mm is determined to be unfused status. The weld seam width at back of weld part between 1.5 mm and 1.6 mm is determined to be fused status, and the weld seam width at back of weld part more than 1.7 mm is determined to be overfused status. The weld penetration status for training and testing data set is shown in Table 1.

Table 1.

Definition of Weld penetration status for training and testing data set.

Weld penetration status	Unfused	Fused	Overfused
Range value	<1.5 mm	≤1.7 mm and ≥1.5	>1.7 mm

Weld penetration prediction model based on deep learning

Obtainment of experimental data

Weld experiments have been caught by the set up experimental system. Table 2 below showed a weld experiment technology parameters. So several weld pool images can be acquired. The weld pool images is then used to be the training and testing sample data for the deducing of penetration prediction model.

Table 2.

Weld experiments technology parameters.

Material and size of steel plate (mm)	Gas flow (ml/s)	Speed of weld experiment (mm/min)	Weld image sample cycle (s)
Q235 (300×150×3)	150	75	0.04

The welding experiments operations process can be shown as following. At the start point, the weld gun is controlled to point to the weld seam. Then step motor along y axis is launched, the work table performs the feed motion along y axis. CCD visual sensor is controlled to capture pool images at sample period, which is 40 ms. The snapped pool images is then sent to the control computer and be saved in the hard disk. In a weld experiment, 400 pool images can be captured. The weld technology character parameters are shown in table 1. And the weld image after weld experiment is shown in Figure 7.

Figure 7.

Weld finished image.

Set up of weld penetration prediction model based on deep learning neural network

Convolution learning neural network is a new technology developed from 80 or 90 years in 20 century, which is actually a deep learning network. There are many hidden layers in a convolution learning neural network, and it is the simulation of the working mechanism of living brain visual cortex. In recent years, with the development of information and artificial technology, the convolution neural network has been widely applied in the field of flaw detection, agriculture, security and protection, and so on.

The convolution neural network for weld penetration prediction model is constructed, which is shown in Figure 8. The convolution neural network contains four layers. The input is the 215×215 weld image, which is shown in Figure 4, and the input nodes are 215×215. The convolution layer is composed by 20 9×9 convolution filers, Some major convolution filters are shown in Figure 9. Then the image characters can be extracted.

Figure 8.

Construction of the weld penetration prediction convolution neural network.

Figure 9.

Convolution filters.

The activation function for the convolution layer is the ReLU function. The output of convolution layer is then input to pooling layer. In the pooling layer , 2×2 average pooling operation is performed. And the dimension of image can be compressed. In the classifier neural network, there are a hidden layer and a output layer. The hidden layer contains 100 neurodes, and the ReLU function is the activation function for the layer. The output layer contains 3 nodes. The softmax function is the activation function for the output layer. The output values of the deep learning network are 1,2,3, which stands the unfused, fused and over fused status for welding. The expression of ReLU and softmax are shown as following^14,15:

Re L U (X) = {\begin{cases} W^{T} X + b, X > 0 \\ 0, X < 0 \end{cases}

(1)

Soft \max (v) = \frac{e^{v}}{\sum_{i = 1}^{3} e^{v}}

(2)

Several group of weld experiments are then caught out. The weld experimental system is shown in Figure 1. The weld technology parameters are shown in table 1.400 weld images can be captured. The 300 images are determined to be training data. The training images run through the input layer, the convolution layer, the pooling layer, the hidden layer and the output layer. The train result is shown in Figure 10.

Figure 10.

Training result.

Confirmatory experiment

In order to test accuracy of the set up convolution neural network, another 100 group of data have been selected from the weld experiment sample data and input to the prediction model. The measure and prediction penetration status can be obtained, which is shown in Figure 11. The precision rate of the setup model can be calculated by the following:

\bar{e} = \frac{n}{N}

(3)

In the formula, N is the verification points amount, which is 100 in the example. And the correct prediction points is n in the paper.

Figure 11.

Accuracy test experiment result of weld penetration prediction model.

From the experiment result in Figure 11, it can be found that there are 8 data failed to prediction the penetration status, which may be disturbed by the noise, such as arc light, weld splash, and so on. The accuracy rate of the model is 92%, which is showed that the prediction status calculated by the deep learning network model is consistent with the measure value.

Conclusion

To settle the question that the metal structure in the weld seam is difficult to test in real time, a novel method is studied in the paper, which is by on the deep learning and can be used for weld parts penetration status prediction. TIG weld is determined to be the studied welding technique. A weld experiment system is firstly set up. To filter the weld arc light, a composited filtering system is setup, which contains a narrow band and a neutral light reducing filter. The training and testing data set is captured in a weld experiment, which contains 400 weld images. The median filter and gray transform operations are then performed to the capture weld images, and the contrast of weld images can be enhanced.

A convolution neural network is constructed, which contains four layers. There are 215×215 nodes in the input layer. Character extraction neural network has one convolution layer. The filter for the convolution layer is the 9×9 filters. The ReLU function is determined to be activation function for the convolution layer. The pool layer is a 2×2 average pooling layer. And the classified neural network contains one hidden layer and one output layer The hidden layer of classified neural network has 100 neurodes, and the active function for hidden layer is also the ReLU function. The output layer contained three nodes, which stand for the unfused status, the fused status and the over fused status. The active function for output layer is softmax function. So the CNN model can be deduced, which is for the penetration status prediction of weld parts. A accuracy verifying experiment is caught out to the setup prediction model. The precision rate of the model is up to 92%. 8 predictions failed, which showed that the weld penetration prediction values are fit to the measure values.

Footnotes

Acknowledgments

The authors acknowledge 2024 Guangdong Province higher vocational colleges production and education integration innovation platform project of Guangdong Province Department of Education-Industrial robot production and education integration innovation platform (No. 2024CJPT027), Guangdong Provincial Department of Education 2023 high-end equipment (intelligent robots, new materials) key special areas—Research of weld control for robot based on the deep learning and multimodal information fusion (No. 2023ZDZX3092), Dongguan social development of science and technology key project (No. 20231800939952), Dongguan Sci-tech Commissioner Project of 2021-Research and development of intelligent flexible production line (No. 20211800500042), intelligent terminal and intelligent manufacturing special project of Dongguan Polytechnic: Platform and team project (No. ZXA003), intelligent terminal and intelligent manufacturing special project of Dongguan Polytechnic (No. ZXD202307),Guangdong HUST Industrial Technology Research Institute, Guangdong Provincial Key Laboratory of Manufacturing Equipment Digitization project (No. 2023B1212060012), and the project for the innovative platform in industry and education integration for higher vocational college of Guangdong province—the innovative platform in industry and education integration of testing technology for intelligent terminal product (No. 2020CJPT014).

ORCID iD

Ding Dukun

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: 2024 Guangdong Province higher vocational colleges production and education integration innovation platform project of Guangdong Province Department of Education-Industrial robot production and education integration innovation platform (No. 2024CJPT027), Dongguan social development of science and technology key project (No. 20231800939952), Guangdong Provincial Department of Education 2023 high-end equipment (intelligent robots and new materials) key special areas—Research of weld control for robot based on the deep learning and multimodal information fusion; No. 2023ZDZX3092, Guangdong HUST Industrial Technology Research Institute, Guangdong Provincial Key Laboratory of Manufacturing Equipment Digitization project (No.2023B1212060012), Key Laboratory Project of Manufacturing System and Advanced Manufacturing Technology of Guangxi Province (No.22-035-4S018).

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Farhan

Guven

. Application of friction stir spot welding on additive manufactured carbon fiber composite. Mater Sci Forum 2022; 6674: 23–28.

Suyama

Delgado

da Silva

, et al. Deep neural networks based approach for welded joint detection of oil pipelines in radio graphic images with Double Wall Double Image exposure.NDT E Int 2019; 105: 46–55.

Luo

, et al. Autonomous decision-making of welding positions based on analytic hierarchy process in T-joint GMAW. J Jilin Univ (Eng Technol Ed) 2022; 10: 1–7.

Wang

Sun

Liu

, et al. Effect of pulsed ultrasonic on arc acoustic binding in pulsed ultrasonic wave-assisted pulsed gas tungsten arc welding. Sci Technol Weld Join 2017; 22(6): 465–471.

Chen

Xue

Zhai

, et al. Study on penetration sensing method based on pool oscillation and arc voltage during pulsed GMAW. Appl Sci 2020; 10(8): 1–15.

Kazuki

Yosuke

Satoshi

, et al. Study on welding phenomena observation method based on arc and molten pool light emission characteristics in visible and infrared wavelength region – development of image sensing technology for in-process welding monitoring technology. Weld Int 2019; 33(1): 83–95.

Kumar

Mishra

Singh

, et al. Development of an autonomous vision sensor-actuator-based circumferential seam path tracker welding machine/device for LPG cylinders. Memories- Materials, Devices, Circuits and Systems 2022; 2: 1–11.

Aiwen

. Study on visual inspection of molten pool-keyhole and penetration state recognition of K-TIG welding. Master thesis of South China University of Technology; 2021, pp. 23–83.

Shengpeng

Zhihong

Jia

, et al. Visual detection of TIG weld forming of thin sheet based on deep learning. Hot Work Technol 2022; 51(3): 115–121.

10.

Duan

. On-line monitoring of welding qualith of stianless steel GTAW in spent fuel pool based on deep learning. Master thesis of Harbin Institute of Technology; 2021, pp. 26–61.

11.

Zihui

. Image analysis and welding quality research of robot welding pool based on deep learning. Master thesis of Guangdong Polytechnic Normal University; 2022, pp. 12–60.

12.

Xie

. In situ studies of full field strain and residual stress of welding heat affect zone using digital image correlation. Master thesis of Shanghai Jiao Tong University; 2020, pp. 39–75.

13.

Jiang

. Multi-source information representation and penetration prediction and control for robotic TIG welding of aluminum alloy. Master thesis of Shanghai Jiao Tong University; 2020, pp. 25–64.

14.

Michael Palaszek write . Practical MATLAB Deep Learning - based on project approach[M]. Beijing: Tsinghua University Press; 2021.

15.

Kim write

. Deep learning:design example based on MATLAB[M]. Beijing: Beijing University of Aeronautics and Astronautics Press; 2018.