Spark plug defects detection based on improved Faster-RCNN algorithm

Abstract

The objective of this study is to apply an improved Faster-RCNN model in order to solve the problems of low detection accuracy and slow detection speed in spark plug defect detection. In detail, an attention module based symmetrical convolutional network (ASCN) is designed as the backbone to extract multi-scale features. Then, a multi-scale region generation network (MRPN), in which InceptionV2 is used to achieve sliding windows of different scales instead of a single sliding window, is proposed and tested. Additionally, a dataset of X-ray spark plug images is established, which contains 1,402 images. These images are divided into two subsets with a ratio of 4:1 for training and testing the improved Faster-RCNN model, respectively. The proposed model is transferred and learned on the pre-training model of MS COCO dataset. In the test experiments, the proposed method achieves an average accuracy of 89% and a recall of 97%. Compared with other Faster-RCNN models, YOLOv3, SSD and RetinaNet, our proposed new method improves the average accuracy by more than 6% and the recall by more than 2%. Furthermore, the new method can detect at 20fps when the input image size is 1024×1024×3 and can also be used for real-time automatic detection of spark plug defects.

Keywords

Defect detection attention module inceptionV2 faster-RCNN spark plug

1 Introduction

Spark plugs, commonly known as the fire nozzle, are often used to discharge the high-voltage electrical pulse which is sent by a high-voltage wire. This process penetrates the air between the two electrodes of the spark plug, producing an electric spark to ignite the cylinder gas mixture. Automotive spark plugs often serve in corrosive, high-temperature, high-pressure environments. Due to the impact of production equipment, production technology, environmental factors, the spark plug oil channels often suffer cold shut defects [1]. In the case of fatigue, radiation and impact conditions, car spark plugs containing such defects may result in starting difficulties and fuel consumption increase; and idling, automatically stalling, and body shaking in worse situations. As shown in Fig. 1, the X-ray images are captured and processed by using the following defect detection system, which contains three components: image acquisition system, file server, and image processing system. In the image acquisition system, the X-ray scans the spark plug and transmits the X-ray image to the file server. Our image processing system obtains the X-ray image from the file server and uses the image inspection method for defect detection. However, in the spark plug X-ray map among them, the cold shut defect is characterized by small size and low contrast, which is easy to be ignored and makes the detection work more difficult. Cold shut defects bring serious hidden risks to property and personal safety, therefore, how to quickly and accurately detect spark plug defects is of great significance to production and life.

Fig. 1

X-ray image defect detection system.

The traditional methods require image pre-processing, then the defect regions are segmented manually and finally classified by a classifier trained by traditional machine learning methods. Moosavian et al. [2] proposed an intelligent diagnosis method for spark plug faults in internal combustion engines based on acoustic and vibration signals. In his method, first the wavelet denoising technique was used to process the signal, and then artificial neural networks and least squares support vector machines were used in the classification stage, to improve the effectiveness of the diagnostic system in engine condition monitoring, the difficulty of the fault detection was due to the type and severity of the fault considered. Zhao et al. [3] proposed a robust detection system based on sparse representation for the detection and classification of casting defects in X-Ray images. The mean accuracy of the method based is 94.29%, which is higher than that of the SVM, but this method has many steps, and each step takes a long time. Hashim et al. [4] proposed a fault diagnosis method for spark plugs, where the selection of wavelet and decomposition layers was optimized in order to reduce the noise of the captured signal. They also proposed a feature extraction method based on discrete wavelet transform and energy spectrum, and evaluated the effect of appropriate selection of denoising wavelet basis and decomposition layers on the performance of supervised and unsupervised artificial neural networks (ANN), but every signal has an optimum wavelet to use depending on the nature of the signal so it must be selected perfectly because it may lost too much information from signal due to the improper selection. These methods have the disadvantages of high computational cost, noise sensitivity, and dependent on the designed template, thus not only perform poorly in terms of detection accuracy and generalization ability, but also make it difficult to achieve real-time detection because of their reliance on manual feature extraction. Therefore, a real-time detection algorithm that can achieve high accuracy is needed to change the status quo.

With the development of convolutional neural networks (CNN), methods for detecting spark plug defects using deep learning-based detection algorithms became dominant in the field of target detection. At present, deep learning-based target detection is mainly divided into two categories, one is one-stage detection algorithms such as SSD [5, 6], YOLO series [7 –9], retinaNet [10]; the other is the two-stage detection algorithm based on RCNN [11 –13], represented by Faster-RCNN [14] and Mask-RCNN algorithm [15]. Faster-RCNN divides the target detection task into generating proposal boxes in Region Proposal Network (RPN) and doing classification and position regression for the proposal boxes.

Due to the target detection task being divided into two stages, the two-stage algorithm has higher accuracy but lower detection speed than the one-stage algorithm. Li Wigang et al. [16] proposed an improved YOLOv3 algorithm to achieve high accuracy classification of defects, the experiment is carried out on the NEU-DET dataset. The detection speed is 50fps when the input image size is 256×256, but the average accuracy is only 80%. Xing et al. [17] proposed a symmetric structured convolutional network (SCN) as a feature extractor to improve the performance of the network for detecting defects on rigid surfaces, the model achieves 78.44% mAP on a detection dataset of surface defects on the hearth of raw aluminum casting, but detection speed is 23fps when the input image size is 416×416×3. Walters et al. [18] used neural networks to classify spark voltage waveforms and confirmed the great potential of neural networks for spark plugs detection. Lin et al. [19] addressed the problem of insufficient computational resources in most manufacturing companies by introducing efficient networks and deeply differentiable convolutions, thus saving computational resources while maintaining detection accuracy. Ji et al. [20] proposed a new two-stage FSS/ED-CNN method for detecting and identifying defects in aerospace titanium castings from X-Ray images, the accuracy is 87.65 % in the label prediction step. Although the accuracy of this method is high, FSS has many steps and is not a convolutional neural network, so the method is not an end-to-end method and takes a long time. Liu et al. [21] proposed a visual defect detection network based on CNN with better detection accuracy by introducing deformable convolution module, balanced feature pyramid module and cascade head module to solve the problems of large variation range of defects and low quality of defect localization. Shu et al. [22] proposed a method to detect and identify commutator surface defects, which improves the speed of network detection of commutator surface defects. To detect fine defects on X-ray images of aluminum alloy castings, Wang et al. [23] proposed a new deep model which consists of a general feature network (GFN) and a subtle feature network (SFN), experiments demonstrated the proposed model had good detection performance. Zhang et al. [24] proposed an improved Fast-RCNN algorithm that integrates clustering-generated anchor boxes and migration learning with ResNet101, had a great improvement in all aspects. However, in some studies mentioned above, more attention is paid to improving the detection accuracy while neglecting the detection efficiency, resulting in real-time detection failure during production, such as [17, 20]; while in other studies, the speed is pursued without improving the accuracy, which cannot be applied in practice, such as [16]. We achieve a more flexible balance between speed and accuracy.

This paper presents an attention symmetric convolutional network (ASCN) for spark plugs defect detection. The proposed network uses a symmetric structure as a multi-scale feature extraction network in which the attention mechanism [25, 26] is embedded. The symmetric structure can improve the feature reuse rate, the attention mechanism on the bridge can make the model focus on the defects. It should be noted that neither max pooling nor average pooling in the attention mechanism do not have parameters, like [27 –29] is also to reduce the network parameters to make the network faster, and we reduce the image size before entering the ASCN, these improvements make the model improve the performance without significantly increasing the time consumption, and the model is more superior. In addition, a multi-scale region generation network (MRPN) in which the InceptionV2 module is introduced is also proposed. Differ from the original single sliding window in the original RPN, the InceptionV2 module [30] in MRPN achieves sliding windows of different scales and feature extraction of different types. Finally, to improve the model’s ability to detect defects at various scales, the model is transferred and learned on the pre-training model by MS COCO dataset [31], which is a large open-source image dataset developed and maintained by Microsoft for object detection, object segmentation and other tasks. The experimental results show that the improved Faster-RCNN algorithm achieves end-to-end spark plug defect detection in real time with higher accuracy and better real-time performance.

The rest of the paper is structured as follows. Section II presents the algorithm’s innovation and implementation details. Section III provides experiments for testing the network, and compares with some competitive models, and analyzes their advantages and disadvantages. Section IV concludes the paper.

2 Proposed methodology

The method in this paper is an improved version of Faster-RCNN, and it is illustrated in Fig. 2. This method uses ASCN as the backbone for feature extraction. Due to the location and shape characteristics of spark plug defects, the attention module as well as symmetric structure is designed in backbone to improve the performance of spark plug defect detection. ASCN feeds feature maps with four different scales into the MRPN as well as the Fast-RCNN part marked in the dashed box in Fig. 2 after entering the MRPN, feature maps will generate proposal boxes, then these boxes and feature maps are fed into the ROI pooling layer to output ROI features, finally after a series of fully connected layers, the classification and regression parameter prediction are done according to these ROI features.

Fig. 2

Schematic diagram of the proposed algorithm.

The training stage mainly included data processing, network training, and parameter optimization. In the testing stage, the effectiveness of the trained model was tested with test samples. Figure 3 shows the specific training-test process. We train the model in three steps. In the first step, we first load the pre-trained model of the original Faster-RCNN whose training set is the MS COCO dataset. The second step is to replace the backbone with ASCN, and then use the sparkplug training set to train the model. The third step replaces the original RPN network with MRPN, and then use the sparkplug training set again to train the model and output our complete model. Finally, we use the test set to verify the performance of the model.

Fig. 3

Flowchart for training and testing.

2.1 Feature extraction network

2.1.1 Symmetric modules

It is well known that the emergence of feature pyramids network (FPN) [32] has led Faster-RCNN to have a higher level of accuracy as FPN has the advantage of fusing features at different scales. Similarly, the U-Net [33] also has the advantage of fusing features at different scales since the skip connections in U-Net. In addition, the U-Net can be trained on small sample datasets and get accurate target localizations, while the FPN cannot. Inspired by this, we designed a symmetric structure similar to the U-Net and introduced the attention module as the backbone of feature extraction, which is called ASCN. As shown in Fig. 4, the symmetric structure in ASCN is the main reason for improving the performance of extracted features, which consists of a systolic channel for extracting feature semantics (the left path in the figure) and an extended channel for achieving feature location accuracy (the right path in the figure). The symmetric module includes convolution layers, down-sampling and up-sampling layers. In each down-sampling layer, a 2×2 pooling is used with a sampling step of 2, so that the output size of the down-sampling is half the size of the input. The up-sampling layer corresponding to the pooling layer is realized by linear interpolation, so that the size of both ends of the skip connection is the same, and the high-resolution features from the contraction channel are concatted together with the output features of the up-sampling through the skip connection to obtain a new feature map. Before the concatted features enter the subsequent network, a 1×1 convolution layer is added to ensure that the number of feature channels is uniformly 256, thereby obtaining the final output feature map. This module is abbreviated as sym_n, where n is the number of down-samples. The symmetric structure combines the deep feature map with rich semantic information and the shallow feature map to consider both the shallow feature and the deep feature, and the detection accuracy of small targets is improved.

Fig. 4

Structure diagram of sym_n.

2.1.2 Attention Modules

When the resolution of the original image is large, it is inevitable that there will be some irrelevant information in the image, such as background, textual information contained in the image, etc. These image areas are not helpful to network training and may even have a negative impact on the detection model. Therefore, the network should learn how to automatically focus its attention on the tested object rather than uniformly across the image, i.e., suppressing the regions of the input image that are irrelevant to the tested object while highlighting the features of the tested object. Given the advantage of attention modules, in this paper, the attention module is embedded in the skip connection. Figure 5 shows the attention module embedded in the ASCN, the feature maps are successively passed through the channel attention module and the spatial attention module. The channel attention module tells the network “what to look at” and the spatial attention module tells the network “where to look”, thus the two attention modules complement each other. The input feature map X ∈ R^W×H×C is adaptively pooled by computing maximum and mean values, and then summed after a series of fully connected layers to generate the channel attention coefficients α. The intermediate feature map X′ ∈ R^W×H×C is obtained by multiplying the channel attention coefficients α ∈ R^1×1×C with the input feature map X by pixel. As shown in Equation (1), after passing through the channel attention module, the spatial attention coefficients β ∈ R^W×H×1 are generated by pooling the maximum and average values on the channel, respectively. Then $\hat{X}$ is obtained by multiplying the spatial attention coefficients ß with the intermediate features by pixel, as shown in Equation (2). $X'_{w, h, i} = α_{i} \cdot X_{W, h, i} (i = 1, 2, \dots, C; w = 1, 2, \dots, W; h = 1, 2, \dots, H)$ (1) ${\hat{X}}_{z, k, c} = X^{'} z, k, c * β_{z, k} (c = 1, 2, \dots, C; z = 1, 2, \dots, W; k = 1, 2, \dots, H)$ (2)

Fig. 5

Schematic diagram of attention network structure.

It should be noted that the sigmoid function determines the channel attention coefficients and spatial attention α_i, β_z,k ∈ [0, 1]. The attention module is a pixel-wise multiplication of the input features and attention coefficients, i.e., the dimensionality of the feature map is guaranteed to be constant. The optimized feature map is then concatted with the upsampled feature map of the next layer. For example, in Fig. 4, F_0 is first fed into the attention module and its output is concatted with the upsampled feature map of feature map F^′_1 to obtain the feature map F^′_0. Therefore, the model automatically highlights the salient features in the feature map that are useful for the task and avoids the interference of irrelevant information to the network.

2.1.3 ASCN in detail

The shallow layers of CNN can provide more information in detail, which belongs to a detailed feature but lacks high-level semantic information. On the contrary, after multiple convolution and down-sampling, the deeper layers although lose the detailed information but retain the overall information. Leading to richer semantic information, which belongs to more abstract information. In the original Faster-RCNN network structure, the feature map output from the last convolution layer of the backbone is usually used as the input image of the subsequent network, which has richer semantic information, however, the ability to characterize small-scale defects is diminished. Therefore, the original Faster-RCNN network has more promising detection results for larger size targets, but the detection results for smaller size targets are poor. ASCN uses the output features of the symmetric structure as the input feature map of the subsequent network, and the validity of this feature map determines the significant predictive effect of the model.

Figure 6 shows the ASCN structure. C is the integration of the convolution, the batch normalization and the activation Rectified Linear Units (ReLu) layer. Batch normalization is added after each layer of convolution to avoid gradient disappearance problems and to speed up convergence. Skip connection concats the feature map on the left with the up-sampled feature map on the right to achieve multi-scale feature fusion. A is an attention module, embedded in the skip connection, which makes the output feature map more helpful for model training. Purple arrows indicate the maximum pooling layer of 2×2 size and 2 steps, which can increase the robustness to some small perturbations of the input image, such as image translation, rotation, etc. It also reduces the risk of overfitting and the amount of operations and increases the size of the perceptual field. Red arrows indicate the upsampling layer, using linear interpolation to enlarge the image to twice its original size with the same number of channels. Inspired by the network of ResNet50 [34], the ASCN network reduces the number of parameters and increases the detection speed by a 7×7 convolution kernel and a 2×2 pooling layer. The multi-scale feature maps output from four branches are used in the subsequent network, thus retaining the detailed features such as image edges and textures for the subsequent steps and providing high-level semantic abstract features. The feature maps extracted from different layers are different due to the perceptual fields of the neurons in each layer, and thus the anchor point size and scale parameters of each RPN need to be set separately. From top to bottom, the anchor point sizes are 32, 64, 128, 256, and the ratios are 1:2, 1:1 and 2:1. Table 1 shows the characteristics of the ASCN structure in detail.

Fig. 6

Structure diagram of ASCN.

Table 1

Characteristics of the ASCN’s structure

Type	Size/Stride	Output size	Remark
conv	7 ×7/2	512×512, 32
pool	2×2/2	256×256, 32
conv	3×3/1	256×256, 64
conv	3×3/1	256×256, 64
sym_4		256×256, 256	Branch1
pool	2×2/2	128×128, 64
conv	3×3/1	128×128, 128
conv	3×3/1	128×128, 128
sym_3		128×128, 256	branch2
pool	2×2/2	64×64, 128
conv	3×3/1	64×64, 256
conv	3×3/1	64×64, 256
sym_2		64×64, 256	branch3
pool	2×2/2	32×32, 256
conv	3×3/1	32×32, 512
conv	3×3/1	32×32, 512
sym_1		32×32, 256	branch4
pool	2×2/2	16×16, 256
conv	3×3/1	16×16, 1024
conv	3×3/1	32×32, 512

In the training phase, feature map extraction is very important for the detection effect. A good feature map is conducive to training an excellent model. The maximum pooling and average pooling in the attention mechanism have no parameters, and when the image is input, the image size is reduced, so it will not increase too much time consumption. ASCN network ensures the validity of the feature map while making the backbone network as light as possible. The symmetric structure itself also has the advantages of accurate target localization and small sample training, which significantly improved the accuracy of model detection and model time performance.

2.2 MRPN Network Structure

In the RPN of the original Faster-RCNN, a 3×3 sliding window is used to extract features and then pass a 1×1 convolution respectively to obtain the detection classification and location results of the RPN. As shown in Fig. 7, instead of a single 3×3 convolution layer, we use an InceptionV2 module marked with the dashed box to extract features. In the InceptionV2 module, each convolution layer is an integration of convolution, batch normalization, and activation function ReLu. In detail, the 1×1 sliding window on the left side in the dashed box reduces the loss of detailed features and helps small target detection, and the other convolution layers are used to extract coarse-grained features. Except the 1×1 sliding window mentioned above, the other 1×1 sliding windows are used to reduce the number of channels in the feature map and in the right side, two 3×3 convolution kernels are set to capture coarse-grained semantics whose perceptual field size is equivalent to that of one 5×5 convolution kernel, this setting can reduce the number of parameters and mitigate overfitting compared with one 5×5 convolution, and the use of a larger perceptual field not only makes the semantics of the features richer, but also helps in the detection of large size feature maps. In addition, the branch of the pooling layer is also used to extract features in order to increase the diversity of feature fusion. Then feature maps of four different scales are obtained using the InceptionV2 module and are fused. Such improvement makes multi-scale feature extraction, thus can capture more fine-grained and coarse-grained feature information on the ROIs (region of interest). Compared with the original 3×3 receptive fields, the improved network structure adds two different receptive fields of 1×1 and 5×5, while the depth and width of the network are also increased to improve the generalization ability of the network.

Fig. 7

MRPN structure.

3 Experiments

The detection performance of ASCN is first compared with ResNet50 and VGG16 on our proprietary datasets. Then to further validate the superiority of the proposed method, the proposed method is compared with three classical detection networks including YOLOv3, retinaNet, and SDD. Finally, ablation experiments are conducted to prove the effectiveness of the proposed method. To accelerate the convergence speed and improve the detection performance of the model, all experiments use pre-trained models for transferred learning.

3.1 Experimental datasets

In this paper, the proprietary dataset includes 1402 X-Ray images of spark plug defects with the image size of 1024×1024. Figure 8 (a) shows a spark plug, and one X-Ray image contains the front and the side of two spark plugs, respectively. as shown in Fig. 8 (b) and (c). Figure 8 (d) and (e) show the zoomed images of defect regions in Fig. 8 (b) and (c). The dataset is randomly divided into a training set and a test set. The division ratio is 4:1, i.e., there are 1121 images in the training set and 281 in the test set, and the defects in images are marked by means of LabelImg software in VOC2007 format, and the specific number of dataset is shown in Table 2.

Fig. 8

Samples of spark plugs.

Table 2

Information of Dataset

Defect class	Label	Training set	Test set	Total
cold shut(front)	1	624	156	780
cold shut(side)	1	258	65	323
none-defect	0	239	60	299

The sample size of the dataset largely affects the training effect of the model. Too few samples in the dataset will lead to underfitting, and even if the detection network is excellent, the detection effect will be unacceptable. The common operation of data enhancement is to change the dataset samples (like Translation and rotation) and save them as a new dataset. This operation certainly increases the number of samples, but only by a fixed amount and takes up storage space. Therefore, instead of saving the dataset after enhancement, we preprocess the samples by flipping images horizontally, vertically, panned and cropping them with 50% probability, and then the changed samples are input into the network for training, thus achieving data enhancement. Since the angle of flipping, the distance and direction of panning, and the location and size of cropping are random, the new dataset has a theoretically infinite number of samples and has more generalization ability than the model trained by the former.

3.2 Analysis of experimental results

The hardware devices used in the experiments are Core i9-10900K processors, NVIDIA RTX 3090 graphics cards, and software environments CUDA11.1 and cuDNN11.3. In all detections, the degree of overlap between the actual target and the predicted target, i.e., Intersection over Union (IoU) ratio, is used to evaluate whether the location of the defect is successfully predicted. The prediction target with IoU > 0.5 is considered to contain defects, otherwise it is considered to contain no defects. When evaluating the performance of the network model, it is necessary to consider both the precision and recall rates. Equation (3) is the formula for the precision rate, and Equation (4) is the formula for the recall rate. $Precision = \frac{TP}{TP + FP}$ (3) $Recall = \frac{TP}{TP + FN}$ (4) where, TP (True Positive) is the number of detected frames with IoU > 0.5, FP (False Positive) is the number of detected frames with IoU≤0.5, and FN (False Negative) is the number of true frames that are not detected.

Average Precision (AP) is to sort the detection results of a certain category of targets according to the confidence level, and then, under different confidence threshold conditions, get the integration of the full rate and the accuracy rate and calculate its average value, which can obtain the average precision, the formula for its calculation is in Equation (5).; Fps is the number of images that the network model can detect per second, which is used to evaluate the detection speed of the network model, the formula for its calculation is in Equation (6). $AP = \frac{\sum_{i = 1}^{N} P_{i} (R_{i} R_{i - 1})}{N}$ (5) $fps = \frac{FigureNum}{TotalTime}$ (6) where N is the number of trials, R_i is the confidence level of detection results after sorting, P_i is the value of precision, FigureNum is the total number of detected images, and TotalTime is the total detection time.

Firstly, we compared the proposed method (marked as ➂) with other classical Faster-RCNN networks with ➀ ResNet50 and ➁ VGG as the backbone, respectively, on the spark plug dataset for evaluation. All the Faster-RCNN networks are also transfer trained on the spark plug dataset with the pre-training model by MS COCO dataset. The detection results are shown in Table 3. Compared with ➀ and ➀, the attention module suppresses the background region of the feature map because sym_n effectively fuses deep and shallow features. In addition, MRPN repeatedly extracts and fuses features of different scales, which improves the detection performance of the model. Compared with ➀, the precision and recall rates of ➂ increased by 6% and 4%, respectively, and compared with increased by 9% and 5%, respectively. This indicates that our proposed method has a more significant detection effect. Although ➁ has the lowest fps value, the detection speed of 20 fps fully meets the requirement of real-time detection.

Table 3

Comparison of detection performance of different Faster-RCNNs

Index	Method	AP	fps	Recall
➀	Faster-RCNN(ResNet50 + RPN)	83%	21	93%
➁	Faster-RCNN(VGG16 + RPN)	80%	26	92%
➂	Faster-RCNN (ASCN+MRPN)	89%	20	97%

In training, the SGD optimizer was used to optimize the network parameters with a learning rate of 0.005; momentum factor and weight decay factor of 0.9 and 0.0005, respectively; learning rate decreased to 0.5 times every 80 epoch. The loss value converged after 500 epochs, as shown in Fig. 9.

Fig. 9

Plot of learning rate vs. loss function.

The performance of the method was analyzed in terms of both the regression bounding box and the classification confidence. Figure 10 shows the local enlarged images of the defect detection results of five groups. We can see from group A that ➀ has a missed detection, ➁ detects the defect location but the confidence of detection is only 0.73, while ➂ not only accurately identifies the defect location but also obtains a confidence of 0.96. As shown in group B, ➀, ➁ and ➂ all accurately detect the defect location, but ➀ and ➁ have redundant regression bounding boxes, while the regression bounding box of ➂ is more accurate. As shown in groups C and D, the results of ➀ appear to be misdetected, predicting the location without defects as containing defects. As shown in group E, the confidence level of ➂ is higher than those of ➀ and ➁. Due to the use of symmetric structures, attention modules, and multi-scale information, our algorithm performed better in terms of accuracy and confidence in the regression bounding box.

Fig. 10

Local enlargement of defect detection results for different Faster-RCNN.

The defect location and the degree of attention of defects with respect to the background are also crucial. We took two approaches for the analysis: the first approach is to visualize the feature maps on both sides of the attention module directly; the second approach is to visualize the Gradient Class Activation Map (Grad-CAM) [35] for the feature maps from the networks with ASCN, and ASCN without attention module (SCN), respectively. Figure 11 is an example to show the good performance of the attention module by using the first approach and Fig. 12 is another example by using the second approach.

Fig. 11

Feature map visualization.

Fig. 12

Grad-CAM visualization results.

In Fig. 11, (a) is the input image, (a1)-(a4) show the feature maps of the left side of the four skip connections in ASCN, (b1)-(b4) show the feature maps of the attention module that pass through the skip connection. We can find that, after passing the attention module the color of the upper left region of the feature map becomes darker and bluer; while the spark plug region becomes brighter and greener, i.e., the irrelevant picture text information is suppressed and the spark plug to be detected is highlighted. As can be seen in Fig. 11 (a2) and (b2), (a3) and (b3), (a4) and (b4), the background region darkens, becomes blue and is suppressed.

We also analyzed the attention module visually with Grad-CAM in Fig. 12, which was formed by weighting each channel of the feature map based on the gradient mean of the feature map as the weight and superimposing it. Figure 12(a) is the original input image, (a1)-(a4) show the 4 visualizations of SCN after the skip connection, and (b1)-(b4) show the 4 visualizations of ASCN after the attention module. The Grad-CAM of ASCN has a more complete spark plug region compared with SCN, indicating that it can retain more useful spatial information features. In summary, the attention module embedded in the skip connection effectively suppresses useless information while highlighting salient features that are useful for the task.

To further validate the defect detection performance of the improved Faster-RCNN model in this paper, we compared it with YOLOv3 [9] and SSD [5] and retinaNet [10], respectively. Figure 13 illustrates the detection results of the cold shut regions using different detection networks. It shows that retinaNet has a lower confidence level and shows missed detections, YOLOv3 has poor regression accuracy and missed detections, while our proposed method is the best in terms of confidence level and regression accuracy.

Fig. 13

Local enlargement of defect detection results for different models.

Besides, we also show a performed quantitative analysis of different detection results, and the average values of multiple experimental results are shown in Table 4. Since YOLOv3, SSD, and retinaNet simplify multiple detection stages into a one-stage neural network, they have fast detection speed, but they are not applicable to the detection of spark plug cold shut defects, and their AP and Recall values of defect detection are low. Compared with other algorithms, the precision rate of our method was improved by more than 8% and the recall rate was improved by more than 2%.

Table 4

Comparison of detection performance of different network models

Method	AP	Recall
YOLOv3	81%	94%
SSD	78%	93%
retinaNet	75%	95%
Our method	89%	97%

Finally, to further explore the reliability of the method and evaluate the effectiveness of ASCN and MRPN, ablation experiments were conducted. Training was performed using two methods: only the ASCN was added to the Faster-RCNN method and both the ASCN and MRPN were simultaneously added to the Faster-RCNN method. The test results obtained by the different strategies were analyzed, respectively. Table 5 shows the results of the AP and Recall values obtained by different strategies, and it shows that the highest AP and Recall were obtained by adding ASCN and MRPN simultaneously. Adding MRPN increased both AP and Recall by 2%, proving the effectiveness of the proposed method. In conclusion, the proposed method is an effective method for detecting the cold shut defects of spark plugs.

Table 5

Ablation experiment on our network

Method	ASCN	MRPN	AP	Recall
Faster-RCNN	√	×	87%	95%
	√	√	89%	97%

4 Discussions and Conclusion

In this paper, a fast and accurate end-to-end spark plug defect detection algorithm is proposed. The method is an improved version of the original Faster-RCNN network. First, the symmetrical convolutional neural network with embedded attention module is used as the feature extraction network, and then the RPN network structure is improved by using InceptionV2, to increase the information of different scales.

Through the experiment, the following conclusions are obtained: (1) In the improved Faster-RCNN, the proposed feature extraction network ASCN increases the AP value by more than 6%. (2) The MRPN network increases the AP value by more than 2% compared with the original Faster-RCNN, since it adds sliding windows of different sizes to improve the detection ability and generalization ability of the algorithm. However, the MRPN model brings the problems of parameter number increase and detection speed reduction. (3) When our model is tested on the spark plug data set, it achieves 89% AP and 97% Recall, furthermore, our model can detect at 20fps when the input image size is 1024×1024×3, achieving an end-to-end fast and accurate detection of spark plug defects.

Compared with other algorithms [5 , 14], the precision rate of our method was improved by more than 6% and the recall rate was improved by more than 2%. Our contributions are mainly as follows: (1) We propose ASCN to extract feature maps. (2) We propose MRPN to extract multi-scale candidate boxes. (3) We propose an efficient method for spark plug defect detection. A limitation of this work is that the proposed model requires a longer detection time than the original Faster-RCNN, though it fully meets the requirement of real-time detection. As the workpiece is usually not uniform, the thick parts of the workpiece show well in the X-ray image when using high-energy X-rays, but the thin parts are overexposed in the X-ray image. On the contrary, when low-energy X-rays are used, the thick parts are underexposed, and the thin parts of the workpieces are shown well in the X-ray images. In such cases, defections are difficult to be detected. Therefore, we will study how to further improve the effect of defect detection by using fusion model of the X-ray images with different energies.

Footnotes

Acknowledgments

This work was supported by the National Natural Science Foundation of China (61801438), Science and Technology Innovation Project of Colleges and Universities of Shanxi Province (2020L0282), Top Young Academic leader Project of North University of China (QX201801), the Natural Science Foundation of Shanxi Province (201901D111153), Natural Science Foundation of Shanxi Province of China(No. 201901D111161, No. 201901D211246), Open Research Fund Project of Fundamental Science on Underground Target Damage Technology Laboratory(No. DXMBJJ2021-02), the Research Project Supported by Shanxi Scholarship Council of China (2021-111).

References

W Book Review: The Spark Plug, Journal of Education 99(14) (1924), 387–387.

Moosavian

, Khazaee

, Najafi

, et al., Spark plug fault recognition based on sensor fusion and classifier combination using Dempster–Shafer evidence theory, Applied Acoustics 93 (2015), 120–129.

Zhao

, He

, et al., A sparse-representation-based robust inspection system for hidden defects classification in casting components, Neurocomputing 153 (12015), 1–10.

Hashim

M.A.

, Nasef

M.H.

, Kabeel

A.E.

, et al., Combustion fault detection technique of spark ignition engine based on wavelet packet transform and artificial neural network, Alexandria Engineering Journal 59(5) (2020), 3687–3697.

Liu

L.W.

, et al., Ssd: single shot multibox detector, European Conference on Computer Vision, Springer, Cham, 2016; 21–37.

C.Y.

, Liu

, Ranga

, et al., DSSD: deconvolutional single shot detector,” arXiv:1701.06659, 2017.

Redmon

, Divvala

, Girshick

, et al., You only look once: Unified, real-time object detection, Proc IEEE Conf Comput Vis Pattern Recognit (CVPR) (2016), 779–788.

Redmon

, Farhadi

, YOLO9000: better, faster, stronger, Proc IEEE Conf Comput Vis Pattern Recognit (2017), 7263–7271.

Redmon

, Farhadi

, Yolov3: an incremental improvement, arXiv: 1804.02767, 2018.

10.

Lin

, Goyal

, Girshick

, et al., Focal Loss for Dense Object Detection, IEEE Trans Pattern Anal Mach Intell 42(2) (2020), 318–327.

11.

Girshick

, Donahue

, Darrell

, et al., Region-based convolutional networks for accurate object detection and segmentation, IEEE Trans Pattern Anal Mach Intell 38 (2015), 142–158.

12.

Ren

, He

, Girshick

, et al., Faster r-CNN: towards real-time object detection with region proposal networks, Advances in Neural Information Processing Systems (2015), 91–99.

13.

Feng

, Cai

, Wang

, et al., Dual residual convolutional neural network (DRCNN) for low-dose CT imaging, J Xray Sci Technol 29(1) (2021), 91–109.

14.

Ren

, He

, Girshick

, Sun

, Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, IEEE Trans Pattern Anal Mach Intell 39 (2017), 1137–1149.

15.

, Gkioxari

, Dollár

, Girshick

, Mask R-CNN, IEEE Trans Pattern Anal Mach Intell 42(2) (2020), 386–397.

16.

, Ye

, Zhao

, Wang

, Strip Steel Surface Defect Detection Based on Improved YOLOv3 Algorithm, Acta Electronica Sinica 48 (2020), 1284–1292.

17.

Xing

, Jia

, A convolutional neural network-based method for workpiece surface defect detection, Measurement 176(1) (2021), 109185.

18.

Walters

SD.

, Howson

P.A.

, Howlett

, Production testing of spark plugs using a neural network, International Conference on Knowledge-Based and Intelligent Information and Engineering Systems, Springer, Berlin, Heidelberg, 2005; 74–80.

19.

Xue

, Hei

, Chen

, et al., An Efficient Method of Casting Defects Detection Based on Deep Learning, IEEE International Conference on Advances in Electrical Engineering and Computer Applications (AEECA), (2020), 480–483.

20.

, Yan

, et al., Filtered selective search and evenly distributed convolutional neural networks for casting defects recognition, Journal of Materials Processing Technology 292 (2021), 117064.

21.

Liu

, Tang

, Duan

, Tan

, TruingDet: Towards high-quality visual automatic defect inspection for mental surface, Optics and Lasers in Engineering 138 (2021), 106423.

22.

Shu

Y.F.

, Li

, et al., Deep learning-based fast recognition of commutator surface defects, Measurement 178 (2021), 109324.

23.

Wang

, Hu

, Chen

, Yin

, Self-attention guided model for defect detection of aluminium alloy casting on X-ray image, Computers & Electrical Engineering 88 (2020), 106821.

24.

Zhang

, Shen

, Solder Joint Defect Detection in the Connectors Using Improved Faster-RCNN Algorithm, Appl Sci 11(2) (2021), 576.

25.

Woo

, Park

, Lee

, Kweon

, CBAM: Convolutional Block Attention Module, ECCV, 2018.

26.

Chen

, Huang

, Xia

, Zhang

, Nonlocal spatial attention module for image classification, International Journal of Advanced Robotic Systems 17 (2020), 1–10.

27.

, He

, Xia

, et al., X-CTRSNet: 3D cervical vertebra CT reconstruction and segmentation directly from 2D X-ray images, Knowledge-Based Systems 236 (2022), 107680.

28.

Liu

, Yang

, Gao

, et al., Remove appearance shift for ultrasound image segmentation via fast and universal style transfer, IEEE 17th International Symposium on Biomedical Imaging (ISBI) (2020), 1824–1828.

29.

Chen

, Liu

, Achuthan

, WODIS: Water Obstacle Detection Network Based on Image Segmentation for Autonomous Surface Vehicles in Maritime Environments, IEEE Transactions on Instrumentation and Measurement 70 (2021), 7503213.

30.

Szegedy

, Vanhoucke

, et al., Rethinking the Inception Architecture for Computer Vision, arXiv: 1512.00567, 2015.

31.

Lin

T.Y.

, Maire

, Belongie

, et al., Microsoft COCO: Common Objects in Context, European Conference on Computer Vision (ECCV) (2014), 740–755.

32.

Lin

, Dollár

, Girshick

, He

, et al., Feature Pyramid Networks for Object Detection, arXiv: 1612.03144, 2016.

33.

Ronneberger

, Fischer

, Brox

, U-Net: Convolutional networks for biomedical image segmentation, Proc Int Conf Med Image Comput Comput-Assist Intervent Cham, Switzerland: 2015; 234–241.

34.

, Zhang

, Ren

, et al., Deep residual learning for image recognition, Proc IEEE Conf Comput Vis Pattern Recognit, (CVPR) (2016), 770–778.

35.

Selvaraju

R.R.

, et al., Grad-CAM: Visual explanations from deep net-works via gradient-based localization, Proc IEEE Conf Comput Vis Pattern Recognit, (2017), 618–626.