Component identification and defect detection in transmission lines based on deep learning

2.1 Traditional target detection method

The traditional target detection method usually uses a sliding window to obtain a large number of potential candidate regions [29]. The size of the candidate regions generated by sliding frames is fixed, and a large number adds a large burden to subsequent calculations [30]. In addition, in the aspect of feature extraction, traditional object detection algorithms will consider designing many very complicated manual features to perform feature extraction on images. When encountering complex and changeable data sets, they are often poorly robust. Deep learning has certain universality, so this chapter mainly discusses related research on object detection based on deep learning [31].

Object recognition and localization is an important research direction in the field of machine vision and pattern recognition. The subject involves multiple research contents such as data processing, feature extraction, classifiers, and so on. The data needs to be pre-processed before it enters the network. It is the processing of the detected image in the early stage of object recognition, such as image denoising, image cropping, brightness transformation, normalization, and other operations [32]. After data pre-processing, feature extraction operations are performed. There are many traditional feature extraction methods, such as SIFT and SURF for local feature extraction; Haar and LBP for face feature extraction [33]; HOG for pedestrian detection, and so on. The role of the classifier is to reasonably classify the extracted features. Common classifiers include SVM, Bayes. [34].

However, in practical applications, the background of the image is diverse, the shape of the object is variable, and the angle of view of the camera changes a lot, which makes the task of object detection more difficult. In recent years, deep learning has made rapid progress in the field of security, especially in the areas of object recognition, detection, and tracking. This chapter mainly studies how to use the results of deep learning in computer vision to achieve the target detection of transmission lines. Generally speaking, object recognition and positioning are divided into two parts. First, find the positions of all foreground targets in the scene to get the position information of the candidate frame of the target; the other part is to determine which object the object belongs to based on the found object category information. As shown in Fig. 1, this process is target detection based on deep learning algorithms [35].

OPEN IN VIEWER

Fig. 1

Deep learning-based object detection steps.

Deep learning-based object detection algorithms usually have four steps.

The first step is to perform image preprocessing on the input image or video frame, including: normalizing the data to reduce the difference between different pictures; and performing noise reduction, rotation, and scaling operations on the image.

The second step is to generate candidate frames that potentially contain targets by sliding frames or RPN networks.

The third step is feature extraction to obtain the feature vector, and the fourth step is to use the feature vector to classify the candidate frames.

The order of the second and third steps directly affects the entire calculation of the model. If the candidate frames are generated directly first since the image will contain many objects with different sizes and shapes, the general frame selection method needs to traverse the entire image, so many redundant frames will be generated. If feature extraction is performed on these redundant frames separately, the calculation pressure will be greatly increased. So the idea of the RPN network is to first use a convolutional network for feature extraction, then share features, and perform sliding frame selection on the extracted feature layer. Common candidate region methods include selective search, edge boxes, and RPN based on convolutional neural networks.

The first use of convolutional neural networks for feature extraction in target detection is in the R-CNN network the scholar Rossi. Girshick proposed an R-CNN network using SS plus CNN, which opened up the use of deep learning to solve target detection of new ideas. In this method, R-CNN uses a selective search algorithm to extract 2000 candidate frames from the original image. Then each candidate box is scaled to a fixed size, all feature maps uniformly input to a CNN convolutional network to extract features, and the fourth step is to input the extracted features to an SVM classifier for classification. R-CNN’s subsequent SPP-Net, FastRCNN, FasterRCNN, and other networks, the basic process is similar to the above process, except that the region proposal extraction method, and the region selection and feature extraction order are different.

At present, deep learning-based object detection methods are mainly divided into two categories. The first category is one-stage deep learning methods, such as SSD, YOLO. The second category is based on two-stage methods, mainly represented by FasterR-CNN.

(1) Framework introduction of Faster R-CNN

The framework flow of FasterR-CNN is shown in Fig. 2. It is mainly composed of the following parts: The first part is the RPN network. This module is responsible for generating the coordinates of the candidate box and whether it is a foreground score. The second part is the FastR-CNN detection module responsible for detection. In the overall structure, the previous convolutional network is used to extract features, and the later RPN network and FastR-CNN detection network share the previous features. The RPN network will generate candidate boxes of different sizes on the anchor points of each feature map. The detection network will detect these candidate regions and identify the target category in the candidate regions.

OPEN IN VIEWER

Fig. 2

Introduction to Faster R-CNN framework.

(2) Regional generation network of Faster R-CNN

The innovation of FasterR-CNN network lies in the RPN network. The target detection network before this method first uses the SS algorithm to perform sliding frame selection on the entire feature map. For example, the SPP-Net network and FastR-CNN network both improve the traditional sliding marquee detection network and improve the efficiency of the detection network. However, both of these methods take a long time to generate candidate region boxes. But the region generation network (RPN) method shares the features generated by the convolutional network, so the total detection region generation consumes very little time. The basic idea of the area generation network is shown in Fig. 3.

OPEN IN VIEWER

Fig. 3

Faster R-CNN generated anchor schematic.

The network generates anchors for each anchor point on the feature map and performs regression and classification of the anchors and foreground backgrounds. Anchor bounding boxes are some candidate regions defined in advance: each anchor point on the feature map will generate 9 different candidate regions with an area of {128 * 128, 256 * 256, 512 * 512} and three ratios of length and width {1:1, 1:2, 2:1}. These candidate regions are called anchors.

The structure of the regional generation network is a full convolution structure. This structure can make the entire network do end-to-end training. Its operation process is as follows:

Generate anchors, and use border regression for all anchors. The anchors generated here are exactly the same as during training.

The RPN network outputs the coordinates of the last foreground box and the score value of the probability.

(3) Loss function design

The loss during network training consists of two parts, one is the loss of the regression position, and the other is the classification loss. The total loss function can be expressed as: L ( x , c , l , g ) = 1 N ( L conf ( x , c ) + aL loc ( x , l , g ) )(1)

Where N is the number of matches between the default box and the real box, α is the weight factor and is generally set to 1, C is the confidence of each class, l and g are the parameters of the default box and the real box, including the center position coordinates and width and height. _Conf (x, c) is the loss of classification confidence, using multi-class loss Softmax, L _loc is used to return the center position and width and height of bounding boxes, and using Smooth_L1 loss, Smooth_L1 is calculated as follows: Smooth L 1 ( X ) = f ( X ) = { | X | - 0.5 | X | ≥ 1 0.5 X 2 | X | < 1(2)

The image size of aerial images is very large, and the deep learning object detection framework often performs a compression process on the images [36]. Therefore, before the picture is input into the deep learning framework, the image is generally compressed.

A typical SSD training sample size is 300×300. However, the insulator aerial images are often ultra-clear pictures with large pixels, so it is necessary to compress the pictures to improve the training speed. This article compresses the original image to 1024×1024. Image compression refers to reducing the original image into a new image at a specified ratio.

If you directly scale the images by scaling the images, the coordinates of many points are calculated as decimals. Therefore, this paper uses the interpolation algorithm to approximate the coordinates. Interpolation algorithms usually include nearest neighbor interpolation, bilinear interpolation, and higher-order interpolation. Nearest neighbor interpolation is the simplest interpolation method. The output pixel value of this method is the pixel value of the nearest sampling point in the input image. Higher-order interpolation can save more details of the original image than neighboring interpolation, but this method requires longer calculation time. Bilinear interpolation is the algorithm used in this paper. It specifies the output pixel value as the weighted average of the pixel gray values of the sample points in the neighborhood nearest to it in the input image [37]. Considering that it takes a lot of time to process large batches of samples, this paper adopts the bilinear interpolation compression method.

The steps for bilinear interpolation are as follows:

(1) Calculating the scaling factor

First, calculate the horizontal compression factor H _f and the vertical compression factor V _f . Assuming that the pixel point before scaling is P₀(x₀,y₀), the corresponding pixel point of the scaled image is P₁(i,j). The scaling formula is as in formula (3): { H f = srcWidth / dstWidth V f = srcHeight / dstHeight(3)

Where srcWidth and srcHeigh represent the length and width of the original image; dstWidth and dstHeigh represent the length and width of the new scaled image.

(2) Calculate the target point in the original image

Since the coordinates of the scaled points are not necessarily integers after they are mapped back to the original image, the original pixel coordinates P₀(x₀, y₀) corresponding to the scaled pixels is first obtained here. { x 0 = i × H f y 0 = j × V f(4)

(3) Calculate the coordinates of the 4 points closest to P0

The third step needs to calculate the coordinates (x₁, y1), (x₁, y2), (x₂, y1), (x₂, y2) of the four pixels closest to P₀(x₀, y0) in the original image.

(4) Calculate weight { w 1 = ( x 0 - x 1 ) ( y 0 - y 1 ) w 2 = ( x 2 - x 0 ) ( y 1 - y 0 ) w 3 = ( x 0 - x 1 ) ( y 2 - y 0 ) w 4 = ( x 2 - x 0 ) ( y 2 - y 0 )(5)

(5) Calculate P₀ point coordinates

P 0 ( x 0 , y 0 ) = w 1 P 1 ( x 1 , y 1 ) + w 2 P 2 ( x 2 , y 1 ) + w 3 P 3 ( x 1 , y 2 ) + w 4 P 4 ( x 2 , y 2 )(6)

According to the above formula, all the sample pictures are traversed in order to obtain the compressed image.

This article does not crop all the sample pictures when processing the samples, so we must first make a judgment on the proportion of the objects in the sample pictures. This article divides the goals into small goals, medium goals, and large goals according to the target occupying less than 5%, between 5%   12%, and greater than 12%. Finally, only the pictures classified as the small target and medium target are operated. For the pictures containing samples of the large target, no cropping operation is performed in this article.

The details of the sample pretreatment process used in the actual experiments in this paper are shown in Fig. 4 below:

OPEN IN VIEWER

Fig. 4

Sample expansion and preprocessing process.

First, calculate the size of the target frame, and then crop the small and medium targets. The processed image and the original image are subjected to conventional operations such as image compression, brightness conversion, and horizontal flip. Then this part of the expansion and the original picture set together form a training sample library.

3.1 Experimental data preparation

The data set in this article is mainly based on the photos taken during the sampling inspection. This photo covers the four seasons of spring, summer, autumn, and winter. The shooting locations are diverse and the resolution is high. This article focuses on the identification and defect diagnosis of transmission line components and chooses five different types of components: equalizing ring 1, equalizing ring 2, complete vibration hammer, bad vibration hammer, and bird’s nest model training and sampling are selected as the default values. Among them, the model training samples have 1,000 training samples in each category, a total of 5000 training samples, and the image size of each sample is 6000×4000.

3.2 Experimental data set processing

First, the training samples are uniformly scaled to 1200×800, and then the complete power component target displayed in each image of the training set is labeled as a frame, the frame coordinates are recorded, and a classification label is provided.

In the test sample, 500 images of each category were selected as the test set, and the test set did not contain training samples. When identifying and classifying the test group, all electrical components (including incomplete and electric components) included in the training sample must be marked and classified and scored. Think of this as a successful identification.

3.3 Experimental model training

This paper is based on the MXNET framework and uses Faster-RCNN to implement network model training. The VGG16 network and the ResNet-101 network were used to initialize the pre-trained ImageNet network to obtain three models with different network layers. The data set of region 2.2 is used as a training sample. The training number of each model training is 20, the batch size is 128, the learning rate is 0.001, the weight attenuation rate is 0.0005, and the number of candidate regions before and after non-maximum suppression is 6000 and 300, respectively. In this experiment, as a criterion for evaluating the quality of the model, the correct rate, recall rate, and missing recognition rate were used. The correct percentage indicates the ratio of the number of targets that are correctly identified to the number of all targets that are identified. The ratio of the number of sample targets. The unrecognized recognition rate indicates the ratio of the number of unrecognized targets to the number of all sample targets.

4.1 Comparison of model recognition effect

In the experiment, the test set in section 3.3 was used as the test set for 3 different models. The two models obtained in Section 3.3 were used as the test models. The accuracy rate, recall rate, and missed recognition rate were used as the criterion for evaluating the quality of the model. The application of the two models in the identification of transmission line components is shown in Tables 1 and 2.

The accuracy of the five types of targets detected under two different models is shown in Fig. 5.

OPEN IN VIEWER

Fig. 5

Accuracy of 5 types of targets detected under two different models.

According to the experimental results shown in Tables 1 and 2 and Fig. 5, the recall rate, correct rate, and missed the recognition rate of the 5 types of targets detected under 2 different models can be obtained. For the recall rate, the ResNet-101 model recall rate is the highest, followed by the VGGl6 model. In terms of accuracy, the ResNet-101 model is the best, followed by the VGGl6 model. For the leak recognition rate, the ResNet-101 model has the lowest leak recognition rate, followed by the VGGl6 model. It can be seen from Tables 1 and 2 that the recognition effect for the pressure-equalizing ring 1, the pressure-equalizing ring 2, and the bird’s nest is far superior to that of the shockproof hammer. The reasons may be as follows: 1) the structural characteristics of the vibration hammer are not obvious enough; 2) the training sample data of the vibration hammer is insufficient, which leads to the model’s generalization ability is not good enough.

According to the experimental results in Tables 1 and 2, aiming at the problem of unsatisfactory seismic hammer recognition and single picture recognition time optimization, the optimal model ResNet-101 model is selected for experiments. The experimental direction is to modify the convolution kernel of its convolution structure. It is mainly to modify the size of the first layer convolution kernel. According to the content in Section 1, it can be known that different convolution kernel sizes have an impact on recognition accuracy and recognition time.

The size of the convolution kernel in the first convolution layer of the ResNet-l01 model network is 7, according to the experimental idea, gradually reduce the size of the convolution kernel for model training, and the parameter settings are consistent with the parameter settings in Section 3.3, and then use the trained model, the samples were tested, and the recall rate and recognition time were used as experimental indicators. The number of all samples was 500. The experimental results are shown in Table 3.

The recall rate of the detected 5 types of targets at different convolution kernel sizes is shown in Fig. 6.

OPEN IN VIEWER

Fig. 6

Recall rates for different convolution kernel sizes.

According to Fig. 6, it can be seen that different convolution kernel sizes have an impact on the accuracy of the detection, and it can be obtained that as the size of the convolution kernel decreases, the recall rate continues to decrease. Kernels have large receptive fields, and convolution kernels with receptive fields have high recognition accuracy. As the size of the convolution kernel decreases, the recognition time of a single picture also decreases. This is because the number of parameters obtained by the recognition of different convolution kernels is different, and the parameters obtained by small convolution kernels. The number is small, and the number of parameters obtained by the large convolution kernel is large. Therefore, the model can be optimized by adjusting the size of the convolution kernel to meet its actual needs-whether a higher recall rate or a lower recognition time is required.

In this experiment, the images of the five different components in the training set in section 3.2 were rotated by 90°, 180°, and 270°, respectively, and a total of 20,000 training samples were obtained. The set is used as a training sample for the ResNet-101 model. The model is trained. The training parameter settings are consistent with the parameter settings in Section 3.3. Then the tested model is used to test the test set in Section 3.2. The recall rate, correct rate, and missed the recognition rate of the images of 5 different parts are shown in Fig. 7.

OPEN IN VIEWER

Fig. 7

ResNet-101 model verification results for 5 types of targets after data expansion.

According to the experimental results in Table 4 and Fig. 7, it can be obtained that the training model obtained by using the expanded data set for model training, the recall rate of any category identified using this training model is better than the recall rate before expansion, and the correct rate, the sum recognition rate is also better than that before the expansion, so it can be obtained that expanding the data set by the rotation transformation method can improve the recognition accuracy rate.

In the experimental part, this paper experimentally verifies the adaptive cropping algorithm proposed in the previous article. The self-cropping algorithm crops the sample picture into one or more small pictures according to the position of the target frame marked by the sample. In this algorithm, the crop coefficient K needs to be calculated based on the annotation information of the image XML. The coefficient K needs to be compared with the threshold Q to determine whether the original image is cut into one or two independent small images. Therefore, the value of the threshold Q directly affects the experimental results. In this set of comparative experiments, the basic network used in the experiment is resnet101 + FPN-SSD, and the iteration algebra is set to 100,000 generations. The experimental results are shown in Fig. 8.

OPEN IN VIEWER

Fig. 8

Experimental results at different thresholds Q.

It can be found from Fig. 8 that the experimental results increase first and then decrease with the change of the threshold. The reason for this phenomenon is that when the threshold is increased, more small images of small-scale targets are cropped. However, because the detection samples are scattered at different scales, improving the accuracy of target recognition at small scales does not necessarily change the targets at other scales. The final experimental results show that the threshold Q is better than 0.24 [38].