Pixel-level pruning deep supervision UNet++ for detecting fabric defects

Abstract

Due to the complexity of fabric texture and the diversity of defect types, fabric defect detection is quite challenging. At present, fabric defect detection algorithms based on deep learning have achieved good detection results, but there are still some key issues to be solved. First, the existing network models remain unchanged once they have been built. When they are used to detect defects of different kinds of fabrics, the network model cannot be adjusted flexibly according to the characteristics of the fabrics, which reduces the efficiency of algorithm detection. Second, the imbalanced category of fabric defect samples makes model training more challenging. Moreover, the number of defective pixels is very small compared with the number of pixels in the whole image, which further increases the difficulty of fabric defect detection methods. To solve these problems, we propose a pixel-level pruning end-to-end deep supervision DSUNet++ architecture for fabric defect detection. The DSUNet++ architecture consists of an encoder, a decoder, and a series of cascade operations for fusing the detailed features of the shallow layer and the abstract features of the deep layer. The deep supervision is embedded into the outputs of different depths in the DSUNet++, which can prune the network reasonably according to the characteristics of different kinds of fabrics, so as to balance the depth, speed and precision of the network. Furthermore, the cross-entropy loss function weighted by the median frequency CEloss_MFB is introduced to overcome the problem of imbalanced fabric defect sample categories and detection rate decrease of small pixel defects. The experimental results show that the average detection rate of the method is 97.68% and 99.01% in the defect detection of raw fabric and patterned fabric, respectively.

Keywords

Fabric defect detection deep Learning deep supervision UNet++CEloss_MFB

The textile industry is a pillar industry of the national economy and social development in China. Effective detection and guarantee of textile surface quality is one of the key links for modern textile enterprises to control cost and improve product competitiveness.¹ How to make textile enterprises more competitive, among which ensuring the quality of fabrics, is very important.^2–4 At present, most of textile enterprises still use the traditional manual detection method for fabric defect detection. Due to the limitation of human physiological characteristics, the speed, accuracy and consistency of test results provided by traditional manual detection methods cannot meet the demand of modern mass production. With the rapid development of artificial intelligence and computer science, computer vision and pattern recognition technology have been widely used in the field of industrial surface defect detection.^5,6 These technologies not only provide a good guarantee for the objective, stable and efficient fabric defect detection, but also conform to the development trend of automation and intelligence in the textile industry.

In the early stages, the traditional machine vision detection method has been widely used in the field of fabric defect detection. It extracts the fabric texture feature of the defect-free fabric sample as the template, and then carries out defect detection according to the similarity between the image to be detected and the template generated in the detection stage. Li et al.⁷ proposed a novel and robust fabric defect detection method based on the low-rank representation technique. The time complexity of this method was very high because the eigenvalue decomposition on the blocked matrix was used for dimension reduction. Kang and Zhang⁸ proposed a fabric defect detection method based on sparse dictionary learning. This method can excellently detect defects of fabric with small period textures, but it is not accurate enough to detect the defects of fabric with large period textures. Zhang et al.⁹ proposed a fabric defect detection method based on the salience metric for color dissimilarity and positional aggregation. The algorithm can excellently detect the patterned fabric, but the time complexity needs to be further improved because of the large number of similarity calculations involved. For these traditional approaches, their features must be manually designed to fit a particular task. In the industrial environment, due to the variety of fabric defects and the uncertainties of the shape, size and type of defects, it is difficult for the traditional machine vision algorithm to complete the modeling and migration of defect features, and the reusability is not great, which will waste a lot of labor and material costs. Compared with traditional methods, deep learning methods do not require manual feature design, they can automatically extract features in an end-to-end manner and are more robust to changes.¹⁰ Therefore, the defect detection method based on deep learning has higher accuracy and faster detection speed, and has been widely used in the defect detection of industrial fabrics.

The convolutional neural network (CNN) is the most representative model of deep learning, which has been widely applied in many fields, such as image classification, object detection and image segmentation.^11,12 In recent years, many defect detection methods based on CNN have been widely used in the textile industry. This kind of method mainly learns the labeled defect samples, constructs the detection model, applies the detection model to the tested samples for detection, and finally gets the detection result.

Wang et al.¹³ proposed a deep CNN model for defect detection, so as to control product quality. This model can automatically extract powerful features for defect detection with less prior knowledge of images. The proposed method has better detection accuracy and speed when tested on the baseline datasets. Hu et al.¹⁴ proposed a new unsupervised fabric defect detection method based on a deep convolutional generative adversarial network (DCGAN). The method extends the standard DCGAN by introducing a new encoder component. With the assistance of the encoder, a given query image can be reconstructed, and only the normal texture can be preserved in the reconstruction. Therefore, when subtracting the reconstructed image from the original image, a residual map can be created to highlight the potential regions. Li et al.¹⁵ proposed a compact CNN structure for detecting some common fabric defects. Because the proposed architecture uses several micro-architectures with multi-layer perceptron to optimize the network, compared with mainstream CNNs, it achieves excellent detection accuracy. Considering the CNN cannot handle small sample problems well in classification tasks, Wei et al.¹⁶ combined compressive sensing and the convolutional neural network (CS-CNN) to compress and augment small sample data by using the compressive sampling theorem. The data features are classified directly by a CNN through compressed sampling. Experiments show that this method can effectively improve the classification accuracy of defect samples even when the number of defect samples is small. Zhang et al.¹⁷ proposed an attention-gate-based U-shaped reconstruction network (AGUR-Net) and a dual-threshold segmentation post-processing method. AGUR-Net consists of an encoder, an atrous spatial pyramid pooling module and an attention gate weighted fusion residual decoder. The experimental results show that the proposed method can better detect and locate the defects of color-patterned fabrics compared with several other state-of-the-art unsupervised fabric defect detection methods. Dlamini et al.¹⁸ proposed introducing a real-time machine vision system with the aim of detecting defects in functional textile fabrics with good precision at relatively fast detection speeds to assist in textile industry quality control. Filtering is employed to denoise and enhance some features. YOLOv4 is used for localization where the system is trained with weights of a pretrained model. Zhang et al.¹⁹ proposed a U-shaped Swin transformer network based on the quadtree attention framework for unsupervised yarn-dyed fabric defect detection. The method was by a U-shaped network based on a Swin transformer, the Swin transformer adopts local attention effectively to learn features, and the U-shaped network realizes pixel-level reconstruction of images. The proposed method effectively avoids the difficulty of collecting a large number of defective samples and manual labeling. Experimental results show that the proposed method can achieve higher accuracy of fabric defect detection and location compared with other methods. Zheng et al.²⁰ found an objective evaluation method for sizing effects on strand integrity and performance of belt-forming of E-glass filament yarn. The image of the cross-section of yarn is captured by Keyence VK-X110 and analyzed by Image-Pro Plus to obtain the central position and diameter of each filament. By using the least squares method, the minimum rectangle containing all groups of filaments is obtained to describe the basic contour of the yarn cross-section. The outer envelope polygon is divided into several equal parts to calculate the coefficient variation of thickness distribution and yarn distribution. These objective values well reflect the strand integrity and feature of the belt-forming of yarn, and are consistent with the subjective evaluation. Bao et al.²¹ proposed a comprehensive defect detection system of tube yarn by combining machine vision and deep learning methods. In the whole system, we inspect the weight through a weight sensor first. Then, we propose a multi-scale cross-fusion attention module to improve the MobileNetV2, and combine with a machine vision image feature extraction method for hairiness detection. Finally, the modified MobileNetV2 network is used as the backbone of the YOLOX network, making the YOLOX is lighter and achieves more efficient stain detection of tube yarn. The experimental results show that the system is efficient, precise, and can be applied to actual production.

The above deep learning methods have achieved better defection results to some extent. However, there are still some challenging issues to be solved. First, there are various kinds of fabric products in the actual production, their backgrounds are complex and changeable, and their defect patterns are different. This requires the structure of the designed network model to be adjusted according to the characteristics of different fabric products. Yet the existing network models remain unchanged once they are built. Second, there are many defective samples in which the proportion of defective pixels in the image is very low.²² Figure 1 shows some samples with a small number of defective pixels. Figure 2 gives the ratio of samples with a small number of defective pixels to the whole samples in the raw fabric dataset and patterned fabric dataset, in which the proportion of defective pixels in the image is less than or equal to 5%. The defect detection performance of existing methods is greatly affected by these samples with a small number of defective pixels. Third, the number of samples for different types of defects is different. The number of common fabric defect samples is relatively large, while the defect samples of some fabrics are not easy to obtain. In particular, for some new types of fabrics, the number of defect samples in the sample set is almost zero. The imbalanced distribution of defect samples will bring great resistance to the learning of the network.

Figure 1.

Partial samples with small proportion of defect pixels.

Figure 2.

The proportion of small pixel defects in raw fabric and patterned fabric sample sets.

In this paper, we propose a pixel-level pruning end-to-end deep supervision UNet++ architecture (DSUNet++) for fabric defect detection, which can be adjusted according to different types of fabrics, and the defect detection performance can be improved under imbalanced samples and samples with a few defective pixels. The contributions of the adapted method are as follows:

The adapted DSUNet++ architecture can reasonably prune the network according to the characteristics of different fabric products, such as the background texture, the similarity of defects and backgrounds, so as better to design the depth of the network, and balance the depth, speed and precision of the network.

The median frequency weighted loss function, CEloss_MFB, is introduced to deal with the imbalance of fabric defect sample categories, and the problem of reduced detection rate with fewer defective pixels.

The structure of this paper is organized as follows. In the Related works section, the related works are introduced. In the Adapted algorithm section, the algorithm principle of the UNet++ network and the principle of DSUNet++ architecture are described. In the Detection results and analysis section, the detection results and the analysis of experiments are provided to validate the adapted method on databases of real fabric images, and the influence of DSUNet++ network depth on detection results is analyzed. In the Comparisons with other methods section, the detection results of the adapted method are compared with the U-Net and other algorithms. Finally, the final section is the conclusion.

Related works

With the further development of deep learning algorithms, many deep learning models are emerging. For example, DenseNet,^23,24 SegNet,^25,26 FCN,^27,28 and so on. These models have been widely used in image segmentation and object detection. Among these models, U-Net^29–31 has been widely considered for its unique structure and segmentation performance. The original intention of the U-Net network is to solve the problems in biomedical image segmentation. Because of its good effect, it has been widely used in various aspects of semantic segmentation. The U-Net network is a variant of fully convolutional networks (FCNs),³² which is a completely symmetric encoder-decoder structure. The encoder part is down-sampling, responsible for feature extraction. The decoder part generally adopts up-sampling, uses the abstract features of the encoder part to restore the original image size, and finally gets the segmentation result. In the U-Net structure, the feature graph before the first down-sampling is directly connected with the convolution layer rather far back through a skip connection. The corresponding two convolution layers before and after are connected to play the role of the information supplement. The model can rely on more information to deal with the problem of insufficient information during up-sampling, so as to improve the segmentation accuracy. However, it will also cause the input of the two convolution layers to have a big difference in features, increasing the difficulty of network learning.

The feature map before the first down-sampling in the U-Net structure is sent to the convolution layer quite far back for direct feature connection. However, in a deep convolutional network, with the deepening of the network layer, the features outputted by the convolutional layer will become more and more abstract. The receptive fields of the first several layers are relatively small, so the extracted features must only be locally related, such as lines and edges. However, the receptive fields of the last several layers are relatively large, and the extracted features are likely to be more abstract features. Therefore, it is inevitable that the extracted features are very different from those of the first several layers. In the U-Net network, the corresponding two convolution layers in the encoder and decoder parts are directly combined through a skip connection. However, the features extracted by these two convolution layers have a large characteristic difference. If the shallow features with great differences are directly connected with the deep features, it will inevitably increase the difficulty of network learning when the shallow features with such great differences are used to supplement the deep features. Although the U-Net network has achieved ideal results in biomedical and industrial defect detection, due to the skip-connection mode in U-Net, it is easy to cause the input of the two convolution layers connected to have a large semantic difference, resulting in increased difficulty in network learning.³³

Li et al.³⁴ proposed the attention UNet++ network, which introduced the attention mechanism between the nested convolution blocks of UNet++, so that the features extracted at different levels could be combined into task-related choices. Furthermore, due to the introduction of deep supervision, the prediction speed of the pruned network is accelerated at the cost of modest performance degradation. Hoorali et al.³⁵ proposed an improved UNet++ architecture to segment microscopic images of patient tissue samples. The proposed model combines multi-scale features by adding skip connections in two paths. Several convolutional networks are used as the backbone to extract powerful representations in the encoder section. To overcome the problem of data imbalance of different classes, a weighted hybrid loss function is proposed. Experimental results show that despite many challenges of microscopic image analysis, the proposed model is a reliable system for the automatic diagnosis of anthrax and other tissue diseases. Li et al.³⁶ proposed a residual-attention UNet++, which is an extension of the UNet++ model with a residual unit and attention mechanism. The residual unit improves the degradation problem and the attention mechanism can increase the weight of the target area and suppress the background area irrelevant to the segmentation task. Three medical image datasets such as skin cancer, cell nuclei, and coronary artery in angiography were used to validate the proposed model. The results showed that the residual-attention UNet++ achieved superior evaluation scores.

From the above analysis, it can be seen that the UNet++ architecture has a good capability in feature extraction and has achieved good application results in medical detection. Many medical image features have high similarity with the features of fabric texture and defect. Therefore, the UNet++ architecture with strong feature extraction ability suitable for medical detection is very suitable for surface defect detection of fabric with complex background and diverse defect types.

Adapted algorithm

UNet++

The UNet++ structure is formed by integrating U-net structures of different depths into one network. Its advantage is that it can capture different levels of features and integrate them using feature superposition. The UNet++ network has receptive fields of different sizes, which can complement each other's shortcomings caused by receptive fields of different sizes, and can well meet the needs of defect detection.

The decoder with the UNet++ structure is densely connected with the same resolution through the redesigned skip connection, changing the connectivity of the encoder and decoder network. In the UNet++ structure, the encoder feature map goes through a dense convolution block, and it is easier for the optimizer to optimize when the received encoder feature map is similar to the corresponding decoder feature map. The jump path in the UNet++ structure is expressed as follows:

x^{i, j} = {\begin{matrix} C (D (x^{i - 1, j})), & j = 0 \\ C ([{[x^{i, k}]}_{k = 0}^{j - 1}, U (x^{i + 1, j - 1})]), & j > 0 \end{matrix}

(1)

where

x^{i, j}

denotes the output of node

X^{i, j}

i

represents the down-sampling layer along the encoder and

j

is the convolution layer of the dense block along the skip pathway.

C (\cdot)

is the convolution layer immediately following the activation function,

D (\cdot)

refers to the down-sampling layer,

U (\cdot)

represents the up-sampling layer, and

[\cdot]

denotes the concatenation layer. The schematic diagram of the UNet++ network is shown in Figure 3. Among them, UNet++ L1 represents one-layer UNet++ network, UNet++ L2 represents two-layer UNet++ network, UNet++ L3 represents three-layer UNet++ network, and UNet++ L4 represents four-layer UNet++ network.

Figure 3.

The schematic diagram of the UNet++ network.

Deep supervision UNet++ architecture

The adapted deep supervision UNet++(DSUNet++) structure is developed by fusing the deep supervision information into the UNet++ structure. The deep supervision makes statistics on the output of different depth models, and then observes the output of different depths for pruning the network. Therefore, according to the texture, color, defect, and other characteristics of different kinds of fabrics, reasonable pruning is carried out for adapting different kinds of fabrics and different types of defect detection. The specific operation is to add a 1 × 1 convolution kernel after the output of each UNet++ sub-network branch to supervise the output of each sub-network branch in the UNet++ network. The corresponding loss function of each sub-network branch in the verification set determines how many branches should be cut, that is the depth of the DSUNet++ network structure is determined. The network design of the deep supervision UNet++ is shown in Figure 4. Among them, L1, L2, L3 and L4, respectively, represent one-layer, two-layer, three-layer, and four-layer depth DSUNet++ network.

Figure 4.

The network design of the deep supervision UNet++.

In the process of deep supervision, the output of each sub-network is the segmentation result of the input image, but the results are different. Therefore, if the output result of the low-level sub-network meets the requirement of defect segmentation, the redundant network can be cut off at will. The deep supervision function can weigh the depth, speed and precision of the network. For example, in Figure 5, if the effect of the three-layer network is good, the four-layer network can be cut directly without any effect on the other layer networks, as shown in the gray part cut out in Figure 5.

Figure 5.

The pruning process from L4 layer to L3 layer.

Figure 6 is a structure diagram of the designed DSUNet++ network. Table 1 lists the detailed operations process of each layer, among which ConvTranspose represents the deconvolution layer, k, and s represents convolution kernel and stride, respectively; the input of the model is the defect sample to be tested and the output is the pixel-level segmentation result.

Figure 6.

The structure diagram of the DSUNet++ network.

Table 1.

Definition and operation of each layer of DSUNet + +

Operation	Input	Output	Output size	k	s
Input image	–	–	(3,256,256)	–	–
(Conv + ReLU) × 2	Input image	C0_0	(64,256,256)	3	1
Pooling	C0_0	P0_0	(64,128,128)	2	2
(Conv + ReLU) × 2	P0_0	C1_0	(128,128,128)	3	1
Pooling	C1_0	P1_0	(128,64,64)	2	2
ConvTranspose	C1_0	U1_0	(64,256,256)	2	2
Skip connection	C0_0 + U1_0	S0_1	(128,256,256)	–	–
(Conv + ReLU) × 2	S0_1	C0_1	(64,256,256)	3	1
(Conv + ReLU) × 2	P1_0	C2_0	(256,64,64)	3	1
Pooling	C2_0	P2_0	(256,32,32)	2	2
ConvTranspose 2	C2_0	U2_0	(128,128,128)	2	2
Skip connection	C1_0 + U2_0	S1_1	(256,128,128)	–	–
(Conv + ReLU) × 2	S1_1	C1_1	(128,128,128)	3	1
ConvTranspose 3	C1_1	U1_1	(64,256,256)	2	2
Skip connection	C0_0 + C0_1 + U1_1	S0_2	(192,256,256)	–	–
(Conv + ReLU) × 2	S0_2	C0_3	(64,256,256)	3	1
(Conv + ReLU) × 2	P2_0	C3_0	(512,32,32)	3	1
Pooling	C3_0	P3_0	(512,16,16)	2	2
ConvTranspose	C3_0	U3_0	(256,64,64)	2	2
Skip connection	C2_0 + U3_0	S2_1	(512,64,64)	–	–
(Conv + ReLU) × 2	S2_1	C2_1	(256,64,64)	3	1
ConvTranspose	C2_1	U2_1	(128,128,128)	2	2
Skip connection	C1_0 + C1_1 + U2_1	S1_2	(384,128,128)	–	–
(Conv + ReLU) × 2	S1_2	C1_2	(128,128,128)	3	1
ConvTranspose	C1_2	U1_2	(64,256,256)	2	2
Skip connection	C0_0 + C0_1 + C0_2 + U1_2	S0_3	(256,256,256)	–	–
(Conv + ReLU) × 2	S0_3	C0_3	(64,256,256)	3	1
(Conv + ReLU) × 2	P3_0	C4_0	(1024,16,16)	3	1
ConvTranspose	C4_0	U4_0	(512,32,32)	2	2
Skip connection	C3_0 + U4_0	S3_1	(1024,32,32)	–	–
(Conv + ReLU) × 2	S3_1	C3_1	(512,32,32)	3	1
ConvTranspose	C3_1	U3_1	(256,64,64)	2	2
Skip connection	C2_0 + C2_1 + U3_1	S2_2	(768,64,64)	–	–
(Conv + ReLU) × 2	S2_2	C2_2	(256,64,64)	3	1
ConvTranspose	C2_2	U2_2	(128,128,128)	2	2
Skip connection	C1_0 + C1_1 + C1_2 + U2_2	S1_3	(512,128,128)	–	–
(Conv + ReLU) × 2	S1_3	C1_3	(128,128,128)	3	1
ConvTranspose	C1_3	U1_3	(64,256,256)	2	2
Skip connection	C0_0 + C0_1 + C0_2 + C0_3 + U1_3	S0_4	(320,256,256)	–	–
(Conv + ReLU) × 2	S0_4	C0_4	(64,256,256)	3	1
Conv + ReLU	C0_4	Output	(1,256,256)	1	1

Loss function

CEloss_MFB

There is great randomness in fabric defect samples. For example, in the existing sample database, the quantity of defect samples of raw fabric and patterned fabric is quite different, which causes the problem of unbalanced classification of fabric defect samples. At the same time, some small pixel defect samples are included in these defect samples, and the defects in such small pixel defect fabric samples are not easy to detect, which directly affects the detection rate of the algorithm.

Cross-entropy loss function is the sum and accumulation of the loss of each pixel. The size of different categories of objects is different; that is, the number of pixels is different. Therefore, different weights should be given to the loss of different types of objects during detection, so that the algorithm can also obtain better detection results for small pixel defects.

In order to solve this problem, the median frequency balance (MFB) method is introduced on the basis of cross-entropy loss function. MFB reweights losses for each class based on the distribution of real labels to reduce the impact of class imbalance on the model. For a multi-category image database, each category will have a category frequency. Divide the number of pixels in the category by the total number of pixels in the database to find the median frequency of all categories, and then divide by the frequency corresponding to the category to get the weight. This can ensure that the category with a small proportion has a weight greater than 1, and the category with a large proportion has a weight less than 1 to achieve a balanced effect. Using MFB can significantly improve network performance. The MFB weighted loss function CEloss_MFB³⁷ is specifically expressed as follows:

CEloss_MFB = - \frac{1}{N} \sum_{n = 1}^{N} \sum_{c = 1}^{C} w_{c} l_{c}^{(n)} \log (p_{c}^{(n)})

(2)

w_{c} = \frac{median ({f_{c} | c \in C})}{f_{c}}

(3)

f_{c} = \frac{sum_c}{sum_image_c}

(4)

where N is the number of pixels in a training sample,

w_{c}

is the weight for class c,

f_{c}

is the frequency of pixels in class c, median

(f_{c})

is the median value of each

f_{c}

, c represents the category to which the pixel belongs, sum_c represents the sum of all the pixels in that category, sum_image_c represents the sum of pixels for all images,

p_{c}^{(n)}

is the probability of pixel n being in class c;

l_{c}^{(n)}

corresponds to the label of sample n for class c, when the label is given in a batch of encoding, and C is the set of all classes.

BCEloss

To prune the network more accurately according to the characteristics of different kinds of fabrics, binary cross-entropy loss function (BCEloss) is added to the output end of each network branch of the DSUNet++ architecture. To balance the depth, speed, and precision of the network better. The detailed description of BCEloss is as follows:

L = \frac{1}{N} \sum_{i} - [y_{i} \log (p_{i}) + (1 - y_{i}) \log (1 - p_{i})]

(5)

where

y_{i}

represents the label of sample

i

, the positive class is 1, and the negative class is 0.

p_{i}

is the probability that sample

i

is predicted as a positive class. N is the total number of samples. L represents the loss of UNet++ networks of different depths.

Detection results and analysis

In this section, a series of experiments are conducted to evaluate the performance of the adapted DSUNet++. All experiments were conducted on a server using an Intel Core i7-8700K processor with 3.2 GHz dominant frequency, 128 GB of memory, and GeForce1080Ti graphics processor (GPU). The software part uses Windows 10 operating system and Pytorch deep learning framework.

Evaluation index

To evaluate the performance of fabric defect detection results, the indexes of Accuracy, Precision, Recall and F-measure³⁸ are used in our experiment. Here, Accuracy represents the proportion of all predictions that are correct, Precision indicates how much of the predicted samples are correctly predicted, Recall indicates how many samples have been correctly predicted from the actual samples, F-measure represents the comprehensive evaluation of Precision and Recall index simultaneously. The calculation formulas are as follows:

Accuracy = \frac{T P + T N}{T P + F P + T N + F N} \times 100 %

(6)

Precision= \frac{T P}{T P + F P} \times 100 %

(7)

Recall= \frac{T P}{T P + F N} \times 100 %

(8)

F - measure = \frac{2 \times Precision \times Recall}{Precision + Recall} \times 100 %

(9)

where TP means the defect detected correctly, FP means error detection of normal texture as defect, TN means the normal texture detected correctly, and FN means missing detection of defect.

Experiments

In order to test the feasibility of the algorithm, the experiment was carried out on raw fabric and patterned fabric. The image size used in this paper is 256 pixels × 256 pixels. The epochs of the algorithm in this paper are 200, and the learning rate is 0.01.

The defect detection of raw fabric

The feasibility of the adapted algorithm is verified by the TILDA fabric sample database. The TILDA fabric sample database is a standard defect sample database, and it is also a common verification database for experts and scholars to verify detection algorithms. The database contains two types of raw fabric samples, C1 and C2. Class C1 fabric image has uniform texture, fine yarn, thin texture, high fabric density, and relatively simple background texture, including two types of fabrics: C1R1 and C1R3. Class C2 fabrics contain visual, grid texture fabric images, thicker yarn, grid structure, small fabric density, relatively complex background texture, which also includes two types of fabrics: C2R2 and C2R3. Both C1 and C2 fabric samples contain four defect types: holes, oil spots, thread errors, and objects on the surface. C1R1, C1R3, C2R2 and C2R3 are composed of 200 defective samples and 50 non-defective samples, respectively. Therefore, the raw fabric sample database used in this paper contains 800 defective samples and 200 non-defective samples, totaling 1000 samples.

During the experiment, according to different defect types, 50% of the images were selected as training samples and the other 50% as testing samples. In the experiment, a three-layer depth network is used for detection, and the defect detection results of raw fabric are shown in Figure 7.

Figure 7.

The defect detection results of raw fabric.

As the yarn of C2 fabric is thicker than C1 fabric, its background texture is more complex than C1 fabric, and the similarity and fusion degree between defects and texture are higher than C1 fabric. Thus, the characteristics of C2 fabric itself will have a certain impact on the accuracy of its defect detection, and the detection of the fabric edge is not complete. The specific evaluation indexes of the experimental results are given in Table 2.

Table 2.

Defect detection evaluation indexes (%) of raw fabric

Fabric types	Defective names	Accuracy	Recall	Precision	F-measure
C1R1	Holes	99.5	94.2	89.7	91.9
	Oil spots	97.7	93.6	90.4	92.0
	Thread errors	98.9	86.2	85.4	85.8
	Objects on the surface	97.6	88.4	78.2	83.0
C1R3	Holes	97.9	91.1	84.9	87.9
	Oil spots	96.8	88.4	83.2	85.7
	Thread errors	97.7	90.3	86.6	88.4
	Objects on the surface	96.2	87.0	83.4	85.2
C2R2	Holes	98.3	93.7	88.6	91.1
	Oil spots	97.5	89.9	91.7	90.8
	Thread errors	96.7	89.3	83.3	86.2
	Objects on the surface	97.7	91.7	86.7	89.1
C2R3	Holes	97.3	90.8	85.7	88.2
	Oil spots	97.7	91.0	87.3	89.1
	Thread errors	97.6	90.6	86.2	88.4
	Objects on the surface	97.7	91.3	87.1	89.2

The defect detection of patterned fabric

To test better the performance of the algorithm, the patterned fabric is taken as the second type of detection object. The samples included three types of fabrics: dot-, star- and box-. Among them, star-patterned and box-patterned fabrics contain five kinds of defect types: broken end, hole, netting multiple, thick bar and thin bar. In addition to the five kinds of defect types above, dot-patterned fabric also includes the defect of knots, a total of six kinds of defect types. Each defect type contains five images, so a total of 80 defect images are included. According to different defect types, 50% of the images were selected as training samples and the remaining 50% as testing samples. A four-layer depth network is used for detection in the experiment. The defect detection results of patterned fabrics are shown in Figure 8.

Figure 8.

The defect detection results of patterned fabric.

The defect detection of patterned fabric also achieved good results. First, the four-layer DSUNet++ network structure extracts more feature information. Second, the defects of patterned fabrics are relatively small, and can be easily distinguished compared with the fabric texture. The specific evaluation indexes of the experimental results are listed in Table 3.

Table 3.

Defect detection evaluation indexes (%) of patterned fabric

Fabric types	Defective names	Accuracy	Recall	Precision	F-measure
Dot-patterned	Broken end	99.4	94.7	94.3	94.5
	Hole	99.4	92.8	97.6	95.1
	Netting multiple	100.0	99.9	100.0	100.0
	Thick bar	91.9	76.1	94.5	84.3
	Thin bar	99.9	98.1	99.6	98.8
	Knots	99.2	91.1	92.5	91.8
Star-patterned	Broken end	99.1	83.3	63.8	72.3
	Hole	99.7	92.4	85.3	88.7
	Netting multiple	99.9	93.3	91.6	92.4
	Thick bar	99.2	87.3	77.1	81.9
	Thin bar	98.9	80.7	59.8	68.7
Box-patterned	Broken end	99.9	98.2	93.4	95.7
	Hole	99.5	79.1	74.9	76.9
	Netting multiple	99.9	99.5	98.6	99.1
	Thick bar	98.6	80.5	89.6	84.8
	Thin bar	99.6	92.3	99.1	95.6

The influence of DSUNet++ network depth on detection results

The UNet++ network integrates the U-net structures of different scales into one network. The deeper the network depth is, the longer the training time will be required, and it is easy to cause overfitting. On the contrary, the shallower the network depth is, the fabric texture information needed for detection cannot be extracted, which reduces the accuracy of detection results. Therefore, the selection of network depth plays a key role in the results of fabric defect detection.

The selection of network depth of raw fabric

For different types of fabrics, the depth selection of the DSUNet++ network depth is different. It depends on the texture of the fabric, the background, the defect and the complexity of the fabric. Because the background texture of raw fabric is relatively simple, the key information of fabric can be extracted without too deep a network depth, and good detection results can be obtained. Figure 9 shows the comparison of detection results of raw fabric with different DSUNet++ network depths. Among them, L2, L3 and L4, respectively, represent two-layer, three-layer, and four-layer depth DSUNet++ networks.

Figure 9.

The defect detection results of raw fabric with different DSUNet++ network depths.

As can be seen from Figure 9, the detection results of the three-layer DSUNet++ network on raw fabric defects are better than those of the two-layer and four-layer. In particular, in terms of detailed information, the edges are clearer, and the detected defect shape is more consistent with the actual defect. Figure 10 draws the line chart of evaluation indexes of four kinds of raw fabrics in the two-layer, three-layer and four-layer DSUNet++ network detection. It can be seen from Figure 10 that the evaluation indexes of three-layer deep network detection are better.

Figure 10.

Comparison of raw fabric defect detection evaluation index (%) with L2, L3 and L4.

The selection of network depth of patterned fabric

For patterned fabric, the fabric background texture is more complex and darker. When extracting features, if the network depth is too shallow, fabric texture features cannot be well extracted. Therefore, the network depth of patterned fabric is deeper than that of raw fabric in defect detection. Figure 11 shows the comparison of detection results of patterned fabric with different DSUNet++ network depths.

Figure 11.

The defect detection results of patterned fabric with different DSUNet++ network depths.

As can be seen from Figure 11, for the same patterned fabric, the detection results of the four-layer DSUNet++ network have obvious advantages over the results of the two-layer and three-layer network structures. In particular, the details of defects are more prominent, and the outline of the defect edge is more accurate. Figure 12 draws the line chart of evaluation indexes of three kinds of patterned fabrics in the two-layer, three-layer and four-layer DSUNet++ network detection. It can be seen that the evaluation indexes of four-layer depth network detection are better than those of two-layer and three-layer DSUNet++ networks.

Figure 12.

Comparison of patterned fabric defect detection evaluation index (%) with L2, L3 and L4.

The influence of CEloss_MFB on detection results

The median frequency balance weighted loss function CEloss_MFB is introduced in this paper to solve the problem of uneven classification of fabric defect samples and low detection rate of small pixel defects. Figure 13 shows the test results before and after the introduction of the loss function.

Figure 13.

The influence of CEloss_MFB on detection results.

It can be seen from the test results that after adding CEloss_MFB loss function, the defect detection of small pixels is more accurate, the defect edge is smoother, and the continuity of the defect is better.

Comparisons with other methods

To show better the superiority of the adapted algorithm, the performance of the adapted algorithm is compared with the detection results of other existing algorithms. In order to make a fair comparison, we only use the defect images employed in other algorithms for comparison. The comparison results of the adapted method and other methods are shown in Figure 14.

Figure 14.

Comparative analysis of the defect detection results.

LGM-FC³⁹ is a traditional defect segmentation method, which first removes the texture of the fabric image, and then uses fuzzy C means to segment the defect. LGM-FC has obtained good performance in the detection of fabrics with simple texture or obvious defects. However, when the defect is not obvious and the patterned fabric defect detection, there will be some loss of defect details.

As can be seen from Figure 14, for the raw fabric, the Frangi filter combined with the fuzzy C means algorithm⁴⁰ and the maximization of expectation (EM) algorithm⁴¹ can accurately detect defects, but the defects profile detected is incomplete, and there may even be some noise. In contrast, the algorithm adapted in this paper is more prominent in defect detail detection.

The detection results of the ER algorithm⁴² and IIER algorithm⁴³ are similar. The IIER algorithm does a good job in the details of general types of defect detection. However, for fabrics with a larger or smaller defect area, the accuracy of detection results will be different to some extent.

The Mobile-Unet⁴⁴ algorithm achieves high segmentation accuracy in raw fabric and patterned fabric defects. However, due to the structure characteristics of the U-net network itself, the input of the two convolution layers connected will have great semantic differences, which will increase the difficulty of network learning and affect the real-time detection.

In brief, the detection results of the DSUNet++ algorithm adapted in this paper are more prominent in the details of defects, and the outline is more distinct. It has a good detection effect for some small pixel defects, and can also completely detect defects when the background texture and the defect are high approximation.

Conclusions

In this paper, we propose an end-to-end deep supervision UNet++(DSUNet++) network architecture for detecting different kinds of fabric defects, which can realize the barrier-free fusion of shallow and deep features. First, the idea of deep supervision is added to the algorithm, which makes the algorithm more flexible for detecting the defects of different kinds of fabrics. It can carry out reasonable pruning based on the background texture of the fabric itself, the obvious degree of defects, and the similarity between the background and the defects. Second, the median frequency weighted loss function CEloss_MFB is introduced to overcome the problem of imbalanced fabric defect sample category and the decrease of detection rate when a small pixel defect is detected. Finally, the selection principle of deep supervised UNet++ network depth is analyzed, which provides algorithm support for better targeted network selection, better detection of fabric defects and the improvement of defect detection efficiency. The datasets of raw fabric and patterned fabric are tested, respectively, with high detection accuracy. In addition to fabric defect detection, the algorithm can also be applied to many aspects of the industry, such as metal surface defect detection, rail surface defect detection, tile surface defect detection, and circuit board surface defect detection, etc. Because the defects of these industrial products have similar characteristics, and the defects appear on the surface of the object, and the state of the defects is similar. Therefore, the same defect detection method can be used for detection. Although this paper has carried out some research and optimization on the fabric defect detection algorithm, machine learning is used to solve some problems. However, there are still many unsolved problems and room for further improvement in the field of fabric defect detection. First of all, as a supervised detection method, its training samples require a large number of manually labeled data. Second, the actual site changes a lot, the emergence of new defect types, the emergence of new fabrics to the off-line training model is not suitable, resulting in a high miss rate. It is difficult to meet the requirements of the textile industry for fabric defect detection. Third, how to design a more efficient network learning method and significantly reduce model complexity and model parameters while keeping the detection accuracy unchanged. Look for cost-effective field edge computing equipment better to realize real-time detection of industrial sites. Therefore, developing weakly supervised or unsupervised defect detection methods, exploring online learning fabric defect detection methods and improving the detection speed of the algorithm on embedded edge devices are our next research direction.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Key Project of Shaanxi Provincial Department of Science and Technology of China (grant no. 2019GY-014), the Key R&D Program of Shaanxi Province, China (grant no. 2022GY-274), and the Science and Technology Project of Shaanxi Province, China (grant no. 2023-YBSF-505).

ORCID iD

Erhu Zhang

References

Tao

Hou

A survey of surface defect detection methods based on deep learning. Acta Automatica Sinica 2021; 47: 1017–1034.

Pang

Shen

Cao

et al. Deep learning for anomaly detection: a review . ACM Computing Surveys (CSUR) 2021; 54: 1–38.

Shen

Zhang

et al. Review of image anomaly detection. Acta Automatica Sinica 2022; 48: 1402–1428.

Xiao

Tian

et al. A review of object detection based on deep learning. Multimedia Tools Appl 2020; 79: 23729–23791.

Rasheed

Zafar

Rasheed

et al. Fabric defect detection using computer vision techniques: a comprehensive review. Mathemat Problems Engng 2020; 16: 1–24.

Zhang

et al. Review of surface defect inspection based on visual perception. Comput Integr Manufact Syst 2023; 29: 169–191.

Liang

Shen

et al. Textile fabric defect detection based on low-rank representation. Multimedia Tools Appl 2019; 78: 99–124.

Kang

Zhang

A universal and adaptive fabric defect detection algorithm based on sparse dictionary learning. IEEE Access 2020; 8: 221808–221830.

Zhang

Yan

et al. Fabric defect detection using salience metric for color dissimilarity and positional aggregation. IEEE Access 2018; 6: 49170–49181.

10.

Bhatt

Malhan

Rajendran

Image-based surface defect detection using deep learning: a review. J Comput Inform Scie Engng 2021; 21: 1–23.

11.

Zhang

Davaasuren

et al. Multi-region saliency-aware learning for cross-domain placenta image segmentation. Pattern Recognition Letters 2020; 140: 165–171.

12.

Huang

Jing

Wang

Fabric defect segmentation method based on deep learning. IEEE Trans Instrument Measurem 2021; 70: 1–15.

13.

Wang

Chen

Qiao

A fast and robust convolutional neural network-based defect detection model in product quality control. Int J Adv Manufact Technol 2018; 94: 3465–3471.

14.

Huang

Wang

et al. Unsupervised fabric defect detection based on a deep convolutional generative adversarial network. Text Res J 2020; 90: 247–270.

15.

Zhang

Lee

Automatic fabric defect detection with a wide-and-compact network. Neurocomputing 2019; 329: 329–338.

16.

Wei

Hao

Tang

et al. A new method using the convolutional neural network with compressive sensing for fabric defect classification based on small sample sizes. Text Res J 2018; 89: 3539–3555.

17.

Zhang

Wang

et al. Attention-gate-based U-shaped reconstruction network (AGUR-Net) for color-patterned fabric defect detection. Text Res J 2023; 93: 3459–3477. doi: 10.1177/00405175221149450.

18.

Dlamini

Kao

et al. Development of a real-time machine vision system for functional textile fabric defect detection using a deep YOLOv4 model. Text Res J 2022; 92: 675–690.

19.

Zhang

Xiong

et al. QA-USTNet: yarn-dyed fabric defect detection via U-shaped Swin transformer network based on quadtree attention. Text Res J 2023; 93: 3492–3508. doi: 10.1177/00405175231158134.

20.

Zheng

Yang

Wang

et al. Objective evaluation of sizing morphology of E-glass filament yarn. J Ind Text 2022; 51: 6563–6589.

21.

Bao

Jing

Xie

A defect detection system of glass tube yarn based on machine vision. J Ind Text 2023; 53: 1–23.

22.

Cui

Jiang

et al. SDDNet: a fast and accurate network for surface defect detection. IEEE Trans Instrument Measurem 2021; 70: 1–13.

23.

Cao

Liu

Peng

et al. DenseUNet: densely connected UNet For electron microscopy image segmentation. IET Image Process 2020; 14: 2682–2689.

24.

Zhou

Huo

et al. NSCR-based DenseNet for lung tumor recognition using chest CT image. Biomed Res Int 2020; 2020: 1–9.

25.

Abdollahi

Pradhan

Alamri

An ensemble architecture of deep convolutional Segnet and Unet networks for building semantic segmentation from high-resolution aerial images . Geocarto Int 2020; 3: 1–22.

26.

Manickam

Rajan

Subramanian

et al. Person identification with aerial imaginary using SegNet based semantic segmentation. Earth Sci Informat 2020; 13: 1293–1304.

27.

Chen

Zhou

et al. Image splicing localization using residual image and residual-based fully convolutional network. J Visual Commun Image Represent 2020; 73: 1–9.

28.

Sun

Geng

et al. Full convolutional neural network with multi-scale residual model for optic cup and disc segmentation. J Med Imag Health Informat 2020; 10: 2733–2738.

29.

Yang

Feng

Wang

Automatic segmentation model combining U-Net and level set method for medical images. Expert Syst Appl 2020; 153: 1–9.

30.

Pasban

Mohamadzadeh

Zeraatkar-Moghaddam

et al. Infant brain segmentation based on a combination of VGG-16 and U-Net deep neural networks. IET Image Process 2020; 14: 4756–4765.

31.

Que

Lee

Residual dense U-Net for abnormal exposure restoration from single images. IET Image Process 2021; 15: 115–126.

32.

Chen

Zhou

et al. Image splicing localization using residual image and residual-based fully convolutional network . J Visual Commun Image Represent 2020, 73: 1–9.

33.

Zhou

Siddiquee

Tajbakhsh

et al. UNet++: a nested U-net architecture for medical image segmentation. In 4th Deep Learning in Medical Image Analysis (DLMIA), Workshop, Spain, 2018, pp. 3–11.

34.

Tan

Chen

et al. Attention Unet++: a nested attention-aware U-net for liver CT image segmentation. In 2020 IEEE International Conference on Image Processing (ICIP), IEEE, 2020.

35.

Hoorali

Khosravi

Moradi

Automatic microscopic diagnosis of diseases using an improved UNet++ architecture. Tissue Cell 2022; 76: 101816. Doi: 10.1016/j.tice.2022.101816

36.

Zhang

Residual-attention UNet++: a nested residual-attention U-net for medical image segmentation. Appl Sci 2022; 12: 7149–7168. Doi: 10.3390/app12147149

37.

Kampffmeyer

Salberg

Jenssen

Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Las Vegas, NV, USA. IEEE, 2016, pp. 680–688.

38.

Luo

Cai

Yang

A review of deep learning methods for industrial defect detection. Scientia Sinica Informationis 2022; 52: 1002–1039.

39.

Zhang

Jing

et al. Fabric defect detection using L0 gradient minimization and fuzzy C-means. Appl Sci 2019; 9: 3506–3521.

40.

Zhang

Jing

et al. Fabric defect detection based on Frangi filter and fuzzy C-means algorithm in combination. J Text Res 2015; 36: 120–124.

41.

Zhang

Jing

et al. Fabric defect detection based on gaussian mixture models of EM algorithm. J Comput Engng Appl 2014; 50: 184–187.

42.

Tsang

Ngan

Pang

Fabric inspection based on the Elo rating method. Pattern Recognit 2016; 51: 378–394.

43.

Kang

Zhang

A universal defect detection approach for various types of fabrics based on the Elo-rating algorithm of the integral image. Text Res J 2019; 89: 4766–4793.

44.

Jing

Wang

Rätsch

et al. Mobile-Unet: an efficient convolutional neural network for fabric defect detection. Text Res J 2020; 92: 30–42.