Recognition of the layout of colored yarns in yarn-dyed fabrics

Abstract

The layout of colored yarns in yarn-dyed fabrics is a significant part of designing and production in the textile industry, which is still analyzed manually at present. Existing methods based on image processing have some limitations in accuracy and stability. Therefore, an automatic method is proposed to recognize the layout of colored yarns and some other basic fabric structure parameters: the fabric density and weave pattern. First, a large dataset with fabric structure parameters is constructed. The fabric images are captured by a wireless portable device. Then the yarns and floats are accurately located using a novel multi-task and multi-scale convolutional neural network. Finally, a density-based color clustering algorithm is proposed to recognize the layout of colored yarns. The results of extensive experiments show that the proposed method can automatically identify the basic structure parameters with high effectiveness and robustness.

Keywords

layout colored yarns yarn-dyed fabrics deep neural network color clustering analysis fabric structure parameters

In the process of design and production, the analysis of fabric structure parameters is an indispensable step in the textile industry. The layout of colored yarns, as an essential structure parameter, is especially vital for the production of yarn-dyed fabrics. However, its analysis still relies on manual operations, which are labor-intensive and time-consuming. Therefore, it is essential to find a competent and objective automatic method for the recognition of the layout of colored yarns.

With the improvement in image processing technologies, scholars have presented many automatic recognition methods and achieved much impressive progress.^1–3 Their methods show high performance for specific kinds of fabric and can be applied in certain environments. However, there are still some problems:

Some existing methods usually depend on a high image resolution, which makes the image acquisition system cumbersome and expensive.

Some methods need to tune parameters according to different fabric types, causing low adaptability.

The diameters of yarns and weave patterns are diverse, which causes some errors in locating and recognizing the colored yarns.

In our previous work,⁴ we creatively utilized a convolutional neural network to accurately locate yarns, showing high accuracy in the location of yarns and fabric density measurement. In this study, a novel hybrid method is proposed to recognize the layout of colored yarns and some other parameters: the fabric density and weave pattern. The flow chart of the proposed method is shown in Figure 1. First, a database with a wide range of fabrics is established by a portable device. Then, a new multi-task and multi-scale convolutional neural network (MTMSnet) is proposed to locate yarns and floats jointly. Using the results, every single float is located, and its color information is extracted. Next, a density-based color clustering algorithm (DBCCA) is introduced to cluster the color of yarns. Finally, the layout of the colored yarns and the color effect are recognized by combining the weave pattern and the color clustering results. During the process, the fabric density and the weave pattern are also obtained.

Figure 1.

Flow chart of the proposed method.

The framework of the paper is organized as follows. In the next section, some related work is reviewed. Then, the image acquisition system and the establishment of the dataset are introduced, and the proposed method, which utilizes the MTMSnet for yarn and float location and the DBCCA for the clustering of colored yarns, is described. In the following sections, the experiments are then described in detail, the results are given, and the effectiveness of the proposed method is discussed. Finally, conclusions are drawn.

Related works

In general, to recognize the layout of colored yarns, existing methods mainly include two procedures: the location of yarns or floats, and the clustering of colored yarns.

For the location of yarns or floats, existing methods can be divided into two types: frequency-domain methods and spatial-domain methods. The frequency-domain methods convert the image to the frequency domain by utilizing the periodical nature of the yarn layout. The typical method is the fast Fourier transform,^5,6 which selects characteristic frequency points to reconstruct feature images. Zheng et al.⁷ used the discrete wavelet transform to extract edge information and locate yarns. However, the power spectrum is often noisy, and the results are often marred by the irregular distribution of the yarns and the color of yarns. Moreover, these methods show poor performance for sophisticated fabrics, especially yarn-dyed fabrics and jacquard fabrics, owing to their complex periodical nature.

The main principle behind the spatial-domain methods is the analysis of brightness variation to determine the location of yarns and the crevices. Zhong et al.,⁸ Huang et al.,⁹ and Huang and Liu¹⁰ used the gray projection to locate yarns, but the method depends heavily on the ability to locate the local peak, which is easily affected by the slanted placement of the fabrics. Thus, Pan et al.¹¹ used the Hough transform to detect the skew angle of the placement, increasing the accuracy of yarn location. Other widely used spatial-domain methods utilize the co-occurrence matrix,¹² the quadratic local extremum,¹³ and morphology analysis.¹⁴ However, owing to the interstices between the yarns, the subsequently identified color of yarns may produce interference.

For the clustering of colored yarns, Luo et al.¹⁵ described how the instrumental color may be influenced by the yarn texture and used a modified k-means clustering approach to analyze the fabric structure parameters. Han et al.¹⁶ proposed a lightness-biased cartoon-and-texture decomposition method for color segmentation of fabric images. Zheng et al.¹⁷ used a multi-region fuzzy segmentation process to realize the color classification and then, based on the segmentation results, detect the yarn location. Although the color classification results of their method are relatively precise, there is always some error in judging the yarn position. Xin et al.¹⁸ proposed an active grid model to identify the color pattern by acquiring duel-side fabric images. Zhang et al.^19–21 conducted a series of studies of the automatic detection of the layout of colored yarns of different kinds of yarn-dyed fabric. They divided the fabric images into some sub-images to locate yarns and used the fuzzy c-means (FCM) clustering method to classify the colored yarns. The drawbacks of their method are that they have to deal with different types of yarn-dyed fabrics individually and that the number of yarn colors need to be specified in advance. Pan et al.²² adopted a genetic algorithm and Xu and Lin²³ used neural networks for automatic yarn color classification. There might be some errors in their results, and Zhou et al.²⁴ introduced an error-correcting method to improve the accuracy of the recognition. Although their methods show certain adaptability and accuracy, some weaknesses are shown in the actual production, owing to the cumbersome image acquisition system.

In the past few years, convolutional neural networks have been widely used to solve pattern recognition problems. With the help of the excellent feature extraction capability of the convolutional neural network, we develop the MTMSnet to locate yarns and floats accurately. Meanwhile, inspired by the success of a density-based algorithm (DBSCAN)²⁵ that does not need the number of clusters to be specified and can discover clusters from the dataset with arbitrary shape, we propose a DBCCA to cluster colored yarns. The algorithm does not consider noises and uses a color difference formula (CIEDE2000)²⁶ to calculate the distance for the aim of color clustering. These improvements increase the accuracy and adaptability of the method for fabric structure analysis.

Image acquisition system and dataset establishment

Image acquisition system

To extend the application range, a new portable wireless device is used to capture fabric images in sRGB mode. The details of the device are shown in Figure 2. The device is equipped with a prime lens, and its distance to the fabric is fixed to ensure a fixed spatial resolution. Below the lens, it has eight small LEDs to provide constant illumination. The captured image is transferred wirelessly to a server with the help of a WIFI module to analyze the fabric structure parameters further. During image acquisition, it is necessary to ensure that the device is close to a clean and flat fabric surface. The spatial resolution is set as 4680 pixel/inch, and the image size is 1280 pixels × 720 pixels.

Figure 2.

Portable wireless fabric image acquisition device.

Compared with other image acquisition systems, the device has the advantages of small size and wireless connection, which allows us to capture images and establish a dataset quickly. Moreover, the application range and portability are significantly increased. It should be noticed that the proposed method does not rely heavily on the image acquisition system, as long as the spatial resolution of the image is within a suitable range.

Dataset establishment

We collected about 400 kinds of fabric with detailed parameter information from a factory. The information contained the fabric density, the weave pattern, the layout of the colored yarns, and some other parameters. Then, by using the image acquisition system, an elaborate dataset was established, which covered a wide range of fabric types, such as solid color fabrics, yarn-dyed fabrics, and jacquard fabrics. There were no identical or quite similar images in the dataset. Figure 3 shows the distribution of the warp and weft densities and fabric types in the dataset. In Figure 3, the axes indicate the distribution of fabric densities, and different symbols signify different basic fabric types. The number of different fabric types are given in the figure key.

Figure 3.

Distribution of warp and weft densities and fabric types in the dataset.

Label generation

To train the network, which will be introduced in the next section, we labeled the locations of yarns and floats in every fabric image and tried different label generating strategies.

For yarn location, we labeled the outline of every yarn in the dataset and used a smooth label strategy to generate yarn location maps. A Gaussian distribution was adopted to express the information about the location of yarns. Its effect has been proved in our previous work.⁴

For the location of floats, we randomly labeled a basic weave repeat and combined the locations of yarns to generate every float location. If the image did not have a basic weave repeat, we labeled all floats. To generate float location maps, we used a hard-label strategy, which adopts 0 or 1 to represent whether a pixel belongs to a float or not. This is because float classification is a binary classification problem, and it shows better performance in experiments.

Because we do not train the network to learn the layout of colored yarns, it is not necessary to label the color of each yarn, but we still marked the color of every yarn in the test set, for evaluation.

A labeled fabric image sample, two generated yarn location maps, and two float location maps are shown in Figure 4. The four location maps were used to train the model. Finally, we established an elaborate fabric dataset with 800 images, which not only covers a wide range of fabric types but also contains the location of yarns and floats, as well as the structure parameters. The establishment of the dataset allows us to better train and evaluate our network.

Figure 4.

Labeled image and its generated location maps for training: (a) labeled image; (b) warp location map; (c) weft location map; (d) warp float location map; (e) weft float location map.

The proposed method

In this section, we introduce the architecture of the multi-task and multi-scale convolutional neural network (MTMSnet). Then we describe the steps involved in processing the location maps to predict the fabric density and weave pattern. Finally, we give the details of the DBCCA used to recognize the layout of the color pattern.

Multi-task and multi-scale convolutional neural network

The architecture of the proposed MTMSnet is illustrated in Figure 5. It contains three components: a shared multi-scale feature encoder, which extracts features of the image, a yarn location map decoder, which decodes the extracted feature maps to predict warp and weft yarn location maps, and a float location map decoder to predict warp and weft float location maps.

Figure 5.

Structure of the proposed MTMSnet.

By combining the warp and weft yarn location maps, we can calculate the fabric density and locate the intersection points. Then, because the relative location between the four predicted location maps has not been changed, the location of intersection points plus the warp and weft float location maps ensures recognition of the weave pattern. Furthermore, the encoder and decoder enable the network to address some general object location problems, such as defect detection, face detection, and crowd counting. Owing to the multi-task structure, the network is also suitable for extracting related features, such as the extraction of defect features and facial landmarks.

Shared multi-scale feature encoder

As shown in Figure 6, the initial shared multi-scale feature encoder is made up of four multi-scale modules to extract features. Owing to the diversity of yarn diameters and float sizes, each multi-scale module has four different sizes of filter, to ensure that the net has a more extensive local receptive field. A 1 × 1 filter is added before each filter to reduce the feature dimensions. Each interior layer has the same feature maps for concatenation. A 2 × 2 maximum pooling layer is applied to eliminate most texture noises and retain robust features. A rectified linear unit is adopted as the activation function after every layer. The existence of shared convolutional features improves the field of focus to extract texture features of yarns and floats and reduces the number of parameters required.

Figure 6.

Structure of the multi-scale module.

Yarn location map decoder

The yarn location map decoder contains four transposed convolutional modules to refine the details of feature maps step by step. Three 2 × 2 transposed convolutional layers are added between the four convolutional layers to ensure that the outputs have the same size as the input image. Rectified linear unit activation is applied in each layer to avoid gradient vanishing. The mean square error (MSE) and structural similarity index (SSIM) are used as loss functions; their effectiveness has been proved in our previous work.⁴ The definitions are

MSE = \frac{1}{N} \sum_{i = 1}^{N} (G_{i} - Y_{i})^{2}

(1)

where N denotes the total pixel number of an image, Y_i denotes the predicted value, and G_i denotes the ground truth, and

SSIM = \frac{(2 μ_{Y} μ_{G} + C_{1}) (2 σ_{YG} + C_{2})}{(μ_{Y}^{2} + μ_{G}^{2} + C_{1}) (σ_{Y}^{2} + σ_{G}^{2} + C_{2})}

(2)

μ_{Y} = \frac{1}{N} \sum_{i = 1}^{N} Y_{i}

(3)

μ_{G} = \frac{1}{N} \sum_{i = 1}^{N} G_{i}

(4)

σ_{Y}^{2} = \frac{1}{N - 1} \sum_{i = 1}^{N} Y_{i} - μ_{Y}

(5)

σ_{G}^{2} = \frac{1}{N - 1} \sum_{i = 1}^{N} G_{i} - μ_{G}

(6)

σ_{YG} = \frac{1}{N - 1} \sum_{i = 1}^{N} (Y_{i} - μ_{Y}) (G_{i} - μ_{G})

(7)

where

μ_{Y}

and

σ_{Y}^{2}

denote the local mean and variance of the predicted value,

μ_{G}

and

σ_{G}^{2}

denote the local mean and variance of the ground truth,

σ_{YG}

denotes the local covariance, and

C_{1}

and

C_{2}

are small constants to avoid division by zero. In addition

\begin{matrix} L = MSE + λ (1 - SSIM) \end{matrix}

(8)

where L is the combined loss function and λ is a weight to balance the MSE and

SSIM

. In this paper, we set λ as 0.001, by experiment.

Float location map decoder

There are minor differences between the floats location map decoder and the yarn location map decoder. Because a hard-label strategy is adopted to generate float location maps as the ground truth, the activation function of the output layer is set as sigmoid and the loss function is set as binary cross entropy (BCE), which is defined as

BCE = - \frac{1}{N} \sum_{i = 1}^{N} [G_{i} \ln Y_{i} + (1 - G_{i}) \ln (1 - Y_{i})] x

(9)

Using sigmoid as the activation function and BCE as the loss function will accelerate convergence and reduce the classification error for the binary classification problem.

Float location and classification

The trained model takes a fabric image of arbitrary size, and outputs two sets of location maps: warp and weft yarn location maps, and warp and weft float location maps. The size of the images in each set is the same. Figure 7(a) to (f) shows a fabric image sample and its predicted location maps. It can be seen that the trained model successfully extracts the yarn and float features and eliminates most of the noises. Next, the following steps are introduced to realize fabric density measurement and weave pattern recognition.

Figure 7.

Predicted location maps and subsequent processes for fabric density measurement and weave pattern recognition: (a) test fabric image sample; (b) predicted warp location map; (c) predicted weft location map; (d) predicted warp float location map; (e) predicted weft float location map; (f) combination of image for weft float location map and original image to show the predicted effect; (g) skeletonized image of warp location map; (h) skeletonized image of weft location map; (i) rotated original image and its region of interest, the red rectangle region; (j) smoothed warp projection curve; (k) smoothed weft projection curve; (l) rotated original image and boundary box of floats; (m) heat map for probabilities of each float as a warp float; (n) schematic diagram of recognized weave pattern; (o) distribution of colored yarns in CIELAB color space, and clustering results; (p) recognized layout of colored yarns.

Skew angle detection

Owing to the placement of the fabric samples, the skew of the warp and weft yarns cannot be ignored. Therefore, the skew angle is detected based on the predicted yarn location maps. First, the Otsu algorithm²⁷ is applied to convert the yarn location maps to binary images, and the Zhang–Suen thinning algorithm²⁸ is used to skeletonize the binary images to save computation time. Next, the Hough transform²⁹ is conducted to detect the skew angles of warp ( $θ_{j}$ ) and weft ( $θ_{w}$ ) threads, respectively. Figure 7(g) and (h) shows the skeletonized image and the detected skew angles. Finally, the predicted warp yarn location map and warp float location map are rotated by $- θ_{j}$ , and the predicted weft yarn location map and weft float location map are rotated by $- θ_{w}$ . To remove the incomplete yarns, the original image and predicted images are cropped according to the skew angle to an inner rectangle as the region of interest, as shown in Figure 7(i). Because the images are rotated around the image center and the sizes of the final rotated images are the same, the relative location among the predicted location maps is not changed.

Fabric density measurement

Image projection is used to locate the warp and weft yarns. The rotated warp location map is projected along the column, and the rotated weft location map is projected along the row to get the projection curves. A 5 × 1 minimum filter is adopted to smooth the curve. The final smoothed projection curves are shown in Figure 7(j) and (k). Because the predicted location maps mainly retain the yarn information, the smoothed projection curves are almost noiseless. By locating the local maxima (P_c), the fabric density (d) can be derived⁴ as

d_{w} = \frac{2.54 \times PPI}{p}

(10)

d_{j} = \frac{2.54 \times PPI}{p}

(11)

where d_w is the weft yarn density (thds/inch), d_j is the warp yarn density (thds/inch), PPI is the spatial resolution (in pixels per inch), and p is the pixel distance between two adjacent yarns (in pixels).

Float location

The two minima (P_l, P_r) on either side of a local maximum (P_c) represent the boundary of the yarn. The boundary box of each intersection point can finally be obtained by combining the P_l and P_r of each warp and weft. Figure 7(l) shows the predicted boundary box of every intersection point in the original image. It can be seen that the boundary box retains the core of floats and that the interstices are not included; this ensures the accuracy of the subsequent float classification and color recognition.

Float classification

Because the relative location between all the predicted location maps has not been changed, once the location of the intersection point has been obtained by combining the yarn location maps, the float type can be classified by summing the values within the boundary box in the corresponding location of the float location maps. The float type is determined as

F_{p} = \frac{S_{j}}{S_{j} + S_{w} + C}

(12)

F = {\begin{matrix} WEFT, F_{p} T \\ WARP, F_{p} \geq T \end{matrix}

(13)

where

F_{p}

is the probability that the float is a warp float, S_j is the sum value within the float boundary box in the warp float location map, S_w is the sum value within the float boundary box in the weft float location map, C is a small constant to avoid division by zero, F is the float type, and T is a threshold, which we naturally set T as 0.5. Figure 7(m) and (n) shows the heat map of probabilities of each float as a warp float and the final schematic diagram of the recognized weave pattern.

Basic weave repeat recognition

Because the number of yarns is relatively small, we calculate the size of the basic weave repeat using the Su index, which is proposed by Pan et al.³⁰ The main principle is to calculate the probabilities of different repeat sizes from two to the number of yarns. Once the size is obtained, the basic weave repeat can be extracted by randomly choosing the same size for the whole predicted weave pattern.

Yarn color features extraction

After these steps, every float is obtained in RGB mode. However, the RGB color space is not a uniform color space and does not matched the characteristics of the human vision system (HVS). Because CIELAB is a relatively uniform color space and is widely used for color difference evaluation,³¹ especially in the textile industry,³² all the floats are converted to the CIELAB mode.³³ The color specification of a single strand of yarn is ambiguous because the pixel values vary with positions, owing to the effect of the three-dimensional shape of yarns. In this paper, the mean values of L, A, and B of the warp floats on each warp yarn, and the mean values of L, A, and B of the weft floats on each weft yarn are calculated as the color features. The layout of colored yarns can be analyzed by clustering the color features of yarns.

Recognition of colored yarns

Current clustering methods include FCM,³⁴ k-means,¹⁵ and neural networks.^35,36 The main drawbacks of these methods are that the number of clusters must be specified. Although some improved color clustering methods^24,30 do not need the number of clusters to be specified, those methods are not aimed at color clustering, and their calculation times are relatively long. Inspired by the success of density-based methods^25,37 in discovering clusters from datasets with arbitrary shape, and without specifying the number of clusters, we improve the DBSCAN and propose a DBCCA.

Since the clustering of colored yarns can be considered noise-free, the minimum number of points (minPts) is set as one. At the same time, a color difference formula (CIEDE2000) is used to calculate the distance of two colors instead of the Euclidean distance. The DBCCA is relatively parameter-free and more suitable for color clustering. The pseudo code of the DBCCA is shown in Figure 8, which has some modifications compared with DBSCAN.

Figure 8.

Pseudo code of DBCCA.

The main principle of the DBCCA is that the collection of all the density-reachable points is set as a cluster. The collection of density-reachable points is defined as follows. If point q is within distance ε from p, the point q is directly reachable from p. If there is a path p₁, p₂, …, p_n where each p_i + 1 is directly reachable from p_i, all the points within the path make up a collection of density-reachable points.

The calculation of the distance can be considered the calculation of color difference ( $Δ E^{*}$ ) for color clustering. The 1976 formula (CIE76)³⁸ is a known set to measure the color difference of CIELAB. It uses the Euclidean distance to calculate the color difference, but this cannot address the problems of the perceptual non-uniformity of human vision. Therefore, the International Commission on Illumination (CIE) redefined the definition (CIEDE2000) to measure the color difference,²⁶ which is commonly used in the textile industry.³² In the DBCCA, CIEDE2000 is used to calculate the distance, and addresses the problem that a certain $Δ E^{*}$ may be inconsistent with human sensitivity; this renders the notion of a just noticeable difference meaningful. In general, $Δ E^{*} \approx 5$ corresponds to a noticeable difference.³⁹ Thus, in the DBCCA, we also set ε, which can be regarded as the radius of the distance domain, as 5.

Figure 7(o) shows the distribution of colored yarns in the CIELAB color space and the clustering results based on the DBCCA. Figure 7(p) shows the recognized layout of colored yarns. We can finally recognize the layout of colored yarns and the color effect by combining the weave pattern and the results of the clustering of colored yarns.

Experiments

The hardware used in this study included a server with Intel Core i9-7900x CPU, GTX 1080Ti GPU, and 32 GB RAM. The algorithm was implemented on the Keras 2.2.4 framework with Tensorflow 1.13.0 as the backend. A series of experiments were conducted to evaluate the effectiveness of our proposed method.

Training details

The dataset was randomly divided into three parts: 600 images were used for training, 100 images for validation, and the remaining 100 images for testing. We used the hand-out method to split the dataset and try to ensure that the distributions of the training and test sets were similar. The distribution of the dataset of the fabric types is given in Table 1. Part of the dataset will be available in the future. The images in the training set were horizontally flipped, vertically flipped, and the channel intensity was shifted, for data augmentation. All the pictures were converted to [0, 1] for normalization. An Adam optimizer⁴⁰ with a small initial learning rate of 10⁻⁵ was used to train the model. Some of the hyperparameters were tested and recommended based on the test in the validation set. After 200 epochs of training, we obtained the final trained model. Based on the trained model, we used the test set to evaluate the performance of the method in predicting the fabric density, and weave pattern, and the layout of colored yarns.

Table 1.

Number of fabric types in the training and test sets

Dataset	Plain	Twill	Satin	Other
Training set	203	66	43	391
Test set	35	16	5	41
Whole set	238	82	48	432

Evaluation details

To evaluate the fabric density measurement, we also estimated the mean-absolute-percentage error (MAPE) and the mean-squared-percentage error (MSPE),⁴ which are defined as

MAPE = \frac{1}{N} \sum_{i = 1}^{N} \frac{| G_{i} - Y_{i} |}{G_{i}} \times 100 %

(14)

MSPE = \sqrt{\frac{1}{N} \sum_{i = 1}^{N} (\frac{G_{i} - Y_{i}}{G_{i}}) 2} \times 100 %

(15)

where N is the number of test images, G_i is the manual measurement value, and Y_i is the automatic measurement value.

For the weave pattern recognition, we followed the convention of existing works, which used the recognition error of weave pattern (WRE), calculated as

WRE = \frac{N - R}{N} \times 100 %

(16)

where N is the number of all test images and R is the number of correctly recognized images. Because of the periodical nature of the weave pattern, a correctly recognized image means that the recognized weave pattern is the same as the ground truth, or the basic weave repeat is a subset of all possible basic weave repeats.

For recognition of the layout of the colored yarns, we calculated the recognition error of color pattern (CRE) to evaluate the model, which is based on the yarn-dyed fabrics in the test set. The definition is

CRE = \frac{M - R}{M} \times 100 %

(17)

where M is the number of yarn-dyed fabrics in the test images and R is the number of correctly recognized images. A correctly recognized image means that the recognized layout of colored yarns is the same as the ground truth.

To further evaluate the effectiveness of the DBCCA for the clustering of colored yarns, the commonly used indexes for evaluating the clustering methods—adjusted Rand index (ARI)⁴¹ and Fowlkes–Mallows index (FMI)⁴²—were adopted.

The ARI is calculated based on the test set, to evaluate the similarity between the clustering results and the real situation. Its values range from −1 to 1. The larger the value, the more consistent the clustering results with the ground truth. The calculation is defined as

RI = a + b C 2 n samples

(18)

ARI = \frac{RI - E [RI]}{max (RI) - E [RI]}

(19)

where a is the number of pairs of elements that are in the same set in the true labels and in the same set in the predicted labels, b is the number of pairs of elements that are in different sets in the true labels and in different sets in the predicted labels,

C 2 samples

is the total number of possible pairs in the dataset, and

E [RI]

is the expected value of RI.

The FMI is defined as the geometric average of the pairwise precision and recall of the clustering results, which is more comprehensive for qualitative analysis. The range of scores is [0, 1]. A high value indicates a good similarity between two clusters. The definition is

FMI = \frac{TP}{\sqrt{(TP + FP) (TP + FN)}}

(20)

where

TP

is the number of true positives, which is the number of pairs of points that belong to the same clusters in both the true labels and the predicted labels;

FP

is the number of false positives, which is the number of pairs of points that belongs to the same clusters in the true labels but not in the predicted labels; and

FN

is the number of false negatives, which is the number of pairs of points that belong to the same clusters in the predicted labels but not in the true labels.

To evaluate the effectiveness of the method, we record the time consumption. A hot-loading method is used to eliminate the time for loading the weights of the model.

Results and discussion

Results

In the experiment, some representative fabric samples in the test set were displayed for qualitative evaluation. Figure 9 shows the original fabric samples, predicted location maps, recognized weave patterns, and recognized color effects. The results illustrate that the method can correctly locate yarns and recognize most of the weave patterns and color patterns. However, it has minor errors in dealing with complex weave patterns, such as jacquard fabrics.

Figure 9.

Some representative fabric samples and the recognized weave patterns and the layouts of colored yarns in the test set: the largest complete weave pattern is marked with a red rectangle, and misjudged floats are marked in red.

From a quantitative perspective, the estimated CRE of the yarn-dyed fabrics in the test set (77 in 100) is 10.39% for the recognition of the layout of colored yarns. For weave pattern recognition, the estimated WRE of the whole test set is 8%. Conversely, the proposed method can accurately measure fabric density. The estimated MAPEs of warp and weft densities in the whole test set are 1.38% and 1.65%, respectively, and the MSPEs are 2.28% and 2.43%, respectively. All these results demonstrate that the proposed method can jointly realize fabric density measurement, weave pattern recognition, and the recognition of the layout of colored yarns with high accuracy and robustness.

In terms of the computation time, the method takes about 6.73 s in all for an image to output the warp density, weft density, full weave pattern, basic weave pattern, and layout of the colored yarns and color effect. However, when the model is hot-loading, the computation time is less than 1.80 s. This is because the hot-loading does not load the weights every time, and the method outputs the schematic values of the patterns instead of the images of the weave pattern, basic weave pattern, and color effect. Moreover, by using a portable device, the fabric structure parameters can be conveniently obtained within 10 s.

The method shows errors when dealing with some more complex fabrics, resulting from faults in locating the floats or yarns. This is because the Hough transform is used to calculate the average skew angle, but the skew angle of each yarn is not the same, while some yarns are curved. Moreover, even the human eye can scarcely tell the float type apart in some images.

It should be noted that the method does not rely heavily on the image acquisition device. If the yarns and floats in the image can be clearly distinguished by human vision, the MTMSnet can achieve favorable results. Figure 10 shows the predicted weft yarn and float location maps based on different resolutions. When the resolution is too large or too small, there are some deficiencies in the predicted location maps. In practice, the recommended range of the fabric density is 50–250 thds/inch, and the fixed spatial resolution of the fabric image is better between 3000 pixel/inch and 12000 pixel/inch. The fabric should not have large curved or overlapped yarns, such as nap fabrics, or two-layer fabrics.

Figure 10.

Predicted yarns and floats location maps with different image resolutions: (a) 3078 PPI, (b) 4680 PPI, (c) 5925 PPI, (d) 12312 PPI.

Evaluation of dataset size

Traditionally, the size of the dataset should be large enough to improve the generalization of the model. Therefore, different sizes of training sets were tested to evaluate the model. The training sets were selected to cover the most common fabric types, and there were not the same images in the training and test sets. Table 2 illustrates the evaluation indexes. Because a number of transposed convolution layers are adopted in the decoder stage, and the fabric images have similar patterns, the model can obtain a relatively favorable performance, even if trained with as few as 20 images. However, the error is still high when trained with a small set. The results illustrate that the larger the train set, the better the performance. So, it is necessary to establish a relatively large dataset to improve the robustness of the model.

Table 2.

Evaluation indexes of MTMSnet trained with different sizes of training set; the MAPE and MSPE are calculated for the mean values of the estimates of warp and weft densities

Dataset size	MAPE (%)	MSPE (%)	WRE (%)	CRE (%)
20	12.34	18.43	28	32.47
100	5.80	9.51	22	25.97
300	3.68	4.70	14	15.58
600	1.52	2.36	8	10.39

Boldface indicates the best values.

Evaluation of ε

The threshold of distance ε = 5 was adopted by an experience value, which is considered as a noticeable difference for human vision. To select a suitable value of ε, a series of experiments with different values of ε were carried out. The range of ε was [0.1, 10], and the step was set as 0.1. The variation curves of ARI and FMI are shown in Figure 11. The results show that when ε = 5.2, the cluster results of the DBCCA have maximal ARI and FMI. In this study, ε was set as 5.2 for the clustering of colored yarns based on our image acquisition system. The illumination was not strict, and the target of our problem was the clustering of colored yarns, so ε is still recommended as 5 for the general color clustering problem.

Figure 11.

Evaluation based on different ε.

Evaluation of different color spaces

The floats in the original images were converted into RGB, HVS, and CIELAB, respectively, to explore suitable color spaces for the clustering of colored yarns. Different values of ε were adopted to match different color spaces. In HVS and RGB color spaces, Euclidean distance with the best ε was set for comparison. As given in Table 3, the clustering results in the CIELAB color space are, overall, better than those in other color spaces, as reflected in the larger values of ARI and FMI. This is mainly because the CIELAB is a relatively uniform color space, and the calculation of color difference conforms better to human vision. The conversion of the color space resulted in minor increases in time consumption, but the influences for the whole method can be ignored.

Table 3.

Clustering results based on different color spaces and distance calculation methods

Color space	Distance calculation	ε	ARI	FMI	Processing time (s)
RGB	Euclidean distance	5	0.6528	0.7658	0.6437
RGB	Euclidean distance	9.5	0.7786	0.8064	0.6049
HVS	Euclidean distance	5	0.7605	0.8438	0.6256
HVS	Euclidean distance	9.6	0.8491	0.8644	0.6300
CIELAB	Euclidean distance	5	0.7549	0.8164	0.6071
	Euclidean distance	9.3	0.8723	0.8820	0.6486
	CIEDE2000	5	0.9408	0.9629	0.6812
	CIEDE2000	5.2	0.9538	0.9706	0.6703

Boldface indicates the best values.

Evaluation of different distance calculation methods

In calculating the color difference, we used the CIEDE2000 instead of the CIE76. The nature of the CIE76 is to calculate the Euclidean distance. Considering that different distance calculation methods have different best values of ε, the best value of ε is calculated using the same method as mentioned previously. The results are also shown in Table 3. The CIEDE2000 gives larger ARI and FMI than the CIE76 in general. This is mainly because the perceptual ability of human vision for different colors is different. The same color distance may give a huge difference from human sensitivity. The results verify that the CIEDE 2000 is more suitable for the method to cluster colored yarns.

Comparisons between different color clustering methods

To demonstrate the effectiveness of the proposed DBCCA for the clustering of colored yarns, some other methods were carried out in Python with the help of Scikit-learn.⁴³ The FCM is one of the widely used methods for the clustering of colored yarns, but it is unacceptable to specify the number of clusters. Therefore, an improved FCM³⁰ method was carried out for comparing with our proposed DBCCA. The improved FCM needs to use iteration to find a better cluster number. Meanwhile, the DBSCAN²⁵ and Mean-Shift⁴⁴ were also used to cluster colored yarns for comparisons. They do not need the number of clusters to be specified. All the parameters were set according to the recommendations in the literature,^25,30,44 and our experiments, to ensure as fair a comparison as possible. Table 4 shows the evaluation details. It can be seen that the FCM and DBCCA both achieve better values of ARI, FMI, and CRE. But the consumption time of the FCM is quite high. Conversely, although we tried to select optimal parameters for the DBSCAN and Mean-Shift, the results show a certain limitation. However, the DBCCA is relatively parameter-free and more suitable for color clustering, and shows excellent performance in the experiment.

Table 4.

Comparisons between different clustering methods

Method	Parameter settings	ARI	FMI	CRE (%)	Process time (s)
FCM³⁰	m = 2, max_iter = 10	0.9433	0.9723	10.39	1.834
DBSCAN	Eps = 10, minPts = 3	0.8196	0.9122	18.18	0.6193
Mean-Shift	bandwidth = auto	0.8945	0.9325	11.69	0.6054
DBCCA	ε = 5.2	0.9538	0.9706	10.39	0.6703

Boldface indicates the best values.

Comparisons between different methods

Comparisons between different methods were made to evaluate the performance of our method. Some methods were not realized, owing to the requirement of the image acquisition system, such as transmitting images⁴⁵ or dual-side captured images.¹⁸ In this study, we realized the method of Pan et al.,³⁰ which used the Hough transform and FCM to recognize the layout of colored yarns. Moreover, we applied the study of Wang et al.⁴⁶ to locate the intersection points and recognize the weave pattern. In addition, we used the MSnet⁴ to locate floats and use the FCM to classify colored yarns. Some parameters were set according to these other studies and our experiments to ensure a fair comparison. In the experiment, some fabrics could not be recognized, and many outliers occurred in the test set; this is because our dataset was complex and the fine textures in the image were not clear enough. As shown in Table 5, the proposed method reaches greater accuracy than other methods. It can also be noted that the accuracy of the FCM in recognizing the layout of the colored yarns can be improved with the help of a convolutional neural network.

Table 5.

Evaluation indexes and processing times of different methods: (1) step angle, min angle, and min distance of the Hough transform are 1°, 10°, 9, respectively; (2) fuzzy weighting exponent of FCM are all set as 2; NAN indicates that the method was not used to recognize weave patterns, as it was based on a weave pattern database

Method	MAPE (%)	MSPE (%)	WRE (%)	CRE (%)	Process time (s)
Hough transform + FCM³⁰	17.98	28.33	NAN	66.23	5.44
Hough transform + PCA + FCM⁴⁶ + FCM	17.98	28.33	73	75.32	5.80
MSnet⁴ + FCM	1.56	2.60	38	49.35	6.40
MTMSnet + DBCCA	1.52	2.36	8	10.39	6.73

Boldface indicates the best values.

Conclusion

In this article, a novel hybrid method is proposed to automatically recognize a series of fabric structure parameters: the fabric density, weave pattern, and layout of colored yarns. An elaborate fabric dataset is established using a portable device, which ensures the development of the proposed MTMSnet for the locating of yarns and floats. The colored yarns are then clustered with the help of the DBCCA, which is aimed at color clustering. Experiments demonstrate that the method has the following advantages:

Low errors. The errors for fabric density measurement, weave pattern recognition, and recognition of the layout of colored yarns are relatively low, compared with other methods.

Good robustness. The evaluation results of the method in the test set, which covers different kinds of fabric, verify the adaptability of the method.

High efficiency. The portable wireless device makes it fast and convenient to capture images. The application range is significantly increased.

Although the proposed method reaches high performance, it still has some limitations:

The establishment of the dataset is time-consuming.

The method cannot deal well with fabrics that contain large curved or overlapped yarns.

In the future, the detailed directions of our research are as follows:

The dataset will be further expanded.

The enhancement of the structure of the MTMSnet or new end-to-end methods will be studied.

An online web service will be established for the automatic detection of fabric structure parameters.

Footnotes

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China (Grant No. 61976105).

ORCID iDs

Shuo Meng

Wentao He

References

. Identifying fabric structures with fast Fourier transform techniques. Text Res J 1996; 66: 496–506.

Khan

Wang

Han

, et al. Fabric weave pattern and yarn color recognition and classification using a deep ELM network. Algorithms 2017; 10: 117.

Xin

Zhang

, et al. Color texture classification of yarn-dyed woven fabric based on dual-side scanning and co-occurrence matrix. Text Res J 2017; 87: 1883–1895.

Meng

Pan

Gao

, et al. Woven fabric density measurement by using multi-scale convolutional neural networks. IEEE Access 2019; 7: 75810–75821.

Wood

. Applying Fourier and associated transforms to pattern characterization in textiles. Text Res J 1990; 60: 212–220.

Jie

Ruru

Weidong

. Automatic inspection of density in yarn-dyed fabrics by utilizing fabric light transmittance and Fourier analysis. Appl Opt 2015; 54: 966–972.

Zheng

Han

. A new method for classification of woven structure for yarn-dyed fabric. Text Res J 2014; 84: 78–95.

Zhong

Shi

, et al. Research on computer-aided analysis and reverse reconstruction for the weave pattern of fabric. Text Res J 2013; 83: 298–310.

Huang

C-C

Liu

S-C

W-H

. Woven fabric analysis by image processing: Part I: Identification of weave patterns. Text Res J 2000; 70: 481–485.

10.

Huang

C-C

Liu

S-C

. Woven fabric analysis by image processing. Part ii: Computing the twist angle. Text Res J 2001; 71: 362–366.

11.

Pan

Gao

Liu

, et al. Automatic inspection of woven fabric density of solid colour fabric density by the Hough transform. Fibres Text East Eur 2010; 18: 81.

12.

Benco

Hudec

Kamencay

, et al. An advanced approach to extraction of colour texture features based on GLCM. Int J Adv Rob Syst 2014; 11: 104.

13.

Wang

. Recognition of fabric density with quadratic local extremum. Int J Clothing Sci Technol 2012; 24: 328–338.

14.

Ajallouian

Tavanai

Palhang

, et al. A novel method for the identification of weave repeat through image processing. J Text Inst 2009; 100: 195–206.

15.

Luo

Shen

Shao

, et al. An effective method to segment weft and warp yarns in a multi-colour region of yarn dyed fabric images. Color Technol 2015; 131: 165–171.

16.

Han

Baciu

, et al. Lightness biased cartoon-and-texture decomposition for textile image segmentation. Neurocomputing 2015; 168: 575–587.

17.

Zheng

Wang

Zhou

, et al. Color pattern recognition for yarn-dyed fabrics. Color Res Appl 2019; 44: 88–97.

18.

Xin

Baciu

, et al. Investigation on the classification of weave pattern based on an active grid model. Text Res J 2009; 79: 1123–1134.

19.

Zhang

Pan

Gao

, et al. Automatic detection of layout of color yarns of yarn-dyed fabric. Part 1: Single-system-mélange color fabrics. Color Res Appl 2015; 40: 626–636.

20.

Zhang

Pan

Gao

, et al. Automatic detection of layout of color yarns of yarn-dyed fabric. Part 2: Region segmentation of double-system-mélange color fabric. Color Res Appl 2016; 41: 626–635.

21.

Zhang

Pan

Gao

, et al. Automatic detection of layout of color yarns of yarn-dyed fabric. Part 3: Double-system-mélange color fabrics. Color Res Appl 2017; 42: 250–260.

22.

Pan

Gao

Liu

, et al. Genetic algorithm-based detection of the layout of color yarns. J Text Inst 2011; 102: 172–179.

23.

Lin

. Automatic color identification in printed fabric images by a fuzzy-neural network. AATCC Rev 2002; 2: 42–45.

24.

Zhou

Sun

. A novel feedback error-correcting algorithm for automatic recognition of the color and weave pattern of yarn-dyed fabrics. Text Res J 2013; 83: 1673–1689.

25.

Ester

Kriegel

H-P

Sander

, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. KDD 1996; 96(34): 226–231.

26.

Sharma

Dalal

. The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Res Appl 2005; 30: 21–30.

27.

Ohtsu

. A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 1979; 9: 62–66.

28.

Zhang

Suen

. A fast parallel algorithm for thinning digital patterns. Comm ACM 1984; 27: 236–239.

29.

Duda

Hart

. Use of the Hough transform to detect lines and curves in pictures. Comm ACM 1972; 15: 11–15.

30.

Pan

Gao

Liu

, et al. Automatic detection of the layout of color yarns for yarn-dyed fabric via a FCM algorithm. Text Res J 2010; 80: 1222–1231.

31.

Luo MR. CIELAB. In: Luo R (ed.) Encyclopedia of color science and technology. Berlin: Springer, 2014, pp. 1–7.

32.

Luo

Cui

Rigg

. The development of the CIE 2000 colour-difference formula: CIEDE2000. Color Res Appl 2001; 26: 340–350.

33.

Ford

Roberts

. Colour space conversions, London: Westminster University, 1998.

34.

Zhang

Pan

Gao

, et al. Automatic recognition of the color effect of yarn-dyed fabric by the smallest repeat unit recognition algorithm. Text Res J 2015; 85: 432–446.

35.

Su T-L, Kung F-C, and Kuo Y-L. Application of back-propagation neural network fuzzy clustering in textile texture automatic recognition system. In: 2008 international conference on wavelet analysis and pattern recognition, Hong Kong, China, 30–31 August 2008, pp. 46–49. Piscataway, NJ: IEEE.

36.

Kuo

C-FJ

T-L

Huang

Y-J

. Computerized color separation system for printed fabrics by using backward-propagation neural network. Fibers Polym 2007; 8: 529–536.

37.

Rodriguez

Laio

. Clustering by fast search and find of density peaks. Science 2014; 344: 1492–1496.

38.

Robertson

. The CIE 1976 color-difference formulae. Color Res Appl 1977; 2: 7–11.

39.

Sharma

Bala

. Digital color imaging handbook, Boca Raton, FL: CRC Press, 2002.

40.

Kingma DP and Ba. J. Adam: A method for stochastic optimization. arXiv 2014; arXiv:14091556.

41.

Gates

Ahn

Y-Y

. The impact of random models on clustering similarity. J Mach Learn Res 2017; 18: 3049–3076.

42.

Fowlkes

Mallows

. A method for comparing two hierarchical clusterings. J Am Stat Ass 1983; 78: 553–569.

43.

Pedregosa

Varoquaux

Gramfort

, et al. Scikit-learn: Machine learning in Python. J Mach Learn Res 2011; 12: 2825–2830.

44.

Cheng

. Mean shift, mode seeking, and clustering. IEEE Trans Pattern Anal Mach Intell 1995; 17: 790–799.

45.

Meng

Wang

, et al. Intelligent recognition of the patterns of yarn-dyed fabric based on LSRT images. J Eng Fibers Fabr. Epub ahead of print 22 April 2019. DOI: 10.1177/1558925019840659.

46.

Wang X, Georganas ND, and Petriu EM. Automatic woven fabric structure identification by using principal component analysis and fuzzy clustering. In: 2010 IEEE instrumentation & measurement technology conference proceedings, Austin, TX, USA, 3–6 May 2010, pp. 590–595. Piscataway, NJ: IEEE.