Wool fabric image retrieval based on soft similarity and listwise learning

Abstract

As a special case in content-based image retrieval, fabric retrieval has high potential application value in many fields. However, fabric retrieval has higher requirements for results, which makes it difficult for common retrieval methods to be directly applied to fabric retrieval. It is also a challenging issue with several obstacles: variety and complexity of fabric appearance, and high requirements for retrieval accuracy. To address this issue, this paper presents a novel method for fabric image retrieval based on soft similarity and pairwise learning. First, a soft similarity between two fabric images is defined to describe their relationship. Then, a convolutional neural network with compact structure and cross-domain connections is designed to learn the fabric image representation. Finally, listwise learning is introduced to train the convolutional neural network model and hash function. The generated hash codes are used to index the fabric image. The experiments are conducted on a wool fabric dataset. The experimental results show that the newly proposed method has a greater improvement than our previous work.

Keywords

Content-based image retrieval fabric retrieval deep hashing listwise learning similarity learning

Recently, with the advent of the fourth consumption era (fast consumption era), the ‘small-batch’ and ‘multi-variety’ have gradually become the production modes of the textile industry. Under this production mode, the textile companies have accumulated a large amount of historical production data, which makes it more difficult to find similar fabrics. The traditional search method is carried out in the form of manual comparison, which is consuming and labor intensive. Keyword-based image retrieval^1,2 methods have been widely used in the textile industry. However, such methods rely too much on manually labeled keywords, which leads to a strong subjectivity of the results. Content-based image retrieval (CBIR)^3

–7 is undoubtedly an effective method to address this issue.

Fabric image retrieval, a special case in CBIR, has high potential application value in many fields, such as e-commerce, inventory management, and textile product design. Fabric retrieval plays a very important role in product search work. For example, in textile order production, the processing will analyze the process parameters of the samples provided by the customer, and then find the visually identical or similar products from the historically produced products. If the historical production data are huge, it will be very difficult to complete this work manually. An accurate image retrieval system can accomplish this task well.

The image retrieval system receives an input query and is expected to output a list of images that are related or similar to the query. Technically speaking, there are two core components of CBIR: image representation and similarity measurement, respectively. Image representation vectorizes the images (including the queries and images in the database), the second component consists of ranking the images in the database based on the relevancy to the query image. The most challenging task in CBIR is to associate pixel-based low-level features with the high-level semantic feature from human perception,⁸ which is called the ‘semantic gap’. CBIR builds an index on the visual content of images and searches for similar images by defined similarity. The choice of visual features directly affects the performance of the retrieval algorithm, which is the key technology of CBIR.

Image retrieval methods have gone through significant development in the past decade, starting with descriptors based on hand-crafted feature,⁹ first organized in bag-of-words,¹⁰ and further expanded by spatial verification,¹¹ Hamming embedding,¹² and query expansion.¹³ The traditional CBIR methods often use global feature descriptors, such as color histogram,¹⁴ color moment,¹⁵ and local binary pattern^16,17 to represent the visual content of the image. With the proposal of scale-invariant feature transform (SIFT),¹⁸ bag-of-words technology¹⁹ was grafted into CBIR, so many local features were proposed for CBIR. The SIFT algorithm is highly robust to geometric deformations such as deformation, rotation, and scaling. Around SIFT, there has been a lot of work that has great influence in the field of CBIR and even in computer vision. Although having achieved certain success, these hand-crafted-based methods depend heavily on feature engineering, which leads to their limitation.

In 2012, AlexNet²⁰ won the ImageNet image recognition competition far exceeding second place. Deep learning technology has attracted great attention in the field of machine vision. Alex²⁰ tried to use the 4096-dimensional output of the fully connected layer in AlexNet as the image index, and directly retrieved it on the ImageNet dataset and achieved very good results, verifying that convolutional neural network (CNN) has a good representation of the image content. However, the extracted high-dimensional representation also causes huge computational overhead, causing the problem of ‘dimensionality disaster’. Many subsequent studies centered on how to reduce the dimensionality of the features extracted by deep CNNs.

There are two main problems in the current research on fabric image retrieval: (a) many methods are just for a specific type of fabric, making them less adaptable; (b) the existing methods only represent the fabric from one or two dimensions, which is difficult to describe fully the visual characteristics of the fabric.

As mentioned before, fabric image retrieval is a special case of common image retrieval. However, retrieval methods for general images are difficult to apply directly to fabric image retrieval, because general image retrieval tends to pay more attention to the local information in the image, but fabric retrieval pays more attention to small elements such as texture primitive shape, primitive size, color, and area composition. For example, the focus of the image shown in Figure 1(a) is often the zebra in the image, but the focus of the fabric image shown in Figure 1(b) is the global color and texture features. Moreover, the content of general images is often easier to describe, in Figure 1(a), a zebra is eating, while the content of fabric images is more difficult to describe. Therefore, it is more difficult to represent fabric images with fine textures.

Figure 1.

General image and fabric image. (a) An image of a zebra and (b) an image of a striped fabric.

Most of the features in the woven fabric image are some global low-level features such as color and texture, as well as the middle and high-level features produced by their combination. Studies^21
–23 have shown that the last layer of the deep CNN contains the highest-order features that can be extracted from this model. The output of this layer is the deep features learned after several convolutional operations. The visual content is highly abstract and contains rich semantic information, such as the location, size, and category of the target. These features are the abstraction of the output of the previous layer, so the output of the previous layer has a lower degree of abstraction. Therefore, the features extracted by deeper convolutional layers are more abstract. Figure 2 shows the representation effect of VGG convolutional layers with different depths on fabric images. Generally, the output of the first and second layers often contains rich color and texture features. In this paper, we designed a deep CNN with a compact structure and cross-domain connections to bridge the ‘semantic gap’. To train the proposed network, a soft similarity between fabric images and the listwise loss is defined. Moreover, the hashing layer is introduced to solve the problem of ‘dimensionality disaster’. This paper takes worsted woven fabrics as the main research object. However, the proposed framework can be extended to other types of fabrics, provided that the CNN model needs to be retrained on datasets of other types of fabrics.

Figure 2.

Feature activation map output by the first four convolutional units of VGG-16.

Network architecture

As shown in Figure 3, the proposed end-to-end framework for fabric image representation consists of five convolutional blocks, two short-circuit connections, two fully connected layers, and a hash layer. The network is based on VGG-16,²⁴ but is more like the combination of VGG-16 and ResNet.²⁵ As mentioned before, most of the features in the woven fabric image are some global low-level features such as color and texture, as well as the middle and high-level features produced by their combination. So the short-circuit connection was added to preserve the low-level feature in the deep layer.

Figure 3.

The architecture of the proposed convolutional neural network (CNN) model.

There are two types of convolutional blocks used in the proposed model, which are shown in Figure 4(a) and (b). The convolution kernels all use a small size of 3 × 3. Under the condition of ensuring the perceptual field, the number of nonlinear layers is increased, and the network depth can be increased to optimize the learning model. Meanwhile, it also reduces the trainable parameters of the model, which reduces the risk of overfitting to a certain extent. To prevent the problem of gradient disappearance during training, a batch normalization²⁶ layer is added before Relu activation in each convolution block to normalize the output of the previous layer. Among the convolutional layers, the batch normalization layer treats each feature map output by the previous layer as a neuron. To prevent the loss of information in the normalization, two trainable parameters γ and β are added to make the normalization have different changes. The normalization process is shown in the following equation (1):

y = \frac{γ}{\sqrt{v ar [x] + ε}} x + \frac{γ E [x]}{\sqrt{v ar [x] + ε}} β

(1)

where y is the normalized result, var[x] and E[x] denote the variance and mean of the output of the previous layer, respectively, and

ε

is a small positive number to prevent the denominator from being 0.

Figure 4.

Two convolutional blocks which are used in the network architecture. (a) Conv block 1 and (b) conv block 2.

Each convolution block contains a 2 × 2 max-pooling layer, and the size of the feature map will be reduced to half of the previous one after the maximum pooling operation. In Figure 2, the thick solid arrow indicates the forward propagation process, and the dashed arrow indicates the short-circuit connection, also known as the cross-domain connection, and the purpose is to transfer the shallow and low-level features to the deep layer. To make the feature maps of the two layers have the same shape, the feature maps of the shallow layer undergo a 2 × 2 max-pooling operation when they are transferred to the deep layer. $\oplus$ represents the feature stacking between two layers of feature maps, not the addition of elements. After the convolution operation of five convolution blocks, a woven fabric image I with an input size of 224 ×224 × 3 (pixels) is converted into 7 × 7 × 512 feature maps. These feature maps are highly abstract on the image features of the fabric. The feature maps are flattened into a very long one-dimensional vector and then transformed into a fixed-length feature vector R after two layers of fully connected linearity. To complete efficient retrieval, the bi-half layer is used to encode continuous features, and its specific derivation is shown in Appendix 1.²⁷

Soft similarity learning of fabric images

The definition of soft similarity

For the two fabric images, I _i and I _j , with multidimensional labels, the similarity is commonly defined as $s_{i j} = 1$ if they completely share the same labels, and $s_{i j} = 0$ if they have at least one label different. The similarity is also called hard similarity. This definition does not consider multilabel information, nor can it sort images with multiple class labels for similarity. According to some related studies,^28,29 in our design, the pairwise similarity is defined as follows: (a) calculate the similarity matrix $M_{i j} = {\{m_{d}\}}_{d = 1}^{D}$ of the multilabels, Y _i and Y _j , corresponding to the pair of images, I _i and I_j, where D is the dimension of the label, and $m_{d} = 0$ if the dth dimension of Y _i and Y _j are the same, otherwise $m_{d} = 1$ ; (b) then the similarity of the two images can be computed by equation (2):

S_{i j} = \frac{D - \underset{d = 1}{\sum^{D}} m_{d}}{D}

(2)

Assuming that $Y_{i} = \{4, 1, 2, 5\}$ and $Y_{j} = \{3, 1, 2, 7\}$ are the annotations of two images, the similarity matrix $M_{i j} = \{1, 0, 0, 1\}$ can be obtained through the first step, and then their similarity is calculated as $S_{i j} = 0.5$ according to equation (2).

As shown in Figure 5, the similarity matrix M has 16 different situations when D = 4. According to equation (2), the similarity of pairwise images can be passed into five levels. For approximate neighbor nearest (ANN) search, the generated hash codes should preserve the similarity of the pairwise fabric images. To be specific, for a pair of generated binary codes b_i and b_j, if $S_{i j} = 0$ , which means the pairwise images do not share any label, the Hamming distance should be large; if $S_{i j} = 1$ , which means the pairwise images completely share the same labels, the Hamming distance is expected to be small; otherwise, the binary codes b_i and b_j should have a suitable Hamming distance complying with the soft definition of similarity S _ij .

Figure 5.

The similarity matrix when D = 4.

Listwise learning

In this study, the fabrics in the dataset are all labeled from four dimensions. The similarity between two fabric images may have five cases, 0, 0.25, 0.5, 0.75, and 1, respectively. During training, the model receives a set of five fabric images, $X = \{x_{1}, x_{2}, x_{3}, x_{4}, x_{5}\}$ . $x_{1}$ is regarded as an anchor, then the similarity matrix $S = \{s_{11}, s_{12}, s_{13}, s_{14}, s_{15}\}$ should meet the condition:

s_{11} < s_{12} < s_{13} < s_{14} < s_{15}

(3)

where s_ij is calculated by equation (2). The goal of listwise learning^30,31 is to preserve this relationship in the generated hash codes. Let

\{r_{1}, r_{2}, r_{3}, r_{4}, r_{5}\}

and

\{b_{1}, b_{2}, b_{3}, b_{4}, b_{5}\}

be the image representations and hash codes output by the model. Euclidean distance and Hamming distance are used to measure the distance between the image representations and the hash codes, respectively.

\{d_{11}^{r}, d_{12}^{r}, d_{13}^{r}, d_{14}^{r}, d_{15}^{r}\}

represents the distance matrix of image representations, and

\{d_{11}^{b}, d_{12}^{b}, d_{13}^{b}, d_{14}^{b}, d_{15}^{b}\}

represents the distance matrix of hash codes. The two distance matrices are also expected to meet the following conditions:

d_{11}^{r} < d_{12}^{r} < d_{13}^{r} < d_{14}^{r} < d_{15}^{r}

(4)

d_{11}^{b} < d_{12}^{b} < d_{13}^{b} < d_{14}^{b} < d_{15}^{b}

(5)

Let $P = \{P (1), P (2), P (3), P (4), P (5)\}$ be the true permutation of the similarity matrix of the five input images, and $ϕ (x)$ is an increasing and strictly positive function. Then, the probability of permutation³⁰ $P$ is defined as:

P_{P} = \prod_{j = 1}^{5} \frac{ϕ (d_{P (j)})}{\underset{k = j}{\sum^{5}} ϕ (d_{P (k)})}

(6)

where

d_{P (j)}

denotes the distance of the image at position j of permutation

P

. In this study,

ϕ (x) = \exp (x)

. During training, the true permutation of the input images is known, then according to the idea of maximum likelihood estimation, by maximizing the log likelihood corresponding to the true permutation (or equivalently minimizing the negative log likelihood), the parameters of the model can then be optimized, and the loss function can be written as:

L_{P} = - \log (P_{P}) = - \log (P_{P} = \prod_{j = 1}^{5} \frac{ϕ (d_{P (j)})}{\underset{k = j}{\sum^{5}} ϕ (d_{P (k)})})

(7)

To ensure the quality of learning, the outputs r of the last fully connected layer of the model and the final hash codes b are restricted. Moreover, to reduce the loss of information in the hashing process, the quantitative loss is introduced into the objective function. The output of the last fully connected layer of the model is mapped by tanh activation function in the closed interval [–1, +1]. Quantitative loss is used to encourage numbers less than 0 to be closer to –1, and numbers greater than 0 to be closer to +1, which can be written as:

L_{q} = \sum_{i = 1}^{k} {(|r_{i}| - 1)}^{2}

(8)

where i is the index of this element in the k-bit output r. In summary, the total loss function of the model can be expressed as:

L = λ_{1} L_{P}^{r} + λ_{2} L_{P}^{b} + λ_{3} L_{q}

(9)

where

L

is the total loss,

L_{P}^{r}

is the listwise loss of feature r,

L_{P}^{b}

is the listwise loss of hash code b,

L_{q}

is the quantitative loss.

λ_{1}, λ_{2}, λ_{3}

are three weighting parameters to balance the effects of different loss functions. In this study,

λ_{1} = 1, λ_{2} = 1, λ_{3} = 0.001

. Each sub-item in the objective function is differentiable, so this optimization problem can be regarded as a convex optimization problem. Like other deep learning models, the stochastic gradient descent algorithm and the backpropagation algorithm are used to optimize the parameters in the proposed model.

The framework of the fabric image retrieval system

The parameters in the proposed CNN model are optimized under the supervision of the defined soft similarity. The trained model will be applied to build the fabric image retrieval system. Generally, the CBIR system contains two parts: an offline module and an online module, as shown in Figure 6. In the offline module, the images in the retrieval database are converted into fixed-length vectors. In this work, the trained model will generate a k-bit hash code for each fabric image in the retrieval database, and the generated hash code will be regarded as the index of the corresponding fabric image. The black line with arrows in Figure 6 indicates the process of the offline module.

Figure 6.

The framework of the fabric image retrieval system.

The offline module covers all the logic of the system from input to output. The retrieval system receives a query image, then inputs it into the trained CNN model to generate the corresponding hash code. Finally, the system calculates the Hamming distances between the generated hash code with all indexes in the index library and outputs the relevant fabric images. The Hamming distance is computed by:

Sim (b^{q}, b^{i}) = 1 - \frac{\underset{k}{\sum^{j}} b_{j}^{q} \oplus b_{j}^{i}}{k}

(10)

where

Sim (b^{q}, b^{i})

denotes the similarity between the two k-bit hash codes and

\oplus

is the XOR operator. Note here that the larger the similarity value, the more similar the two hash codes are. The method proposed in this paper can be grafted into the management system of the enterprise to improve the efficiency of the enterprise to find similar fabrics.

Experimental configuration

Dataset

To study fabric image retrieval, in our previous works,^32

–37 a fabric dataset named WFID³³ has been established, which contains 82,073 fabric images. The images in this dataset are labeled from four views: (a) coarse texture, which is classified according to the presence or absence or type of pattern on the surface of the fabric image, is divided into four categories: monochrome fabric, stripe fabric, lattice fabric, and patterned fabric; (b) fine texture is simply divided into three categories according to the weave structure of the fabric; (c) fabric style refers to the subjective feel of a fabric (monochrome fabric: dark and bright; others: casual and business); (d) the pattern forming method is to distinguish according to its literal meaning. In this paper, the proposed CNN model and compared learning-based methods are all trained on the training set with 33,645 fabric images, and the performance of all methods is evaluated on the validation set (consisting of 1029 sets of data, each of which is an image and the 20 most relevant images in the dataset). The images were captured in a red, green, and blue model using a scanner (Canon 9000F Mark II). The light source of the scanner was a white light-emitting diode, which can guarantee a stable capture environment, and the resolution was set to 200 dpi.

Implementation details

In this paper, the proposed retrieval framework and CNN are implemented by using the Pytorch toolkit. The hardware environment is as follows: CPU = E5 2623V4@2.60 GHz, RAM = DDR4 32G, GPU =GeForce RTX 3090 (24G) × 2. It is stated here that all compared deep learning-based methods are implemented using the Pytorch toolkit and based on the bone of VGG-16, and the other methods, which are based on the hand-crafted descriptor, are implemented by using MATLAB 2018b.

During the training, the hyper-parameter configuration is as follows: batch_size = 32, weight_decay =5 × 10^–5, optimizer = ADAM, and initial learning_rate =1 × 10^–3. Specifically, the learning rate of the parameters of the convolutional layers inherited from VGG-16 is set to one-tenth of the subsequent layers, and the strategy for adjusting the learning rate can be denoted by:

{l r}_{n} = \{\begin{array}{l} {l r}_{n - 1} if n \leq a; \\ {l r}_{n - 1} \times (1 - \frac{n - a}{N - a}) if n > a \end{array}

(11)

where

{l r}_{n}

is the learning_rate of the nth epoch and N is the total number of epochs configured. a Denotes the starting epoch where the learning rate begins to decay. p Is a parameter used to control the intensity of each decay within (0,1). In this work, a = 20, N = 200, p = 0.7, respectively. In the comparative experiments, mean average precision (mAP), normalized discounted cumulative gain (NDCG), and precision-recall (PR) curves were used as evaluation criteria, and their detailed calculation methods are shown in Appendix 2.

Results and discussion

Parameter analysis

There are three parameters $λ_{1}, λ_{2}, λ_{3}$ to balance different losses. $L_{P}^{r}$ is the listwise loss of feature r, and is used to align the similarity of features with the similarity of annotations. $L_{P}^{b}$ is the listwise loss of hash code b, and is used to align the similarity of hash codes with the similarity of annotations. $L_{q}$ is the quantitative loss and is used to reduce the loss of information in the hashing process.

Table 1 shows the results of the proposed method by using different parameters configuration. In the proposed model, the quantitative loss is used as an auxiliary component, so it only needs to be equipped with a smaller weight. When its weight is too large, it will affect the performance of fabric image feature learning. Therefore, this paper set the weight of quantitative loss as 0.001. $L_{P}^{r}$ and $L_{P}^{b}$ are the main objective functions in the proposed model, whose main function is to control the quality of feature learning and hash code generation. As the generated hash code is directly used to retrieve images, removing it will have a great impact on the retrieval performance of the proposed model. The experimental results also demonstrate that the interaction of these three losses improves the retrieval performance of the model.

Table 1.

Results of mean average precision (mAP) and NDCG₅₀ for different parameter configurations

$λ_{1}$	$λ_{2}$	$λ_{3}$	mAP				NDCG₅₀
$λ_{1}$	$λ_{2}$	$λ_{3}$	32-bit	64-bit	128-bit	256-bit	32-bit	64-bit	128-bit	256-bit
1	1	0.0001	0.851	0.863	0.915	0.927	0.795	0.803	0.843	0.851
1	1	0.001	0.879	0.896	0.952	0.962	0.812	0.825	0.884	0.890
1	1	0.01	0.862	0.884	0.926	0.933	0.807	0.813	0.869	0.873
1	1	0.1	0.821	0.824	0.843	0.859	0.784	0.796	0.826	0.830
1	1	1	0.781	0.791	0.816	0.820	0.769	0.774	0.806	0.811
1	1	0	0.773	0.796	0.811	0.818	0.753	0.769	0.781	0.783
1	0	0.001	0.714	0.768	0.789	0.793	0.704	0.713	0.763	0.764
0	1	0.001	0.836	0.852	0.901	0.913	0.798	0.803	0.809	0.813

Notes: Bold values indicate the best results in the experiment.

Comparison with different settings

To prove the rationality of the proposed configuration, this study compared the performance of different network configuration settings on the WFID dataset. Specifically, the compared network configurations include: (a) entire VGG-16, abbreviated as EV; (b) shallow VGG-16 without short-circuit connection, abbreviated as SV; (c) entire ResNet-50, abbreviated as ER; (d) shallow ResNet-50, abbreviated as SR; (e) entire AlexNet, abbreviated as EA; (f) shallow AlexNet with short-circuit connection, abbreviated as SAS; (g) proposed setting, shallow VGG-16 with short-circuit connection. It is stated here that only the convolutional layer is different in the above configurations, and the fully connected layer and the hash layer are the same.

This study first compared the PR curves, mAP, and NDCG₅₀ of the several models mentioned above on the validation set, and the results obtained are shown in Figure 7 and Table 2. The larger the area enclosed by the PR curve and the coordinate axis, the better the performance of the retrieval method. As shown in Figure 6, the performance gap of different models is not large, which illustrates the rationality of the proposed network framework to a certain extent. Comparing EV, ER, and EA, it can be seen that the representation ability of ResNet is better than that of AlexNet and VGG. However, the performance of SV is stronger than that of SR and SA, which shows that the representation ability of VGG is more robust. In addition, after the short-circuit connection is equipped in the SV model (proposed), the performance of the model has been significantly improved, indicating that this structure is beneficial to the transmission of shallow features to deep layers without obstacles. It is for these reasons that this study chose shadow VGG-16 with short-circuit connection as the stem of the proposed framework.

Figure 7.

Precision-recall curves of models with different configurations at four different code lengths: (a) 32 bits; (b) 64 bits; (c) 128 bits and (d) 256 bits.

Table 2.

The mAP and NDCG₅₀ results of models with different configurations at four different code lengths.

Model	mAP				NDCG₅₀
Model	32-bit	64-bit	128-bit	256-bit	32-bit	64-bit	128-bit	256-bit
EV	0.822	0.823	0.884	0.896	0.755	0.752	0.816	0.824
SV	0.847	0.849	0.913	0.918	0.780	0.778	0.845	0.846
ER	0.851	0.851	0.920	0.930	0.784	0.780	0.851	0.858
SR	0.801	0.808	0.859	0.866	0.735	0.737	0.790	0.794
EA	0.822	0.829	0.878	0.894	0.755	0.758	0.810	0.822
SAS	0.849	0.854	0.920	0.935	0.782	0.783	0.852	0.863
Proposed	0.879	0.896	0.952	0.962	0.812	0.825	0.884	0.890

Ablation study for listwise learning

The proposed method of fabric image retrieval mainly includes two technical components: short-circuit connection and listwise learning. The previous section has verified that the short-circuit connection component improves the model retrieval performance. Among image representation methods based on similarity learning, pairwise learning using triple loss is the most commonly employed. The idea of triplet loss can be described as:

‖f (x^{a}) - f (x^{p})‖ < ‖f (x^{a}) - f (x^{n})‖

(12)

where

x^{a}

is regarded as the anchor point,

x^{p}

is the relevant sample, which is completely similar to the label of the anchor point, and

x^{n}

is an irrelevant sample, which is not completely similar to the label of the anchor point. Pairwise learning only uses the relevant information of anchor points and very few samples for feature learning. The proposed method adopts listwise learning to extract more relevant information from the training set.

This section conducts ablation research to verify the superiority of list learning in the proposed method. In this experiment, pairwise and listwise learning methods are used to train the proposed CNN network model. To make a fair comparison, the other configurations of the model are the same in the two pieces of training. The results of the comparative experiment are shown in Figure 8. The two learning methods have little difference in mAP indicators, and both achieved good performance, which shows that the proposed CNN model has a good representation effect on fabric images. However, the NDCG₅₀ indicators of the two learning methods are quite different, and the performance of listwise learning is significantly better than pairwise learning. NDCG₅₀ reflects the sorting performance of the search algorithm for the results. The experimental results show that listwise learning can supervise model learns with more similarity of information, and thus achieve better retrieval performance.

Figure 8.

Comparative experiment results of listwise learning and pairwise learning.

Comparison with several retrieval methods

To demonstrate the rationality of the proposed method. In this section, we compare the retrieval performance of the proposed model with several retrieval methods, including three retrieval methods (1–3) based on similarity learning and four methods (4–7) dedicated to fabric image retrieval. Specifically, a brief introduction and implementation of the comparison methods are as follows:

Improved deep hashing network (IDHN)²⁸ is the first deep hashing method that directly uses pairwise quantified similarity, which can reflect the fine-grained similarity between a pair of multilabel images for supervised learning. The authors implemented this model by using Tensorflow. According to the code provided by the authors, this model was reimplemented using PyTorch, where VGG-16 was configured as the basic network.

Central similarity quantization (CSQ)³⁸ was proposed to optimize the central similarity between data points. The authors developed the center similarity with CNNs to learn a hash function. This study directly uses the authors’ public code for comparison experiments, where the feature learning network is VGG-16.

Deep pairwise-supervised hashing (DPSH)³⁹ is a model that can perform simultaneous feature learning and hash code learning for applications with the pairwise label. The authors provided us with the code implemented by PyTorch. Similarly, this paper also uses VGG as the stem of its model.

Fabric retrieval using hierarchical search (FRHS)³² employs a hierarchical search strategy that includes coarse-level retrieval and fine-level retrieval (our previous work). Its image representation is based on a deep sparse network driven by a classification task.

Fabric retrieval based on multitask learning (FRMT)³³ proposed a multitask learning framework to represent the fabric image (our previous work). The learning of this model is guided by four classification tasks.

Wool fabric retrieval based on CNN and ANN (FRCA).³⁶ This method is proposed for fabric images based on Fourier transform and local binary pattern (texture feature). This study implements this method by using Matlab.

Learning deep similarity models with focus ranking (FRDS).⁴⁰ This novel embedding method is termed focus ranking that can be easily unified into a CNN for jointly learning image representations and metrics in the context of fine-grained fabric image retrieval. This study implements this method by using PyTorch.

The above methods are all trained on the training set of WFID. In the performance evaluation of the algorithm, each fabric image in the verification set is used as query input to the retrieval system, and then a string of retrieval results is output. This study calculates the mAP and NDCG₅₀ index of each retrieval based on the output results, and the final result is the average of all retrieval mAP and NDCG₅₀.

The quantitative results are presented in Table 3. Both IDHN, CSQ, and DPSH use pairs of similar or dissimilar images to drive learning image representation and coding. These methods simply define the relationship between the two images as similar or dissimilar, while ignoring part of the same annotations (hard similarity), so that the NDCG₅₀ scores of these methods are not high. The length of the code has little effect on FRHS because it uses a two-step search strategy, and its poor retrieval performance is because it only uses one dimension of label information. When comparing FRMT (previous work) and the method proposed in this study, it can be found that the difference in their mAP indicators is not obvious, but the NDCG₅₀ indicator has been greatly improved. Although FRMT learns all the label information, it does not consider the relationship between labels of different dimensions. The newly proposed method defines soft similarity and uses the soft similarity between a list of images to supervise the model to learn fabric representation and coding. This approach makes the retrieval results appear such a trend: the more similar the fabric to the query, the higher the position in the results, and the farther back the retrieval results that are completely different from the query. This is also the reason why the newly proposed method can achieve a better NDCG₅₀ score.

Table 3.

Comparison of experiment results with several retrieval methods.

Methods	mAP				NDCG₅₀
Methods	32-bit	64-bit	128-bit	256-bit	32-bit	64-bit	128-bit	256-bit
IDHN	0.817	0.826	0.874	0.891	0.732	0.741	0.756	0.763
CSQ	0.832	0.847	0.892	0.913	0.753	0.768	0.794	0.806
DPSH	0.849	0.862	0.904	0.927	0.761	0.772	0.799	0.813
FRHS	0.827	0.831	0.837	0.839	0.747	0.755	0.763	0.766
FRMT	0.875	0.892	0.947	0.958	0.785	0.797	0.815	0.824
FRCA	0.847	0.879	0.901	0.924	0.769	0.779	0.802	0.811
FRDS	0.831	0.852	0.889	0.907	0.772	0.785	0.793	0.803
Proposed	0.879	0.896	0.952	0.962	0.812	0.825	0.884	0.890

This study also presents the retrieval performance of the compared methods on different classifications of fabric images in Figure 9. The results show that the retrieval performance of all methods for pure color, stripe, and lattice fabrics is better than that of patterned fabrics because the features in patterned fabric images are more complicated. It can also be found that our method has significantly better performance than other methods, especially the NDCG evaluation index. The experimental results once again demonstrate the effectiveness and superiority of the proposed method for fabric image retrieval. Some retrieval samples are shown in Figure 10. It is particularly pointed out that the retrieval time of a single image of the method in this paper is only 0.24 seconds.

Figure 9.

Experimental results on different classification fabrics. D1, D2, D3, and D4, respectively, represent monochrome, stripe, lattice, and pattern fabric.

Figure 10.

Some retrieval examples.

Conclusions

In this paper, a novel method for fabric image retrieval based on listwise learning was presented. To narrow the gap between the fabric images and annotations, a CNN with a compact structure and cross-domain connections is designed. Then, the soft similarity is defined to describe and quantify the relationship between paired fabric images. The listwise learning is introduced to train the proposed model. The objective function consists of three parts: listwise loss of features $L_{P}^{r}$ , listwise loss of codes $L_{P}^{b}$ and quantitative loss $L_{q}$ . The experiments are conducted on the self-built wool fabric dataset. The experimental results demonstrate that: (a) when the weight ratio between the three-loss functions is 1:1:0.001, the model can achieve better performance; (b) compared with AlexNet and ResNet, the shallow VGG network is more robust to represent fabric; (c) compared with previous work FRMT, the mAP of the proposed method in this study only increases for some bits, but the NDCG₅₀ score has greatly improved – for example, when the code length is equal to 128, NDCG₅₀ is improved by 0.069; (d) the performance of the proposed method is better than other methods suitable for general image retrieval, which proves that they are difficult to be directly applied to fabric retrieval. The proposed method can help textile companies find similar fabrics quickly and precisely to improve production efficiency. When multilabel datasets are available, the retrieval framework proposed in this paper can be extended to other types of fabrics.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This work was supported in part by the National Natural Science Foundation of China under grant 61976105, in part by the National Key R&D Program of China under grant 2017YFB0309200, and in part by the Fundamental Research Funds for the Central Universities under grant JUSRP52007A.

ORCID iD

Jun Xiang

References

Unar

Wang

Zhang

, et al. Detected text-based image retrieval approach for textual images. IET Image Processing 2019; 13: 515–521.

Barrios

Diaz-Espinoza

Bustos

Text-based and content-based image retrieval on Flickr. In: 2009 Second International Workshop on Similarity Search and Applications. August 29-30, Prague, Czech Republic, 2009, pp. 156–157.

Gudivada

Raghavan

VV.

Content based image retrieval systems. Computer 1995; 28: 18–22.

Wan

Wang

Hoi

SCH

, et al. Deep learning for content-based image retrieval: a comprehensive study. In: Proceedings of the 22nd ACM International Conference on Multimedia. November 3-7, Orlando, Florida, USA, 2014, pp. 157–166.

Zhou

Tian

Recent advance in content-based image retrieval: a literature survey. arXiv preprint arXiv:1706.06064 2017; 01: 1–22.

Tzelepi

Tefas

Deep convolutional learning for content based image retrieval. Neurocomputing 2018; 275: 2467–2478.

Ahmed

Ummesafi

Iqbal

Content based image retrieval using image features information fusion. Informat Fusion 2019; 51: 76–99.

Radenović

Iscen

Tolias

, et al. Revisiting oxford and paris: large-scale image retrieval benchmarking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. June 18-22, Salt Lake City, Utah, USA, 2018, pp. 5706–5715.

Lei

Fuzong

A CBIR method based on color-spatial feature. In: Proceedings of IEEE. IEEE Region 10 Conference. TENCON 99. ‘Multimedia Technology for Asia-Pacific Information Infrastructure’ (cat. no. 99CH37030). September 15-17, Cheju Island, South Korea, 1999, pp. 166–169.

10.

Zhang

Jin

Zhou

Z-H.

Understanding bag-of-words model: a statistical framework. Int J Machine Learning Cybernet 2010; 1: 43–52.

11.

Marsigli

Montani

Paccangnella

A spatial verification method applied to the evaluation of high‐resolution ensemble forecasts. Meteorolog Appl: a journal of forecasting, practical applications, training techniques and modelling 2008; 15: 125–143.

12.

Jegou

Douze

Schmid

Hamming embedding and weak geometric consistency for large scale image search. In: European Conference on Computer Vision. October 12-18, Marseille, France, 2008, pp. 304–317.

13.

Efthimiadis

EN.

Query expansion. Ann Rev Informat Sci Technol (ARIST) 1996; 31: 121–187.

14.

Jain

Vailaya

Image retrieval using color and shape. Pattern Recognit 1996; 29: 1233–1244.

15.

Huang

Z-C

Chan

, et al. Content-based image retrieval using color moment and Gabor texture feature. In: 2010 International Conference on Machine Learning and Cybernetics. July 11-14, Qingdao, China, 2010, pp. 719–724.

16.

Yuan

Qin

, et al. A SIFT-LBP image retrieval model based on bag of features. In: IEEE International Conference on Image Processing. September 11-14, Brussels, Belgium, 2011, pp. 1061–1064.

17.

Liu

Guo

J-M

Chamnongthai

, et al. Fusion of color histogram and LBP-based features for texture image retrieval and classification. Informat Sci 2017; 390: 95–111.

18.

Lowe

DG.

Distinctive image features from scale-invariant keypoints. Int J Computer Vision 2004; 60: 91–110.

19.

Sivic

Zisserman

Video Google: a text retrieval approach to object matching in videos. In: Computer Vision, IEEE International Conference. October 13-16, Nice, France, 2003, pp. 1470–1470.

20.

Krizhevsky

Sutskever

Hinton

GE.

Imagenet classification with deep convolutional neural networks.

Adv Neural Informat Processing Systems 2012; 25: 1097–1105.

21.

Wang

Turko

Shaikh

, et al. CNN explainer: learning convolutional neural networks with interactive visualization. IEEE Transact Visualiz Computer Graphics 2020; 27: 1396–1406.

22.

Qin

Liu

, et al. How convolutional neural network see the world – a survey of convolutional neural network visualization methods. arXiv preprint arXiv:1804.11191 2018; 01: 1–14.

23.

Hänggi

Moser

Pfaffhauser

, et al. Simulation and visualization of CNN dynamics. Int. J Bifurcat Chaos 1999; 9: 1237–1261.

24.

Simonyan

Zisserman

Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014; 01: 1–14.

25.

Zhang

Ren

, et al. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. June 27-30, Las Vegas, NV, USA, 2016, pp. 770–778.

26.

Ioffe

Szegedy

Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning. July 6 -11, Lille, France, 2015, pp. 448–456.

27.

van Gemert

Deep unsupervised image hashing by maximizing bit entropy.

arXiv preprint arXiv:2012.12334 2020; 01: 1–17.

28.

Zhang

Zou

Lin

, et al. Improved deep hashing with soft pairwise similarity for multi-label image retrieval. IEEE Transact Multimedia 2019; 22: 540–553.

29.

Sidorov

Gelbukh

Gómez-Adorno

, et al. Soft similarity and soft cosine measure: similarity of features in vector space model. Computación y Sistemas 2014; 18: 491–504.

30.

Cao

Qin

Liu

T-Y

, et al. Learning to rank: from pairwise approach to listwise approach. In: Proceedings of the 24th International Conference on Machine Learning. June 20-24, Corvallis, Oregon, 2007, pp. 129–136.

31.

Shi

Larson

Hanjalic

List-wise learning to rank with matrix factorization for collaborative filtering. In: Proceedings of the Fourth ACM Conference on Recommender Systems, September 26-30, Barcelona, Spain, 2010, pp. 269–272.

32.

Xiang

Zhang

Pan

, et al. Fabric image retrieval system using hierarchical search based on deep convolutional neural network. IEEE Access 2019; 7: 35405–35417.

33.

Xiang

Zhang

Pan

, et al. Fabric retrieval based on multi-task learning. IEEE Transact Image Processing 2020; 30: 1570–1582.

34.

Zhang

Xiang

Wang

, et al. Image retrieval of wool fabric. Part I: Based on low-level texture features. Text Res J 2019; 89: 4195–4207.

35.

Zhang

Xiang

Wang

, et al. Image retrieval of wool fabric. Part II: Based on low-level color features. Text Res J 2020; 90: 797–808.

36.

Zhang

Xiang

Wang

, et al. Image retrieval of wool fabric. Part III: Based on aggregated convolutional descriptors and approximate nearest neighbors search. Text Res J 2021; 92(3-4): 434–445.

37.

Xiang

Zhang

Pan

, et al. Patterned fabric image retrieval using relevant feedback via geometric similarity. Text Res J 2021; 92(3-4): 409–422.

38.

Yuan

Wang

Zhang

, et al. Central similarity quantization for efficient image and video retrieva. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 13-19, Seattle, WA, USA, 2020, pp. 3083–3092.

39.

W-J

Wang

Kang

W-C.

Feature learning based deep supervised hashing with pairwise labels. arXiv preprint arXiv:1511.03855 2015; 01: 1–19.

40.

Deng

Wang

, et al. Learning deep similarity models with focus ranking for fabric image retrieval. ImageVision Computing 2018; 70: 11–20.