Patterned fabric image retrieval using relevant feedback via geometric similarity

Abstract

Due to the potential value in many areas, such as e-commerce and inventory management, fabric image retrieval, which is a special case of content-based image retrieval, has recently become a research hotspot. As a major category of textile fabrics, patterned fabrics have a diverse and complex appearance, making the retrieval task more challenging. To address this situation, this paper proposes a novel approach for patterned fabric based on the non-subsampled contourlet transform (NSCT) feature descriptor and relevance feedback technique. To integrate the color information into the NSCT feature descriptor, we extract the feature of patterned fabric images in HSV color space. An outlier rejection-based parametric relevance feedback algorithm is employed to adjust the similarity matrix to improve the retrieval results. The experimental results not only show the effectiveness of the proposed approach but also demonstrate that it can significantly improve the performance of the retrieval system compared to other state-of-the-art algorithms.

Keywords

Content-based image retrieval relevance feedback patterned fabric geometrical features geometric similarity

With the improvement of people’s living standards, consumer demands for goods are no longer limited to their practical performance but tend to be attractive and diversified. Therefore, “small-batch and multi-variety” has increasingly become a new production mode for the textile industry. Under this mode of production, companies have accumulated a large amount of historical production data. In textile imitation design, it is usually necessary to manually analyze the sample process parameters and find the same or similar textiles from the warehouse or historical production records. Image retrieval plays a very important role in this work. However, in practice, most companies still use manual search methods or KBIR (keyword-based image retrieval) systems to search for similar fabrics. The first type of method requires companies to store and simply classify fabric samples, and search them by means of manual comparison. Storing the fabric samples not only takes up storage space but also incurs the cost of managing the fabric samples. Besides, the search method of manual comparison is highly subjective. In contrast, the KBIR system used in the textile industry builds indexes for images through manually annotated keywords, and users search for relevant fabric images through the annotated keywords. Although this system has the characteristics of high retrieval speed and easy construction, the search method is too simple and can provide limited functions. Moreover, manual labeling of fabric images is also highly subjective, leading to instability of retrieval results, and further leading to the inefficiency and inaccuracy of KBIR.

The content-based image retrieval (CBIR) system use image content to index images, which can avoid the influence of human subjectivity on the results. In addition, the CBIR system can continuously optimize and improve the retrieval results according to the user's needs. Many researchers in this field have also shifted their research^1–9 focus to CBIR. However, the current CBIR system for natural images generally pays more attention to macro features, such as the outline and color of the image, while the CBIR system applied in patterned fabric retrieval needs to pay more attention to image details and textures, such as the shape of the texture primitives, the size of the primitives, the color, and the composition of regions. Therefore, CBIR research in the textile industry is a more challenging task.

Generally, a CBIR^1,10–12 system receives the query from user input and then retrieves images in a database by analyzing the characteristics and representations of the visual content. The whole process is conducted by presenting a visual query to the CBIR system and by selecting a set of images from the dataset that have the highest resemblance to the query image. This query-by-example process compares the visual content of images in terms of extracted features by computing the distance between the features of the query image and the images in the target database.

Technically speaking, there are three key issues in CBIR: image representation, image similarity measurement, and relevance feedback (RF) mechanism. The first phase allows one to obtain visual content and compact representations for the query and images in the database, which possibly summarize their most distinctive features. The second phase consists of sorting the images in the database based on their relevancy to the query image. The last relevant feedback phase involves user intervention to tag the images in the result set as relevant or irrelevant. After received feedback information, the system will re-rank the images in the database by updating the parameters. Multiple feedback rounds can follow until user satisfaction is achieved.

Various fabric image presentation methods have been applied in the literature to narrow the gap between pixel-level low-level features and high-level semantic features, which can be effectively adopted to represent the underlying image’s characteristics.^13–17 It has been demonstrated that image representation techniques working in the frequency domain are more effective in representing significant and subtle details of the image than the conventional spatial domain schemes. Among various frequency-based methods, wavelet transform (WT),¹⁸ Fourier transform (FT),¹⁹ and their variants have been extensively applied in CBIR.^20–22 Low-level image representations based on WT or FT provide a unique description of the image, and they are highly suitable for representing the texture of the image. However, the main disadvantage of these frequency-based features is the inherent lack of support for directionality and anisotropy. To overcome the limitations, a new technique named Multi-scale Geometric Analysis (MGA)²³ has been introduced and several MGA tools have been developed, such as Beamlet,²⁴ Wedgelet,²⁵ Bandelet,²⁶ Contourlet,²⁷ Curvelet,²⁸ and Ridgelet,²⁹ with applications to different task domains. It has been proved that CBIR systems based on MGA tools are more effective than traditional frequency-based CBIR schemes. In Da Cunha et al.,²⁷ the improvement of contourlet transform (CNT) is proposed to alleviate the shift sensitivity and aliasing problems of CNT in the space and frequency domains. This solution combines both the non-subsampled directional filter bank (NSDFB) and non-subsampled pyramid (NSP), so it is also known as non-subsampled contourlet transform (NSCT). Based on NSCT, many applications in CBIR have been developed. Zhang and Gao³⁰ proposed an approach based on NSCT and fractal dimension for medical image retrieval. Xu et al.³¹ presented an efficient method for human recognition and retrieval using NSCT and the support vector machine (SVM). In Ng-Duc et al.,³² the authors proposed an image retrieval method by combining interest point detection and NSCT. Patterned fabric, such as that for wallpaper and ceramics, is generated by a repetitive unit-motif, through a set of predefined symmetry rules. The pattern appears as lines or contours on the fabric image, so we propose to apply NSCT to represent the content of fabric images.

However, low-level features and distance metrics are not sufficient to narrow the gap between pixel-level features and semantic features; thus, it is necessary to re-rank the retrieval results based on user feedback. Since the mid-1990s, a relevant feedback mechanism has been adopted to retrieve images by using the human visual impression as a feedback signal used to iteratively correct errors generated by the CBIR system. Commonly used RF techniques can be broadly categorized into two: geometric similarity-based methods and probabilistic similarity-based methods.

In geometric similarity-based methods, it is assumed that the parameter distance metric between the two image feature vectors X and Y is $d (X, Y) = (X - Y) A {(X - Y)}^{T}$ , where A is the cross-correlation metric. When the features are independent, A is diagonal. In a typical CBIR scenario, given a vector that is extracted from the query, a distance is computed between the query and an image in the target database. Several images with the least distances are retrieved, and the user’s opinion is sought. The cross-correlation metric entries are improved based on some feature properties of the relevant images. The Multimedia Analysis and Retrieval System (MARS)³³ is the earliest CBIR system employing relevant feedback based on the intuitive argument; however, its importance in the user’s mind must be small. MindReader³⁴ formalizes the relevant feedback method in MARS, and obtains the weight update rule for off-diagonal A by solving the minimization problem. Rui et al.³³ proposed a feedback method based on the concept of a feature class and showed its performance over MindReader for image retrieval. Most of these methods based on geometric similarity only deal with unimodal queries, while ignoring the information in non-relevant image sets. However, the information hidden in non-relevant image sets is essential to prevent the algorithm from falling into the local minima.

On the other hand, probabilistic-based methods utilize the concept of likelihood-based similarity measures. Given the user's preferred initial image set, the probability density functions of related and unrelated images can be obtained. Next, these distributions are used to compute the likelihood that two image feature vectors in the database are similar or dissimilar; one is the query feature vector q and the other is the image feature vector x. Finally, the most likely relevant images are shown in successive iterations. Generally, such methods provide greater robustness than other methods, but at the cost of flexibility.

In this paper, we proposed a new CBIR system for fabric retrieval based on relevant feedback. Our system exploits feature representation for the patterned fabric images given in terms of first-order statistics computed from NSCT.

Related works

Fabric image representation is the most important component in a fabric retrieval system. The image representation methods used in general image retrieval systems tend to pay more attention to three-dimensional (3D) shape features and local features. However, there are few 3D shape features and local features in fabric images, and features that people pay attention to are often global. So, the commonly used image retrieval algorithms often fail to achieve ideal performance on fabric datasets.

There are three types of low-level features on the fabric images: color, texture, and two-dimensional (2D) shape. Most studies described fabrics from these three aspects and generally combined these features to retrieve images. Jing et al.⁵ proposed a method to combine a weighted color histogram with image segmentation for patterned fabric retrieval. In their follow-up study, an algorithm based on the Gist feature and color moment (CM) was presented for printed fabric.⁶ Both methods combine the color and shape feature to represent fabric images. Zhang et al.⁷ proposed a multi-scale and rotation invariant local binary pattern (MRI-LBP) feature, which is a texture descriptor for lace fabric image retrieval. Li et al.³⁵ presented a content-based lace fabric image retrieval system using Haralick features (for texture) and shape features. Suciati et al.² developed a fabric retrieval system using a combination of fractal-based texture features and HSV features. In Li et al.,³⁶ an image retrieval framework combining CMs and coding features (extracted by the perceptual hashing algorithm) was proposed. Recently, Zhang et al.^8,9 studied the retrieval of wool fabric using color and texture features respectively and proposed two methods: a method based on Fourier transformation and the Local Binary Pattern (Part I) and a method based on dominant colors (DCs) and CMs (Part II). The above retrieval algorithms all use two feature descriptors to represent the fabric images. Although having achieved certain success in fabric retrieval, the weights between different features are still difficult to determine.

Significant breakthrough has been achieved on image analysis by moving from hand-crafted feature-based algorithms to deep learning-based frameworks. Therefore, many researchers in the textile industry have tried to apply deep learning techniques to achieve the task of fabric retrieval. Deng et al.¹ presented a novel embedding method, which can be easily integrated into a CNN (Convolutional Neural Network) to jointly learn image representation and metrics in the context of fine-grained fabric image retrieval. Cai et al.³⁷ applied a triplet CNN to learn image representation under the criterion of a similarity metric. Wang et al.³⁸ used a pre-trained CNN model with center loss to describe yarn-dyed fabric patterns for fabric retrieval. Xiang et al.³ proposed a deep learning-based method that employed a hierarchical search strategy. In their follow-up study,⁴ they presented a multi-task learning-based fabric retrieval framework. These two algorithms both learn fabric image representation driven by annotated images (supervised learning). These methods mainly use two types of data to drive the learning of image representation through two types of data, namely (1) fabric images with structured annotations; (2) pair-wise similar or dissimilar fabric images. However, when labeled data is not available, these methods will not work.

Therefore, this paper proposes a patterned fabric image representation method based on NSCT, which can endow the fabric retrieval system with the ability to capture similarities within images in either shape or color in an unsupervised way. To overcome the limitations of hand-crafted descriptors, we employ a RF technique, which is based on geometric similarity, to optimize and improve the performance of the retrieval system.

Feature extraction of fabric images

In this section, we briefly introduce NSCT, which is employed in our system to extract features for patterned fabric image representation. Then we present the detailed process of fabric image representation based on NSCT.

Non-subsampled contourlet transformation

CNT is a new 2D image representation method, with multi-resolution, local positioning, multi-directionally, nearest-neighbor sampling, and anisotropy, and its basic function is distributed in multi-scale and multi-direction. This method can effectively capture the contour features in an image with a small number of coefficients. However, the contour feature is an important feature in the patterned fabric image. Non-Subsampled Contourlet Transform (NSCT) NSCT can thus be divided into two shift-invariant parts: (a) a NSP structure that ensures the multi-scale property of the NSCT; (b) a NSDFB gives the directionality. Figure 1(a) displays an overview of NSCT. We first use the NSP (introduced in the next section) to perform a multi-scale decomposition of the input fabric image. The lowpass sub-band is extracted from the first-level decomposition. Then we continue to decompose the results of the first layer of decomposition to obtain multi-scale bandpass directional sub-bands. The structure consists of a bank of filters that split the 2D frequency plane into the sub-bands, as illustrated in Figure 1(b).

Figure 1.

Non-subsampled contourlet transform (NSCT): (a) Non-subsampled Filter Bank (NSFB) structure that implements the NSCT; (b) idealized frequency partitioning obtained with the NSFB structure.

Non-subsampled pyramid

Similar to the sub-band decomposition of the Laplacian pyramid, the multi-scale property of the NSCT is attributed to a shift-invariant filtering structure. However, unlike the Laplacian pyramid, NSCT uses two-channel non-subsampled 2D filter banks, which produce a low-frequency and high-frequency image at each NSP decomposition level. Figure 2 illustrates the NSP decomposition with J = 3 stages. In the first stage, the input image is composed of two components, low-pass L₀(z) and high-pass H₀(z). So, such expansion has J + 1 redundancy, where J denotes the number of decomposition stages. The ideal passband support for the low-pass filter in the jth stage is the region [–(π/2 ^j ), (π/2 ^j )]². Correspondingly, the ideal support for the equivalent high-pass filters is complementary to those for low-pass filters, that is, the region [–(π/2 ^j ^–1), (π/2 ^j ^–1)]²\[–(π/2 ^j ), (π/2 ^j )]², which is presented in Figure 2(b). The filters for the subsequent stages are obtained by upsampling the filter of the first stage. This provides multi-scale properties without the need for other filter designs. In particular, one bandpass image is generated at each stage, resulting in J + 1 redundancy. Decomposition can be achieved by removing the upsampler and downsampler in the Laplacian pyramid and then upsampling the filter accordingly. These reconstruction systems can be regarded as a special case of the more general structure. Therefore, they filter certain parts of the noise spectrum in the processed pyramid coefficients.

Figure 2.

Non-subsampled pyramid: (a) non-subsampled pyramid filter bank with three-stage decomposition, where L₀(z) is the low-pass and H₀(z) is the high-pass; (b) sub-bands on the two-dimensional frequency plane.

Non-subsampled directional filter bank

The NSDFB is constructed by removing the downsamplers and upsamplers of the directional filter bank (DFB) and by upsampling the filters accordingly. These operations result in a tree composed of two-channel NSFBs. Figure 3 represents a four-channel NSDFB, which is constructed with two-channel fan filter banks. Firstly, the input image is divided into four directional sub-bands using the corresponding filter bank (shown in Figure 3(b)). The parallel filter bank is used to iterate sub-bands in different directions. The sub-band of each direction only includes the information in that direction. In the second level, the upsampled fan filters U_i(z^Q), i = 0,1 have checker-board frequency support. When used in conjunction with the filter in the first stage, the four-way frequency decomposition shown in Figure 3 will be performed.

Figure 3.

Four-channel non-subsampled directional filter bank, which is constructed with two-channel fan filter banks: (a) filtering structure; (b) corresponding frequency decomposition.

At each stage of the NSP, the NSDFB can be decomposed into any number of 2 ^l directions, where l denotes the number of levels in the NSDFB. This provides NSCT with multi-directionality and offers accurate directional information. The NSCT is actually a combination of the NSP and the NSDFB, which can be seen in Figure 1(a). After NSCT, the size of the directional sub-image at each scale is the same size as the original image, and the sum of all sub-bands is equal to the original image.

Fabric image representation based on NSCT

In CBIR, feature storage and computational complexity must be considered when designing the image representation methods. Many researchers applied image transformation to obtain such a representation. However, conventional transformation approaches, such as WT and FT, suffer from image discontinuities, such as edge-induced discontinuities. To avoid this situation, this study applies NSCT, which has been introduced in the previous section, to represent the image.

In this study, we mainly focus on the pattern in the fabric image. To weaken the influence of the texture of the fabric image on the shape feature extraction, we first smooth the fabric image. The smooth method is based on relative total variation.³⁹ As shown in Figures 4(a) and (e), the texture information in the smoothed image is greatly weakened. In addition to the shape and contour information in the patterned fabric image, another important component that cannot be ignored is the color information in the image. To deal with a color metric that better matches the human perception, we map the RGB color space images to HSV color space. This is important, as humans will play a key role during the retrieval phase. Moreover, by decoupling the brightness intensity and the chromatic channels, we ensure the independence between shape features and color features. Generally, the NSCT decomposition over the V channel extracts the shape information, and the same decomposition over H and S channel captures the color information. The results in Figure 3 once again prove that the proposed image smoothing method can well eliminate the influence of impurities on image representation.

Figure 4.

A fabric image sample and its three HSV channels: (a) original image; (b) H channel; (c) S channel; (d) V channel; (e) image with t-smooth; (f) H channel of image with t-smooth; (g) S channel of image with t-smooth; (h) V channel of image with t-smooth.

The shape features and color features are extracted by using NSCT on H, S, and V channels with a four-level decomposition (configuration: 1, 2, 4, 4). Such a decomposition provides 11 (= 1 + 2 + 4 + 4) sub-bands per channel (three channels, in total 33 sub-bands per image) for each image in the database. The reason we consider a four-level decomposition is that when the decomposition level is greater than 4, the peak signal-to-noise ratio (PSNR), which measures the quality of the decomposed image, will not improve significantly. Besides, high-level decomposition will lead to more computational costs, resulting in low efficiency. Consequently, the designed decomposition in this study can make a good compromise between less information loss and compact image representation. Then, each sub-band S ^j (0 < j ≤ 33) is summarized in terms of its mean, standard deviation, and energy as follows

\begin{array}{l} f_{mean}^{j} = \frac{1}{h * w} \sum_{m = 1}^{h} \sum_{n = 1}^{w} S_{m n}^{j} # \end{array}

(1)

f_{std}^{j} = \sqrt{\frac{1}{h * w} \sum_{m = 1}^{h} \sum_{n = 1}^{w} (S_{m n}^{j} - f_{mean}^{j})}

(2)

f_{energy}^{j} = \frac{1}{h * w} \sum_{m = 1}^{h} \sum_{n = 1}^{w} {{(S}_{m n}^{j})}^{2}

(3)

where

S_{m n}^{j}

denotes the NSCT coefficient at the spatial location (m, n) of sub-band S ^j, which is an h × w-dimensional matrix. Thus, each image in the database is represented by a 99-dimensional vector

X_{i} = (f_{mean}^{1}, \dots, f_{mean}^{33}, f_{std}^{1}, \dots, f_{std}^{33}, \dots, f_{ener g y}^{1}, \dots, f_{energy}^{33})

(4)

Relevance feedback and retrieval system

Relevance feedback technique

The proposed RF technique, which is based on geometric similarity, explicitly uses the information about feature points of the non-relevant image. The proposed method iteratively updates the parameters of the geometric similarity measure to fit the relevant feature vectors while excluding non-relevant ones. The RF is achieved by modifying the weights associated with the relevant examples. Specifically, relevant points that are far from non-relevant points are given greater weight, while relevant points that are close to non-relevant points are given outliers and are given less weight.

Assume $X_{1}, X_{2} {, \dots X}_{n} \in R^{d}$ represent a set of image feature vectors (where d is the dimension of the feature space) and $Y_{1}, Y_{2} {, \dots Y}_{n}$ denote the corresponding user responses (when X_i is relevant to the user, Y_i = 1; otherwise, Y_i = –1). Then let X^r and Xⁿ respectively denote the set of relevant and non-relevant feature vectors from the user response. In addition, ω ^r∈ $R^{| X^{r} |}$ , whose initial value is set to 1, denotes the weights associated with relevant examples. As mentioned above, the parameter distance metric between the two image feature vectors X and Y is represented by

d (X, Y) = (X - Y) A {(X - Y)}^{T}

(5)

This method is based on the similarity measure of the generalized ellipsoid. When given the feature vector $X$ in the database and the feature vector $μ$ of a query, the relevance can be computed as

d (X, μ, A) = (X - μ) A {(X - μ)}^{T}

(6)

The cross-correlation metric A is initialized to 1, and the distance between feature vector X and $μ$ can be regarded as the Euclidean distance. Here, we take d = 2 as an example to introduce the proposed RF technique. In Figure 5, the diamonds represents the input query and the dots indicate the retrieval results of the CBIR system (blue indicates non-relevant, green indicates relevant).

Figure 5.

Iterative learning of the similarity metric on a synthetic two-dimensional dataset: (a) original dataset; (b) the first iteration; (c) the third iteration. (Color online only.)

Our goal is to estimate $μ$ and A so that the sum of distances between the vector of the query and the relevant vectors from the target concept is minimized. In terms of the example shown in Figure 5, the goal is to estimate the location of the red diamond and the shape of the boundary, so as to obtain the sum of distances between the red diamond and green dots. We express this optimization problem with the following formula

J = \sum_{i = 1}^{n_{g}} ω_{i}^{r} {(X - q)}^{T} A (X - q)

(7)

where n_g denotes the dimension of ω. Here, q and A, subject to the constraint det(A) = 1, can be represented by (8) and (9)

μ (X^{r}, ω^{r}) = \frac{\sum_{x_{i} \in X^{r}} ω_{i}^{r} x_{i}}{\sum ω_{i}^{r}}

(8)

A (X^{r}, ω^{r}) = C^{- 1}

(9)

where C is the covariance matrix for the set X^r of relevant images, which can be denoted by

C (X^{r}, ω^{r}) = \frac{\sum_{j = 1}^{n_{g}} ω_{j} (X_{j} - μ) {(X_{j} - μ)}^{T}}{\sum_{i = 1}^{n_{g}} ω_{i}^{r}}

(10)

where

ω_{}^{r}

is provided by the users, generally. Existing systems require users to rank images according to their relevance, from very relevant to less relevant. This kind of multi-level correlation assignment is too difficult for users and is error-prone. So, we propose a new method that only seeks the binary correlation assignment from the user, and the weights (

ω_{}^{r}

) are automatically computed based on their distances from the non-relevant images, that is, the user just needs to feedback to the system whether each result and query are relevant.

The proposed method iteratively updates the weights (

ω_{}^{r}

) and the parameters of similarity metrics, so that the ellipse representing the similarity matrix can better capture relevant images and exclude non-relevant ones. The pseudo-code of the proposed optimization process is shown in Algorithm 1. We initialize all weights

ω_{}^{r}

to 1, which means that all the relevant examples are considered to be equally important. In each iteration,

μ

and A can be calculated by using the current weights. The

X_{\max}^{r}

denotes the farthest positive example with a non-zero weight and

d_{\max}^{r}

is its distance from query

μ

. Let

ε

be the ellipsoid with a radius of

d_{\max}^{r}

determined by

(μ, A)

. Then, let

X_{new}^{n}

denote the set of non-relevant examples that fall inside the ellipsoid

ε

. The purpose of the proposed algorithm is to update the parameters

(μ, A)

to reduce the number of elements in

X_{new}^{n}

. In each iteration, the weight of the farthest positive sample

X_{\max}^{r}

is set to 1. The weights of other non-negatively weighted positive samples are updated to the sum of their quadratic distances from the samples in

X_{new}^{n}

. Then, the updated weight is used to obtain a new estimate of the similarity measurement parameter, and then the iterative. When the size of

X_{new}^{n}

becomes zero, the iteration stops. The algorithm proceeds by deleting the positive examples with the farthest distance in each iteration. This positive example is called an “outlier” because including it in the estimation of the similarity measure will cause the distance of the negative example to be less than the distance of the positive example. This would lead to the examples in the

X_{new}^{n}

being retrieved again in the next iteration. To avoid this problem, the remaining relevant examples will be weighted by the cumulative distance from the examples in

X_{new}^{n}

. Therefore, the estimated metric in the next iteration will be forced away from

X_{new}^{n}

Algorithm 1: Proposed weight optimization algorithm
Inputs: $X^{r}$ —the relevant examples, $X^{n}$ —the non-relevant examples.
Outputs: A and $μ$ , the parameters of the similarity metric.
Assume targetd is the dimension of the feature space class with the smallest number of components;
Assume ω r∈ $R^{\| X^{r} \|}$ ; ω r = 1.
while (1) {
Calculate $A (X^{r}, ω^{r})$ and $μ (X^{r}, ω^{r})$ using (9) and (8);
$X_{\max}^{r} = {argmax}_{X \in X^{r}} \{d (X; μ, A)\}$ , where d (*) is given by (6);
$d_{\max}^{r} = d (X_{\max}^{r}; μ, A)$ ;
$X_{new}^{n} = {X : X \in X^{r}; d (X; μ, A) < d_{\max}^{r}}$ ;
$ω^{r} [X_{\max}^{r}] = 0$
$ω_{i}^{r} = \{\begin{array}{l} 0 i f (ω_{i}^{r} = 0) \\ \sum_{z \in X_{new}^{n}} d (X; Z, A) \end{array}$
if ( $\|X_{new}^{n}\| = = 0 or \|\{ω^{r} : ω_{i}^{r}! = 0\}\| < target d$ )
break;
}
return A and $μ$

The proposed algorithm is greedy in nature since the positive example with the farthest distance is removed in each iteration. Refer to Figure 5 to see how the proposed algorithm seeks outliers and iteratively generates the similarity metric.

Fabric image retrieval system

In this section, we summarize the algorithm underlying our CBIR system with the parametric RF technique. We first preprocess the fabric image to reduce the influence of micro-texture on image representation. Then we convert the image from RGB color space into HSV color space. By applying the NSCT-based image representation method, we extract a 99-dimensional vector (using (1)–(4)) for each image from the image dataset. The extracted vectors constitute the feature dataset for image retrieval. By using these two datasets, we build the fabric CBIR system, whose similarity measurement method applies Euclidean distance.

Then, the images in the testing dataset are input to the CBIR system and output the retrieval results, which contain relevant examples X^r and non-relevant examples Xⁿ. The results are used to improve the similarity measurement method by optimizing $μ$ and A. Once the parameters $μ, A$ are computed by the optimization algorithm, the system updates its similarity measurement method for outputting better results.

Experimental details

Experimental setup

In this section, we demonstrate the performance of our proposed fabric image retrieval framework on a real fabric image dataset. We compare the performance of our CBIR system with several state-of-the-art CBIR systems.

We evaluate the proposed retrieval method on the patterned fabric image dataset. Then, we first introduce the dataset used in this study. The dataset consists of 8000 patterned fabric images, including 400 sets of fabric images, and each set contains 20 related fabric images. The size of the images in the dataset is 512 × 512 × 3. There are three main criteria to judge whether the fabrics are similar, namely, the forming process, the color, and the subjective evaluation. The fabrics are all woven fabrics. The images are captured in an RGB model using a scanner (Canon 9000F Mark II). The light source of the scanner was a white light-emitting diode (LED), which can guarantee a stable capture environment. The resolution was set to 200 dpi. The patterned fabric can be formed by printing, yarn-dyed weaving, or jacquard. The collected fabric images are then are classified into four categories: Lattice (148 sets), Jacquard (61 sets), Printed (49 sets), and Strip (142 sets). Figure 6 shows some samples of captured images in the dataset. We will discuss the influence of different formation methods on the retrieval results. To verify the robustness of the proposed NSCT method, we also perform some transformations on the query image, including rotation, scale, and flipping.

Figure 6.

Some samples of patterned fabric images in the dataset.

The proposed CBIR framework is implemented by using Python software (lib: sklearn, numpy, PIL, opencv-python). The hardware environment of the experiments is as follows: CPU E5-2623 v4 @ 2.60 GHz, RAM 32 G. The proposed framework adopts a four-level NSCT decomposition, which is described in the second section. The decomposition is conducted on HSV color space. The reasons for these choices are analyzed in the following section. The performance of the proposed method for fabric image retrieval is evaluated by considering each fabric image as the query image on the testing set and by measuring the precision–recall curve and mean of average precision (mAP). Precision and recall are defined as

P = \frac{N_{RIR}}{N_{RIR} + N_{IRIR}}

(11)

R = \frac{N_{RIR}}{T_{RID}}

(12)

where N_RIR denotes the number of relevant images output by the retrieval framework, N_IRIR is the number of irrelevant images output by the retrieval framework, and T_RID denotes the total number of relevant images in the dataset. The average precision (AP) can be calculated by

A P = \frac{1}{N_{T r} (q) @ n} = \sum_{i = 1}^{n} T r (q, i) \frac{N_{T r} (q) @ i}{i}

(13)

where

T r (q, i) \in {0, 1}

is an indicator function that if the query Iq and the ith retrieval result Ii are relevant images,

T r (q, i) = 1

; otherwise,

T r (q, i) = 0

. Here,

N_{T r} (q) @ i

denotes the number of relevant images with the top i images. Let Q represent the number of the query set. Then the mAP can be calculated by

mAP = \frac{1}{Q} \sum_{q}^{Q} A p (q)

(14)

Feature evaluation

In this experiment, we evaluate the quality of the proposed NSCT-based low-level features computed using different color spaces, namely HSV, RGB, CIELab, and YCbCr. Figure 7 presents the retrieval performance obtained with the different color spaces on the patterned fabric dataset. The results clearly show that the proposed method using HSV color space can outperform the others by a large margin. The results demonstrate that the NSCT-based features computed on HSV color space can obtain a better representation of patterned fabric images, and then achieve a better retrieval performance. The reason for these results may be that the HSV color space expresses colors more in line with the way humans perceive colors, while RGB and YCbCr suffer from color closeness, which is not supported by the human visual system.

Figure 7.

Precision–recall curves for the patterned fabric retrieval.

Qualitative results

In this section, we present three sample queries from the testing set before and after some feedback steps, as shown in Figures 8 –10. Figures 8(a), 9(a), and 10(a) show the initial retrieval results of different fabrics (Houndstooth, lattice, and stripe) on the dataset. Here, the top-left image is the input query image. The precisions of these three retrievals are 15/20, 17/20, and 17/20. In Figure 8(a), the input query is a Houndstooth, also known as dogstooth, dogtooth, dog’s tooth, or pie-de-poule, which is a duotone textile pattern characterized by broken checks or abstract four-pointed shapes, often in black and white, although other colors are used. Most of the retrieval results belong to this type of fabric. The remaining ones have incorrectly retrieved images from other categories, which are mainly due to the prominent role of small-period texture features in these categories. Nevertheless, also the shape component is partially causing retrieval errors, as some erroneous images do have a Houndstooth-like shape. After three relevant feedback iteration, only one error remains; the proposed method achieves 100% precision within the 10th feedback iteration. The same situation applies to the following two retrieval examples. In summary, the fabric images in the initial retrieval results are very similar in shape, and the number of irrelevant fabrics in the retrieval results keeps decreasing with the iteration of relevant feedback, which qualitatively demonstrates the effectiveness of the proposed fabric retrieval method.

Figure 8.

Retrieval results of a Houndstooth fabric on the testing set with the query image in the top-left corner: (a) initial result (75% precision); (b) third iteration (90% precision); (c) 10th iteration (100% precision).

Figure 9.

Retrieval results of a lattice fabric on the testing set with the query image in the top-left corner: (a) initial result (85% precision); (b) third iteration (95% precision); (c) 10th iteration (100% precision).

Figure 10.

Retrieval results of a stripe fabric on the testing set with the query image in the top-left corner: (a) initial result (85% precision); (b) third iteration (95% precision); (c) 10th iteration (100% precision).

Quantitative results

In this section, we conduct some quantitative experiments, including the comparison of initial retrieval results and the comparison of feedback adjustment capabilities, to prove the effectiveness of the proposed approach.

We first do not involve the user in the retrieval loop and, therefore, we simply measure the performance of the initial retrieval results. In this section, we compare the retrieval performance of the proposed method and several CBIR methods on fabric, including PFCS⁵ (a fabric image retrieval method based on color histogram and image segmentation), PFCG⁶ (a fabric image retrieval based on CM and gist feature), and LFML⁷ (a fabric image retrieval based on multi-scale LBP). In addition, we also compare the proposed method with our previous works, including WFBT⁹ (a fabric image retrieval method based on the LBP feature), WFBC⁸ (a fabric image retrieval method based on the CM), and FRHS³ (a fabric image retrieval method based on CNNs). Here we apply top-20 precision (P-top20), top-10 precision (P-top10), and mAP to measure the retrieval performance of the mentioned methods. The experimental results are presented in Table 1. The results clearly show that the retrieval performance of all methods on yarn-dyed fabrics is better than that on jacquard fabrics and printed fabrics. Moreover, the retrieval results for lattice fabrics are better than that for stripe fabrics. This phenomenon shows that these methods perform better for the representation of simple pattern fabrics. Among these methods, WFBC, LFML, FRHS, and the proposed method have rotation invariance and scale invariance. As shown in Table 1, the transformation of the input query has little effect on the retrieval results of these methods. These results also prove the robustness of these methods. However, other methods are very sensitive to the transformation, and the retrieval performance is significantly reduced. We can also find that the learning-based method FRHS achieves a good performance for lattice and stripe fabrics, but poor performance for jacquard and printed fabrics, which demonstrates the dependence of the CNN method (supervised) on labeled data. In addition, it can be obviously observed that our approach achieves the best performance on the patterned fabric dataset, compared to the other state-of-the-art approaches. The results demonstrate the superiority of the proposed approach over the compared methods. In addition, the results also prove that the proposed method can achieve good performance on patterned fabric retrieval, especially yarn-dyed fabric.

Table 1.

Results of quantitative comparison experiment results that do not involve user feedback. The definitions of precision (top-20 precision (P-top20) and top-10 precision (P-top10)) and mean of average precision (mAP) are presented in Equations (11) and (14), respectively. The second column “Transform” has two options: “YES” means random scaling, rotation, or flipping of the input query; “NO” means no transformation of the input query

Method	Transform	P-top20	P-top10	mAP of each categories				mAP
Method	Transform	P-top20	P-top10	Lattice	Stripe	Jacquard	Printed	mAP
PFCS⁵	NO	0.839	0.862	0.791	0.798	0.781	0.791	0.792
PFCS⁵	YES	0.719	0.757	0.701	0.716	0.739	0.743	0.717
WFBT⁹	NO	0.841	0.858	0.799	0.806	0.739	0.746	0.786
WFBT⁹	YES	0.634	0.718	0.655	0.679	0.601	0.637	0.653
WFBC⁸	NO	0.839	0.864	0.815	0.818	0.769	0.775	0.804
WFBC⁸	YES	0.826	0.857	0.810	0.813	0.741	0.754	0.793
PFCG⁶	NO	0.815	0.837	0.771	0.788	0.753	0.762	0.773
PFCG⁶	YES	0.713	0.753	0.643	0.652	0.694	0.705	0.662
LFML⁷	NO	0.792	0.821	0.758	0.783	0.721	0.732	0.758
LFML⁷	YES	0.786	0.813	0.744	0.769	0.710	0.719	0.744
FRHS³	YES	0.845	0.881	0.839	0.834	0.789	0.759	0.815
FRHS³	NO	0.836	0.869	0.824	0.821	0.768	0.746	0.810
Proposed	NO	0.847	0.877	0.819	0.827	0.806	0.809	0.819
Proposed	YES	0.839	0.871	0.813	0.824	0.801	0.799	0.813

We conduct a second experiment on the patterned fabric dataset involving also the relevant feedback. The compared RF methods include ESM⁴⁰ (auto set adaptive weights of similarity measurement for each dataset image from the user feedback), GBRF⁴¹ (a graph-theoretic approach to rank the images following the user’s feedback), and SSL⁴² (a semi-supervised long-term RF algorithm). We also use mAP, P-top10, and P-top20 to measure the performance of the four methods. The comparison results are shown in Figure 11. It can be observed that, from the first iteration to the 10th iteration, all RF algorithms improve the retrieval performance of the proposed retrieval method. In addition, the proposed RF algorithm achieves a better improvement (+0.156 mAP, +0.11 P-top10, +0.138 P-top20) over the compared algorithm. These results demonstrate the superiority of the proposed RF algorithm. Moreover, from the Figure 11(a), we can observe that, the proposed and GBRF algorithm converges after eight iterations and the SSL algorithm converges after six iterations, while ESM does not coverage after 10 iterations.

Figure 11.

Plots of the mean of average precision (mAP), top-10 precision (P-top10), and top-20 precision (P-top20) of different relevance feedback algorithms: (a) mAP-iteration curve; (b) P-top10 iteration curve; (c) P-top20 iteration curve.

Conclusion

In this paper, we present an efficient interactive CBRI framework, which is based on a NSCT feature descriptor and an RF algorithm, for patterned fabric image retrieval. The NSCT feature descriptor can capture the geometrical features of the patterned fabric images. To integrate color information into NSCT, we extract features in HSV color space. When given a query, the extracted features are used to retrieve similar images from the dataset. Experimental results have shown that our retrieval framework is effective. In more detail, the adoption of a feature representation, which derives from a flexible multi-scale, multi-directional, and shift-invariant image decomposition method like NSCT, endows the fabric retrieval system with the ability to capture similarities within images in either shape or color in an unsupervised way. In addition, an outlier rejection-based parametric RF algorithm is employed to adjust the similarity matrix, and thus fit the retrieval results to the user’s needs. The qualitative and quantitative experiments of the proposed retrieval system with relevant feedback that we have conducted on a patterned fabric dataset, and reported in Figures 8 –11 and Table 1, have shown that the proposed RF algorithm effectively exploits the user’s feedback to improve the quality of the retrieved results. Furthermore, the comparative experimental results demonstrate the superiority of the proposed patterned fabric image retrieval framework.

Footnotes

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by the National Key R&D Program of China (Grant 2017YFB0309200), in part by the Fundamental Research Funds for the Central Universities (Grant JUSRP52007A), and in part by the National Natural Science Foundation of China (Grant 61976105).

ORCID iD

Jun Xiang

References

Deng

Wang

, et al. Learning deep similarity models with focus ranking for fabric image retrieval. Image Vis Comput 2018; 70: 11–20.

Suciati

Herumurti

and Wijaya AY. Fractal-based texture and HSV color features for fabric image retrieval. In: 2015 IEEE international conference on control system, computing and engineering (ICCSCE), Mohd Nasir Taib, Penang, Malaysia, 27–29 Nov.2015, 16037595, NY, IEEE, pp.178–182.

Xiang

Zhang

Pan

, et al. Fabric image retrieval system using hierarchical search based on deep convolutional neural network. IEEE Acc 2019; 7: 35405–35417.

Xiang

Zhang

Pan

, et al.

Fabric retrieval based on multi-task learning.

IEEE Trans Image Proc 2020; 30: 1570–1582.

Jing

, et al. Patterned fabric image retrieval using color and space features. J Fiber Bioeng Informat 2015; 8: 603–614.

Jing

, et al. A new method of printed fabric image retrieval based on color moments and gist feature description. Text Res J 2016; 86: 1137–1150.

Zhang

Liu

, et al. Lace fabric image retrieval based on multi-scale and rotation invariant LBP. In: proceedings of the 7th international conference on internet multimedia computing and service, Ramesh Jain, Hunan China, August 19 – 21, 2015, 74, NY, ACM, pp.1–5.

Zhang

Xiang

Wang

, et al. Image retrieval of wool fabric. Part II: based on low-level color features. Text Res J 2020; 90: 797–808.

Zhang

Xiang

Wang

, et al. Image retrieval of wool fabric. Part I: based on low-level texture features. Text Res J 2019; 89: 4195–4207.

10.

Smeulders

Worring

Santini

, et al. Content-based image retrieval at the end of the early years. IEEE Trans Patt Anal Mach Intell 2000; 22: 1349–1380.

11.

Liu

Zhang

, et al. A survey of content-based image retrieval with high-level semantics. Patt Recognit 2007; 40: 262–282.

12.

Wan

Wang D Hoi SCH , et al. Deep learning for content-based image retrieval: a comprehensive study. In: proceedings of the 22nd ACM international conference on multimedia, Kien A. Hua, Orlando Florida USA, Nov 3-4, 2014, Vol 18,NY, ACM 2014, pp.157–166.

13.

Kuo SC-Y, Shih

C-Y

Lee

J-Y.

Automatic recognition of fabric weave patterns by a fuzzy C-means clustering method.

Text Res J 2004; 74: 107–111.

14.

Kang

Kim

KW.

Automatic recognition of fabric weave patterns by digital image analysis.

Text Res J 1999; 69: 77–83.

15.

Guo

, et al. Automatic recognition method for the repeat size of a weave pattern on a woven fabric image. Text Res J 2019; 89: 2754–2775.

16.

Kuo

C-FJ

Tsai

C-C.

Automatic recognition of fabric nature by using the approach of texture analysis. Text

Res J 2006; 76: 375–382.

17.

Pan

Gao

Liu

, et al. Automatic recognition of woven fabric patterns based on pattern database. Fiber Polym 2010; 11: 303–308.

18.

Sifuzzaman

Islam

Ali

Application of wavelet transform and its advantages compared to Fourier transform. J Physical Sci 2009; 13: 121–134.

19.

Bracewell

RN.

The Fourier transform and its applications. Vol. 31999. New York: McGraw-Hill, 1986.

20.

Zhang

Generic Fourier descriptor for shape-based image retrieval. In proceedings of the IEEE international conference on multimedia and expo, Lausanne, Switzerlang, 26-29, Aug.2002, 7540663, NY, IEEE, pp.425–428.

21.

Zhang

Shape-based image retrieval using generic Fourier descriptor.

Sign Proc Image Commun 2002; 17: 825–848.

22.

Ardizzoni

Bartolini

Patella

Windsurf: Region-based image retrieval using wavelets. In: proceedings. tenth international workshop on database and expert systems applications (DEXA 99), Florence, Italy, 3rd Sept, 1999, Vol 6359062, NY, IEEE, pp.167–173.

23.

Allard

Chen

Maggioni

Multi-scale geometric methods for data sets II: geometric multi-resolution analysis.

Appl Computat Harmon Anal 2012; 32: 435–462.

24.

Ying

Salari

Beamlet transform‐based technique for pavement crack detection and classification. Comput Aid Civ Infrastruct Eng 2010; 25: 572–580.

25.

Donoho

DL.

Wedgelets: nearly minimax estimation of edges.

Ann Stat 1999; 27: 859–897.

26.

Le Pennec

Mallat

Bandelet image approximation and compression.

Multiscale Model Simulat 2005; 4: 992–1039.

27.

Da Cunha

Zhou

MN.

The nonsubsampled contourlet transform: theory, design, and applications.

IEEE Trans Image Proc 2006; 15: 3089–3101.

28.

Candes

Demanet

Donoho

, et al. Fast discrete curvelet transforms. Multiscale Model Simulat 2006; 5: 861–899.

29.

Vetterli

The finite ridgelet transform for image representation.

IEEE Trans Image Proc 2003; 12: 16–28.

30.

Zhang

Gao

Medical image retrieval based on nonsubsampled contourlet transform and fractal dimension. In: 2009 3rd international conference on bioinformatics and biomedical engineering, Beijing China, June 11-13, 2009, Vol 10777984, NY, IEEE, pp.1–4.

31.

Zhang

An efficient method for human face recognition using nonsubsampled contourlet transform and support vector machine.

Optica Applicata 2009; 39: 601–615.

32.

Ng-Duc

Le-Tien

Do-Hong

, et al. Image retrieval using contourlet based interest points. In: 10th international conference on information science, signal processing and their applications (ISSPA 2010), Kuala Lumpur, Malaysia, May 10-13, 2010, Vol 11596669, NY, IEEE, pp.93–96.

33.

Rui

Huang

Mehrotra

Content-based image retrieval with relevance feedback in MARS. In: proceedings of international conference on image processing, Santa Barbara, CA, USA, Oct 26-29, 1997, Vol 5899998, NY, IEEE, pp.815–818.

34.

Ishikawa

Subramanya

Faloutsos

MindReader: querying databases through multiple examples. Jennifer Widom, NY, USA, Aug 24-27, 1998, Vol 18, pp 218-227, NY, ACM.

35.

Luo

Jiang

, et al. Content-based lace fabric image retrieval system using texture and shape features. J Text Inst 2019; 110: 911–915.

36.

Xiang

Wang

, et al.

Yarn-dyed fabric image retrieval using colour moments and the perceptual hash algorithm.

Fibre Text East Eur 2019, 27(5(137)): 60–69.

37.

Cai

Gao

, et al. Feature extraction with triplet convolutional neural network for content-based image retrieval. In: 2017 12th IEEE conference on industrial electronics and applications (ICIEA), Siem Reap, Cambodia, June 18-20, 2017, Vol 17576299, NY, IEEE, pp.337–342.

38.

Wang

Zhong

Fabric identification using convolutional neural network. In: international conference on artificial intelligence on textile and apparel, Wai Keung Kong, Hong Kong, China, July 3-6, 2018, Vol 849, Cham, Springer, pp.93–100.

39.

Yan

Xia

, et al. Structure extraction from texture via relative total variation. ACM Trans Graph 2012; 31: 1–10.

40.

Lee

C-H

Lin

M-F.

Ego-similarity measurement for relevance feedback.

Exp Syst Appl 2010; 37: 871–877.

41.

Kundu

Chowdhury

Bulò

SR.

A graph-based relevance feedback mechanism in content-based image retrieval.

Knowl Based Syst 2015; 73: 254–264.

42.

Yang

Nie

, et al.

A multimedia retrieval framework based on semi-supervised ranking and relevance feedback.

IEEE Trans Patt Anal Mach Intell 2011; 34: 723–742.