Cross-modal variable-length hashing based on hierarchy

Abstract

Due to the emergence of the era of big data, cross-modal learning have been applied to many research fields. As an efficient retrieval method, hash learning is widely used frequently in many cross-modal retrieval scenarios. However, most of existing hashing methods use fixed-length hash codes, which increase the computational costs for large-size datasets. Furthermore, learning hash functions is an NP hard problem. To address these problems, we initially propose a novel method named Cross-modal Variable-length Hashing Based on Hierarchy (CVHH), which can learn the hash functions more accurately to improve retrieval performance, and also reduce the computational costs and training time. The main contributions of CVHH are: (1) We propose a variable-length hashing algorithm to improve the algorithm performance; (2) We apply the hierarchical architecture to effectively reduce the computational costs and training time. To validate the effectiveness of CVHH, our extensive experimental results show the superior performance compared with recent state-of-the-art cross-modal methods on three benchmark datasets, WIKI, NUS-WIDE and MIRFlickr.

Keywords

Cross-modal retrieval hash learning hierarchy structure discrete optimization

1. Introduction

Due to the explosive growth of big data, cross-modal retrieval has occupied an increasingly important status in information retrieval. However, due to the heterogeneous between different modalities, it is impossible to directly represent other modalities data in its original feature space. Therefore, how to find the correlations between different modalities, and to achieve accurately retrieval has become increasingly vital. At present, classic mainstream cross-modal retrieval algorithms can be divided into three categories: subspace learning-based methods, deep learning-based methods and hashing learning-based methods.

A kind of methods based on subspace learning [9, 10, 12, 11, 13], which first use the paired symbiosis information of different modalities to learn projection matrix. Then project the different features into a latent common subspace by projection matrix, and obtain the similarity of different modalities in this subspace. Finally, sort similarity and implement cross-modal retrieval process. Although existing cross-modal retrieval based on subspace learning have made some progress, there still suffered from problems of differences in spatial structure and matching problems between different modalities. On the other hand, subspace learning can only learn the linear mapping, which cannot effectively model the higher-order correlations of different modalities.

Another kinds of retrieval methods are based on deep learning [14, 15, 16], which mainly use deep learning to extract the underlying features of heteromodal data, and then use high-level network to represent the semantic correlation between the modalities. Although deep learning-based methods have better retrieval performance than subspace methods, it does not match the spatial structure within the modal and the semantic connection between different modalities, also because of its complex network structure and high training costs, we only use several classic traditional methods in this paper.

To address these problems above, another kind of methods are based on hashing learning [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35] has been increasingly used in machine learning, computer vision and information retrieval due to its efficient retrieval performance and low computational costs. Cross-modal hashing methods generate compact binary codes for fast and efficient retrieval by mapping the original high-dimensional data into a low-dimensional feature space. However, most hashing-based methods map original data to fixed-length hash codes, which may not represent different modalities data well. Therefore, the accuracy of cross-modal retrieval cannot be guaranteed. In addition, solving the binary hash codes is an NP-hard problem [17, 18, 25]. Most of methods try to relax the binary constraints of hash codes, which resulting the learned binary codes are not accurate enough.

In order to solve these issues, we initially propose a novel cross-modal retrieval method called Cross-modal Variable-length Hashing Based on Hierarchy (CVHH). CVHH uses variable-length hashing codes to represent different modalities data to learn more accurate representation of different modalities data. Furthermore, in order to take into account the correlation between different modalities data on the global manifold structure, we construct a hierarchy structure based on algebraic multigrid (AMG) model [31, 33]. We assume that high-dimensional neighbor data is also similar in projected low-dimensional space. In fact, the AMG model is currently only used for unimodal retrieval. Therefore, in order to further improve the retrieval performance of cross-modal data and reduce the computational costs of training process, we apply AMG model into our cross-modal variable-length hashing method. The main contributions of CVHH can be summarized as follows:

(1)
We utilize variable-length hash codes to represent different modalities data, which reflects the original structure in their respective feature spaces.
(2)
We construct a cross-modal hierarchical structure based on AMG model, and select the core instances to represent the original entire training set, which can greatly reduce the computational costs and training time.
(3)
Experimental results on three benchmark cross-modal datasets show that our proposed CVHH is superior to other comparison methods in cross-modal retrieval.

The rest of this paper is organized as follows. In Section 2, we introduce some related work in the field of cross-modal retrieval. Section 3 presents the theoretical analysis and algorithm optimization of our CVHH in detail. Section 4 analyses and compares the experimental results of CVHH with other comparison methods. Finally, the conclusion is summarized in section 5.
2. Related work

In this section, we introduce two kinds of cross-modal methods, called latent subspace learning [9, 10, 12, 11, 13] and binary representation learning [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35], namely hashing learning.

2.1 Latent subspace learning

Latent subspace learning methods attempt to eliminate heterogeneity between different modalities, and match different features by the same subspace. The most typical one is named Canonical Correlation Analysis (CCA) [11], which finds the mapping vector of each modal to the common subspace by maximizing the correlation coefficient between two modalities. However, since the features of original data are greatly different, how to express the correlation between different original data has become the most important problem. In order to more accurately obtain the correlation between different modalities and implement better retrieval performance, hash learning methods are widely used in cross-modal retrieval tasks in recent years.

2.2 Hashing learning

Hashing learning can effectively improve the efficiency by mapping the original high-dimensional data into low-dimensional binary strings, which can significantly reduce the computational costs. Hashing learning is to learn the binary hash codes representation of original data, and to maintain the neighbor relationship in the original space, i.e., to maintain the similarity of the original data. Specifically, each sample is encoded by a compact binary string, so that two similar samples in the original feature space should remain similarity in compact binary space. Currently, hashing learning-based methods can be divided into unimodal hashing learning [18, 19, 20, 21, 22, 23, 24, 25, 26, 27], and cross-modal hashing learning [28, 29, 30, 31, 32, 33, 34, 35].

2.2.1 Unimodal hashing learning

Unimodal hashing learning can be generally divided into data-independent hashing methods and data-dependent hashing methods. As the most famous data-independent method, the main idea of Locality Sensitive Hashing (LSH) [18] is that the neighbor data in high-dimensional space can still maintain the neighboring relationship by mapping into hashing space. However, LSH uses random projection matrix to generate hash functions, which cannot represent the original data well. In contrast, data-dependent methods learn hash codes through the relationship between original data, which can generate hash functions more accurately compared with data-independent methods.

According to whether use label information in training process, data-dependent hashing methods can be roughly divided into unsupervised hashing, semi-supervised hashing and supervised hashing. Spectral Hashing (SH) [19] leverages Principal Component Analysis (PCA) [20] to calculate and order the eigenvalues of each principal component direction of image data. On the basis of SH, Anchor Graph Hashing (AGH) [21] uses the neighbor graph between the data clustering center and each instance to replace the original neighbor graph, and uses neighbor graph to replace the original neighbor matrix. Discrete Graph Hashing (DGH) [22] solves the discrete constrains by directly establishing anchor graph and the relaxation constrains. These methods are all unsupervised hashing algorithms by using feature vectors or anchor graphs to solve hash codes. The advantage of unsupervised hashing algorithms is that it reduces the training time, but the retrieval results are not satisfactory enough because the label information is not been used. Semi-supervised hashing methods use both original data information and a small amount of completed label information to generate hash codes. Semi-supervised hashing methods reduce the training time because they do not use all label information, while maintaining the relationship of the original data. Semi-Supervised Hashing (SSH) [23] combines metric similarity and semantic similarity, uses singular value decomposition to establish semi-supervised hashing function, and obtains non-orthogonal and orthogonal solutions of the coefficient matrix. Another one called Semi-Supervised Discriminant Hashing (SSDH) [24] uses unlabeled data for regularization, and uses labeled data to distinguish different categories associated with binary codes to avoid overfitting. Although semi-supervised methods have achieved better results than unsupervised methods, since the original data features cannot fully represent the semantic correlation between different data, complete label information appears to be significant. Under these circumstances, many label-based supervised hashing methods are more used in retrieval. Kernel-based Supervised Hashing (KSH) [25] leverages inner product of vectors to generate the kernel-base hashing function. FastHash [26] solves hashing codes by the classic graph cuts algorithm. Supervised Discrete Hashing (SDH) [27] maps the hash codes to the label information and solves hash codes bit-by-bit under discrete constraints using discrete cyclic coordinate descent (DCC) algorithm proposed in [27].

2.2.2 Cross-modal hashing learning

Cross-modal hashing learning methods are based on unimodal hashing learning to find the correlation between different modalities and implement the retrieval process. Specifically, cross-modal hashing retrieval methods first extract effective features from different modalities by using their own feature extraction methods. Each original high-dimensional feature is then projected into a low-dimensional Hamming space. Finally query data by ordering the “Top n” nearest neighbors in this low-dimensional Hamming space.

Cross-modal hashing methods can be divided into unsupervised cross-modal hashing and supervised cross-modal hashing according to whether use the label information. Unsupervised cross-modal hashing methods mainly refer to different modalities data having similarities in feature representation, and they are considered to have the same semantic information. Typical unsupervised cross-modal hashing methods are Cross-View Hashing (CVH) [28], Inter-Media Hashing (IMH) [29], and Collective Matrix Factorization Hashing (CMFH) [31], etc. CVH [28] extends from unimodal method SH [19], which is essentially to construct the objective function by minimizing the Euclidean distance with weights, and obtain approximate hash codes by mapping the nearest neighbor instances. IMH [29] maintains the consistency of hash function between and within modalities, i.e., the similar data in their original different modalities are have similar hash codes in cross-modal retrieval, and neighbor data in same modal are still similar in Hamming space. CMFH [31] assumes that hash codes of different modalities in common latent Hamming subspace are consistent, and uses cooperative matrix factorization method to find the potential representation of each modal. Another supervised cross-modal hashing models learn common subspaces by making full use of label information. On account of the category information is more distinguished in cross-modal data and can better represent original information, the supervised methods usually have higher retrieval performance. The classic supervised cross-modal hashing algorithms are Semantic Correlation Maximization (SCM) [33], Semantics-Preserving Hashing (SePH) [34], Double Alignment Based Hashing (DASH) [30], and Generalized Semantic Preserving Hashing (GSPH) [35]. SCM [33] is a linear cross-modal hashing method which using learned hash codes to reconstruct the similarity matrix, and using label information to improve the retrieval performance. SePH [34] first transforms the semantic matrix into a probability distribution and makes it as close as possible by minimizing the Kullback-Leibler divergence. Then uses a logistic regression with a kernel to learn the non-linear hash function for each modal. Because the first step of the training process is so complicated, it takes much longer training time than other comparison methods [34]. DASH [30] first uses unimodal hash methods to obtain the hash codes of different modalities, then directly uses linear regression model to learn the hash function of each modal. GSPH [35] first learns the optimal hash codes of each modal, while the hash codes maintain the semantic similarity between different modalities. Then learns the hash function that maps original data to binary space.

Although cross-modal hashing methods have made some progress, there still have some unsolved issues. (1) How to effectively find the correlations between heterogenous data; (2) How to efficiently learn hash functions and hash codes; (3) How to reduce the training time of cross-modal hashing retrieval. In this paper, we propose CVHH to obtain hash functions more efficiently and also reduce the computational costs. The specific implementation and optimization process is in Section 3.

3. The proposed CVHH method

In this section, we describe the algorithm analysis and optimization process of CVHH in detail. First we construct hierarchical structure to obtain the representative image-text pairs. Then we use these pairs to learn the hash functions by variable-length hashing model. Finally, we implement the cross-modal retrieval by ordering the similarity of hash codes in common Hamming space.

3.1 Notations

Suppose we have a set of $n$ bimodal training instances including image-text pairs $Z=(X,Y)$ . $X=\{x_{1}^{T},x_{2}^{T},\ldots,x_{g}^{T},x_{g+1}^{T},\ldots,x_{n}^{T}\}\in R^{% d_{1}\times n}$ is image set, and $Y=\{y_{1}^{T},y_{2}^{T},\ldots,y_{g}^{T},y_{g+1}^{T},\ldots,y_{n}^{T}\}\in R^{% d_{2}\times n}$ is text set, where $d_{1}$ and $d_{2}$ are feature dimensions of image data and text data, and $n$ is the size of image-text pairs. We set $X^{G}=\{x_{1}^{T},x_{2}^{T},\ldots,x_{g}^{T}\}$ as $g$ training images, and $X^{H}=\{x_{g+1}^{T},x_{g+2}^{T},\ldots,x_{n}^{T}\}$ represents $(n-g)$ testing images. Similarly, $Y^{G}=\{y_{1}^{T},y_{2}^{T},\ldots,y_{g}^{T}\}$ as $g$ training texts, and $Y^{H}=\{y_{g+1}^{T},y_{g+2}^{T},\ldots,y_{n}^{T}\}$ represents $(n-g)$ testing texts. We let $Z_{G}=(X_{G},Y_{G})$ to represent the training image-text pairs. For label matrix $L=\{l_{1},l_{2},\ldots,l_{g},l_{g+1},\ldots,l_{n}\}\in R^{d_{l}\times n}$ , we set $L_{G}=\{l_{1},l_{2},\ldots,l_{g}\}\in R^{d_{l}\times g}$ as label of training set, and $L_{H}=\{l_{g+1},l_{g+2},\ldots,l_{n}\}\in R^{d_{l}\times(l-g)}$ as label of testing set. $G^{[0]}({U^{[0]},V^{[0]}})$ as a neighbor graph of training image-text pairs, where $U^{[0]}$ represents g-pair instances of training set, and $V^{[0]}=V_{X}^{[0]}+V_{Y}^{[0]}$ represents similarity matrix between each pair. First we find the representative $m$ image-text pairs $U^{[M]}$ from $g$ training instances, $S_{ij}$ is the similarity matrix constructed by label matrix $L_{M}$ of represent $m$ data. Then we get $X_{M}\in R^{{d_{1}}\times{m}}$ and $Y_{M}\in R^{{d_{2}}\times{m}}$ to respectively represent original image data and text data, ${B_{X}}\in R^{{q_{1}}\times{m}}$ , ${B_{Y}}\in R^{{q_{2}}\times{m}}$ are corresponding hash codes of image and text modalities respectively, where $q_{1}$ and $q_{2}$ are the image and text length of hash codes. $P_{X}$ , $P_{Y}$ are projection matrix of image and text sets, separately, and $W$ is correlation matrix of two modalities.

3.2 CVHH retrieval algorithm

This section we focus on the implementation process of CVHH. First, we combine the manifold learning hypothesis and algebraic multigrid (AMG) method [36, 37] to construct the hierarchical structure of entire training set, which can reduce the size of instances that actually participate in hash training process. Then we construct cross-modal variable-length hashing model to accommodate the different modalities in different length of hash codes. Finally we optimize the objective function and obtain the hash functions of each modal. Figure 1 shows the framework of our proposed CVHH.

Figure 1.

The framework of proposed CVHH. First construct hierarchical structure and select the top-layer representative image-text pairs, then project top-layer pairs to the common Hamming space, finally generate the entire hash codes by similarity transfer matrix.

Manifold learning assumes that the data we observe is actually mapped from a low-dimensional manifold to a higher-dimensional space. By means of manifold learning, we can get the manifold distribution of data in the original space more accurately. In the process of constructing the hierarchical model, we first assume that the image-text pairs in training set are distributed on a manifold structure, and their local neighbor pairs have similar features. In each local structure of current layer, we select representative pairs as original data for the next layer. According to the same way, we can construct a bottom-up pyramid hierarchical structure for training set. The top layer is the most representative pairs with great differences between each pair. The bottom layer are all instances of training set. Figure 2 shows the overall flow of CVHH algorithm.

Figure 2.

The overall flow chart of CVHH, example of using a new image to retrieve similar images or texts.

The challenge of constructing hierarchy structure is how to choose representative pairs strongly connected to the unselected pairs, which can ensure that the representative pairs can substitute all unselected pairs. The specific process is as follows. First, the bottom layer is the entire training set. We first construct a k-nearest neighbor graph based on the bottom layer, as layer 0, and calculate the weight matrix of layer 0. Then build a strongly-connected graph to select the data from bottom layer which are strongly connected to unselected data. Thus, each representative data can represent entire data in this layer. We store the selected data into layer 1, then we construct a new weight matrix based on layer 1, and new representative data are selected as layer 2 according to the strongly-connected graph by layer 1, and so on. Finally, we select the top layer data as the representative data for the entire training set.

First construct a k-nearest neighbor graph $G^{[0]}({U^{[0]},V^{[0]}})$ by image-text pairs, $U^{[0]}$ represents $g$ -pair image-text data from $Z_{G}$ , and $V^{[0]}=V_{X}^{[0]}+V_{Y}^{[0]}$ represents weight matrix between each pair of image and text. Where affinity matrix $V_{X}^{[0]}={[V_{X_{ij}}^{[0]}]}_{g\times g}$ represents the similarity between paired images $x_{i}$ and $x_{j}$ , which is defined as follows:

$\displaystyle{V_{X_{ij}}}^{[0]}=\left\{\begin{array}[]{ll}\exp({\|x_{i}-x_{j}% \|}_{2}/\sigma_{1}^{2})+(l_{i}\odot l_{j}),&x_{j}\in\textit{knn}(x_{i})\mbox{ % or }x_{i}\in\textit{knn}(x_{j})\\ l_{i}\odot l_{j},&\mbox{otherwise }\\ \end{array}\right.$ (1)

where $\sigma_{1}$ is the kernel function parameter, and we choose $\sigma_{1}$ a the average of all Euclidean distance values. $\textit{knn}(x_{i})$ and $\textit{knn}(x_{j})$ represent the k-nearest neighbor sets of the images $x_{i}$ and $x_{j}$ , respectively. We utilize the local manifold structure and label information to represent the similarity between paired image instances as $V_{X_{ij}}$ . $(l_{i}\odot l_{j})$ means if $x_{i}$ and $x_{j}$ have the same label, then it takes 1, if they have different labels takes 0. Similarly, the similarity between paired texts $y_{i}$ and $y_{j}$ are defined as follows:

$\displaystyle{V_{Y_{ij}}}^{[0]}=\left\{\begin{array}[]{ll}\exp({\|y_{i}-y_{j}% \|}_{2}/\sigma_{2}^{2})+(l_{i}\odot l_{j}),&y_{j}\in\textit{knn}(y_{i})\mbox{ % or }y_{i}\in\textit{knn}(y_{j})\\ l_{i}\odot l_{j},&\mbox{otherwise }\\ \end{array}\right.$ (2)

where $\textit{knn}(y_{i})$ and $\textit{knn}(y_{j})$ respectively represent k-nearest neighbor sets of texts $y_{i}$ and $y_{j}$ . We use $V^{[s]}=V_{X}^{[s]}+V_{Y}^{[s]}$ to represent the affinity matrix of image-text pairs in layer $s$ , where ${s=1,2,\ldots,M}$ , and $M$ is the number of layer. According to literatures [36], we construct the strongly-connected graph matrix, which is defined as:

$\displaystyle{V_{ij}}\geqslant\theta\max_{k\neq i}\{V_{ik}\},0<\theta\leqslant 1,$ (3)

where $\theta$ is a strong connection threshold, generally let $\theta=0.99$ , which indicate the strength of the connection between two instances. When Eq. (3) is satisfied, the image-text pair $z_{i}$ is selected as representative pair if the degree of its node is larger than $z_{j}$ , otherwise choose $z_{j}$ as representative pair. Similarly, we can select representative pairs from layer 0 as layer 1 data, and continue until layer $s$ , where ${s=1,2,\ldots,M}$ , $M$ is the number of layers. To directly establish the connection between two layers, we construct similarity transfer matrix applied in [37]. We can directly obtain the matrix $V^{[s]}$ by $V^{[s-1]}$ using similarity transfer matrix $Q^{[s-1]}$ , and $Q^{[s-1]}$ is defined as:

$\displaystyle\left\{\begin{array}[]{l}{Q_{ik}}^{[s-1]}=\frac{{V_{ik}}^{[s-1]}}% {{\sum_{k}}{V_{ik}}^{[s-1]}},x_{i}\notin U^{[s]},x_{k}\in U^{[s]}\\ \left\{\begin{array}[]{l}{Q_{ii}}^{[s-1]}=1\\ {Q_{ij}}^{[s-1]}=0\\ \end{array}\right.,x_{i}\in U^{[s]},j\neq i\\ \end{array}\right..$ (4)

Similarity transfer matrix can transfer similarities between any two adjacent layers. Therefore, the affinity matrix $V^{[s]}$ of each layer can be calculated by:

$\displaystyle V^{[s]}=Q^{[s-1]T}\ldots Q^{[1]T}Q^{[0]T}V^{[0]}Q^{[0]}Q^{[1]}% \ldots Q^{[s-1]}.$ (5)

We now have a hierarchical structure for the entire training set, and also obtain representative pairs for each layer. So after debugging and continuing the above, we can select the representative pairs of data from appropriate layers. The complete process can be summarized as Algorithm 1.

Construction of Hierarchical StructureImage-text pairs $Z_{G}=\{{z_{1}^{T}},{z_{2}^{T}},\ldots,{z_{g}^{T}}\}$ , the number of layers $M$ Representative image-text pairs of each layer $\{{V^{[1]}},{V^{[2]}},\ldots,{V^{[M]}}\}$ Compute knn neighbor graph ${G^{[0]}}({U^{[0]}},{V^{[0]}})$

Let $s=1$

${s=M}$ Select representative pairs from layer $s-1$ as layer $s$ via Eq. (3).

Calculate similarity transfer matrix ${Q^{[s-1]}}$ via Eq. (4).

Update affinity matrix ${V^{[s]}}$ via Eq. (5).

$s=s+1$ . $\{{V^{[1]}},{V^{[2]}},\ldots,{V^{[M]}}\}$

Through Algorithm 1, we obtain representative pairs and introduce them into cross-modal variable-length hashing retrieval process. $V^{[M]}$ represents selected top layer pairs, $X_{M}\in R^{{d_{1}}\times{m}}$ and $Y_{M}\in R^{{d_{2}}\times{m}}$ represent the selected image and text respectively, ${B_{X}}\in R^{{q_{1}}\times{m}}$ , ${B_{Y}}\in R^{{q_{2}}\times{m}}$ are corresponding hashing codes, $P_{X}$ , $P_{Y}$ are projection matrices, and $W$ is correlation matrix between two modalities. In order to make full use of label information, we define the similarity matrix $S_{ij}$ by label matrix $L_{M}$ as:

$\displaystyle S_{ij}=\left\{\begin{array}[]{ll}1,&l_{x}^{i}=l_{y}^{j}\\ 0,&l_{x}^{i}\neq l_{y}^{j}\\ \end{array}\right.$ (6)

We assume that there has a common latent semantic space $K$ according to [38], which can directly query different modalities by projecting original data into $K$ . The hash codes of each modal are projected into $K$ as follows:

$\displaystyle B_{X}\xrightarrow{W_{1}}K_{X},B_{Y}\xrightarrow{W_{2}}K_{Y}.$ (7)

The similarity between images and texts in $K$ can be obtained by inner product:

$\displaystyle{K_{X}^{T}}{K_{Y}^{T}}={({W_{1}}{B_{X}})^{T}}{({W_{2}}{B_{Y}})}={% B_{X}^{T}}{W_{1}^{T}}{W_{2}}{B_{Y}}.$ (8)

Let $W={W_{1}^{T}}{W_{2}}$ , then $W$ is the correlation matrix between different modalities. The overall objective function is as follows:

$\displaystyle\min_{{B_{X}},{B_{Y}},W,{P_{X}},{P_{Y}}}{\|{B_{X}}-{P_{X}X_{M}}\|% }_{F}^{2}+{\|{B_{Y}}-{P_{Y}Y_{M}}\|}_{F}^{2}+{\|S-{B_{X}^{T}}W{B_{Y}}\|}_{F}^{2}$ (9) $\displaystyle s.t.{B_{X}}\in{\{-1,+1\}}^{{q_{1}}\times m},{B_{Y}}\in{\{-1,+1\}% }^{{q_{2}}\times m}.$

The first two terms project original images and texts data into their own low-dimensional binary spaces, respectively. The last one represents the semantic similarity of original data in cross-modal latent space. Then we can optimize the objective function and obtain the projection matrices $P_{X}$ , $P_{Y}$ , hash codes $B_{X}$ , $B_{Y}$ and correlation matrix $W$ simultaneously.

Since the objective function is non-convex, we first solve for one variable by fixing others, and then iteratively solve all variables until the objective function converges.

(1) Fix other variables and solve $P_{X}$ , $P_{Y}$ , separately

The objective function can be simplified as follows:

$\displaystyle\min_{P_{X}}{\|{B_{X}}-{P_{X}X_{M}}\|}_{F}^{2},\min_{P_{Y}}{\|{B_% {Y}}-{P_{Y}Y_{M}}\|}_{F}^{2}.$ (10)

Therefore, $P_{X}$ , $P_{Y}$ are respectively calculated by the regression function:

$\displaystyle P_{X}={B_{X}}{X_{M}^{T}}{(X_{M}X_{M}^{T})}^{-1},P_{Y}={B_{Y}}{Y_% {M}^{T}}{(Y_{M}Y_{M}^{T})}^{-1}.$ (11)

(2) Fix other variables and solve $W$

The objective function can be simplified as follows:

$\displaystyle\min_{W}{\|S-B_{X}^{T}WB_{Y}\|}_{F}^{2}.$ (12)

Then correlation matrix $W$ can be calculated as:

$\displaystyle W={(B_{X}B_{X}^{T})}^{-1}B_{X}SB_{Y}^{T}{(B_{Y}B_{Y}^{T})}^{-1}.$ (13)

(3) Fix other variables and solve $B_{X}$

The objective function can be simplified as follows:

$\displaystyle\min_{B_{X}}{\|{B_{X}}-{P_{X}X_{M}}\|}_{F}^{2}+{\|S-B_{X}^{T}WB_{% Y}\|}_{F}^{2},s.t.B_{X}\in{\{-1,+1\}}^{{q_{1}}\times m}.$ (14)

Due to the binary constraint problem, we learn the binary codes $B_{X}$ by discrete cyclic coordinate descent (DCC) method bit by bit according to reference [27], and deform the function into:

$\displaystyle\min_{B_{X}}{\|D{B_{X}}\|}_{F}^{2}-2Tr(B_{X}^{T}E),s.t.B_{X}\in{% \{-1,+1\}}^{{q_{1}}\times m},$ (15)

where ${D}={B_{Y}^{T}}{W^{T}}$ and $E=(W{B_{Y}}{S^{T}}+{P_{X}}X)$ . Let $b^{T}$ be the $i^{\text{th}}$ row of $B_{X}$ and $B^{\prime}_{X}$ is the matrix of $B_{X}$ excluding $b^{T}$ . Similarly, let $e^{T}$ be the $i^{\text{th}}$ row of matrix $E$ , $E^{\prime}$ is the matrix of $E$ excluding $e^{T}$ , let $d^{T}$ be the $i^{\text{th}}$ row of $D$ and $D^{\prime}$ is the matrix of $D$ excluding $d$ . Then we get:

$\displaystyle b=\textit{sgn}(e-B^{\prime}_{X}D^{\prime T}d).$ (16)

We can see from Eq. (16) that, every bit in $b$ is learned by $q_{1}-1$ bits $B^{\prime}_{X}$ which is pre-learned, i.e., we can generate hash codes $B_{X}$ by iteratively update each bit, until it converges.

(4) Fix other variables and solve $B_{Y}$

The solution of $B_{Y}$ is similar to $B_{X}$ , please refer to step (3).

In summary, the overall algorithm can be summarized as Algorithm 2.

Cross-modal Variable-length Hashing Based on HierarchyThe training set $V^{[M]}=(X_{M},Y_{M})$ , label matrix $L_{M}$ Variable-length hashing codes $B_{X}$ , $B_{Y}$ , projection matrix $P_{X}$ , $P_{Y}$ , correlation matrix $W$ Initialize correlation matrix $W$ as one Initialize variable-length hashing codes $B_{X}$ , $B_{Y}$ both as zero Initialize iterative control parameter $\textit{iter}=0$ Construct the semantic similarity matrix $S$ by Eq. (6) Eq. (10) tends to converge Update the projection matrix $P_{X}$ , $P_{Y}$ via Eq. (11) Update the correlation matrix $W$ via Eq. (13) Calculate $B_{X}$ , $B_{Y}$ via Eq. (16) $\textit{iter}=\textit{iter}+1$ $B_{X}$ , $B_{Y}$ , $P_{X}$ , $P_{Y}$ , $W$

In the query phase, assume there is a new image instance $x^{\prime}$ , the hash codes of $x^{\prime}$ in text space can be represented as $b_{y^{\prime}}=\textit{sgn}(WP_{X}x^{\prime})$ . Then we construct Hamming distance matrix $H$ by $b_{y^{\prime}}$ and the top layer text data $B_{Y}$ . Next we use similarity transfer matrix $Q^{[s]}$ to get Hamming distance matrix $H^{[0]}$ by $H^{[0]}=HQ^{[s-1]T}\ldots Q^{[1]T}Q^{[0]T}$ by $b_{y^{\prime}}$ and the entire text training set in layer 0. Finally, ascending ordering the distances of $H^{[0]}$ and obtain the top $n$ required texts. If we query a new text instance $y^{\prime}$ , then the hash codes of $y^{\prime}$ in image space can be obtained by $b_{x^{\prime}}=\textit{sgn}(W^{T}P_{Y}y^{\prime})$ .

3.3 Time complexity analysis

The complexity analysis of CVHH is as follows. The hierarchy process can be processed offline, so the main time cost is cross-modal hashing retrieval process, i.e., calculate and update projection matrices $P_{X}$ , $P_{Y}$ , hash codes $B_{X}$ , $B_{Y}$ and correlation matrix $W$ . The corresponding time complexity is $O(d^{2}qm)$ , $O(dq^{2}m)$ , $O(q^{2}m^{2})$ , respectively. Let $T$ be the number of iterations, so the time complexity of the entire algorithm is $O((d^{2}+qm+dq)\textit{qmT})$ , where $d=\max\{d_{1},d_{2}\}$ , $q=\max\{q_{1},q_{2}\}$ , $d_{1}$ , $q_{1}$ are the dimensions of image feature and the length of image hash codes, $d_{2}$ , $q_{2}$ are the dimensions of text feature and the length of text hash codes, respectively, and $m$ is the number of instances. The query time complexity for a new instance is $O(dq)$ .

4. Experiments

In this section, we evaluate the effectiveness of CVHH, and compare CVHH with several state-of-the-art cross-modal retrieval algorithms. The evaluation metrics are precision-recall curves and MAP values, indicating that the performance of CVHH is superior to other comparison methods.

4.1 Datasets and features

In order to verify the performance of CVHH, we adopt three benchmark large-scale datasets, Wiki [38], MIRFlickr [39] and NUS-WIDE [40], which commonly used for cross-modal retrieval methods [31, 32, 33, 34, 35]. These datasets are all provide bimodal data, images and texts. The statistics of the three datasets are summarized in Table 1.

Wiki [38] is a common bimodal dataset contains 2866 instances, which collected from Wikipedia website. All instances are separated by 10 classes and each instance has its own semantic label. For each instance, its image is represented as a 128-dimensional SIFT feature vector, and its text is represented by a 10-dimensional feature vector generated by Latent Dirichlet Allocation (LDA). Wiki was initially split into 2173 instances as training set and 693 instances as test set.

MIRFlickr [39] is a public large-scale dataset collected from Flickr website which includes 25000 instances. each instance has several correlative labels from 24 classes. According to [31], we select 16738 image-text pairs to be used in our experiments. We randomly select 5% instances as the testing set and others as the training set. Each image is represented as a 150-dimensional edge histogram and each text is represented as a 500-dimensional feature vector generated from PCA on its index tagging vector.

NUS-WIDE [40] is a real-world large-scale dataset which contains 269648 instances, and every instance is annotated by at least one label of 81 ground truth semantic categories. According to the settings in [30], we choose the most common 10 categories and totally 186577 instances for our experiments. Each image is represented as a 500-dimensional Bag-of-Words feature vector and each text is represented as a 1000-dimensional binary tagging vector of the most frequent labels. We randomly select 1% instances as the testing set and others as the training set.

Table 1
Statistics of three benchmark datasets

	Wiki [38]	MIRFlickr [39]	NUS-WIDE [40]
Training set size	2173	15902	186577
Query set size	693	836	1866
Number of labels	10	24	10
Type of labels	Uni-label	Multi-label	Multi-label
Dimension of image feature	128	150	500
Dimension of text feature	10	500	1000

4.2 Evaluation metrics and comparison methods

This section mainly describe the evaluation matric and comparison methods of our experiments. We focus on two typical tasks: 1) Img2Txt use images to query texts; 2) Txt2Img use texts to query images. During the query phase, we sort the distances of Hamming distance matrix in ascending order, and return the top 10 instances as the results. Furthermore, we adopt two common-used performance measurements, mean average precision (MAP) value and precision-recall curve to evaluate our CVHH with other comparison methods. For these two measurements, higher values indicate better performance. The MAP value is defined as:

$\displaystyle\textit{MAP}=\frac{1}{|F|}\sum_{i=1}^{|F|}\frac{1}{G}\sum_{j=1}^{% G}P(ij),$ (17)

where $|F|$ is the size of query set, $G$ is the number of ground truth instances of training set in $i^{\text{th}}$ query, $P_{(ij)}$ is the precision of top $j$ retrieved instance in $i^{\text{th}}$ query, which we set $j=10$ . Furthermore, we test all methods on different lengths of hash codes, $\{16,32,64,128\}$ , and the iterative parameter is set to $\textit{iter}=5$ .

We choose several typical cross-modal retrieval algorithms as our comparison methods, namely CCA [32], SCM [33], SePH [34] and GSPH [35]. Our experiments use the codes published by the authors with the same parameter settings as the original papers. In original articles [34, 35], both have two methods to learn hash functions in SePH and GSPH models: (1) SePH ${}_{\textit{rnd}}$ and GSPH ${}_{\textit{rnd}}$ randomly select instances to train the hash functions; (2) SePH ${}_{\textit{knn}}$ and GSPH ${}_{\textit{knn}}$ by clustering to select instances to train the hash functions. The original experiments show that the performances by these two methods are basically the same. Therefore, we select the first way to train the hash functions of SePH and GSPH models. There also have two different methods in SCM model, SCM ${}_{\textit{seq}}$ and SCM ${}_{\textit{orth}}$ . The original experiments in article [33] show that the former one is generally superior to the latter one, so we choose SCM ${}_{\textit{seq}}$ as one of our comparison method.

Figure 3.

The precision-recall curves of CVHH and other comparison methods in Img2Txt and Txt2Img in 64-bit hashing length. Subgraphs (a), (b) are the results on Wiki, (c), (d) are the results on MIRFlickr, and (e), (f) are the results on NUS-WIDE.

4.3 Experimental results of CVHH with comparison methods

In this section, we compare CVHH with four typical cross-modal retrieval methods on three benchmark datasets. For comparison, the experimental results show the retrieval performance of CVHH and all comparison methods under 64-bit length of hash coding.

The overall precision-recall curves are shown in Fig. 3. Subgraph (a), (b) show the experimental results on Wiki dataset, (c), (d) show the results on MIRFlickr, and (e), (f) show the results on NUS-WIDE. It can be seen that the performance of CVHH is generally better than other comparison methods, for both Img2Txt and Txt2Img. Figure 3 also shows that CVHH improving significantly over comparison methods on MIRFlickr and NUS-WIDE, probably because CVHH is more suitable for multi-label datasets than other comparison methods.

Tables 2 and 3 show the MAP values of Img2Txt and Txt2Img, respectively. We mark the maximum MAP values for each column as black. It can be seen that our proposed CVHH has the highest retrieval performance among all comparison methods in both Img2Txt and Txt2Img tasks, which proves our proposed CVHH has higher retrieval accuracy. And we can also observe that with the increase of hash codes length, the MAP values of all methods have generally increased, which represent the retrieval performance will vary with the length of hash codes.

Table 2
MAP values of CVHH and comparison methods in Img2Txt with different hashing length on three datasets

Methods	Wiki [38]				MIRFlickr [39]				NUS-WIDE [40]
	16-bit	32-bit	64-bit	128-bit	16-bit	32-bit	64-bit	128-bit	16-bit	32-bit	64-bit	128-bit
CCA	0.184	0.170	0.150	0.140	0.579	0.574	0.571	0.568	0.373	0.366	0.361	0.358
SCM	0.234	0.241	0.246	0.257	0.610	0.631	0.647	0.641	0.501	0.542	0.553	0.551
SEPH	0.276	0.296	0.300	0.313	0.671	0.652	0.681	0.648	0.560	0.578	0.582	0.581
GSPH	0.272	0.290	0.305	0.307	0.665	0.676	0.687	0.692	0.571	0.582	0.585	0.593
CVHH	0.303	0.346	0.344	0.358	0.777	0.776	0.774	0.772	0.641	0.642	0.647	0.650

Table 3

MAP values of CVHH and comparison methods in Txt2Img with different hashing length on three datasets

Methods	Wiki [38]				MIRFlickr [39]				NUS-WIDE [40]
	16-bit	32-bit	64-bit	128-bit	16-bit	32-bit	64-bit	128-bit	16-bit	32-bit	64-bit	128-bit
CCA	0.168	0.159	0.154	0.150	0.579	0.574	0.572	0.570	0.371	0.365	0.362	0.360
SCM	0.226	0.246	0.249	0.253	0.615	0.624	0.628	0.631	0.535	0.540	0.542	0.539
SEPH	0.631	0.658	0.659	0.669	0.710	0.744	0.727	0.744	0.683	0.695	0.693	0.708
GSPH	0.645	0.663	0.671	0.674	0.726	0.742	0.748	0.764	0.681	0.697	0.686	0.714
CVHH	0.686	0.745	0.752	0.763	0.797	0.808	0.813	0.816	0.743	0.773	0.782	0.785

To visually compare the training time of CVHH with other comparison methods, Table 4 shows the training time of several methods on the three large data sets with 16-bit, 32-bit and 64-bit hash coding lengths. In Table 4, the training time of all methods increases with the increase of the hash code length, and even the training time of many algorithms increases exponentially. It can also be seen from Table 4 that the training time of two comparison methods SePH and GSPH is costs too much, which because these two methods need to construct pairwise similarity matrices. Compared with other algorithms, the computational costs are quite high. At the same time, it can be seen that compared with other comparison algorithms, CVHH has an absolute advantage in training time and is in an advantageous position in large-scale cross-modal data set.

Table 4

The training time (seconds) of CVHH with all comparison methods in 16-bit, 32-bit and 64-bit hashing lengths on three datasets

Methods	Wiki [38]			MIRFlickr [39]			NUS-WIDE [40]
	16-bit	32-bit	64-bit	16-bit	32-bit	64-bit	16-bit	32-bit	64-bit
CCA	0.23	0.32	0.39	1.89	2.44	3.40	4.93	8.66	16.55
SCM	0.20	0.38	0.67	2.95	3.39	4.50	10.57	21.35	39.12
SePH	189.06	235.39	275.78	701.45	796.53	866.68	1566.78	3393.67	3729.01
GSPH	159.25	189.05	234.33	621.37	795.30	871.58	2936.75	3587.61	3895.20
CVHH	0.12	0.17	0.43	1.09	1.85	6.90	2.89	4.20	10.01

Table 5

Comparison of MAP values on every layer of CVHH for text query image and image query text in three datasets with 64-bit hashing length

	Hierarchy number	Size of training set	Training time (s)	MAP of Img2Txt	MAP of Txt2Img
Wiki [38]	0	2173	5.071	0.364	0.760
	1	1260	1.365	0.364	0.762
	2	759	0.852	0.346	0.755
MIRFlickr [39]	0	15902	171.839	0.776	0.809
	1	9132	74.050	0.779	0.815
	2	5325	36.703	0.775	0.798
NUS-WIDE [40]	0	10000	98.099	0.653	0.807
	1	5548	50.595	0.656	0.786
	2	3130	17.204	0.641	0.743

Figure 4.

The precision-recall curves of CVHH in Img2Txt and Txt2Img with different hashing length. Subgraphs (a), (b) are the results on Wiki, (c), (d) are the results on MIRFlickr, and (e), (f) are the results on NUS-WIDE.

Figure 4 shows the experimental results of our proposed CVHH in different length of hash codes (image hashing length * text hashing length). Subgraphs (a), (b) show the retrieval performance of different length of hash codes on Wiki dataset, (c), (d) show the performance on MIRFlickr, and (e), (f) show the performance on NUS-WIDE. In order to show the variation trend of experimental results in different length of hash codes, the colors of curves from 16*16 to 128*128 gradually change from dark blue, light blue, light red to dark red. In general, with the increasing of hash codes length, the query results become more effective, especially in the subgraphs (b), (d) and (f) in Fig. 4. The experimental results also show that when the codes length is from 16*16 to 128*128, the query performances is gradually improving.

4.4 Evaluation indicators of CVHH on each layer

To demonstrate the validity of the hierarchical structure, Table 4 shows the experimental results of every layer on the above three datasets. The evaluation indicators are the number of hierarchical layers, the size of training sets, training time and MAP values. In order to reduce the computational costs, we take 10000 instances as layer 0 on NUS-WIDE.

The results in Table 4 fully demonstrate that with the increase of the number of layers, the training time is greatly reduced while the retrieval performance is guaranteed. As an example, we can see that the MAP values on layer 1 show a competitive advantage in both Img2Txt and Txt2Img on Wiki dataset. However, the MAP values on layer 2 is significantly reduced compared to layer 1. So we select layer 1 data to participate in subsequent cross-modal variable-length hashing. Similarly, we all select the layer 1 as the training set in MIRFlickr and NUS-WIDE. In the query phase, we can obtain the hash codes of entire training set, thus ensuring the authenticity of the experimental results.

5. Conclusion

In this paper, we propose a novel cross-modal retrieval method, called Cross-modal Variable-length Hashing based on Hierarchy (CVHH). First we construct the hierarchical structure based on training set, and select the representative image-text pairs from optimal layer. Then we build the variable-length hashing structure to project high-dimensional original data into low-dimensional hashing space. Finally we compare the distances between queried instances with the entire training set by similarity transfer matrix, and return the top 10 instances as final query results. The experiments show that our proposed CVHH can greatly reduce the training time, and also improve the retrieval efficiency. In addition, the performance of CVHH is superior to other typical comparison methods. Furthermore, we will try to apply this method to multi-modal domain in future researches.

Footnotes

Acknowledgments

The authors would like to thank the anonymous reviewers for their help. This work was supported by the National Natural Science Foundation of China (Grant No. 62076044), Chongqing Research Program of Basic Research and Frontier Technology (Grant No. cstc2019jcyj-zdxm0011), and The Postgraduate Scientific Research and Innovation Project of Chongqing (Grant No. CYS18248).

References

Setti

Stapleton

Leahy

et al., Improving the efficiency of multisensory integration in older adults: audio-visual temporal discrimination training reduces susceptibility to the sound-induced flash illusion, Neuropsychologia 61 (2014), 259–268.

Deng

Dong

socher

et al., Magenet: A large-scale hierarchical image database, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2014, in: 248–255.

Feng

Wang

and Li

, Cross-modal retrieval with correspondence autoencoder, in: Proceedings of the 22nd ACM International Conference on Multimedia, ACM, 2014, pp. 7–16.

Mandal

and Biswas

, Generalized coupled dictionary learning approach with applications to cross-modal matching, IEEE Transactions on Image Processing 25(8) (2016), 3826–3837.

Tang

Wang

and Shao

, Supervised matrix factorization hashing for cross-modal retrieval, IEEE Transactions on Image Processing 25(7) (2016), 3157–3166.

Jiang

Q.Y.

and Jun

L.W.

, Deep cross-modal hashing, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3232–3240.

Wang

Yin

Wang

et al., A comprehensive survey on cross-modal retrieval, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, arXiv preprint arXiv:1607.06215, 2016.

Peng

Huang

and Zhao

, An overview of cross-media retrieval: concepts, methodologies, benchmarks, and challenges, IEEE Transactions on Circuits and Systems for Video Technology 28(9) (2017), 2372–2385.

Rasiwasia

Pereira

J.C.

Coviello

et al., A new approach to cross-modal multimedia retrieva, in: Proceedings of the 18th ACM International Conference on Multimedia, ACM, 2010, pp. 251–260.

10.

Hwang

S.J.

and Grauman

, Learning the relative importance of objects from tagged images for retrieval and cross-modal search, International Journal of Computer Vision 100(2) (2012), 134–153.

11.

Hardoon

D.R.

Szedmak

and Shawe-Taylor

, Canonical correlation analysis: an overview with application to learning methods, Neural Computation 16(12) (2004), 2639–2664.

12.

Sharma

Kumar

Daume

et al., Generalized multiview analysis: a discriminative latent space, in: 2012 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2012, pp. 2160–2167.

13.

Amal

L.H.

and Giraud

A.L.

, Cortical oscillations and sensory predictions, Trends in Cognitive Sciences 16(7) (2012), 390–398.

14.

Wang

Yang

and Meinel

, Deep semantic mapping for cross-modal retrieval, in: 2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI), IEEE, 2015, pp. 234–241.

15.

Andrew

Arora

Bilmes

et al., Deep canonical correlation analysis, in: International Conference on Machine Learning, 2013, pp. 1247–1255.

16.

Wei

Y.C.

Zhao

C.Y.

et al., Cross-modal retrieval with CNN visual features: a new baseline, IEEE Transactions on Cybernetics 47(2) (2016), 449–460.

17.

Manku

G.S.

Bawa

and Raghavan

, Symphony: Distributed Hashing in a Small World, in: USENIX Symposium on Internet Technologies and Systems, Vol. 10, 2003.

18.

Datar

Immorlica

Indyk

et al., Locality-sensitive hashing scheme based on p-stable distributions, in: Proceedings of the Twentieth Annual Symposium on Computational Geometry, ACM, 2004, pp. 253–262.

19.

Weiss

Torralba

and Fergus

, Spectral hashing, Advances in Neural Information Processing Systems, 2009, 1753–1760.

20.

Wold

Esbensen

and Geladi

, Principal component analysis, Chemometrics and Intelligent Laboratory Systems 2(1–3) (1987), 37–52.

21.

Liu

Wang

Kumar

et al., Hashing with graphs, 2011.

22.

Liu

Kumar

et al., Discrete graph hashing, Advances in Neural Information Processing Systems, 2014, 3419–3427.

23.

Liu

Wang

et al., Semi-supervised hashing for large-scale search, IEEE Transactions on Pattern Analysis and Machine Intelligence 34(12) (2012), 2393–2406.

24.

Kim

and Choi

, Semi-supervised discriminant hashing, in: 2011 IEEE 11th International Conference on Data Mining, IEEE, 2011, pp. 1122–1127.

25.

Liu

Wang

et al., Supervised hashing with kernels, in: 2012 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2012, pp. 2074–2081.

26.

Song

Dharmapurikar

Turner

et al., Fast hash table lookup using extended bloom filter: an aid to network processing, ACM SIGCOMM Computer Communication Review. ACM, 35(4) (2005), 181–192.

27.

Shen

F.M.

Shen

C.H.

Liu

et al., Supervised discrete hashing, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 37–45.

28.

Kumar

and Udupa

, Learning hash functions for cross-view similarity search, in: Twenty-Second International Joint Conference on Artificial Intelligence, 2011.

29.

Song

Yang

and Yang

, Inter-media hashing for large-scale retrieval from heterogeneous data sources, in: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, ACM, 2013, pp. 785–796.

30.

Lin

Ding

et al., A survey of sequence alignment algorithms for next-generation sequencing, Briefings in Bioinformatics 11(5) (2010), 473–483.

31.

Ding

Guo

and Zhou

, Collective matrix factorization hashing for multimodal data, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2075–2082.

32.

Rasiwasia

Pereira

J.C

Coviello

et al., A New Approach to Cross-Modal Multimedia Retrieval, in: International Conference on Multimedia. ACM, 2010.

33.

Zhang

and Li

W.J.

, Large-scale supervised multimodal hashing with semantic correlation maximization, in: Twenty-Eighth AAAI Conference on Artificial Intelligence, 2014.

34.

Lin

Ding

et al., Semantics-preserving hashing for cross-view retrieval, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3864–3872.

35.

Mandal

Chaudhury

K.N.

and Biswas

, Generalized semantic preserving hashing for n-label cross-modal retrieval, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4076–4084.

36.

Sharon

Brandt

and Basri

, Fast multiscale image segmentation, in: Proceedings IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2000 (Cat. No. PR00662), pp. 70–77.

37.

Yang

Matei

and Davis

L.S.

, Re-ranking by multi-feature fusion with diffusion for image retrieval, in: 2015 IEEE Winter Conference on Applications of Computer Vision, IEEE, 2015, pp. 572–579.

38.

Pereira

J.C.

Covirllo

Doyle

et al., On the role of correlation and abstraction in cross-modal multimedia retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence 36(3) (2013), 521–535.

39.

Huiskes

M.J.

Thomee

and Lew

M.S.

, New trends and ideas in visual concept detection: the MIR flickr retrieval evaluation initiative, in: Proceedings of the International Conference on Multimedia Information Retrieval, ACM, 2010, pp. 527–536.

40.

Chua

T.S.

Tang

Hong

et al., NUS-WIDE: a real-world web image database from National University of Singapore, in: Proceedings of the ACM International Conference on Image and Video Retrieval, ACM, Vol. 48, 2009.

Cross-modal variable-length hashing based on hierarchy

Abstract

Keywords

1. Introduction

2.1 Latent subspace learning

2.2 Hashing learning

2.2.1 Unimodal hashing learning

2.2.2 Cross-modal hashing learning

3. The proposed CVHH method

3.1 Notations

3.2 CVHH retrieval algorithm

(1) Fix other variables and solve P X , P Y , separately

(2) Fix other variables and solve W

(3) Fix other variables and solve B X

(4) Fix other variables and solve B Y

4. Experiments

4.1 Datasets and features

Table 1 Statistics of three benchmark datasets

Table 2 MAP values of CVHH and comparison methods in Img2Txt with different hashing length on three datasets

5. Conclusion

Footnotes

Acknowledgments

References

(1) Fix other variables and solve $P_{X}$ , $P_{Y}$ , separately

(2) Fix other variables and solve $W$

(3) Fix other variables and solve $B_{X}$

(4) Fix other variables and solve $B_{Y}$

Table 1
Statistics of three benchmark datasets

Table 2
MAP values of CVHH and comparison methods in Img2Txt with different hashing length on three datasets