Deep hash networks with cross-scale feature fusion for optimal binary encoding

3.1 Problem Definition

As mentioned above, deep hash networks can learn a compact high-dimensional feature representation and transform it into binary hash codes through a hash function, enabling fast image retrieval. Given N images χ = { x i } i = 1 N and corresponding one-hot semantic labels y = { y i } i = 1 N ∈ { 0 , 1 } C × N , where C is the category numbers of the whole dataset. For each training sample (x _i , y _i ), Deep Hashing learns a feature extractor ψ parameterized by ð to extract representative latent code representations h i = { h 1 h 2 . . . h k ∣ h i ∈ R } from the sample. Subsequently,the latent code h _i is quantized into a compact K-bit binary code b _i  ={ b₁b₂ . . . b _k  ∣ b _i  ∈ { ± 1 }} by the hashing function H : b i ← H ( h i ) . Our objective in this task is to learn a reliable feature extractor ψ and a robust hash function H , to cluster images of different categories into appropriately and distinctly hash positions. Given a query image x _q , we sort the hash codes of all samples in the database based on their Hamming distance and return the top N images as query results. It is reasonable to assume that the hash code b in Hamming space has similar semantic information to the representation h _i in feature space, enabling similarity matching between images. Moreover, to improve retrieval performance, we need to choose an appropriate hash code length K based on the size of the dataset and the complexity of the featurerepresentation.

3.2 Overview of CSF-Net

The CSF-Net framework proposed by us is shown in Fig. 2. Building upon the state-of-the-art image recognition model ResNet50, as in [1, 28], we introduce a dual-branch architecture, CSF-Net, to address the issue of insufficient representation of ambiguous objects in image encoding by deep learning networks. As is well-known, shallow layers of deep neural networks tend to capture fine-grained details such as edges and corners, while deeper layers are more sensitive to global and semantic information. Inspired by this, the dual-branch design of CSF-Net aims to leverage semantic context in Res4 to align and filter better local features, thus suppressing noise interference. The approach showcases a sophisticated fusion of deep learning and image retrieval techniques, resulting in a coherent and elegant solution for improving the representation of non-obvious targets in image encoding.

OPEN IN VIEWER

Fig. 2

Overall structure of proposed CSF-Net, built upon ResNet50. The multi-scale features f _global are obtained by the global branch which takes the output of Res4 block as input. The local branch, which models more local details f _local after Res3 block, uses spatial pyramid pooling (SPP) and self-attention. The CFS module takes both f _global and f _local as inputs to generate the final representation. The metric loss( L HGM ) and quantization loss( L Q )are used for optimizing the network.

As shown in Fig. 2, our global branch remains the same as the original ResNet50, but all pooling and fully connected layers after the Res4 block are removed to preserve spatial information in the feature maps. In addition, a multi-scale sliding window is introduced to capture multi-scale spatial global information by applying sliding windows of different sizes to the feature map. This allows the network to capture contextual information at multiple scales and helps to improve the overall performance of the model. The global features, abbreviated as f _g , can be expressed as: f g = cat ( F . unfold ( f 4 1 , s w 1 ) , F . unfold ( f 4 2 , s w 2 ) )(1)

In Equation (1), the feature maps f₄₁ and f₄₂ are obtained by processing the output of the Res4 layer with Conv2d, but they have different sizes. We use 3 × 3 and 5 × 5 as the window size (denoted as s _{w 1} and s _{w 2} , respectively) to apply multi-scale sliding windows on them. To enhance the ability of neural networks to extract detailed information from images and jointly extract local descriptors, we also introduced a local branch after the Res3 block, which consists of spatial pyramid pooling layers and a self-attention module. Specifically, we perform max pooling on the output feature f3 of the Res3 block at different spatial scales to obtain multiple feature maps with different spatial receptive fields. Each feature map is then flattened into a one-dimensional vector, and these vectors are concatenated along the channel dimension. By performing pooling operations at different scales on the image, a fixed-size feature representation is generated, allowing the model to capture multi-scale information of the target. This enables the model to extract relevant information from features at different scales, even when the position and size of the target vary in the image, thus maintaining effective recognition capabilities. Therefore, the spatial pyramid pooling layer contributes to enhancing the performance and robustness of the model, enabling it to adapt to targets of different scales and positions [29]. Finally, the concatenated feature map is passed to the self-attention module to further model the importance of each local feature point. The self-attention mechanism computes the similarity between input features and allocates weights accordingly. This allows the model to focus more on the areas that have the greatest impact on the results, reducing the influence of other distracting information. Since it takes into account and emphasizes the importance of meaningful information areas, it can better capture representative, distinguishable local features. This mechanism enhances the model’s ability to extract local features and contributes to improving the accuracy of the model’s prediction [16].

The global feature component f _g and the local feature tensor f _l , as two important feature sources, undergo feature fusion and elaborately designed processing through the Cross-Feature Synergy module, resulting in a compact descriptor that effectively preserves spatial information from both branches and achieves adaptive context alignment, which will be discussed in detail later. The design of the Cross-Feature Synergy module aims to fully leverage the strengths of global and local features, achieve complementarity and synergy, and better capture semantic information from images.

Our Cross-Feature Synergy (CFS) module operates as illustrated in Fig. 3. The input to CFS includes two components: the multi-scale global feature f _g based on sliding windows and the local feature f _l based on self-attention. To start, we align the local feature tensor and global feature tensor by calculating the contextual repetition of each local feature point f l ( i , j ) on the global feature f _g using matrix multiplication (torch.mm). Mathematically, this can be expressed as: f t 1 ( i , j ) = f l ( i , j ) · f g | f l ( i , j ) | 2 f l ( i , j )(2)

OPEN IN VIEWER

Fig. 3

Framework of our proposed Cross-Feature Synergy Module(CFS).

Due to the fact that global feature tensor contains more semantic contextual information, while local feature tensor contains more detailed local contextual information, redundant context in complex backgrounds may contain unnecessary semantic information. Therefore, to better extract fine-grained local features with multi-scale spatial context, we adopted a strategy of eliminating redundant features. Specifically, when extracting local information from global features, we use subtraction operation to cancel out the duplicated information with local features, thus retaining information that only appears in the global scope. This effectively separates global and local features, and extracts more valuable features. The whole process can be expressed as follows: f t 2 ( i , j ) = f l ( i , j ) - f t 1 ( i , j )(3)

By following this procedure, we obtain a tensor of size C × H × W, where each element is orthogonal to the global feature f _g . We then concatenate the C×1 vector f _gc to each element of this tensor, resulting in a new tensor of size C₀ × H × W. Finally, we aggregate this tensor into a C₀ × 1 vector. Finally, a K×1 descriptor is generated using a fully connected layer. Here, we choose to use pooling to aggregate the cascaded tensors. Through our Cross-Feature Synergistic (CFS) module processing, the extracted features align detailed local features with global spatial context, thus focusing on the key instance regions.

Metric Loss. In general, for any K-dimensional binary codes b₁ and b₂, the Hamming distance measures their similarity by counting the number of different bits between them using XOR operation. However, in practical applications, XOR operation is not differentiable and cannot be used for training neural networks with backpropagation. It is also not suitable for measuring distances between continuous floating-point features used for training. Given that Hamming distance is related to cosine similarity, we use cosine similarity to approximate the distance between continuous floating-point hash codes h₁ and h₂, and enable backpropagation. Therefore, the relationship between cosine similarity and Hamming distance can be expressed as Equation 4, where K represents the dimensionality of the hashcodes. D Ham ( b 1 , b 2 ) = ∑ i = 1 K | b 1 i - b 2 i | = 1 2 K - D cos ( h 1 , h 2 ) · K 2(4) In Equation 4,D _cos  (h₁, h₂) represents the cosine distance between h₁ and h₂, which can be expressed as: D cos ( h 1 , h 2 ) = 1 - h 1 · h 2 | h 1 | | h 2 |(5)

As is well known, the goal of metric loss is clustering, which is a distance-based learning method designed to learn a mapping function that maps input data from a high-dimensional feature space to a low-dimensional embedding space, and makes samples of the same class more compact in the embedding space, while making the distance between different class samples more distinct. This technique is widely used in the field of image retrieval and is used to optimize the performance of models. Existing techniques [30–33] typically use cosine similarity estimated in mini-batches to design the metric loss term, which can be represented as: L Metric = ∑ h 1 ∈ 𝕊 ∑ h 2 ∈ 𝕊 ′ 1 * ( y 1 = y 2 ) F + ( - cos ( h 1 , h 2 ) ) + ∑ h 1 ∈ 𝕊 ∑ h 2 ∈ 𝕊 ′ 1 * ( y 1 ≠ y 2 ) F - ( cos ( h 1 , h 2 ) )(6) In Equation 6, F₊ and F_- represent the loss functions designed for positive and negative samples, respectively, which can be estimated using contrastive-based or class-proxy-based methods to measure similarity. 𝕊 represents the latent codes of the given batch of images, and 𝕊 ′ represents the latent code set of other samples or class proxies. y _i represents the class label.

Quantization loss. The metric loss only considers the intra-class compactness and inter-class separability of continuous features in the original space, while neglecting the requirements for efficient query and easy storage. Therefore, the metric loss itself cannot guarantee that the binary hash codes are located in the ideal hash positions. Directly binarizing the latent code h _i will lead to a large amount of feature information loss, such as some bits close to 0 being binarized as ±1, which is not ideal for image clustering with similar hash codes. In order to alleviate the problem of information loss, the quantization loss was proposed in reference [7]. It constrains the latent code by calculating the L2 regularization loss between the latent code h _i and its corresponding binary vector b _i . Subsequent works [3, 25] have demonstrated that adding quantization loss can significantly improve the performance of image retrieval, bringing new opportunities for the development of image retrieval technology.

L Q = 1 N ∑ i = 1 N ∥ h i - b i ) ∥ 2 2(7)

In image retrieval, deep learning models are commonly used to learn feature representations, which are then subjected to distance measurement to enable matching of similar images. The purpose of metric loss is to optimize this feature representation so that similar images are closer to each other in feature space, while dissimilar images are further apart. On the other hand, quantization loss aims to quantize continuous feature vectors into discrete binary codes for more efficient storage and retrieval. However, the learning objectives of metric loss and quantization loss are contradictory, since metric loss requires the feature representation to be more compact and continuous in feature space, while quantization loss requires the feature representation to be more discrete in the binary code space.

To illustrate this contradiction more clearly, we use an example of a two-dimensional feature space shown in Fig. 4, which contains three different categories. To better demonstrate their distribution in the two-dimensional feature space, all samples are scaled to a feature circle with a radius of r = 2 , where each point on the feature circle corresponds to a feature vector representing a sample’s feature representation. As there are only four quadrants in the two-dimensional space, the binarization operation directly pulls all samples to four binary points (i.e., (±1, ±1)) corresponding to the four quadrants. The farther away the sample is from the binary point, the more information is lost after the hashingfunction.

OPEN IN VIEWER

Fig. 4

(a) The objective of metric loss is to maximize the inter-class distance while minimizing the intra-class distance in the feature space. (b) Once the quantization loss is incorporated, the class centroids tend to shift towards the binary points. However, the metric loss that repels the classes from one another prevents the centroids from aligning with the optimal hashing positions. (c) Upon integrating hash-guided metric loss, the solution obtained is the global optimum for this problem.

Figure 4(a) illustrates the goal of metric learning by showing samples of three different classes in a 2D feature space, where the centers of these classes are pushed as far apart from each other as possible to form an equilateral triangle, representing the best solution where each cluster is separated as much as possible from each other. In contrast, in Fig. 4 (b), the goal of quantization learning is to pull the class centers to one of the four binary points as closely as possible to minimize information loss during the binarization process, resulting in the optimal quantized solution (i.e., minimizing information loss as much as possible). It is visually clear that in metric learning, we want the center points of different classes to be separated as much as possible, while in quantization learning, we want the center points of different classes to be pulled towards one of the four binary points, to minimize information loss in the hashing process. Therefore, there is no solution that simultaneously satisfies the best metric and the best quantization goals. In fact, the ideal solution is to have the three class centers strictly located at the three of the four binary points, which is the optimal global solution with the least information loss during the binarization process (as shown in Fig. 4 (c)). When both metric loss and quantization loss are integrated into a method, an intermediate state is obtained (as shown in Fig. 4 (b)), which lies between the optimal metric solution and the optimal global solution. This integration leads to more information loss during the binarization process, thus reducing retrieval accuracy. Specifically, samples that are further away from the cluster center are more likely to be misclassified into other binary points.

The Proxy-Anchor [37] was previously the state-of-the-art method, which used the cosine similarity estimated in small batches to design the metric loss term. Our proposed HGM-loss modifies the linear computation part of the cosine similarity in the Proxy-Anchor metric loss to restrict the learning scope of the metric term. The metric loss of Proxy-Anchor is calculated as Equation 8: M loss = 1 | ℙ | ∑ p ∈ ℙ log ( 1 + ∑ h ∈ h p - e α ( cos ( h , p ) + γ ) ) + 1 | ℙ + | ∑ p ∈ ℙ + log ( 1 + ∑ h ∈ h p + e - α ( cos ( h , p ) - γ ) )(8) In Equation 8, alpha (α) is a scaling factor, while gamma (γ) is a margin. Let ℙ denote the set of proxies and ℙ + represent the positive proxies corresponding to samples in a mini-batch. In the Proxy-Anchor method, each class has a proxy vector, which is continually updated during training to maximize the distance between positive and negative classes while minimizing the distance between the same class. The latent code set ℍ for a proxy is divided into two categories: h p + represents all samples of the same category, while the remaining set h p - = ℍ - h p + represents samples of different categories to the proxy.

In Equation 8, the exponential term is linearly related to the cosine similarity of all samples, which contradicts the principles of quantization learning discussed earlier. Therefore, our proposed HGM-loss uses a specific threshold value ν to correct this part of the equation. The HGM-loss is shown in Equation 9: L HGM = 1 | ℙ | ∑ p ∈ ℙ log ( 1 + ∑ h ∈ h p - ( e α F ( cos ( h , p ) , - ν - δ ) - 1 ) ) + 1 | ℙ + | ∑ p ∈ ℙ + log ( 1 + ∑ h ∈ h p + ( e - α F ( cos ( h , p ) , - δ ) ) )(9)

where F(a,b) = max(0,a + b), δ is a margin hyperparameter to prevent the model from fitting too closely to the training data. The proposed HGM-loss modifies the error component calculation part in the Proxy-Anchor metric loss with a specific threshold value to alleviate the conflict between metric loss and quantization loss. The value of threshold ν depends on the binary code length and the number of categories, which indicates the minimum distance d _min of the optimal global solution. The goal of HGM-loss is not to have an infinite metric distance between different classes, but to push them apart until they reach a specific distance ν derived from the minimum distance d _min . When the distance between different classes satisfies ν, the optimal metric solution is considered to have been achieved. Overall, the HGM-loss aims to balance the separation of different class clusters and the minimization of information loss in the quantization process.

The calculation method of threshold ν is equivalent to minimizing the distance between given binary codes, however, previous work [34] has proven that computing the ideal minimum distance requires exponential time complexity, which is an NP-hard problem. However, researchers [35] proposed a method to estimate the minimum distance of binary linear codes and provided a table of lower bounds for the minimum distance of binary linear codes, denoted as 𝕋 . Specifically, they limit the minimum distance through strict inequality constraints and propose some upper and lower bound estimation methods to solve the NP-hard problem. Among them, the most important is the Gilbert-Varshamov bound [36], which can be used to calculate the upper and lower bounds of the minimum distance of binary linear codes and can be computed in polynomial time. Inspired by [35], we set the minimum distance d _min according to the following rules: if the value obtained from 𝕋 is a constant, we use that value as d _min ; if the value obtained is a range of values, we choose the median of that range as d _min . Finally, we normalize d _min according to the length K of the hash code to obtain the optimal threshold ν, ensuring a certain degree of separation between differentcategories. ν = 1 - d min 2 K(10) Following [27, 28], our approach combines the metric loss with the quantization loss to reduce information loss during the binarization process. Our learning objective can be expressed as: min ( L HGM + β L Q ) , where we use β=1 to train CSF-Net in an end-to-end manner.

4.1 Dataset

We conduct extensive experiments on four representative benchmarks (CIFAR-10 [38], CIFAR-100 [38], MS-COCO [39], and ImageNet [40]) to validate the performance of our proposed method and to demonstrate the effectiveness of the proposed CSF-Net.

ImageNet is a widely used large-scale image dataset containing over 14 million labeled images, covering more than 20,000 categories. We adopt the same dataset and setting as in [13], where the main challenge is the large number of categories and limited number of training samples.

CIFAR-10 is a public dataset commonly used for image recognition, but it is also suitable for image retrieval tasks. It contains 60,000 32x32 color images belonging to 10 classes, with 6,000 images per class. We selected 50,000 images as the training set and the database, and the remaining 10,000 images as the query set.

CIFAR-100 dataset is an image classification dataset that consists of 100 classes, with each class containing 600 32x32 pixel color images. There are 50,000 images for training and 10,000 images for testing. The 100 classes in the dataset are divided into 20 superclasses, with each superclass containing 5 subclasses. CIFAR-100 is an extension of the CIFAR-10 dataset, with more classes and a more diverse set of images.

MS-COCO dataset is a commonly used multi-label image recognition dataset, which contains over 120,000 images, each labeled with several of 80 categories. To remove images without any labels, according to [28], the dataset is first preprocessed, and then 5,000 images that contain at least one label are randomly selected from the remaining 122,218 images as the query set, 10,000 images as the training set, and the rest as the database. The main challenges of this dataset are its multi-label nature and its large scale of images.

4.2 Evaluation protocols and implementation details

The CSF-Net is implemented in the PyTorch framework. We use pre-trained ResNet-50 on ImageNet as the backbone to show the superiority of our solution. The convolutional layers with the proposed CFS module and HGM-loss are optimized using backpropagation. We use stochastic gradient descent (SGD) as the optimizer with a momentum of 0.9 and weight decay of 1e-5. The initial learning rate is set to 0.01. As for hyperparameters, we set δ to 0.2 based on empirical experience. All experiments use batch size of 128 trained on a NVIDIA TITAN X (Pascal) GPU for 150 epochs. We employ the widely used metric of mean average precision (mAP) to evaluate the retrieval performance, which is a metric ranging between 0 and 1, used to measure the accuracy and reliability of a retrieval system. mAP @ N = 1 | Q | ∑ i = 1 | Q | 1 N ∑ j = 1 N P ( j ) · rel ( j )(11) Where |Q| represents the size of the query set, N represents the maximum number of results returned for each query, P (j) represents the precision of the top j retrieved results, i.e., the proportion of relevant samples among the top j retrieved results. rel (j) is usually set to 1 or 0 to indicate whether the jth retrieved result is relevant or not.

Table 1 details the comparisons of mean average precision (mAP) across various hash bit lengths, allowing us to juxtapose the performance of our proposed CSF-Net method with that of alternative leading-edge methods. Intriguingly, CSF-Net exhibits superior performance when compared to other techniques, particularly when we consider the 32-bit and 48-bit hash codes within the CIFAR-10 and CIFAR-100 datasets. While the breadth of categories present in the CIFAR-100 dataset makes image identification a more demanding task, and results in a more notable performance decline for all methods when compared to CIFAR-10, CSF-Net remains robust. It’s worth noting the stark performance drop exhibited by the DPSH and DSDH methods when transitioning from CIFAR-10 to CIFAR-100, yet our CSF-Net shows resilience amid these challenges. This highlights our solution’s capability to mitigate observed conflict between metric learning and quantization learning, thereby demonstrating the robustness of our approach.

When discussing the large-scale ImageNet and MS-COCO datasets, CSF-Net consistently outshines. With a 64-bit hash code on ImageNet, an uptick of 0.5%, 0.6%, and 1.5% is observed over the next best performing method—CSCE-Net for 16-bit, 32-bit, and 64-bit hash bits respectively. Our CSF-Net implementation on the MS-COCO dataset also outperforms CSCE-Net, barring a minor exception where the performance dips with 32-bit hash bits. Amid varying datasets and variable hash bit lengths, CSF-Net remains steady and impressive. It confirms the algorithm’s inherent strength for extracting and isolating crucial granular fine-grained information. We also recognize the role of our newly proposed HGM-Loss, which plays a significant part in minimizing informational loss during the binarization process to the farthest extent.

In Fig 5, we demonstrate the comparative results between our method and others in terms of precision when employing 64-bit binary codes. Our method stands out with a significant advantage in precision among all compared. This advantage is attributed to the effective use of 64-bit binary codes in our method to acquire and retain information, as well as the exceptional stability and accuracy of our algorithm while handling high-dimensional data and complex scenarios.

OPEN IN VIEWER

Fig. 5

Precision@top-500 curves on CIFAR-10 and MS-COCO datasets with binary codes @ 64-bits.

We also observed that although our algorithm performs well under the settings of 64-bit binary coding, there might be minimal precision drop under certain circumstances, such as complexity in the image patterns or in cases of high noise. This indicates that there is room for improvements in our method to handle these challenges.

On the other hand, in Fig 6, we present the retrieval results of our CSF-Net on the CIFAR-10 dataset, using only 12-bit hash codes. The graph suggests that our method can execute accurate image retrievals even when only applying 12-bit hash codes. This further emphasizes the strong capability of our method in dealing with the issue of information loss during the binarization process.

OPEN IN VIEWER

Fig. 6

Shown here are the top 7 images retrieved with our proposed method using 12 bits on the CIFAR-10 dataset

Investigating the Impact of Network Components on Key Retrieval Information Extraction. We conduct a series of ablation experiments to investigate which parts of the network are most beneficial for extracting key retrieval information. In these experiments, we use the same loss function( L HGM + L Q ) and build a baseline model based on the fourth layer of ResNet50, which represents the global context information of the image. In Table 2, we show the performance of different models, where “baseline+local” represents the concatenated output of global and local features. We find that this model performs better than the baseline results on both datasets, possibly because the local features contain geometric discriminative information about specific image regions. However, performing a single sliding window operation or concatenating sliding window features of different scales on global features does not achieve significant performance improvement, as verified in rows 3-5 of Table 2. In addition, when considering the fusion of global and local features, we introduce a CFS module for feature processing. Table 2 shows the performance of features processed by CFS, where “Baseline+CFS” represents the final feature after global and local features are processed by the CFS module. We find that compared to all other conditions, the CFS module performs best on both the CIFAR-10 and ImageNet datasets, indicating that the CFS module has strong fusion ability, can filter duplicate information, and retain important representative region information. It is particularly noteworthy that even when only a small amount of information can be retained, our CFS module still has strong representative ability, which has been demonstrated in the case of 12-bit hash codes.

Analysis on the Effects of loss functions. In our previous discussions, we elaborately explained the incompatibility issues between the metric learning and the quantization learning. To tackle these issues, we proposed the HGM-Loss, which explicitly restricts the learning range of the metric term, thereby preventing infinite separation of class centers. Furthermore, drawing upon the minimum distance theory of binary linear codes, our HGM-Loss pays more attention to quantization learning while ensuring distinguishable clustering, thereby achieving a balance between metric learning and quantization learning. To verify the effectiveness of our method, we conducted several tests on the large-scale ImageNet dataset and made comparisons with other methods.

We adopted various loss settings in the experiment represented in Table 4. “Proxy-NCA” introduced a set of class centers that can be learnt using proxy-based loss. “Proxy-Anchor” is also a proxy-based loss, which amalgamates the advantages of proxy-based methods and pair-based methods. “Qua” stands for Quantization loss. “CE” refers to the Cross Entropy loss utilized, and “CF” represents the CosFace loss implemented.

From Table 4, it is evident that our proposed HGM-Loss+Qua algorithm displays remarkable performance on the Imagenet100 dataset, with average mAP highest performances of 0.874, 0.906, and 0.912 respectively. This substantiates the robustness and effectiveness of our HGM-Loss, even when the increase in the number of categories within the dataset exacerbates the conflict issues between metric learning and quantization learning. Our method effectively ameliorates these conflicts and manifests its resilience.

“CE+Qua” and “CF+Qua” represent the combined results of Cross-Entropy loss and CosFace loss. Their respective mAP performances for 16bits, 32bits, and 64bits stand at 0.862, 0.886, 0.893 and 0.863, 0.887, 0.894. Although these scores are slightly lower than when we utilize our proposed HGM-Loss, they still verify that implementing Quantization loss can help improve the precision of image retrieval.

In conclusion, these experimental outcomes validate the superiority of our proposed HGM-Loss in achieving an effective balance between metric learning and quantization learning and thereby facilitating efficient and precise image retrieval.

Sensitivity Analysis of Hyperparameters. We conduct a sensitivity analysis of hyperparameters to explore the influence of δ on the model. As shown in Table 3, we perform experiments on three standard datasets with a hash code length of 32 bits and set the range of the margin hyperparameter δ from 0 to 0.4. The results indicate that the optimal performance occurs around δ of approximately 0.2. When using the HGM-Loss, the main role of δ is to prevent similar samples from being simply mapped to the same latent codes, thus ensuring that the sample distribution is in the ideal hypersphere. It should be noted that without adding δ, the mAP of each dataset will generally decrease due to the problem of overfitting, which will harm the retrieval performance. Additionally, if δ is increased uncontrollably, that is, when δ is far from 0.2, we observe that the performance will rapidly decline, and this cannot guarantee consistent performance improvements across all datasets. Therefore, experimental results demonstrate that choosing an appropriate δ is advantageous for improving hash performance.

In HGM-Loss, the alpha value has a direct impact on the model’s performance. The alpha value controls the weight of the two parts of the loss in HGM-Loss, namely, the metric loss and the quantization loss. Proper adjustment of the alpha value can affect the model’s performance in two aspects: 1. Metric learning of the data, that is, how to separate the distance between different categories in the feature space; 2. Quantization learning of the hash code, that is, how to map the continuous feature vectors to the discrete binary code. By adjusting the alpha value, the impact of these two aspects can be balanced, thereby optimizing the overall model performance. Therefore, choosing the appropriate alpha value is a key factor in optimizing the performance of HGM-Loss. As shown in Table 6, we conducted experiments on the model on three standard datasets (CIFAR-10, CIFAR-100, and ImageNet), with the hash code length set to 32 bits, and the scaling factor α ranging from 0.15 to 0.6. The experimental results show that the optimal performance for each dataset occurs at a specific value of α. Specifically, for CIFAR-10, the optimal α value is 0.15; for CIFAR-100, the optimal α value is 0.25; for ImageNet, the optimal α value is 0.45.

In the CIFAR-10 dataset, with the increase in the value of α, the model’s mAP performance exhibited a significant decrease. When α is set at 0.15, the mAP reaches the maximum value of 0.889. This suggests that for this dataset, a smaller α value can help the model achieve higher performance. This result may be due to the greater tolerance of error as a result of a smaller α, which allows the model to better learn the underlying patterns of the data and to avoid over-concentrating on noise and outliers. However, in the CIFAR-100 dataset, the model’s performance reaches its highest value when α is 0.25. This may be due to the fact that for datasets with more categories, a slightly higher α value can be beneficial in distinguishing more classifications, thereby improving the model’s performance. On the ImageNet dataset, with the increase in α, the model’s performance initially shows an upward trend and achieves its peak when α is 0.45. Afterward, as α continues to increase, the model’s performance starts to decline. This may be related to ImageNet’s higher image complexity and the number of categories, which allows the model to improve performance within a certain range of increased α, but an overly large α may cause the model to overfit, leading to performance degradation.

In summary, this ablation study reveals the impact of the α value on the model’s classification performance, and indicates that the selection of α needs to take into account the characteristics and complexity of the dataset. A specific α value cannot adapt to all datasets, thus it needs to be chosen and adjusted according to the specific dataset and task at hand.

Effect analysis of different feature fusion methods. In the ablation study, we compared different feature fusion methods on the ImageNet dataset. As shown in Table 5, we evaluated the performance of FFA-Net, CFNet, CFP, Concat, Add, and our proposed CFS under different hash bit lengths.

Firstly, it can be observed that our CFS method achieves competitive performance across all hash bit lengths. Particularly, at 64-bit hash bit length, our CFS method achieves the highest performance with an mAP@1000 of 0.912, surpassing all other methods. This indicates a significant advantage of our CFS module in cross-scale feature fusion, enabling better capture of both global and local feature information in images at 64-bit hash length. Secondly, although in some cases, such as at 32-bit hash length, our CFS method slightly trails behind CFNet, it still demonstrates high mAP@1000 (0.906). This suggests that while CFNet may excel in capturing global feature information at 32-bit hash length, our CFS method remains competitive in local feature fusion.

Compared to other feature fusion methods, our CFS method exhibits stable performance across different hash bit lengths. In particular, our CFS method outperforms CFP in terms of mAP@1000, further confirming the superiority of our CFS module in extracting key fine-grained information. Overall, the results of the ablation study validate the effectiveness and competitive performance of our CFS module across different hash bit lengths. While in some cases, other feature fusion methods may demonstrate better performance, our CFS module’s advantage in cross-scale feature fusion is more pronounced, providing a more flexible and robust feature representation for image retrieval tasks.