Multi-branch selection fusion fine-grained classification algorithm based on coordinate attention localization

Abstract

Object localization has been the focus of research in Fine-Grained Visual Categorization (FGVC). With the aim of improving the accuracy and precision of object localization in multi-branch networks, as well as the robustness and universality of object localization methods, our study mainly focus on how to combine coordinate attention and feature activation map for target localization. The model in this paper is a three-branch model including raw branch, object branch and part branch. The images are fed directly into the raw branch. Coordinate Attention Object Localization Module (CAOLM) is used to localize and crop objects in the image to generate the input for the object branch. Attention Partial Proposal Module (APPM) is used to propose part regions at different scales. The three classes of input images undergo end-to-end weakly supervised learning through different branches of the network. The model expands the receptive field to capture multi-scale features by Selective Branch Atrous Spatial Pooling Pyramid (SB-ASPP). It can fuse the feature maps obtained from the raw branch and the object branch with Selective Branch Block (SBBlock), and the complete features of the raw branch are used to supplement the missing information of the object branch. Extensive experimental results on CUB-200-2011, FGVC-Aircraft and Stanford Cars datasets show that our method has the best classification performance on FGVC-Aircraft and also has competitive performance on other datasets. Few parameters and fast inference speed are also the advantages of our model.

Keywords

FGVC CAOLM SB-ASPP SBBlock coordinate attention

1. Introduction

The purpose of Fine-Grained Visual Categorization (FGVC) is to distinguish various more fine-grained object categories through subtle and indistinguishable visual differences between subordinate categories of the same kind of objects, such as CUB-200-2011 [21], Stanford Cars [13], FGVC-Aircraft [15] are all commonly used datasets in the FGVC field. There are usually only indistinguishable visual differences between different sub-categories with the influencing factors such as deformation, occlusion, care, and shooting angle, making FGVC become a challenging task. convolutional neural network (CNN) still dominates the image classification, but traditional single convolutional neural networks do not perform well in FGVC. The SOTA methods usually choose a convolutional neural network with strong feature extraction capability as the backbone network, and introduce some improved methods to enable the network to capture fine-grained differences in FGVC tasks to improve classification performance.

[17] proposed an object-part attention method, which exploits objects and part attention to extract local subtle differences and achieves excellent results in distinguishing subordinate categories. It demonstrates the effectiveness of methods which focus on different part regions of object in fine-grained visual classification tasks by using more than one deep learning models. Recurrent Attention Convolutional Neural Network (RA-CNN) [5] uses three branches to iteratively generate regional attention from coarse-grained to fine-grained, and uses a mutually reinforcing way to recursively learn the discriminative regional attention and the feature representation based on the region. It concatenates feature vectors of multiple region parts for classification. However, RA-CNN can only amplify the most important features and pay insufficient attention to other finer-grained features, resulting in limited effects. MMAL-Net [29] designs the Attention Partial Proposal Module (APPM) according to the sliding window and uses the non-maximum suppression (NMS) method to crop out multiple feature regions of different scales and different parts. It can focus on other more fine-grained features while focusing on the main feature regions like RA-CNN. The fully connected layer and the convolutional layer in three branches of MMAL-Net share the parameters with each other, so they can have good classification performance for objects of different scales and parts. However, the attention object localization module AOLM used in MMAL-Net needs to intercept the output features of two different layers in the convolutional neural network to achieve good localization performance. Our experimental results show that even if the backbone network of MMAL-Net is replaced from the original ResNet-50 [8] to ResNeXt-50 [28] which has the same architecture but stronger classification ability, the classification accuracy of the model will drop significantly. In addition, the localization accuracy of AOLM will drop significantly as the network focuses more and more on discriminative regions. MMAL-Net’s reliance on the classification results output by a single branch also leads to its underutilization of the performance of multi-branch models.

Fig. 1.

The structure of multi-branch selection fusion fine-grained classification algorithm based on coordinate attention localization.

Based on the advantages and weaknesses of the MMAL-Net, we propose our fine-grained classification algorithm. Our model structure is shown in Fig. 1. In the training phase, our model consists of three branches. The raw branch directly uses the original image for training, and learns the overall features according to the complete information of the object. The output of the last convolutional layer in the raw branch is sent to CAOLM as the input to obtain the bounding box information. After the fusion of multi-scale information and the weight redistribution of the coordinate attention module, the object image not only contains all the structural features of the object but also more fine-grained features, what’s more, background that is not helpful for classification are cropped. The APPM is used to crop out multiple different parts from the object image that have the greatest degree of differentiation and the least degree of mutual inclusion as part images and enter the part branch for training. In this way, the network can learn the features of different parts and scales of the target. Inspired by Selective Kernel Net (SKNet) [14], we propose a selective branch block (SBBlock), which is used to select and fuse the output features of different branches to avoid the cropped image missing the discriminative information in the raw image. Selective Branch Atrous Spatial Pooling Pyramid (SB-ASPP) is used to reduce the impact of the network’s increasing focus on more discriminative regions on the localization ability of Coordinate Attention Object Localization Module (CAOLM).It can expand the receptive field of the network and capture multi-scale features. The three branches of our model share the parameters of the fully connected layer and the convolutional layer. In this way, they can have good classification performance for objects of different scales and parts in learning process.

What follows is our summary of our work. (1) CAOLM is proposed to solve the disadvantage that AOLM can only play a good localization ability under certain conditions, and improve the localization ability by using the attention to the coordinate position. (2) SB-ASPP is proposed to alleviate the disadvantage that the AOLM localization accuracy declines after a certain period of network training and the grid effect problem of ASPP. (3) SBBlock is proposed to selectively fuse features from different branches for better classification. (4) On all datasets used in the experiments, our method has competitive classification performance compared to the best algorithms and outperforms MMAL-Net steadily.And the amount of parameters of our entire model is only comparable to Resnet-101.

2. Related work

2.1. Attention mechanism

In the field of image classification, attention mechanisms are special structures embedded in models that mimic the characteristics of human visual observations that are biased towards dominant features. It is used to automatically learn and calculate the input data contribution size and adjust it. (SENet) [10] adaptively redistributes weights in the channel dimension through squeezing and excitation operations, enabling the network to focus on more efficient channels. SKNet [14] applies group convolution and branch convolution to make the network adaptively adjust the appropriate receptive field size. Spatial attention is used in (CBAM) [26]. CBAM calculates the attention map in the spatial and channel dimensions in turn, after which it is multiplied by the input feature map to optimize the feature map weights. The recent method [7] adopts vertical max-pooling combined with horizontal max-pooling, and finally stitches them together to form an attention map. In this way, important location information can be obtained more effectively. Inspired by these methods [7,14,26], we design the coordinate attention part in CAOLM.

2.2. Localization-classification subnetworks

A model using this approach usually consists of a classification sub-network and an object localization sub-network. The accuracy of object localization by the object localization sub-network is the focus of research in this direction. The classification sub-network is classified by learning more fine-grained object area features generated by the object localization sub-network. Earlier methods such as Mask-CNN [25] required the use of part-level annotations to help train classification subnetworks. It generates multiple informative partial images and achieves classification by concatenating these part-level feature vectors to represent the entire object. The generation and maintenance of additional annotations is labor-intensive and affects practical applications. Therefore, methods such as NTS-Net [2], TASN [3], RA-CNN [5] and [7] use the attention mechanism to locate the object region to avoid additional overhead. MMAL-Net [29] is used to localize the target image simply by calculating the regions in the feature map where the response value is greater than the average value.

2.3. Atrous convolution

Atrous convolution [2] is widely used in tasks such as semantic segmentation and object detection. It refers to the interval sampling of the input image during the convolution process, which can expand the receptive field and capture multi-scale context information without reducing the resolution. Atrous Spatial Pooling Pyramid [3] (ASPP) uses different intervals in the same layer, which can precisely control the receptive field of the network and capture multi-scale information. However, due to the sparse sampling of the input signal by atrous convolution, the problem of local information loss and lack of correlation of information will be caused. Hybrid Dilated Convolution [22] (HDC) alleviates this problem by using dilated convolutions with different intervals in different convolutional layers. [23] Periodically sample the original feature map to form 4 sets of feature maps with decreasing resolution, and then use the original atrous convolution parameters to perform convolution respectively, and then upsample and combine the results of the convolution to collect inconsistent local information. By introducing an attention method, SB-ASPP selectively fuse the outputs of convolutions at different intervals and add shortcuts to allow our model to capture multi-scale features while avoiding the loss of local information.

3. Method

3.1. Selective branch block

Fig. 2.

The structure of selective kernel convolution.

In SKNet [14], a nonlinear method is proposed to adjust the receptive field size of neurons by aggregating information from multiple convolution kernels adaptively. There are three operators in selective kernel (SK) convolution, which are split, fusion, and selection. The split operator generates different branches with convolution kernels of different sizes, corresponding to receptive fields of different sizes. By combining and aggregating information from multiple paths, the fusion operator obtains a global representation of the selection weights. According to selection weights, the selection operator aggregates the feature maps of different paths and redistributes the weights. The SK convolution is computationally lightweight and only slightly increases the parameters and computational cost. Figure 2 shows the structure of the SK convolution. We want to be able to combine and aggregate information from different branches. The fusion and selection part of SK convolution are reserved, and we call it the selective branch block (SBBlock), the structure is shown in Fig. 3.

$M_{1}$ and $M_{2}$ represent feature maps output by the images in the raw branch and the object branch after passing through the last convolutional layer of the model. The element-wise summation of $M_{1}$ and $M_{2}$ is used to integrate the information from all branches. To generate $s \in R^{C}$ channel-level statistics, global average pooling is used for U to generate global information. Specifically, the cth element of s is obtained by shrinking the cth spatial dimension $H \times W$ corresponding to the feature map U, as shown in formula (1) $\begin{matrix} (1) & S_{c} = F_{g p} (U_{c}) = \frac{1}{H \times W} \sum_{i = 1}^{H} \sum_{j = 1}^{W} U_{c} (i, j) . \end{matrix}$ In order to guide the next adaptive selection of features in the subsequent selection operation, the feature descriptor $z \in R^{d + 1}$ is generated by a fully connected layer. This method reduces the feature dimension and increases the computational efficiency. How to generate the feature descriptor z is shown in formula (2) $\begin{matrix} (2) & z = F_{f c} (s) = δ (ξ (W s)) . \end{matrix}$ Among them, δ represents the ReLU [16] function, and ξ represents the batch normalization [11], $W \in R^{d \times C}$ . In order to reduce the effect of d on the efficiency of model, the reduction ratio r is used in the experiment to control the size of its value, as shown in formula (3), where L represents the minimum value of d $\begin{matrix} (3) & d = max (\frac{C}{r}, L) . \end{matrix}$ After that, we perform a Softmax operation on the data in the channel direction, as shown in Eq. (4) $\begin{matrix} (4) & a_{c} = \frac{e^{A_{c^{Z}}}}{e^{A_{c^{Z}}} + e^{B_{c^{Z}}}}, b_{c} = \frac{e^{B_{c^{Z}}}}{e^{A_{c^{Z}}} + e^{B_{c^{Z}}}} . \end{matrix}$ Where $A, B \in R^{C \times d}$ and a, b respectively represent the attention vectors of $M_{1}$ and $M_{2}$ . It is worth noting that $A_{c} \in R^{1 \times d}$ is the cth row of the A vector, $a_{c}$ represents the cth element of the vector a, and the same is true for $B_{c}$ and $b_{c}$ . Because $a_{c} + b_{c} = 1$ , matrix B is redundant in the two-branch case. The output V is obtained by the attention weights on each kernel, as shown in Eq. (5) $\begin{matrix} (5) & V_{c} = a_{c} \cdot M_{1_{c}} + b_{c} \cdot M_{2_{c}}, a_{c} + b_{c} = 1 . \end{matrix}$ Where $V = [V_{1}, V_{2}, V_{3}, \dots, V_{C}]$ , $V_{c} \in R^{H \times W}$ .

Fig. 3.

The structure of selective branch block.

3.2. Selective branch atrous spatial pooling pyramid

SB-ASPP is inspired by ASPP [3], which samples the input signal in parallel with multiple different sampling intervals in the same layer. This method can accurately control the spatial resolution and capture features at different scales. We hope that the model can capture features of different scales in different parts of the object before CAOLM localizes the object. Our proposed SB-ASPP is still based on atrous convolution, which allows us to adjust the response in one layer at any resolution desired. When we consider a one-dimensional input $x [i]$ , the output $y [i]$ of the atrous convolution of a filter $w [k]$ of length K is defined as Eq. (6), where r corresponds to the stride at which we sample the input signal $\begin{matrix} (6) & y [i] = \sum_{k = 1}^{K} x [i + r * k] w [k] . \end{matrix}$ The convolution result obtained by a certain layer comes from an independent set of the former layer without interdependence, because the calculation method of atrous convolution is close to the checkerboard format. In this layer, there is no correlation between the convolution results, that is, local information is lost. At the same time, due to the sparse sampling input signal of atrous convolution, the long-distance convolution gets information with no correlation, which affects the classification result. To solve the problem of local information loss and lack of correlation, we do not directly concatenate the output results of different convolutions like ASPP. We use SBBlock to fuse convolutions of different strides, and the output results of different convolution strides complement each other’s missing information. When setting the convolution stride, the processing method of HDC is introduced to avoid information loss, and set the stride to be continuous. As shown in Fig. 4, we set three sampling intervals of $r = 1$ , $r = 2$ , and $r = 3$ .

Fig. 4.

Selective branch atrous spatial pooling pyramid.

3.3. Coordinate attention object localization module

By aggregating the feature maps in the channel dimension, the AOLM in [29] obtains the activation map. The activation map is used to generate the bounding box to obtain the object image. We optimized the acquisition method of the activation map, and the specific process is as follows. $X \in R^{C \times H \times W}$ is the output generated by the last convolutional layer of the raw branch, and its channel number is C and the spatial size is $H \times W$ . As shown in Eq. (7), $x_{i}$ represents the ith channel corresponding to the output feature map $\begin{matrix} (7) & A = \sum_{i = 0}^{C - 1} x_{i} . \end{matrix}$ The activation feature map A is generated by aggregating the output feature map X, we divide the activation feature map A equally into four regions with $H ? 2$ and $W ? 2$ as boundaries. The average $\bar{a}$ of the activation feature map is used to determine whether each region belongs to the key region in the recognition task as the threshold. The process of obtaining the average value of a single region is shown in Eq. (8) $\begin{matrix} (8) & \bar{a} = \frac{\sum_{i = 0}^{W / 2 - 1} \sum_{j = 0}^{H / 2 - 1} A (i, j)}{H \times W}, \end{matrix}$ where $(i, j)$ is a specific location in a feature map of spatial size $H ? 2 \times W ? 2$ . Then according to Eq. (9), we obtain the mask map $\tilde{m}$ . Finally, the mask maps of each region are stitched together according to their corresponding activation feature map A to obtain the final mask map $\tilde{M}$ $\begin{matrix} (9) & {\tilde{M}}_{(i, j)} = \{\begin{matrix} 1 & if A (x, y) > \bar{a} \\ 0 & otherwise . \end{matrix} \end{matrix}$

Inspired by AttNet [7], CBAM [26], Coordatt [9], we design the coordinate attention part in CAOLM. The purpose of the attention module is to spatially give greater weight to the region where the recognized object is located. $X \in R^{C \times H \times W}$ is used to represent the output of the CAOLM. Referring to the method of [9], we use two spatially scaled pooling kernels $(H, 1)$ and $(1, W)$ to obtain the overall representation of the horizontal and vertical information separately. As shown in Fig. 5, we compute the horizontal coordinate attention and vertical coordinate attention parts on two branches, respectively. The cth channel of the output of different pooling layers on different branches can be expressed as: $\begin{array}{l} (10) & z_{c}^{h_avg} (h) = \frac{1}{W} \sum_{0 ⩽ i < W} x_{c} (h, i) \\ (11) & z_{c}^{h_max} (h) = Max (x_{c} (h, i)), 0 ⩽ i < W \\ (12) & z_{c}^{h_avg} (w) = \frac{1}{H} \sum_{0 ⩽ j < H} x_{c} (j, w) \\ (13) & z_{c}^{w_max} (h) = Max (x_{c} (j, w)), 0 ⩽ j < H \end{array}$

Fig. 5.

The structure of split coordinate attention module.

Taking a single branch as an example, we send $z^{h} (h)$ , which is obtained by concatenating $z^{h_{avg}} (h)$ and $z^{h_{max}} (h)$ , to a $1 \times 1$ convolutional transformation function $F_{1}$ that shares parameters between the two branches: $\begin{matrix} (14) & f_{h} = δ (F_{1} (z_{c}^{h} (h))) \end{matrix}$ Where $f_{h} \in R^{C ? r \times 2 H}$ is the intermediate feature map that encodes spatial information in the vertical direction. δ stands for nonlinear activation function. r is a scaling parameter (for reducing the overhead, we usually reduce the number of channels of $f_{h}$ with an appropriate reduction ratio r (such as 32). Then we split $f_{h}$ into $f_{h}^{avg} \in R^{(C ? r) \times H}$ and $f_{h}^{max} \in R^{(C ? r) \times H}$ . Taking the average pooling branch encoded in the horizontal direction as an example, as shown in formula (15), we use another $1 \times 1$ convolutional transformation function $F_{h}$ ( $f_{h}^{avg}$ shares the parameters of $F_{h}$ with $f_{h}^{max}$ ) to transform the scaled $f_{h}^{avg}$ into a vertical attention weight $g^{h_{avg}}$ with the same number of channels as the input X $\begin{matrix} (15) & g^{h_avg} = σ (F_{h} (f_{h}^{avg})) . \end{matrix}$ Where σ represents the sigmoid function. Similarly, we can get $g^{h_{max}}$ , $g^{w_{avg}}$ and $g^{w_{max}}$ . Finally, as shown in Eqs (16), (17), we get the feature maps $Y^{avg}$ and $Y^{max}$ obtained using different down-sampling methods. Where $y_{c}^{avg}$ and $y_{c}^{max}$ are the feature maps corresponding to the output feature map channel c $\begin{array}{l} (16) & y_{c}^{avg} (i, j) = x_{c} (i, j) \times g_{c}^{h_{avg}} (i) \times g_{c}^{w_{avg}} (j) \\ (17) & y_{c}^{max} (i, j) = x_{c} (i, j) \times g_{c}^{h_{max}} (i) \times g_{c}^{w_{max}} (j) . \end{array}$ Finally, we input $Y^{max}$ , $Y^{avg}$ , and X into SMBlock to get the output Y. The final mask map M of CAOLM is the intersection of ${\tilde{M}}_{X}$ and ${\tilde{M}}_{Y}$ .

3.4. Structure of multi-branch selection fusion fine-grained classification algorithm based on coordinate attention localization

In order for the model to sufficiently learn the image features obtained by CAOLM and APPM. As shown in Fig. 1, our model with a three-branch structure which consists of raw branch, object branch and part branch. The raw image has three branches after the raw branch is used for feature extraction CNN. The first branch outputs the classification results of the raw branch directly through the fully connected layer. In the second branch, the feature map is sent to SBBlock together with the output result of the last convolutional layer of the object branch, and then outputs the classification result of the object branch through the fully connected layer. In the third branch, the feature map is sent to CAOLM after passing through the SB-ASPP module to generate a mask map for cropping the raw image. The cropped image is used as the input of object branch, and it has two branches after passing through the CNN of the object branch. The feature map is sent to SBBlock in one branch, and in another branch, the feature map is sent to APPM for cropping the part area as the input of the part branch. Cross-entropy loss is used as the classification loss of all three of our branches, as shown in Eqs (18), (19) and (20) $\begin{array}{l} (18) & L_{raw} = - log (P_{r} (c)) \\ (19) & L_{object} = - log (P_{o} (c)) \\ (20) & L_{parts} = - \sum_{n}^{N - 1} log (P_{p^{n}} (c)) . \end{array}$ Where c is used to represent the ground-truth label of images. $P_{r}$ is the possibilities of classification generated by the last Softmax layer of the raw branch. $P_{o}$ and $P_{p^{n}}$ are generated in the same way from the object branch and part branch respectively. $p^{n}$ represents the nth part image, and N is the total number of cropped areas in each image. The definition of the total loss is shown in Eq. (21) $\begin{matrix} (21) & L_{total} = α * L_{raw} + β * L_{object} + γ * L_{parts} . \end{matrix}$ Where α, β, γ are the hyperparameters we use to set the loss weights for each branch (usually set to $α = 0.75$ , $β = 1$ , $γ = 0.75$ . The total loss is the sum of three weight hyperparameters multiplied by losses of three branches respectively. These three branches work together during backpropagation to optimize our model’s performance. In this way, the model achieves a better accuracy of classification predictions with the overall structural features of objects and fine-grained features of parts. SBBlock strengthens the connection between branches. It utilizes the feature that the raw branch possesses the complete features of the object, and supplements the missing information of the object branch when the CAOLM localization is inaccurate. SB-ASPP not only captures multi-scale information but also alleviates the problem of missing information in ASPP. The model has good target scale adaptability, and SB-ASPP and CAOLM improve the robustness of localization when the CNN focuses more on those discriminative features during training. All the used modules in the model are designed to be lightweight and efficient, and in the testing phase, the part branch is removed to reduce a lot of computation. Because of that, our method takes a short time to predict in practical applications. Due to our reasonable and efficient architecture, our method is able to outperform MMAL-Net and has comparable performance to the SOTA algorithms. It outperforms almost all methods based on convolutional neural network architectures on the datasets used in the experiments.

4. Experiments

4.1. Implementation details

In all the experiments we conducted, we fixed the image size to $448 \times 448$ to obtain the input of the raw branch. Object images are also scaled to $448 \times 448$ , but for part branches, the sizes of all part images are $224 \times 224$ . To alleviate overfitting, random flipping and Mixup [30] with a mixing parameter of 0.1 are used as our image enhancement strategy. For activation maps of size $14 \times 14$ , we use windows of three scales: ${[4 \times 4, 3 \times 5], [6 \times 6, 5 \times 7], [8 \times 8, 6 \times 10, 7 \times 9, 7 \times 10]}$ . The number of part images of the raw image is $N = 7$ , where $N_{1} = 2$ , $N_{2} = 3$ and $N_{3} = 2$ . $N_{1}$ , $N_{2}$ and $N_{3}$ are the numbers of the above three types of scale windows. The backbone of our model uses ResNext-50 [8] pre-trained on ImageNet. We do not use any additional annotation information during training and testing. SGD is used as our optimizer with a momentum of 0.9% and a weight decay of 0.0001. On an NVIDIA V100 GPU, our model has a batch size of 6 with an initial learning rate of 0.001, which is multiplied by 0.1 after 60 epochs. Our method is implemented in Pytorch.

4.2. Datasets

Fig. 6.

Dataset samples.

The samples of the datasets used in the experiments (CUB-200-2011 [21], Stanford Cars [13], FGVC-Aircraft [15]) are shown in Fig. 6. The CUB-200-2011 dataset contains a total of 11788 annotated images, of which 5909 images are used for training and 5879 images are used for testing. The dataset images contain a total of 200 different categories of birds. The Stanford Cars dataset contains a total of 16,185 annotated images, of which 8,144 were trained and 8,041 were tested. The dataset images contain a total of 196 cars of different brands and models. The FGVC-Aircraft dataset contains a total of 10,000 annotated images, of which 6,667 images are used for training and 3,333 images are used for testing. These images are typically grouped into 100 different types of aircraft in fine-grained visual classification. Classification tasks on these datasets are still challenging.

4.3. Comparison to state-of-the-art

Table 1
Comparison results with state-of-the-art algorithms

Method Accuracy [%]

CUB-200-2011 Stanford Cars FGVC Aircraft

ResNext 79.4 84.9 80.1

ResNet 78.9 84.7 79.7

Inceptionv4 [20] 81.2 87.6 82.3

Xception [4] 80.3 86.2 81.7

RA-CNN [5] 85.3 92.5 –

NTS-Net [2] 87.5 93.9 88.4

Spatial RNN [27] 89.7 93.4 88.4

ACNet [12] 88.6 94.6 –

Stacked LSTM [6] 90.4 – –

API-Net [31] 90.0 95.3 93.9

CAL [19] 90.6 95.5 94.2

AttNet [7] 88.9 95.6 94.1

MMAL-Net(ResNet-50) 89.6 94.7 94.5

MMAL-Net(ResNext-50) 87.1 93.3 92.7

Inceptionv4+NB tranfer [18] – 95.3 95.1

CAP(Xception) [1] 91.6 95.6 94.1

CAP(ResNet-50) [1] 90.9 94.9 94.9

Ours(ResNet-50) 90.0 95.3 95.1

Ours(ResNext-50) 90.2 95.6 95.4

Method	Accuracy [%]
ResNext	79.4	84.9	80.1
ResNet	78.9	84.7	79.7
Inceptionv4 [20]	81.2	87.6	82.3
Xception [4]	80.3	86.2	81.7
RA-CNN [5]	85.3	92.5	–
NTS-Net [2]	87.5	93.9	88.4
Spatial RNN [27]	89.7	93.4	88.4
ACNet [12]	88.6	94.6	–
Stacked LSTM [6]	90.4	–	–
API-Net [31]	90.0	95.3	93.9
CAL [19]	90.6	95.5	94.2
AttNet [7]	88.9	95.6	94.1
MMAL-Net(ResNet-50)	89.6	94.7	94.5
MMAL-Net(ResNext-50)	87.1	93.3	92.7
Inceptionv4+NB tranfer [18]	–	95.3	95.1
CAP(Xception) [1]	91.6	95.6	94.1
CAP(ResNet-50) [1]	90.9	94.9	94.9
Ours(ResNet-50)	90.0	95.3	95.1
Ours(ResNext-50)	90.2	95.6	95.4

As shown in Table 1, our method is used to compare with state-of-the-art algorithms (this SOTA information comes from “https://paperswithcode.com”) that do not use additional annotations like ours on the CUB-200-2011, Stanford Cars and FGVC Aircraft datasets. From the results, our method stably outperforms similar methods [5,29] that crop images through multi-branch loop training. The obvious drop in the classification performance of MMAL-Net using ResNext-50 as the backbone network proves the universality of our CAOLM to the backbone network. On CUB-200-2011, CAP (AAAI 2021) achieves the state-of-the-art classification performance. It proposes a context-aware attentional pooling (CAP) module, which can efficiently encode the location and appearance information of local features. This module takes the features output by the convolutional network as input, and learns to adjust the importance of different regions in the features, so as to obtain the rich appearance features and spatial features of local regions, and then perform accurate classification. On Stanford Cars, our model has similar classification performance to CAP, but we have significantly better performance gains compared to the backbone used. When it uses the same backbone as ours, CAP is only 94.9% accurate on Stanford Cars. Among all the algorithms used for comparison, we achieve the best classification performance on the FGVC Aircraft dataset. The only method [18] with similar classification performance uses Inceptionv4 as the backbone, and it uses a complex training strategy instead of end-to-end training. Apart from that, we have clearly outperformed the other models on the FGVC-Aircraft. Our method achieves 10.7% and 15.4% improvement over the baseline (ResNext-50) on Standford Cars and FGVC-Aircraft, respectively. And when we use ResNet-50 as the baseline, there are also 10.6% and 15.4% improvements on these two datasets, respectively. As far as we know, our method currently has the most improvement over the baseline. It is attributed to the accurate object localization of our proposed CAOLM and the use of different branch information by SBBlock to complement each other. SB-ASPP avoids the problem of information loss caused by atrous convolution while capturing multi-scale information.

4.4. Ablation studies

We perform ablation studies on the Stanford Cars dataset.

The effectiveness of the three-branch structure. The effectiveness of the recurrent multi-branch model has been demonstrated in [5,29]. In our model (using ResNext-50 as the backbone network), if only the raw branch is used, the classification accuracy is 85.3%. On the basis of the above, if the model has the object branch, the classification accuracy can reach 93.5%. Our model with three branches can achieve a classification accuracy of 95.7%.The experimental results show that if the part branch is not used in the inference stage, the model can double the inference speed while the classification results are still excellent. The classification accuracy of our model in this case is 95.6%.

The effectiveness of CAOLM. It can achieve 94.7% accuracy when using ResNet-50 as the backbone in MMAL-Net. But when the backbone was replaced with ResNet-50, the accuracy of MMAL-Net dropped to 93.3% instead. The performance of AOLM degrades when convolutional neural networks focus on discriminative features very efficiently. Also using ResNet-50 as the backbone network, we replaced the AOLM in MMAL-Net with our CAOLM, and the accuracy increased to 95.1%. When using SBBlock, the accuracy of MMAL-Net improves to 95.3%. It is proved that our method can improve the localization and classification performance effectively.

The effectiveness of changing the backbone. The backbone used by our method is ResNext-50, which achieves 95.6% accuracy classification performance on Stanford cars. This is better than the classification accuracy (95.3%) when our method uses ResNet-50 as the backbone. It proves that CAOLM has better generality to the backbone network of the model.

The effectiveness of SB-ASPP. If SB-ASPP is removed in our method, the accuracy drops from 95.6% to 95.0%. SB-ASPP can adjust the network receptive field and focus on global multi-scale information, which alleviates the disadvantage that the localization performance of AOLM will decrease when the convolutional neural network is very effective in focusing on discriminative features.

4.5. Object localization performance

Compared to the ground-truth bounding boxes, the percentage of predicted boxes which correctly locate more than 50% of the IOU (Intersection over Union) is PCP (Percent of Right Localized Parts). As shown in Table 2, the localization ability of CAOLM is significantly improved compared with SCDA [18] and AOLM [29]. Although multi-layer integration significantly improves the performance of AOLM, in actual experiments, we find that which two layers are used for integration in different networks requires a lot of experiments to find. This method reduces the universality of AOLM, and the effect of single-layer AOLM is not ideal in practical experiments. The positioning accuracy of AOLM can reach a maximum of 85.1%, but it will actually decrease to 71.1% with the training of the network. It is worth mentioning that if the Mixup image enhancement strategy is used, the AOLM cannot find the intersection of the activation maps of the two layers, which proves that the robustness of the AOLM to the noise of the input signal is not good. The accuracy of our proposed CAOLM can reach up to 87.4%, and it will drop to 77.1% after network training. This figure rises to 79.1% when using SB-ASPP. The part branch of the cropped image by APPM makes the model adaptable to the scale of the object, and it still achieves good classification performance.

Table 2
Object localization accuracy on CUB-200-2011

Method Localization accuracy [%]

SCDA [24] 76.8%

AOLM(conv_5c) [29] 82.2%

AOLM(conv_5b & conv_5c) 85.1%

CALOM 87.4%

Method	Localization accuracy [%]
SCDA [24]	76.8%
AOLM(conv_5c) [29]	82.2%
AOLM(conv_5b & conv_5c)	85.1%
CALOM	87.4%

5. Conclusion

In this paper, we propose an efficient fine-grained classification method that can be trained end-to-end without additional annotations. The multi-branch structure can fully utilize the images obtained by APPM and CAOLM to achieve excellent performance, and use SBBlock to make the information between branches complement each other. SB-ASPP can capture multi-scale information to alleviate the problem of CAOLM localization ability decline. We propose a recurrent multi-branch model utilizing multiple outputs for classification, and a new coordinate attention-based localization method. Experimental results on CUB-200-2001, FGVC-Aircraft and Stanford Cars datasets show that our method is very competitive with the state-of-the-art algorithms.The way CAOLM combines attention mechanisms and feature activation maps cannot be corrected by the results. The next research focus can be on studying trainable methods for locating objects through coordinate attention.

Footnotes

Acknowledgements

This work was supported by National Natural Science Foundation of China (No. 62062007, 62272198) and Guangdong Key Laboratory of Data Security and Privacy Preserving (No. 2017B030301004).

References

Behera,

Wharton,

P.R.

Hewage and

Bera, Context-aware attentional pooling (cap) for fine-grained visual classification, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, 2021, pp. 929–937.

L.-C.

Chen,

Papandreou,

Kokkinos,

Murphy and

A.L.

Yuille, Semantic image segmentation with deep convolutional nets and fully connected crfs, 2014, arXiv preprint arXiv:1412.7062.

L.-C.

Chen,

Papandreou,

Schroff and

Adam, Rethinking atrous convolution for semantic image segmentation, 2017, arXiv preprint arXiv:1706.05587.

Chollet, Xception: Deep learning with depthwise separable convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1251–1258.

Fu,

Zheng and

Mei, Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4438–4446.

Ge,

Lin and

Yu, Weakly supervised complementary parts models for fine-grained image classification from the bottom up, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3034–3043.

Hanselmann and

Ney, Fine-grained visual classification with efficient end-to-end localization, 2020, arXiv preprint arXiv:2005.05123.

He,

Zhang,

Ren and

Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

Hou,

Zhou and

Feng, Coordinate attention for efficient mobile network design, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 13713–13722.

10.

Hu,

Shen and

Sun, Squeeze-and-excitation networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132–7141.

11.

Ioffe and

Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: International Conference on Machine Learning, PMLR, 2015, pp. 448–456.

12.

Ji,

Wen,

Zhang,

Du,

Wu,

Zhao,

Liu and

Huang, Attention convolutional binary neural tree for fine-grained visual categorization, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10468–10477.

13.

Krause,

Stark,

Deng and

Fei-Fei, 3d object representations for fine-grained categorization, in: Proceedings of the IEEE International Conference on Computer Vision Workshops, 2013, pp. 554–561.

14.

Li,

Wang,

Hu and

Yang, Selective kernel networks, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 510–519.

15.

Maji,

Rahtu,

Kannala,

Blaschko and

Vedaldi, Fine-grained visual classification of aircraft, 2013, arXiv preprint arXiv:1306.5151.

16.

Nair and

G.E.

Hinton, Rectified linear units improve restricted Boltzmann machines, in: Icml, 2010.

17.

Peng,

He and

Zhao, Object-part attention model for fine-grained image classification, IEEE Transactions on Image Processing 27(3) (2017), 1487–1500. doi:10.1109/TIP.2017.2774041.

18.

Plested,

Shen and

Gedeon, Non-binary deep transfer learning for image classification, 2021, arXiv preprint arXiv:2107.08585.

19.

Rao,

Chen,

Lu and

Zhou, Counterfactual attention learning for fine-grained visual categorization and re-identification, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1025–1034.

20.

Szegedy,

Ioffe,

Vanhoucke and

A.A.

Alemi, Inception-v4, inception-resnet and the impact of residual connections on learning, in: Thirty-First AAAI Conference on Artificial Intelligence, 2017.

21.

Wah,

Branson,

Welinder,

Perona and

Belongie, The caltech-ucsd birds-200-2011 dataset, 2011.

22.

Wang,

Chen,

Yuan,

Liu,

Huang,

Hou and

Cottrell, Understanding convolution for semantic segmentation, in: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, 2018, pp. 1451–1460. doi:10.1109/WACV.2018.00163.

23.

Wang and

Ji, Smoothed dilated convolutions for improved dense prediction, Data Mining and Knowledge Discovery 35(4) (2021), 1470–1496. doi:10.1007/s10618-021-00765-5.

24.

X.-S.

Wei,

J.-H.

Luo,

Wu and

Z.-H.

Zhou, Selective convolutional descriptor aggregation for fine-grained image retrieval, IEEE Transactions on Image Processing 26(6) (2017), 2868–2881. doi:10.1109/TIP.2017.2688133.

25.

X.-S.

Wei,

C.-W.

Xie and

Wu, Mask-cnn: Localizing parts and selecting descriptors for fine-grained image recognition, 2016, arXiv preprint arXiv:1605.06878.

26.

Woo,

Park,

J.-Y.

Lee and

I.S.

Kweon, Cbam: Convolutional block attention module, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 3–19.

27.

Wu,

Wang,

Li and

Gao, Deep attention-based spatially recursive networks for fine-grained visual recognition, IEEE transactions on cybernetics 49(5) (2018), 1791–1802. doi:10.1109/TCYB.2018.2813971.

28.

Xie,

Girshick,

Dollár,

Tu and

He, Aggregated residual transformations for deep neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1492–1500.

29.

Zhang,

Li,

Zhai and

Liu, Multi-branch and multi-scale attention learning for fine-grained visual categorization, in: International Conference on Multimedia Modeling, Springer, 2021, pp. 136–147. doi:10.1007/978-3-030-67832-6_12.

30.

Zhang,

Cisse,

Y.N.

Dauphin and

Lopez-Paz, mixup: Beyond empirical risk minimization, 2017, Preprint arXiv:1710.09412.

31.

Zhuang,

Wang and

Qiao, Learning attentive pairwise interaction for fine-grained classification, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, 2020, pp. 13130–13137.

Method	Accuracy [%]

	CUB-200-2011	Stanford Cars	FGVC Aircraft
ResNext	79.4	84.9	80.1
ResNet	78.9	84.7	79.7
Inceptionv4 [20]	81.2	87.6	82.3
Xception [4]	80.3	86.2	81.7
RA-CNN [5]	85.3	92.5	–
NTS-Net [2]	87.5	93.9	88.4
Spatial RNN [27]	89.7	93.4	88.4
ACNet [12]	88.6	94.6	–
Stacked LSTM [6]	90.4	–	–
API-Net [31]	90.0	95.3	93.9
CAL [19]	90.6	95.5	94.2
AttNet [7]	88.9	95.6	94.1
MMAL-Net(ResNet-50)	89.6	94.7	94.5
MMAL-Net(ResNext-50)	87.1	93.3	92.7
Inceptionv4+NB tranfer [18]	–	95.3	95.1
CAP(Xception) [1]	91.6	95.6	94.1
CAP(ResNet-50) [1]	90.9	94.9	94.9
Ours(ResNet-50)	90.0	95.3	95.1
Ours(ResNext-50)	90.2	95.6	95.4

Multi-branch selection fusion fine-grained classification algorithm based on coordinate attention localization

Abstract

Keywords

1. Introduction

2.1. Attention mechanism

2.2. Localization-classification subnetworks

2.3. Atrous convolution

3. Method

3.1. Selective branch block

4. Experiments

4.1. Implementation details

4.2. Datasets

4.5. Object localization performance

Table 2 Object localization accuracy on CUB-200-2011 Method Localization accuracy [%] SCDA [24] 76.8% AOLM(conv_5c) [29] 82.2% AOLM(conv_5b & conv_5c) 85.1% CALOM 87.4%

Footnotes

Acknowledgements

References

Table 2
Object localization accuracy on CUB-200-2011

Method Localization accuracy [%]

SCDA [24] 76.8%

AOLM(conv_5c) [29] 82.2%

AOLM(conv_5b & conv_5c) 85.1%

CALOM 87.4%