Multi-feature fusion of convolutional neural networks for Fine-Grained ship classification

2.1 Fine-grained image recognition

Fine-grained visual categorization has been widely studied. There are some works depend on the hand-crafted feature description and encoding methods for fine-grained classification. In [13], Hierarchical Structure Learning (HSL) algorithm and Geometric Phrase Pooling (GPP) algorithm are used to capture mid-level structures for fine-grained classification. The method of [14] detected volumetric part models based on Poselets and used Stacked Evidence Trees to aggregate information about part properties.

Due to the success of deep learning, Convolutional neural networks (CNN) have been widely used in fine-grained image recognition. TsungYu Lin et al. [11] built a bilinear CNN model, which extracted two local features and multiplied these features to obtain an image descriptor. Zhicheng Yan et al. [15] proposed a Hierarchical Deep Convolutional Neural Network (HD-CNN) that decomposed fine-grained classification task into two steps.

In many recent studies, many works incorporated precise part information to improve fine-grained classification. Jonathan Krause et al. [16] proposed a method based on generating parts using co-segmentation and alignment without part annotations. Philippe-Henri Gosselin et al. [17] proposed a search-based architecture to search for more informative parts and thus improved recognition. Chen Huang et al. [18] explored a task-driven approach for progressive part detection, which provided the most discriminative visual features for the subsequent object classification. Jonathan Krause et al. [19] proposed an object representation that detects important parts in a fully unsupervised manner. Jianlong Fu et al. [20] considered the relationship between region detection and fine-grained feature learning and recursively learned discriminative region attention and region-based feature representation at multiple scales in a mutually reinforced way.

Although there are a significant performance improvement by using CNN, from the above methods we can find out that global CNN features extracted from full images are too spatially rigid to be optimal for fine-grained classification. Therefore, we consider classifying images using local information combined with global information. Results from these works show that the local features are more competitive than CNN features based on full image and can provide important complementary information.

2.2 Multi-feature fusion

Recently, several feature fusion methods have been proposed. Jiang Liu et al. [21] adopted a weighted fusion scheme to combine the content and the context features without need of human interaction. Anran Wang et al. [22] proposed a modality and component aware feature fusion framework to combine regressoes for FV features and global CNN features. Heechul Jung et al. [23] used two models to extracts temporal appearance features and temporal facial landmark points, then adopted a new integration method to boost the performance of the facial expression recognition. Zuxuan Wuet al. [24] proposed a regularized DNN to lefusion layer is adopted to impose regularizatarn feature relationships an class relationships jointly. Xin Lu et al. [25] incorporated heterogeneous inputs generated from the image, which included a global view and a local view, and unified the feature learning and classifier training in a double-column deep convolutional neural network.

The basic idea for designing deep coarse component network is to extract global high-level features. There are many exist deep networks used for large-scale image datasets which have achieved outstanding performances on fine-grained classification. Their main disadvantage needs to train a large number of weights. So a complex system cannot be learned properly on a mid-scale fine-grained ship dataset. To deal with this issue, we present deep coarse component network with a moderate depth and a moderate number of parameters to avoid overfitting.

The deep coarse component network consists of three convolutional and two fully connected layers. The first convolutional layer is composed of 64 filters, filters of the first convolutional layer have size 7×7 and compute convolution at a global scale. Then the ReLU activation function is applied to the convolved patch. Next, we use a 2×2 max-pooling function in each response map with stride 2, which yields position invariance over the patches. We additionally apply convolution layer with a 5×5×64 kernel and the activation function ReLU. Max-pooling is performed the same way as in the first convolutional layer. The last convolutional layer comprises 64 filters of size 5×5. Finally, these output values are passed through the two fully connected layers and then classified using softmax. Batch Normalization (BN) is applied to the activations of these convolutional layers, following by the Rectified Linear Unit (ReLU) for non-linearity. The outputs of this network are finally normalized through a softmax step, computed as follows: f soft max ( v ) = [ exp ( v ) ∑ i n exp ( v ) ] T(1)

Where v is the score of filter i from previous layer and f_softmax is the corresponding output. This structure is summarized in Table 1. For training our network, the stochastic gradient descent method is used for optimization, and dropout [26] and weight decay methods are utilized for regularization.

The DCCN extract global high-level features, but may discard local details, such fine-scale object parts, which are significant cues to discriminate fine-grained categories. So we propose deep fine component network to capture local visual information.

In practical application, many datasets don’t have bounding box or part annotation, besides, human-defined regions may not be optimal for image classification. Bolei Zhou et al. [27] pointed out that convolutional units in convolutional neural networks can play the role of target detector without supervised information of target location, but the ability of the convolutional layer to locate the target is lost when adding the full connection layer to the target classification. They used the global average pooling (GAP) layer to instead of the full connection layer. And the use of GAP can identify the regions in the images which are significant to the classification only through a single forward-pass.

In deep fine component network, we use class activation maps (CAM) which can identify the discriminative regions of the image by projecting back the weights of the output layer on to the convolutional feature maps. We use a simple threshold technique to segment the heatmap, and generate a bounding box. Lijun Wang et al. [28] proposed that CNN features at different levels/depths have different properties. Such a top convolutional layer captures more abstract and high-level semantic features, and a lower layer provides more detailed local features which help to separate the target from distracters. So we map the box to the lower convolution layer of the convolution neural network to get the local features information.

By using a RoI pooling layer, we extracts a fixed-size feature map from the feature maps for the corresponding box, as shown in Fig. 2. The features inside the box are converted to a small feature map with a fixed spatial extent of of the RoI pooling layer. Then the small feature map is feed into a fully connected layer. A final fully connected layer computes the score of the input image for each class. In our framework, we train three deep fine component network by using bounding boxes with three scales separately. For the training of DFCN, we use transfer learning [29] to overcome the deficit of training samples for fine-grained categories. So we use ImageNet-trained CNN and fine-tune the parameters of CNN on target datasets.

OPEN IN VIEWER

Fig. 2

The framework of deep fine component network (DFCN).

A lot of method concatenated multiple extracted features together simply to form a high dimensional feature, usually result in limited performance since the intrinsic relations among the features extracted from the multiple models are overlooked. Therefore, we use an integration method [24] to fuse the features from DCCN and DFCN that fully leverages the complementary clues from various features.

Below we use a regularized variant which is able to accommodate the deep fusion process of multiple features from these trained networks. We utilize one additional layer for the fusion of all the features, as shown in Fig. 3. Let X = [ x 1 1 , ⋯ , x i j , ⋯ , x N M ] ∈ R D × N denote the M features from N input images.

OPEN IN VIEWER

Fig. 3

The fusion layer.

OPEN IN VIEWER

Fig. 4

Examples from (left) Vessel dataset, (center) FGVC-Aircraft dataset, and (right) Stanford Cars dataset used in our experiments.

This fusion layer can be written as the following:

Y F = σ ( ∑ m = 1 M W P m Y P m + B P )(2)

Where Y _F , Y P m denotes the output of the fusion layer and the last layer of feature extraction network respectively, W P m , B _P are the trainable parameters of the last layer of feature extraction network. Y _F uses a sigmoid function. In addition, as mentioned earlier, different features could also be complementary because they have distinct characteristics. Instead of concatenating multiple features simply, we specifically formulate an objective function that can regularize the fusion process to explore such correlations among the multiple features simultaneously. Finally we formulate the following objective as the following:

min W , Ψ L + λ 1 2 ( ∑ l = 1 P ∑ m = 1 M ∥ W l m ∥ F 2 + ∑ l = F L - 1 ∥ W l ∥ F 2 ) + λ 2 2 tr ( W p Ψ - 1 W P T ) s . t . Ψ ≥ 0 ,(3)

Where L = ∑ i = 1 N ℓ ( y ˆ i , y i ) is an tranditional objective, we add one regularization term. The matrix W _P represents the coefficients over all the features, and the matrix Ψ ∈ R^M×M is used to model the feature correlation. Note that the entries with large values in Ψ indicate strong feature correlations, while small-valued entries denote different features are less correlated. The coefficients λ₁ and λ₂ control the contributions from different regularization terms.

The learning speed of DCCN is different from DFCN, because they have different layers. To address this issue, we develop a learning algorithm for optimizing the model parameters efficiently in two steps. First, the DCCN and DFCN are trained seperatly. Next, we retrain the entire network.

(W _P , Ψ) are coupled with each other. Therefore, we adopt the alternative optimization method to iteratively minimize the objective with respect to W l m ( l = 1 , ⋯ , L , m = 1 , ⋯ , M ) and Ψ.

By fixing Ψ, we first consider the minimization problem over W l m . Denote G l m as the gradient with respect to W l m , the weight matrix for l-th layer and m-th feature is updated as: W l m = W l m - η G l m(4) where η is the step length of the gradient descent.

We then minimize the objective function over Ψ with other variables being fixed. The problem in Equation 3 degenerates to: min Ψ tr ( W P Ψ - 1 W P T ) , s . t . Ψ ⩾ 0 tr ( Ψ ) = 1 .(5)

The optimization pipeline is summarized in Algorithm 1.

4.1 Comparison to previous works

Implementation details In the DMCFN, all input images are resized to 224×224. We train DCCN and DFCN respectively. The architecture of DCCN is mentioned in Section 3.1. We initialize DFCN by the pre-trained CNN network from ImageNet as mentioned in Section 3.2. Then we replace the last fully connected layer with actual number of categories, and fine tune DFCN with our fine-grained datasets. For the initialization of DCCN, the weights are drawn from a Gaussian distribution of mean 0 and standard deviation 0.01. An equal learning rate 0.001 is set for all layers. Momentum and weight decay are set as 0.9, 0.0005 correspondingly. A 50% ‘Dropout’ regularization method is used to reduce overfitting in fully-connected layers.

Vessels dataset The results of fine-grained recognition on Vessels dataset is given in Table 3. Because the images of vessels dataset is collected from internet, there are no exist works having recognition results for comparison. So we fine-tune the three CNN architectures – AlexNet, VGGnet-16 and GoogLeNet for comparison. In our experiments, we fine-tune the weaker architecture of Krizhevsky et al. and still reach 92.4% accuracy. We fine-tune the architecture of their very deep GoogleLeNet to obtain 89.5% accuracy. In comparison, DFCMN uses conv4-3 features from VGGnet-16 obtains 93.5% accuracy. Our approach with lower layer CNN features outperforms the fine-tuned models by a significant margin.

FGVC-Aircraft The results of fine-grained recognition on FGVC-Aircraft are shown in Table 4. In the area of fine-grained recognition, there are many approaches relying on additional annotation like ground-truth part locations or bounding boxes. Some of them required annotation in training is distinguished from the annotation required at test time. As can be seen in the Table 4, our approach with the conv4-3 layer features from vgg-16 improves the work of Tsung-Yu Lin et al. [11] by 1.5% significantly. Besides, our approach with the conv3 layer features from AlexNet can reduce computation redundancy and achieves 83.3% accuracy with smaller architecture. It is important to note that our work only need image labels for training, which outperform these methods by a significant margin.

Stanford Cars The classification accuracy on Stanford Cars are summarized in Table 5. The previous works [16] [36] have remarkable results, which can achieve 92.8% and 91.3% accuracy respectively. Without object bounding-boxes the Jianlong Fu et al.[20] and Tsung-Yu Lin et al. [11] achieve accuracy of 92.5% and 92.6% respectively. Our method DMCFN- VGGNet (conv5-1) obtain the highest recognition accuracy 94.3% by leveraging the power of feature fusion, which integrates gloabal features and local features. In FGVC-Aircraft dataset, features from conv5-1 of VGGnet-16 does well on this dataset compared to features from conv4-3. We elaborate the effect of CNN features at different levels/depths in next experiment.

The previous works [28, 37] show that the low-level CNN features is more sensitive to appearance variations, and compared to high-level CNN features which can balance feature representativeness and generalization ability. Therefore, we verify this on these three datasets ranging from low-level to high-level vision. The classification accuracy using the feature maps for these three datasets are demonstrated in Table 6. In our experiments, we found that in FGVC-Aircraft datasets, features from conv4-3 does well on this dataset achieving 92.8% accuracy. However, in Stanford Cars dataset, features from conv5-1 achieving 94.3% accuracy. Maybe compared to aircrafts, cars are smaller and appear in a more cluttered background. The feature maps of conv5-1 may better separate car from non-car objects and preserve more middle-level information to achieve recognition that is more accurate.

There are a few methods proposed for fine-grained classification. Most of them directly concatenated multiple features together prior to classification. Such a direct combination does not adequately explore the relationship between global features and local features. Therefore, we conduct the experiment to test the fusion performance.

The λ₁ and λ₂ parameters of DMCFN used on three datasets are given in Table 7.

In order to test the ability of the fusion layer based on our method, we evaluate the performance using regularizations on fusion layers and concatenating multiple extracted features on three datasets based on Vgg-16. We plot the performance w.r.t. the level of features in Fig. 5. Using regularizations on fusion layer clearly achieves higher performance than concatenating features straightly. And we also found when the low level features are selected, the improvement of our method is even more significant.

OPEN IN VIEWER

Fig. 5

Classification performance on the three datasets using feature maps from different levels. We plot the results of DMCFN without regularization (red), DMCFN with regularization (blue).