Res2Net with spectral-spatial and channel attention for hyperspectral image classification

Abstract

Representing features at multiple scales is of great significance for hyperspectral image classification. However, the most existing methods improve the feature representation ability by extracting features with different resolutions. Moreover, the existing attention methods have not taken full advantage of the HSI data, and their receptive field sizes of artificial neurons in each layer are identical, while in neuroscience, the receptive field sizes of visual cortical neurons adapt to the neural stimulation. Therefore, in this paper, we propose a Res2Net with spectral-spatial and channel attention (SSCAR2N) for hyperspectral image classification. To effectively extract multi-scale features of HSI image at a more granular level while ensuring a small amount of calculation and low parameter redundancy, the Res2Net block is adopted. To further recalibrate the features from spectral, spatial and channel dimensions simultaneously, we propose a visual threefold (spectral, spatial and channel) attention mechanism, where a dynamic neuron selection mechanism that allows each neuron to adaptively adjust the size of its receptive fields based on the multiple scales of the input information is designed. The comparison experiments on three benchmark hyperspectral image data sets demonstrate that the proposed SSCAR2N outperforms several state-of-the-art deep learning based HSI classification methods.

Keywords

Hyperspectral image classification deep learning convolutional neural networks (CNNs)Res2Net visual attention mechanism

1 Introduction

Hyperspectral images (HSIs), acquired from special sensors, are composed of numerous narrow spectral bands, which are the electromagnetic radiation emitted, or reflected, by the scene under observation. With high spectral resolution, HSIs have absolute advantages for fine land-cover recognition. HSI classification which attempts to assign a specific class to each pixel has crucial and broad applications in the areas of globle environmental monitoring [1], national defense security [2], precision agriculture [3], materials analysis [4]. However, it has imposed serious challenges on HSI classification task, because HSI data has rich spectral information and high correlated spatial information. Over the last few decades, a vast amount of machine learning methods have been proposed for HSI classification. The earliest of these did classification by extracting the spectral signatures of hyperspectral data for feature representation and using classical classifiers, such as k-nearest neighbor approach [5], support vector machine (SVM) [6], random forest [7] and decision trees [8]. These methods are conceptually simple and easy to implement, but they ignored the spatial information of HSI. As HSI data lies in a geographical manifold and its neighboring pixels are spatial dependent, many methods are proposed to incorporate the spatial information for HSI classification, including the Markov Random Fields (MRFs) [9], sparse representation [10], partitional clustering techniques [11], extended morphological profiles (EMPs) [12] and multiple kernel learning [13]. Nevertheless, these conventional machine learning HSI classification methods, considered as only one or two layers processing methods, are all dependent on hand-craft descriptors which usually need expert knowledge.

In recent decade, deep learning, a subfield of machine learning, which is inspired by the structure and functions of the biological brain has shown more promising performance in HSI classification in comparison with the conventional machine learning methods, such as stacked autoencoder [14], convolutional neural network(CNN) [15], deep belief network [16], and recurrent neural network (RNN) [17]. In the deep learning based HSI classification methods, a hierarchical feature representation of HSIs is learned automatically by multiple non-linear networks, which exhibits better classification performance than the hand-craft descriptors [18, 19]. The early deep learning methods for HSIs classification need to organize the spatial information into a vector before training, and can not represent the spatial information efficiently [15 , 21]. To extract the spatial feature, Yue et al. firstly proposed a spectral-spatial deep learning architecture using 2D CNN and logical regression for HSI classification [22], which demonstrates that CNNs can effectively learn the local spatial structure features of the HSI data. Although 2D spectral-spatial classification methods incorporate the spatial information into the feature representation, their processings of data transformation tend to deform the spatial structure and can not learn both the spectral and spatial feature of HSI. Therefore, it is reasonable to extend the deep neural networks model to a 3D structure to learn the higher-level 3D spectral-spatial features which conform to the structure of the HSI data. In [23], Chen et al. proposed to employ several 3D convolutional and pooling layers to extract nonlinear, discriminant and invariant deep features of HSIs and verified the superiority of 3D spectral-spatial classification method over 1D spectral and 2D spectral-spatial classification methods. To achieve an efficient and accute classification, Paoletti et al. proposed a novel 3D-CNN composed of 5 layers with a specific strategy for dealing with the border pixels in the image for HSI classification [24]. To address the over-train problem caused by the insufficient training samples, in [25], a 3D generative adversarial network was proposed for HSI classification, which contains a generative CNN to generate fake inputs and a discriminative CNN to classify the real and fake inputs. More recently, in [26], the spectral bands are firstly clustered to obtain several different groups which are fed into multiple 3D-CNNs respectivley, and then the multiple 3D-CNNs are fused for spectral and spatial feature fusion to improve the precision of HSI classification.

The HSI classification accuracy declines due to the degradation problem of the deep CNN architecture. To address the problem, a considerable number of residual architecture based HSI classification methods were proposed [27 –29]. For example, Zhong et al. proposed a spectral-spatial residual network (SSRN), which uses two consecutive residual blocks to extract spectral-spatial features separately and employs residual connections to mitigate network degradation and improve HSI classification accuracy [29]. In [28], a depthwise separable residual neural network for HSI classification was proposed, which embeds the ResNet into a maximum posteriori framework to leverage the conditional random field model to preserve the class boundaries and edges. To reduce the high parameter numbers and feature redundancy, Paoletti et al. proposed a densely connected CNN for HSI classification, which enhances the generalization ability of the network and improves feature extraction capability of HSI [30]. In order to preserve the original spatial-spectral information in HSI, Yang et al. proposed a dual-channel network based on DenseNet to extract spectral and spatial features separately [31]. Zhang et al. proposed a multi-scale dense network (MSDN) for HSI classification, using dense networks in the horizontal dimension to extract deep features [32].

The aforementioned 3D-CNN based HSI classification methods all use single-scale convolution for the spectral-spatial feature extraction, of which the size of receptive field is fixed. It should be noted that relatively large receptive fields produce global and semantic features relatively easily, while relatively small receptive fields encode shape and geometric information quickly. In order to obtain receptive field of different sizes, researchers have developed deep methods with multi-layers to extract multi-scale fusion features, which have shown powerful feature representation ability for HSI classification [33]. In [34], a multi-scale 3D deep convolutional neural network was proposed, which could jointly learn both 2D multi-scale spatial feature and 1D spectral feature of HSI data. Subsequently, other forms of multi-scale convolutional neural networks were proposed, including the multi-scale densely 3D CNN [35] and the faster multi-scale capsule networks [36].

Nevertheless, the existing multi-scale feature extraction networks represent multi-scale features by increasing parallel convolution, which causes feature redundancy and increases the parameters amount. Therefore, recently, based on the Res2Net [37], which does not enhance the layer-wise multi-scale representation strength of CNNs, rather, improves the multi-scale representation ability at a more granular level, Zhang et al. proposed to consecutively extract spectral and spatial features of HSIs and an effective hinge cross-entropy loss function for HSI classification [38]. However, the network only uses the 3D CNN for feature extraction, which leads to feature redundancy. In [39], an advanced Capsule Network named RS-CapsNet was proposed for HSI classification, which uses Res2Net modules and small convolutional kernels to mitigate the problem of CapsNet’s large number of parameters, and adopts squeeze-and-excitation (SE) block to extract channel features. However, RS-CapsNet adds a convolution layer with small convolutional kernel of a single scale after each Res2Net module, which reduces the receptive field, and the SE module only focuses on spectral features, which is not facilitate extracting discriminative features.

In addition, recently, the attention mechanism was introduced in the feature representation of HSIs to suppress the clutter in HSIs and characterize the feature interactions of HSIs, attention mechanism can dynamically highlight the salient features. For instance, to suppress the influence of disturbing edge pixels, a spectral and spatial attention module is proposed and embedded in the feature extraction module [40]. To encode the long range spatial dependencies, Pande et al. proposed a hybrid attention composed of 1D and 2D CNNs which are attention masks for enhancing the spectral and spatial characteristics of HSIs [41]. In [42], different from the existing attention strategy, a spectral-spatial connected attention mechanism was proposed to process the raw HSI, which explores the diversity of spectral bands and the spatial relationship between the neighboring pixels. Hang et al. proposed a spectral attention subnetwork and a spatial attention subnetwork to enhance the learning capacity of CNNs for HSI classification [43]. Similar to [42], Zhu et al. proposed to directly process the raw HSI by spectral and spatial attention modules successively [44]. The spectral-spatial attention modules are also used for the following feature refinement and training acceleration.

Although great progress has been achieved in HSI classification by deep learning-based methods, there exist some problems unsolved.

The existing attention based HSI classification methods either only consider the optimization of channel dimension, or only consider the optimization of spatial-spectral dimension, have not fully exploited the 3D information of spatial, spectral and channel.

In the existing CNNs based attention models, the receptive field size of each artificial neuron layer is the same. But in neuroscience, the receptive field size of visual cortical neurons depends upon the stimulation, this is rarely considered in the convolutional networks construction [45].

To solve the aforementioned problems, in this paper, we propose Res2Net with spectral-spatial and channel attention (SSCAR2N) for HSI classification. To effectively extract the multi-scale features of hyperspectral images at a fine-grained level, Res2Net is employed which uses a set of smaller filter groups, each with several channels, to replace the traditional 3 × 3 filters, and also guarantees faster calculation speed. To further optimize the features obtianed by the Res2Net from the spectral-spatial and channel dimensions simultaneously, we propose a threefold attention block,i.e., spectral-spatial and channel attention block, where a dynamic neuron selection mechanism that allows each neuron to adaptively adjust the size of its receptive fields based on multiple scales of the input information is designed. Specifically, in the channel attention block, we design multi-scale selective kernel convolution in order to respectively adaptively weight the neurons and obtain a global and aggregated feature representation. In the spectral-spatial attention block, multi-scale 3D asymmetric convolutions are utilized to enhance the center backbone position weight and solve the problem of unequal features learning. Then to alleviate the degradation problem and gradient vanishing problem of the network, and enhance the generalization ability of the network, a residual connection is constructed which adds the optimized feature maps and the original feature map. To ensure the validity of feature, the network repeats the above operation. To the best of our knowledge, our study is the first exploration of the attention mechanism from the spectral-spatial and channel dimensions for HSI classification.

The contributions of this paper can be summarized as follows:

Different from the existing architectures for HSI classification which involve Res2Net, the proposed network optimizes the feature maps of Res2Net by a novel spectral-spatial and channel attention (SSCAR2N) block, and residual structure is built on the optimized features and the original features, which can enhance the generalization ability of the network, and the feature extraction and optimization are repeated for feature effectiveness.

To calibrate the features from spectral, spatial and channel dimensions simultaneously and adaptively, a spectral-spatial and channel attention block is designed. Inspired by neuroscience, a dynamic neuron selection mechanism that allows each neuron to adaptively adjust the size of its receptive fields based on multiple scales of the input information is developed.

In the channel attention block, we design multi-scale selective kernel convolution in order to respectively adaptively weight the neurons and obtain a global and aggregated feature representation. In the spectral-spatial attention block, multi-scale 3D asymmetric convolutions are utilized to enhance the center backbone position weight and solve the problem of unequal features learning.

The rest of this paper is laid out as follows. The innovative Res2Net with spectral-spatial and channel attention (SSCAR2N) approach is described in detail in Section 2. The experimental results are presented and discussed in Section 3. This study is finally summarized in Section 4.

2 The proposed method

In this section, the proposed SSCAR2N for HSI classification is described in detail, including the backbone structure of our network, the principle and implementation of the Res2Net block and threefold attention block.

2.1 The proposed architecture of SSCAR2N for HSI classification

In the proposed SSCAR2N, as shown in Fig. 1, initially, for dimensionality reduction, the HSI initial data is analyzed using principal component analysis (PCA) along the spectral dimension,which makes the spectral feature extraction more efficient without affecting the spatial dimension. To extract the correlational spectral and spatial information, the input of network is a 3D HSI cube that is a center pixel with its adjacent pixels in a square area (using mirror fill for edge pixels) selected from the HSI. Let x = d × d × band denote the 3D HSI cube, with d × d representing the spatial dimension and band representing the spectral bands. The cube data is firstly fed to a 3D convolution layer whose kernel size is 3 × 3 ×3. The batch normalization (BN) algorithm [46] and rectified linear units (ReLU) activation function [47] are used to process the feature map after convolution. The following formula describes it in detail, $y = H (F (x + b))$ (1)

Fig. 1

Flowchart of the proposed SSCAR2N for HSI classification.

where x is the input HSI cube data, b is the bias which is randomly initialized, the convolution operation is F (·), H (·) is a subsequent processing function composed of batch normalization (BN) and rectified linear units (ReLU), and y is output feature map of this convolution layer. Then, the feature map y is fed to the Res2Net block and the threefold attention block to perform the deep feature extraction and optimization. The corresponding detailed content are described in Section 3.2 and Section 3.3. The corresponding feature maps and the corresponding elements of the original feature map are added after optimization. By constructing the residual structure, the degradation problem and the gradient vanishing problem of the network are solved to some extent, and the generalization ability of the network is enhanced. To ensure the feature effectiveness, the network repeats the above operation. Finally, a convolution with the kernel size of 5 × 5 ×5 is used to reduce the spatial size. As a classifier, three fully connected layers (FC) and a softmax classifier are utilized. Dropout technology, which can efficiently reduce over-fitting without introducing a significant number of parameters, is employed in the fully-connected layer. As a loss function, the category cross entropy is used.

2.2 The Res2Net block

In order to effectively exploit the multi-scale features of HSI at a more granular level, the Res2Net module is adopted, which is in contrast to the existing methods that explore the multi-scale features by layer-wise operations. The architecture of the Res2Net block is shown in Fig. 2.

Fig. 2

The architecture of the Res2Net block.

Suppose the input feature map of Res2Net block has c channels,i.e. y = d × d × band × c, then we divide y into s equal subsets along the channel dimension. Each subset y_i (1 ≤ i ≤ s) has the same spatial and spectral size as the input data y, but has 1/s number of channels in comparison with the input data y. Then, to obtain receptive field of different scales, the subsets are processed and combined differently, specifically, except for y₁, each y_i is operated by a convolution kernel of size 3 × 3 ×3 and connected in a hierarchical residual-like style, which results in multiple diverse receptive fields at a more granular level. The individual output of the Res2Net block z_i (1 ≤ i ≤ s) are,

$z_{i} = {\begin{matrix} y_{i} & i = 1; \\ F (y_{i}) & i = 2; \\ F (y_{i} + z_{i - 1}) & 2 < i \leq s . \end{matrix}$ (2) where F (·) is the convolution operation. Finally, to fuse the feature at different scales, all the output z_i are concatenated to obtain the final output of the block.

In this block, division and concatenation strategies allow the network to fit the residual, thereby can enhance the features convolution processing more effectively. The omission of the convolution of the first division reduces the number of parameters, and can also be regarded as a form of feature reuse. The Res2Net block is the key part of our network, which is an architecture with strong multi-scale feature extraction capability and low computational complexity similar to a single-scale.

2.3 Spectral-spatial and channel attention module

To further recalibrate the features extracted by Res2Net block, we attempt to mine the interdependence relationship between the spatial, spectral, and channel in the feature map and optimize the spatial, spectral, and channel features simultaneously and adaptively as the hyperspectral images have abundant spectral and spatial information. To this end, a spectral-spatial and channel attention module (SSCA) is proposed for HSI according to the characters of HSI data. SSCA consists of two parallel attention subblocks: spatial-spectral attention and channel attention subblocks, which use the adaptive selection mechanism of multi-branch to realize the adaptive weighting of neurons respectively and obtain a global, comprehensive selection weight representation. Fig. 3 depicts the structure of the spectral-spatial and channel attention mechanism block.

Fig. 3

The structure of the 3D asymmetric convolutions attention mechanism block.

To improve the classification performance of our network, in the spatial-spectral attention subblock, we leverage the asymmetric convolution [48] to enhance the representational ability of a standard square-kernel. Specifically, we construct three parallel layers with 3 × 1 ×1, 1 × 3 ×1 and 3 × 3 ×3 kernels, among which the 3 × 1 ×1 and 1 × 3 ×1 kernels are non-square and referred to as the asymmetric convolutional layers. The asymmetric convolutional layers can enhance the learned center backbone position weight through horizontal and vertical kernels, which fit the nature of square kernels and derive no extra computation time. Then, to avoid the influence of the channel dimension on the spatial-spectral features we use a convolution with a convolution kernel size of 1 × 1 ×1, which also reduces the redundancy. Finally, the outputs of three parallel layers are summed up. A sigmoid operator is used for spatial-spectral-wise activation. The designed spatial-spectral attention subblock reliably extracts the feature effectively, which is beneficial to enhance the spatial feature of HSI.

In the channel attention subblock, to extract the multi-scale convolutional feature, two parallel convolution operations with 3 × 3 ×3 and 5 × 5 ×5 kernels and activation operations are applied to the input feature map x₀ = d × d × band × c. Then we obtain u₁ and u₂, $u_{1} = H (F_{1} (x_{0}))$ (3) $u_{2} = H (F_{2} (x_{0}))$ (4) To enable the neurons to adaptively adjust their sizes according to the stimulus content, we embed the global information by using global average pooling F_gap (u₁) and global maximum pooling F_gmp (u₂) to generate channel-wise statistics $s_{1} \in R^{c}$ and $s_{2} \in R^{c}$ , which are calculated by $\begin{matrix} s_{1} & = F_{gap} (u_{1}) \\ = \frac{1}{d \times d \times band} \sum_{i = 1}^{d} \sum_{j = 1}^{d} \sum_{k = 1}^{band} u_{1} (i, j, k) \end{matrix}$ (5) $s_{2} = F_{gmp} (u_{2}) = \max (u_{2} (i, j, k)) .$ (6) To guide the feature selections of feature map adaptively and efficiently, two fully connected layers, which are designed for dimension reduction and feature reconstruction, and a softmax function for feature fusion are employed. Then the attention weight vector of channel dimension v_a and v_b are obtained by fusing the weights of two scales, which are calculated by $v_{a} = softmax (F_{fc} (σ (F_{fc} (s_{1}))) + F_{fc} (σ (F_{fc} (s_{2}))))$ (7) $v_{b} = 1 - v_{a}$ (8) where σ is the ReLU activation function. The final feature map V is obtained through the attention weights (on various kernels): $V = v_{a} \cdot u_{1} + v_{b} \cdot u_{2},$ (9) Consequently, the spatial-spectral attention subblock can adaptively select the multi-scale features.

Finally, the spatial-spectral attention enhanced feature map U and the channel attention enhanced feature map V are fused with each other by BN operation to obtain the final output x₁ of the threefold attention block.

3 Experiments and analysis

To comprehensively evaluate the performance of our proposed SSCAR2N for HSI classification, exhaustive quantitative and qualitative experiments are conducted on three standard HSI benchmark data sets, which are Indian Pines, Pavia University and Salinas Valley, in comparison with five state-of-the-art networks, 2DCNNs [15], 3DCNNs [49], ResNet [29], MSDN-SA [50], Res2Net. Four metrics, class accuracy, overall accuracy (OA), average accuracy (AA), kappa coefficient (κ) are used to measure the pros and cons of our approach. Overall accuracy (OA) is the ratio of the number of correctly predicted samples to the total number of samples. Average accuracy (AA) is the average of the ratio of the number of correctly predicted samples per class to the total number of samples per class. Kappa coefficient (κ) is used to judge the degree of image consistency. The networks are written in the TensorFlow 2.1.0 with Python 3.7 architecture. Moreover, all the experiments are executed on i7-8700 CPU and NVIDIA GTX1060 GPU.

3.1 Experimental data sets

Indian Pines [51] Indian Pines image was collected by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS). The original data set has 224 spectral bands with a wavelength range of 0.4 to 2.5 μm and includes 145×145 pixels with a spatial resolution of 20m per pixel. This data set has 10249 labeled pixels and 16 classes in total. Fig. 4 are the false color image, the corresponding ground truth map and the number of available samples.

Fig. 4

False color composite, ground reference map, and a number of available samples of the Indian Pines data sets.

Pavia University [51] Pavia University was collected in the urban area around the University of Pavia in Italy through the Reflective Optical System Imaging Spectrometer (ROSIS). The ROSIS optical sensor captured a total of 115 spectral bands in the range of 0.43 to 0.86 μm, of which 12 noise bands are removed in the experiments. The spatial size of this data set is 610×340 pixels, and the spatial resolution is 1.3m per pixel. This scene contains a total of 9 classes. The false color, reference ground truth image and a number of available samples are shown in Fig. 5.

Fig. 5

False color composite, ground reference map, and a number of available samples of the Pavia University data sets.

Salinas Valley [51] Salinas Valley was acquired by the AVIRIS instrument with 224-band over Salinas Valley, California. The available data set is composed of 204 bands after the low signal-to-noise ratio (SNR) bands were removed. The HSI data set contains 512 × 217 pixels with a high spatial resolution of 3.7-meter pixels and total 16 classes. Fig. 6 are the false color image, the corresponding ground truth map and a number of available samples.

Fig. 6

False color composite, ground reference map, and a number of available samples of the Salinas data sets.

3.2 Experimental setup

The compared methods in our experiments are as follows:

2DCNNs [15]: A deep convolutional neural network was employed to classify hyperspectral images directly in the spatial domain.

3DCNNs [49]: A 3D CNNs method for HSI classification. CNNs are used to encode the spectral-spatial information of pixels, and an MLP is used to perform the classification task.

ResNet [29]: A residual structure spectral-spatial 3D deep learning network that successfully mitigates over-fitting.

MSDN-SA [50]: A densely connected framework for 3D CNNs with spectral-wise attention mechanism.

Res2Net: Our proposed method without the spectral-spatial and channel attention mechanism block.

For all the adopted three data sets mentioned in Section 3.1, 50 samples per class are randomly selected for training, and a half of the total is selected for training if the corresponding class contains less than 50 samples. The remaining samples in each class are regarded as unlabeled examples during the training process and are used as the test set to evaluate the classification performance.

The dimension of the 3D cube is fixed at 12 × 12 × 12 in all experiments, with the spectral bands reduced to 12 using PCA. The number of convolution kernels in the first layer is set to 32, and the convolution mode is adopted ’valid’. In the first Res2Net block, the number of kernels in each convolution has a quarter of the number of kernels of the first convolution, and the convolution method uses ’same’. In the first channel attention module, the number of channels in the convolution process is 32. We use a 2-layer fully connected layer in the channel attention module to reduce the number of channels to half first and then restore the original size. In the spatial-spectral attention module, the number of feature maps remains unchanged, and convolution kernels are equal to the input data channels. The final fully connected layer is composed of 512 neurons and 256 neurons, sigmoid is used as the activation function, and the dropout rate is 0.5.

We utilize Adam as the network optimizer in this experiment. The optimizer’s learning rate is set to 1e-3 in order to ensure that the network has a fast convergence speed. The training epoch represents the number of times the network has been iterated. The experiment’s number of training epochs is 35, and the number of training batches is 128. Each iteration procedure will update the parameters of the whole network. Because each pick is made at random, each categorization will provide different results. We do five trials for each class in each of the three data sets to guarantee that the experiments are accurate. We determined the mean precision and standard deviation for each class of the six techniques in the three data sets, as well as the overall accuracy (OA), average accuracy (AA), and kappa coefficients (κ) for each data set.

3.3 Experimental results and discussion

1) Indian Pines: Table 1 shows the quantitative comparative findings of the classification accuracy on the Indian Pines data set. As the Indian Pines data set’s sample distribution is unbalanced. Some categories such as Corn-notill has 1428 sample, while Oats only has 20 samples. Moreover, its spatial sacle is only 145 × 145, which is the smallest among the three data sets. These make it a challenging data set. Table 1 suggests that the standard 3D CNN performs significantly better than CNNs. The superiority of the ResNet and MSDN-SA methods demonstrate that the residual 3D CNN’s spatial-spectral feature is more powerful than the standard 3D CNN’s. It also shows that SSCA has a good effect from the results of the proposed SSCAR2N in comparison with the ResNet, MSDN-SA and Res2Net methods. Among the six methods, the proposed SSCAR2N acquires the highest OA, AA, and κ, which are 94.13%, 94.54%, and 93.27% respectively. To display the results of the visual categorization, the classification maps generated by the proposed method in comparison with other different methods are shown in Fig. 7. From Fig. 7, we can see that the classification map produced by SSCAR2N method is closest to the ground truth map, and many regions of the classification maps achieved by SSCAR2N method are obviously less noisy than those of the CNNs, 3D CNNs, ResNet, MSDN-SA and Res2Net.

Table 1
Classification accuracy comparison for Indian Pines data set

Class Samples Methods

Train/Test CNNs 3DCNNs ResNet MSDN-SA Res2Net SSCAR2N

Alfalfa 23/23 70.43 ± 12.45 99.13 ± 1.94 96.52 ± 3.64 99.13 ± 1.94 100.0 ± 0.00 100.0 ± 0.00

Corn-notill 50/1378 37.65 ± 8.38 83.06 ± 7.79 82.08 ± 9.04 86.04 ± 5.51 87.11 ± 3.36 90.19 ± 2.09

Corn-min 50/780 52.36 ± 3.27 75.72 ± 37.39 95.26 ± 2.72 93.92 ± 3.14 96.18 ± 1.80 89.21 ± 8.88

Corn 50/187 77.01 ± 6.01 81.07 ± 26.88 98.93 ± 1.56 96.79 ± 2.65 99.47 ± 0.93 97.86 ± 2.42

Grass-Pasture 50/433 83.51 ± 5.07 93.95 ± 5.65 97.64 ± 0.34 96.40 ± 2.04 95.20 ± 2.48 95.94 ± 2.52

Grass-Trees 50/680 92.71 ± 0.89 95.65 ± 4.98 98.62 ± 0.89 98.97 ± 0.75 99.41 ± 0.23 97.76 ± 1.57

Grass-Pasture-mowed 14/14 85.71 ± 8.75 100.0 ± 0.00 100.0 ± 0.00 100.0 ± 0.00 98.57 ± 3.19 100.0 ± 0.00

Hay-windrowed 50/428 93.74 ± 3.61 99.21 ± 1.32 99.91 ± 0.21 99.53 ± 1.04 99.67 ± 0.73 99.77 ± 0.52

Oats 10/10 72.00 ± 20.49 98.00 ± 4.47 100.0 ± 0.00 100.0 ± 0.00 100.0 ± 0.00 100.0 ± 0.00

Soybeans-notill 50/922 54.56 ± 12.86 91.11 ± 7.52 95.94 ± 2.14 90.13 ± 8.30 94.82 ± 1.63 94.01 ± 2.64

Soybeans-min 50/2405 51.26 ± 7.14 81.72 ± 9.66 83.11 ± 3.53 85.14 ± 4.11 84.40 ± 2.73 90.79 ± 3.33

Soybeans-clean 50/543 52.49 ± 8.18 88.21 ± 4.94 89.17 ± 6.21 85.23 ± 7.11 92.71 ± 1.70 97.24 ± 1.79

Wheat 50/155 98.58 ± 1.15 99.35 ± 1.44 100.0 ± 0.00 100.0 ± 0.00 100.0 ± 0.00 99.87 ± 0.29

Woods 50/1215 84.08 ± 3.06 98.78 ± 0.68 97.05 ± 1.49 98.77 ± 0.57 97.22 ± 0.88 99.05 ± 0.58

Building-Grass-Trees-Drives 50/336 66.07 ± 4.22 92.92 ± 10.80 99.52 ± 0.54 99.70 ± 0.52 98.69 ± 1.91 99.94 ± 0.13

Stone-steel Towers 46/47 95.74 ± 3.98 99.57 ± 0.95 99.15 ± 1.77 98.72 ± 1.90 99.15 ± 1.17 99.57 ± 0.95

OA(%) 62.39 ± 1.74 88.04 ± 2.63 91.14 ± 1.02 91.46 ± 0.80 92.30 ± 0.63 94.13 ± 0.57

AA(%) 61.02 ± 3.25 91.22 ± 0.98 91.80 ± 0.73 91.48 ± 0.97 92.83 ± 1.92 94.54 ± 0.65

κ×100 57.45 ± 1.71 86.25 ± 3.24 89.88 ± 1.19 90.23 ± 0.93 91.20 ± 0.71 93.27 ± 0.66

Class	Samples	Methods
Alfalfa	23/23	70.43 ± 12.45	99.13 ± 1.94	96.52 ± 3.64	99.13 ± 1.94	100.0 ± 0.00	100.0 ± 0.00
Corn-notill	50/1378	37.65 ± 8.38	83.06 ± 7.79	82.08 ± 9.04	86.04 ± 5.51	87.11 ± 3.36	90.19 ± 2.09
Corn-min	50/780	52.36 ± 3.27	75.72 ± 37.39	95.26 ± 2.72	93.92 ± 3.14	96.18 ± 1.80	89.21 ± 8.88
Corn	50/187	77.01 ± 6.01	81.07 ± 26.88	98.93 ± 1.56	96.79 ± 2.65	99.47 ± 0.93	97.86 ± 2.42
Grass-Pasture	50/433	83.51 ± 5.07	93.95 ± 5.65	97.64 ± 0.34	96.40 ± 2.04	95.20 ± 2.48	95.94 ± 2.52
Grass-Trees	50/680	92.71 ± 0.89	95.65 ± 4.98	98.62 ± 0.89	98.97 ± 0.75	99.41 ± 0.23	97.76 ± 1.57
Grass-Pasture-mowed	14/14	85.71 ± 8.75	100.0 ± 0.00	100.0 ± 0.00	100.0 ± 0.00	98.57 ± 3.19	100.0 ± 0.00
Hay-windrowed	50/428	93.74 ± 3.61	99.21 ± 1.32	99.91 ± 0.21	99.53 ± 1.04	99.67 ± 0.73	99.77 ± 0.52
Oats	10/10	72.00 ± 20.49	98.00 ± 4.47	100.0 ± 0.00	100.0 ± 0.00	100.0 ± 0.00	100.0 ± 0.00
Soybeans-notill	50/922	54.56 ± 12.86	91.11 ± 7.52	95.94 ± 2.14	90.13 ± 8.30	94.82 ± 1.63	94.01 ± 2.64
Soybeans-min	50/2405	51.26 ± 7.14	81.72 ± 9.66	83.11 ± 3.53	85.14 ± 4.11	84.40 ± 2.73	90.79 ± 3.33
Soybeans-clean	50/543	52.49 ± 8.18	88.21 ± 4.94	89.17 ± 6.21	85.23 ± 7.11	92.71 ± 1.70	97.24 ± 1.79
Wheat	50/155	98.58 ± 1.15	99.35 ± 1.44	100.0 ± 0.00	100.0 ± 0.00	100.0 ± 0.00	99.87 ± 0.29
Woods	50/1215	84.08 ± 3.06	98.78 ± 0.68	97.05 ± 1.49	98.77 ± 0.57	97.22 ± 0.88	99.05 ± 0.58
Building-Grass-Trees-Drives	50/336	66.07 ± 4.22	92.92 ± 10.80	99.52 ± 0.54	99.70 ± 0.52	98.69 ± 1.91	99.94 ± 0.13
Stone-steel Towers	46/47	95.74 ± 3.98	99.57 ± 0.95	99.15 ± 1.77	98.72 ± 1.90	99.15 ± 1.17	99.57 ± 0.95
OA(%)		62.39 ± 1.74	88.04 ± 2.63	91.14 ± 1.02	91.46 ± 0.80	92.30 ± 0.63	94.13 ± 0.57
AA(%)		61.02 ± 3.25	91.22 ± 0.98	91.80 ± 0.73	91.48 ± 0.97	92.83 ± 1.92	94.54 ± 0.65
κ×100		57.45 ± 1.71	86.25 ± 3.24	89.88 ± 1.19	90.23 ± 0.93	91.20 ± 0.71	93.27 ± 0.66

Fig. 7

HSI classification maps of Indian Pines. (a) False-color image. (b) Ground truth map. (c) CNN (62.39%). (d) 3DCNN (88.04%). (e) ResNet (91.14%). (f) MSDN-SA (91.46%). (g) Res2Net (92.30%). (h) SSCAR2N (94.13%).

2) Pavia University: The comparison results of classification accuracy of Pavia University data set are reported in Table 2, from which we can see that the proposed SSCAR2N achieved the best OA, AA, and κ, which are 95.94%, 94.73% and 94.60% respectively. The SSCAR2N’s categorization results for each class were far superior than those of others, like the three classifications Bricks, Trees, and Asphalt. The OA, AA, and κ all improve when compared to the Res2Net approach without the attention module, indicating the usefulness of the suggested tripartite attention block. With regard to the qualitatively classification accuracies, the classification maps of different methods are illustrated in Fig. 8. According to Fig. 8, the suggested method’s classification maps are the most similar to the ground truth map. In our feature map, for example, the yellow region in the middle of the Pavia University data set, which represents the class ’Baresoil’, is pure yellow with no other colors, indicating that there are no misclassified pixels.

Table 2

Classification accuracy comparison for Pavia University data set

Class	Samples	Methods
	Train/Test	CNNs	3DCNNs	ResNet	MSDN-SA	Res2Net	SSCAR2N
Asphalt	50/6581	85.63 ± 7.59	89.07 ± 6.33	88.40 ± 7.08	83.61 ± 6.50	89.06 ± 7.70	92.05 ± 4.70
Meadows	50/18599	80.36 ± 6.91	91.17 ± 4.63	93.43 ± 2.92	97.37 ± 2.42	96.63 ± 3.39	98.18 ± 1.30
Gravel	50/2049	54.83 ± 22.37	98.80 ± 1.61	97.68 ± 3.49	95.60 ± 6.57	99.11 ± 0.86	94.69 ± 7.64
Trees	50/3014	87.70 ± 4.09	94.17 ± 2.24	92.63 ± 5.98	90.75 ± 8.69	93.31 ± 5.53	97.26 ± 1.42
Sheets	50/1295	99.24 ± 0.56	100.0 ± 0.00	99.98 ± 0.03	100.0 ± 0.00	100.0 ± 0.00	100.0 ± 0.00
Baresoil	50/4979	86.72 ± 8.84	96.49 ± 5.16	98.37 ± 1.08	97.46 ± 4.04	92.85 ± 4.60	92.73 ± 8.80
Bitumen	50/1280	78.78 ± 10.98	96.33 ± 5.82	95.81 ± 3.91	99.50 ± 0.74	98.41 ± 1.90	98.17 ± 2.22
Bricks	50/3632	82.73 ± 17.93	68.05 ± 14.11	71.93 ± 13.75	85.88 ± 11.90	72.21 ± 12.29	92.36 ± 5.75
Shadows	50/897	99.89 ± 0.14	99.73 ± 0.15	99.44 ± 0.76	98.39 ± 2.80	99.49 ± 0.57	99.82 ± 0.10
OA(%)		82.36 ± 2.47	90.67 ± 3.23	91.93 ± 2.41	93.86 ± 1.39	93.01 ± 1.52	95.94 ± 1.40
AA(%)		82.44 ± 1.75	89.78 ± 2.48	91.24 ± 1.75	91.32 ± 2.25	92.03 ± 1.84	94.73 ± 1.85
κ×100		77.26 ± 2.83	87.85 ± 4.04	89.41 ± 3.15	91.88 ± 1.85	90.76 ± 1.93	94.60 ± 1.88

Fig. 8

Classification maps provided for the Pavia University data set by different methods. (a) A false color map. (b) The ground truth map. (c) CNN (82.36%). (d) 3DCNN (90.67%). (e) ResNet (91.93%). (f)MSDN-SA (93.86%). (g) Res2Net (93.01%). (h) SSCAR2N (95.94%).

3) Salinas Valley: Table 3 and Fig. 9 show the classification maps of all approaches’ classification results for the Salinas Valley data set, respectively. The suggested SSCAR2N approach produced the best OA, AA, and κ, as can be observed. The OA, AA, and κ were 94.10%, 97.06% and 93.43% respectively from the Table 3. In this data set, ’Grapes-untrained’ and ’Vinyard-untrained’ are difficult to distinguish. It can be seen that our method has a classification accuracy of 84.97% and 82.27% in these two classes. It has better classification results than other methods. For the visual results, from Fig. 9, we can see that the data in the orange part and the dark blue part in the figure are easily confused, but the result graph of our method is closer to the ground truth map.

Table 3

Classification results for the Salinas data set

Class	Samples	Methods
	Train/Test	CNNs	3DCNNs	ResNet	MSDN-SA	Res2Net	SSCAR2N
Brocoli-green-weeds-1	50/1959	91.34 ± 18.06	99.91 ± 0.21	99.99 ± 0.02	99.87 ± 0.19	99.87 ± 0.19	98.24 ± 2.26
Brocoli-green-weeds-2	50/3676	92.23 ± 8.81	99.61 ± 0.49	99.24 ± 1.01	99.79 ± 0.34	99.95 ± 0.10	99.99 ± 0.01
Fallow	50/1926	94.91 ± 5.63	99.84 ± 0.25	99.42 ± 1.19	99.73 ± 0.26	99.83 ± 0.24	99.95 ± 0.12
Fallow-rough-plow	50/1344	98.97 ± 1.19	99.78 ± 0.32	99.90 ± 0.08	99.69 ± 0.66	99.63 ± 0.64	99.88 ± 0.23
Fallow-smooth	50/2628	96.87 ± 1.63	98.01 ± 0.71	96.90 ± 2.74	98.73 ± 1.40	98.68 ± 1.36	98.74 ± 0.83
Stubble	50/3909	98.65 ± 1.16	99.99 ± 0.02	100.0 ± 0.00	99.99 ± 0.01	99.99 ± 0.01	100.0 ± 0.00
Celery	50/3529	99.48 ± 0.08	99.97 ± 0.05	99.74 ± 0.17	99.86 ± 0.17	99.76 ± 0.24	99.78 ± 0.39
Grapes-untrained	50/11221	70.00 ± 13.45	67.69 ± 23.82	69.12 ± 27.18	71.00 ± 20.71	81.03 ± 12.76	84.97 ± 4.31
Soil-vinyard-develop	50/6153	96.66 ± 1.96	99.90 ± 0.23	99.93 ± 0.15	99.31 ± 1.30	99.31 ± 1.30	100.0 ± 0.00
Corn-senesced-green-weeds	50/3228	89.80 ± 1.69	97.09 ± 1.76	98.31 ± 1.45	98.71 ± 0.89	98.55 ± 1.12	97.11 ± 1.48
Lettuce-romaine-4wk	50/1018	71.65 ± 29.74	99.31 ± 1.00	99.57 ± 0.32	98.82 ± 1.11	98.80 ± 1.08	99.86 ± 0.16
Lettuce-romaine-5wk	50/1877	99.39 ± 0.68	99.83 ± 0.38	99.96 ± 0.10	99.97 ± 0.05	99.97 ± 0.05	100.0 ± 0.00
Lettuce-romaine-6wk	50/866	95.73 ± 4.70	99.84 ± 0.25	99.84 ± 0.19	99.91 ± 0.21	99.88 ± 0.26	99.98 ± 0.05
Lettuce-romaine-7wk	50/1020	96.12 ± 1.80	99.78 ± 0.28	99.57 ± 0.38	99.65 ± 0.43	99.67 ± 0.41	99.67 ± 0.37
Vinyard-untrained	50/7218	64.56 ± 13.37	73.89 ± 29.62	76.30 ± 32.46	79.54 ± 22.36	74.44 ± 19.57	82.27 ± 4.67
Vinyard-vertical-trellis	50/1757	97.30 ± 1.17	99.57 ± 0.50	98.95 ± 2.06	99.83 ± 0.61	99.90 ± 0.14	99.80 ± 0.22
OA(%)		85.74 ± 0.92	89.30 ± 1.72	89.88 ± 3.79	90.82 ± 2.33	92.24 ± 1.44	94.10 ± 0.25
AA(%)		90.02 ± 0.78	96.15 ± 0.34	96.32 ± 3.79	96.50 ± 0.39	96.68 ± 0.53	97.06 ± 0.22
κ×100		84.14 ± 0.99	88.11 ± 1.85	88.76 ± 4.16	89.81 ± 2.55	91.36 ± 1.60	93.43 ± 0.27

Fig. 9

Classification maps provided for the Salinas data set by different methods. (a) A false color map. (b) The ground truth map. (c) CNN (85.74%). (d) 3DCNN (89.30%). (e) ResNet (89.88%). (f) MSDN-SA (90.82%). (g) Res2Net (92.24%). (h) SSCAR2N (94.10%).

4) Parameters Analysis: To further figure out the effectiveness of parameters in our deep learning framework, more experiments are carried out. The first set of experiments where different numbers of the spectral band are selected from the three data sets via PCA while the spatial sizes are fixed, is to explore the influence of PCA dimension. The corresponding overall classification accuracies (OA) of the three data sets are provided in Table 4. From the table, we can see that large spectral dimension contains more features, which is beneficial to network training. However, due to the influence of noise, it is necessary to reduce the spectral dimension properly.

Table 4

OA (%) of the proposed method with different PCA Band

PCA Band	Indian Pines	Pavia University	Salinas
12	94.13	95.94	94.10
20	96.78	98.03	97.31
40	96.43	99.12	96.67
80	93.54	95.34	93.89

To see the effect of the spatial size of the input image patches, another set of experiments is conducted on three data sets for comparing classification accuracy of different spatial sizes, while the spectral dimension is fixed to 12. The overall classification accuracies (OA) results are provided in Table 5. From the table, we can observe that 12 × 12 and 16 × 16 have a better performance. This is because large image patches will introduce excessive noise, and small image patches may not include enough information. Therefore, after comprehensive experiments analysis, the 12 × 12 spatial size is adopted in our experiments.

Table 5

The OA (%) of the proposed method with different spatial sizes

Spatial Size	Indian Pines	Pavia University	Salinas
8 × 8	86.56	92.58	90.55
12 × 12	94.13	95.94	94.10
16 × 16	96.43	97.21	95.34
20 × 20	96.78	98.94	93.29

We also conducted further experiments to explore how the ultimate model was affected by the number of samples in the training set. By modifying the proportion of samples in the training set to the total data set samples, we compared the five methods. Firstly, in the Indian Pines data set, take 4%to 19% of the samples as the training set, and choose 1% to 6% of the samples as the training set in the Pavia University data set and the Salinas data set. Experiments were carried out to obtain the OA results shown in Figs. 10–12. Further analysis of the experimental results shows that among the five methods, the OA results of the data set improve with the increase of training samples proportion. Compared with other state-of-the-art methods, our method achieves better OA results.

Fig. 10

OA (%) of different training set sizes (a) Indian Pines data set.

Fig. 11

OA (%) of different training set sizes (b) Pavia University data set.

Fig. 12

OA (%) of different training set sizes (c) Salinas data set.

4 Conclusions

Hyperspectral images have the dilemma of high dimensional spectral versus limited samples. The discriminative feature representation is of great importance for HSI classification. However, the traditional methods extracted the multi-scale feature of HSI data by the layer-wise multi-scale architecture which explores the multi-scale features with different resolutions and consumes a lot of computation. Moreover, the existing attention methods have not considered optimizing the extracted features from spectral, spatial and channel aspects simultaneously. Therefore, in this paper, a novel deep learning framework which consists of Res2Net and threefold attention mechanism is proposed for HSI classification. The proposed architecture explores the multi-scale features of HSI data at a fine-grained level by the Res2Net which is orthogonal to the traditional methods. To further enhance the feature representation, a threefold attention block is designed to select the visual discriminative and important features from the spectral-spatial and channel dimensions adaptively and simultaneously. The experiments on three benchmark hyperspectral data sets have shown the superiorities of the proposed approach in comparison with the state-of-the-art deep methods.

The advantage of such an approach is that the designed threefold attention block can solve the problem of unequal features learning and adaptively optimizes the multi-scale features from the spectral-spatial and channel dimensions simultaneously in which the neurons are dynamically and adaptively weighted. The proposed backbone is general and can be extended to other deep learning-based feature engineering applications. The potential limitation of our approach is the computation caused by the adopted multiple 3D convolution layers in the architecture. As future work, we will resort to new strategy to improve our threefold attention mechanism and the computational efficiency of the architecture.

Footnotes

Acknowledgments

The authors would like to thank the anonymous referees for their constructive comments which have helped improve the paper. The research is supported by the National Natural Science Foundation of China (Nos. 61860206004, 72071001, U20B2068), Natural Science Foundation of Anhui Province (Nos. 2008085MG226, 2008085QG334), Natural Science Foundation for the Higher Education Institutions of Anhui Province (No. KJ2021A0038).

References

Richard Ellis

and Peter Scott

, Evaluation of hyperspectral remote sensing as a means of environmental monitoring in the St. Austell China clay (kaolin) region, Cornwall, UK, Remote Sensing of Environment 93(1-2) (2004), 118–130.

Yuan Yuan , Qi Wang and Guokang Zhu , Fast hyperspectral anomaly detection via high-order 2-d crossing filter, IEEE Transactions on Geoscience and Remote Sensing 53(2) (2014), 620–630.

Stanton Martin

and Thomas George

, Applications of hyperspectral image analysis for precision agriculture, In Micro-and Nanotechnology Sensors, Systems, and Applications X, volume 10639, page 1063916. International Society for Optics and Photonics, 2018.

Saeid Homayouni and Michel Roux , Hyperspectral image analysis for material mapping using spectral matching, In ISPRS Congress Proceedings, volume 20, page 28, 2004.

Enrico Blanzieri and Farid Melgani , Nearest neighbor classification of remote sensing images with the maximal margin principle, IEEE Transactions on Geoscience and Remote Sensing 46(6) (2008), 1804–1811.

Farid Melgani and Lorenzo Bruzzone , Classification of hyperspectral remote sensing images with support vector machines, IEEE Transactions on Geoscience and Remote Sensing 42(8) (2004), 1778–1790.

Jisoo Ham , Yangchi Chen , Melba Crawford

and Joydeep Ghosh

, Investigation of the random forest framework for classification of hyperspectral data, IEEE Transactions on Geoscience and Remote Sensing 43(3) (2005), 492–501.

Delalieux

, Ben Somers , Haest

, Spanhove

, Vanden Borre

and Mücher

C.A.

, Heathland conservation status mapping through integration of hyperspectral mixture analysis and decision tree classifiers, Remote Sensing of Environment 126 (2012), 222–231.

Yuliya Tarabalka , Mathieu Fauvel , Jocelyn Chanussot and Jón Atli Benediktsson , SVM-and MRF-based method for accurate classification of hyperspectral images, IEEE Geo-science and Remote Sensing Letters 7(4) (2010), 736–740.

10.

Jianjun Liu , Zebin Wu , Zhihui Wei , Liang Xiao and Le Sun , Spatial-spectral kernel sparse representation for hyperspectral image classification, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 6(6) (2013), 2462–2471.

11.

Yuliya Tarabalka , Jón Atli Benediktsson and Jocelyn Chanussot , Spectral-spatial classification of hyperspectral imagery based on partitional clustering techniques, IEEE Transactions on Geoscience and Remote Sensing 47(8) (2009), 2973–2987.

12.

Wenzhi Liao , Jocelyn Chanussot , Mauro Dalla Mura , Xin Huang , Rik Bellens , Sidharta Gautama and Wilfried Philips , Taking Optimal Advantage of Fine Spatial Resolution: Promoting partial image reconstruction for the morphological analysis of very-high-resolution images, IEEE Geoscience and Remote Sensing Magazine 5(2) (2017), 8–28.

13.

Yanfeng Gu , Tianzhu Liu , Xiuping Jia , Jón Atli Benediktsson and Jocelyn Chanussot , Nonlinear multiple kernel learning with multiple-structure-element extended morphological profiles for hyperspectral image classification, IEEE Transactions on Geoscience and Remote Sensing 54(6) (2016), 3235–3247.

14.

Yushi Chen , Zhouhan Lin , Xing Zhao , Gang Wang and Yanfeng Gu , Deep learning-based classification of hyperspectral data, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 7(6) (2014), 2094–2107.

15.

Wei Hu , Yangyu Huang , Li Wei , Fan Zhang and Hengchao Li , Deep convolutional neural networks for hyperspectral image classification, Journal of Sensors, 2015, 2015.

16.

Ping Zhong , Zhiqiang Gong , Shutao Li and Carola-Bibiane Schönlieb , Learning to diversify deep belief networks for hyperspectral image classification, IEEE Transactions on Geoscience and Remote Sensing 55(6) (2017), 3516–3530.

17.

Lichao Mou , Pedram Ghamisi and Xiao Xiang Zhu , Deep recurrent neural networks for hyperspectral image classification, IEEE Transactions on Geoscience and Remote Sensing 55(7) (2017), 3639–3655.

18.

Paoletti

M.E.

, Haut

J.M.

, Plaza

and Plaza

, Deep learning classifiers for hyperspectral imaging: A review, ISPRS Journal of Photogrammetry and Remote Sensing 158 (2019), 279–317.

19.

Ava Vali , Sara Comai and Matteo Matteucci , Deep learning for land use and land cover classification based on hyperspectral and multispectral earth observation data: A review, Remote Sensing 12(15) (2020), 2495.

20.

Shaohui Mei , Jingyu Ji , Qianqian Bi , Junhui Hou , Qian Du and Wei Li , Integrating spectral and spatial information into deep convolutional neural networks for hyperspectral classification, In 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pages 5067–5070, IEEE, 2016.

21.

Shaohui Mei , Jingyu Ji , Junhui Hou , Xu Li and Qian Du , Learning sensor-specific spatial-spectral features of hyperspectral images via convolutional neural networks, IEEE Transactions on Geoscience and Remote Sensing 55(8) (2017), 4520–4533.

22.

Jun Yue , Wenzhi Zhao , Shanjun Mao and Hui Liu , Spectral-spatial classification of hyperspectral images using deep convolutional neural networks, Remote Sensing Letters 6(6) (2015), 468–477.

23.

Yushi Chen , Hanlu Jiang , Chunyang Li , Xiuping Jia and Pedram Ghamisi , Deep feature extraction and classification of hyperspectral images based on convolutional neural networks, IEEE Transactions on Geoscience and Remote Sensing 54(10) (2016), 6232–6251.

24.

Paoletti

M.E.

, Haut

J.M.

, Plaza

and Plaza

, A new deep convolutional neural network for fast hyperspectral image classification, ISPRS Journal of Photogrammetry and Remote Sensing 145 (2018), 120–147, Deep Learning RS Data.

25.

Lin Zhu , Yushi Chen , Pedram Ghamisi and Jón Atli Benediktsson , Generative adversarial networks for hyperspectral image classification, IEEE Transactions on Geoscience and Remote Sensing 56(9) (2018), 5046–5063.

26.

Akrem Sellami , Ali Ben Abbes , Vincent Barra and Imed Riadh Farah , Fused 3-d spectral-spatial deep neural networks and spectral clustering for hyperspectral image classification, Pattern Recognition Letters 138 (2020), 594–600.

27.

Swalpa Kumar Roy , Suvojit Manna , Tiecheng Song and Lorenzo Bruzzone , Attention-Based Adaptive Spectral-Spatial Kernel ResNet for Hyperspectral Image Classification, IEEE Transactions on Geoscience and Remote Sensing 59(9) (2021), 7831–7843.

28.

Kui Li , Zhiguo Ma , Linlin Xu , Ye Chen , Yiyi Ma , Wei Wu , Fang Wang and Zihao Liu , Depthwise Separable ResNet in the MAP Framework for Hyperspectral Image Classification, IEEE Geoscience and Remote Sensing Letters 19 (2022), 1–5.

29.

Zilong Zhong , Jonathan Li , Zhiming Luo and Michael Chapman , Spectral-spatial residual network for hyperspectral image classification: A 3-D deep learning framework, IEEE Transactions on Geoscience and Remote Sensing 56(2) (2018), 847–858.

30.

Mercedes Paoletti

, Juan Haut

, Javier Plaza and Antonio Plaza , Deep & dense convolutional neural network for hyperspectral image classification, Remote Sensing 10(9) (2018), 1454.

31.

Gefei Yang , Utsav Gewali

, Emmett Ientilucci , Micheal Gartley and Sildomar Monteiro

, Dual-channel densenet for hyperspectral image classification, In IGARSS 2018–2018 IEEE International Geoscience and Remote Sensing Symposium, pages 2595–2598, 2018.

32.

Chunju Zhang , Guandong Li and Shihong Du , Multi-Scale Dense Networks for Hyperspectral Remote Sensing Image Classification, IEEE Transactions on Geoscience and Remote Sensing 57(11) (2019), 9201–9222.

33.

Zhiqiang Gong , Ping Zhong , Yang Yu , Weidong Hu and Shutao Li , A cnn with multiscale convolution and diversified metric for hyperspectral image classification, IEEE Transactions on Geoscience and Remote Sensing 57(6) (2019), 3599–3618.

34.

Mingyi He , Bo Li and Huahui Chen , Multi-scale 3D deep convolutional neural network for hyperspectral image classification, In 2017 IEEE International Conference on Image Processing (ICIP), pages 3904–3908. IEEE, 2017.

35.

Yong Xiao , Qin Xu , Dongyue Wang , Jin Tang and Bin Luo , Multi-scale Densely 3D CNN for Hyperspectral Image Classification, In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 596–606. Springer, 2019.

36.

Qin Xu , Dongyue Wang and Bin Luo , Faster multiscale capsule network with octave convolution for hyperspectral image classification, IEEE Geoscience and Remote Sensing Letters 18(2) (2020), 361–365.

37.

Shang-Hua Gao , Ming-Ming Cheng , Kai Zhao , Xin-Yu Zhang , Ming-Hsuan Yang and Philip Torr , Res2Net: A New Multi-Scale Backbone Architecture, IEEE Transactions on Pattern Analysis and Machine Intelligence 43(2) (2021), 652–662.

38.

Zhongqiang Zhang , Danhua Liu , Dahua Gao and Guangming Shi , A novel spectral-spatial multi-scale network for hyperspectral image classification with the Res2Net block, International Journal of Remote Sensing 43(3) (2022), 751–777.

39.

Shuai Yang , Feifei Lee , Ran Miao , Jiawei Cai , Lu Chen , Wei Yao , Koji Kotani and Qiu Chen , RS-CapsNet: An advanced capsule network, IEEE Access 8 (2020), 85007–85018.

40.

Hao Sun , Xiangtao Zheng , Xiaoqiang Lu and Siyuan Wu , Spectral-Spatial Attention Network for Hyperspectral Image Classification, IEEE Transactions on Geoscience and Remote Sensing 58(5) (2020), 3232–3245.

41.

Shivam Pande and Biplab Banerjee , Adaptive hybrid attention network for hyperspectral image classification, Pattern Recognition Letters 144 (2021), 6–12.

42.

Wenhui Guo , Hailiang Ye and Feilong Cao , Feature-Grouped Network With Spectral-Spatial Connected Attention for Hyperspectral Image Classification, IEEE Transactions on Geoscience and Remote Sensing, pages 1–13, 2021.

43.

Renlong Hang , Zhu Li , Qingshan Liu , Pedram Ghamisi and Shuvra Bhattacharyya

, Hyperspectral image classification with attention-aided cnns, IEEE Transactions on Geoscience and Remote Sensing 59(3) (2021), 2281–2293.

44.

Minghao Zhu , Licheng Jiao , Fang Liu , Shuyuan Yang and Jianing Wang , Residual spectral-spatial attention network for hyperspectral image classification, IEEE Transactions on Geoscience and Remote Sensing 59(1) (2021), 449–462.

45.

Xiang Li , Wenhai Wang , Xiaolin Hu and Jian Yang , Selective kernel networks, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 510–519, 2019.

46.

Sergey Ioffe and Christian Szegedy , Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv 2015, pages 448–456, 2015.

47.

Vinod Nair and Geoffrey Hinton

, Rectified linear units improve restricted boltzmann machines, In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814, 2010.

48.

Xiaohan Ding , Yuchen Guo , Guiguang Ding and Jungong Han , Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks, In Proceedings of the IEEE International Conference on Computer Vision, pages 1911–1920, 2019.

49.

Ying Li , Haokui Zhang and Qiang Shen , Spectral-spatial classification of hyperspectral imagery with 3d convolutional neural network, Remote Sensing 9(1) (2017), 67.

50.

Bei Fang , Ying Li , Haokui Zhang and Jonathan Cheung-Wai Chan , Hyperspectral Images Classification Based on Dense Convolutional Networks with Spectral-Wise Attention Mechanism, Remote Sensing 11(2) (2019), 159.

51.

Ayerdi

, Graña

, Veganzons

M.A.

, Hyperspectral remote sensing scenes, https://www.ehu.eus/ccwintco/_index.php/Hyperspectral_Remote_Sensing_Scenes#Pavia_Centre_and_University