Cross-form efficient attention pyramidal network for semantic image segmentation

Abstract

Although convolutional neural networks (CNNs) are leading the way in semantic segmentation, standard methods still have some flaws. First, there is feature redundancy and less discriminating feature representations. Second, the number of effective multi-scale features is limited. In this paper, we aim to solve these constraints with the proposed network that utilizes two effective pre-trained models as an encoder. We develop a cross-form attention pyramid that acquires semantically rich multi-scale information from local and global priors. A spatial-wise attention module is introduced to further enhance the segmentation findings. It highlights more discriminating regions of low-level features to focus on significant location information. We demonstrate the efficacy of the proposed network on three datasets, including IDD Lite, PASCAL VOC 2012, and CamVid. Our model achieves a mIoU score of 70.7% on the IDD Lite, 83.98% on the PASCAL VOC 2012, and 73.8% on the CamVid dataset.

Keywords

Attention mechanism convolutional neural networks cross-form spatial pyramid multi-scale features semantic segmentation

1. Introduction

Since the beginning of the digital image processing field, image segmentation has been a major issue in computer vision. Semantic segmentation assigns per-pixel labels to object categories in a given image, which is a difficult process that necessitates precise object category, position, and shape prediction. Due to the rich semantic and spatial information provided by semantic segmentation, it becomes a fundamental component of many visual understanding systems. It has a large impact on a diverse range of intelligent applications such as video surveillance, urban planning, traffic analysis, autonomous driving, augmented reality, robot sensing, etc. From the simplest methods, such as thresholding, k-means clustering, watershed methods, etc., to more sophisticated methods, such as graph cuts, conditional and Markov random fields, etc., several image segmentation methods have been developed in the past.

Recently, deep Convolutional Neural Networks (CNNs) have been widely used for their powerful feature extraction, representation learning, and end-to-end trainable framework capabilities in semantic segmentation tasks. Particularly, fully convolutional networks (FCNs) [27] have become the foundation of using CNNs for semantic segmentation. Since then based on FCN, many state-of-the-art (SOTA) works have been proposed in the literature. Among many SOTA methods of segmentation, the backbones are often retrieved from classification models such as VGGNet [33], ResNet [17], DenseNet [19], and Xception [8], etc. These classification models extract similar features several times at different levels of the network, resulting in redundant information flow. Furthermore, previous approaches still have some issues including how to efficiently extract multi-scale information and focus on crucial channel and spatial information to improve the discrimination ability of learned features.

Several approaches such as pyramid pooling [48] and dilated convolution [4,5,7] have been introduced to extract multi-scale information. Furthermore, encoder-decoder architectures [2,31] are proposed to capture varying scaled objects by combining low-level and high-level semantic features. However, the interdependence between distinct channels in multiple-scale features receives less attention in these approaches. As a result, the model overlooks the difference between local representation and contextual dependencies for various categories. The dependencies of distinct channel mappings can significantly improve the feature maps to reflect specific semantics.

In response to the above-discussed problems, we propose an encoder-decoder based architecture named cross-form efficient attention pyramidal network (CEAPNet) for semantic segmentation. Unlike previous studies that have used a single pre-trained model for the encoder, we combine two effective pre-trained models as a backbone for the encoder: a deep residual network (ResNet) [17] and an efficient channel attention network (ECANet) [36]. ResNet architecture is deep enough to extract semantic information while still being able to manage the vanishing gradient problem and ECANet focuses on modeling local cross-channel dependencies, providing rich informative features. Further, we propose an enhanced spatial pyramid module named context-aware cross-form attention pyramid (CCFAP) that combines multi-scale and channel attention to generate effective contextual features at different scales. Generally, the spatial pyramid modules [5,48] extract multi-scale features using either parallel dilated convolutions or pooling layers. Dilated convolution uses local information, whereas the pooling layer uses global information to obtain multi-scale features. Our spatial pyramid pooling module is extended to a cross-form level with four parallel branches, each including a parallel dilated conv-layer and average pooling layer. Thus, multi-scale feature extraction through local to-global pyramid aggregates global information and local information. Moreover, we employ efficient channel-wise attention (ECA) strategy to model the channel-interdependencies of multi-scale features. Different from existing channel attention [18,38], ECA records local cross-channel interactions through fast 1D convolution without involving dimensionality reduction (DR). Similar to DeeplabV3+ [7], we also fuse low-level and high-level features in the decoder part. Before that, we use spatial-wise attention to filter the low-level features with spatial information. As a result of processing through distinct attention mechanisms, the network has improved channel and spatial feature dependencies and has obtained more accurate results. The following is a summary of our major contributions:

We propose a cross-form efficient attention pyramidal network (CEAPNet) that processes the encoded high-level and low-level features to obtain segmentation prediction. As an encoder, it uses two pre-trained models: ResNet 101, winner of the ILSVRC 2015 competition, and ECANet.

Aiming at high-level features, a context-aware cross-form pyramid module is proposed to extract more effective multi-scale features. Moreover, channel attention is employed to fully utilize the semantic information of multi-scale features by weighting more informative channel features.

Aiming at low-level features, we apply spatial-wise attention to get more effective location information which we then combine with the high-level multi-scale features to ensure precise segmentation results.

We prove the validity of the proposed network on three datasets: PASCAL VOC 2012, IDD Lite, and CamVid.

2. Related works

In this section, we will review several studies on image segmentation that are closely related to our work based on the effectiveness of the multi-scale context and the attention mechanism.

2.1. Multi-scale feature fusion

The first way is to combine features from several scales using deep neural networks. For this, encoder-decoder networks have proven to be effective. FCN [27] and U-Net [31] have utilized skip connections to gradually concatenate the decoder and corresponding encoder features from multiple scales. Exfuse [46] uses enhanced feature fusion to lessen the gap between encoder and decoder features. To improve the multi-scale feature learning capability of U-Net, FCDenseNet [21] and GC-DenseNet [34] replace the original blocks with densely connected blocks. UNet $+ +$ [50] extended the U-Net by restructuring the encoder-decoder skip-connections and combining the features of multiple semantic scales. HFCN [42] is a highly fused network that reuses the feature information more thoroughly with several cost functions, whereas HRNet [35] provides a multi-scale backbone for information sharing between coarse and fine-scale branches.

However, with only this encoder-decoder structure, it is difficult to interpret visual context and multi-scale information adequately.

2.2. Multi-scale context extraction

Another option is to use a module to collect multi-scale context from the same feature maps. An image pyramid is a simple method of obtaining multi-scale image representations by feeding several resized input images into the same model and then fusing the results [12,44]. However, the image pyramid approaches are inefficient in particular due to the additional time spent in training and inference. Some works [1,5,7,30] have exploited dilated convolutions with various rates to embed multi-scale contexts. Meanwhile, PSPNet [48] has adopted a pyramid pooling module to capture contextual data at various scales for object recognition. Furthermore, RefineNet [26] and ICNet [47] both combine the feature maps generated from the input images of multiple resolutions for multi-scale information. By concatenating residual feature maps extracted from consecutive convolution blocks, ESSN [22] is capable of generating multi-scale information. CE-Net [14] utilizes a context extractor module that consists of a multi-kernel pooling and dense atrous convolution block to obtain wider context and global information, respectively.

These solutions can solve the issue of scale variations of objects in complicated scenarios to some extent, but they are less effective due to a lack of understanding of contextual dependencies.

2.3. Multi-scale attention mechanism

A multi-scale network with attention mechanism not only obtains a larger receptive field but also assists the model to retain important information on different scales. Several studies [9,11,23,25,39,40,49] have exploited the attention mechanism in different ways to achieve remarkable performance. Specifically, Chen et al. [6] has used an attention mechanism to select more discriminating multi-scale features and predict each pixel in a single image individually. Zhang et al. [45] proposed an attention-directed network that uses the channel and spatial attention methods for saliency detection. The most recent non-local/self-attention approaches connect each pair of pixels to enhance context aggregation [16,37,43]. By summing the two attention modules from distinct branches, DANet [13] adaptively merged local features with their global dependencies. CCNet [20] introduces recurrent criss-cross attention that can more efficiently and effectively gather context information from remote dependencies, thus reducing the computational complexity of self-attention. However, these approaches are usually memory and processing intensive because there are so many pixels and connections between them. EMANet [24] rethinks the attention mechanism from the perspective of the expectation-maximization (EM) algorithm, computing the attention map iteratively in the same way that the EM algorithm does. EAPNet [41] uses an efficient channel attention pyramid module for high-level features which can acquire effective channel attention maps by skipping dimensionality reduction and residual attention fusion block for low-level features to improve the performance.

Different from the above-discussed papers, we use an attention mechanism not only in the multi-scale feature extraction module but also in the encoder to enhance the feature representations.

3. Proposed architecture

For semantic segmentation, researchers have preferred encoder-decoder architectures with pre-trained models as an encoder. We also propose an encoder-decoder architecture named CEAPNet for semantic segmentation. The overall network is illustrated in Fig. 1. Our proposed network has mainly three parts: (i) a combined pre-trained encoder for feature extraction, (ii) a context-aware cross-form attention pyramid (CCFAP) module for multi-scale feature extraction through global and local pyramids, and (iii) a decoder. The following sub-sections will discuss each part in detail.

Fig. 1.

Overview of our proposed CEAPNet.

3.1. Combined pre-trained encoder

We employ a pre-trained model for feature extraction in the encoder rather than training from scratch. There are various pre-trained models in the literature that have been trained on a large ImageNet dataset. Previous works like [1,5,7,13,22,34,35,41,42,48] have used a single pre-trained model as an encoder while in our proposed architecture, we have combined two pre-trained models: residual network (ResNet) [17] and efficient channel attention network (ECANet) [36]. The ResNet architecture has a depth of 101 so the pre-trained model is referred to as ECA-ResNet 101 as shown in Fig. 1. Generally, the final feature responses of Deep CNNs used for the image classification task have output stride = 32 i.e. features are 32 times smaller than the original input size. These small resolution feature maps are not very useful for semantic segmentation. That’s why we change the stride of a convolution layer from 2 to 1 in the last encoder block and adopt $3 \times 3$ dilated convolutions of rates 2, 4, and 2 correspondingly. Thus we obtain rich feature maps having output stride = 16 at the end of the encoder as shown in Fig. 2. We have preferred ECANet rather than SENet [18] because it involves channel dimension reduction which has side effects on channel attention estimation while ECANet learns cross-channel interaction without dimension reduction while significantly decreasing model complexity. The basic structure of ECA-ResNet 101 is illustrated in Fig. 3. We now briefly discuss the ResNet architecture and then ECANet.

Fig. 2.

Pre-trained architecture with residual units and the size of their outputs. (a) ResNet 101: going deeper without dilated convolution in the last block, (b) ECA-ResNet 101: going deeper with dilated convolution and setting dilation rates to 2, 4, 2 for an output stride = 16. Key: the notation $k \times k$ , C in each block denotes the filter size k and number of channels.

Fig. 3.

Basic building block for, (a) ResNet 101 model, (b) ECA-ResNet 101 model.

Fig. 4.

Structure of efficient channel attention (ECA) block.

3.1.1. ResNet 101 architecture

In the research community, there is a growing consensus that deeper networks are better at learning complicated visual representations, which could lead to greater performance. However, simply stacking layers on top of each other will not improve performance. The familiar vanishing gradient problem makes deep networks challenging to train. So, as the network expands in size, its performance gets saturated, if not severely deteriorated. He et al. [17] proposed a solution to this problem by introducing residual blocks which allow training hundreds or even thousands of layers while still achieving excellent results. Its main idea is to introduce an “identity shortcut or skip connection” that bypasses one or more layers, as indicated in Fig. 3. With ResNets, the gradients can travel directly downstream from hidden layers to early layers using skip connections. Without the skip connection, input ‘x’ is multiplied by the weights (w) of the layer followed by adding a bias term (b) and then applying an activation function. Mathematically, we can represent it as follows: $\begin{matrix} (1) & Y (x) = f (w x + b) or F (x) . \end{matrix}$ Now, with the introduction of skip-connection, the output $Y (x)$ can be written as: $\begin{matrix} (2) & Y (x) = F (x) + x . \end{matrix}$ This skip-connection strategy tackles the problem of vanishing gradient in deep CNNs by enabling the gradient to flow along an additional shortcut path.

3.1.2. ECANet architecture

The channel-wise attention mechanism has recently been proven to hold a lot of promise for improving deep CNN performance [18,29,36,38]. To enhance the characterization capability of feature maps at different levels of the encoder, ECANet [36] introduces efficient channel attention (ECA) block which can be integrated with any model without increasing the model complexity, see Fig. 3(b). For convenience, we consider high-level features of a ResNet block as $f^{h} \in R^{H \times W \times C}$ , where H, W and C are height, width, and channel dimension. As shown in Fig. 4, the ECA block first generates a feature map $f^{C} = C \times 1 \times 1$ by applying a global average pooling on $f^{h}$ . Then we compress and exchange dimensions of $f^{C}$ and obtain a feature map $f^{S} = C \times 1$ . Next, we utilize $1 D$ convolution of k kernel size to capture the coverage of local cross-channel interaction followed by the sigmoid function (σ) to obtain the weight of each channel. The whole process can be summarized as below: $\begin{matrix} (3) & ω = σ (C 1 D_{k} (f^{S})), \end{matrix}$ where $C 1 D$ denotes $1 D$ convolution operation. Here, Eq. (3) is called the ECA method which has only k parameters.

The channel dimension C is proportional to the kernel size k of $1 D$ convolution. In other words, there may be a mapping between k and C as in Eq.: $\begin{matrix} (4) & C = γ (k) = 2^{α \cdot k - β} . \end{matrix}$ Then, the kernel size k can be determined adaptively if channel dimension C is given as follows: $\begin{matrix} (5) & k = {| \frac{C}{α} + \frac{β}{α} |}_{odd} . \end{matrix}$ where $| A |_{odd}$ denotes the closest odd number of A. The values of α and β are set at 2 and 1, respectively. Employing a non-linear mapping, high-dimensional channels have long-range interaction while low-dimensional channels have short-range interaction. To the best of our knowledge, ECA blocks have not been used in the backbone of encoder parts for segmentation problems. Thus by combining two efficient architectures, the proposed network can extract rich informative features.

3.2. Context-aware cross-form attention pyramid module

The main purpose of this module is to capture efficient multi-scale features through global and local information to identify objects of varied sizes. After the last encoder block, many works have utilized different pyramid methods [5,7,48] which either focus on dilated convolutions or pooling operations for multi-scale feature extraction. On the other hand, our suggested pyramid module integrates both strategies as shown in Fig. 5. The proposed cross-form attention pyramid is made up of four parallel cross-form branches that correspond to the four levels of the pyramid. Every branch is made up of two parallel sub-branches namely dilated sub-branch and pooling sub-branch, and the channel-wise attention mechanism.

Dilated sub-branch: It applies a $3 \times 3$ depth-wise dilated conv-layer followed by a $1 \times 1$ conv-layer to reduce the channel dimension to 256. We use dilation rates for each branch as 3, 6, 12, and 18. After that, we use a batch-normalization (BN) layer.

Pooling sub-branch: It consists of an average pooling layer with corresponding pyramid scales. Similar to dilated branch, we apply a $1 \times 1$ conv-layer to minimize the number of channels to 256 and a BN layer. After that, the bilinear interpolation is applied to keep the output size constant. For every pooling sub-branch corresponding to a dilated branch, we utilize a fixed-size average pooling scale of $4 \times 4$ , $8 \times 8$ , $16 \times 16$ , and $32 \times 32$ .

Channel-wise attention mechanism: After merging the outputs of both sub-branches, the ECA block is used to rescale important feature maps and discard unnecessary ones.

Fig. 5.

Structure of context-aware cross-form attention pyramid (CCFAP) module.

Dilated branch allows us to extract dense features of multiple scales without increasing the number of parameters. This is done by inserting the holes (zeroes) within the filter’s weight matrix, resulting in a larger filter’s field of view. In 2D, dilated convolution mathematically can be computed as follows [5]: $\begin{matrix} (6) & y [i] = Σ_{k} x [i + d \cdot k] \cdot w [k] . \end{matrix}$ where the output y is obtained by convolving every location i of the input x with weight matrix w and dilation rate d is the stride with which input is convolved. When $d = 1$ , it becomes a special case of standard convolution. We can modify the fields of view by increasing the dilation rate value. The pooling branch considers the global context of the image to estimate the local level prediction. We employed global average pooling to reduce model size while simultaneously preventing overfitting. Although max pooling is another pooling strategy, it only allows for a single maximum value, which is insufficient for our network. Different from [48] where an adaptive average pooling strategy is used to compress the features into four different scales, we use fixed-size pooling scales. Thus, both sub-branches combine context information with global and local priors. However, there might be a possibility of information redundancy among different branches. So, it is significant to filter these features and focus on the most important ones. For this, we employ the ECA block to enhance the characterization ability of fused features by learning cross-channel inter-dependencies. Further, the feature maps from all four parallel branches are concatenated along with the corresponding encoder’s feature maps. The final features of this module are then fed to the decoder. In this module, we have used both addition and concatenation processes for feature aggregation. We use addition to aggregate the features of sub-branches, then concatenation to aggregate the features of all four parallel branches, see Fig. 5. Because sub-branches have local and global priors, addition appears to be advantageous, whereas main parallel branches have substantial variances in feature scales; concatenation is more powerful, preserving all information of multi-scale features.

3.3. Decoder

The main purpose of the decoder is to restore the resolution of down-sampled feature maps. We use a simple decoder structure like DeeplabV3+ [7] with an additional spatial attention strategy as shown in Fig. 1. Similar to DeeplabV3+, the high-level features obtained from the above-mentioned pyramidal module, are upsampled by a factor of four using bilinear interpolation and then merged with equivalent low-level features of the encoder. Before that, these low-level features are passed through $1 \times 1$ convolution to minimize the channel dimension and make the training easier, followed by a spatial-wise attention module. As we know that low-level features are responsible for location (spatial) information rather than semantic, spatial-wise attention allows us to process information by prioritizing a specific area within the visual field. As a result, a specific area is given attention, and information inside that area is processed further. After the concatenation, we apply two $3 \times 3$ conv-layers along with BN and an activation layer to upgrade the features. We again apply bilinear upsampling by a factor of four before the final prediction.

Fig. 6.

Design of spatial attention (SA) block.

3.3.1. Spatial-wise attention

Since low-level features provide a lot of location data, it’s best to concentrate on the salient regions that will help you to discover the object’s location and identify the target structure. For this, we apply spatial-wise attention to low-level features, giving closer attention to the ‘where’ is an important region to predict. As illustrated in Fig. 6, we first perform max pooling, average pooling, and $1 \times 1$ convolution across the channel dimension and thereafter concatenate them to generate significant feature representation. Utilizing average and max pooling, as suggested by [38], and using a $1 \times 1$ convolution, as suggested by [32], are excellent ways to convey necessary information. The concatenated features are then applied to conv-layers to create a spatial attention map $ω^{'} \in R^{H \times W \times 1}$ . The details of the operations are given below.

We aggregate input tensor $f^{l} \in R^{H \times W \times C}$ along channel dimension by using pooling and $1 \times 1$ convolution and generate $f_{avg}^{s} \in R^{H \times W \times 1}$ , $f_{max}^{s} \in R^{H \times W \times 1}$ and $f_{1 \times 1}^{s} \in R^{H \times W \times 1}$ maps. These maps are then concatenated and passed through convolution layers of kernel size $1 \times 9$ and $9 \times 1$ along with a BN layer. The channel dimension after this is reduced to 1. These convolution layers are better at deciding spatially important regions [38]. To reduce computations, we chose factorized kernels over $9 \times 9$ large kernel. Finally, the sigmoid activation is applied to the resultant to generate the attention map. The entire procedure can be stated as follows: $\begin{array}{l} (7) & M (f^{l}) = F_{9 \times 1} (F_{1 \times 9} ([f_{avg}^{s}, f_{max}^{s}, f_{1 \times 1}^{s}])), \\ (8) & ω_{^{'}} = σ (M (f^{l})), \end{array}$

4. Experiments

We perform extensive experiments on three demanding semantic segmentation and scene parsing datasets including IDD Lite [28], PASCAL VOC 2012 [10], and CamVid [3] to validate our proposed architecture. We begin with the description of the datasets in Section 4.1, followed by a description of the implementation in Section 4.2. Thereafter, we conduct many ablation experiments on the IDD Lite validation set to demonstrate the efficacy of every module of our proposed design in Section 4.3. Finally, in subsequent sections, we compare our method to SOTA models in terms of mean intersection over union (mIoU) on all three datasets.

4.1. Datasets

IDD Lite: It is an unstructured road scene dataset that accurately reflects the road infrastructure of Asian countries. It includes 1404 training images, 204 validation images, and 408 test images. Only training and validation images have their respective ground truths. The original images have a resolution of $320 \times 227$ pixels, which we adjust to $320 \times 224$ pixels. It has seven categories: vehicles, drivable, far objects, non-drivable, roadside objects, living things, and sky. We use the augmentation techniques including random rotation (angle bw 15 and 45), horizontal flip, brightness, Gaussian blur, contrast, shear, etc. to avoid overfitting.

PASCAL VOC 2012: It is a semantic segmentation benchmark dataset that originally included 1,464 images for training, 1,449 for validation, and 1,456 for testing. It has a total of 20 foreground objects classes and a background class. We use the augmentation to expand the dataset to 10582 images for training by [15]. During the training, we use a crop size of $512 \times 512$ .

CamVid: This is another challenging urban street scene dataset in the domain of autonomous driving. It contains a total of 701 images, including 376 for training, 101 for validation, and 233 for testing. It has 11 classes like building, car, tree, fence, sky, etc. and the original image resolution is $960 \times 720$ but it was resized to $360 \times 480$ pixels before training to ensure a fair comparison. We flip left-right randomly with the probability of 50% and scale the image ranging from 0.5 to 2 as our data augmentation.

4.2. Implementation details

All the experiments are executed on the Pytorch framework with a 16GB NVIDIA TESLA P100 GPU. We use ECA-ResNet 101 that has been trained on the ImageNet dataset for the starting point of all our networks. As discussed in Section 3, we modify the backbone network so that output features are $\frac{1}{16}$ th of the original input size. We employ poly learning rate policy, where $l r_{rate} = {init}_{rate} \times {(1 - \frac{epoch}{max_epoch})}^{0.9}$ . The learning rate is initially set to 0.001 for PASCAL VOC 2012 and CamVid; and $3 e - 4$ for the IDD Lite dataset. SGD with a momentum of 0.9 and weight decay of 0.0001 is used as an optimizer for PASCAL VOC 2012 and Adam algorithm for IDD Lite and CamVid. We train the networks for a total of 200 epochs with a batch size of 16 for Pascal VOC 2012 and, 150 epochs with a batch size of 15 for IDD Lite, and 180 epochs with a batch size of 8 for the CamVid dataset. We use the cross-entropy loss for CamVid and PASCAL VOC 2012, and the sum of binary cross-entropy and IoU loss for the IDD Lite dataset to train our network.

4.3. Ablation experiments

4.3.1. Ablation for context-aware cross-form attention pyramid module

The first experiment was conducted to validate the effectiveness of our introduced CCFAP block. We compare it with different pyramid methods. For better comparison, we set three different spatial pyramid structures of the CCFAP module as in [5,48], and [4]: modified ASPP (mASPP) deletes all pooling branches; modified Spatial Pyramid Pooling (mSPP) deletes all dilated branches and modified Atrous Image Cascade (mAtIC) sequentially connects the dilated and pooling branches while keeping the remaining parameters constant, respectively. The outcomes of the experiments are shown in Table 1. The results demonstrate that the CCFAP module outperforms other methods. When compared to mSPP and mASPP, CCFAP improves the performance by 2% and 1.2% mIoU. Although mAtIC uses both operations in a series manner, it didn’t perform well. The reason might be pooling operation which is applied after dilated convolution not on the input image itself, so it is not able to add global information of original input. In our module, the different sizes of objects and the relationship between an object (e.g. car) and its sub-regions (e.g. windows, tires, etc.) are effectively learned by the CCFAP module. This is due to the inclusion of both ASPP and SPP structure that helps in extraction of rich multi-scale features through local and global information. It can also be seen in Fig. 7 that global information allows capturing the context of the whole image to correctly classify the ‘drivable’ and ‘roadside object’ classes in CCFAP. Also, different size vehicles and riders are clearer in the segmentation results of the CCFAP module than others.

Table 1
Comparison of different spatial pyramid methods on IDD lite validation set

Method mIoU (%) #Param.

ECA-ResNet101 + mSPP 68.7 45.97M

ECA-ResNet101 + mASPP 69.5 46.57M

ECA-ResNet101 + mAtIC 69.0 46.57M

ECA-ResNet101 + CCFAP 70.7 48.67M

Method	mIoU (%)	#Param.
ECA-ResNet101 + mSPP	68.7	45.97M
ECA-ResNet101 + mASPP	69.5	46.57M
ECA-ResNet101 + mAtIC	69.0	46.57M
ECA-ResNet101 + CCFAP	70.7	48.67M

Fig. 7.

Segmentation results of different pyramid methods on IDD lite validation set. Here, the first row represents the ground truths and the remaining rows represent the predicted outputs of specified bounding box areas (in the first row) for compared methods.

We further do several experiments to address how the dilation rate and pooling scale affect the CCFAP module. Different combinations of dilation rates and pooling scales are applied and results are recorded in Table 2. We end up setting the dilation rates as 3, 6, 12, 18 and corresponding pooling scales as 4, 8, 16, 32 which produces the best results for our network (we explored a variety of dilation rates and pooling scales but only the most accurate were reported).

4.3.2. Ablation for the effect of attention modules

In this section, we investigate the importance of channel attention and spatial attention on the segmentation performance of the proposed network. Specifically, we have used ECA for the high-level features in the CCFAP module and spatial attention for low-level features. The results are reported in Table 3. Results demonstrate that adding channel attention to the pyramid structure enhances performance by 2.5% in comparison to the baseline. We may conclude that higher-level features of certain channels have a significant impact on segmentation results and the channel attention amplifies this impact, hence boosting network performance. When the SA block is added, the mIoU increases by nearly 1.1% from 67.4% to 68.5%. The spatial attention tells the network which pivotal part should be focused on and suppressed. Thus, it improves the low-level feature representations which have an abundance of spatial information. By visually examining the segmentation results in Fig. 8, we can see those attention modules have a significant decrease in the false classification of pixels.

Table 2
Results of ablation study on IDD lite validation set for different pooling scales and dilation rates

Method Dilation rate Pooling Scale mIoU (%)

CCFAP 3, 5, 7, 17 4, 8, 16, 32 67.9

3, 5, 7, 11 6, 10, 14, 22 68.4

3, 5, 7, 11 4, 8, 16, 32 68.9

3, 6, 12, 18 6, 12, 24, 36 69.6

3, 6, 12, 18 4, 8, 16, 32 70.7

Method	Dilation rate	Pooling Scale	mIoU (%)
CCFAP	3, 5, 7, 17	4, 8, 16, 32	67.9
3, 5, 7, 11	6, 10, 14, 22	68.4
3, 5, 7, 11	4, 8, 16, 32	68.9
3, 6, 12, 18	6, 12, 24, 36	69.6
3, 6, 12, 18	4, 8, 16, 32	70.7

Table 3

Results of ablation study on IDD lite validation set for the effect of attention mechanisms on network performance

Backbone	Channel attention	Spatial attention	mIoU (%)
ECA-ResNet101(baseline)	–	–	67.4
ECA-ResNet101	Yes	–	69.9
ECA-ResNet101	–	yes	68.5
ECA-ResNet101	Yes	yes	70.7

Fig. 8.

Segmentation results with or without attention mechanisms on IDD lite validation set.

Further, we have conducted several experiments to choose the kernel size of spatial attention, and results are recorded in Table 4. We steadily increased the kernel size and discovered that performance improved as the kernel size rose to 9. Further increment degraded the performance, so we settled on 9 as kernel size which achieved the best result.

Table 4

Results of ablation study on IDD lite validation set for kernel size of spatial attention

Method	Kernel size	mIoU (%)
SA	5	67.9
	7	69.4
	9	70.7
	11	68.3

4.3.3. Comparison with other channel attention structures

In this section, we compare the ECA block with three attention modules including squeeze and excite (SE) [18], convolutional block attention module (CBAM) [38], and Triplet attention (TA) [29]. The accuracy results (mIoU) are reported in Table 5. Although SE and CBAM blocks provide significant performance improvements, these blocks don’t account for efficient cross-dimension interaction. Both approaches first project channel descriptors into a low-dimensional space before mapping them back into a high-dimensional space which causes loss in inter-channel interaction. On the other hand, TA incorporates cross-dimensional interaction without involving any dimensionality reduction but has more parameters than ECA. As it has been discussed in Section 3.1.2, ECA records not just cross-dimensional interdependence and interactions of features for rich feature representations but also avoids channel dimensionality reduction without increasing the parameters. It has utilized $1 D$ convolution of kernel size k to capture cross-channel interaction by determining each channel and its k neighbors. It can be seen in Table 5 that ECA introduces the lowest additional parameters and negligible computation while achieving 3.1%, 2.7%, and 1.5% gain over SE, CBAM, and TA, respectively.

Table 5
Comparison of different channel methods on IDD lite validation set. (Here, C: channel dimension, r: reduction ratio, and k: kernel size)

Method (#)Param. mIoU (%)

SE $2 C^{2} / r$ 67.6

CBAM $2 C^{2} / r + 2 k^{2}$ 68.0

Triplet attention $6 k^{2}$ 69.2

ECA k 70.7

Method	(#)Param.	mIoU (%)
SE	$2 C^{2} / r$	67.6
CBAM	$2 C^{2} / r + 2 k^{2}$	68.0
Triplet attention	$6 k^{2}$	69.2
ECA	k	70.7

Fig. 9.

Comparison of segmentation results of different models on IDD lite validation set. Specific areas are highlighted in bounding boxes for better comparison.

4.4. Results on IDD lite dataset

The proposed network is compared with several SOTA models on the IDD Lite dataset. All SOTA models were benchmarked on different datasets, so we re-evaluated these models on the IDD Lite dataset and report the results of the validation set in Table 6. Our proposed model yields 70.7% mIoU which is 5.1% and 4.6% greater than DRN DeepLabV3+ [1] and ERFNet [30], respectively. All previous works have used mainly single pre-trained architecture as a backbone for feature extraction whereas CEAPNet has exploited two effective pre-trained models for the feature extraction process which has an advantage over them. Among previous works, DANet [13], DF-DAM [39], and GLNet [25] adopt attention mechanisms to improve the performance but those are not very effective. Although EAPNet [41] adopts an efficient channel attention mechanism but the absence of global information during multi-scale feature extraction, makes it inferior to our proposed model by 2.6% mIoU. For visual comparison, we consider only the best two models among previous works, and the results are shown in Fig. 9. As can be seen from the figure, CEAPNet is more detailed than GLNet and EAPNet. For example, light poles and small size vehicles are more visible in the segmentation results of CEAPNet. For roadside objects and far objects, the other two models are confused whereas CEAPNet is more distinguishable between them.

Table 6
Results on the IDD lite validation set

Model Year Drivable Non-drivable Living things Vehicles Far obj. Road-side obj. Sky mIoU (%)

DF-DAM [39] 2019 92.0 36.4 43.6 70.2 72.7 44.7 91.0 64.4

DRN DeeplabV3+ [1] 2019 93.0 37.9 44.4 72.2 73.7 43.4 94.5 65.6

ERFNet [30] 2017 93.2 37.8 47.6 73.5 73.5 43.6 93.3 66.1

DANet [13] 2019 93.1 39.9 46.8 75.0 74.5 46.0 93.4 66.9

GLNet [25] 2020 93.5 40.4 46.9 74.5 74.4 47.2 92.6 67.1

EAPNet [41] 2021 94.6 41.7 47.3 75.5 75.4 48.3 93.7 68.1

CEAPNet (Ours) – 95.9 45.1 52.5 76.3 78.5 51.1 95.4 70.7

Model	Year	Drivable	Non-drivable	Living things	Vehicles	Far obj.	Road-side obj.	Sky	mIoU (%)
DF-DAM [39]	2019	92.0	36.4	43.6	70.2	72.7	44.7	91.0	64.4
DRN DeeplabV3+ [1]	2019	93.0	37.9	44.4	72.2	73.7	43.4	94.5	65.6
ERFNet [30]	2017	93.2	37.8	47.6	73.5	73.5	43.6	93.3	66.1
DANet [13]	2019	93.1	39.9	46.8	75.0	74.5	46.0	93.4	66.9
GLNet [25]	2020	93.5	40.4	46.9	74.5	74.4	47.2	92.6	67.1
EAPNet [41]	2021	94.6	41.7	47.3	75.5	75.4	48.3	93.7	68.1
CEAPNet (Ours)	–	95.9	45.1	52.5	76.3	78.5	51.1	95.4	70.7

4.5. Results on Pascal VOC 2012 dataset

We also run the experiment on the PASCAL VOC 2012 dataset to ensure that the proposed CEAPNet is generalizable. Comparisons with previous SOTA models on the validation set are recorded in Table 7. As can be observed, our suggested CEAPNet produces results that are comparable to previous attention-based models including DF-DAM [39], DANet [13], and EAPNet [41]. We also compared our model with other models such as HFCN [42], DeeplabV3 [5], DeeplabV3+ [7], Exfuse [46], and APCNet [16]. Results show that our model obtains the best performance of 83.98% mIoU without pre-trained on the MS-COCO dataset and outperforms baseline DeeplabV3+ by a mIoU value of 4.13%. We also compare the segmentation effects of the best two models among compared SOTA models with our proposed CEAPNet in Fig. 10. It can be seen that we achieve more obvious segmentation results for target objects than APCNet and EAPNet. Our proposed CEAPNet classifies objects such as a person, horse, sofa, sheep, etc. with better edge segmentation.

Table 7
Results on the Pascal VOC 2012 validation set

Model Year mIoU (%)

HFCN [42] 2019 75.0

DeepLabv3 [5] 2017 77.21

DF-DAM [39] 2019 77.9

DeeplabV3+ [7] 2018 79.85

Exfuse [46] 2018 80.0

DANet [13] 2019 80.4

APCNet [16] 2019 80.71

EAPNet [41] 2021 81.70

CEAPNet (ours) – 83.98

Model	Year	mIoU (%)
HFCN [42]	2019	75.0
DeepLabv3 [5]	2017	77.21
DF-DAM [39]	2019	77.9
DeeplabV3+ [7]	2018	79.85
Exfuse [46]	2018	80.0
DANet [13]	2019	80.4
APCNet [16]	2019	80.71
EAPNet [41]	2021	81.70
CEAPNet (ours)	–	83.98

Fig. 10.

Comparison of segmentation results of different models on Pascal VOC 2012 validation set.

Table 8

Results on the CamVid test set

Model	Building	Tree	Sky	Car	Sign	Road	Pedes.	Fence	Pole	Sidewalk	Bicyclist	mIoU (%)
Fan et al. [11]	88.4	84.5	93.4	84.9	48.8	95.6	61.8	54.8	38.4	90.5	47.7	60.1
FC-DenseNet [21]	83	77.3	93.0	77.3	43.9	94.5	59.6	37.1	37.8	82.0	50.5	66.9
GC-DenseNet [34]	84.7	77.1	92.3	84.9	46.7	94.7	56.2	46.2	35.9	82.2	61.9	69.3
AGLNet [49]	82.6	76.1	91.8	87.0	45.3	95.4	61.5	39.5	39.0	83.1	62.7	69.4
PPANet [9]	84.3	77.8	92.1	86.1	51.2	94.9	62.8	41.9	36.9	82.2	61.1	70.1
Dense decoder [2]	–	–	–	–	–	–	–	–	–	–	–	70.9
ESSN [6]	85.5	77.9	91.9	85.7	53.9	95.7	63.7	51.1	33.3	85.2	64.6	71.6
CEAPNet (ours)	89.0	85.2	93.6	87.4	54.0	96.2	64.2	55.3	36.1	86.3	65.3	73.8

4.6. Results on CamVid dataset

We also evaluated CEAPNet on another challenging road scene dataset i.e. CamVid to further examine the validity and reliability of the proposed model. Our model is compared with several SOTA models including ESSN [22], Dense Decoder [2], GC-DenseNet [34], and FC-DenseNet [21] as shown in Table 8. Compared with SOTA models, our proposed method achieves the best performance of 73.8% mIoU. It also outperforms attention-based methods including PPANet [9], AGLNet [49], and Fan et al. [11] by 3.7%, 4.4%, and 13.7% mIoU, respectively. Note that some of the approaches in Table 8 make use of high-resolution images, which expectedly would further improve our results. From Fig. 11, we can see that our model has improved accuracy in some important categories like sidewalk, car, and small categories like a sign, pedestrian, bicyclist, etc.

Fig. 11.

Comparison of segmentation results of different models on the CamVid test set. Specific areas are highlighted in bounding boxes for better comparison.

5. Conclusion

In this paper, our proposed CEAPNet segmentation model integrates two effective pre-trained models such as ResNet and ECANet for the encoder and channel attention into the pyramid module. The utilization of two pre-trained models as backbone architecture for feature extraction has a more positive impact on the segmentation performance. Different from other works which have only used pyramid structures for multi-scale feature extraction without giving importance to relevant features, our proposed pyramid structure with channel attention has further improved the performance. Additionally, spatial attention further complements the low-level features for fusion with high-level features. Finally, experiments show that our proposed network achieves 70.7%, 83.98%, 73.8% mIoU on IDD Lite, PASCAL VOC 2012, and CamVid datasets, respectively. In the future, we plan to combine efficient channel attention with alternative CNN architectures to improve semantic segmentation accuracy.

Footnotes

Acknowledgements

The authors would like to acknowledge the University Grants Commission (UGC) for fellowship throughout this research work.

References

Baheti,

Gajre and

Talbar, Semantic scene understanding in unstructured environment with deep convolutional neural network, in: TENCON 2019 – 2019 IEEE Region 10 Conference (TENCON), IEEE, 2019, pp. 790–795. doi:10.1109/TENCON.2019.8929376.

Bilinski and

Prisacariu, Dense decoder shortcut connections for single-pass semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 6596–6605.

G.J.

Brostow,

Shotton,

Fauqueur and

Cipolla, Segmentation and recognition using structure from motion point clouds, in: European Conference on Computer Vision, Springer, 2008, pp. 44–57.

Chen,

Wang and

Tong, AtICNet: Semantic segmentation with atrous spatial pyramid pooling in image cascade network, EURASIP Journal on Wireless Communications and Networking 2019(1) (2019), 1–7. doi:10.1186/s13638-018-1318-8.

L.-C.

Chen,

Papandreou,

Schroff and

Adam, Rethinking atrous convolution for semantic image segmentation, 2017, arXiv preprint arXiv:1706.05587.

L.-C.

Chen,

Yang,

Wang,

Xu and

A.L.

Yuille, Attention to scale: Scale-aware semantic image segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 3640–3649.

L.-C.

Chen,

Zhu,

Papandreou,

Schroff and

Adam, Encoder-decoder with atrous separable convolution for semantic image segmentation, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 801–818.

Chollet, Xception: Deep learning with depthwise separable convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1251–1258.

M.A.

Elhassan,

Chen,

Huang,

Yang,

Yao,

Yang and

Cheng, PPANet: Point-Wise Pyramid Attention Network for Semantic Segmentation, Wireless Communications and Mobile Computing 2021 (2021).

10.

Everingham,

Zisserman,

C.K.

Williams,

Van Gool,

Allan,

C.M.

Bishop,

Chapelle,

Dalal,

Deselaers,

Dorkó et al., The 2005 Pascal visual object classes challenge, in: Machine Learning Challenges Workshop, Springer, 2005, pp. 117–176.

11.

Fan,

W.-C.

Wang,

Zha and

Yan, Exploring new backbone and attention module for semantic segmentation in street scenes, IEEE Access 6 (2018), 71566–71580. doi:10.1109/ACCESS.2018.2880877.

12.

Farabet,

Couprie,

Najman and

LeCun, Learning hierarchical features for scene labeling, IEEE transactions on pattern analysis and machine intelligence 35(8) (2012), 1915–1929. doi:10.1109/TPAMI.2012.231.

13.

Fu,

Liu,

Tian,

Li,

Bao,

Fang and

Lu, Dual attention network for scene segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3146–3154.

14.

Gu,

Cheng,

Fu,

Zhou,

Hao,

Zhao,

Zhang,

Gao and

Liu, Ce-net: Context encoder network for 2d medical image segmentation, IEEE transactions on medical imaging 38(10) (2019), 2281–2292. doi:10.1109/TMI.2019.2903562.

15.

Hariharan,

Arbeláez,

Bourdev,

Maji and

Malik, Semantic contours from inverse detectors, in: 2011 International Conference on Computer Vision, IEEE, 2011, pp. 991–998. doi:10.1109/ICCV.2011.6126343.

16.

He,

Deng,

Zhou,

Wang and

Qiao, Adaptive pyramid context network for semantic segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 7519–7528.

17.

He,

Zhang,

Ren and

Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

18.

Hu,

Shen and

Sun, Squeeze-and-excitation networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132–7141.

19.

Huang,

Liu,

Van Der Maaten and

K.Q.

Weinberger, Densely connected convolutional networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4700–4708.

20.

Huang,

Wang,

Huang,

Wei and

Liu, Ccnet: Criss-cross attention for semantic segmentation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 603–612.

21.

Jégou,

Drozdzal,

Vazquez,

Romero and

Bengio, The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 11–19.

22.

D.S.

Kim,

Arsalan,

Owais and

K.R.

Park, ESSN: Enhanced semantic segmentation network by residual concatenation of feature maps, IEEE Access 8 (2020), 21363–21379. doi:10.1109/ACCESS.2020.2969442.

23.

Li,

Tan,

Chen,

Luo,

Gao,

Jia and

Wang, Attention unet++: A nested attention-aware U-net for liver CT image segmentation, in: 2020 IEEE International Conference on Image Processing (ICIP), IEEE, 2020, pp. 345–349. doi:10.1109/ICIP40778.2020.9190761.

24.

Li,

Zhong,

Wu,

Yang,

Lin and

Liu, Expectation-maximization attention networks for semantic segmentation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 9167–9176.

25.

C.-Y.

Lin,

Y.-C.

Chiu,

H.-F.

Ng,

T.K.

Shih and

K.-H.

Lin, Global-and-local context network for semantic segmentation of street view images, Sensors 20(10) (2020), 2907. doi:10.3390/s20102907.

26.

Lin,

Milan,

Shen and

Reid, Refinenet: Multi-path refinement networks for high-resolution semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1925–1934.

27.

Long,

Shelhamer and

Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440.

28.

Mishra,

Kumar,

Kalluri,

Varma,

Subramaian,

Chandraker and

Jawahar, Semantic segmentation datasets for resource constrained training, in: National Conference on Computer Vision, Pattern Recognition, Image Processing, and Graphics, Springer, 2019, pp. 450–459.

29.

Misra,

Nalamada,

A.U.

Arasanipalai and

Hou, Rotate to attend: Convolutional triplet attention module, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 3139–3148.

30.

Romera,

J.M.

Alvarez,

L.M.

Bergasa and

Arroyo, Erfnet: Efficient residual factorized convnet for real-time semantic segmentation, IEEE Transactions on Intelligent Transportation Systems 19(1) (2017), 263–272. doi:10.1109/TITS.2017.2750080.

31.

Ronneberger,

Fischer and

Brox, U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015, pp. 234–241.

32.

A.G.

Roy,

Navab and

Wachinger, Recalibrating fully convolutional networks with spatial and channel “squeeze and excitation” blocks, IEEE transactions on medical imaging 38(2) (2018), 540–549. doi:10.1109/TMI.2018.2867261.

33.

Simonyan and

Zisserman, Very deep convolutional networks for large-scale image recognition, 2014, arXiv preprint arXiv:1409.1556.

34.

C.K.

Wang and

L.-W.

Chang, Semantic Segmentation via Global Convolutional Network and Concatenated Feature Maps in: ICPRAM, 2019, pp. 292–297.

35.

Wang,

Sun,

Cheng,

Jiang,

Deng,

Zhao,

Liu,

Mu,

Tan,

Wang et al., Deep high-resolution representation learning for visual recognition, in: IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.

36.

Wang,

Wu,

Zhu,

Li,

Zuo and

Hu, ECA-net: Efficient channel attention for deep convolutional neural networks, 2020 IEEE, in: CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2020.

37.

Wang,

Girshick,

Gupta and

He, Non-local neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7794–7803.

38.

Woo,

Park,

J.-Y.

Lee and

I.S.

Kweon, Cbam: Convolutional block attention module, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 3–19.

39.

Xiong,

Wang,

Yu and

Gu, Differentiating Features for Scene Segmentation Based on Dedicated Attention Mechanisms, 2019, arXiv preprint arXiv:1911.08149.

40.

Xu,

Peng,

Li,

Li and

Du, Pyramid feature attention network for monocular depth prediction, in: 2021 IEEE International Conference on Multimedia and Expo (ICME), IEEE, 2021, pp. 1–6.

41.

Yang,

Ku and

Hu, Efficient attention pyramid network for semantic segmentation, IEEE Access 9 (2021), 18867–18875. doi:10.1109/ACCESS.2021.3053316.

42.

Yang,

Wu,

Zhao and

Guan, Semantic segmentation via highly fused convolutional network with multiple soft cost functions, Cognitive Systems Research 53 (2019), 20–30. doi:10.1016/j.cogsys.2018.04.004.

43.

Yuan,

Huang,

Guo,

Zhang,

Chen and

Wang, Ocnet: Object context network for scene parsing, 2018, arXiv preprint arXiv:1809.00916.

44.

Zhang,

Tang,

Zhang,

Li and

Yan, Scale-adaptive convolutions for scene parsing, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2031–2039.

45.

Zhang,

Wang,

Qi,

Lu and

Wang, Progressive attention guided recurrent network for salient object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 714–722.

46.

Zhang,

Peng,

Xue and

Sun, Exfuse: Enhancing feature fusion for semantic segmentation, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 269–284.

47.

Zhao,

Qi,

Shen,

Shi and

Jia, Icnet for real-time semantic segmentation on high-resolution images, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 405–420.

48.

Zhao,

Shi,

Qi,

Wang and

Jia, Pyramid scene parsing network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2881–2890.

49.

Zhou,

Wang,

Fan,

Wu,

Zhang,

Kang and

L.J.

Latecki, AGLNet: Towards real-time semantic segmentation of self-driving images via attention-guided lightweight network, Applied Soft Computing 96 (2020), 106682. doi:10.1016/j.asoc.2020.106682.

50.

Zhou,

M.M.R.

Siddiquee,

Tajbakhsh and

Liang, Unet++: A nested u-net architecture for medical image segmentation, in: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer, 2018, pp. 3–11. doi:10.1007/978-3-030-00889-5_1.

Cross-form efficient attention pyramidal network for semantic image segmentation

Abstract

Keywords

1. Introduction

2. Related works

2.1. Multi-scale feature fusion

2.2. Multi-scale context extraction

2.3. Multi-scale attention mechanism

3. Proposed architecture

3.1.2. ECANet architecture

3.2. Context-aware cross-form attention pyramid module

4. Experiments

4.1. Datasets

4.2. Implementation details

4.3. Ablation experiments

4.3.1. Ablation for context-aware cross-form attention pyramid module

Table 1 Comparison of different spatial pyramid methods on IDD lite validation set Method mIoU (%) #Param. ECA-ResNet101 + mSPP 68.7 45.97M ECA-ResNet101 + mASPP 69.5 46.57M ECA-ResNet101 + mAtIC 69.0 46.57M ECA-ResNet101 + CCFAP 70.7 48.67M

Table 5 Comparison of different channel methods on IDD lite validation set. (Here, C: channel dimension, r: reduction ratio, and k: kernel size) Method (#)Param. mIoU (%) SE 2 C 2 / r 67.6 CBAM 2 C 2 / r + 2 k 2 68.0 Triplet attention 6 k 2 69.2 ECA k 70.7

Table 7 Results on the Pascal VOC 2012 validation set Model Year mIoU (%) HFCN [42] 2019 75.0 DeepLabv3 [5] 2017 77.21 DF-DAM [39] 2019 77.9 DeeplabV3+ [7] 2018 79.85 Exfuse [46] 2018 80.0 DANet [13] 2019 80.4 APCNet [16] 2019 80.71 EAPNet [41] 2021 81.70 CEAPNet (ours) – 83.98

Footnotes

Acknowledgements

References

Table 1
Comparison of different spatial pyramid methods on IDD lite validation set

Method mIoU (%) #Param.

ECA-ResNet101 + mSPP 68.7 45.97M

ECA-ResNet101 + mASPP 69.5 46.57M

ECA-ResNet101 + mAtIC 69.0 46.57M

ECA-ResNet101 + CCFAP 70.7 48.67M

Table 5
Comparison of different channel methods on IDD lite validation set. (Here, C: channel dimension, r: reduction ratio, and k: kernel size)

Method (#)Param. mIoU (%)

SE $2 C^{2} / r$ 67.6

CBAM $2 C^{2} / r + 2 k^{2}$ 68.0

Triplet attention $6 k^{2}$ 69.2

ECA k 70.7

Table 7
Results on the Pascal VOC 2012 validation set

Model Year mIoU (%)

HFCN [42] 2019 75.0

DeepLabv3 [5] 2017 77.21

DF-DAM [39] 2019 77.9

DeeplabV3+ [7] 2018 79.85

Exfuse [46] 2018 80.0

DANet [13] 2019 80.4

APCNet [16] 2019 80.71

EAPNet [41] 2021 81.70

CEAPNet (ours) – 83.98