Urban street scene analysis using lightweight multi-level multi-path feature aggregation network

Abstract

Urban street scene analysis is an important problem in computer vision with many off-line models achieving outstanding semantic segmentation results. However, it is an ongoing challenge for the research community to develop and optimize the deep neural architecture with real-time low computing requirements whilst maintaining good performance. Balancing between model complexity and performance has been a major hurdle with many models dropping too much accuracy for a slight reduction in model size and unable to handle high-resolution input images. The study aims to address this issue with a novel model, named M2FANet, that provides a much better balance between model’s efficiency and accuracy for scene segmentation than other alternatives. The proposed optimised backbone helps to increase model’s efficiency whereas, suggested Multi-level Multi-path (M2) feature aggregation approach enhances model’s performance in the real-time environment. By exploiting multi-feature scaling technique, M2FANet produces state-of-the-art results in resource-constrained situations by handling full input resolution. On the Cityscapes benchmark data set, the proposed model produces 68.5% and 68.3% class accuracy on validation and test sets respectively, whilst having only 1.3 million parameters. Compared with all real-time models of less than 5 million parameters, the proposed model is the most competitive in both performance and real-time capability.

Keywords

DCNN semantic segmentation encoder-decoder feature map dilated convolution

ï»¿

1. Introduction

Semantic scene segmentation is an important task in scene analysis wherein each pixel of an input image is assigned a class label. These labels or classes could include a wide range of objects, such as people, car, table, building, train, etc. depending on the data sets. This study focuses on urban street images which contain objects relevant to outdoor street scenes, such as traffic light, traffic sign, car, train, rider, etc. As with any research in this area, segmentation of partially occluded objects will be limited to the parts of these objects visible in the image.

Over the last decade, extensive research has been conducted in this field and it has been shown that in robotics, video surveillance and autonomous car driving industries, semantic segmentation plays a key role. It helps the machine segment an image automatically, but how efficiently and accurately machine can perform this task, is always a critical question. Scene segmentation models can be divided into two categories: off-line models [4, 5, 6, 32, 46] that require a large number of parameters and real-time models [24, 27, 37, 38, 41, 45] that have much less number of parameters. Real-time models are therefore more suitable for resource-constrained devices and are the main focus of our work. It is still a challenge in the research to develop real-time models that achieve good segmentation performance. We attempt to address this challenge in this work.

Previous studies have shown that deep convolution neural networks (DCNNs) are widely used in various domains such as classification [20, 34], object detection [35], instance and semantic segmentation [1, 14, 18, 21, 32]. Due to their high robust nature and handling capability of rich semantic features, they often produce promising results on various publicly available benchmarks [2, 8], mostly under an offline setting with very expensive GPU computing clusters. However, achieving a good performance in a real-time environment, especially for resource-constrained devices, is still a big challenge for all researchers. Over the decade, several real-time semantic segmentation models [23, 24, 41, 45] have been developed to address semantic segmentation for computer vision embedded devices, but the inability of handling high-resolution input images hampers model performance. High-resolution input demands large resources if the model is too deep. To maintain a balance between model complexity and input resolution, the authors of [26, 41, 45] introduce a multi-branch approach in which the deep branch is used for capturing rich contextual information whereas the shallow branch is responsible for retaining boundaries/edges details. Nevertheless, mutually independent branches hardly contribute to the learning ability of the model and also the addition of the shallow branch with high-resolution input image slows down its performance. To speed up the inference speed, some studies [1, 24] prune the redundant channels from the model architecture. Such approach boosts inference speed of the model by costing model’s performance.

For semantic segmentation, rich contextual details along spatial and channel dimensions are important. These contextual information can be extracted by a series of convolution layers. The common approach is to adopt a popular DCNN model as a feature extractor to extract details from the input. Many segmentation models use ResNet [36] as a backbone due to its high scalability and robustness. Moreover, the weights of ResNet models trained by ImageNet [9] dataset are easily available on the web which can boost segmentation performance. Due to its popularity in the field of image classification, several variants of ResNet model are proposed [39, 44]. However, with the increase in depth and width of the model, the number of parameters and floating point operations (FLOPs) also increase drastically which makes the model incapable of handling high-resolution input in real-time environments. Various off-line segmentation models, such as OCR [42], DeepLab (all variants), [4, 5] use ResNet as an encoder. However, for real-time execution, segmentation model needs a light-weighted backbone to reduce the computational cost and memory usage. The MobileNet architecture [13, 30] fulfills this requirement, thus it is suitable for embedded devices having low hardware specifications. Due to the optimised structure of residual blocks of MobileNet, it always results in less parameters and FLOPs, hence less memory usage. For that reason, many real-time scene segmentation models, such as ContextNet [26], FAST-SCNN [27], FANet [33], have adopted MobileNet residual blocks. Even in the field of image classification [34] and object detection [35], MobileNet is also widely used. Inspired by the optimized and robust design of MobileNet’s architecture, current research designs a light-weighted backbone network by assembling a series of residual bottleneck blocks found in MobileNet [13]. It will help the proposed semantic segmentation model to achieve the desired balance for real-time applications using resource-constrained devices.

Traditionally, an encoder-decoder design is employed in semantic segmentation in which rich contextual feature maps are extracted by the encoder and a series of upsampling layers are used in the decoder to reconstruct the scene and produce the output of the same size as the input. When using upsampling layers, a model can lose certain rich contextual details due to large semantic gap between levels. To reduce this semantic gap and localize the contextual information better, this research proposes a new technique, called Multi-level Multi-path Feature Fusion (M2-FF) method for semantic segmentation. This technique is discussed in detail subsequently.

Figure 1.

Number of parameters vs class mIoU on Cityscapes test set – proposed M2FANet sets a new state-of-the-art performance among semantic segmentation models of having less than 5M parameters.

In pursuit of better contextual representation, this study exploits different feature scaling techniques such as Pyramid Pooling Module (PPM) [46] and Atrous Spatial Pyramid Pooling (ASPP) module [5]. Due to the optimized design and usage of dilated convolutions, ASPP performance is better than PPM and it also provides better object localization facility by maximizing the size of the receptive field. In ablation study section, the details about the model’s performance using both PPM and ASPP are discussed.

The key contributions of the proposed study are as follows: firstly, this research presents an optimized light-weighted backbone by exploiting residual blocks of MobileNetV2 to extract rich contextual details from high-resolution input images. Due to its’ optimized architecture and less number of parameters, the proposed model is suitable for resource-constrained real-time semantic scene segmentation applications. Secondly, it also introduces a new, yet effective, feature aggregation technique (M2-FF) at the decoder end for better region localization and context engrossment. Top-down paths in M2-FF module ensures better context assimilation whereas bottom-up path enhances the entire feature hierarchy by propagating the local signals from bottom layer to top layer. Thirdly, by introducing a feature scaling module, the proposed model shows a better way to maximize the receptive field for capturing the contextual details from the feature map without adding any extra parameters. Fourthly, the proposed model is tested and evaluated on two different publicly available benchmarks on urban street scenes. Finally, the results of the study clearly state that compared to the existing semantic segmentation models of less than 5 million parameters, the proposed model achieves state-of-the-art results on Cityscapes [8] dataset. The plot in Fig. 1 displays the position of the proposed model in the real-time category. The number of parameters and GFLOPs of other competitive real-time models are at least five times higher than the proposed model which makes the proposed model (M2FANet) superior than the existing real-time scene segmentation models for resource constrained embedded devices.

The following table shows all the acronyms used in this paper.

Table 1

Acronyms

Acronym	Full form
DCNNs	Deep Convolution Neural Networks
CNN	Convolution Neural Network
GPU	Graphics processing unit
FLOPs	Floating Point Operations
ASPP	Atrous Spatial Pyramid Pooling
PPM	Pyramid Pooling Operation
FPS	Frame Per Second
mIoU	mean Intersection over Union

2. Literature review

Semantic scene segmentation models typically follow a pyramid architecture. In the encoder stage, a deep CNN typically computes a feature hierarchy layer by layer and develops an inherent multi-scale pyramid shape. At the decoder end, the high-semantic feature map is up-sampled and fused with the previous layers’ feature maps through lateral connections to recover higher spatial dimensions. Once spatial details are completely extracted, the model will predict the class label for each pixel to complete the segmentation process. Different DCNN models, such as VGG [32], ResNet [36], Xception [7], and MobileNet [13], are often used as a feature extractor in the encoder end. Due to the deep network design, most of these DCNNs need extensive hardware support to generate output. However, due to the overwhelming use of mobile devices, developing real-time semantic segmentation model became an important research area in the field of computer vision. Therefore, in the following subsections, first, we discuss the models which are designed for offline computation, then we provide the details of semantic segmentation models targeting real-time computation.

2.1 Off-line segmentation

Traditional semantic scene segmentation models are off-line models due to their deep architectural design. Inspired by the outstanding performance of DCNNs in the field of image classification, Fully Convolutional Network (FCN) [21] has shown a revolutionary approach to deploying a DCNN model as an encoder. It replaces the fully connected layers from the top of the DCNN by a convolution layer to generate a spatial map and feeds it to the decoder circuit in order to produce segmentation output. Based on this foundation, several offline models, such as UNet [29], SharpMask [25], DeepLabV2 [4], RefineNet [16], PSPNet [46], DeepLabV3 [5], OCR [42], HANet [6] are then developed by introducing lateral connections between the low-level feature maps across resolutions and semantic levels. Most of these off-line semantic segmentation models use ResNet-101 or higher variants [38] as the feature extractor due to their capability to capture many vision features. The literature has also shown that with the increase of network depth and width, the performance of the model improves [5, 39, 46]. For instance, with the use of deep ResNet-101 as the encoder, DeepLab [5] generates a better accuracy than using VGG-16 [32] or ResNet-50 [36] or Xception [7]. However, due to the large depth (101 layers) and large width (2048 channels), ResNet-101 contributes a large number of parameters and GFLOPs which makes all semantic segmentation models using it [5, 6, 16, 42, 46] computationally inefficient to run in real time. This elevates the need for designing lightweight scene segmentation models for resource-constrained embedded devices.

2.2 Real-time segmentation

Due to the growing demand of designing real-time applications in the field of classification [22], clustering [3], segmentation [14, 24] and object detection [35], considerable researches have been conducted to optimize the existing off-line models and to produce acceptable real-time performance on resource-constrained devices. SegNet [1] was one of the pioneering models targeting real-time computation by introducing a small architecture. Although, it reduced model overall parameters as compared to offline models, but model still requires extensive hardware support to run. Later on, ENet [24] proposed an extremely efficient framework by reducing the number of down-sampling operations and introducing high dilation rate in bottleneck blocks. It helps the model reduce the number of parameters and GFLOPs drastically; however, the accuracy of this model is severely compromised. Few more studies, such as ICNet [45], ContextNet [26], BiSeNet [41], GUN [43], were developed to improve the real-time performance. All these models introduced a new approach, called multi-branch approach targeting real-time computation. Figure 2b demonstrates the design of multi-branch encoders. In this method, one dedicated deep branch is introduced in the encoder to extract rich contextual information from input. To reduce the computational cost due to the depth of the branch, it accepts a lower resolution input whereas other branches, named as shallow branch, receive a higher resolution input. The shallow branch is mainly deployed for capturing boundary and texture details from the input which will be used at the decoder to fuse with deep features. Thus, this multi-branch approach satisfies the goal for real-time computation with high resolution input images. However, independent branches at the encoder did not contribute to the learning ability of the model. Moreover, the independent multi-branch approach introduces a large semantic gap between shallow and deep features.

Figure 2.

Different approaches. From left to right: (a) One-branch encoder, (b) Multi-branch encoder, (c) Feature reuse in sub-encoder (d) M2-FF decoder. This approach uses semantic features at different levels from encoder network and progressively map them in the multiple paths at decoder end. Dotted arrows (green color) in the last path defines the presence of feature aggregation path in which features from encoder will be mapped with the features at current path by skip connections.

By considering the drawback of the independent branches, FAST-SCNN [27] introduced a new technique, called Down-Sampling. Instead of using a completely separate branch from the beginning, it deployed a down-sampling module at the initial stage of the pipeline. This module reduced the input resolution to a quarter of the original input size. After this module, it created two branches – a deep branch for feature extraction and a shallow branch for preserving texture and pattern details. This approach reduced the computational cost and enhances the performance. However, FAST-SCNN could not reduce the large semantic gaps between local and global features. Moreover, the model loses its ability to retain boundaries information due to scaling up the final feature by a large factor at the decoder.

In contrast to these approaches, DFANet [15] introduces the concept of sub-networks in the encoder. Traditionally, an encoder follows pyramid structure in which a stack of convolution layers processes an input image and reduces its spatial dimensions as the input reaches to the end. This low resolution high semantic feature map is used as an input for decoding. In contrast to this traditional approach, DFANet [15] uses this semantic feature as an input for next sub-encoder network and thus the process is repeated for the third sub-encoder. Layers at the same level but in different sub-encoders share their gradient information among themselves. This approach produced new state-of-the-art results in real-time scene segmentation. However, it still has as many as 7.8 million parameters and large GFLOPs due to multiple sub-networks at the encoder end.

The first three diagrams in Fig. 2 show the architectural difference of existing approaches. It clearly displays that all existing methods upsample the final semantic feature map by a large factor in the decoder side which causes a loss in spatial details. Thus, the boundary degeneration effect can be noticed in the output. To address this issue, few existing real-time models such as FAST-SCNN [27], DFANet [15] introduced a simple feature fusion module which fuses deep features with shallow features at a particular stage before upsampling the final feature map. This approach helps the model to fuse contextual details with spatial details at lower resolution. However, it could not resolve the boundary degeneration effect. Moreover, mapping shallow features at higher resolution with deep features at lower resolution introduces a large semantic gap. To reduce this semantic gap between the spatial dimensions of the local features and global features, features at different levels need to be added in multiple directions. Hence, this study proposes a new technique, called M2-FF, in the decoder side to fuse features from top five stages in multiple directions. Furthermore, introduction of this module eliminates the need for upsampling the final feature map by a large factor. Thus, it addresses boundary degeneration effect in scene parsing. The far right diagram (d) of Fig. 2 illustrates the proposed approach.

The idea to fuse features at different levels was firstly introduced by FPN [17] and it was a multi-level feature fusion technique. By adding a top-down path and lateral connections among layers at same level, it tries to reduce the semantic gap between semantic feature maps of different levels. Later on, PAN [18] shows that adding a top-down path at decoder reduces semantic gap but introduces a localization issue unfortunately. To boost the localization capability of the entire feature hierarchy, [18] proposes another path from bottom to top to fuse local features with global features. By adopting this technique, an efficient, scalable object detection model, named EfficientDet [35], has recently been introduced by the Google Brain team. Both studies [18, 35] have shown that the addition of a bottom-up path provides better object localization and context accumulation in the feature maps. Motivated by this approach, current study introduces a novel Multi-level Multi-path feature fusion approach at the decoder end. Thus, the proposed study addresses all the issues of the existing real-time scene segmentation models.

3. Proposed method of real-time semantic segmentation model

In this section, the complete architecture of the proposed model is discussed. By exploiting dilated convolution, depth-wise separable convolution (DsConv), feature scaling and multi-stage feature fusion techniques, this study significantly extends the preliminary research [33] without drastic an increase of parameters and computational cost.

3.1 Network architecture

The overall architecture of the proposed M2FANet model is shown in Fig. 3. In what follows, the detail of every component of the model, including the backbone network, M2-FF, ASPP, classifier module, dilated convolution, depth-wise separable convolution, and non-linearity functions is addressed.

Figure 3.

Complete pipeline of M2FANet.

3.1.1 Encoder network

Since the main focus of this research is to design an optimized scene segmentation model for resource-constrained devices, it employs the mobile residual bottleneck block (MBConv) of MobileNetV2 [30]. In the preliminary investigation FANet [33], a down-sampling module is deployed at the beginning of the network to reduce the input resolution to 1/8 of the original input size before passing the tensor to residual blocks. However, in the proposed design, it is replaced by MBConv blocks with less channels. Utilization of down-sampling module generates less parameters and GFLOPs. It mainly controls the input resolution and boundary details by not contributing much for holding the spatial details from the input scene. On the other hand, it has shown that deploying MBConv of different expansion ratios at the initial stage preserves more contextual and spatial details due to its squeeze and excitation architecture [30]. Although MBConv block generates more parameters and GFLOPs compare to the down-sample module, it still has less channels at the initial stage and hence controls the increase of model parameters and GFLOPs.

The proposed study uses two types of bottleneck residual block of MobileNetV2 [30] – MBConv1 and MBConv6, which are named based on their expansion ratio. It is the ratio between the size of the input bottleneck and the inner size, and is either 1 or 6. Each block contains an input followed by several bottlenecks, then followed by an expansion. To reduce the number of parameters, Depth-wise Conv (DwConv) layer is deployed at the expansion stage. Table 2 displays the structure of an MBConv block. Here, Non-linearity activation is deployed in the first two layers as as suggested in [30]. However, to retain meaningful information, it is skipped after the last layer of each MBConv. The filter size of the first block is 24 and it is progressively increased in the successive blocks. It is shown in [30] that the use of width multipliers from 0.35 to 1.25 for all resolutions generally produces a better performance. Based on this strategy, the proposed study uses multipliers between 0.35 and 0.5 to set the number of channels in the successive MBConv blocks. It avoids higher values of the width multiplier to control model width and make it computationally efficient for real-time applications. To exploit the architectural advantage of [12], this research uses a squeeze and excitation module in each residual bottleneck block and test the model’s performance. In the result section, it compares the performance of MobileNetV2 and MobileNetV3 residual blocks. Previous publications [12, 40] demonstrate that the addition of the squeeze and excitation module in the bottleneck architecture improves performance; however, the complete architecture of the network totally depends on Network Architecture Search (NAS) [10], which makes network design unpredictable. It also requires high CPU/GPU power to leverage neural architecture search for automatically predicting feature network design. To keep the proposed network design simple, predictable and usable for real-time computation, the final proposed model uses MBConv blocks of MobileNetV2. Experimental results also confirm that the inverted residual block of MobileNetV2 performs better than MobileNetV3 blocks. Moreover, the addition of squeeze and excitation module of MobileNetV3 results in a large number of parameters and GFLOPs, which makes the model more difficult to run in a real-time environment.

Table 2
Bottleneck residual block

Input	Operator	Output
$h\times w\times c$	1 $\times$ 1 Conv, 1/1, Relu	$h\times w\times tc$
$h\times w\times tc$	3 $\times$ 3 DwConv, 3/s, Relu	$h/s\times w/s\times tc$
$h/s\times w/s\times tc$	1 $\times$ 1 Conv, 1/1, -	$h/s\times w/s\times c^{\prime}$

Table 3

Layer architecture of backbone network

Stage (i)	Input	Operators	Layers (n)	stride(s)	Output
1	1024 $\times$ 2048 $\times$ 3	Conv, k3 $\times$ 3	1	2	512 $\times$ 1024 $\times$ 32
2	512 $\times$ 1024 $\times$ 32	MBConv1, k3 $\times$ 3	1	2	256 $\times$ 512 $\times$ 24
	256 $\times$ 512 $\times$ 24	MBConv6, k3 $\times$ 3	2	1	256 $\times$ 512 $\times$ 32
3	256 $\times$ 512 $\times$ 32	MBConv6, k3 $\times$ 3	3	2	128 $\times$ 256 $\times$ 48
4	128 $\times$ 256 $\times$ 48	MBConv6, k3 $\times$ 3	3	2	64 $\times$ 128 $\times$ 64
5	64 $\times$ 128 $\times$ 64	MBConv6, k3 $\times$ 3	3	2	32 $\times$ 64 $\times$ 96
6	32 $\times$ 64 $\times$ 96	MBConv6, k3 $\times$ 3	2	2	16 $\times$ 32 $\times$ 128
7	16 $\times$ 32 $\times$ 128	MBConv6, k3 $\times$ 3	1	2	8 $\times$ 16 $\times$ 160

The layer architecture of the backbone network is displayed in Table 3. It clearly displays that an input image passes through maximum seven stages in the encoder side to produce a global feature map. For an input image of 1024 $\times$ 2048, the final feature map at encoder end will be 8 $\times$ 16, which may lose contextual details due its low resolution. To address this issue, an effective technique, called Multi-level Multi-path feature fusion is introduced which fuses features of last five stages in the multiple paths. Previous work [5] suggests that an output stride of $2^{4}$ in the encoder would be the best for semantic segmentation. However, with $2^{4}$ output stride and a full resolution input, real-time computation would be prohibitive. A global feature map of high resolution (64 $\times$ 128 px for $2^{4}$ stride) leads to a high consumption of memory, making model infeasible to run in resource constrained devices. The work in [5] proposes a very deep off-line semantic segmentation model which can only handle 513 $\times$ 513 px input, whereas the proposed model accepts input of up to 1024 $\times$ 2048 px. Therefore, to reduce the computational cost and enable the model to run in a resource-constrained computer vision system such as embedded devices, the proposed method uses maximum $2^{7}$ stride in the encoder side for high resolution input. Most of the real-time scene segmentation models with high input resolution, has a large output stride. The benchmark Cityscapes dataset [8] provides images of 1024 $\times$ 2048 px, whereas CamVid [11] supplies comparably low-resolution images. To handle low resolution datasets, stride can be reduced up to $2^{5}$ .

3.1.2 Multi-level multi-path feature fusion

The literature has shown that fusing features at different levels helps the model combine rich contextual information with spatial details and reconstruct the image. Higher-level neurons strongly respond to entire objects while low-level neurons more likely capture local texture and patterns which stimulates the necessity of adding a top-down path to propagate rich semantic features from high to low level and enrich all features at different levels with sensible classification knowledge.

Traditionally, a high-level rich semantic feature map $F_{i}(C_{i}\times H_{i}\times W_{i})$ is up-sampled and mapped with the former low-level features $F_{i-1}(C_{i-1}\times H_{i-1}\times W_{i-1})$ [1, 29, 17] to regenerate the scene. Such up-sampling and mapping process will continue as long as the final prediction of the original input size is not produced. After every upsampling, the model loses certain gradient information. Moreover, one direction propagation does not ensure the localization of each object in the scene. This phenomena clearly manifests the need of another bottom-top path to enable the localization capability of the entire feature hierarchy. By adding a bottom-up path augmentation, PAN [18] preserves the local context. Inspired by the path augmentation techniques presented in [17, 18], a new feature aggregation technique, called Multi-level Multi-path feature fusion is introduced at the decoder end. The design of this module is also influenced by the design of a technique known as BiFPN, which is used in [35] for object detection. The backbone network of the proposed model produces semantic feature maps at multiple stages. In each stage, the input will be convolved to reduce its size by half. For M2-FF, feature maps from the third stage (S3) to the seventh stage (S7) are used. The complete structure of the feature fusion module is shown in Fig. 4. Different from [18, 35], the proposed method introduces another top-down path to aggregate rich semantic feature maps of M2-FF with the feature maps generated by the encoder through lateral connections. This helps the suggested model to retain gradient information from the previous layers.

Figure 4.

Design of Multi-level Multi-path feature fusion module at decoder. a) Top-down path for multi-scale feature fusion (blue arrows), b) Bottom-up path for better localization (green arrows), c) Top-down path for better feature aggregation with the help of skip connections (red arrows) from backbone. There are lateral connections (black arrows) among the layers at same levels.

Formally, given a list of semantic features $F^{in}=(F_{l_{3}}^{in},F_{l_{4}}^{in},\ldots,F_{l_{i}}^{in})$ generated by different stages (S3 to S7) in the encoder stage, the aim is to find a transformation $f$ that can effectively map different features at different semantic levels and produce rich semantic feature maps $F^{\textit{out}}=(F_{l_{3}}^{\textit{out}},F_{l_{4}}^{\textit{out}},\ldots,F_{% l_{i}}^{\textit{out}})$ at all levels. In the first top-down path, coarse features from the top level (S7) will be up-sampled by factor of 2 and then mapped with the former stage features. Here, simple add function is utilized to map coarse features of two consecutive levels. Thus, this top-down path generates features ( $N_{l_{7}}^{in}$ , $N_{l_{6}}^{in}$ , $N_{l_{5}}^{in}$ , $N_{l_{4}}^{in}$ , $N_{l_{3}}^{in}$ ) for the next path. In the later path, it starts from the bottom layer (S3) and gradually down-samples features by a factor of 2 before it gets mapped with next higher level feature maps. Thus, this bottom-up path creates features ( $M_{l_{3}}^{in}$ , $M_{l_{4}}^{in}$ , $M_{l_{5}}^{in}$ , $M_{l_{6}}^{in}$ , $M_{l_{7}}^{in}$ ). Finally, feature aggregation path aggregates these features in top-down direction and also maps with features ( $F_{l_{7}}^{in}$ , $F_{l_{6}}^{in}$ , $F_{l_{5}}^{in}$ , $F_{l_{4}}^{in}$ , $F_{l_{3}}^{in}$ ) generated by encoder network. Hence, M2-FF produces the output as follows:

$\displaystyle F_{l_{7}}^{\textit{out}}=\textit{DsConv}(M_{l_{7}}^{in}),$ (1) $\displaystyle F_{l_{6}}^{\textit{out}}=\textit{DsConv}(F_{l_{6}}^{in})+\textit% {DsConv}(M_{l_{6}}^{in})+\textit{DsConv}(\textit{Upsample}(F_{l_{7}}^{\textit{% out}})),$ (2) $\displaystyle F_{l_{5}}^{\textit{out}}=\textit{DsConv}(F_{l_{5}}^{in})+\textit% {DsConv}(M_{l_{5}}^{in})+\textit{DsConv}(\textit{Upsample}(F_{l_{6}}^{\textit{% out}})),$ (3) $\displaystyle F_{l_{4}}^{\textit{out}}=\textit{DsConv}(F_{l_{4}}^{in})+\textit% {DsConv}(M_{l_{4}}^{in})+\textit{DsConv}(\textit{Upsample}(F_{l_{5}}^{\textit{% out}})),$ (4) $\displaystyle F_{l_{3}}^{\textit{out}}=\textit{DsConv}(F_{l_{3}}^{in})+\textit% {DsConv}(M_{l_{3}}^{in})+\textit{DsConv}(\textit{Upsample}(F_{l_{4}}^{\textit{% out}})).$ (5)

To reduce number of operations and number of parameters, the standard convolution layer is replaced by depth-wise separable convolution layer.

3.1.3 Atrous spatial pyramid pooling

ASPP, introduced by [5], is a powerful tool to explicitly control the spatial dimensions of semantic feature maps produced by DCNNs. It also enhances the ability of DCNNs to handle both large and small objects efficiently by providing a robust mechanism to control the field-of-view of filters in convolution layers.

A dataset usually contains objects of different classes with various sizes. Although DCNNs have shown astonishing capability of classifying an arbitrary region of scene by employing a small kernel, typically 3 $\times$ 3, less frequent and small objects are always overlooked by the presence of large objects in the scene. To overcome this issue, the authors of [4, 5] have shown that objects of an arbitrary scale can be precisely classified by resampling convolutional features with different sampling rates. Motivated by this approach, this study adopts dilated convolution technique with different dilation rates. Formally, an ASPP layer with dilation rate $r$ introduces $(r-1)$ zeros between two consecutive values of the filter in order to enlarge $t$ times the kernel size of a $k\times k$ filter to $k_{e}=k+(k-1)\cdot(r-1)$ without increasing parameters or GFLOPs. Thus, it provides the best trade-off between small field-of-view and large field-of-view. For an input feature map $x$ , each location $i$ on the output feature map $y$ will be sampled by ASPP as follows:

$\displaystyle y[i]=\sum_{k=0}^{K}x[i+r\times k]w[k]$ (6)

where the dilation rate $r$ determines the stride with which it convolves the input feature. The filter’s field-of-view will be adaptively updated by changing the dilation rate. In this study, ASPP module uses multiple parallel dilated convolution branches with three different dilation rates: 6, 12, and 18. The features extracted from multi-path multi-scale feature fusion module are further processed separately in each atrous branch and then concatenated to each other to produce the final result. Figure 3 demonstrates the whole process of the ASPP module.

Typically, ASPP is deployed on top of the encoder network to control the spatial resolution of the resulting feature maps if the output stride of encoder is $2^{4}$ . In the proposed model output stride is $2^{7}$ which means rich semantic feature map with low spatial dimensions. Engrossing rich contextual details with the help of ASPP will be effective if the feature map size is eight to sixteenth times smaller than the original input. High dilation rate at the dilated branch of ASPP helps creating a large receptive field for better localization of small and large objects in the scene. Moreover, it also reduces number of operations drastically in the convolution process. Hence, by looking at the motivation factors of utilizing ASPP, it is deployed on top of the M2-FF module. M2-FF generates final aggregated feature map of 128 $\times$ 256 px which is eight times smaller than the original input. Moreover, high dilation rates (6, 12, and 18) at different branches of feature scaling module process the input quickly with less GFLOPs.

3.2 Classifier module

This is the last module of the whole pipeline. Features produced by the ASPP module are upsampled by a factor of 4 and then simply mapped with the second level (S2) features ( $F_{l_{i}}^{in}$ ) of the encoder through lateral connections. To avoid the loss of semantic details, every upsample layer is followed by one DwConv layer and a point-wise standard convolution (1 $\times$ 1 Conv). Two 3 $\times$ 3 DsConv layers are exploited in the classifier module to refine the feature maps, followed by one more upsample layer to produce an output of the same size as the input. Finally, proposed model employs the softmax activation function to assign a class label to every individual pixel. This study uses 19 classes for Cityscapes and 11 classes for CamVid.

3.2.1 Depth-wise separable convolution

To achieve the target of designing an efficient scene segmentation model for resource-constrained devices, this study replaces the standard Conv layer by DsConv in many places. In the feature aggregation module, after fusing features of two consecutive levels, it deploys a separable convolution block in which a depth-wise separable convolution is performed, followed by a batch-normalization layer. DsConv factorizes a standard convolution into two stages – a depth-wise convolution (DwConv) and a point-wise convolution (1 $\times$ 1 Conv). It reduces a large number of parameters and GFLOPs. Proposed study also uses dilated depth-wise convolution as it is support by TensorFlow DsConv layer [31]. It uses different dilation rates to increase the size of receptive field.

3.2.2 Nonlinearities

The selection of the activation function in DCNN models has a significant impact on model prediction. It helps the model preserve non-linearity while passing knowledge from one layer to the next layer. The most widely used non-linearity function is Rectified Linear Unit (ReLU) due to its strong convergence rate during gradient descent optimization. Although, other activation functions such as Sigmoid, Tanh are smoother than ReLU, but they have not been as popular as ReLU. Recently, the Google Brain team has proposed a new activation, called swish [28]. Unlike ReLU, swish is smooth and non-monotonic. Empirically, [28] has shown that swish boosts MobileNet [13] performance on ImageNet [9] dataset by 2.2% over ReLU and concludes that for light-weight model, swish is the best activation function. It also comes with non-zero cost in real-time environment. Motivated by [28, 34, 35], the proposed study employs swish non-linearity after every convolution layer. It is defined as:

$\displaystyle\textit{swish}(x)=x\cdot\sigma(x),$ (7)

where $\sigma(x)$ defines sigmoid activation function.

4. Experiment

To evaluate the proposed model in urban street scene analysis, extensive experiments are carried out on two different benchmark datasets: Cityscapes [8], CamVid [2]. Experimental results clearly illustrate that the proposed model outperforms many existing semantic segmentation models which have less than 5 million parameters. In the following subsections, this paper first discusses the datasets and implementation details, followed by a series of ablation studies on the Cityscapes benchmark dataset. Finally, it compares the proposed model with some existing off-line and real-time segmentation models and report the results on both validation and test sets. Consistent with previous work, this study reports model parameters, GFLOPs, class and category meanIoU.

4.1 Datasets

Cityscapes For urban street scene analysis, Cityscapes [8] is the most popular choice for researchers. It is a large-scale dataset, mainly used for object detection, instance, semantic and panoptic segmentation. It provides annotated data for more than 30 classes grouped into 8 categories. The dataset consists of around 5,000 fine annotated images and 20,000 coarse annotated ones.

The proposed study mainly used fine-tune images and considered only 19 classes for pixel annotations. The whole dataset is divided into three parts- training set (2,975 images), validation set (500 images) and test set (1,525 images). The labels for the training set and validation set are supplied by the benchmark whereas the labels for test set are not provided. However, test set predictions are submitted to the Cityscapes evaluation server and the results are discussed in the results section.

CamVid It is a small dataset [2], mainly designed for object detection in automated driving vehicle. The images of this dataset were generated from a recorded video, then annotated frames were created by assigning a class colour to each object of the frame by human operators. This dataset contains only 267 images for training, 101 for validation and 233 for testing. Out of 32 classes, current study uses 11 classes (excluding void) for performance evaluation. Due to the small size of the dataset, models were usually under-learned.

4.2 Implementation details

To conduct this experiment, this study uses a dual Nvidia GeForce RTX 2080Ti GPUs system, each GPU has 11 GB of memory. To exploit the parallel processing power of GPUs, it uses CUDA 10.2. The proposed model is developed using tensorflow 2.1.0 and keras 2.3.1. To utilize both GPUs in data-parallel distributed training environment, it employs the horovod framework [31]. Horovod takes a single-GPU tensorflow program and trains it on multiple GPUs. For instance, in this research, horovod divides the whole training set into two sets and runs the training script on individual set in each GPU. Thus, it boosts the run-time performance by effectively utilizing all resources. Stochastic gradient decent (SGD) is used as model optimizer with a momentum of 0.9.

Figure 5.

Finding optimal learning rate.

Inspired by [4, 13, 46], the proposed study uses the ‘poly’ learning rate by setting 0.045 as the base value and 0.9 as the power. To find out the optimal learning rate in each epoch during training the model, model is trained for 5 epochs using a polynomial scheduler and the corresponding losses against different learning rates are plotted. Thus, the upper and lower bound of learning rate for training are set. Figure 5 illustrates the plot of learning rate vs model loss as an example. To calculate model loss, the categorical cross entropy loss function is exploited.

Following the suggestion by MobileNetV2, the proposed model uses $l_{2}$ regularization with value 0.00004 for top layers except depth-wise convolution layer. To avoid model over-fitting due to limited data, various data augmentation techniques such as random horizontal flip, vertical flip, random crop, resizing and adjusting brightness, saturation and contrast of images are applied. Finally, a dropout layer is deployed before the final classification in the training process. Adding dropout layer helps to regulate model over-fitting issue and also improve validation accuracy. After training the model several times with different dropout rates, it has been noticed that a dropout rate less than 0.2 cannot address the over-fitting issue whilst a value more than 0.6 value causes under-fitting. Therefore, taking the average of these two values as the dropout rate would be beneficial for model prediction.

4.3 Ablation study

Previous study [33] has shown the effectiveness of using modified design of BiFPN for region localization and context aggregation. Taking that as a starting point for this study, the design of FANet is updated for the better scene segmentation. Table 4 shows the evaluation of the proposed M2FANet model. Clearly, it produces better results than the existing real-time scene segmentation models having less than 5 million parameters.

Table 4
Results of ablation study on Cityscapes validation set

Method	Backbone	Feature fusion module	Feature scaling method	Mean IoU
(%)	Parameters (Million)
FANet	9 MBConv6	FPN	–	62.6	1.2
FANet	9 MBConv6	BiFPN	–	65	1.1
FANet	9 MBConv6	Modified BiFPN	–	65.9	1.1
M2FANet	MobileNetV2	M2-FF ${}^{*}$	ASPP	67.7	2.7
M2FANet	MobileNetV3 ${}^{*}$	M2-FF ${}^{*}$	ASPP	3.7	67.5
M2FANet	15 MBConv6	M2-FF ${}^{*}$	–	67.4	1.25
M2FANet	15 MBConv6	M2-FF ${}^{*}$	PPM	68.0	1.37
M2FANet	15 MBConv6	M2-FF ${}^{*}$	ASPP	68.5	1.28

${}^{*}$ In MobileNetV3, it uses MBConv blocks with squeeze and excitation blocks. This study redesigns the architecture of the modified BiFPN design and proposes a new design: M2-FF.

The first three rows of Table 4 show the preliminary result of FANet already presented in [33]. From the forth row, it shows the additional results obtained from the ablation study using the proposed model. At the initial stage, the backbone of FANet is replaced by an ImageNet pre-trained MobileNetV2 model and performance is measured on Cityscapes validation set. MobileNetV2 [30] has 12 MBConv blocks with increasing channel sizes (16 to 320). The literature shows that increasing the depth and width of the model will likely enhance the performance. However, it also increases model parameters and number of operations. Specifically, MobilenetV2 has more than 2.6M parameters, and this increases model’s overall parameters and GFLOPs. Similarly, the fifth row of Table 4 shows that the use of MobileNetV3 [12] as feature extractor of the new design supplements model performance, but at the cost of a drastic increase in computation. Moreover, the addition of squeeze and excitation modules in residual blocks of MobileNetV3 is determined by Neural Architecture Search (NAS) [40] which makes model structure unpredictable. Therefore, optimizing the backbone architecture is really difficult for MobileNetV3.

The above observation motivates this study to stick with MBConv blocks of MobileNetV2. The depth and width of the FANet model is modified by increasing the number of residual blocks and width multiplier. The proposed model replaces down-sampling module of FANet by three MBConv blocks of different expansion ratios and last two MaxPooling layers of FANet model are also replaced by three MBConv blocks. Table 4 clearly illustrates that due to the addition of new blocks, the number of parameters is increased by 0.2M; however, the model’s performance is boosted by 2.6%. To control the number of parameters of MBConv blocks, the proposed design uses the width multiplier between 0.35 and 0.5 to increase model’s width. Table 4 also shows that without using any feature scaling technique, M2FANet enhances the performance by 1.5% compare to FANet. However, empirically it has been proved that feature scaling at different branches with different scaling rates can further escalate the model’s performance.

This study explores two powerful feature scaling techniques: PPM [46] and ASPP [5]. It is noticeable from Table 4 that both techniques boost the performance by a recognised percentage. Compare to PPM, utilizing ASPP at the decoder side is more effective due to its design and the presence of dilated convolution branches. In PPM, features are processed by four ImagePooling branches with different rates whereas in ASPP, one ImagePooling branch, one 1 $\times$ 1 Convolution, and three 3 $\times$ 3 dilated convolution branches process the features with different dilated rates. Such an architecture provides a large receptive field for localization and context assimilation. The design of ASPP can be visualized in Fig. 3. By exploiting all these techniques, the current study finally extends the design of FANet with a new backbone, M2-FF and ASPP modules, and produces 68.5% mean IoU on the Cityscapes validation set.

4.4 Model evaluation

This section presents the performance of the proposed model on the Cityscapes dataset and compares model’s performance with other existing off-line and real-time scene segmentation models. It presents model parameters, GFLOPs, pixel accuracy, Class and category mean Intersection Over Union (mIoU) on validation and test sets. It also demonstrates inference time and FPS (frame per second) of the models which are trained under the same system configuration. This study did not pre-train the model with ImageNet [9] dataset. Note that the domain of ImageNet dataset is different from urban street scenes. Due to this different domain knowledge, this study focuses urban street scene benchmarks and trains the model with related domain knowledge datasets. Cityscapes [8] provides 5,000 fine-tune and 20,000 coarse annotated images for training the model. The current study reports both results of fine-tune and fine-tune with weakly annotated data (Table 5) at 512 $\times$ 1024 px. Due to hardware limitation, the proposed model could not be trained with fine-tune and coarse datasets together at full input resolution. It is evident from Table 5 that the performance is slightly improved by 0.3% due to the additional coarse dataset. It is expected that the performance could be further improved by 0.5–1% on the validation and test sets if the model is trained with both coarse and fine-tune datasets together at full resolution. This study aims to address this in the future.

Table 5
Model performance on Cityscapes fine-tune and coarse datasets

Model	Input	Dataset	Class meanIoU (%)	C ategory meanIoU (%)
FANet [33]	512 $\times$ 1024	Fine-Tune only	59.7	81.4
M2FANet	512 $\times$ 1024	Fine-Tune only	62.3	83.6
M2FANet	512 $\times$ 1024	Fine-Tune $+$ Coarse	62.6	83.8

It also trains the proposed M2FANet on the fine-tune datasets at different input resolutions and present the results in Table 6. At 512 $\times$ 1024 px input resolution, the feature extractor extracts feature maps of 4 $\times$ 8 px, which is very small. Therefore, study suggests not to use input image smaller than this size. Basically, the proposed model targets high-resolution input (1024 $\times$ 2048) for real-time computation. For small resolution input, it can replace stride by 1 in last two blocks on encoder network. However, little modification is also required in the M2-FF design. Table 6 illustrates accuracy of M2FANet for each class at different input resolutions. In the top 5 classes (road, sky, vegetation, building, car), the model’s accuracy is more than 90%, and its accuracy is less than 50% only in four classes (wall, fence, rider and motorcycle). Due to uneven class distribution in the dataset, the accuracy among all classes varies like many other methods. For instance, motorcycle class has less than 0.1% pixel in whole dataset which causes low prediction accuracy (33.4%) for this class. Table 6 also illustrates that its performance is also improved at higher input resolutions.

Table 6

Class-wise M2FANet performance on validation set at different input resolutions

Input size	Road	S.walk	Build.	Wall	Fence	Pole	T.light	T.sign	Veg.	Terrain
1024 $\times$ 2048	94.9	76.6	90.1	45.7	48.5	55.8	58.9	70.3	90.7	56.1
	Sky	Person	Rider	Car	Truck	Bus	Train	M.cycle	Bicycle	mIoU
	93.3	74.2	47.8	92.1	69.1	73.6	61.8	33.6	68.1	68.5
Input size	Road	S.walk	Build.	Wall	Fence	Pole	T.light	T.sign	Veg.	Terrain
768 $\times$ 1536	94.7	74.1	88.2	43.5	45.1	48.6	53.9	68.3	90.2	53.3
	Sky	Person	Rider	Car	Truck	Bus	Train	M.cycle	Bicycle	mIoU
	90.8	74.5	46.2	90.4	50.5	68.9	48.2	36.5	66.8	64.9
Input size	Road	S.walk	Build.	Wall	Fence	Pole	T.light	T.sign	Veg.	Terrain
512 $\times$ 1024	94.2	73.7	87.6	34.2	41.8	54.3	50.9	66.2	90.4	52.2
	Sky	Person	Rider	Car	Truck	Bus	Train	M.cycle	Bicycle	mIoU
	89.4	73.4	44.6	90.2	37.7	61.0	34.7	38.2	68.6	62.3

Table 7

Category-wise M2FANet performance on Cityscapes validation dataset

Input size	Flat	Construction	Object	Nature	Sky	Human	Vehicle	mIoU
1024 $\times$ 2048	94.7	89.6	62.4	91.0	93.3	75.5	90.3	85.3
768 $\times$ 1536	94.4	88.5	60.3	90.7	90.8	75.6	88.6	84.1
512 $\times$ 1024	94.5	87.7	59.7	90.4	89.4	75.8	88.1	83.7

Table 7 displays the category-wise model performance at different input resolutions. In the Cityscapes dataset, all classes are distributed into seven categories: flat, construction, object, nature, sky, human, and vehicle. The proposed model’s performance is outstanding in five categories (flat, construction, nature, sky and vehicle) across all resolutions. However, the object category has a low accuracy (59.7–62.4%) across all input sizes, possibly due to asymmetrical class distribution and tinny shape of traffic sign, traffic light and pole.

4.5 Performance comparison on Cityscapes

4.5.1 Results on validation set

To compare the proposed model with other existing models, this study trained one off-line model (DeepLab) and three real-time segmentation models (SegNet, ContextNet, FAST-SCNN) under same system configuration. Results of comparison on the validation set are displayed in Table 8. Models marked with the sign * are trained during this study where either public code is available or sufficient implementation details are known. For other models, results are extracted from the literature. Due to a large number of parameters and GFLOPs, standard convolution layers of DeepLab and Segnet are replaced by DsConv layers. For DeepLab, instead of using ResNet101 or VGG-16, a pre-trained Xception model is used as a backbone. This study also incorporated ASPP on top of the encoder as a dense feature extractor. Due to the large size of DeepLab, this study could not train the model at full input resolution. It trained Deeplab with 512 $\times$ 1024 px input and achieved 58.2% validation mean IoU. However in [4], the authors claimed 62.97% accuracy using VGG-16 and large field of view. Similarly, this study achieved 56.8% and 54.2% class accuracy for SegNet [1], ContextNet [26] respectively with 512 $\times$ 1024 px input size. It trained FAST-SCNN [27] model with full input resolution for 1000 epochs with fine-tune dataset of Cityscapes. Table 8 clearly depicts that FASt-SCNN [27] produces better results among the existing models listed in the table, although its performance is lagged by 5–6% compared to the proposed model. This study did not achieve the result claimed by [27]. The implementation of FAST-SCNN by this study is consistent with the information provided in the paper. The implementation of all compared models presented here is readily available on the official GitHub repository. To enable a fair comparison based on the models only, this research did not use any post-processing techniques.

Table 8
Performance evaluation of different models on Cityscapes validation set

Type	Model	Input size	Parameters (million)		GFLOPs		Class meanIoU (%)	Category meanIoU (%)	Pixel accuracy (%)
Off-line	PSPNet [46]	713 $\times$ 713	250	.8	516		81.2	–	–
	HANet [6]	768 $\times$ 768	65	.4	2138	.0	80.3	–	–
	OCR [42]	769 $\times$ 769	10	.5	340		81.2	–	–
	DeepLab ${}^{*}$ [4]	512 $\times$ 1024	37	.9	845	.9	58.2	87.7	88.6
Real-time	SegNet ${}^{*}$ [1]	512 $\times$ 1024	29	.5	680	.4	56.8	77.2	87.3
	ContextNet ${}^{*}$ [26]	512 $\times$ 1024	1	.0	9	.3	54.2	78.3	87.3
	FAST-SCNN ${}^{*}$ [27]	1024 $\times$ 2048	1	.2	14	.9	63.3	82.2	89.4
	ENet [19]	1024 $\times$ 2048	0	.4	3	.8	58.3	–	–
	ICNet [45]	1024 $\times$ 2048	6	.68	58	.5	67.7	–	–
	DFANet-B [15]	1024 $\times$ 1024	4	.7	31	.3	58.2	–	–
	FANet ${}^{*}$ [33]	1024 $\times$ 2024	1	.1	11	.4	65.9	83.6	89.6
Real-time	M2FANet ${}^{*}$	1024 $\times$ 2048	1	.3	37	.3	68.5	85.3	90.2

${}^{*}$ Models marked with ${}^{*}$ sign are trained by us and empty cell means that data are not found in literature.

Table 9

Efficiency comparison of all trained models

Model	Input size	Inference time (sec.)	FPS	Model size (MB)
DeepLab	512 $\times$ 1024	0.35	2.9	90.2
SegNet	512 $\times$ 1024	0.76	1.3	225.4
ContextNet	512 $\times$ 1024	0.09	11.1	8.8
FAST-SCNN	512 $\times$ 1024	0.11	9.1	9.4
FANet	512 $\times$ 1024	0.1	10.0	9.4
M2FANet	512 $\times$ 1024	0.11	9.1	11

It is noticeable from Table 8 that the proposed M2FANet generates better segmentation results on Cityscapes validation set among all the listed models. Although, ICNet [45] produces an accuracy (67.7%) very close to M2FANet, it has much higher GFLOPs and model parameters than the proposed one. This makes the proposed M2FANet superior than other real-time scene segmentation alternatives. This study achieves 68.5% mean IoU on the Cityscapes validation set without using any post processing techniques. In [15], the authors claimed around 70% class accuracy on Cityscapes validation set; however, the number of parameters is 6 times higher than M2FANet and thus it is not as efficient as the proposed model. Moreover, after studying [15] architecture, it is observed that this model has a much higher GFLOPs than the proposed model. To have a reasonable comparison, current work compares the proposed model performance with DFANet-B variant which has 4.8 million parameters. Table 8 shows that the proposed model performs much better while having less parameters and GFLOPs. This study also found that the authors of [15] claimed much smaller GFLOPs of their best model. However, the current study suggests that it is not possible to have low GFLOPs count with large number of parameters (7.8 million). It calculated the GFLOPs count and reported in Tables 8–10. In off-line scene segmentation model, PSPNet [46], HANet [6, 42] achieve outstanding results (above 81%) due to its deep neural architecture.

The current work also presents the inference time and the rate in terms of FPS (frame per second) of the models which are trained under the same system configuration. It is often the case in the literature that such comparison of inference time and FPS is based on different hardware, which makes it less insightful as these indicators can vary significantly. It also depends on the size of input image and different hyper-parameters of the model. Therefore, comparing the inference time of different models ran in different hardware platforms with different input resolutions does not provide a clear picture model’s superiority in terms of efficiency. Therefore, to present a balanced comparison, this study measures the inference time of all the trained models under the same system configuration and with similar input size. The results are reported in Table 9. It demonstrates that ContextNet [26] is quite efficient overall among all the trained models. It is understandable as ContextNet has the least number of parameters among all the listed models in Table 9. It can process on an average 11 frames per second where as the proposed model can process 9.1 frames per second. The inference time is measured using a single Nvidia GeForce RTX 2080Ti GPU system with the same input size. Table 9 also presents model size which is the size of the model after completing the whole training process.

Table 10

Performance evaluation of different models on Cityscapes test set

Model	Input size	Class meanIoU (%)	Category meanIoU (%)	Number of parameters (in Million)	GFLOPs
DeepLabV3 $+$ [5]	512 $\times$ 1024	82	91.6	54.8	344.9
DeepLabV2 [4]	512 $\times$ 1024	70.4	86.4	37.9	845.9
PSPNet [46]	713 $\times$ 713	81.2	90.6	65.5	516
BiSeNet [41]	768 $\times$ 1536	74.7	–	5.8	14.8
SegNet extended [1]	360 $\times$ 640	56.1	79.8	29.5	286
ENet [24]	360 $\times$ 640	58.3	80.4	0.4	3.8
ICNet [45]	1024 $\times$ 2048	69.5	–	6.68
FCN 8S [21]	512 $\times$ 1024	65.3	85.7	57	136.2
SQ [37]	1024 $\times$ 2048	59.8	84.3	–	–
CRFasRNN [47]	512 $\times$ 1024	62.5	82.7	–	–
ESPNet [23]	512 $\times$ 1024	60.3	82.2	0.4	–
ContextNet [26]	1024 $\times$ 2048	66.1	82.8	1.0	18.4
DFANet-B [15]	1024 $\times$ 1024	67.1	–	4.9	31.3
FAST-SCNN [27]	1024 $\times$ 2048	63.0	84.7	1.2	14.9
FANet [33]	1024 $\times$ 2048	64.1	83.1	1.1	11.4
M2FANet	1024 $\times$ 2048	68.3	86.6	1.3	37.3

${}^{*}$ Empty cell means that data are not found in literature.

4.5.2 Results on test set

To compare with other existing models, this study exhibits the test set results in Table 10. The results of other models are extracted from their original papers and Cityscapes leader-board. It displays that all off-line and real-time semantic segmentation models except ContextNet [26], FAST-SCNN [27], ENet [24], have at least four times higher number of parameters and GFLOPs than the proposed model. In addition, they all achieved a lower prediction accuracy network than M2FANet did. Table 10 also demonstrates that very few existing real-time models can handle full-resolution input images. Among all listed models, ICNet [45] produced 69.5% class accuracy on full-resolution test images of Cityscapes, whereas the proposed model generates 68.3% accuracy. However, it is distinctly noticeable that ICNet has almost 5 times higher number of parameters than M2FANet. This makes ICNet less favourable for real-time applications. The proposed M2FANet has less parameters and less GFLOPs compared to most models.

Table 10 also presents GFLOPs count of all the models. It is noticeable that instead of counting GFLOPs at full input resolution, most of the models report their GFLOPs count at low input resolution. As GFLOPs count depends on input size, it is expected that the GFLOPs count will be lower in this setting. This study reports 37.3 GFLOPs count of the proposed model with 1024 $\times$ 2048 input resolution, whereas most of the models reported it at low resolution. ENet [24] has the lowest GFLOPs count, however it is reported at 360 $\times$ 640 input resolution.

4.6 Performance comparison on CamVid

Current research also evaluates M2FANet on the CamVid dataset. Table 11 shows the class accuracy and pixel accuracy of M2FANet and other compared models on this dataset. It is clearly evident that M2FANet outperforms the other models: it achieves 60.0% class accuracy despite of having small number of training images. To have a fair and reproducible comparison between the different models based on their core architecture, the present study did not use any data augmentation techniques and any other similar datasets in this experiment. Of course, one would expect an increase in performance should any other techniques are used. For example, some literature achieved such improvement by many extras, one of which was additional 3,000 training images from other datasets outside CamVid. However, these results are difficult to reproduce due to the lack of details in these papers. For completeness, this study quotes the original figure from previous publications for those models which are not trained by the current study. It is expected that the proposed model will definitely achieve even better performance when trained with these extra images from related data with similar annotations.

Table 11
Performance evaluation on validation set of CamVid dataset

Model	Input	Class meanIoU (%)	Pixel accuracy (%)
SegNet	360 $\times$ 480	55.6	–
ENet	360 $\times$ 480	51.3	–
DFANet-B	720 $\times$ 960	59.3	–
FAST-SCNN ${}^{*}$	512 $\times$ 1024	57.5	86.9
FANet ${}^{*}$	512 $\times$ 1024	57.8	87.1
M2FANet ${}^{*}$	640 $\times$ 896	60.0	88.7

${}^{*}$ Empty cell means that results are not found in the literature. Models marked with ${}^{*}$ sign are trained by this study.

Figure 6.

Output by different models using Cityscapes validation image.

Figure 7.

Output by M2FANet using Cityscapes test images.

Figure 8.

Output by FAST-SCNN, FANet and M2FANet using CamVid test image.

4.7 Qualitative performance comparison

4.7.1 Prediction on Cityscapes benchmark

Figure 6 shows segmented output produced by different models on the validation set of Cityscapes benchmark. The first image is the RGB input followed by ground truth, colour code and predicted outputs by different models. It clearly displays that the output image quality generated by M2FANet is much better than other models and this has been already verified by the quantitative results in Table 8. Similar observation can be drawn from Fig. 7. The first row shows RGB test images from Cityscapes, whereas second row displays segmented output by M2FANet. It is observed that the proposed model is capable of identifying and localizing every objects in the scene whilst not overlooking tinny objects such as traffic light, traffic sign, and pole.

4.7.2 Prediction on CamVid benchmark

Figure 8 exhibits the output produced by FAST-SCNN, FANet and the proposed M2FANet on selected images from CamVid validation set. In contrast to Cityscapes, here models are trained with 12 classes (including background class) on this dataset. It is distinctly visible that output by M2FANet is much better than the output generated by FAST-SCNN and FANet. Especially, boundaries of every class are much smoother in the proposed model output compared to others. Though the current study notes that Camvid is much smaller than other scene segmentation datasets and this generally reduces the performance of any models, an observation that has also been established in previous works. Again, one may address this limitation by using extra images from other datasets or using data augmentation techniques, which will benefit all models. However, the focus of the current research is on a fair comparison between the models and thus this direction could be pursued elsewhere.

5. Summary and conclusion

In semantic segmentation, offline models often produce outstanding results. However, due to its large network design, these models are inefficient on mobile devices. To fulfill the real-time computational requirement for resource-constrained embedded devices, over the few years several lightweight scene parsing models have been proposed. Although these models produce competitive results in real-time environment, but still noisy feature, class overlapping, miss-classification, boundary degeneration effects can be seen in the model’s output. Moreover, some existing real-time models producing state-of-the-art results have still large number of parameters and GFLOPs. Therefore, by exploiting new modern techniques such as feature scaling, multi-directional feature fusion, semantic segmentation model architecture can be optimized without hampering model performance and efficiency.

Thus, focusing on our objectives, this study presented a light semantic scene segmentation model, named M2FANet for resource-constrained devices, capable of handling high-resolution input images in the real-time environment. The goal of this research was to develop a model that achieves the best performance among models with low computational requirements, which is defined as having less than 5 million parameters. The optimized backbone network and proposed multi-level multi-path feature aggregation module at decoder efficiently produce a new state-of-the-art results compare to existing models in the same category. Current research also demonstrates the usefulness of feature scaling and multi-directional feature fusion techniques for better object localization and contextual engrossment. The proposed model is evaluated on two public scene segmentation benchmarks. The results show that the proposed model is suitable for urban street scene analysis in real-time mode. In the future, this study plans to extend the proposed model for indoor scene analysis and expand the evaluation on other public benchmark datasets. We believe that results on indoor scene datasets will be consistent with our current results as our proposed model is for semantic segmentation for any complex scene. For reproducing the results presented in this work, this study has made the implementation of the proposed model and selected models on the official Github repository https://github.com/tanmaysingha/M2FANet.

Footnotes

Author’s Bios

	Tanmay Singha is currently a PhD research scholar in Computer Science at Curtin University, Australia. He has double Master’s Degrees-Master of Technology in Information Technology and Master of Computer Applications from University of Calcutta, India. Before joining Curtin University, he served nine years as a lecturer in the department of IT at Royal University of Bhutan, Bhutan. His current research interest focuses on computer vision tasks such as semantic and instance segmentation, object detection, scene graph generation, indoor and outdoor scene analysis, human pose estimation, facial landmark detection and medical image processing using deep neural networks.
	Duc-Son Pham received the PhD degree from Curtin University of Technology in 2005. He is currently a Senior Lecturer with the Discipline of Computing, Curtin University, Perth, Western Australia. His current research interests include sparse learning theory, large-scale data mining, convex optimization, and advanced deep learning with applications to computer vision and image processing. He is a Senior Member of the IEEE. He is a recipient of the Young Author Best Paper Award 2010 for a publication in IEEE Transactions on Signal Processing.
	Aneesh Krishna is currently an Associate Professor with the School of Electrical Engineering, Computing and Mathematical Sciences, Curtin University, Australia. He holds a PhD in computer science from the University of Wollongong, Australia. He was a lecturer in software engineering at the School of Computer Science and Software Engineering, University of Wollongong, Australia (from February 2006–June 2009). His research interests include AI for software engineering, model-driven development/evolution, requirements engineering, agent systems, formal methods, data mining, computer vision, machine learning, bio-informative and renewable energy systems. He has published more than 130 articles in reputed journals and international conferences. His research is (or has been) funded by the Australian Research Council (ARC), and various Australian government agencies (like NSW State Emergency Service) as well as companies such as Woodside Energy, Amristar Solutions, Autism West Incorporated, BW Solar Australia, Western Australia Dementia Training Center and Andrew Corporation. He serves as an assessor (Ozreader)
for the ARC. He has been on the organising committee, served as invited technical program committee member of many conferences and workshops in the areas related to his research.

References

Badrinarayanan

Kendall

and Cipolla

, Segnet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 39(12) (2017), 2481–2495.

Brostow

G.J.

Fauqueur

and Cipolla

, Semantic object classes in video: A high-definition ground truth database, Pattern Recognition Letters 30(2) (2009), 88–97.

Callister

Lazarescu

and Pham

D.-S.

, Robustrepstream: Robust stream clustering using self-controlled connectivity graph, Intelligent Data Analysis 24(4) (2020), 799–830.

Chen

L.-C.

Papandreou

Kokkinos

Murphy

and Yuille

A.L.

, Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 40(4) (2017), 834–848.

Chen

L.-C.

Zhu

Papandreou

Schroff

and Adam

, Encoder-decoder with atrous separable convolution for semantic image segmentation, in: Proceedings of the European Conference on Computer Vision (ECCV), pages 801–818, Munich, Germany, September 2018. Springer International Publishing.

Choi

Kim

J.T.

and Choo

, Cars can’t fly up in the sky: Improving urban-scene segmentation via height-driven attention networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9373–9383, Seattle, WA, USA, June 2020. IEEE.

Chollet

, Xception: Deep learning with depthwise separable convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1251–1258, Honolulu, HI, USA, 2017. IEEE.

Cordts

Omran

Ramos

Rehfeld

Enzweiler

Benenson

Franke

Roth

and Schiele

, The cityscapes dataset for semantic urban scene understanding, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, June 2016. IEEE.

Deng

Dong

Socher

L.-J.

and Fei-Fei

, Imagenet: A large-scale hierarchical image database, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, Miami, Florida, USA, June 2009. IEEE.

10.

Ghiasi

Lin

T.-Y.

and Le

Q.V.

, Nas-fpn: Learning scalable feature pyramid architecture for object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7036–7045, Long Beach, California, USA, June 2019. IEEE.

11.

Zhang

Ren

and Sun

, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, Las Vegas, NV, USA, June 2016. IEEE.

12.

Howard

Sandler

Chu

Chen

L.-C.

Chen

Tan

Wang

Zhu

Pang

Vasudevan

et al., Searching for mobilenetv3, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1314–1324, Seoul, Korea, October 2019. IEEE Xplore.

13.

Howard

A.G.

Zhu

Chen

Kalenichenko

Wang

Weyand

Andreetto

and Adam

, Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv preprint arXiv:1704.04861, 2017.

14.

Lalchhanhima

Saha

Nunsanga

M.V.

and Kandar

, Synthetic aperture radar image segmentation using supervised artificial neural network, Multiagent and Grid Systems 16(4) (2020), 397–408.

15.

Xiong

Fan

and Sun

, Dfanet: Deep feature aggregation for real-time semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9522–9531, Long Beach, California, USA, June 2019. IEEE.

16.

Lin

Milan

Shen

and Reid

, Refinenet: Multi-path refinement networks for high-resolution semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1925–1934, Honolulu, HI, USA, 2017. IEEE.

17.

Lin

T.-Y.

Dollár

Girshick

Hariharan

and Belongie

, Feature pyramid networks for object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2117–2125, Honolulu, HI, USA, 2017. IEEE.

18.

Liu

Qin

Shi

and Jia

, Path aggregation network for instance segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8759–8768, Salt Lake City, UT, USA, June 2018. IEEE.

19.

Liu

Rabinovich

and Berg

A.C.

, Parsenet: Looking wider to see better, arXiv preprint arXiv:1506.04579, 2015.

20.

Liu

Huang

Ming

Gao

and Zhang

, An attention-gated convolutional neural network for sentence classification, Intelligent Data Analysis 23(5) (2019), 1091–1107.

21.

Long

Shelhamer

and Darrell

, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3431–3440, Los Alamitos, CA, USA, June 2015. IEEE.

22.

Mallick

P.K.

Ryu

S.H.

Satapathy

S.K.

Mishra

Nguyen

G.N.

and Tiwari

, Brain mri image classification for cancer detection using deep wavelet autoencoder-based deep neural network, IEEE Access 7 (2019), 46278–46287.

23.

Mehta

Rastegari

Caspi

Shapiro

and Hajishirzi

, Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation, in: Proceedings of the European Conference on Computer Vision (ECCV), pages 552–568, Munich, Germany, September 2018. Springer International Publishing.

24.

Paszke

Chaurasia

Kim

and Culurciello

, Enet: A deep neural network architecture for real-time semantic segmentation, arXiv preprint arXiv:1606.02147, 2016.

25.

Pinheiro

P.O.

Lin

T.-Y.

Collobert

and Dollár

, Learning to refine object segments, in: Proceedings of the European Conference on Computer Vision (ECCV), pages 75–91, Amsterdam, Netherlands, October 2016. Springer.

26.

Poudel

R.P.

Bonde

Liwicki

and Zach

, Contextnet: Exploring context and detail for semantic segmentation in real-time, in: Proceedings of the British Machine Vision Conference (BMVC), Newcastle, UK, 2018. BMVC.

27.

Poudel

R.P.K.

Liwicki

and Cipolla

, Fast-scnn: Fast semantic segmentation network, in: Proceedings of the British Machine Vision Conference (BMVC), Cardiff, UK, September 2019. BMVC.

28.

Ramachandran

Zoph

and Le

Q.V.

, Searching for activation functions, arXiv preprint arXiv:1710.05941, 2017.

29.

Ronneberger

Fischer

and Brox

, U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), pages 234–241, Munich, Germany, October 2015. Springer.

30.

Sandler

Howard

Zhu

Zhmoginov

and Chen

L.-C.

, Mobilenetv2: Inverted residuals and linear bottlenecks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4510–4520, Salt Lake City, UT, USA, June 2018. IEEE.

31.

Sergeev

and Del Balso

, Horovod: fast and easy distributed deep learning in tensorflow, ArXiv, abs/1802.05799, 2018.

32.

Simonyan

and Zisserman

, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556, 2014.

33.

Singha

Pham

D.-S.

and Krishna

, Fanet: Feature aggregation network for semantic segmentation, in: 2020 Digital Image Computing: Techniques and Applications (DICTA), pages 1–8, Melbourne, Australia, November 2020. IEEE.

34.

Tan

and Le

, Efficientnet: Rethinking model scaling for convolutional neural networks, in: International Conference on Machine Learning (ICML), pages 6105–6114, Long Beach, California, USA, June 2019. PMLR.

35.

Tan

Pang

and Le

Q.V.

, Efficientdet: Scalable and efficient object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10778–10787, June 2020.

36.

Targ

Almeida

and Lyman

, Resnet in resnet: Generalizing residual architectures, ArXiv, abs/1603.08029, 2016.

37.

Treml

Arjona-Medina

Unterthiner

Durgesh

Friedmann

Schuberth

Mayr

Heusel

Hofmarcher

Widrich

et al., Speeding up semantic segmentation for autonomous driving, in: Proceedings of the MLITS, NIPS Workshop, Vol. 2, Barcelona, Spain, 2016.

38.

Shen

and Hengel

A.v.d.

, Real-time semantic image segmentation via spatial sparsity, ArXiv, abs/1712.00213, 2017.

39.

Shen

and Van Den Hengel

, Wider or deeper: Revisiting the resnet model for visual recognition, Pattern Recognition 90 (2019), 119–133.

40.

Yang

T.-J.

Howard

Chen

Zhang

Sandler

Sze

and Adam

, Netadapt: Platform-aware neural network adaptation for mobile applications, in: Proceedings of the European Conference on Computer Vision (ECCV), pages 285–300, Munich, Germany, September 2018. Springer International Publishing.

41.

Wang

Peng

Gao

and Sang

, Bisenet: Bilateral segmentation network for real-time semantic segmentation, in: Proceedings of the European Conference on Computer Vision (ECCV), pages 325–341, Munich, Germany, September 2018. Springer International Publishing.

42.

Yuan

Chen

and Wang

, Object-contextual representations for semantic segmentation, in: Proceedings of the European Conference on Computer Vision (ECCV), pages 173–190, Glasgow, UK, August 2020. Springer.

43.

Zhang

Wang

and Xie

, Co-occurrent features in semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 548–557, Long Beach, CA, USA, June 2019. Computer Vision Foundation/IEEE.

44.

Zhang

Change Loy

and Lin

, Polynet: A pursuit of structural diversity in very deep networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 718–726, Honolulu, HI, USA, June 2017. IEEE.

45.

Zhao

Shen

Shi

and Jia

, Icnet for real-time semantic segmentation on high-resolution images, in: Proceedings of the European Conference on Computer Vision (ECCV), pages 405–420, Munich, Germany, September 2018. Springer International Publishing.

46.

Zhao

Shi

Wang

and Jia

, Pyramid scene parsing network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2881–2890, Honolulu, HI, USA, 2017. IEEE.

47.

Zheng

Jayasumana

Romera-Paredes

Vineet

Huang

and Torr

P.H.

, Conditional random fields as recurrent neural networks, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1529–1537, Santiago, Chile, December 2015. IEEE.

48.

Zhang

Ren

and Sun

, Spatial pyramid pooling in deep convolutional networks for visual recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 37(9) (2015), 1904–1916.

49.

Shrivastava

Gupta

and Girshick

, Training region-based object detectors with online hard example mining, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 761–769, Las Vegas, NV, USA, June 2016. IEEE.

50.

Szegedy

Liu

Jia

Sermanet

Reed

Anguelov

Erhan

Vanhoucke

and Rabinovich

, Going deeper with convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–9, Boston, MA, USA, June 2015. IEEE.

51.

Chen

Wang

Xian

Chen

Liu

Madhavan

and Darrell

, Bdd100k: A diverse driving dataset for heterogeneous multitask learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2636–2645, Seattle, Washington, USA, June 2020. IEEE.

Urban street scene analysis using lightweight multi-level multi-path feature aggregation network

Abstract

Keywords

1. Introduction

2.1 Off-line segmentation

2.2 Real-time segmentation

3.1 Network architecture

Table 2 Bottleneck residual block

3.2.1 Depth-wise separable convolution

3.2.2 Nonlinearities

4.1 Datasets

4.2 Implementation details

Table 4 Results of ablation study on Cityscapes validation set

Table 5 Model performance on Cityscapes fine-tune and coarse datasets

4.5.1 Results on validation set

Table 8 Performance evaluation of different models on Cityscapes validation set

4.6 Performance comparison on CamVid

Table 11 Performance evaluation on validation set of CamVid dataset

4.7.1 Prediction on Cityscapes benchmark

4.7.2 Prediction on CamVid benchmark

5. Summary and conclusion

Footnotes

Author’s Bios

References

Table 2
Bottleneck residual block

Table 4
Results of ablation study on Cityscapes validation set

Table 5
Model performance on Cityscapes fine-tune and coarse datasets

Table 8
Performance evaluation of different models on Cityscapes validation set

Table 11
Performance evaluation on validation set of CamVid dataset