TextureMask: A merged architecture for low-resolution instance segmentation

Abstract

Instance segmentation has a wide range of applications, including video surveillance, autonomous driving, and behavior analysis. Nevertheless, as a type of pixel-level segmentation, its prediction performance in practice is substantially affected by low-resolution (LR) images resulting from the limitations of image acquisition equipment and poor acquisition conditions. Moreover, because their immense computational costs prevent the implementation of existing segmentation models on embedded devices, the development of a lightweight segmentation model has become an urgent necessity. However, it is challenging to achieve sound results with high efficiency and portability. From another perspective, to improve understanding of detailed objects, an architecture is needed that promotes an advanced interpretation of the segmentation, that is, a refined mask with texture. Our main contribution, called TextureMask, consists of the MobileNet-FPN for Mask R-CNN methods, segmentation with cropping, and a gradient sensitivity map, which are then merged into a unified map to refine and enrich the mask with texture information. Furthermore, preprocessing and post-processing algorithms are incorporated. Experiments demonstrated that our technique exhibits good pixel-level segmentation performance in terms of both accuracy and computational efficiency for a given LR input, and it can be easily implemented in embedded platforms.

Keywords

Mask R-CNN MobileNet light weight sensitivity map instance segmentation

1. Introduction

Video images represent one of the most common data types in modern life, and instance segmentation is a key research area in the field of human-computer interaction. Nevertheless, various real-world factors (such as poor image acquisition equipment, unfavorable acquisition conditions, and lens shake) impair the quality of the collected images. Therefore, the obtained data are mostly low-resolution (LR) images. These obstacles have restricted further research on instance segmentation considerably, including that for applications such as video surveillance, autonomous driving, and behavior analysis. However, few studies have focused on segmentation for LR data so far. Furthermore, LR segmentation algorithms must be lightweight for practical front-end applications. These necessities make LR segmentation a serious challenge.

In general, previous works focused on the use of normal images that are usually of high quality. In contrast, this paper proposes models that are compatible with (LR) input, a common characteristic in many real-world image collecting devices. In addition, instance segmentation, compared to detection and recognition, is considered to be a difficult and complicated task for pixel-level segmentation, because of the large number of parameters involved and the considerable computing power needed.

Recently, convolutional neural networks (CNNs) [1] have been employed for a variety of object detection tasks and have achieved ideal performance. Although some researchers have attempted to increase system accuracy by adding layers and complicating the network, the resulting elaborate model structure usually requires substantial memory and computing power for storage and processing. Such operational prerequisites have considerably constrained their popularization and application. Therefore, few of the aforementioned algorithms are readily applied to a range of real-world platforms such as mobile or embedded devices. Furthermore, few studies have addressed this challenge so far. In this light, the aim of this study was not only enhanced accuracy, but also lightweight algorithms for practical front-end applications.

Technically, the existing typical segmentation models, such as Mask R-CNN [2], U-Net [3], PSPNet [4], and other networks, do not perform well on LR images, and the predicted mask is often regarded as incomplete and of low quality. For practical applications such as behavior analysis and object tracking, we consider not only the single mask, but even further detailed and advanced information analysis. Therefore, it is not ideal to directly use a single mask for practical applications and further research. In view of this, we aimed to establish a new approach to propose a novel type of high-quality mask with fused texture information in order to better understand and analyze images.

Several papers on instance segmentation [5, 6, 7, 8, 9, 10, 11, 12, 13, 14] have proposed methods that embrace either instance or semantic segmentation; our algorithm, in essence, merges and keeps the merits of both ideas. To overcome the limitations of existing approaches, a novel mask generation framework is proposed, called TextureMask. More specifically, the operation of the mask combination refines and enriches the pixel-level detailed mask with texture information. Our study addressed the problems of difficulty in building a lightweight instance segmentation model for LR and the pursuit of obtaining merged masks with texture-level details. Accordingly, we constructed a compound model that consists of MobileNet-FPN for Mask R-CNN, segmentation with cropping, and a gradient sensitivity map. All these pieces are then integrated into a unified model to refine and enrich the mask with the inclusion of texture information. Furthermore, for LR, preprocessing is employed, and a motion vector is proposed as post-processing for the challenge of imputing incomplete mask prediction.

2. Related work

To address the lightweight application concern, two streams of ideas have been exploited. The first is to compress and simplify the trained CNN model to remove redundant parameters and increase the processing speed. The techniques for optimization involve pruning [15, 16, 17], quantization [18, 19, 20], and Huffman coding [21]. In contrast, the second idea is to replace the computation of a convolutional network by lightweight frameworks such as MobileNet V1 [22], MobileNet V2 [23], ShuffleNet [24, 25], and EfficientNet [26]. The goal is also to reduce the network parameter set while maintaining its performance. Currently, MobileNet V1 [22] and EfficientNet [26] have been widely adopted in portable devices because of their easier implementation without sacrificing accuracy. From another perspective, the CNN also motivates a cohort of novel and influential detection methods including R-CNN [27], Fast R-CNN [28], Faster R-CNN [29], and Mask R-CNN [2]. R-CNN (regions with CNN features) is a classical algorithm for object detection. Fast R-CNN improves R-CNN by incorporating SPPNet [30]; this combination largely solves the issue of high memory and processing requirements. Faster R-CNN is built upon Fast R-CNN and RPN [29], through which it generates a regional suggestion box. Subsequently, Mask R-CNN is improved by adding a branch of segmentation on the basis of Faster R-CNN.

The Mask R-CNN method is regarded as state-of-art instance segmentation, owing to its high precision and versatility. Moreover, by subjoining disparate branches, it meets various demands such as instance segmentation and human pose recognition. Nonetheless, the immense computation and memory resource requirements are still a hindrance for Mask R-CNN.

U-net is a small-footprint, state-of-the-art network for semantic segmentation; it is based on full convolutional network expansion and modification. The network consists of two parts: a contracting path to gather contextual information, and a symmetrical expanding path for precise positioning. U-Net has the merits of light weight and high semantic segmentation performance for large targets.

In addition to these solutions, a wide range of related studies has been published on segmentation in video sequences. Ventura [31] introduced an end-to-end recurrent network for video segmentation. Zhu Yi [32] presented a segmentation model that improves semantic segmentation by video propagation and label relaxation. Other algorithms have also exhibited superior performance [33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43]. Although there is a body of high-performing algorithms for common video datasets, few mature LR segmentation frameworks with easy portability and high efficiency have been designed.

3. Proposed approach

3.1 Overall pipeline

To address the aforementioned challenges, we integrated, sequentially, MobileNet-FPN for Mask R-CNN, semantic segmentation with cropping, and a gradient sensitivity map. The first two components, Mask R-CNN and U-net, are widely known, respectively, as state-of-art instance and semantic segmentation models, and they do not interfere with each other. Therefore, the merits of each method are preserved: the former exhibits a high detection rate, and the latter has excellent semantic segmentation. In addition, a sensitive map alone that provides the texture characteristics of the target is also regarded as a well-recognized and popular approach for attaching texture information to mask prediction. Our combined model, therefore, takes advantage of the component at each step.

To be more specific, we aimed to incorporate the following criteria as the core TextureMask architecture, as shown in Fig. 1:

1.
MobileNet-FPN for Mask R-CNN
2.
Segmentation with cropping
3.
Gradient sensitivity map

Beyond the power of these three elements, we also incorporated a pre-processing step, namely, Super-resolution (SR) reconstruction (to address low-resolution data), and a post-processing step, called SR mask prediction and refinement (using temporal correlation among consecutive video frames) to further enhance the reliability of mask prediction.

MobileNet-FPN for Mask R-CNN: In order to retain the excellent detection and segmentation of Mask R-CNN and to address its large number of computational problems, we modeled the Mask R-CNN on the basis of MobileNet as the first stage for defining the initial mask. In essence, the experiment proved that Mask R-CNN using ResNet-FPN as the feature extraction backbone exhibits high accuracy. Because MobileNet is a lightweight structure with excellent compatibility on portable devices, we combined MobileNet V1 with FPN (called MobileNet-FPN) for feature extraction.

Segmentation with cropping: The goal of U-Net is to refine the initial mask after cropping, called the second stage, because U-Net not only is a small model but also has the merits of superior semantic segmentation for large objects. Furthermore, researches have shown that U-net is more suitable for large target segmentation than for small target segmentation. Because U-Net is suitable for cutting specific parts, the network was selected as an optimal segmentation method. The second stage is mainly used to compensate the initial mask on the basis of the detection and initial segmentation in the first stage, and to further improve the quality and integrity of the mask by combining the first stage and the second stage.

Gradient sensitivity maps: The last stream, gradient sensitivity maps, is employed to reflect texture details. Henceforth, the operations are fused for mask combination to refine and enrich the mask with texture. We further aimed to pursue texture-level information, that is, a textured mask. Gradient sensitivity maps are regarded as an excellent technique that can reflect the sensitivity characteristics of objects. Technically speaking, it can track the areas in the image that have the greatest impact on the final classification result to understand what the neural network has learned.

Figure 1.
Overall pipeline.

3.2 SR reconstruction as preprocessing

Because of the lack of high-quality physical equipment, lens shaking, and other issues, most acquired images have poor quality or even mosaic characteristics. Moreover, the entire architecture is faced with the challenge of LR data, which directly affects the segmentation results. These conditions motivated us to rebuild a high-resolution (HR) dataset by utilizing SR reconstruction on the initial data.

We adopt deep networks for image super-resolution with sparse prior (SCN) [44] as the preprocessing for the reconstruction. SCN takes advantage of sparse prior and deep learning, and incorporates three independent optimized modules of sparse representation, mapping, and sparse reconstruction into a sparse network. Technically, training with SCN is equivalent to co-optimizing the three modules in order to gain the global optimal solution, and the reconstructed image has acceptable visual quality. In fact, the sparse coding network makes full use of the prior information of images. First, the method acquires the sparse prior information from images through the feature extraction layer. Thereafter, it establishes a feed-forward SCN for sparse encoding and decoding of data through the learned iterative shrinkage and threshold algorithm (LISTA) [45]. Finally, the approach utilizes a cascade network to reconstruct the images. In terms of network structure, SCN retains the image block extraction, representation, and reconstruction layer in a super resolution using convolution neural network (SRCNN) [46], and it includes the LISTA network in the middle layer.

As shown in Fig. 2, the input image $I_{y}$ first enters a convolution layer $H$ that produces features $y$ for each LR patch. Delivering each $y$ to the LISTA network, the algorithm goes through two linear layers, obtains the sparse code $\alpha$ , and multiplies $\alpha$ by HR dictionary $D x$ in another linear layer to generate the HR patch $x$ . After all patches are reconstructed from LR to HR, the final layer $G$ assembles them to their corresponding positions in the output HR image $I_{x}$ . In the training process, mean square error is adopted as the cost function. After the HR is reconstructed, the outcomes are set to the subsequent fusion network for further segmentation. The optimization function can be formulated as:

$\displaystyle\min\sum_{i}\left\|\textit{SCN}(I_{y}^{(i)};\Theta)-I_{x}^{(i)}% \right\|_{2}^{2}$ (1)

where $I_{y}^{(i)}$ and $I_{x}^{(i)}$ are the LR/HR training data, and $\textit{SCN}(I_{y}^{(i)};\Theta)$ generates the HR image with parameter set $\Theta$ .

Figure 2.

SCN pipeline.

3.3 MobileNet-FPN for Mask R-CNN

In the first stage, Mask R-CNN is a cutting-edge model for segmentation. Nonetheless, MobileNet V1 [22] is employed as the backbone to reduce the amount of calculation, because ResNet-based approaches require excessive computing resources. Specifically, MobileNet V1 is a type of lightweight network that sets up an efficient factorization to shrink the model size and meet the hardware limitations on small devices. Its core unit is a depthwise separable convolution that integrates the filtering step (by 3 $\times$ 3 depthwise convolutions) and the combining step (by 1 $\times$ 1 pointwise convolution). Because 95% of the calculation and 75% of the parameters in MobileNet algorithms are related to those 1 $\times$ 1 convolutions, a set of highly optimized math libraries can be applied to the computation, making it notably efficient in contrast to the traditional approaches.

The computational details are illustrated in Fig. 3. Suppose there is an input data of size $D_{F}\times D_{F}\times M$ , where $D_{F}\times D_{F}$ can be the image size and $M$ is the number of channels (3 for an RGB image). There are $N$ kernels of size $D_{K}\times D_{K}\times M$ . Then, we obtain an output of size $D_{P}$ with $N$ channels. In this setting, the standard convolutional kernel $K$ is of size $D_{K}\times D_{K}\times M\times N$ . As a result, the computational cost for standard convolutions is $D_{K}\times D_{K}\times M\times N\times D_{F}\times D_{F}$ . In contrast, the calculation of depthwise convolution is defined as $D_{P}^{2}\times D_{K}^{2}\times M+D_{P}^{2}\times M\times N$ . In addition, the traditional convolution calculation is $D_{P}^{2}\times D_{K}^{2}\times M\times N$ . Thereafter, the calculation of depthwise convolution/traditional convolution can be written as $\frac{D_{P}^{2}\times D_{K}^{2}\times M+D_{P}^{2}\times M\times N}{D_{P}^{2}% \times D_{K}^{2}\times M\times N}=\frac{1}{N}+\frac{1}{D_{K}^{2}}$ .

This evidence proves that the depthwise convolution can considerably reduce the amount of calculation. With regard to the structure, MobileNet can be divided into the “Depthwise Conv Block” and “Conv Block,” as shown in Fig. 4. In short, MobileNet is designed with five stages, and the output for each stage is 32, 64, 128, 256, 512, and 1024, respectively. The details of the MobileNet architecture are listed in Table 1.

Figure 3.

Details of convolution.

Figure 4.

MobileNet architecture.

Table 1

Details of MobileNet architecture

Type	Kernel	M	N	Operator	Stage
conv1	(3,3)	3	32	Conv	Stage 1
conv_dw_1	(3,3)	32	1	DW_Conv _Block
conv_pw_1	(1,1)	32	64
conv_dw_2	(3,3)	64	1	DW_Conv _Block	Stage 2
conv_pw_2	(1,1)	64	128
conv_dw_3	(3,3)	128	1	DW_Conv _Block
conv_pw_3	(1,1)	128	128
conv_dw_4	(3,3)	128	1	DW_Conv _Block	Stage 3
conv_pw_4	(1,1)	128	256
conv_dw_5	(3,3)	256	1	DW_Conv _Block
conv_pw_5	(1,1)	256	256
conv_dw_6	(3,3)	256	1	DW_Conv _Block	Stage 4
conv_pw_6	(1,1)	256	512
conv_dw_7	(3,3)	512	1	DW_Conv_Block
conv_pw_7	(1,1)	512	512
conv_dw_8	(3,3)	512	1	DW_Conv _Block
conv_pw_8	(1,1)	512	512
conv_dw_9	(3,3)	512	1	DW_Conv _Block
conv_pw_9	(1,1)	512	512
conv_dw_10	(3,3)	512	1	DW_Conv _Block
conv_pw_10	(1,1)	512	512
conv_dw_11	(3,3)	512	1	DW_Conv_Block
conv_pw_11	(1,1)	512	512
conv_dw_12	(3,3)	512	1	DW_Conv_Block	Stage 5
conv_pw_12	(1,1)	512	1024
conv_dw_13	(3,3)	1024	1	DW_Conv _Block
conv_pw_13	(1,1)	1024	1024

3.3.1 Pre-classifiers

In practical applications, when implementing ResNet-based and MobileNet-based algorithms (the two candidate networks in this study), experiments showed that different images usually exhibit considerably different segmentation outcomes in the two candidates. In the experiment, there were three outcomes that ResNet-FPN and MobileNet-FPN may exhibit in practice. These included: (i) the former is superior in segmentation results, (ii) the latter has better results, and (iii) both networks deliver almost indistinguishable results. A natural problem is to know how to select one of the two networks. In this section, we present a pre-classifier branch designed to select the more favorable segmentation network.

The feature extraction process is as follows. Images may possess a rich set of information. To better extract the details (such as the texture and the edge gradient features) of the images, in our application, we extract the significant features from two perspectives: the gray level co-occurrence matrix (GLCM) [47] and the histogram of the oriented gradient (HOG) [48]. The GLCM is a texture analysis method, exploring the spatial distribution relationship between pixels. In our algorithm, GLCM considers the following four features:

Contrast: The value represents the gray level pixel difference to gauge how pixel pairs look, contrasting to each other. It is proportional to the deviation of the off-diagonal from diagonal elements, expressed as $\sum_{i,j}|i-j|^{2}P(i,j)$ . Correlation: The cross-correlation between a pixel and the rest of the entire image as demonstrated by $\sum_{i,j}\frac{(i-\mu i)(j-\mu j)P(i,j)}{\sigma_{i}\sigma_{j}}$ . Energy: The sum of squares of all elements in GLCM. This is expressed as $\sum_{i,j}P(i,j)^{2}$ . Homogeneity: The value reflects the tightness of the distribution of elements in the GLCM relative to its diagonal entries. $\sum_{i,j}\frac{P(i,j)}{1+|i-j|}$ .

Figure 5.

MobileNet-FPN for Mask R-CNN.

Figure 6.

The training procedure.

In total, the features consist of these four properties with four angles (0 ${}^{\circ}$ , 45 ${}^{\circ}$ , 90 ${}^{\circ}$ , 135 ${}^{\circ}$ ) and their mean and standard deviation.

$\displaystyle\textit{Feature}(\textit{GLCM})=[\textit{mean}(\textit{GLCM}),% \textit{std}(\textit{GLCM})]$ (2)

The HOG characterizes the local gradient direction and the gradient intensity distribution of an image. To construct the HOG, we first partition the image into small and connected regions. For each region, the algorithm collects histograms of the gradient or edge direction of each pixel inside, and then merges these histograms to generate the HOG. We combine the texture features of GLCM and the gradient features of HOG as the final features (Eq. (3)). Thence, we can effectively capture the entire feature of the image when these two complementary aspects are employed in combination.

$\displaystyle\textit{Feature}(\textit{all})=[\textit{Feature}(\textit{HOG}),% \textit{Feature}(\textit{GLCM})]$ (3)

We chose four high-performing classifiers as pre-classifier candidates in order to measure performance, and then selected the best of them in the proposed framework: naive Bayes (NB) [49], random forest (RF) [50], adaptive boosting (Adaboost) [51], and support vector machine (SVM) [52]). We designated two labels, 0 and 1, indicating that the image could be better employed by MobileNet-FPN or ResNet-FPN, respectively. In particular, when MobileNet and ResNet behave similarly, we label the target as 0 because MobileNet usually requires less time and resources. We then split the dataset into training and testing sets and apply the four classifiers to the dataset as follows.

The entire role of stage one, namely, MobileNet-FPN for Mask R-CNN with pre-classifiers, is presented in Fig. 5. In detail, the pre-classifiers output 0 or 1, indicating which candidates to set for a given input image on the subsequent segmentation network. The initial segmentation results of stage one are called Mask 1. The training procedure is shown in Fig. 6.

3.4 Segmentation with cropping

At this stage, U-Net exhibits considerable virtues, including light weight, fast training speed, suitability for large-object segmentation, easy implementation, and high robustness. In turn, EfficientNet is a novel type of lightweight model with high efficiency; its modeling process balances all dimensions of depth, width, and resolution in the network to improve accuracy and efficiency. Considering the advantages of portability and easy transplantation, EfficientNet was adopted as the encoder feature extractor and decoder of U-Net.

3.5 Gradient sensitivity maps

The gradient sensitivity map in the third stage is employed in extracting the texture feature of each object. The technical details are as follows. Given the input image $x$ , the network calculates a class activation function $A_{c}$ for each class.

$\displaystyle\textit{Class}(x)=\textit{argmax}A_{c}(x)$ (4)

For any image $x$ , we can construct a vanilla sensitivity map, $S_{c}(x)$ . In particular, we can define

$\displaystyle S_{c}(x)=\frac{\partial A_{c}(x)}{\partial x}$ (5)

Here $\partial A_{c}$ represents the gradient of $A_{c}$ . The initial sensitivity map of a label shows the relevance of the area of that label.

Smoothed gradient [53]: The noise seen in the sensitivity map can be due to a substantially meaningless local change in the partial derivative. In fact, the networks currently discussed as $A_{c}$ are often not even continuously distinguishable. An improved method of sensitivity mapping, called smoothed gradient, can smooth the mapping on the basis of Gaussian check $\partial A_{c}$ instead of visualizing it directly resting on the gradient.

Figure 7.

Gradient sensitivity maps. (a) Data. (b) Vanilla gradient. (c) Smoothed vanilla gradient. (d) Guided backProp. (e) Smoothed guided backProp.

We can calculate a simple random approximation. Random samples are taken near the input $x$ and the resulting sensitivity maps are averaged.

$\displaystyle\hat{S_{c}(x)}=\frac{1}{n}\sum_{1}^{n}S_{c}(x+N(0,\sigma^{2}))$ (6)

Where $n$ is the number of samples, and $N(0,\sigma^{2})$ represents Gaussian noise with standard deviation $\sigma$ .

Guided backpropagation [54]: Because guided back-propagation includes an additional pilot signal from the upper layer to the usual back-propagation, the backflow of negative gradients is prevented, which corresponds to neurons that reduce activation. Here, higher-level information must be visualized. For further description, deconvolution is equivalent to reverse transmission through the network. The two methods, back-propagation and deconvolution, are combined to better visualize advanced functions.

The activation equation is defined as:

$\displaystyle f_{i}^{l+1}=\textit{relu}(f_{i}^{l})=\textit{max}(f_{i}^{l},0)$ (7)

The backpropagation:

$\displaystyle R_{i}^{l}=(f_{i}^{l})>0)\cdot R_{i}^{l+1}=\textit{max}(f_{i}^{l}% ,0),\text{where }R_{i}^{l+1}=\frac{\partial f^{\textit{out}}}{\partial f_{i}^{% l+1}}$ (8)

The guided backpropagation is shown:

$\displaystyle R_{i}^{l}=(f_{i}^{l})>0)\cdot(R_{i}^{l+1}>0)\cdot R_{i}^{l+1}$ (9)

Figure 8.

Merging and post-processing.

Figure 9.

Results of the mask sequences. (a)–(b) list two sets of the results with and without post-processing.

Finally, the smoothed gradient and guided back-propagation are incorporated together as smoothed guided back-propagation and called the gradient sensitivity map. It is applied to extract the texture features in stage three. As illustrated in Fig. 7, the comparison of the four methods reveals that the smoothed guided back-propagation algorithm can realize higher performance for the texture information.

3.6 Merging

The sub-results from each of the three branches are then integrated into a unified result to refine the mask with the texture information. Technically, for each part, we first perform an erosion and then dilation algorithm to remove noise. Then, the final result can be incorporated by multiplying and adding among the outcomes. Specifically, as illustrated in Fig. 8, the first stage can predict the basic mask, but ignores detailed information about certain locations, such as wheels, hands, and feet. For the second subdivision, more attention can be absorbed and supplemented. Furthermore, technically, the sensitivity map can be perfectly extracted to describe the texture features. As shown in Fig. 8, the merged information from three different levels can further enrich the advanced and texture-level masks.

3.7 Post-processing

An additional problem to be addressed. To solve the challenge of undetected and incomplete objects, we further pattern a post-processing algorithm called SR mask prediction and refinement, as shown in Fig. 9, after merging the branches. Fundamentally, considering the time-series nature of the frame-by-frame images of the video, one can predict subsequent images by relying on the strong correlation within (i.e., intra-frame) and among images (i.e., inter-frame) defined as the spatio-temporal features in disparate frames [55, 56, 57]. In particular, we apply a motion vector [58, 59] to predict and purify these incomplete and missing masks to build up the succeeding mask sequences. Moreover, for each object, we produce the initial mask sequence by finding the center of gravity of each mask and sorting the masks for the adjacent frames to determine the minimum distance between the centers of gravity. Thus, prediction is performed in view of the sequence. Our matching algorithm locates the pixel block inside $(m+2\textit{dxmax})(n+2\textit{dymax})$ that has the minimum sum of absolute difference (MAD) (Eq. (10)). dxmax, dymax are the maximum displacement that can be searched in the horizontal and vertical directions of the reference block.

$\displaystyle\textit{MAD}(x,y)=\frac{1}{MN}\sum_{m=1}^{M}\sum_{n=1}^{N}\left|f% _{k}(m,n)-f_{k-1}(m+x,n+y)\right|$ (10)

Thanks to its power to easily find the globally best-matched block, the block-based full search motion estimation method [60] is employed in this case. Similarly, we gain the final mask sequence from the prediction and refinement of the neighboring frames. The results, including those with and without post-processing, are shown in Fig. 9. The two panels present two examples of how incomplete (the dotted blue rectangle) or undetected (the yellow block) mask prediction could be improved or imputed by the post-processing stage of our model. The first and second row of each panel illustrate the model output without and with post-processing, respectively, and images from left to right on each row are consecutive frames with time. In detail, in the third column of the second row, the human body mask was completed from its primal version (i.e., the blue rectangle) according to the predicted masks in the preceding and subsequent frames. In addition, the two masks in the third and last column on the second row of the bottom panel were also imputed on the basis of predicted masks in the surrounding frames.

4. Experiments

4.1 Dataset

We used the three datasets, UCF Sport Action, Street_View_MB, and Visdrone2018, for the experiments because they exhibit a diverse collection of segmentation examples with LR and varied scenes, such as street views, vehicles, pedestrians, and sports action. These datasets are LR because of lens shake, hardware conditions, and other situations (see Fig. 10 for sample images). The datasets were obtained from BBC, ESPN’s TV news, drone video, and other sources. We randomly partitioned each category into training, verification, and test sets with a fixed ratio of $5:3:2$ . Data augmentation was employed to expand the datasets, such as flipping, cropping, and other methods. The dataset details are listed in Table 2.

Table 2
Details of datasets

Name	Number of frames
UCF Sport	5200
Street_View_MB	5500
Visdrone2018	2500

Figure 10.

Examples of dataset.

Table 3

Experimental comparison of the SR reconstruction

	IQA	NEA	BIL	BIC	SCN
	PSNR	28.0394	28.0556	29.9956	30.2199
Data_01	SSIM	0.8200	0.8397	0.8416	0.8549
	FSIM	0.8830	0.8862	0.8910	0.8963
	PSNR	23.4615	23.9908	24.1967	24.8281
Data_02	SSIM	0.6266	0.6332	0.6516	0.6913
	FSIM	0.8401	0.8403	0.8424	0.8464
	PSNR	26.0116	26.9237	26.9594	27.8923
Data_03	SSIM	0.7516	0.7774	0.7799	0.8068
	FSIM	0.8400	0.8403	0.8462	0.8519

Figure 11.

Examples of SR reconstruction.

In our experiments, we deployed the NVIDIA Jetson TX2 as mobile devices because of its 256-core NVIDIA PascalTM architecture and its energy efficiency in users’ embedded AI computing devices.

4.2 Experimental analysis

4.2.1 Preprocessing

For preprocessing, we further demonstrated the power of SCN reconstruction using three rules: peak signal-to-noise ratio (PSNR) [61], structural similarity index (SSIM) [62], and FeatureSIM [63]. PSNR is most commonly used for measuring the quality of lossy compression code reconstruction, and it is efficient for image quality assessment (IQA). A higher PSNR generally indicates higher-quality reconstruction. Similarly, for the other two IQA measures (SSIM and FeatureSIM), a higher value indicates better image quality.

As shown in Table 3, SCN reconstruction is superior to that of three traditional methods, namely, nearest interpolation (NEA), bilinear interpolation (BIL), and bicubic interpolation (BIC), in terms of the three criteria. From Table 3, we observe that SCN outperforms the others under the three criteria. It can be demonstrated that images generated by the SCN have higher quality compared to the other models; this comparison is shown in Fig. 11. As seen in Fig. 12, compared to LR, more objects are detected and more complete masks are achieved in the SCN-reconstructed HR image segmentation. The dotted blue rectangles are marked as incomplete objects, the yellow blocks are called undetected masks, and the red ones represent better HR performance results.

4.2.2 Pre-classifiers

We further randomly partitioned each category into training data and test data with a fixed ratio of 8:2. We used three indicators – precision (Eq. (11)), recall (Eq. (12)), and F 1-score [64] – to measure the performance of the classifiers. Defined for each label, precision evaluates the proportion of true positives relative to false positives, and recall gauges the number of true positives compared to the false negatives. As shown in Table 4, because SVM achieves the best scores under all three criteria (precision, recall, and F1 score) among all candidate classifiers and it also enjoys the numerous advantages discussed in Section 3.3.1, we eventually chose SVM as the pre-classifier method.

$\displaystyle\textit{Precision}=\frac{\textit{True Positive}}{\textit{True % Positive}+\textit{False Positive}}$ (11) $\displaystyle\textit{Recall}=\frac{\textit{True Positive}}{\textit{True % Positive}+\textit{False Negative}}$ (12) $\displaystyle F_{1}=2\times\frac{\textit{precision}\times\textit{recall}}{% \textit{precision}+\textit{recall}}$ (13)

Table 4

Results of pre-classifiers

Classifiers	Precision	Recall	F1 score
RF	0.915	0.914	0.914
NB	0.858	0.853	0.855
Adaboost	0.932	0.933	0.932
SVM	0.956	0.956	0.956

Figure 12.

Examples of segmentation in LR and HR.

Table 5

Experimental comparison for two candidate networks

GPU	Backbone	Kernels	FPS	Throughput	Model size
NVIDIA 1080 Ti	ResNet-FPN	125	0.92	65.2	260M
NVIDIA 1080 Ti	MobileNet-FPN (Ours)	125	0.92	65.2	260M
NVIDIA Tx2	MobileNet-FPN (Ours)	48	0.65	92.3	98M

Table 6

Experimental comparison of mAP and mIoU

Model	Backbone	mAP (%)	mIoU (%)
Mask R-CNN	ResNet-FPN	69.4	60.2
Mask R-CNN (Ours)	MobileNet-FPN	71.9	60.3
SR as preprocessing
$+$ Mask R-CNN	MobileNet-FPN
$+$ U-Net	EfficientNet	72.6	63.1
$+$ Sensitivity map	VGG
with post-processing (Ours)

Figure 13.

Details of mAP results.

Figure 14.

Examples of comparison results.

4.2.3 Overall

In our experiment, we set the learning rate to 0.0001 and the number of training steps to 1000. For the first stage, the comparison of the two candidate networks is listed in Table 5. It can be seen that MobileNet-FPN exhibits a multitude of merits compared with the ResNet-based model. Experimental results listed in the table indicate that the MobileNet and ResNet results are close to each other. First, running the pre-classifier takes little time and computing resources compared to the main segmentation, so it can be ignored. In addition, the MobileNet-based approach is small; it contains only 48 kernels, whereas the uncompressed model has 125 kernels. Next, our algorithm also improves throughput and running time. Ultimately, the method consumes only 98 MB of memory after compression, whereas the original takes 260 MB. The results imply that the MobileNet-based approach can be easily transplanted to mobile devices, owing to its remarkably low computation and memory cost. The details of statistical results are shown in Fig. 13.

A comparison of the Mask R-CNN, U-Net, refined mask, and refined mask with texture approaches is illustrated in Fig. 14. As demonstrated in Fig. 14, U-net exhibits excellent refinement of the detailed target. Furthermore, the texture details in each object can be perfectly extracted by gradient sensitivity maps. In particular, note that the refined mask is more complete and perfect compared to the mask predicted by a single Mask R-CNN. The textured mask not only presents strong segmentation ability, but also shows the object details well. Overall, the texture part enables users to perfectly understand the detailed information about each object, which can strongly and effectively promote further research.

Additional results (Table 6) show that the MobileNet-FPN model has slightly higher mean average precision (mAP) than ResNet-FPN; this means that the MobileNet-based model has strong robustness and portability. Moreover, by combining post-processing, the accuracy can be further improved. Meanwhile, the mean intersection over union (mIoU) in the proposed method is improved compared to other models. Hence, these results prove that our model exhibits superior performance and robustness.

In brief, the proposed TextureMask model represents an entirely new approach to drawing the mask by concatenating five major modules: preprocessing, MobileNet for Mask R-CNN, segmentation with cropping, sensitivity mapping, and merger with post-processing. Experiments prove that the novel model achieves the superior performance of masking with texture in terms of high efficiency, high accuracy, and low resource utilization, so that it can be easily ported to front-end applications.

5. Conclusion

We constructed a robust merged segmentation architecture (TextureMask) by integrating three different perspectives to generate textured masks, which can provide an advanced explanation for understanding objects from a new perspective. Meanwhile, TextureMask combines the merits of different deep learning algorithms. Our experiments demonstrated that the proposed model exhibits the superior performance of textured masks in the LR dataset. Furthermore, it is notable that TextureMask can be easily transplanted to mobile devices for segmenting because of its remarkably low memory cost and high precision. Therefore, the algorithm proposed in this paper has both high efficiency and portability. In future research, we aspire to make further contributions to make the model even more lightweight and improve the effectiveness of the masks.

Footnotes

Acknowledgments

This research was funded by National “13th Five-Year” Science and Technology Projects in China, and the funding number is JSZL2017203B023.

References

Krizhevsky

Sutskever

and Hinton

G.E.

, ImageNet classification with deep convolutional neural networks, Communications of the ACM 60 (2012), 84–90.

et al., Mask R-CNN, IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (2020), 386–397.

Ronneberger

Fischer

and Brox

, U-Net: Convolutional networks for biomedical image segmentation, 2015, ArXiv, abs/1505.04597.

Zhao

et al., Pyramid scene parsing network, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6230–6239.

Akilan

et al., A 3D CNN-LSTM-based Image-to-Image foreground segmentation, IEEE Transactions on Intelligent Transportation Systems 21 (2020), 959–971.

Gao

et al., Salient object detection in the distributed cloud-edge intelligent network, IEEE Network 34 (2020), 216–224.

Banerjee

and Maji

, A spatially constrained probabilistic model for robust image segmentation, IEEE Transactions on Image Processing 29 (2020), 4898–4910.

Perazzi

et al., Learning video object segmentation from static images, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 3491–3500.

Caelles

et al., One-shot video object segmentation, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5320–5329.

10.

Cheng

et al., SegFlow: Joint learning for video object segmentation and optical flow, in: IEEE International Conference on Computer Vision (ICCV), 2017, pp. 686–695.

11.

Cheng

et al., Fast and accurate online video object segmentation via tracking parts, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 7415–7424.

12.

Huang

and Schwing

, VideoMatch: matching based video object segmentation, 2018, ArXiv, abs/1809.01123.

13.

and Loy

C.C.

, Video object segmentation with joint re-identification and attention-aware mask propagation, ECCV, 2018.

14.

Bhatti

A.H.

Rahman

A.U.

and Butt

, Unsupervised video object segmentation using conditional random fields, Signal, Image and Video Processing 13 (2019), 9–16.

15.

Lin

et al., Towards optimal structured CNN pruning via generative adversarial learning, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 2785–2794.

16.

Xin

and Lin

S.H.

, Pruning and retraining method for a convolution neural network, U.S. Patent Application 15/699, 438, filed March 14, 2019.

17.

Moon

et al., Memory-reduced network stacking for edge-level CNN architecture with structured weight pruning, IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9 (2019), 735–746.

18.

Nogami

et al., Optimizing weight value quantization for CNN inference, in: International Joint Conference on Neural Networks (IJCNN), 2019, pp. 1–8.

19.

Banner

Nahshan

and Soudry

, Post training 4-bit quantization of convolutional networks for rapid-deployment, NeurIPS, 2019.

20.

Zhang

and Li

, Unleashing the power of soft logic for convolutional neural network acceleration via product quantization, in: Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, 2019.

21.

Chandra

, Data bandwidth reduction in deep neural network SoCs using history buffer and Huffman coding, in: International Conference on Computing, Power and Communication Technologies (GUCON), 2018, pp. 1–3.

22.

Howard

A.G.

et al., MobileNets: Efficient convolutional neural networks for mobile vision applications, 2017, ArXiv, abs/1704.04861.

23.

Sandler

et al., MobileNetV2: Inverted residuals and linear bottlenecks, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 4510–4520.

24.

Zhang

et al., ShuffleNet: An extremely efficient convolutional neural network for mobile devices, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 6848–6856.

25.

et al., ShuffleNet V2: Practical guidelines for efficient CNN architecture design, ECCV, 2018.

26.

Tan

and Le

Q.V.

, EfficientNet: Rethinking model scaling for convolutional neural networks, 2019, ArXiv, abs/1905.11946.

27.

Girshick

R.B.

et al., Rich feature hierarchies for accurate object detection and semantic segmentation, in: IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 580–587.

28.

Girshick

R.B.

, Fast R-CNN, in: IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1440–1448.

29.

Ren

et al., Faster R-CNN: towards real-time object detection with region proposal networks, IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (2015), 1137–1149.

30.

et al., Spatial pyramid pooling in deep convolutional networks for visual recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 37 (2015), 1904–1916.

31.

Ventura

et al., RVOS: End-to-end recurrent network for video object segmentation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 5272–5281.

32.

Zhu

et al., Improving semantic segmentation via video propagation and label relaxation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 8848–8857.

33.

Vuola

Akram

and Kannala

, Mask-RCNN and U-net ensembled for nuclei segmentation, in: IEEE 16th International Symposium on Biomedical Imaging (ISBI), 2019, pp. 208–212.

34.

Johnander

et al., A generative appearance model for end-to-end video object segmentation, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 8945–8954.

35.

Zhang

et al., Context encoding for semantic segmentation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 7151–7160.

36.

Zeng

et al., Dmm-net: Differentiable mask-matching network for video object segmentation, in: IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 3928–3937.

37.

Wang

et al., RDSNet: A new deep architecture for reciprocal object detection and instance segmentation, AAAI, 2020.

38.

Chen

et al., Hybrid task cascade for instance segmentation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4969–4978.

39.

Wang

et al., Understanding convolution for semantic segmentation, in: IEEE Winter Conference on Applications of Computer Vision (WACV), 2018, pp. 1451–1460.

40.

Xie

et al., Polarmask: Single shot instance segmentation with polar representation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 12190–12199.

41.

Maninis

et al., Video object segmentation without temporal information, IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (2019), 1515–1530.

42.

et al., See more, know more: Unsupervised video object segmentation with co-attention siamese networks, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 3618–3627.

43.

Ying

et al., Embedmask: Embedding coupling for one-stage instance segmentation, 2019, ArXiv, abs/1912.01954.

44.

Wang

et al., Deep networks for image super-resolution with sparse prior, in: IEEE International Conference on Computer Vision (ICCV), 2015, pp. 370–378.

45.

Gregor

and LeCun

, Learning fast approximations of sparse coding, in: International Conference on International Conference on Machine Learning (ICML), 2010, pp. 399–406.

46.

Dong

et al., Image super-resolution using deep convolutional networks, IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (2016), 295–307.

47.

BinoSebastian

Unnikrishnan

and Balakrishnan

, Gray level co-occurrence matrices: generalization and some new features, 2012, ArXiv, abs/1205.4831.

48.

Dalal

and Triggs

, Histograms of oriented gradients for human detection, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) 1 (2005), 886–893.

49.

Zhang

, The optimality of naive Bayes, in: FLAIRS Conference, 2004.

50.

Breiman

, Random forests, Machine Learning 45 (2004), 5–32.

51.

Freund

and Schapire

, A decision-theoretic generalization of on-line learning and an application to boosting, J. Comput. Syst. Sci. 55 (1995), 119–139.

52.

Suykens

and Vandewalle

, Least squares support vector machine classifiers, Neural Processing Letters 9 (2004), 293–300.

53.

Smilkov

et al., SmoothGrad: removing noise by adding noise, 2017, ArXiv, abs/1706.03825.

54.

Springenberg

J.T.

et al., Striving for Simplicity: The all convolutional net, 2015, arXiv, abs/1412.6806.

55.

Taylor

G.W.

et al., Convolutional learning of spatio-temporal features, ECCV, 2010.

56.

Hara

Kataoka

and Satoh

, Learning spatio-temporal features with 3D residual networks for action recognition, in: IEEE International Conference on Computer Vision Workshops (ICCVW), 2017, pp. 3154–3160.

57.

Sun

et al., Human action recognition using factorized spatio-temporal convolutional networks, in: IEEE International Conference on Computer Vision (ICCV), 2015, pp. 4597–4605.

58.

Moschetti

and Debes

, Data adapting motion estimation and subsampling, Visual Communications and Image Processing, 2000.

59.

Luo

D.E.

et al., Patent No. 7,978,770, Washington, DC: U.S. Patent and Trademark Office, 2011.

60.

Chen

and Chiueh

, One-dimensional full search motion estimation algorithm for video coding, IEEE Trans, Circuits Syst. Video Technol 4 (1994), 504–509.

61.

Huynh-Thu

and Ghanbari

, Scope of validity of PSNR in image/video quality assessment, Electronics Letters 44 (2008), 800–801.

62.

Wang

et al., Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing 13 (2004), 600–612.

63.

Zhang

et al., FSIM: a feature similarity index for image quality assessment, IEEE Transactions on Image Processing 20 (2011), 2378–2386.

64.

Powers

, Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation, ArXiv, abs/2010.16061.

TextureMask: A merged architecture for low-resolution instance segmentation

Abstract

Keywords

1. Introduction

2. Related work

3. Proposed approach

3.1 Overall pipeline

3.5 Gradient sensitivity maps

3.7 Post-processing

4.1 Dataset

Table 2 Details of datasets

4.2.1 Preprocessing

4.2.2 Pre-classifiers

5. Conclusion

Footnotes

Acknowledgments

References

Table 2
Details of datasets