LHRNet: Lateral hierarchically refining network for salient object detection

Abstract

Deep convolutional neural networks (CNNs) have shown outstanding performance in salient object detection. However, there exist two conundrums under-explored. 1) High-level features are beneficial to locate salient objects while low-level features contain fine-grained details. How to combine these two types of features to promote accuracy is the first conundrum. 2) Previous CNN-based methods adopt a convolutional layer after extracting features to infer saliency maps. While encountering images that are different greatly from training dataset, adopting a convolutional layer as a classifier is not robust enough to detect all salient objects. In addition, limited receptive field and lack of spatial correlation will cause salient objects to be incomplete while blurring their boundaries. In this paper, a Lateral Hierarchically Refining Network (LHRNet) is put forward for accurate salient object detection. Firstly, LHRNet efficiently integrates multi-level features, which simultaneously incorporates coarse semantics and fine details. Then a coarse saliency prediction is made from low-resolution features by convolution. Finally, a series of nearest neighbor classifiers are learned to hierarchically restore the missing parts of salient objects while refining their boundaries, yielding a more reliable final prediction. Comprehensive experiments demonstrate that this network performs favorably against state-of-the-art approaches on six datasets.

Keywords

Salient object detection Deep learning Convolutional neural networks

1 Introduction

Salient object detection aims to identify the most visually distinctive objects or regions in an image, which commonly serves as the first step for a variety of computer vision applications such as semantic segmentation [17], image and video compression [18], object recognition [19, 20], visual tracking [21] and scene classification [22].

In many previous traditional methods [23 –26], low-level hand-crafted features, center-surround contrast and heuristic priors play essential roles to determine the position of a prominent object. But it is difficult to use these methods to detect salient objects with diverse scales, shapes and locations in complex scenes especially when image background is similar to foreground.

Recently, deep convolutional neural networks (CNNs) have delivered remarkable performance in many recognition tasks. CNNs can be trained end-to-end through back propagation without the need to extract traditional hand-crafted features and do a lots of heuristic priors. However, repeated subsampling operations such as pooling layers and convolutional layers adopted in earlier CNN-based models will lead to a significant decrease in the initial image resolution, which is fatal for dense prediction tasks. In order to deal with this problem, the Fully Convolutional Network (FCN) [27] adopts de-convolutional layers to recover space information and achieves the best results in semantic segmentation. Motivated by the superiorities of FCN, several attempts have been performed to utilize FCN in salient object detection.

Most previous models based on FCN for salient object detection exist two conundrums.

a) Some models [5, 7] predict saliency maps by mainly using high-level semantic features from deep convolutional layers without taking low-level spatial details into consideration. However, high-level features are beneficial to locate salient objects while low-level features can make the saliency maps to retain fine object boundaries. Although different levels of features are beneficial to the improvement of results, simply concatenating them is not the optimal way. Hou et al. [13] add short connections between multiple side output layers to combine features of different levels. Lin et al. [29] propose a top-down architecture with lateral connections to build high-level semantic feature maps at all scales. Zhang et al. [8] integrate multi-level feature maps into multiple resolutions then adaptively combine these feature maps to predict saliency maps. Zhang et al. [9] propose a model to pass messages among features from different levels and design a mechanism to adaptively fuse these messages. All of these models improve the accuracy by constructing communication between different side outputs layers, essentially expanding the parameter space and complexity of their networks, but their results are still not satisfactory. How to effectively use different level features is the first conundrum.

b) Many methods [5 , 13] adopt convolutional layers with kernel size 1x1 or 3x3 as classifiers at the end of network to infer saliency maps, which is sub-optimal to detect all salient objects with diverse scales, shapes and locations especially those having complex boundaries in complex scenarios. This problem can be explained in three ways. (1) The parameters of convolutional kernels depend on patterns of training dataset. At the end of training, the parameters of the convolutional kernels are fixed. Once the network encounters a pattern that differs greatly from the training dataset, the network will produce an extremely terrible saliency map because it no longer adaptively adjusts the parameters of convolutional kernels according to the input. (2) The receptive field of a convolutional kernel is limited. When a convolutional kernel just covers a small region, it can not determine which of the internal pixels are located in the foreground area and which are in the background area. Salient object detection needs to determine which objects belong to the foreground and which objects belong to the background from global semantic information. This situation can be shown as Figure 1. But it is obviously impractical to set the size of a convolutional kernel to be as large as the input image, which will lead to an explosion of memory and computation.

Fig.1

Red solid dot represents a certain pixel, and red dashed square boxes represent different sizes of receptive fields of convolutional layers. It is not able to determine the saliency of the pixel from small receptive field. In other words, whether the green area or the orange area belongs to salient object is confusing. As the receptive field increases, it is clearly seen that the pixel belongs to a dog running on the lawn. In this case, it is undoubted that the pixel should be masked as salient.

(3) Convolutional layers use sliding window to share weights, but this operation will result in a lack of spatial correlation between different parts of same salient object that are far away. For example, in a complicated scene, a person wearing clothes may have the arm properly segmented, but his legs are lost. If global spatial correlation is introduced in the inference process, the correlation between different parts of the same salient object is enhanced. As long as one of the parts is correctly segmented, the remaining ones can be correctly segmented regardless of their distances. Rather than adopting a convolutional layer to infer results, LPS [12] learns a nearest neighbor classifier to highly refine results generated by previous methods, which can be considered as a substitute for conditional random field (CRF) [30]. Nearest neighbor classifier (NNC) has no parameters, and it can obtain the saliency of each pixel from global contrast, which achieves global spatial correlation to some extent. But the structure proposed in LPS is not an end-to-end model because it needs to adopt results produced by other models as initial saliency maps.

Fig.2

Visual examples of the proposed LHRNet, Amulet [8], DCL [6] and GBMPM [9]. The red dashed square boxes indicate the missing saliency information. All of Amulet, DCL, GBMPM and our S₄ adopt a convolutional layer at the end of network to infer saliency maps. It can be seen that although the salient objects in S₄ are incomplete and have fuzzy boundaries, the missing parts of them are restored and their boundaries are refined through a series of NNCs.

In this paper, motivated by the superiorities of NNC, a Lateral Hierarchically Refining Network (LHRNet) is proposed for accurate salient object detection. Firstly, LHRNet efficiently integrates multi-level features, which simultaneously incorporates coarse semantics and fine details. Then a coarse saliency map is predicted from low-resolution features by a convolutional layer. Finally, a series of nearest neighbor classifiers are learned to hierarchically restore the missing parts of salient objects while refining their boundaries, yielding a more reliable final prediction. Figure 2 lists some saliency maps predicted by our LHRNet and other state-of-the-art methods. The main contributions in this paper are summarized as follows:

A well-designed information flow structure is constructed to better integrate high-level sematic features and low-level fine-grained details.

A lateral hierarchically optimization structure implemented by nearest neighbor classifiers is proposed to restore the missing parts of salient objects while refining their boundaries, yielding a more reliable final prediction.

A Lateral Hierarchically Refining Network (LHRNet) consisting of the two structures described above is put forward for accurate salient object detection. Apart from taking full advantages of abundant multi-level features, LHRNet can avoid the problems caused by adopting convolutional layers as classifiers after extracting features to some extent. Comprehensive experiments demonstrate that the LHRNet performs favorably against state-of-the-art approaches under different evaluation metrics on six datasets.

2 Related work

2.1 Salient object detection

Previous conventional methods [23 –26] implement an unsupervised structure to segment salient regions from the backgrounds. These methods typically extract local pixel-wise or region-wise hand-crafted features such as RGB, grayscale and texture, then compare these features with global features. For example, Yang et al. [24] rank the similarities of the image elements with foreground cues or background cues via graph-based manifold ranking. The saliency of an image element is defined based on its relevance to the given seed or query. Cheng et al. [25] introduce a regional contrast based salient object detection algorithm which simultaneously evaluates global contrast differences and spatial weighted coherence scores. But these methods will produce terrible results while encountering salient objects with diverse scales, shapes, and locations in complex scenarios.

Deep convolutional neural network (CNN) which is trained end-to-end through back propagation has made a major breakthrough in many computer vision tasks. A lot of research efforts have been made to develop various deep architectures for salient object detection. Wang et al. [1] construct two deep neural networks to integrate local pixel estimation and global proposal search. Li et al. [2] extract multi-scale features for each super-pixel and then append fully connected layers on top of model to predict saliency degree. Zhao et al. [3] design a deep model to extract multiple contexts that are integrated into a multi-context deep learning framework in the end for salient object detection. These models only adopt high-level features obtained by stacking convolutional layers and fully connected layers, regardless of spatial information of input images. In order to deal with this problem, Lee et al. [4] encode low-level distance map of super-pixels and high-level semantic features of CNNs, then combine them to predict saliency of each super-pixel. Li et al. [6] propose two complementary components, a pixel-level fully convolutional stream and a segment-wise spatial pooling stream to predict saliency maps. Apart from these methods, some works endeavor to aggregate multi-level convolutional feature maps to detect salient regions. Zhang et al. [8] firstly integrate multi-level feature maps into multiple resolutions, then combine these maps to predict several saliency maps which are fused to generate a final result in the end. Zhang et al. [9] take into consideration information transmission from shallow side output layers to deep ones, so they propose a bi-directional message passing model to integrate multi-level features. Luo et al. [11] combine local and global information through a multi-resolution 4x5 grid structure to promote features with strong local contrast. Hou et al. [13] introduce a series of short connections between shallower and deeper side output layers instead of directly connecting loss layers to the last layer of each stage.

2.2 Boundary refinement of salient object detection

The saliency maps generated by simply using convolutional neural networks tend to have very blurred boundaries. The conditional random field [10] has been used as post-processing instrument to optimize the boundaries of salient regions. CRF is an undirected graph model combining the characteristics of maximum entropy model and hidden Markov model. Besides, Liu et al. [5] firstly make a coarse global prediction by learning various global structured saliency cues, then construct a network to hierarchically refine the details of saliency map step by step. Wang et al. [7] construct a deep recurrent network to incorporate a coarse predicted map as saliency prior and then automatically learn to refine the map by iteratively correcting its previous errors. In addition, recent methods attempt to optimize the loss function to obtain relatively exquisite boundary. Luo et al. [11] add an extra loss function while training their network to penalize errors on the boundary. Cai et al. [35] construct a new topological metric space, with the implicit metric being determined by a deep network. However, all of CNN-based methods mentioned above still adopt a single-scale convolutional layer with kernel size 1x1 or 3x3 to infer saliency maps, which is incapable of fundamentally solving the problems discussed at the beginning. Different from these methods, Zeng et al. [12] believe that learning a unified detector such as a convolutional layer is hard to detect all varieties of salient objects. So given an input image with its coarse saliency map that is produced by other previous methods, Zeng et al. firstly construct a DNN as an embedding function and then learn a nearest neighbor classifier to refine the coarse saliency map, which can be considered as a substitute for conditional random field.

3 Lateral hierarchically refining network

The proposed Lateral Hierarchically Refining Network dubbed LHRNet can restore the missing parts of salient objects while refining their boundaries step by step. Figure 3 shows the intact illustration of the LHRNet. The structure of the proposed network will be described in Section 3.1. The detailed formulas of nearest neighbor classifier (NNC) for salient object detection will be given in Section 3.2. Section 3.3 provides the loss function to train the LHRNet.

Fig.3

The architecture of the LHRNet. All additional convolutional layers in the information flow structure use 3x3 convolution kernels and padding. Each H*W*C annotation denotes the corresponding feature maps’ height, width and channel. For more concise statement, all ReLU layers and up-sample operation are not drawn in this structure diagram. Up-sample operation is adopted to adapt the spatial scale of different convolutional features. In practice, bilinear interpolation is used as up-sample function. Firstly, a coarse saliency map is predicted in the deepest side output layer by a convolutional layer with kernel size 3x3. Then the map is refined step by step by learning a series of NNCs.

3.1 Network architecture

The architecture of LHRNet is built on the pre-trained VGG16 [36] which has been widely used as feature extraction backbone in many deep learning models due to its simplicity and good generalization properties. VGG16 stacks repeated fully connected layers to infer results for image classification, which is not suitable for dense prediction tasks. So all the fully connected layers and the last pooling layer are removed. The input image is resized to 256x256, and feature maps at five levels Conv1_2, Conv2_2, Conv3_3, Conv4_3 and Conv5_3 are selected, which is denoted as f_i, i = 1, 2, 3, 4, 5. VGG16 appends subsampling operation such as a pooling layer to each convolutional layer, so the resolution of f_i is $[\frac{256}{2^{i - 1}}, \frac{256}{2^{i - 1}}]$ .

Visual context is essential to detect salient regions with large variations in scale, shape and position. Szegedy et al. [31] launch inception that contains multiple branch CNNs with different convolutional kernels to extract features from different receptive fields. Inspired by inception, its variants [32, 33] are put forward to achieve competitive results in object detection and classification. Yu et al. [34] propose dilated convolution which supports exponential expansion of the receptive field without loss of resolution or coverage. They applies four parallel dilated convolutions with different dilated rates to systematically aggregate multi-scale contextual information. Zhang et al. [9] design a MCFEM to capture multi-scale contextual information by stacking dilated convolutions which have the same kernel size 3x3 with different dilated rates that are set to 1, 3, 5, 7. In the LHRNet, MCFEM is added to f_i for the sake of obtaining abundant multi-scale information. This operation will generate feature maps $f_{i}^{c}$ . In order to enlarge capacities of dilated convolutional layers, the channel size of $f_{i}^{c}$ is set to 512.

Information Flow Structure. Although abundant multi-scale feature maps $f_{i}^{c}$ have been obtained, there exist large semantic gap between $f_{i}^{c}$ . High-level semantic knowledge can better locate salient objects while low-level spatial details are helpful to accurately detect boundaries. However, directly concatenating $f_{i}^{c}$ is sub-optimal to take full advantages of multi-context features. In addition, there still exist redundant information in $f_{i}^{c}$ , which is unbeneficial to the improvement of accuracy and will increase the amount of memory and calculation in the subsequent operation. So it is essential to efficiently combine $f_{i}^{c}$ while reducing the complexity of the LHRNet. In order to deal with these issues, a solution that consists of two steps is proposed. Firstly, the two adjacent features $f_{i}^{c}$ and $f_{i + 1}^{c}$ are concatenated by cross-channel manner, followed by a convolutional layer to reduce channel size. Secondly, short connections [13] between shallower and deeper side output layers are adopted to integrate multi-level features and improve features’ representational ability. These series of operations are respectively performed as Eqs. (1) and (2). In this way, $h_{i}^{2}$ can incorporate coarse semantic and fine details at the meantime.

$\begin{matrix} h_{i}^{0} & = Concat (f_{i}^{c}, Up (f_{i + 1}^{c})), i = 1, 2, 3, 4 \\ h_{i}^{1} & = Relu (Conv (h_{i}^{0}; θ_{i}^{1})), i = 1, 2, 3, 4 \end{matrix}$ (1) and $\begin{matrix} h_{4}^{2} & = Relu (Conv (h_{4}^{1}; θ_{4}^{2})) \\ h_{3}^{2} & = Relu (Conv (Concat (h_{3}^{1}, Up (h_{4}^{2})); θ_{3}^{2})) \\ h_{2}^{2} & = Relu (Conv (Concat (h_{2}^{1}, Up (h_{3 \sim 4}^{2})); θ_{2}^{2})) \\ h_{1}^{2} & = Relu (Conv (Concat (h_{1}^{1}, Up (h_{2 \sim 4}^{2})); θ_{1}^{2})) \end{matrix}$ (2) where Concat () is cross-channel concatenation; Relu () is ReLU activation function. Compared with sigmoid function and tanh function that have a saturated region which is easy to cause gradient vanishing and slow down the convergence speed, the ReLU function keeps the gradient constant when its input x > 0, which alleviates gradient vanishing, and is much faster than sigmoid and tanh in convergence speed. In addition, most other state-of-the-art methods [4 , 9] use ReLU as an activation function. The LHRNet attempts to confirm the advantages of the information flow structure and the final series of hierarchical NNCs. It can’t be judged whether the advantage of the LHRNet is due to the selection of the activation function if other activation functions such as sigmoid, tanh or leak ReLU is adopted. Conv (* ; θ) is a convolutional layer with parameter θ = {W, b}; All additional convolutional layers in the information flow structure use 3x3 convolution kernels and padding. Up () is an up-sample operation to adapt the spatial scale of different convolutional features. In practice, bilinear interpolation is used as up-sample function. $Up (h_{3 \sim 4}^{2})$ denotes the set of $Up (h_{3}^{2})$ and $Up (h_{4}^{2})$ , while $Up (h_{2 \sim 4}^{2})$ denotes the set of $Up (h_{2}^{2})$ , $Up (h_{3}^{2})$ and $Up (h_{4}^{2})$ . Then, $h_{i}^{2}$ is resized to 256x256.

Hierarchical NNC Refinement. It has been discussed the inadequate capabilities of adopting convolutional layers as classifiers to detect salient objects at the beginning. Inspired by the superiorities of nearest neighbor classification for salient object detection, LHRNet consists of an end-to-end structure which incorporates a series of NNCs to restore the missing parts of salient objects and refine their boundaries at the meantime. The whole process of the inference is performed as Eq. (3).

$\begin{matrix} S_{4} & = Softmax (Conv (h_{4}^{2}; θ_{4}^{3})) \\ S_{i} & = NNC (h_{i}^{2}, S_{i + 1}), i = 3, 2, 1 \end{matrix}$ (3)NNC () represents the formulas of nearest neighbor classification and Softmax () denotes a Softmax layer. At first, a coarse saliency prediction S₄ is made from low-resolution features by using a convolutional layer with kernel size 3x3. S₄ is primarily used to locate the salient regions. $h_{i}^{2}$ and its high-level prediction map S_i+1 are taken as input (i=3,2,1), then a nearest neighbor classifier is learned to restore the missing parts of salient objects while refining their boundaries. In this way, after a series of hierarchical optimizations, an accurate saliency map S₁ is obtained. The detailed formulas of NNC are introduced in next section.

3.2 Detailed formulas of NNC for salient object detection

Given an input image I with its feature maps h and coarse saliency map S, the average feature value of pixels in foreground region and background region are extracted respectively, obtaining two vectors as anchors. This process is performed as Eq. (4).

$\begin{matrix} f_{+} = \frac{1}{| S_{+} |} \sum_{i \in S_{+}} h_{i} \\ f_{-} = \frac{1}{| S_{-} |} \sum_{i \in S_{-}} h_{i} \end{matrix}$ (4) where S₊ denotes the foreground regions, and S_- denotes the background regions, S = S₊ ∪ S_-. f₊ denotes the foreground anchor and f_- denotes the background anchor. Euclidean distances of each pixel and the two anchors are calculated separately. The probability of the i-th pixel belonging to salient region and non-salient region can be given by the softmax over its distance to the anchors, obtaining the refined saliency map Y. These processes are defined as Eqs. (5) and (6).

$\begin{matrix} {dist}_{i}^{+} & = ∥ h_{i} - f_{+} ∥, \\ {dist}_{i}^{-} & = ∥ h_{i} - f_{-} ∥ \end{matrix}$ (5) and

$\begin{matrix} P (l_{i} = 1 | θ) & = \frac{\exp (- {dist}_{i}^{+})}{\exp (- {dist}_{i}^{+}) + \exp (- {dist}_{i}^{-})}, \\ P (l_{i} = 0 | θ) & = \frac{\exp (- {dist}_{i}^{-})}{\exp (- {dist}_{i}^{+}) + \exp (- {dist}_{i}^{-})} \end{matrix}$ (6) where ${dist}_{i}^{+}$ and ${dist}_{i}^{-}$ denote the Euclidean distance of each pixel and the two anchors separately, P (l_i = 1|θ) and P (l_i = 0|θ) represent the label probability of the i-th pixel in Y. Figure 4 shows the visualization of adopting NNC to refine saliency maps.

Fig.4

Visualization of adopting nearest neighbor classification to refine saliency maps. (a) input image. (b) feature maps. For a more concise description, the structure that produces the feature maps is omitted. (c) and (d) represent coarse foreground map and coarse background map respectively, both of them are gray-scales. It should be noticed that (d) =255 - (c). (e) and (f) denote foreground anchor and background anchor. (g) denotes the refined saliency map. After extracting the average feature value of pixels in foreground and background respectively to obtain two vectors as anchors, the Euclidean distances of each pixel and the two anchors are calculated separately. Softmax function is adopted in the end to generate the refined saliency map.

The NNC has two advantages: a) NNC is feature similarity calculation based on distance metrics, not derived from nonlinear mapping of convolution kernels. Its essence is a non-parametric decision mechanism, which can greatly reduce the interference of different data distribution and improve the generalization ability. NNC calculates the feature distance of each pixel and anchor when optimizing the initial result. As long as the feature vector of a pixel can correctly encode the information through ahead information flow structure and is close to its corresponding real anchor in probability, it can be correctly marked. Therefore, the NNC can recover the missed regions and obtain fine boundaries. b) NNC is based on global contrast. For each picture to be detected, the anchors are generated by feature averaging from the global view of the initial saliency map. Therefore, the foreground anchor can encode the central feature of the salient regions, while the background anchor can encode the central feature of the background. The saliency value of each pixel is determined by the feature similarity between the pixel and the anchors. Therefore, NNC is based on global contrast rather than local contrast. This decision mechanism is more in line with the nature of salient object detection: salient object detection should assign labels to pixels based on global context, rather than just their neighborhood.

3.3 Loss function

Now there exist three refined saliency maps S₁, S₂, S₃ and a coarse saliency map S₄. Cross entropy (CE) is adopted to optimize the network. For S_k, the loss function is written as Eq. (7).

$\begin{matrix} L_{CE}^{k} & = - \frac{1}{N \times | Ω |} \sum_{n = 1}^{N} \sum_{i = 1}^{Ω} \sum_{y = 0}^{1} 1 {l_{i}^{n} = y} {lnP}_{k} (l_{i}^{n} = y | θ), \\ k = 1, 2, 3, 4 \end{matrix}$ (7) where 1 {} denotes indicator function; Ω is the pixel domain of the image; N denotes the batch size; y = 1 denotes the salient pixel and y = 0 denotes the non-salient pixel; $P_{k} (l_{i}^{n} = 1 | θ)$ and $P_{k} (l_{i}^{n} = 0 | θ)$ represent the label probability of the i-th pixel in S_k, k = 1, 2, 3, 4. Thus, the joint loss function for all predictions is defined as Eq. (8).

$\begin{matrix} L_{CE}^{all} & = \sum_{k = 1}^{4} a_{k} L_{CE}^{k} \end{matrix}$ (8) where a_k are the loss weights to balance each loss term. For simplicity and fair comparison, all of a_k are set to 1. The above loss function is continuously differentiable, so the stochastic gradient descent (SGD) method can be used to train the LHRNet end-to-end.

4 Experiments

4.1 Experimental setup

Dataset. The LHRNet is evaluated on six benchmark datasets: ECSSD, PASCAL-S, HKU-IS, DUTS, DUT-OMRON and SOD. The ECSSD dataset has 1000 images with various complex scenes. The PASCAL-S dataset consists of 850 images which are more complicated with cluttered backgrounds and multiple objects based on the validation set of the PASCAL VOL 2009 segmentation challenge. DUT-OMRON includes 5168 images in complex scenes. The HKU-IS dataset contains 4447 images with low color contrast and diverse foreground objects in each image. DUTS dataset is the largest salient object detection benchmark dataset at present which contains two parts, DUTS-TRAIN and DUTS-TEST. DUTS-TRAIN has 10553 images and DUTS-TEST has 5019 images, both of which are challenging with salient objects of different locations and scales in different complex backgrounds. SOD contains 300 images which have complex backgrounds and multiple foreground objects.

Evaluation metrics. In order to illustrate the effectiveness of LHRNet, three metrics including precision-recall (PR) curves, maximum F-measure and mean absolute error (MAE) are adopted to evaluate the performance of the LHRNet as well as other 11 state-of-the-art salient object detection methods. In the classification task, the precision value is the ratio of ground truth pixels in the predicted salient region while the recall value is the ratio of the predicted saliency pixels in the ground truth area. Researchers can threshold the predicted map and compare it with the ground truth to draw the PR curves. F-measure which is computed the relationship between precision and recall at different thresholds with an adjustable coefficient β represents the overall performance:

$F_{β} = \frac{(1 + β^{2}) \times Precision \times Recall}{β^{2} \times Precision + Recall}$ (9) where β² is set to 0.3 as suggested in previous work. Maximum F-measure can well reflect the performance of the LHRNet, so it is adopted as one of the evaluation index. Moreover, the mean absolute error (MAE) is calculated to compute the average absolute per-pixel difference between predicted map and the corresponding ground truth. The formula of MAE is defined as Eq. (10).

$MAE = \frac{1}{W \times H} \sum_{x = 1}^{W} \sum_{y = 1}^{H} | S (x, y) - G (x, y) | .$ (10)

Implementation details. ln LHRNet, the parameters of the first 13 convolutional layers are initialized by pre-trained VGG16 net, while the weights of other convolutional layers are initialized by using truncated normal method. Compared with random normal, truncated normal can ensure that the generated initial values are near the mean while introducing randomness, avoiding the introduction of extreme values which will deteriorate the network training. In addition, majority of state-of-the-art methods [5 , 11] use the truncated normal method to initialize kernel weights. In order to avoid the deviation of the initialization methods which affect the performance comparison, truncated normal method is adopted to initialize the weights of other convolutional layers in the LHRNet. DUTS-TRAIN dataset which contains 10553 images with high-quality pixel-wise annotations is utilized to train the LHRNet. In the training phase, the batch size is set to 1. The training set is augmented by horizontal flipping and vertical flipping. LHRNet does not use validation set and is trained until the training loss converges. LHRNet is trained using Adam with an initial learning rate at 1e-6. The overall training process takes about 22 hours and converges after 14 epochs. A NVIDIA TITAN X (Pascal) GPU is used for training and testing. The details are showed in Algorithm 1.

Algorithm 1 The input parameters and the training process for LHRNet

Input: source RGB images I with their ground truths G in DUTS-TRAIN dataset, learning rate = 1e - 6, batch size = 1, epoch = 20;

Output model parameters θ : = {W, b}

1: Augment I by horizontal flipping and vertical flipping

2: last is the loss of the last iteration;

cur is the loss of the current iteration;

LHR denotes the proposed network;

L_CE denotes Cross Entropy Loss;

3: Initialize the model parameters of the first 13 convolutional layers θ₁ by pre-trained VGG16 net and the model parameters of other convolutional layers θ₂ by using truncated normal method, θ = θ₁ ∪ θ₂;

Initialize last = 0, cur = 0;

4: for iter = 1 to epoch do

5: for I_i ∈ I do

6: Obain corresponding predicted salienct map P_i = LHR (I_i) , P_i includes {S₁, S₂, S₃, S₄};

7: Calculate cross entropy loss L_CE (G_i, P_i);

8: Update model parameters θ through back propagation;

9: cur ← cur + L_CE (G_i, P_i);

10: end for

11: if iter = =1 or cur < last then

12: last=cur;

13: cur=0;

14: else

15: Break;

16: end if

17: end for

18: return θ

4.2 Performance comparison

First of all, comparison between S_k is performed. Then LHRNet is compared with 11 state-of-the-art methods, including PiCANet [14], GBMPM [9], Amulet [8], DCL [6], DHS [5], DSS [13], ELD [4], NLDF [11], RFCN [7], SRM [15] and UCF [16]. For fair comparison, the saliency maps of different methods are provided by the authors or achieved by running available codes.

Comparison Between S_k. Comparison between S_k is performed on six datasets. Table 1 illustrates the performance of S_k under the metrics of maximum F-measure and MAE. It can be observed from Table 1 that although F_max is not monotonically increasing and MAE is not monotonically decreasing with the optimization of nearest neighbor classification, but generally they are getting better. In term of HKU-IS and ECSSD, S₁ and S₂ are the same. S₁ performs slightly better on SOD, while S₂ performs slightly better on DUT-OMRON, PASCAL-S and DUTS-TEST. In order to demonstrate better generalization capabilities on the six datasets, the average of S₁ and S₂ is selected as final result. Figure 5 lists some saliency maps of S_k. These images are selected from six datasets for testing. It can be seen that S₄ can only basically locate the salient regions with blurred boundaries. Through continuous refinements by learning a series of nearest neighbor classifiers, the boundaries of salient objects become more and more clear. Moreover, some tiny regions that were missing before, such as the legs of scolopendras, the feet of ants, and the horns of deer, are correctly segmented. This precision is not achieved by many previous deep learning models.

Fig.5

Examples of saliency maps in different side output layers. It can be seen that S₄ can only basically locate the salient regions with blurred boundaries. Some tiny regions that were missing before, such as the legs of scolopendras, the feet of ants, and the horns of deer, are correctly segmented through continuous refinements. This precision is not achieved by many previous deep learning models.

Table 1

The maximum F-measure and MAE of S_k

Branches	SOD		HKU-IS		DUT-OMRON		PASCAL-S		ECSSD		DUTS-TEST
		F _max	MAE	F _max	MAE	F _max	MAE	F _max	MAE	F _max	MAE	F _max	MAE
S ₁	0.875	0.100	0.929	0.033	0.796	0.079	0.872	0.075	0.937	0.038	0.852	0.055
S ₂	0.873	0.099	0.929	0.033	0.800	0.074	0.873	0.074	0.937	0.038	0.855	0.053
S ₃	0.873	0.097	0.926	0.034	0.798	0.066	0.872	0.073	0.934	0.040	0.854	0.050
S ₄	0.868	0.102	0.919	0.038	0.788	0.061	0.867	0.076	0.927	0.043	0.847	0.049
(S₁ + S₂)/2	0.874	0.099	0.929	0.033	0.799	0.076	0.873	0.074	0.937	0.038	0.855	0.054

Quantitative Evaluation. Comparison of the proposed LHRNet and 11 state-of-the-art salient object detection methods is performed on six datasets. The comparison results are shown in Table 2 and Figure 6. Table 2 illustrates the performance of different methods under the metrics of maximum F-measure and MAE. Table 2 demonstrates that LHRNet performs favorably against state-of-the-art approaches in terms of maximum F-measure and MAE. Figure 6 lists the PR curves of different approaches on six datasets. It can be observed that the PR curves of LHRNet perform better than other methods on six datasets. Both of Table 2 and Figure 6 demonstrate the effectiveness of the LHRNet.

Table 2

The performance of different methods under the metrics of maximum F-measure and MAE. The top results are highlighted in red, green, and blue, respectively. It can be seen that the LHRNet performs favorably against 11 state-of-the-art approaches in terms of maximum F-measure and MAE on six datasets

Methods	SOD		HKU-IS		DUT-OMRON		PASCAL-S		ECSSD		DUTS-TEST
	F _max	MAE	F _max	MAE	F _max	MAE	F _max	MAE	F _max	MAE	F _max	MAE
Ours	0.874	0.099	0.929	0.033	0.799	0.076	0.873	0.074	0.937	0.038	0.855	0.054
PiCANet	0.855	0.108	0.921	0.042	0.794	0.068	0.880	0.088	0.931	0.047	0.851	0.054
GBMPM	0.851	0.106	0.920	0.038	0.774	0.063	0.862	0.076	0.928	0.044	0.850	0.049
Amulet	0.808	0.145	0.896	0.052	0.743	0.098	0.858	0.103	0.915	0.059	0.778	0.085
DCL	0.825	0.198	0.885	0.137	0.739	0.157	0.823	0.189	0.901	0.075	0.782	0.150
DHS	0.827	0.133	0.902	0.054	0.758	0.072	0.841	0.111	0.907	0.060	0.829	0.065
DSS	0.846	0.126	0.911	0.040	0.771	0.066	0.846	0.112	0.916	0.053	0.825	0.057
ELD	0.760	0.154	0.839	0.074	0.720	0.088	0.773	0.123	0.867	0.079	0.738	0.093
NLDF	0.837	0.123	0.902	0.048	0.753	0.080	0.845	0.112	0.905	0.063	0.812	0.066
RFCN	0.807	0.166	0.898	0.080	0.738	0.095	0.850	0.132	0.898	0.095	0.783	0.090
SRM	0.845	0.132	0.906	0.046	0.769	0.069	0.847	0.085	0.917	0.054	0.827	0.059
UCF	0.803	0.169	0.886	0.074	0.735	0.132	0.846	0.128	0.911	0.078	0.771	0.117

Fig.6

The PR curves of the LHRNet perform better than other 11 state-of-the-art methods on six datasets.

Qualitative Evaluation. Figure 7 lists some saliency maps generated by the proposed LHRNet as well as other 11 state-of-the-art algorithms. These images are also selected from six datasets for testing. It can be seen that LHRNet can accurately detect salient objects. For example, the first picture contains some empty wine bottles that are very similar to background. People need to observe carefully to distinguish the wine bottles. The corresponding saliency maps produced by almost all state-of-the-art methods are very vague, and the segmented salient objects are fragmented. But LHRNet can perfectly separate these wine bottles from background. Moreover, it can be seen that basically all the state-of-the-art methods mentioned above can not detect the gazebo in the fourth picture and the swimming pool in the fifth picture, but LHRNet can do this. For objects with various scales and shapes, LHRNet can highlight the entire objects with well-defined boundaries. Multi-target segmentation in complex scenes such as the second, third and sixth pictures is a difficult point of current research. The intermediate information flow structure of LHRNet can well integrate high-level semantic features and low-level fine-grained details, which are beneficial to identify multiple salient targets in complex scenes while preserving relatively clear boundaries. In addition, the final series of nearest neighbor classifiers can further optimize the preceding coarse prediction due to non-parametric decision mechanism and global contrast. Therefore, the people masked as salient objects in ground truth are correctly segmented by LHRNet, without missed detection and blurred boundaries compared to previous methods.

Fig.7

Qualitative comparison of the LHRNet and state-of-the-art algorithms.

5 Conclusion

In this paper, a Lateral Hierarchically Refining Network (LHRNet) is put forward for accurate salient object detection. Apart from taking full advantages of abundant multi-level features, LHRNet can avoid the problems caused by adopting convolutional layers as classifiers after extracting features to some extent, such as insufficient generalization ability due to fixed parameters of convolutional kernels at the end of training, the limited receptive field and the lack of spatial correlation. Comprehensive experiments demonstrate that the proposed LHRNet performs favorably against state-of-the-art approaches under different evaluation metrics on six datasets.

Footnotes

Acknowledgments

This research was supported by National Key R&D Program of China (no. 2017YFC0806000) and National Natural Science Foundation of China (Grant Nos. 11627802, 51678249).

References

Wang

, Lu

and Ruan

, Deep networks for saliency detection via local estimation and global search, IEEE Conference on Computer Vision and Pattern Recognition (2015), 3183–3192.

and Yu

, Visual saliency based on multiscale deep features, IEEE Conference on Computer Vision and Pattern Recognition (2015), 5455–5463.

Zhao

, Ouyang

and Li

, Saliency detection by multicontext deep learning, IEEE Conference on Computer Vision and Pattern Recognition (2015), 1265–1274.

Lee

, Tai

W.Y.

and Kim

, Deep saliency with encoded low level distance map and high level features, IEEE Conference on Computer Vision and Pattern Recognition (2016), 660–668.

Liu

and Han

, Dhsnet: Deep hierarchical saliency network for salient object detection, IEEE Conference on Computer Vision and Pattern Recognition (2016), 678–686.

and Yu

, Deep contrast learning for salient object detection, IEEE Conference on Computer Vision and Pattern Recognition (2016), 478–487.

Wang

, Wang

and Lu

, Saliency detection with recurrent fully convolutional networks, European Conference on Computer Vision, Springer, Cham (2016), 825–841.

Zhang

, Wang

and Lu

, Amulet: Aggregating multi-level convolutional features for salient object detection, IEEE International Conference on Computer Vision (2017), 202–211.

Zhang

, Dai

and Lu

, A Bi-Directional Message Passing Model for Salient Object Detection, IEEE Conference on Computer Vision and Pattern Recognition (2018), 1741–1750.

10.

and Koltun

, Efficient inference in fully connected crfs with gaussian edge potentials, Advances in Neural Information Processing Systems (2011), 109–117.

11.

Luo

, Mishra

K.A.

and Achkar

, Non-local Deep Features for Salient Object Detection, CVPR 2(6) (2017), 7.

12.

Zeng

, Lu

and Zhang

, Learning to Promote Saliency Detectors, IEEE Conference on Computer Vision and Pattern Recognition (2018), 1644–1653.

13.

Hou

, Cheng

M.M.

and Hu

, Deeply supervised salient object detection with short connections, IEEE Conference on Computer Vision and Pattern Recognition (2017), 5300–5309.

14.

Liu

, Han

and Yang

H.M.

, PiCANet: Learning Pixel-wise Contextual Attention for Saliency Detection, IEEE Conference on Computer Vision and Pattern Recognition (2018), 3089–3098.

15.

Wang

, Borji

and Zhang

, A stagewise refinement model for detecting salient objects in images, IEEE International Conference on Computer Vision (2017), 4019–4028.

16.

Zhang

, Wang

and Lu

, Learning uncertain convolutional features for accurate saliency detection, ICCV) (2017), 212–221.

17.

L.H.

, Qian

, Lian

, Zheng

W.N.

and Zhou

Y.F.

, Study on semantic image segmentation based on convolutional neural network, Journal of Intelligent & Fuzzy Systems 33(6) (2017), 3397–3404.

18.

Guo

and Zhang

, A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression, IEEE Trans Image Processing 19(1) (2010), 185–198.

19.

Zaki

and Abulwafa

M.M.

, The use of invariant features for object recognition from a single image, Journal of Intelligent & Fuzzy Systems 12(2) (2002), 79–95.

20.

Wei

, Liang

and Chen

, Stc: A simple to complex framework for weakly-supervised semantic segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence 39(11) (2017), 2314–2320.

21.

Maia

J.E.B.

, Barreto

G.A.

and Coelho

A.L.

, Visual object tracking by an evolutionary self-organizing neural network, Journal of Intelligent & Fuzzy Systems 22(2, 3) (2011), 69–81.

22.

Siagian

and Itti

, Rapid biologically-inspired scene classification using features shared with visual attention, IEEE Transactions on Pattern Analysis and Machine Intelligence 29(2) (2007), 300–312.

23.

Zhu

, Liang

and Wei

, Saliency optimization from robust background detection, IEEE conference on computer vision and pattern recognition (2014), 2814–2821.

24.

Yang

, Zhang

and Lu

, Saliency detection via graphbased manifold ranking, IEEE conference on computer vision and pattern recognition (2013), 3166–3173.

25.

Cheng

M.M.

, Mitra

J.N.

and Huang

, Global contrast based salient region detection, IEEE Transactions on Pattern Analysis and Machine Intelligence 37(3) (2015), 569–582.

26.

Itti

, Koch

and Niebur

, A model of saliency-based visual attention for rapid scene analysis, IEEE Transactions on pattern analysis and machine intelligence 20(11) (1998), 1254–1259.

27.

Wang

, Zheng

and Piramuthu

, Grab: Visual saliency via novel graph model and background priors, IEEE Conference on Computer Vision and Pattern Recognition (2016), 535–543.

28.

Long

, Shelhamer

and Darrell

, Fully convolutional networks for semantic segmentation, IEEE conference on computer vision and pattern recognition (2015), 3431–3440.

29.

Lin

Y.T.

, Dollar

and Girshick

B.R.

, Feature Pyramid Networks for Object Detection, CVPR 1(2) (2017), 4.

30.

and Koltun

, Efficient inference in fully connected crfs with gaussian edge potentials, Advances in neural information processing systems (2011), 109–117.

31.

Szegedy

, Liu

and Jia

, Going deeper with convolutions, IEEE conference on computer vision and pattern recognition (2015), 1–9.

32.

Szegedy

, Vanhoucke

and Ioffe

, Rethinking the inception architecture for computer vision, IEEE conference on computer vision and pattern recognition (2016), 2818–2826.

33.

Szegedy

, Ioffe

and Vanhoucke

, Inception-v4, inceptionresnet and the impact of residual connections on learning, AAAI 4 (2017), 12.

34.

and Koltun

, Multi-scale context aggregation by dilated convolutions, arXiv:1511.07122, 2015.

35.

Cai

, Huang

and Zeng

, MEnet: A Metric Expression Network for Salient Object Segmentation, arXiv:1805.05638, 2018.

36.

Simonyan

and Zisserman

, Very deep convolutional networks for large-scale image recognition, arXiv:1409.1556, 2014.