Object segmentation using FCNs trained on synthetic images

1 Introduction

Segmentation techniques could facilitate a large number of applications such as object detection, auto driving, and scene understanding. In the past few decades, traditional approaches such as Active Contour Models [26], Watersheds [23], GrabCut [6] have achieved substantial improvements on segmentation tasks. Recently, Convolutional Neural Networks(CNNs) [37] has surpassed traditional segmentation approaches in terms of accuracy on segmentation tasks, and they are trained more easily in comparison with traditional methods that relied on hand-crafted features. However, deep learning approaches require lots of annotation images and their performance has a close relationship with the amount of training images. All benchmarks such as PASCAL VOC [25], SDB [4], Microsoft COCO [35], Cityscapes [24] require community to annotate based on handwork which is both a waste of time and costly. The problem of data unavailability will become more serious when some specific objects are segmented since training images are hard to collect. There are several approaches in reducing dependency on pixel-level image labeling. The main methods deploy semi-supervised models, which learns segmentation model from existing manual annotations such as bounding boxes or image tags [9]. D lin et al. [10] exploited scribbles, one on each object to learn segmentation models. [1] needed a point on the specific object and then trained to obtain segmentation results. While these semi-supervised approches are not as good as fully supervised methods and even not competitive with traditional segmentation methods yet. Other works have several attempts at taking advantage of easier-to-obtain image bounding boxes to populate objects with the same dye. According to [19], given a ground-truth bounding box, this method uses polygon-RNN to outline the edge of the object through some clicks on this object and further gets the instance segmentation. [6] utilized existing bounding boxes to obtain labels with an iterative GrabCut method. Though segmentation results have improved a lot, they cannot be used as ground-truth because of low accuracy. In this paper, one goal is to obtain the training data and labeling images faster and more accuracy. Another goal is to provide a method to segment object automatically and precisely.

Our method can synthetize images and obtain ground-truth automatically without manual annotations. 3D models are used to complete this task. It is not the first time that researcheres have generated synthetic training data from 3D models. [12] synthetized training data with 3D models to recognize human actions. In [14], authors proposed a pipeline that synthesized RGB-D data of indoor scenes to train their models. But in human eyes, this is the first time to synthetize segmentation data using 3D models. This system firstly scans the object or takes photos in different viewpoint with the object and exploits three dimensional reconstruction technology to acquire a 3D model, and then projects the 3D model with different angels and scales to 2D images with diverse backgrounds to obtain synthetic training data. During this process, the location of the projection in this picture is easily to be acquired, which means the ground-truth labels would be acquired automatically. Though these synthetic images are not realistic enough, the segmentation model trained on these synthetic pictures has strong generalization ability on real images. Synthetic images are fed into existed segmentation network FCN-8s [13] and DeepLab-CRF-LargeFOV [20] and trained models finally obtain relatively accurate segmentation result on real images, which proves that it is credible to train segmentation networks using synthetic dataset. In order to learn more details of the images, a improved network is proposed by us and it reaches 81.7% mIOU on test set(real image), which exceeds the results of DeepLab-CRF-LargeFOV.

Our main contributions are as follows:

(1) A new method is proposed by using 3D models to synthetize segmentation training dataset.

(2) A improved network is proposed that combines existing Fully Convolutional Networks(FCNs), dilated convolution and Conditional Random Filed(CRF) and achieves automatic object segmentation with a remarkable degree of accuracy in real scenes.

2 Related work

In the past few decades, segmentation systems mainly relied on hand-crafted features such as Random Forests [16], Boosting [15], or Support Vector Machines [3]. These systems have achieved substantial improvements but the segmentation result has always limited by expressive power of the features. In recent years, with the development of the computing ability of GPUs, CNNs have made breakthroughs in image classification, object detection, and segmentation. Currently, the Fully Convolutional Networks(FCNs) proposed by Long et al. [13] is the most successful deep learning techniques for segmentation tasks and many modified networks are stemed from it. Since this object segmentation task can be resolved by deep learning methods, a central questions is how to obtain large annotated image datasets. Some approaches such as using annotation tools, semi-automatic annotation methods and synthetic data methods are proposed to solve the problem of data deficiency.

Annotation tools. Torralba et al. proposed a Web-Based Tool for image annotation, LableMe [2]. Using this annotation tool, authors have collected a large dataset which extends object categories and contains multiple instances through clicking along the boundary of the desired object. This system produces a polygon around an object and helps more for annotators’ work but manual marking is not so accurate. In [18], the system extended super-pixels methods to develop a labeling tool that labels clothes in images by performing annotation. This makes labeling garment items more efficient but inherently relies on the super-pixels size and the accuracy rate of super-pixels methods.

Semi-automatic annotation methods. [6] utilized existing bounding box or interaction with users to obtain labels with an iterative GrabCut method. This approach performs a color data model and segments by iterative energy minimization that makes pixel-wise prediction with foreground or background models using EM. Inspired by this view, DeepCut [27] extended GrabCut to deep learning by learning a neural network classifier from weak annotations. It formulate this question as an energy minimisation problem, solving over a densely-connected CRF and update the training targets iteratively to get segmentation results. Recently, Polygon-RNN proposed by Castrejon [19] achieved segmentation result reaching the typical agreement among manual annotators. This method treats object segmentation problem as a polygon prediction task instead of a pixel-labeling problem as most researchers consider, the approach seeks out vertices of the polygon outlining the object and finds a cycle that links the contours into a closed region. These works in segmentation labeling tasks have shown impressive results, however they require a great deal of effort by manual annotations actually. Apart from that, due to its imprecision, the segmentation results cannot be used as ground truth. But our 3D reconstruction method can synthetize images and obtain ground-truth automatically without manual annotations.

Synthetic data methods. There have been several approaches use 3D models to synthetize training data. But our method is the first time to synthetize segmentation training data using 3D models. [12] presented to use 3D point clouds of human models to obtain synthetic human pose images in order to learn a human pose representation model. The main contribution of this paper is that it learns a view-invariant human pose model from synthetic depth images that is expensive to label. Rajpura et al. [31] applied 3D models to generate detection training datasets for detecting packaged food products clustered in refrigerator scenes. Inspire by the view, our method would take advantage of 3D model to synthetize segmentation images. The biggest distinguish between our method and these approches is that the synthetic results are different. We generate segmentation images and ground truth labels, while they are not. The SYNTHIA [11] dataset is a collection of synthetic images for semantic segmentation, but it is generated by a virtual world generator and it is used only for urban scenes. While our method almost can be used for synthetizing segmentation images and ground truth of all objects.

3 Generating synthetic training data

A pipeline(See Fig. 1) for synthesizing training datasets is proposed. For a specific object, 3D reconstruction techniques are applied to acquire a 3D model and then a 3D model is projected to the image plane. As proposed by [34], an image can be synthesized by foreground and background. Our method is able to synthesize a image by merging foreground which is provided by rendered datasets with suitable background. Moreover, in order to promote the degree of realism of synthetic images, shading is drawn to synthetic images. For more details, each step is given as below and some synthetic images are shown in Fig. 2.

OPEN IN VIEWER

Fig.1

A proposed pipeline for synthesizing segmentation train images and ground truth. (a) For a given object, this system firstly use 3D reconstruction to acquire a 3D model. (b) Then, projecting the 3D model to 2D plane to obtain images and merge 2D images with various backgrounds. (c) Synthetic images and ground-truth can be acquired automatically.

OPEN IN VIEWER

Fig.2

Examples images of synthetic images and ground truth generated by our method.

This section talk about our network which is used to train on our synthetic images. This proposed network is based on VGG-16 [17], which won ILSVRC 2014 competition with GoogLeNet [8]. The Fully Convolutional Networks(FCNs) [13] are employed to solve the segmentation problem and the convolution operations have already been placed by atrous convolution. Our model derives from DeepLab-CRF-LarFOV which is stems from FCNs but further efforts are made to improve the segmentation results by adding Conditional Random Field layer to the end of network rather than a post-processing stage and sets greater dilated rates. More details are shown in Fig. 3.

OPEN IN VIEWER

Fig.3

An illustration of our pixel-wise prediction network architecture. The last few convolution layers and fully connected layers are placed by dilated convolution layer. Subsequently, an efficient up-sampling filtering acts on the feature map and followed by a dense CRF layer to obtain dense prediction results.

Long el al. [13] proposed the most successful method for pixel-level segmentation tasks, Fully Convolution Networks(FCNs), which is the forerunner in CNNs, whereafter, many efforts based on this system have been made to improve its accuracy. The biggest contribution of this system is that it replaces convolutional layer with fully connected layer of traditional classification networks and trains end-to-end to solve segmentation tasks.

However, there are some challenges in FCNs. The biggest issue is reduced features caused by applying multiple pooling operations to down-sampling the images. In order to get an output image which is the same size with the input picture rather than a probability value in classification networks for a picture, FCNs exploit deconvolution layer to up-sampling the last convolution layer’s feature map. Thus, some detailed information has dropped during down-sampling and up-sampling stage and segmentation results are rough in FCNs.

To further improve the segmentation accuracy and capture long-range information, atrous convolution and Conditional Random Field (CRF) are integrated into our network.

4.1 Atrous convolution

One challenge for segmentation using CNNs is the problem of reducing spatial resolution of feature maps, causing the dense prediction results is coarse. Due to the limitation of classification nets such as VGG-16 net, it frequently uses pooling and in all a stride of 32 in 5 pooling layers and results in this problem. The representative solutions deploy deconvolution layers such as in [13] or decoder variants in [36], however, these methods require additional memory and higher computational power.

Recently, annother effective approach is to employ atrous convolution, also named dilated convolution, which originates from kronecker layer [33] proposed by S.zhou el al. Deeplab [20–22] successfully deploy dilated convolution to do dense prediction problems and achieves the-state-of-art. Using atrous convolution can not only obtain wider receptive fields without extra computation cost and excessive reduction in feature responses but also adjust to arbitrary resolution at any layer of CNNs. For two-dimensioanl features, the output of dilated convolutions y can be expressed by a weight w [k] and input value x [i] using equation: y [ i ] = ∑ k = 1 K x [ i + r · k ] w [ k ](5) where r known as dilated rates and controls upsampling factor, which is equivalent to fill adjacent element with r - 1 zeros in filters before doing traditional convolution. That is to say, filter are filled with zeros. Specially, dilated convolution is equivalent to usual convolution when r = 1, as illustrated in Fig. 4.

OPEN IN VIEWER

Fig.4

3 × 3 filters(orange) while using multiple different dilated rates, from left to right r = 1, 2, 3. The red rectangel regions are corresponding to the receptive fields, the left ones with 3 × 3 recptive fileds, the middle has a 7 × 7 recptive fileds, and the right one with 11 × 11 receptive fileds.

In this section, common setting is presented for all the experiment and experimental steps and the parameters are elaborated. Synthetic images are used as training set and real images of this object are viewed as test set. Existed network architectures, including FCN-8s, DeepLab-CRF-LargeFOV are firstly used to demonstrate the effectiveness of this synthetic datasets. The experimental results demonstrate that our synthetic images can be used to train and obtain roughly accurate segmentation results in real scenes. Furthermore, To obtain finer segmentation results on real images, a improved network based on DeepLab-CRF-LargeFOV is proposed by us.

Evaluation metic: The performance is measured by Mean Intersection over Union(MIOU) alrogithm [25] to evaluate our system. Experiment environment: All experiments are conducted by Caffe Framework [38] and use one NVIDIA GTX 1080 GPU. Stochastic Gradient Descent(SGD) algorithm is applied to minimize the softmax function over images in all networks.

Dataset: 5 objects are reconstructed and more different colors and shapes of 3D models about these objects are acquired. Using synthetic images method proposed in Section 1, synthetic training images are able to acquired. To improve the predictability of these models, data augmentation methods are applied by random rotation, image scaling(from 350 pixel to 600 pixel), contrast transformation and randomly top-bottom or left-right flip in synthetic training images. At last, about 5000 synthetic training images for each object are generated. As for test set, about 200 images per class in various backgrounds are collected by us and ground-truth lables are acquired by using LabelMe [2]. Additionally, there is some difference between objects in train and testdatasets.

Extra supervised with VOC 2012: Initial weights of VGG-16 model is utilized to pretrained and all modles are firstly trained with PASCAL VOC 2012 dataset [25]. The initial learning rate is set to 0.001 and learning policy is "step", and a total of 20 thousand iterations of training is conducted and trained models yield a 2.1% improvement on testset.

5.2 Results

In order to verify the validity of synthetic data, different number of synthetic training images are fed into FCN-8s and DeepLab-CRF-LargFOV networks and test on real images. It was not until the loss value is convergence and stable that iterative training stopped. Fig. 5 shows that with the increase of training images, segmentation results on test set are better and better. That indicates our synthetic images are able to used for training. When the number of training images for per object reaches 2000, the performance on test set almost reaches the top. A smooth curve is observed when training images start from 2000 to 5000.

OPEN IN VIEWER

Fig.5

Performance on test set when providing different numbers of training images.

Since the performance almost reaches the top when the number of training images reaches 2000, the same number of synthetic images of each object are randomly choosen for training. Segmentation results on real images are shown in Fig. 7 (Compared with FCNs, Deeplab and Ours). Experiemtal results indicate that our models has strong generalization ability on test dataset. The modified network obtain finer segementation results compared to DeepLab (See Fig. 6). In the end, the improved model achieves the performance of 81.7% on test set which outperforms deeplab-CRF-LargeFOV about 2.5%, as shown in Table 1.

OPEN IN VIEWER

Fig.6

An example image of DeepLab-CRF-LargeFOV segmentation result(left) and ours(right).

OPEN IN VIEWER

Fig.7

Visualization results on test set(real images). Input images, ground-truth, FCN-8s results, Deeplab-CRF-LargeFOV results and ours.

OPEN IN VIEWER

Table 1

Method	mIOU
FCN-8s [13]	70.8%
DeepLab-CRF-LargeFOV [20]	79.2%
Ours	81.7%

Experimental results indicate that our system can generate images and obtain ground-truth images automatically by 3D models. Training improved networks with synthetic images by this approach, a relatively accurate segmentation result would be obtained on real images. Meanwhile, the future work is to improve the reality of synthetic images and further improve the segmentation result using fully convolutional networks. Additionally, reducing the consume memory and improve the efficiency of computation are our goals.