Recognition method of outdoor design scene based on support vector machine and feature fusion

Abstract

Based on the impact of epidemic prevention and control, the floating population supervision department classifies and controls the floating population by industry. There are many personnel management and control points. When the computer-aided management system is used, the outdoor environment is complex and the data interference is large. Therefore, the recognition accuracy of outdoor scenery is required to be higher. In this paper, a convolutional neural network with adaptive weights is proposed. In this method, the feature fusion strategy is combined with the network, and the optimal feature weight is obtained by training the network. In addition, this paper uses multiple two classifiers instead of multiple classifiers to achieve accurate target classification. Experiments show that the method proposed in this paper has excellent performance in the detection of similar objects. The strategy of replacing multi classification network with multi classification network improves the accuracy and recall of target detection in known environment.

Keywords

Adaptive weight convolution neural network fusion strategy outdoor environment recognition

1 Introduction

With the development of target detection methods, target recognition accuracy and recall rate have been improved. The rapid development of graphics processing unit (GPU) makes the detection more real-time [1, 2]. However, the upgrading of network structure has limited effect on the improvement of the recall rate, especially target overlapping in the complex urban environment, the problem of low recall rate cannot be solved fundamentally, and the target detection method still has a lot of room for improvement. For example, in many scenarios, people want to be able to detect all the targets, that is, the recall rate is close to 100%, but the current detection method is obviously difficult to achieve [3]. For example, in terms of streetscape recognition, traffic safety and automatic driving, the target detection method has been applied to predict the potential danger. The system obtains the information around the vehicle through cameras and sensors, detects the targets around the vehicle, and makes lane departure warning and collision warning to the driver in time. However, this method can only play an auxiliary role in early warning, because the rate of missing detection is still very high, especially in the complex road section, there is still a certain gap between the target detection distance and the human automatic driving [4, 5]. In the aspect of outdoor design scene recognition, people can easily find and locate the outdoor location through deep learning algorithm, and can also identify the types of outdoor design scene with high accuracy. However, in a complex outdoor environment, even if there is a slight deviation in the detection, it will have a great impact on the design and is also very irresponsible to the designer. In addition, the main focus of the target detection method is the target recognition, and the target location is only an auxiliary step [6]. Using the target detection method alone can only obtain the approximate position of the target in the image. If it is applied to real life, it is also very important to obtain the three-dimensional coordinates of the target in the world coordinate system [7, 8]. Based on the depth learning method, this paper introduces a binocular vision system to calculate the position coordinates of the target, and further improves the accuracy and recall rate of the target detection in the outdoor design scene recognition through prior knowledge as a constraint condition.

2 SENET network

In recent years, convolutional neural networks have made great breakthroughs in many fields. However, it is quite difficult to learn a network with very strong classification ability. The difficulties come from many aspects. With the deepening of research on neural networks by researchers, more and more methods have been proposed to improve various performance indexes of the network. One of them is the Inception network, which adds multi-scale information into the network structure and combines it with the characteristics of multiple receptive fields to improve the identification ability of the network. Inside outside network is to analyze spatial context information and add attention mechanism in spatial dimension. The squeeze and exception network (hereinafter referred to as SENET) proposed by Hu Jie and others considers the relationship between feature channels [9], and the accuracy of network classification has been greatly improved.

In the structure of SENET, squeeze and exception are two very critical operations, and the network is named after them. The purpose of SENET network is to hope that more important features can have greater weight, while secondary features reduce their weight. The network does not introduce a new spatial dimension to fuse feature channels, but adopts a brand-new strategy of recalibrating the weight of feature maps [10, 11]. The network gradually obtains the importance of each feature map through continuous iteration and training, and then uses the obtained importance (i.e., feature map weight) to promote useful features and suppress features that are less useful for the current task.

Figure 1 is a schematic diagram of the SENET network module. Suppose the input is x and the number of characteristic channels is c₁. After a series of convolution, pooling and other transformations, the number of characteristic channels becomes c₂. Compared with the traditional convolution neural network, the next three steps are used to re calibrate the features. The first is the compression operation, which turns each two-dimensional feature channel into a real number, which has a global receptive field to some extent, and the output dimension matches the number of input feature channels. It represents the global distribution of the response on the feature channel, and enables the layer near the input to obtain the global receptive field, which is very useful in many tasks. Next is the excitation operation, which is similar to the gate mechanism in the cyclic neural network. Weight is generated for each feature channel by parameter ω. Finally, it is an operation to recalculate the weight. The weight of the output after excitation is regarded as the importance of each feature channel after feature selection [12]. Finally, the weight of the feature map is recalibrated by weighting the previous features one by one through multiplication.

Fig. 1

SENET network module.

The specific implementation uses the global average pooling as the compression operation, and the global average pooling is shown in Fig. 2. The left figure is a conventional convolution neural network. First, all the feature maps are connected into one-dimensional vectors. Then, through the full connection layer and softmax classification, the right figure is the global average pooling. The feature map of the last layer is pooled into a mean value of the whole map to form a feature point. These feature points constitute the final feature vector, which can greatly reduce the number of parameters.

Fig. 2

Convolution neural network and global mean pooling.

After the global average pooling layer, the correlation between the two fully connected layers is modeled, and the weight of the same number of output and input features is output. Firstly, the feature dimension is reduced to 1 / 16 of the input, and then it is promoted to the same level through a full connection layer after being activated by Relu. Compared with using only one full connection layer, this method has the following advantages: The network has more non-linearity, which can better fit the complex correlation between channels; in addition, the reduction of dimension can greatly reduce the computation of the network. Then the weight is reduced to 0 ∼ 1 by using the sigmoid function. Finally, the weight is weighted to each channel’s feature by multiplying each feature graph with the obtained weight coefficient.

Embed SENET into RESNET to be used in this paper. The structure of network module is shown in Fig. 3:

Fig. 3

SE-RESNET network module.

The SE module is connected after the RSENET convolution module. Through global mean pooling, full connection layer and sigmoid function, the weight vector of convolution eigengraph is obtained, and identity mapping is outside the RES net and SENET modules.

3 Feature fusion strategy

Convolution features have strong robustness. In target classification, the feature map generated by the last convolution layer of the network is usually used. Compared with other convolution layers, the feature map of this layer is more abstract and higher level, and the effect of target classification is better. However, the more abstract features are, the more details of the image will be lost. When the convolutional neural network is used to distinguish objects with similar categories, the classification results are sometimes unsatisfactory. As shown in Fig. 4, there are two kinds of bottles to be distinguished. From the overall category, both belong to water bottles, but the specific categories of the two are different.

Fig. 4

Two different types of water bottles.

Using RES net-50 convolution neural network to extract features, the feature map of one of the convolution layers in this picture is shown in Fig. 5.

Fig. 5

Feature map extracted by convolution neural network.

It can be seen that in the deep feature map, there are many feature maps in which the difference between the two water bottles is very small. The success rate of classification is relatively low by using this feature, while the HOG feature is a global gradient feature and is very sensitive to the gradient change of pixels in the image. As shown in Fig. 6, it is a HOG feature map.

Fig. 6

HOG characteristic map.

If we can make use of the advantages of different features separately and fuse them, it will be helpful to improve the success rate of classification.

3.1 Decision-level fusion strategy for classifiers

The flow of the classifier decision-level fusion strategy algorithm is shown in Fig. 7. For the two types of features CNN and HOG extracted from the input image, two SVM classifiers are established to classify the CNN and HOG features respectively, and then the classifier output result is fitted to the posterior probability of the target category by sigmoid function. The posterior probability matrix equation is shown in Equation (1).

Fig. 7

Flow chart of classifier decision-level fusion strategy algorithm.

$\begin{matrix} prob_estimates (k) \\ = (\begin{matrix} p (c_{1} | x_{1}) & p (c_{2} | x_{2}) & . . . & p (c_{m} | x_{1}) \\ p (c_{1} | x_{2}) & p (c_{2} | x_{2}) & . . . & p (c_{m} | x_{2}) \\ . . . & . . . & . . . \\ p (c_{1} | x_{m}) & p (c_{m - 1} | x_{m}) & . . . & p (c_{m} | x_{m}) \end{matrix}), k = 1, 2 \end{matrix}$ (1)

Where p (c_m |x_n) represents the posterior probability that the nth x sample is identified as c_m class, the number of target classes is m, the number of test samples is n, and prob_ estimates (k) represents the posterior probability matrix of the k(k = 1, 2) th classifier. By using decision weight for information fusion, the classification performance is better than that of a single classifier. The decision information after decision level fusion is shown in Equation (2). $p = (c_{j} | x_{i}, E_{1}, . . . E_{k}) = \sum_{k = 1}^{k} w_{k} p (c_{j} | x_{i}, E_{k})$ $j = 1, 2, . . ., m; i = 1, 2, . . ., n; k = 1, 2; K = 2$ (2)

Among them, E_k represents the k-th classifier, p (c_j|x_i, E_k) represents the posterior probability of the recognition of x_i as c_j class under the k-th classifier, wk represents the weight assigned to the k-th classifier, p (c_j| x_i, E₁,..., E_k) is the final weighted probability of the recognition of x_i as c_j class under all classifiers. Finally, the weight is determined by normalization. The correct probability of each classifier is normalized to get the weight of each classifier, that is, the voting right. The weight determination equation is shown in Equation (3). $w_{k} = \frac{A_{k}}{\sum_{k = 1}^{n} A_{k}}, k = 1, 2 . . . n$ (3)

Where, w_k represents the weight given to the k-th classifier, A_k represents the correct recognition rate of the k-th classifier, and n is the total number of classifiers.

In this method, convolution features and HOG features are extracted and classified respectively, the results of the two classifiers are output, and the weights of the classifiers are calculated according to the posterior probability to obtain the final classifier. This method only integrates different classifiers and integrates the information of the two classifications. It belongs to an integrated learning strategy and can improve the success rate of target recognition in some scenarios. However, the unchanged feature extraction strategy means that the method cannot fundamentally improve the accuracy rate.

3.2 Feature map fusion strategy

Figure 8 shows a feature map fusion strategy. The input image enters a convolution neural network for feature extraction, selects feature maps of some layers, extracts HOG features from the feature maps, adds the feature maps of the same channel, and finally sends the feature map vectors to a support vector machine for classification.

Fig. 8

Feature map fusion strategy.

This method accumulates the convolution feature map and the HOG feature map, which is the real feature fusion. But for different objects, different scenes, different features have different gains to the classification accuracy, and the rigid fusion principle is insufficient.

3.3 The feature fusion strategy of convolution network based on adaptive weight

Different features have different sensitivity to different forms of objects. Only using the same model cannot determine the feature weight that makes the classification accuracy optimal. Based on this problem, this section proposes a feature fusion network resnet-s with adaptive weight, and uses multiple more accurate classifiers instead of multiple classifiers. Each two classifier is only responsible for judging whether the object is the target object.

Another advantage of the two classifiers over the multi-classifiers is that the multi-classifiers do not distinguish between similar categories or objects of the same category sufficiently. For example, the features of A and B objects in the scene are very close. Using the multi-classification method, the probability scores of the categories of the two objects will be almost the same. However, the two classifiers can train objects similar to the target object as negative samples on the premise of knowing some prior knowledge (objects prone to misclassification) to improve the classification accuracy. The loss function of multiclassification network is softmax loss, which is the transformation form of cross entropy loss multiclassification. In this paper, the loss function of biclassification network directly selects cross entropy loss, and its deduction process is as follows:

In the two classification problem, the actual sample label is 1 or 0, representing positive and negative samples respectively. The output of the model usually passes through the sigmoid function, making the result become the probability value between [0,1]. This probability reflects the possibility that the target is a positive sample, and the greater the probability is, the greater the possibility is. The equation of sigmoid function is: $g (s) = \frac{1}{1 + e^{- s}}$ (4)

In the equation, s is the output of the previous model. Sigmoid function is characterized by: when s = 0, g (s)=0.5, when s is far greater than 0, g ≈ 1, when s is far less than 0, g ≈ 0. g (s) is the model prediction probability value of cross entropy equation. This probability value represents the probability that the current sample is a positive sample: $y^{'} = p (y = 1 | x)$ (5)

4 Comprehensive experiment

4.1 Resnet-s and resnet-50 comparative experiment

Taking the streetcape as the scene, the network performance experiment is carried out. In the experiment, 10 kinds of objects such as traffic light, stop sign, car, people, motorbike, laptop, water bottle, chair, desk and backpack were selected for the experiment, as shown in Fig. 9.

Fig. 9

Object detected in this paper.

The data set used in this paper is real-world pictures taken in multiple urban environments. The objects to be tested are common objects in urban environment. There are 800 pictures in the training set and 200 pictures in the test set.

The deep learning framework of this project is python. Python is the python version of torch, a deep learning framework developed by the Facebook team. Different from the static calculation chart of tensorflow, Python’s calculation chart is dynamic. It can change the calculation chart in real time according to the calculation needs, and print the intermediate results at any time, which is convenient for network debugging.

Firstly, the model trained on image net by fast RCNN is applied to the scene of reality directly. Image net is a computer vision system recognition project. It is the largest image recognition database in the world. It was established by Stanford computer scientists to simulate human recognition system. There are more than 14 million images in the dataset, and at least one million images provide a bounding box for target detection. In this paper, the Faster Rcnn network model, which has been pre trained on image net, is used to optimize the parameters. The model has a total of 1000 categories of objects. Reduce the detection threshold, select the 20 detection boxes with the highest score, and the detection results are shown in Fig. 10. Most of the objects selected in the detection box are independent of the objects to be detected in this paper. The objects in the box can be intercepted as the background of the multiple classification network.

Fig. 10

Test results of training network.

Then fine tune the network pre trained by Faster Rcnn on image net combined with its own data set. The training data is shown in Fig. 11. The blue box in the figure is the manually marked category box. Ensure that the number of occurrences of each type of object in the data set is greater than 100.

Fig. 11

Manually labeling training data set.

The training parameters of the network mainly include: learning rate ɛ, momentum factor μ, rotation angle α, number of pictures in each iteration, and the setting parameters are shown in Table 1:

Table 1

Training parameter setting

Parameter	Set value
ɛ	0.00001
μ	0.9
α	–40° ∼ +40°
Batchsize	64

Intercept the target in the box of the fine-tuning network as a positive sample of the two classification network, as shown in Fig. 12 is a positive sample of the green water bottle.

Fig. 12

Water bottle positive sample.

For each type of object, 200 positive samples and 200 negative samples are used respectively, and the accuracy of resnet-50 and RSENET-s are used for ten types of samples, as shown in Table 2.

Table 2

Resnet-50 and RSENET-s comparative experiment

	ResNet-50	RSENET-S
Traffic light	96.53	97.67
Stop sign	92.5	87.5
Car	98.75	98.5
People	97.75	98.75
Motorbike	96.67	98.58
Laptop	98.75	98.75
Water bottle	97.91	98.2
Chair	94.32	95.5
Desk	95.73	97.44
Backpack	98.75	97.46
Average accuracy	96.77	96.74

From the data in Table 2, it can be seen that the average recognition accuracy of RSENET-s is basically the same as that of resnet-50, and the average performance of resnet-50 is slightly higher than that of RSENET-s, and the stability is stronger, which proves that RESNET-50 is more universal. However, in the classification of targets with more similar features, the recognition accuracy of RSENET-s is slightly higher (such as distinguishing different types of bottles, water cups, etc.), which is mainly due to the gain of gradient features of the hog on network performance. Using RSENET-s in the precise classification network of this paper has better effect.

The abscissa in the figure is epoch (one epoch for all data sets after training) and the ordinate is the loss of the network. ResNet-S has a slower convergence speed than ResNet-50. The first reason is that there is global mean pooling in ResNet-S, and this module is not easy to converge. Secondly, HOG feature parameters do not participate in training, and there are large differences in feature distribution and convolution features, which will also lead to slow training. When the two networks finally converge, the losses are basically the same.

4.2 Accurate classification network experiment

First of all, we directly use Faster-Rcnn to carry out an accurate classification experiment, and then we use low threshold detection combined with ResNet-S classifier experiment with coco database. The test results are shown in Fig. 14, where the Fig. 14(a) is the original method and the figure below is the improved method. In the improved method, the probability value is the probability output by the feature extraction network network, and only the detection boxes with probability values above 0.25 are retained.

Fig. 13

Network loss reduction chart.

Fig. 14

Accurate classification network experiment map.

In Fig. 14, the green oval box shows the false detection and missed objects in the scene. As can be seen, the missed detection rate in the Fig. 14 (b) is obviously lower than that in the picture above, and the average probability scores are very close. The improved algorithm has some advantages over the traditional algorithm in the recognition of complex texture targets such as knapsack or traffic sign with good background fusion. However, when the two targets have a large overlap area, the two algorithms have the same degree of target repeat marks. In terms of detection time, since the method in this paper adopts multiple binary classification networks instead of multiple classification networks, the detection time is improved compared with Faster-Rcnn method. When the number of detection categories is 80, the detection time consumption is increased by about 20%.

The accuracy rate and recall rate are calculated with a probability threshold of 0.5. Compare the recall rate and recall rate of 50 pictures in the urban environment by using the recall rate of fast RCNN directly and the accurate classification method combined with prior knowledge. The accuracy rate is only considered as the correct detection if the accurate category of the target is accurately detected by the detection result, as shown in Table 3.

Table 3

Comparison between traditional Faster-RCNN and the method in this paper

	Traditional Faster-Rcnn	This method
Accuracy rate	85.25	92.83
Recall rate	73.63	89.47

It can be seen from the data in the table that under the condition of known part of prior knowledge (such as target space position constraint), the method proposed in this paper can greatly improve the accuracy and recall rate of target detection in the known environment. This method has strong practicability.

5 Conclusions

Firstly, this paper introduces the basic principles of SENET network and hog features, compares a variety of feature fusion strategies, and proposes an adaptive weight target classification network RES net-s. Experiments show that RES net-s has excellent performance in target detection with similar categories. In this paper, a strategy is proposed to replace the multi classification network with multiple accurate two classification networks, which greatly improves the accuracy and recall rate of target detection in the known environment.

Footnotes

Acknowledgment

This paper is supported by “the Fundamental Research Funds for the Central Universities”: Research on the visual evaluation of winter landscape in cold cities in Northeastern China—with examples of Shenyang and Harbin (PID: 02040022120002).

References

Steen

K.A.

, Therkildsen

O.R.

, Karstoft

, et al., Audio-based detection and recognition of conflict species in outdoor environments using pattern recognition methods, Applied Engineering in Agriculture 30(1) (2014), 89–96.

Cho

, Lee

, A new assessment method of outdoor tobacco smoke (OTS) exposure, Atmospheric Environment 87(APR.) (2014), 41–46.

Park

, Lee

, Kim

, et al., Automatic detection and recognition of Korean text in outdoor signboard images, Pattern Recognition Letters 31(12) (2010), 1728–1739.

Zhang

, Zhong

, Lin

, et al., Basis image decomposition of outdoor time-lapse videos, The Visual Computer 29(11) (2013), 1197–1210.

Sato

, Kanbara

, Yokoya

, et al., Dense 3-D reconstruction of an outdoor scene by hundreds-baseline stereo using a hand-held video camera, International Journal of Computer Vision 47(1–3) (2001), 0057.

Grabler

, Agrawala

, Li

, et al., Generating photo manipulation tutorials by demonstration, ACM Transactions on Graphics 28(3) (2009), 427–435.

Zhao

Qun

, Synthetic aperture radar automatic target recognition with three strategies of learning and representation, Optical Engineering 39(5) (2000), 1230.

Khan

J.I.

, Komogortsev

O.V.

, A hybrid scheme for perceptual object window design with joint scene analysis and eye-gaze tracking for media encoding based on perceptual attention, Journal of Electronic Imaging 15(2) (2006), 23–28.

Hughes

G.B.

, Algorithms for sensor chip alignment to blind datums, Journal of Electronic Imaging 15(3) (2006), 3–13.

10.

Liu

, Li

, Qi

, et al., Method for unconstrained text detection in natural scene image, Computer Vision 11(7) (2017), 596–604.

11.

Newman

, Liu

E.T.

, Perspective on BRCA1, Breast Disease 10 (1998), 3–10.

12.

Pilkey

D.F.

, Happy conservation laws, in: Neural Stresses, J. Frost, ed., Controlled Press, Georgia, 1995, 332–391.