A brief review and challenges of object detection in optical remote sensing imagery

Abstract

Traditional object detection algorithms and strategies are difficult to meet the requirements of data processing efficiency, performance, speed and intelligence in object detection. Through the study and imitation of the cognitive ability of the brain, deep learning can analyze and process the data features. It has a strong ability of visualization and becomes the mainstream algorithm of current object detection applications. Firstly, we have discussed the developments of traditional object detection methods. Secondly, the frameworks of object detection (e.g. Region-based CNN (R-CNN), Spatial Pyramid Pooling Network (SPP-NET), Fast-RCNN and Faster-RCNN) which combine region proposals and convolutional neural networks (CNNs) are briefly characterized for optical remote sensing applications. You only look once (YOLO) algorithm is the representative of the object detection frameworks (e.g. YOLO and Single Shot MultiBox Detector (SSD)) which transforms the object detection into a regression problem. The limitations of remote sensing images and object detectors have been highlighted and discussed. The feasibility and limitations of these approaches will lead researchers to prudently select appropriate image enhancements. Finally, the problems of object detection algorithms in deep learning are summarized and the future recommendations are also conferred.

Keywords

Object detection optical remote sensing neural networks region proposal deep learning

1. Introduction

Object detection is the prerequisite of advanced visual tasks such as scene content understanding. It has been applied to intelligent video surveillance, content-based image retrieval, robot navigation and enhancement [1, 2, 3, 4, 5, 6]. Compared with object detection in video, static image object detection is more difficult and challenging [7, 8, 9, 10, 11, 12, 13]. The object detection in optical remote sensing imagery is widely used for surveillance, traffic monitoring, agriculture development, disaster planning, geographical referencing and many more. The processing of optical remote sensing images is more challenging due to several limitations such as less number of datasets, low-resolution images, complex environment and so on [14, 15, 16, 17]. So far, traditional object detection methods mainly use Histogram of Oriented Gradient (HOG) and Scale-Invariant Feature Transform (SIFT) for feature extraction. HOG [18, 19, 20] and SIFT [21, 22, 23] features are used to discriminate sliding windows. The main representative methods are Deformable Part Model (DPM) and its extensions [24, 25, 26, 27]. Because sliding windows require a massive computational burden, therefore, object detection methods based on region proposals have come to the fore. At present, the more promising region proposal methods include Selective Search (SS) [28] and Edge-Box [29].

In recent years, a deep convolution network has made breakthroughs in the field of computer vision. A deep convolution network deepens the network hierarchy through weight sharing strategy, which makes the network more analytical. The establishment of large-scale image databases such as ImageNet [30], COCO [31] greatly promotes the development of deep convolution network. Alex-Net [32] convolutional network with seven layers won the championship with absolute advantage in ImageNet image classification contest and its effectiveness has been more and more verified [33, 34, 35, 36]. Subsequently, VGG [37] network, Google Net [38] and residual network [39] push the convolution network to a deeper level which has greatly improved the performance of the network and improved the accuracy of large-scale image classification to a high level.

At the same time, researchers have initiated to seek the expansion of deep convolution network in other fields. In object detection, the region-based convolutional neural network (R-CNN) [40] successfully connects the object detection and deep convolution network and improves the accuracy of object detection to a new level. R-CNN consists of three independent steps: Candidate windows, feature extraction, SVM classification and then window regressions are generated. R-CNN mainly uses the SS method to generate many favorable candidate windows. All candidate windows are sent to the deep network to extract features at one time. Finally, SVM classifier is trained to classify all candidate windows and to perform regression. The detection efficiency is very low in R-CNN because it is divided into three independent processes. Based on this situation, researchers improved R-CNN and proposed Spatial Pyramid Pooling Net (SPP-net) [41] and Fast Region-based Convolutional Neural Network (Fast-RCNN) [42]. Candidate windows are mapped on a certain layer of the network, which greatly improves the detection speed of the model.

Faster region-based convolutional neural network (Faster-RCNN) [43] generates candidate regions using Region Proposal Network (RPN). Classifies and regresses regions using the same structure as Fast-RCNN. RPN and Fast-RCNN share the main depth of the network. Faster-RCNN combines object detection so a unified deep network framework is established. Based on this, the region-based Fully Convolutional Network (RFCN) [44] is further improved. It is found that the network layer pooled by Region of Interest (ROI) is no longer shift-invariant, and the number of ROI pooled layers will directly affect the detection efficiency. Therefore, RFCN designs a location-sensitive ROI pooling layer, which directly discriminates the results after pooling and greatly improves the detection efficiency. The proposals of YOLO (You Only Look Once) [45] and SSD (Single Shot Multibox Detector) [46] aim at improving the detection efficiency of object detection and try to achieve real-time detection for object detection applications. In particular, SSD can enhance the detection accuracy and detection efficiency of object detection.

Compared with traditional object detection methods, the method based on a deep network has obvious advantages in accordance with accuracy. Firstly, the neural network is a network structure with a self-learning function to simulate the human brain. Secondly, forward propagation of deep network can be regarded as a process of abstracting objects continuously. The upper layer of deep neural network (near the output layer) records more perceptual information of objects [34]. Thirdly, the structure of a deep network can fit large-scale training samples well. The major difficulty of object detection lies in the variety of target objects, which have different colors, sizes and shapes in different scenarios. The deep networks have a large number of parameters which makes them have strong learning ability. A large number of training samples are helpful to activate deep network neurons. Based on the above two points, we can see the inevitability that the deep networks achieve excellent results in the field of object detection and the accuracy of object detection is much higher than that of conventional methods.

The main motive of this contribution is to consistently synopsize the object detection frameworks which are based on a deep convolution network in recent years. At the same time, we describe the performance of regression-based object detection methods. Moreover, we assess the advantages, disadvantages and limitations of distinctive methods in computer vision. We also describe some basic preprocessing approaches which could enhance the accuracy or computational speed of detectors. Finally, we describe the current limitations and future challenges of DCNN in developing context for object detection in optical remote sensing imagery.

The rest of the paper is organized as follows: The general framework of object detection, limitations of optical remote sensing images and object detectors, along with challenges have been described in Section 2. Section 3 presents a review of different approaches for object detection based on deep learning and regression. Finally, Section 4 concludes this paper.

2. General framework of object detection

Object detection becomes challenging due to the deviations of illumination, view angle and partial occlusion during the detection process. Using the same standard database can effectively compare the detection performance of various algorithms. At present, the main databases used are PASCAL VOC [47], Caltech and ImageNet [48] database. Pre-processing is utilized to process the brightness, color and size of the original image, so as to get the accurate features of the object to be detected, so as to reduce the complexity of the algorithm in the subsequent processing, and effectively improve the efficiency. Common image preprocessing operations include image enhancement, binarization, grayscale, etc.

In the feature extraction process, the required features are expressed as much as possible in the numerical form so it becomes convenient to filter out the interference features and obtain the actual features of the image. The main problem of feature generation is how to extract features, which affects the accuracy and timeliness of the detection algorithm. Building a model is a key step of the object detection system, which extracts the similarities of the same category of objects, distinguishes different types of objects effectively, and efficiently processes, stores and utilizes the extracted features and the spatial structure between them. According to the statistical structure, the model construction can be divided into Generative Model and Discriminative Model [49].

When the characteristics and models of the object to be detected are obtained, the training of the model is carried out. The important basis of object detection is to obtain the model parameters of the object by learning and training the specified training image set. This training can be divided into unsupervised training, supervised training and semi-supervised training according to the different training methods. According to the different classifiers, it can be divided into Neural Networks (NNs), KNN, Support Vector Machine (SVM) and Random Forest (RF).

The general process of object detection is to match the model obtained from the training of the sample set with the model extracted from the detected image to obtain the category and orientation information of the object in the image to be detected. Among them, object localization directly affects the final performance evaluation of the detection system, which is the key step. Current object searching methods mainly include image segmentation based search method and sliding window based search method. With the development of computer and communication technology, it is easier to obtain a large amount of monitoring data than before. Now a difficult problem to be solved is how to improve the efficiency of obtaining useful information while having a large amount of data. Taking object detection and recognition based on image and video data as an example, the traditional manual and human-computer interaction methods cannot adapt to the current situation of the “explosion” of data information, especially the recognition efficiency. The generalized steps of the object detection system for optical remote sensing imagery have shown in Fig. 1.

Figure 1.

General implementation steps of the object detection system for optical remote sensing imagery.

2.1 Limitations and improvements

In general, target detection algorithms based on deep learning can be divided into two categories: 1) R-CNN algorithm series based on region nomination; 2) YOLO and SSD algorithm series without region nomination. R-CNN target detection algorithm framework and YOLO target detection algorithm framework provide two basic frameworks for our research on target detection. On this basis, researchers have proposed some methods to improve the performance of target detection: 1) hard negative mining [50]; 2) multi-layer feature fusion [51]; 3) using context information [52, 53]. In addition, a variety of open-source deep learning frameworks are emerging including Theano, Keras, Caffe, Tensorflow and so on. These open-source frameworks accelerate the development of deep learning and provide an efficient learning tool for object detection. It also helps to design a more reasonable network structure, improve the detection efficiency of the recurrent neural network, and achieve multi-scale and multi-category object detection. In general, object detection based on deep learning is still a challenging subject, which is mainly playing a role in the following two aspects: robustness and computational complexity [54, 55, 56]. At present, with the arrival of the era of big data and artificial intelligence, the subject provides new opportunities and challenges, so it is worth further research. The feasibility of methods used in this discussion can be observed in Fig. 6. Among all methods, Faster-RCNN is widely used in object detection due to its simplified structure and robustness.

2.1.1 Limitations of remote sensing images

The optical remote sensing images have several drawbacks with respect to shapes, sizes, occlusions, resolutions, pixels and so on, of objects. The detectors generally utilize the region of interests (ROIs) which are not appropriate frequently. There is a need to improve the selection of required training objects. The exact number of pixels that hold the characteristics of an object should be selected for training. By considering the resolution, if a detector is trained on high-resolution images then it is difficult to get optimal accuracy on low-resolution images and it will generate more false alarms. Nowadays, the researchers are delivering the solutions for this problem in various ways such as fine-tuning, wise selection of layers of the neural net, and scaling of images. The key problems in point form are given below:

(1)
The targeted object frequently appears at diverse scales and scale changes due to different resolution of images. This limitation becomes the reason for the poor performance of object detection.
(2)
The objects in high-resolution images are comparatively very small which reduces the accuracy when CNN features are pooled by the uppermost layer at a lower resolution. After several down-scaling of the sample, the feature map becomes 1/16 of the input image which causes the loss of key information of small objects and creates false alarms.
(3)
In remote sensing imagery, the labeled data is very less and accessible annotated data is not enough for optimal performance of object detection. This consumes much time to collect new labeled data and it is very expensive as well.
(4)
The remote sensing data is generated by distinctive tools (e.g. satellite, airborne, airplane, UAV and SAR, etc.) with varying resolution. It increases the plea for emerging a distinctive object detection method which could be applicable for all kinds of remote sensing imageries.

2.1.2 Image enhancement approaches for object detection

There are several different sized objects in various datasets and this reduces the performance of different classifiers/detectors due to irregular change in size and shape. There is a dire need to improve the optical remote sensing training images for object detection using geometric transformations. The training images have a great impact on object detection performance. Several well-intentioned contributions for image enhancement have been proposed in past decades [57, 58, 59, 60] still there is a gap to make more improvements in training images. Another wise way is to compress training images to reduce the required storage space for high-resolution remote sensing images. While lossless compressed images take similar training as compared to original images but it is advantageous for storage space. On the other hand, lossy images reduce the accuracy of object detection due to a lack of information. The optimal image enhancement approaches for training images are mentioned as under:

a)
Image denoising
b)
Image filtering
c)
Edge sharpening
d)
Image rotation
e)
Image scaling
f)
Image compression etc.

The compression of the object detection model will improve detection efficiency. Because the deep network model will occupy a large amount of storage space and its application in a mobile terminal is limited, compression of the object detection model will also be an important research idea in the future. The research based on analog network aims to design special supervised and unsupervised tasks to make object detection promising. The main idea is to train smaller object detection models by using the trained deep network object detection framework. The training of the small framework becomes more efficient through the training of additional tasks. Therefore, the overall framework of object detection can be compressed on the premise of ensuring the performance of object detection. Specifically, the object detection with deep training can be achieved. In addition, unsupervised training tasks can be added to strengthen network training. Additional tasks can be done to the shallow network by setting up coders and decoders. The training of service will further improve the efficiency of network training.
2.1.3 Limitations of current object detection methods

The search ability for large-scale remote sensing images is weak. At present, most of the algorithms focus on object detection for a small size image containing the object to be interpreted, and the ability to interpret the object from a large-scale remote sensing image is relatively weak. The algorithm has a high time complexity, and cannot fully meet the actual needs of automatic interpretation of the surge data, fast detection and recognition of objects. On the other hand, due to the difference of detector angle and distance from the object, the features presented by the object change greatly, which makes the database of the object characteristic view larger and consumes a lot of storage space. At the same time, the computational complexity of related algorithms increases in the process of object search. Existing interpretation algorithms have strong pertinence. When interpreting different kinds of objects in a remote sensing image, different algorithms need to be designed to make the same remote sensing image that needs to be processed many times, this increases the processing time. Therefore, different types of object interpretation are worthy of attention and research.

Objects in remote sensing images are affected by a variety of random factors, such as the number of unknown objects, the probability of complete or partial occlusion of objects, the probability of unknown objects, etc., which makes the object information incomplete. The research work of knowledge base remote sensing is not comprehensive, and the existing algorithms do not fully integrate the important prior knowledge such as object environment, so it is necessary to further mind the characteristics of the object environment.

3. Object detection based on deep learning

3.1 Reinforcements of CNN’s towards computational burden

In many recent applications of object detection and classification, CNN is the most popular method as compared to traditional feature extraction methods. The previous approaches were based on pre-designed hand-crafted features. The traditional features such as HOG and SIFT etc. are not promising as compared to CNN features due to redundant feature computations of CNN. Feature learning is primarily based on the end-to-end process in CNN, exploiting a gradient descent method to update the parameters in the propagation system. In a conventional CNN, the initial layers are established by convolutional and pooling layers while final layers are made up of fully-connected layers and a soft-max layer. The softmax layer is used as a classification/detection layer which can be replaced by the Support Vector Machine (SVM) classification layer for performance evaluation. In each layer of the CNN, the features are generated from the preceding layer by sharing weights. The authors Han et al. [61, 62] revealed an efficient framework for CNN training by taking advantage of useful connections and pruning useless connections. After pruning, retraining of the network is necessary to fine-tune the weights.

Parallel computing is an effective way to reduce processing time. It has been exploited in Deep Neural Networks (DNNs) that increases the speed ten times as compared with the processing time of an individual DNN although with a slight decay in accuracy [63]. A single-stage Fully Convolutional Network (FCN) has been proposed to enable faster detection compared to multi-stage schemes [64]. This method is performed by interchanging the convolutional layers with fully connected layers. An additional CNN based approach was developed to increase the robustness and accuracy by a combination of Binary Normed Gradients (BING) and CNN in which BING has been implemented for localization and CNN for classification [65]. The computational cost and storage space is the major concern in optical remote sensing imagery so the training images have been compressed and down-scaled to reduce the computational burden with effective accuracy. The comparison of region-based methods shows that the training with compressed images is more beneficial than down-scaled images and the size of these images was reduced significantly to save the storage space [66]. Artificial Neural Networks (ANNs) are playing a vital role in the field of robotics in which DNNs are becoming more attractive. DNNs have various variants, which are mainly differentiated by their number of layers. The structure of DNNs consists of multiple hidden layers between one input and one output layer. The number of neurons can be different or the same in each layer of DNNs that depends upon the architecture.

3.2 Object detection frameworks for region-based CNN methods

3.2.1 Region-based convolutional neural network (R-CNN)

In 2014, Girshick et al. [67] introduced CNN model into the object detection field and proposed the R-CNN model. In SPP-Net proposed by He et al. [41] in 2015 and Fast R-CNN proposed by Girshick [42], respectively, several problems are improved by them such as 1) For each candidate region, to extract feature expression of fixed length, it needs to be adjusted to a uniform size (i.e. resized patches as shown in Fig. 2), which may lead to loss of image information to some extent; 2) Candidate region recommendation, depth feature extraction, step by step SVM training for large-scale data so the process is usually costly. 3) The consideration of the quantity of training time and storage space; each candidate region covers a large proportion of each other which results in repeated feature extraction.

Figure 2.

R-CNN framework for object detection and classification.

According to the intuitive ideas, object detection refers to the location of the object which consists x, y, width and height of bounding box, which covers the object, whereas, classification tells us that which object is located. In other words, classification/recognition gives us the class label for the object. The whole detection process of R-CNN is shown in Fig. 2. R-CNN algorithm can be divided into four steps: 1) Candidate region generation: input a natural image and extract about 2000 candidate regions by Selective Search method; 2) feature extraction: normalize the image of each candidate region, and input the deep convolution network CNN for convolution, pooling and other operations to extract features; 3) Classification judgment: extracted features; Linear SVM classifier is fed to judge the classification; 4) position refinement: the candidate box position is refined by boundary regression to get accurate object region.

To overcome the time-consuming problem of R-CNN feature extraction, SPP-Net [41] proposed that the whole image features should be extracted first, instead of the cumbersome convolution calculation for each candidate region, and then the feature of candidate region should be intercepted on the feature map before classification, and this method needs 0.5 seconds to process an image.

3.2.2 Spatial pyramid pooling network (SPP-Net)

In SPP-Net, He et al. [41] extract feature expression for each candidate region by using spatial pyramid pooling method, which effectively avoids losing information in the process of adjusting the size of the candidate region, and makes the detection accuracy of SPP-Net surpass R-CNN model. Girshick proposed a ROI Pooling layer in Fast-RCNN. Using this layer, only one depth feature is extracted. Then, according to the location of each candidate region mapped to the ROI Pooling layer, the feature expression of uniform length is extracted by combining the idea of SPP-Net. In addition, because of the introduction of the ROI Pooling layer, feature extraction and classification model training are completed in the same network. Therefore, compared with R-CNN and SPP-Net, the performance of Fast-RCNN in training, testing speed and detection accuracy has been improved.

Figure 3.

SPP-net structure for fixed-length image representation.

The detection flow of SPP-Net is shown in Fig. 3. Although the overall detection process remains unchanged, the following improvements have been made on the basis of R-CNN: 1) it is no longer necessary to unify the images into fixed-size by intercepting and normalizing operations in advance, so as to solve the problem of image distortion and information loss caused by normalization operations; 2) using spatial pyramid structure to replace the pooling layer of the last convolution layer.

Therefore, SPP-Net has the following advantages: 1) By defining a scalable pooling layer, SPP-Net can process images of arbitrary aspect ratio and arbitrary scale, produce fixed size output, and improve the robustness of extracted features through multi-scale; 2) Because the features of all candidate regions are extracted directly from the overall feature mapping, the repeated counting of convolution layer is effectively solved. The efficiency is improved by calculating problems. But at the same time, there are still some problems in SPP-Net: 1) the isolated training process makes it impossible to train the network parameters as a whole, and still needs to preserve a large number of intermediate results; 2) the convolution layer and the connection layer located on both sides of the spatial pyramid pooling layer still need to be adjusted separately, which limits the effect of deep convolution network; 3) the extraction of candidate areas is still very important in the whole detection process.

3.2.3 Fast-RCNN

Fast-RCNN [42] is mainly improved by accelerating R-CNN. The detection flow of Fast-RCNN is shown in Fig. 4. In Fast-RCNN, firstly, a series of candidate regions are generated by one candidate region method; secondly, convolution neural network feature maps of input images are generated by multiple convolution layers; secondly, candidate region pooling layer maps candidate regions to convolution neural network feature maps, and extracts feature maps of the same size for candidate regions of different sizes; finally, pooling is located in candidate regions. After the layer, the final feature is extracted, and the classification score and the location of the detection box are predicted and refined based on the feature.

Figure 4.

Fast-RCNN framework for object detection and classification.

The improvement of Fast-RCNN can be summarized as follows: 1) Compared with R-CNN, the detection quality is improved; 2) Multi-task loss layer is introduced by combining classification and border regression loss function, and single-level training process is exploited which improves the accuracy of the algorithm; 3) end-to-end training is utilized except for candidate region extraction and all features are stored in display memory without additional disk space. 4) Referring to the idea of SPP-Net, a single fixed-scale ROI layer is proposed and the full-connection layer is speeded up by SVD decomposition.

However, Fast-RCNN still has the following shortcomings: 1) SS method is still used to extract candidate regions, and the relative feature extraction is still time-consuming; 2) real end-to-end training and testing are not achieved, which cannot meet the requirements of real-time detection; 3) GPU acceleration is used, but the candidate region extraction method is still CPU implementation.

3.2.4 Faster-RCNN

Faster-RCNN is an advanced improvement of Fast-RCNN [68]. The region proposal network (RPN) is proposed in Faster-RCNN. It takes only 10 ms to generate regions by applying the RPN on GPU with the convolutional neural network. The region proposal generation and Fast-RCNN have been merged into a new network model to form a real-time object detection framework which enhances the detection speed and accuracy. The object detection structure of Faster-RCNN has shown in Fig. 5.

Ren et al. proposed a Regional Proposal Network (RPN) based on CNN in 2017 [43], which is known as RPN. The experiment results show that compared to the conventional region proposal algorithm such as Selective Search (SS), RPN achieves higher accuracy by recommending fewer regions (RPN: 300, Selective Search: 2000). The RPN has been combined with Fast-RCNN and trained them alternately, and acquired the unified detection model which is known as Faster-RCNN. In addition, Redmon et al. also proposed a network model without region proposals, known as YOLO (You Only Look Once) [45]. This proposed model predicts the conceivable locations of the objects in the images using regression. The model has achieved optimal real-time performance on GPU and the detection speed is about 45 FPS (frame per second). Nevertheless, Faster-RCNN has maintained its definite advantages in terms of detection accuracy. However, Faster-RCNN has some shortcomings such as it has not reached to the real-time detection. The way of getting region boxes before classification needs a huge amount of computations. This limitation has led to another advance approach, is called as Mask-RCNN [69]. The important features which make the difference between region-based CNN object detectors are given in Table 1.

Table 1
The distinctive characterization of CNN object detectors

Detector	Prediction time/image	Description
CNN	–	• It predicts several regions hence it faces high computations
R-CNN	40–50 seconds	• Less time needed for training but slow detection process • It allows adapted region proposals
Fast-RCNN	2 seconds	• It allows adapted region proposals
Faster-RCNN	0.2 seconds	• Optimal real-time detection performance • It does not allow adapted region proposals

Figure 5.

Faster-RCNN framework for object detection and classification.

In present years, many scholars have presented CNN models for vehicle detection in aerial images because CNN has excellent feature representation ability. Ammour et al. has proposed the detection model based on deep learning for vehicle detection in unmanned aerial vehicle (UAV) images [70]. This proposed model is analogous to the R-CNN framework as mentioned above. The key difference is that they use segmentation rather than the region proposal algorithm. The method of image segmentation describes two advantages: the processing speed of the segmentation is better than selective search, and the extracted regions do not overlap with each other. However, the extracted region by the segmentation method needs post-processing before using it on CNN. Qu et al. has proposed a detection method for UAV images which is based on SPP-Net and region proposal algorithm [71]. However, in these two detection methods, the CNN framework has been used for feature extraction, and there are still analogous problems with R-CNN models. The regions are very important in order to enhance accuracy; therefore, RCNN is utilized to produce accurate bounding boxes [72]. These boxes are horizontal at the first stage and rotated at the second stage which was adopted from cascade RCNN [73] and it verifies that the regression of targets by this approach can produce more precise results. The detection accuracy is also concerned with the dataset’s characteristics. More specifically, the performance of Faster-RCNN has been improved for fixed input size in terms of average precision [74]. In region-based methods, SS is prominently involved to enhance the accuracy. A systematic region proposal method was proposed in comparison with SS to reduce the processing speed with optimal accuracy [75].

3.3 Object detection based on regression methods

Faster-RCNN algorithm is one of the main object detection algorithms at present, but its speed does not meet the real-time requirements. Then algorithms such as YOLO and SSD gradually highlight their superiority. This kind of method makes full use of the idea of regression, returns directly to multiple positions of the original image, and gives the accurate box of the object position and category.

3.3.1 You only look once (YOLO)

YOLO algorithm proposed by Redmon et al. in 2016 is a convolutional neural network that can predict multiple box locations and classes at once. The network design strategy of the YOLO algorithm continues the core idea of Google Net [76] which reveals end-to-end object detection in a real-time scenario. It gives a full role to the advantages of fast speed, but its essence towards accuracy has declined. However, the YOLO 9000 [77] algorithm proposed by Redmon et al. in 2016 improves its accuracy on the speed of the original YOLO algorithm. There are two main improvements: 1) a series of improvements have been made on the original YOLO detection framework to make up for the lack of detection accuracy; 2) a method of combining object detection and training is proposed. The training network of the YOLOv2 algorithm [78] can adjust dynamically under certain circumstances by the down-sampling method. This mechanism can make the network predict different size pictures and balance the speed and accuracy of detection. The YOLO framework carries out the detection likewise regression problem. It transforms an image into S $\times$ S grid then each cell of grid predicts bounding boxes B with their confidence and class probabilities can be described by C. The final tensor for predictions can be expressed as S $\times$ S $\times$ (B $\times$ 5 $+$ C).

3.3.2 Single shot multibox detector (SSD)

In 2016, Liu et al. proposed the SSD algorithm [46], which combines YOLO’s regression idea and Faster-RCNN anchor mechanism to achieve the coexistence of speed and accuracy. The original YOLO algorithm recognizes objects by dividing image into S $\times$ S grid. When small objects are detected with this framework, the accuracy will decrease. In the SSD algorithm, the full connection layer of the YOLO algorithm is removed, so the object of any size can be detected, and the performance is basically not good. The biggest difference between training and training SSD test sets using candidate regions and standard detectors for pooling is that ground truth needs to be assigned a set of fixed sets to detect a particular output in the output. When this assignment is determined, the loss function and backward propagation can achieve end-to-end applications. In a word, SSD combines the regression idea in YOLO and anchor mechanism in Faster-RCNN and uses the multi-scale regional features in all positions of the whole graph to carry out regression. It not only keeps the fast speed of YOLO but also ensures that the region prediction is as accurate as Faster-RCNN. The shallow layers in SSD might not yield sufficient high-level features to execute predictions for small size objects. Furthermore, SSD shows optimal results for a complex environment but it requires a large amount of training data. The training process of SSD is acquired from a multi-scale MultiBox approach which exhibits that the model could deal with a variety of instances for each class [79, 80] and it is modified in order to handle multi-class object categories. The weighted sum of localization loss ( $L_{\textit{loc}}$ ) and confidence loss ( $L_{\textit{conf}}$ ) provides the total objective loss. It the number of matching of default boxes will be zero then the overall loss function will become zero.

Figure 6.

Statistical assessments of feasibility of object detectors.

Finally, based on the previous related work, we investigate that the sliding windows approach has the lowermost difficulty in execution and there is essentially a recurrence of executing image classification while it might be very slow during the process of training and testing. Though the accuracy of image classification relies on the effectiveness of the network it needs the expertise to build such a mature network. The region proposal approach has reduced the training and testing time compared with the conventional sliding window approaches and helps to enhance the detection accuracy. Indeed, Faster-RCNN attains the maximum accuracy as compared to other methods [81]. By means of testing and training time, the well-known SSD is considerably faster than the state of the art methods whereas it eliminates the region proposal approach, but it costs the reduced accuracy as compared to those techniques with region proposals.

4. Conclusion

Object detection is a fundamental and challenging topic in the field of optical remote sensing imagery. Several substantial contributions for object detection have been proposed during the past few years. This paper conveys a brief review of the current progress for object detection. Here, we have broadly discussed the region and regression-based object detection approaches. More specifically, we have briefly discussed R-CNN, Fast-RCNN, Faster-RCNN, SSD and YOLO in the appraisal. Besides, we have summarized the current limitations and challenges which would be considerably useful for the upcoming researchers to find and fill the gaps in this research.

Object detection is the premise of high-level visual tasks such as scene content understanding. There are many difficulties in static image object detection such as object deformation, object occlusion and small object detection. How to solve these problems more effectively and reasonably is an important research direction in the future. The traditional effective object detection methods which deal with some special problems (object deformation, occlusion, etc.) are introduced into the deep networks to increase the ability to deal with these special problems and improve the performance of object detection. The specific neurons of the corresponding network layer (such as some operations need to be carried out for a single neuron and some operations need to be carried out on different areas of the Feature Map) could be organized reasonably to get the required results. In the process of backpropagation, the derivative is needed. The approximation of derivatives is obtained by limited methods. The local network structure can be formed by combining different network-layer operations.

Footnotes

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grants 61471148.

Conflict of interest

The authors declare no conflict of interest.

Author’s Bios

	Shahid Karim received his BS degree in Electronics from Comsats University, Abbottabad, Pakistan, and his MS degree in Electronics and Information Engineering from Xi’an Jiaotong University, China, in 2010 and 2015, respectively. He received his PhD degree from School of Electronics and Information Engineering, Harbin Institute of Technology (HIT), China, in 2019. Currently, he is working as Assistant Professor at Department of Computer Science, ILMA University, Karachi, Pakistan. His current research interests include image processing, computer vision and machine learning towards remote sensing imagery.
	Ye Zhang received his MS degree and PhD in communication and information science from the HIT, Harbin, China, in 1985 and 1996, respectively. He is currently a professor at the Department of Information Engineering, HIT. His research interests include hyperspectral image processing, compression, imaging scout, and vision matching. He is a member of the Chinese Signal and System teaching committee and the Commission of Science, Technology, and Industry for National Defense expert committee.
	Shoulin Yin received a B.S. degree and M.S. degree in Software College from Shenyang Normal University, Shenyang, China, in 2013 and 2015, respectively. He is currently working toward a Ph.D. degree in the School of Electronic and Information Engineering at the Harbin Institute of Technology (HIT), Harbin, China. His research interests include image fusion, target detection and image recognition.

Irfana Bibi received the Ph.D. degree in computer application technology from the School of Artificial Intelligence, Xidian University, Xi’an, China, in 2019. She received the M.Sc. degree and M.S. degree from the University of the Punjab, Lahore, Pakistan, in 2004 and 2012, respectively. Her research interests include image processing, image segmentation, object detection and computer vision.

Ali Anwar Brohi received B.S. degree in Mechanical Engineering from Mehran University of Engineering and Technology, SZAB Campus, Khairpur Mir’s, Sindh, Pakistan in 2010 and MS degree in Mechanical Engineering from Harbin Institute of Technology, Harbin, China in 2015. He is currently working toward a PhD degree in the School of Energy Science and Engineering at Harbin Institute of Technology, Harbin, China. His research interests include entropy generation, heat transfer and fluid flow.

References

Gall

and Lempitsky

, Class-specific hough forests for object detection, in: Decision Forests for Computer Vision and Medical Image Analysis, Springer, 2013, pp. 143–157.

Dollár

Appel

Belongie

and Perona

, Fast feature pyramids for object detection, IEEE Trans. Pattern Anal. Mach. Intell. 36(8) (2014), 1532–1545.

Zou

Zhang

Weng

and Meng

, Multi-vehicle tracking via real-time detection probes and a markov decision process policy, Sensors 19(6) (2019), 1309.

Brookhuis

K.A.

De Waard

and Janssen

W.H.

, Behavioural impacts of advanced driver assistance systems – an overview, Eur. J. Transp. Infrastruct. Res. 1(3) (2019).

Fang

Wang

Yao

Zhao

and Zha

, On-road vehicle tracking using part-based particle filter, IEEE Trans. Intell. Transp. Syst. 12 (2019), 4538–4552.

Caraffi

Vojíř

Trefný

Šochman

and Matas

, A system for real-time detection and tracking of vehicles from a single car-mounted camera, in: 2012 15th International IEEE Conference on Intelligent Transportation Systems, 2012, pp. 975–982.

Chavez-Garcia

R.O.

and Aycard

, Multiple sensor fusion and classification for moving object detection and tracking, IEEE Trans. Intell. Transp. Syst. 17(2) (2016), 525–534.

W.-C.

Chen

C.-H.

Chen

T.-Y.

Huang

D.-Y.

and Wu

Z.-C.

, Moving object detection and tracking from video captured by moving camera, J. Vis. Commun. Image Represent. 30 (2015), 164–180.

Kulchandani

J.S.

and J.Dangarwala

, Moving object detection: Review of recent research trends, in: 2015 International Conference on Pervasive Computing (ICPC), 2015, pp. 1–5.

10.

Oreifej

and Shah

, Simultaneous video stabilization and moving object detection in turbulence, IEEE Trans. Pattern Anal. Mach. Intell. 35(2) (2013), 450–462.

11.

Zhou

Yang

and Yu

, Moving object detection by detecting contiguous outliers in the low-rank representation, IEEE Trans. Pattern Anal. Mach. Intell. 35(3) (2013), 597–610.

12.

Jiang

Wang

Yuan

Zheng

and Li

, Salient object detection: A discriminative regional feature integration approach, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Portland, Oregon), 2013, pp. 2083–2090.

13.

Szegedy

Toshev

and Erhan

, Deep neural networks for object detection, in: Advances in Neural Information Processing Systems, 2013, pp. 2553–2561.

14.

Teng

and Karim

, DMCNN: A deep multiscale convolutional neural network model for medical image segmentation, J. Healthc. Eng. 2019 (2019), 1–10.

15.

Yin

Zhang

and Karim

, Large scale remote sensing image segmentation based on fuzzy region competition and gaussian mixture model, IEEE Access 6 (2018), 26069–26080. doi: 10.1109/ACCESS.2018.2834960.

16.

Yin

Zhang

and Karim

, Region search based on hybrid convolutional neural network in optical remote sensing images, Int. J. Distrib. Sens. Networks 15(5) (2019), 1550147719852036.

17.

Yin

Teng

Jiang

and Karim

, An optimised multi-scale fusion method for airport detection in large-scale optical remote sensing images, Int. J. Image Data Fusion (2020), 1–14.

18.

Wang

Han

T.X.

and Yan

, An HOG-LBP human detector with partial occlusion handling, in: Computer Vision, 2009 IEEE 12th International Conference on, 2009, pp. 32–39.

19.

Dalal

and Triggs

, Histograms of oriented gradients for human detection, in: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, 1, 2005, pp. 886–893.

20.

Zhu

Yeh

M.-C.

Cheng

K.-T.

and Avidan

, Fast human detection using a cascade of histograms of oriented gradients, in: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), 2, 2006, pp. 1491–1498.

21.

Lowe

D.G.

, Distinctive image features from scale-invariant keypoints, Int. J. Comput. Vis. 60(2) (2004), 91–110.

22.

Juan

and Gwun

, A comparison of sift, pca-sift and surf, Int. J. Image Process. 3(4) (2009), 143–152.

23.

Liu

Yuen

and Torralba

, Sift flow: Dense correspondence across scenes and its applications, IEEE Trans. Pattern Anal. Mach. Intell. 33(5) (2011), 978–994.

24.

Divvala

S.K.

Efros

A.A.

and Hebert

, How important are ‘deformable parts’ in the deformable parts model, in: European Conference on Computer Vision, 2012, pp. 31–40.

25.

Felzenszwalb

P.F.

Girshick

R.B.

and McAllester

, Object detection with discriminatively trained part-based models, IEEE Trans. Pattern Anal. Mach. Intell. 32(9) (2010), 1627–1645.

26.

Girshick

Iandola

Darrell

and Malik

, Deformable part models are convolutional neural networks, in: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (Boston, Massachusetts), 2015, pp. 437–446.

27.

Ouyang

and Wang

, Joint deep learning for pedestrian detection, in: Proceedings of the IEEE International Conference on Computer Vision (Sydney, NSW, Australia), 2013, pp. 2056–2063.

28.

Uijlings

J.R.R.

Van De Sande

K.E.A.

Gevers

and Smeulders

A.W.M.

, Selective search for object recognition, Int. J. Comput. Vis. 104(2) (2013), 154–171.

29.

Zhu

Porikli

and Li

, Tracking randomly moving objects on edge box proposals, arXiv Prepr. arXiv1507.08085, 2015.

30.

Russakovsky

et al., Imagenet large scale visual recognition challenge, Int. J. Comput. Vis. 115(3) (2015), 211–252.

31.

Lin

T.-Y.

et al., Microsoft coco: Common objects in context, in: European Conference on Computer Vision, 2014, pp. 740–755.

32.

Krizhevsky

Sutskever

and Hinton

G.E.

, Imagenet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems, 2012, pp. 1097–1105.

33.

Long

Shelhamer

and Darrell

, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Boston, Massachusetts), 2015, pp. 3431–3440.

34.

Hansen

L.K.

and Salamon

, Neural network ensembles, IEEE Trans. Pattern Anal. Mach. Intell. 10 (1990), 993–1001.

35.

Kingsbury

Sainath

T.N.

and Soltau

, Scalable minimum Bayes risk training of deep neural network acoustic models using distributed Hessian-free optimization, in: Thirteenth Annual Conference of the International Speech Communication Association, 2012.

36.

Ren

J.S.J.

Liu

and Jia

, Deep convolutional neural network for image deconvolution, in: Advances in Neural Information Processing Systems, 2014, pp. 1790–1798.

37.

Simonyan

and Zisserman

, Very deep convolutional networks for large-scale image recognition, arXiv Prepr. arXiv1409.1556, 2014.

38.

Szegedy

et al., Going deeper with convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Boston, Massachusetts), 2015, pp. 1–9.

39.

Zhang

Ren

and Sun

, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Las Vegas, NV, USA), 2016, pp. 770–778.

40.

Girshick

Donahue

Darrell

Berkeley

U.C.

and Malik

, Rich feature hierarchies for accurate object detection and semantic segmentation, in: Proc. IEEE Conf. Comput. Vis. pattern Recognit. (Columbus, Ohio), 2014, pp. 2–9. doi: 10.1109/CVPR.2014.81.

41.

Zhang

Ren

and Sun

, Spatial pyramid pooling in deep convolutional networks for visual recognition, in: European Conference on Computer Vision, 2014, pp. 346–361.

42.

Girshick

, Fast r-cnn, in: Proceedings of the IEEE International Conference on Computer Vision (Boston, Massachusetts), 2015, pp. 1440–1448.

43.

Ren

Girshick

and Sun

, Faster R-CNN: Towards real-time object detection with region proposal networks, IEEE Trans. Pattern Anal. Mach. Intell. (6) (2017), 1137–1149.

44.

Dai

and Sun

, R-fcn: Object detection via region-based fully convolutional networks, in: Advances in Neural Information Processing Systems, 2016, pp. 379–387.

45.

Redmon

Divvala

Girshick

and Farhadi

, You only look once: Unified, real-time object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Las Vegas, NV, USA), 2016, pp. 779–788.

46.

Liu

et al., Ssd: Single shot multibox detector, in: European Conference on Computer Vision, 2016, pp. 21–37.

47.

Zhang

Huang

and Huang

, Data decomposition and spatial mixture modeling for part based model, in: Asian Conference on Computer Vision, 2012, pp. 123–137.

48.

Felzenszwalb

P.F.

Girshick

R.B.

and McAllester

, Cascade object detection with deformable part models, in: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010, pp. 2241–2248.

49.

Zhang

Donahue

and Girshick

, Part-based R-CNNs for fine-grained category detection, in: European Conference on Computer Vision, 2014, pp. 834–849.

50.

Shrivastava

Gupta

and Girshick

, Training region-based object detectors with online hard example mining, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Las Vegas, NV, USA), 2016, pp. 761–769.

51.

Kong

Yao

Chen

and Sun

, Hypernet: Towards accurate region proposal generation and joint object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Las Vegas, NV, USA), 2016, pp. 845–853.

52.

Torralba

, Contextual priming for object detection, Int. J. Comput. Vis. 53(2) (2003), 169–191.

53.

Benz

U.C.

Hofmann

Willhauck

Lingenfelder

and Heynen

, Multi-resolution, object-oriented fuzzy analysis of remote sensing data for GIS-ready information, ISPRS J. Photogramm. Remote Sens. 58(3–4) (2004), 239–258.

54.

Belgiu

and Drăguţ

, Random forest in remote sensing: A review of applications and future directions, ISPRS J. Photogramm. Remote Sens. 114 (2016), 24–31.

55.

Cheng

and Han

, A survey on object detection in optical remote sensing images, ISPRS J. Photogramm. Remote Sens. 117 (2016), 11–28.

56.

Hussain

Chen

Cheng

Wei

and Stanley

, Change detection from remotely sensed images: From pixel-based to object-based approaches, ISPRS J. Photogramm. Remote Sens. 80 (2013), 91–106.

57.

Kuang

Sui

Liu

Chen

and Gu

, Single infrared image enhancement using a deep convolutional neural network, Neurocomputing 332 (2019), 119–128.

58.

Suzuki

Horiba

and Sugie

, Neural edge enhancer for supervised edge enhancement from noisy images, IEEE Trans. Pattern Anal. Mach. Intell. 25(12) (2003), 1582–1596.

59.

Sreedhar

and Panlal

, Enhancement of images using morphological transformation, arXiv Prepr. arXiv1203.2514, 2012.

60.

Piao

and Park

, Image resolution enhancement using inter-subband correlation in wavelet domain, in: 2007 IEEE International Conference on Image Processing, 1, 2007, pp. I–445.

61.

Han

Pool

Tran

and Dally

, Learning both weights and connections for efficient neural network, in: Advances in Neural Information Processing Systems, 2015, pp. 1135–1143.

62.

Han

Mao

and Dally

W.J.

, Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv Prepr. arXiv1510.00149, 2015.

63.

Chen

Xiang

Liu

C.-L.

and Pan

C.-H.

, Vehicle detection in satellite images by parallel deep convolutional neural networks, in: Pattern Recognition (ACPR), 2013 2nd IAPR Asian Conference on, 2013, pp. 181–185.

64.

Zhang

and Yang

, Fast Vehicle Detection in Satellite Images Using Fully Convolutional Network, in: Chinese Conference on Intelligent Visual Surveillance, 2016, pp. 122–129.

65.

Wang

Meng

and Pan

, Vehicle detection in satellite images by incorporating objectness and convolutional neural network, J. Ind. Intell. Inf. Vol 4(2) (2016), 158–162.

66.

Karim

Zhang

Yin

Laghari

A.A.

and Brohi

A.A.

, Impact of compressed and down-scaled training images on vehicle detection in remote sensing imagery, Multimed. Tools Appl. 78(22) (2019), 32565–32583.

67.

Girshick

Donahue

Darrell

and Malik

, Rich feature hierarchies for accurate object detection and semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Columbus, Ohio), 2014, pp. 580–587.

68.

Ren

Girshick

and Sun

, Faster R-CNN: Towards real-time object detection with region proposal networks, in: Advances in Neural Information Processing Systems, 2015, pp. 91–99.

69.

Gkioxari

Dollár

and Girshick

, Mask r-cnn, in: Computer Vision (ICCV), 2017 IEEE International Conference on, 2017, pp. 2980–2988.

70.

Ammour

Alhichri

Bazi

Benjdira

Alajlan

and Zuair

, Deep learning approach for car detection in UAV imagery, Remote Sens. 9(4) (2017), 312.

71.

Zhang

and Sun

, Vehicle detection from high-resolution aerial images using spatial pyramid pooling-based deep convolutional neural networks, Multimed. Tools Appl. 76(20) (2017), 21651–21663.

72.

Zhu

and Du

, Rotated cascade R-CNN: A shape robust detector with coordinate regression, Pattern Recognit. 96 (2019), 106964.

73.

Cai

and Vasconcelos

, Cascade r-cnn: Delving into high quality object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Salt Lake City, UT, USA), 2018, pp. 6154–6162.

74.

Ammar

Koubaa

Ahmed

and Saad

, Aerial images processing for car detection using convolutional neural networks: Comparison between faster r-cnn and yolov3, arXiv Prepr. arXiv1910.07234, 2019.

75.

Karim

Zhang

Yin

and Asif

M.R.

, An Efficient Region Proposal Method for Optical Remote Sensing Imagery, in: IGARSS 2018–2018 IEEE International Geoscience and Remote Sensing Symposium, 2018, pp. 2455–2458.

76.

Zhong

Jin

and Xie

, High performance offline handwritten chinese character recognition using googlenet and directional feature maps, in: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), 2015, pp. 846–850.

77.

Redmon

and Farhadi

, YOLO9000: better, faster, stronger, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Honolulu, HI, USA), 2017, pp. 7263–7271.

78.

Zhang

Huang

Jin

and Li

, A real-time chinese traffic sign detection algorithm based on modified YOLOv2, Algorithms 10(4) (2017), 127.

79.

Szegedy

Reed

Erhan

Anguelov

and Ioffe

, Scalable, high-quality object detection, arXiv Prepr. arXiv1412.1441, 2014.

80.

Erhan

Szegedy

Toshev

and Anguelov

, Scalable object detection using deep neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Portland, Oregon), 2014, pp. 2147–2154.

81.

Zhang

Wang

and Sun

, R-CNN-Based Ship Detection from High Resolution Remote Sensing Imagery, Remote Sens. 11(6) (2019), 631.

A brief review and challenges of object detection in optical remote sensing imagery

Abstract

Keywords

1. Introduction

2. General framework of object detection

2.1.1 Limitations of remote sensing images

3. Object detection based on deep learning

3.1 Reinforcements of CNN’s towards computational burden

3.2 Object detection frameworks for region-based CNN methods

3.2.1 Region-based convolutional neural network (R-CNN)

Table 1 The distinctive characterization of CNN object detectors

3.3.1 You only look once (YOLO)

3.3.2 Single shot multibox detector (SSD)

Footnotes

Acknowledgments

Conflict of interest

Author’s Bios

References

Table 1
The distinctive characterization of CNN object detectors