Scale prediction based MDNet for infrared target tracking

Abstract

Infrared target tracking is increasingly becoming important for various applications in recent years. However, it is still a challenging task as limited information can be obtained from the infrared image. Inspired by the excellent performance of deep tracker, a novel tracker based on MDNet is proposed. As the prior information has great value for target tracking, a modified Back-Propagation network is used for predicting the scale of target during tracking. The result of the prediction is used for generating candidate windows for online learning, which can improve the performance of tracker. To evaluate the proposed tracking algorithm, we performed experiments on the VOT-TIR2016 and AMCOM infrared data. The experimental results demonstrate that our algorithm provides a 1.94% relative gain in accuracy and 21.4% in robustness on VOT-TIR2016 when compared with MDNet.

Keywords

Artificial intelligence infrared target tracking convolutional neural network scale prediction

1 Introduction

Visual tracking is sometimes considered a solved task, but many applied projects show that robust and accurate infrared(IR) target tracking method is scarce [1]. IR imaging technology is a non-contact passive detection technology, which producing images by sensing the radiation of target [2]. The main advantages of thermal camera are their ability to see in total darkness, their robustness to illumination changes and shadow effects, and they reduce privacy intrusion [3]. With the development of the IR technologies, they are commonly used not only in military purpose but also in civilian applications, such as the surveillance systems [2, 4]. However, in contrast to visual images, IR images generally have low spatial resolution, poor signal-to-noise ratios(SNR), and lack of textural information (see Fig. 1) [5]. Except the poor imaging quality compared with the visual image, the IR target dataset available is quite limited, which make it even impossible to train a very deep network for tracking [6]. Although great progress has been made, how to track IR target reliably in applied project is still challenging.

Fig. 1

(a) Visual image and (b) IR image.

Scholars have done a lot of work in the area of IR target tracking in past decade [7 –10]. Meanwhile, numerous superior trackers have been proposed in visual tracking. Despite the above-mentioned progresses in either speed or accuracy, real-time, high quality IR target tracking algorithms remain scarce. Multi-Domain Network (MDNet) [11] is the champion of the VOT2015, which is composed of shared layers and multiple branches of domain-specific layers. Where domains correspond to individual training sequences and each branch is responsible for binary classification to identify the target in each domain. The architecture of the network only contains six layers, which is substantially smaller than the common neural networks, such as the AlexNet [12] and VGG-Nets [13].

Our approach builds on two major observations based on prior work. First, the architecture of MDNet is high efficiency for IR target tracking as it only contains six layers. On the one hand, it has been proved that a lower layer carries more discriminative information which is suitable for target localization [14]. On another hand, the tiny network structure usually has a higher speed. Second, the precision of online learning has close relationship with tracking precision, online learning with poor quality even cause tracking failure. However, online tracking in MDNet is performed by evaluating the candidate windows randomly sampled around the previous target state. The scale of the target is highly related to the scales before current frame, which won’t change suddenly. Random sampling policy cannot make full use of the prior information. The main contributions of this paper can be summarized as follows:

As our first contribution, we proposed an IR tracking algorithm called SPMDNet (Scale Prediction based MDNet). It provides a 1.94% relative gain in accuracy and 21.4% in robustness on VOT-TIR2016 when compared with MDNet.

As our second contribution, we designed a modified BP network to predict the scale of the next frame online. The tracking performance can be improved with the feedback of the scale prediction. The scale predicting is meaningful for the research on IR image sequences.

The following sections are organized as follows. In section 2, we reviewed the existing research on both IR trackers and visual trackers, then we explained the motivation why we predicting the scale of the target during the online learning. In section 3, we presented the model structure of our approach and the details of it were introduced. In section 4, the tracking performance of our approach when compared with state-of-the-art algorithms from VOT-TIR2016 is introduced. Comprehensive experiments clearly demonstrate that our approach concurrently improved the tracking performance.

2 Related works

2.1 IR trackers

IR sensors/cameras produce images with a low SNR, which cause differences between IR trackers and visual trackers to some extent. As far as studies to date are concerned, many scholars have made brilliant achievements. In [15], morphological filter is used to remove noise and clutter in tracking window, which can only deal with the background noise. Mean-shift based approaches have been widely used in IR target tracking, which provide a general optimization solution in this field [9]. However, it cannot track the target properly when the target affected by different disturbing issues. In [16], AM-FM consistency checks was used for IR target tracking, which can improve the performance on challenging IR data sequences to some extent, even the famous AMCOM forward-looking infrared (FLIR) dataset. Despite the above-mentioned progresses in IR target tracking, the tracking performance cannot satisfy the requirement of applied projects.

2.2 Deep trackers

The rapid development of deep learning has promoted the development of visual tracking [17]. In [18 –20], the convolutional neural network (CNN) is used as the feature extractor, and they all adopt correlation filter as their base tracker. The state-of-the-art performance of deep trackers benefit from the expressive power of CNN features. C-COT [21] employed the implicit interpolation method to solve the learning problem in the continuous spatial domain. ECO [22] is an improved version of C-COT [21] in both accuracy and speed. While these deep trackers result in high accuracy and robustness, it often suffers from high computational burden. MDNet [11] is composed of five shared layers. The small architecture is suitable for IR target tracking, as the IR images usually have a low quality. The input of the images in IR target tracking is usually much smaller than the visual tracking, the size in AMCOM dataset is 128×128. It tends that the deep feature is less effective in target localization since the spatial information tends to be diluted as the network goes deeper. As mentioned above, we chose the MDNet as the basic architecture of our approach.

2.3 Modified BP neural network for scale prediction

For the MDNet, online tracking is performed by evaluating the candidate windows randomly sampled around the previous target state. The randomly sampled policy cannot make full use of the prior information, the error accumulation of the online learning will decrease the tracking precision. In this paper, we use a modified BP network to predict the scale of the target during the online learning. The input data is a series of tracking result before the current frame, and the output is the predicted scale of the target in current frame. By this mean, the precision and robustness of the tracker can be improved conspicuously.

The artificial neural network does not need to determine the mathematical equation of the mapping relationship between input and output in advance. It only learns some rules through its own training and obtains the result closest to the expected output value when given the input value. BP neural network is a kind of multi-layer feed-forward neural network, the main characteristics of which are signal forward transmission and error back propagation [23]. Since the scale changing on different sequence is vary, we can only predict the scale of next frame based on the tracking result before the current frame. As a nonlinear modeling and forecasting method, the neural network has been widely used in plenty of domains with good nonlinear quality, high fitting accuracy, flexible and effective learning method, completely distributed storage structure and hierarchical model structure [24]. In this paper, we use a modified BP neural network to predicting the scale during the online learning.

3 Scale prediction based MDNet

This section describes the architecture of our algorithm and the evaluation method used. The architecture of our approach is based on the MDNet. The scale prediction during the online learning can make full use of the prior information through generating reasonable learning samples, which can improve the accuracy and robustness of the tracker.

3.1 Network architecture

As shown in the Fig. 2, the model architecture can be divided into two parts, the shared layers and the domain-specific layers. The input of the Network is 107×107 grayscale images. There are five hidden layers including three convolutional layers and two fully connected layers. The domain-specific layers contain K branches responding to K domains is the fully connected layers. Compared with AlexNet [12] and VGG-Nets [13], this network architecture is tiny. Unlike the detection task, the target tracking indeed mainly focuses on distinguish target and background [11]. As a result, the depth of the network doesn’t need to be very deep. A shallow network is obvious effective, especially when we need to online learning during the tracking.

Fig. 2

Illustration of the SPMDNet.

3.2 Scale predictor in SPMDNet

The architecture of the modified BP network used for predicting the scale of the target is illustrated in Fig. 3. The BP network contain two layers, the hidden layer and the output layer. Where the input data is the scale information before the current frame, the output data is the predicting scale of the current frame. The input data dimension is m and the output data dimension are single, which is the predicting result of the target.

Fig. 3

Illustration of the BP network.

As a classical BP network, it suffers from the disadvantages of classical BP network. The perform function of BP neural network mainly employs MSE (Mean Square Error) function, which is good at dealing with gaussian distribution error. However, the scale predicting forecast error does not satisfy the gaussian distribution. Comparing with MSE, MCC (Maximum Correntropy Criteria) is more suitable for scale predicting. Correntropy [25] is a theory used for evaluating the similarity between two random variables such as X, Y. The definition is as Equation (1). $V_{σ} (X, Y) = E [k_{σ} (X - Y)]$ (1) where k_σ (·) is the kernel function, σ is the kernel width.

According to the theory of correntropy, while correntropy equals the maximum, the forecast error is the minimum and can obtain an optimized BP network model. $mcc (e) = max \frac{1}{n} \sum_{i = 1}^{n} k_{σ} (e_{i})$ (2)

As shown in Fig. 4, the input training data are the scales of the target form current frame t - 1 to frame t - m - n. As a result, the training data have 2 (m - n). The ‘2’ here means the width and the length of the target, which were predicted respectively. The output of the BP network has only one result at each time. As shown in Fig. 4, the training data was recombined to n item, each item contain m number. In our algorithm, m = 49 and n = 51. The network predicting 200 times to produce candidate windows for online learning.

Fig. 4

Training data used for training of the BP network.

The overall procedure of the scale prediction algorithm is presented in Algorithm 1.

Algorithm 1: Scale prediction in MDNet
1. Initial target state x₁;
2. if t < 101;
3. samples = random (x_t-1);
4. else;
5. Training data = $x_{t - 100}^{i} : x_{t - 1}^{i};$
6. Predicting the scale x_t by BP network;
7. Output samples x _N based on prediction.
8. End

The further implementation details are described below.

Training data. The training of the BP network was executed online, as the training data is produced with the tracking process. The predicted scale of the target can provide the tracker more scientific and reasonable samples, which can improve the tracking performance.

Network learning. To extract features satisfying common properties of IR image, we trained the shared layers with VOT-TIR 2016. For each tracking sequences, we train the network for 300 iterations with a learning rate 0.05.

3.3 Bounding box regression

Bounding box regression technique has been widely used in object detection. Based on the prediction of the scale of the target, a simple linear regression model was used for predicting the target location. It is a task which mean to find the optical target position ${bf x}_{t}^{*}$ from the candidate samples x _t . Candidate samples can be written as bfx¹, …, bfx^N, _N represents the number of the candidate samples. All of the candidates are evaluated using the network, the result were shown as the positive scores f⁺ (bfxⁱ) and negative scores f^- (bfxⁱ). Then target position ${bf x}_{t}^{*}$ can be calculate by Equation (3). $x^{*} = \underset{x^{i}}{\arg \max f^{+} (x^{i})}$ (3)

3.4 Evaluation method of the VOT-TIR2016

In contrast to the numerous benchmarks that exist in the area of visual tracking, the benchmarks designed for IR tracking is rare. As performance measure, we compared our algorithm with 18 kinds of excellent algorithm from the VOT-TIR2016. All the tracking algorithms are evaluated with the VOT2016 evaluation Toolkit 1 .

3.4.1 Accuracy

The accuracy measurement is computed from the overlap between the predicted bounding box and the ground truth. Base on the definition of IOU (Intersection over Union), the accuracy in VOT is defined as follows. $φ_{t} = \frac{A_{t}^{G} \cap A_{t}^{T}}{A_{t}^{G} \cup A_{t}^{T}}$ (4) where $A_{t}^{G}$ is the ground truth responding bounding box, the $A_{t}^{T}$ represents the tracking box, and t denotes the index of the frame. $Φ_{t} (i) = \frac{1}{N_{rep}} \sum_{i = 1}^{N_{rep}} Φ_{t} (i, k)$ (5)

The Φ_t (i, k) denotes accuracy in the kth repetition of tracker i, and the N_rep denotes the number of repetitions. As a result, the accuracy of the tracker i can be defined as follows. $a (i) = \frac{1}{N_{valid}} \sum_{t = 1}^{N_{valid}} Φ_{t} (i)$ (6)

In order to make the display more intuitive, the accuracy is modified as follows. S is a constant term, which would be shown in the result of the accuracy. $A (i) = e^{- Sa (i)}$ (7)

3.4.2 Robustness

The robustness is used to evaluate the robustness of a tracker. The robustness of the VOT is defined as follows. $R (i) = \frac{1}{N_{rep}} \sum_{k = 1}^{N_{rep}} F (i, K)$ (8)

3.4.3 EAO (Expected Average Overlap)

Because the evaluation method Accuracy-Robustness cannot make full use of the row data, a new index EAO is proposed. The EAO only focus on the accuracy based on definition of overlap. The calculation of EAO is shown as follows. $Φ_{N_{s}} = \frac{1}{N_{s}} \sum_{i = 1}^{N_{s}} Φ (t)$ (9) where N_s is the length of the sequence, the accuracy is valuated with one-pass evaluation.

4 Experiment

4.1 Implementation details and dataset

Our tracker is implemented in MATLAB with MatConvNet toolbox, which runs at round 1.38 fps on a single NVIDIA GTX 1070 GPU with 4GB memory in Linux operating system.

We evaluated the proposed tracking algorithm on the well-known public IR dataset named VOT-TIR 2016 2 , which contain 25 challenging video sequences. The IR sequences contained in the data set were collected from nine different sources using ten different types of sensors, which originate from industry, research institutes and 2 EU (European Union) projects [1]. In the VOT challenge protocol, the tracker is re-initialized whenever tracking fails and the evaluation module reports both accuracy and robustness, which correspond to the bounding box overlap ratio and the number of failures respectively. In addition, we also evaluated our algorithm on two sequences from the famous FLIR dataset named AMCOM. Moreover, the AMCOM sequences obtained via an airborne, moving platform, suffer from abrupt discontinuities in motion. It is more similar with the applied project.

4.2 Evaluation on the VOT-TIR2016

At first, we made a qualitative evaluation of our algorithm and other trackers. In the VOT challenge protocol, a tracker is re-initialized whenever tracking fails. In order to evaluate the performance of the trackers more concise, we do not re-initialize it again. Figure 5 summarizes qualitative comparisons of our approach with 11 ranking trackers of VOT-TIR2016. The tracking box with different color responding to different trackers. The failing frames are labeled with different colors of ×. Compared with other trackers, our approach can track the target more reliably. Even in the cases of the camera motion, severe occlusion, fast motion et al.

Fig. 5

A visualization of tracking result of our algorithm and other 11 ranking trackers on nine challenging sequences.

4.2.1 The robustness-accuracy

The sequence pooled AR rank plot is obtained by concatenating the results from all sequences and creating a single rank list, while the attribute normalized AR rank plot is created by ranking the trackers over each attribute and averaging the rank lists. Table 1 illustrates the accuracy ranks with respect to the visual attributes. Our algorithm outperforms all the compared algorithms in most of attribute, including the camera motion, motion change and size change. All the attributes above have a crucial relationship with the scale of the target, which indicate that our scale prediction is effective and meaningful. The average accuracy of our approach ranking first on VOT-TIR2016. The average scores of the different attributes shows the accuracy of our algorithm increased by 1.94 percent approximately. Table 2 illustrates the robustness ranking result. Our algorithm outperforms all the compared algorithms, which proved the robustness of our algorithm is excellent. Compared with the previous MDNet, the robustness of our algorithm has increased by 21.4 percent roughly. In order to show the ranking unambiguous, the first, second and the third best scores are highlighted in red, blue and green colors respectively. In summary, the accuracy and robustness of our tracker have been improved obvious with the scale prediction.

Table 1
The average scores and ranks of accuracy in different attributes: Camera motion (CM), Dynamics change (DC), Motion change (MC), Occlusion(O), Size change (SC)

CM DC MC O SC Mean Weighted Mean Pooled

SPMDNet 0.6402 0.536 0.6207 0.6051 0.6789 0.6162 0.6454 0.6461

MDNet 0.6023 0.5368 0.6145 0.6366 0.6565 0.6133 0.6331 0.6425

BDF 0.4341 0.5199 0.4482 0.4312 0.3721 0.4411 0.4146 0.4122

BST 0.4483 0.4652 0.4527 0.5444 0.5391 0.4900 0.4945 0.5110

DAT 0.5598 0.5288 0.4808 0.5732 0.5632 0.5412 0.5512 0.5676

deepMKCF 0.6226 0.6119 0.5917 0.6067 0.6432 0.6152 0.6245 0.6220

DPT 0.5774 0.5597 0.4830 0.5769 0.4866 0.5367 0.5296 0.5271

DSST 0.5697 0.5731 0.5551 0.6315 0.6138 0.5886 0.5914 0.6017

FCT 0.415 0.4153 0.4493 0.3510 0.4330 0.4127 0.4196 0.4277

GGT2 0.5536 0.4941 0.5454 0.5788 0.5818 0.5507 0.5641 0.5683

LOFT_Lite 0.2507 0.3400 0.2498 0.2617 0.2471 0.2699 0.2533 0.2645

LT_FLO 0.5394 0.5002 0.4351 0.4852 0.5518 0.5023 0.5236 0.5214

MAD 0.5271 0.5355 0.5151 0.5935 0.5510 0.5444 0.5420 0.5569

MVCFT 0.5203 0.4670 0.5093 0.5617 0.5443 0.5205 0.5307 0.5462

NSAMF 0.5428 0.5718 0.4962 0.6323 0.5834 0.5653 0.5627 0.5667

PKLTF 0.4667 0.4867 0.4717 0.5650 0.4428 0.4866 0.4692 0.4700

SHCT 0.6035 0.5815 0.5381 0.6390 0.5834 0.5891 0.5903 0.5883

SKCF 0.5257 0.5488 0.4627 0.6161 0.5524 0.5411 0.5380 0.5514

Staple_plus 0.5937 0.5883 0.5180 0.6617 0.5970 0.5917 0.5921 0.5911

	CM	DC	MC	O	SC	Mean	Weighted Mean	Pooled
SPMDNet	0.6402	0.536	0.6207	0.6051	0.6789	0.6162	0.6454	0.6461
MDNet	0.6023	0.5368	0.6145	0.6366	0.6565	0.6133	0.6331	0.6425
BDF	0.4341	0.5199	0.4482	0.4312	0.3721	0.4411	0.4146	0.4122
BST	0.4483	0.4652	0.4527	0.5444	0.5391	0.4900	0.4945	0.5110
DAT	0.5598	0.5288	0.4808	0.5732	0.5632	0.5412	0.5512	0.5676
deepMKCF	0.6226	0.6119	0.5917	0.6067	0.6432	0.6152	0.6245	0.6220
DPT	0.5774	0.5597	0.4830	0.5769	0.4866	0.5367	0.5296	0.5271
DSST	0.5697	0.5731	0.5551	0.6315	0.6138	0.5886	0.5914	0.6017
FCT	0.415	0.4153	0.4493	0.3510	0.4330	0.4127	0.4196	0.4277
GGT2	0.5536	0.4941	0.5454	0.5788	0.5818	0.5507	0.5641	0.5683
LOFT_Lite	0.2507	0.3400	0.2498	0.2617	0.2471	0.2699	0.2533	0.2645
LT_FLO	0.5394	0.5002	0.4351	0.4852	0.5518	0.5023	0.5236	0.5214
MAD	0.5271	0.5355	0.5151	0.5935	0.5510	0.5444	0.5420	0.5569
MVCFT	0.5203	0.4670	0.5093	0.5617	0.5443	0.5205	0.5307	0.5462
NSAMF	0.5428	0.5718	0.4962	0.6323	0.5834	0.5653	0.5627	0.5667
PKLTF	0.4667	0.4867	0.4717	0.5650	0.4428	0.4866	0.4692	0.4700
SHCT	0.6035	0.5815	0.5381	0.6390	0.5834	0.5891	0.5903	0.5883
SKCF	0.5257	0.5488	0.4627	0.6161	0.5524	0.5411	0.5380	0.5514
Staple_plus	0.5937	0.5883	0.5180	0.6617	0.5970	0.5917	0.5921	0.5911

Table 2

The average scores and ranks of robustness in different attributes

	CM	DC	MC	O	SC	Mean	Weighted Mean	Pooled
SPMDNet	33.0000	2.000	2.000	5.000	20.000	12.400	19.993	52.000
MDNet	39.667	3.667	3.000	6.000	24.333	15.333	24.262	64.667
BDF	70.000	16.000	7.000	26.000	59.000	35.600	51.148	133.000
BST	48.333	17.667	9.000	15.000	46.667	27.333	38.041	108.667
DAT	55.000	9.000	12.000	16.000	51.000	28.600	42.232	109.000
deepMKCF	40.000	9.000	7.000	13.000	28.000	19.400	27.222	78.000
DPT	49.000	5.000	8.000	12.000	31.000	21.000	31.392	79.000
DSST	50.000	6.000	8.000	18.000	31.000	22.600	32.407	89.000
FCT	36.000	7.000	7.000	14.000	32.000	19.200	27.410	88.000
GGT2	59.000	9.333	9.333	13.000	44.667	27.067	40.534	102.000
LOFT_Lite	85.000	28.000	14.000	31.000	76.000	46.800	64.718	143.000
LT_FLO	83.333	12.000	18.000	17.333	78.333	41.800	63.588	153.000
MAD	54.000	7.000	6.000	16.000	36.000	23.8000	35.2766	86.000
MVCFT	43.000	6.000	5.000	13.000	29.000	19.200	28.289	78.000
NSAMF	71.000	15.000	7.000	20.000	54.000	33.400	48.907	115.000
PKLTF	51.000	7.000	13.000	13.000	48.000	26.400	39.439	121.000
SHCT	43.000	5.000	3.000	11.000	25.000	17.400	26.248	69.000
SKCF	65.000	16.000	5.000	19.000	54.667	31.933	46.747	110.667
Staple_plus	49.000	5.000	3.000	9.000	29.000	19.000	29.656	76.000

Figure 6 shows the robustness-accuracy ranking of all the trackers in different attributes. From this figure, we can see the robustness and accuracy contrastively. Our approach has an excellent performance in the AR plot. Figure 7 shows the failures and overall overlap in different attributes, from which we can see that the failures in nearly all attributes have an excellent performance. The overall overlap of our algorithm ranking first in attributes camera motion, motion change and size change, which indicate the accuracy of our tracker have been proved in different attributes.

Fig. 6

The robustness-accuracy ranking plots for five visual attributes: Camera motion, Dynamics change, Motion change, Occlusion, Size change and Empty.

Fig. 7

The Failures and Overall overlap ranking plots for five visual attributes: Camera motion, Dynamics change, Motion change, Occlusion, Size change and Empty.

As illustrated in Table 1, Table 2, Figs. 6 and 7, our algorithm has an excellent performance in both accuracy and robustness. It demonstrates that our approach has the better accuracy than all other methods, even with fewer re-initializations. It also proved that our tracker is stable in various challenging situations.

4.2.2 The expected overlap

The overall criterion expected overlap also play an important role in evaluation. The expected average overlap curve is given by the average bounding-box-overlap averaged over a set of sequences of certain length, as defined in Equation (9). As shown in Fig. 8(a), our approach is consistently better than other trackers. Expected overlap score was obtained by integrating the EAO curve over an interval of typical sequence expected overlap of 223 to 509 frames. Figure 8(b) summarizes the EAO curves and the ranking of the trackers. The ranking of expected overlap score also proved that our approach is excellent in both accuracy and robustness. The right-most tracker is the top-performing according to the VOT2016 expected average overlap values. As illustrated in Table 3, our ranking second in all the trackers.

Fig. 8

Expected overlap curve(a) and EAO graph(b) with trackers ranked from right to left.

Table 3

Expected overlap analysis

Trackers	all	Trackers	all
SHCT	0.2633	SPMDNet	0.2577
MDNet	0.2447	Staple_plus	0.2407
DSST	0.2364	MVCFT	0.2308
DPT	0.2187	deepMKCF	0.2130
MAD	0.2108	GGT2	0.1920
NSAMF	0.1920	SKCF	0.1863
FCT	0.1855	LT_FLO	0.1629
DAT	0.1618	BDF	0.1466
PKLTF	0.1410	BST	0.1399
LOFT_Lite	0.1071

4.2.3 Overlap

In this section, the evaluation based on no re-initialization, which can also be called as unsupervised evaluation. The performance of overlap of all the trackers in different attributes is shown in Table 4 and Fig. 9. Our algorithm ranking first among 18 kinds of trackers in attribute Camera motion, Motion change, Occlusion, Size change and second in Dynamics change and Empty. In general, the performance of overlap outperforms other trackers. Precision plot, which shows the percentage of image frames whose tracked location is within the given threshold distance of ground truth. Our approach shows the best performance compared with other trackers as well.

Table 4
Overlap overview in different attributes

CM DC Empty MC O SC all

SPMDNet 0.5816 0.5178 0.5069 0.6098 0.6214 0.6468 0.6729

MDNet 0.5728 0.5033 0.5443 0.6045 0.5797 0.6067 0.6018

BDF 0.3580 0.3037 0.3778 0.4208 0.3443 0.3234 0.3517

BST 0.4094 0.3122 0.4613 0.4454 0.5116 0.5044 0.4739

DAT 0.4779 0.4059 0.5064 0.4402 0.5096 0.5000 0.4986

deepMKCF 0.5524 0.4534 0.4852 0.5616 0.5489 0.5997 0.5630

DPT 0.5040 0.4867 0.4911 0.4626 0.5209 0.4538 0.4784

DSST 0.4970 0.4691 0.4714 0.5269 0.5414 0.5730 0.5399

FCT 0.3737 0.3305 0.3834 0.4328 0.3186 0.3989 0.3829

GGT2 0.4879 0.4492 0.4369 0.5268 0.5487 0.5446 0.5190

LOFT_Lite 0.2068 0.1947 0.2536 0.2269 0.2180 0.2119 0.2268

LT_FLO 0.4792 0.4227 0.4266 0.4235 0.4693 0.5072 0.4789

MAD 0.4537 0.4414 0.4695 0.4956 0.5297 0.5065 0.5013

MVCFT 0.4616 0.4034 0.4402 0.4905 0.5075 0.5084 0.4957

NSAMF 0.4481 0.3567 0.4461 0.4757 0.5440 0.5163 0.4950

PKLTF 0.4020 0.4131 0.3848 0.4332 0.5045 0.3981 0.4068

SHCT 0.5345 0.5123 0.4559 0.5264 0.5878 0.5503 0.5393

SKCF 0.4405 0.3589 0.4770 0.4465 0.5286 0.4911 0.4866

Staple_plus 0.5183 0.5183 0.4580 0.5069 0.6103 0.5593 0.5381

	CM	DC	Empty	MC	O	SC	all
SPMDNet	0.5816	0.5178	0.5069	0.6098	0.6214	0.6468	0.6729
MDNet	0.5728	0.5033	0.5443	0.6045	0.5797	0.6067	0.6018
BDF	0.3580	0.3037	0.3778	0.4208	0.3443	0.3234	0.3517
BST	0.4094	0.3122	0.4613	0.4454	0.5116	0.5044	0.4739
DAT	0.4779	0.4059	0.5064	0.4402	0.5096	0.5000	0.4986
deepMKCF	0.5524	0.4534	0.4852	0.5616	0.5489	0.5997	0.5630
DPT	0.5040	0.4867	0.4911	0.4626	0.5209	0.4538	0.4784
DSST	0.4970	0.4691	0.4714	0.5269	0.5414	0.5730	0.5399
FCT	0.3737	0.3305	0.3834	0.4328	0.3186	0.3989	0.3829
GGT2	0.4879	0.4492	0.4369	0.5268	0.5487	0.5446	0.5190
LOFT_Lite	0.2068	0.1947	0.2536	0.2269	0.2180	0.2119	0.2268
LT_FLO	0.4792	0.4227	0.4266	0.4235	0.4693	0.5072	0.4789
MAD	0.4537	0.4414	0.4695	0.4956	0.5297	0.5065	0.5013
MVCFT	0.4616	0.4034	0.4402	0.4905	0.5075	0.5084	0.4957
NSAMF	0.4481	0.3567	0.4461	0.4757	0.5440	0.5163	0.4950
PKLTF	0.4020	0.4131	0.3848	0.4332	0.5045	0.3981	0.4068
SHCT	0.5345	0.5123	0.4559	0.5264	0.5878	0.5503	0.5393
SKCF	0.4405	0.3589	0.4770	0.4465	0.5286	0.4911	0.4866
Staple_plus	0.5183	0.5183	0.4580	0.5069	0.6103	0.5593	0.5381

Fig. 9

Precision in different threshold(left) and overall overlap(right) in different attributes.

4.2.4 Evaluation on the AMCOM

In addition, we also valuated our algorithm in two sequences from the famous FLIR dataset AMCOM. The data sets are available in grayscale format and each frame is 128×128 pixels. We employ one-pass evaluation (OPE) in this subsection. The success plot and precision plot were used for evaluating the tracking result. Figure 10 shows the result of our algorithm and five excellent trackers, including MDNet [11], LSST [26], TLD [27], KCF [28], LCT [29]. For the sequence lwir_1608, our algorithm has an excellent performance on precision and ranking second for successful rate. When it comes to sequence lwir_1913, the location precision ranking second and the success rate ranking first compared with other five trackers. It also indicates that our approach works well on the AMCOM dataset. In especial, our tracker has a better accuracy when compared with MDNet on those two sequences.

Fig. 10

Comparison of DPR and OSR with the 5 excellent trackers on AMCOM.

In the experiment, we also made a qualitative evaluation of our algorithm and other trackers. Fig. 11 summarizes qualitative comparison of our approach with five trackers on two sequences from AMCOM dataset. In sequence lwir_1608, our algorithm can track the target reliable all the time, the tracking precision is excellent. When it comes to the sequence lwir_1913, the SNR of the target decrease severely from frame 139 as the target swerved to right. The low SNR result in the tracking failure of the tracker LCT, KCF and MDNet. Our algorithm can still track the target because the scale predicting can make full use of the prior information. The integrating error during the online learning has been reduced to some extent, which has crucial meaning for the performance of long-term tracking.

Fig. 11

A visualization of tracking result of our algorithm and 5 kinds excellent trackers on two sequences from the AMCOM.

4.3 Detailed analysis of SPMDNet

SPMDNet is a modified tracking algorithm based on MDNet, which predicting the target scale during the tracking by improved BP network. As we can see in the experiment above, our approach has a better precision and robustness in different attributes. The predicting sampled method of online learning can make full use of the prior information, which have crucial effect on the performance of the tracker. Our algorithm also takes advantages of the lightweight architecture of MDNet.

5 Conclusions

In this paper, a novel tracking algorithm based on MDNet was proposed. The scale prediction of target during the tracking is performed by improved BP network, which can make full use of the prior information. The utilization of scale prediction can improve the tracking precision and robustness substantially in comparison to MDNet. Extensive experimental results show that the proposed algorithm performs favorably against the state-of-the-art methods in terms of accuracy and robustness.

Footnotes

Acknowledgments

This research was supported by the National Natural Science Foundation of China (61703337) and the Aerospace Science and Technology Innovation Fund of China (SAST2017-082).

References

Michael

, Matej

, Matas

, et al., The thermal infrared visual object tracking VOT-TIR2016 challenge results, IEEE International Conference on Computer Vision (2016), 824–849.

Liaghat

and Masnadi-Shirazi

M.A.

, Aerial target detection and tracking in infrared image sequences by using morphological operations kalman filtering and elliptical representation, Electrical Engineering (2016), 1841–1847.

, Liang

, Bai

and Feng

, Detecting and tracking dim small targets in infrared image sequences under complex backgrounds, Multimedia Tools & Applications 71 (2014), 1179–1199.

Leitloff

, Rosenbaum

, Kurz

, et al., An operational system for estimating road traffic information from aerial images, Remote Sensing 6 (2014), 11315–11341.

Xin

and Tang

, Target detection and tracking in forward-looking infrared image sequences using multiscale morphological filters, International Symposium on Image & Signal Processing & Analysis (2007), 25–28.

Cao

, Wang

, Yan

, et al., Two algorithms for the detection and tracking of moving vehicle targets in aerial infrared image sequences, Remote Sensing 8 (28) (2015), 1–21.

Y.G.

, Xiao

M.Q. K.

, Zhang , et al., Aerial infrared target tracking in complex background based on combined tracking and detecting, Mathematical Problems in Engineering (2019), 1–17.

Liu

, Huang

, Fan

, et al., Infrared dim target detection and tracking based on particle filter, IEEE Chinese Control Conference (2017), 5372–5378.

Wei

, Zhao

Y.Q.

, Quan

, et al., IR target tracking based on mean shift and particle filter, Journal of Optoelectronics Laser 19 (2) (2008), 213–217.

10.

Kong

, Chen

, Gu

, et al., Particle filter-based modulation domain infrared targets tracking, Optical and Quantum Electronics 51 (13) (2019), 1–15.

11.

Nam

and Han

, Learning multi-domain convolutional neural networks for visual tracking, IEEE Conference on Computer Vision and Pattern Recognition (2015), 1–10.

12.

Krizhevsky

, Sutskever

and Hinton

G.E.

, ImageNet classification with deep convolutional neural networks, International Conference on Neural Information Processing Systems (2012), 1–9.

13.

Simonyan

and Zisserman

, Very Deep Convolutional Networks for Large-Scale Image Recognition, IEEE Conference on Computer Vision and Pattern Recognition (2014), 1–14.

14.

Wang

, Ouyang

, Wang

, et al., Visual tracking with fully convolutional networks, IEEE International Conference on Computer Vision (2014), 3119–3127.

15.

Xin

and Shuo

, Target Detection and Tracking in Forward-looking Infrared Image Sequences Using Multiscale Morphological Filters, International Symposium on Image and Signal Processing and Analysis (2007), 25–28.

16.

Mould

N.A.

, Nguyen

C.T.

, Havlicek

J.P.

, et al., Infrared target tracking with AM-FM consistency checks, IEEE Southwest Symposium on Image Analysis and Interpretation (2010), 5–8.

17.

Le Cun

, Bengio

and Hinton

, Deep learning, Nature 521 (2015), 437–444.

18.

Danelljan

, Hager

, Khan

F.S.

and Felsberg

, Convolutional features for correlation filter based visual tracking, IEEE International Conference on Computer Vision (2015), 621–629.

19.

, Huang

J.B.

, Yang

, et al., Hierarchical convolutional features for visual tracking, IEEE International Conference on Computer Vision (2015), 3074–3082.

20.

, Zhang

, Qin

, et al., Hedged deep tracking, IEEE Conference on Computer Vision and Pattern Recognition (2016), 4303–4311.

21.

Danelljan

, Robinson

, Khan

F.S.

, et al., Beyond correlation filters: Learning continuous convolution operators for visual tracking, European Conference on Computer Vision (2016), 472–488.

22.

Danelljan

, Bhat

, Khan

F.S.

and Felsberg

, ECO: Efficient convolution operators for tracking, IEEE Conference on Computer Vision and Pattern Recognition (2016), 6931–6939.

23.

Zheng

, Bo

, Liu

, et al., Improved BP neural network algorithm to wind power forecast, The Journal of Engineering (2017), 940–943.

24.

Rong

, Wang

, Yan

, et al., Research and application on improved bp neural network algorithm, IEEE Conference on Industrial Electronics & Applications (2010), 1462–1466.

25.

Liu

, Pokharel

and Principe

J.C.

, Correntropy: properties and applications in non-gaussian signal processing, IEEE Transactions on Signal Process 55 (11) (2007), 5286–5298.

26.

Wang

, Lu

and Yang

M.H.

, Least-soft threshold squares tracking, IEEE Conference on Computer Vision and Pattern Recognition (2013), 2371–2378.

27.

Kalal

, Mikolajczyk

and Matas

, Tracking-Learning-Detection, IEEE Transactions on Software Engineering 34 (7) (2011), 1409–1422.

28.

Henriques

J.F.

, Rui

, Martins

, et al., High-speed tracking with kernelized correlation filters, IEEE Transactions on Pattern Analysis & Machine Intelligence 37 (3) (2014), 583–596.

29.

, Yang

, Zhang

, et al., Long-term correlation tracking, IEEE Conference on Computer Vision and Pattern Recognition (2015), 5388–5396.