Tracking objects in video-based education using an enhanced particle filter

2.1 Bayesian tracking

Visual tracking can be considered as Bayesian filtering problem. As the aim of Bayesian filtering is to estimate the system states recursively based on the incoming observations, visual tracking under the Bayesian framework refers to estimating the value of object state variable using the image observations. Thus, the task of our tracking method is to find the best state that can maximize the posterior probability density function (PDF) p (x _t |y_1:t). Here, we define the object state as a 3D vector, that is, x t = ( x t x , x t y , x t s ). The three components represent the position coordinates and scale of the object, respectively. The PDF can be acquired by the following Bayesian filter,

p ( x t | y 1 : t ) ∝ p ( y t | x t ) ∫ p ( x t | x t - 1 ) p ( x t - 1 | y 1 : t - 1 ) d x t - 1(1) where p (x _t |x_t-1) denotes the motion model that produces the predicted state based on x_t-1; p (y _t |x _t ) represents the appearance model which is used to measure the similarity between the target object and observation at the proposed state, and it is also named as likelihood model. With the computed p (x _t |y_1:t), we can obtain the estimated state value through maximum a posteriori (MAP) estimate over a sample set containing N samples, x t MAP = arg max p ( x t ( i ) | y 1 : t ) , i = 1 , 2 , . . . , N(2)

As is known that, it is difficult to obtain the analytical formulation due to the infeasibility to compute the integral in Equation (1). Thus, many real world applications resort to an approximation to p (x _t |y_1:t) using Monte Carlo sampling methods one of which is the particle filter.

The basic idea underlying the particle filter is to approximate PDF using a set of weighted particles { x t i , ϖ t i } i = 1 N, p ( x t | y 1 : t ) ≈ ∑ i = 1 N ϖ t i δ ( x t - x t i )(3) where δ is the Delta-Dirac function. The particles used in such approximation are drawn from a proposal distribution q, and the particle weight ϖ t i satisfies ∑ i = 1 N ϖ t i = 1 which can be recursively computed using the following equation, ϖ t ( i ) ∝ ϖ t - 1 ( i ) p ( y t | x t i ) p ( x t i | x t - 1 i ) q ( x t i | x t - 1 i , y t )(4)

As we have discussed in Section 1, designing an appropriate proposal distribution is of great importance for particle filter performance as well as the likelihood model. We will show the detail of our contribution in Section 3.

3.1 Motion model

In order to robustly track an object, we initially select a region to define the object which provides the prior information for tracking purpose. The motion model adopted is a second order constant acceleration model. x t = A 1 x t - 1 + A 2 x t - 2 + Γ v t(5) where A₁, A₂ and Γ are coefficient matrices. This motion model is widely adopted is the literature.

Many particle filter based tracking methods use color histogram in designing appearance model. As we have discussed above, although it is simple and efficient, color based appearance model cannot handle severe appearance changes induced by illumination change. Thus, we consider combining another efficient histogram based feature, LSH, with the color histogram in order to better modeling the object appearance change. Under the assumption that the features we adopted are independent, the overall appearance model is a product of the likelihoods of the individual features, p ( y t | x t ) = p lsh ( y lsh , t | x t ) · p color ( y color , t | x t )(6)

In our method, we use two distributions to generate particles, the UKF and the transition prior. The motivation behind this contribution contains two aspects. On the one hand, although the transition prior is simple and easy to implement, it neglects to incorporate the recent observations. The observations contain lots of valuable information of the objects. On the other hand, the UPF uses the UKF as the proposal distribution, but it has very high computational cost which prohibits its extensive application in practice. Thus we propose to combine these two proposal distribution in the particle filtering framework. We denote the transition prior as q₁ and the UKF as q₂. The final proposal distribution can be formulated as follow, q ( x t | x t - 1 , y t ) = q 1 ( x t | x t - 1 , y t ) · q 2 ( x t | x t - 1 , y t )(13)

As q₁ is the transition prior p (x _t |x_t-1), according to (6) and (13), the particle weights in (4) can be updated as follow. ϖ t ( i ) ∝ ϖ t - 1 ( i ) p lsh ( y lsh , t | x t i ) . p color ( y color , t | x t i ) q 2 ( x t i | x t - 1 i , y t )(14) When a candidate particle is being evaluated, the weight is calculated by fusing the color feature and the LSH based feature through Equation (14). Such integration of the two features can obviously make better use of the observation information in current frame, which consequently produce positive influence to the tracking results.

Figure 1 shows the schematic view of our proposed particle filter method. At time step t - 1, we use the designed proposal distribution to generate particles and calculate the particle weights. The main procedure contains the following steps:

Predicting step. Using the particle set { x t - 2 i , 1 / N } i = 1 N of time t - 2 to generate the predicted particle set { x t - 1 | t - 2 i , 1 / N } i = 1 N. This particle set is generated through the proposal distribution we designed as in (13).

The above steps will be executed iteratively until the tracking process is ended (the last frame). We summarize our proposed tracker as in Algorithm 1.

Algorithm 1 Enhanced Particle Tracker

Input:

Particle set of p (x_t-1|y_1:t-1), image based observation y _t .

Output:

Particles of p (x _t |y_1:t), estimated object state x t MAP;

1: for (i = 1; i < = N; i++) do

2: – Propagate particle x t i through the motion model to generate the predicted particle set.

3: – Compute the color feature and the LSH feature with the current frame.

4: – Calculate the particle weight ϖ t ( i ) using Equation (14).

5: end for

6: – Obtain the current particle set { x t i , ϖ t ( i ) } i = 1 N

7: – Calculate the estimated object state x t MAP using Equation (2).

8: – Resample the particle set to obtain { x t i , 1 / N } i = 1 N

9: return Particle set { x t i , 1 / N } i = 1 N, x t MAP;

4.1 Qualitative analysis

For qualitative comparison, we use 100 particles for UPF, our method and IUPF, 600 for PF and 300 samples for SSAMC by default.

Face. In this sequence, a man is moving his face quickly and suddenly changes moving direction. The tracking results are shown in Fig. 2. It is clear that our tracker could capture the face successfully with fewer particles compared to the other four trackers. The IUPF and UPF could also track the man’s face, but their performance is obviously worse than our tracker. The SSAMC and PF frequently lose the face throughout the sequence. The results on Face sequence demonstrate that our method is efficient in dealing with fast motions.

Grilface. This sequence refers to a girl face moving with rotation, pose change and partial occlusions. This is a long sequence containing different challenging factors. As shown in Fig. 3, the proposed tracker could efficiently deal with the thorny problems induced by rotation (# 177), pose change (# 306) and partial occlusions (#458 and #468). The SSAMC and IUPF show moderately good performance, but their tracking accuracy is not as good as ours. The UPF and PF show the worst performance, they cannot capture the girl face during rotation and occlusion. From these results, our tracker could efficiently solve the appearance variation induced by rotation and partial occlusions.

SeqMS. The SeqMS sequence contains a man frequently moving his hands to occlude his face frequently which leads to frequent partial and total occlusions. 300 particles are used for the methods except the PF, and 600 for the PF. Sample frames of the tracking results are shown in Fig. 4. It is shown that, when the man’s face is occluded by his hands, our method could accurately localize its position before or after occlusion. Our method shows steadily better performance. The SSAMC shows better performance in dealing with partial occlusions compared to the IUPF and UPF. However, as for total occlusions, only our method could successfully handle the problem while the other four methods fail to deal with them.

David and Singer. In this experiment, 300 particles are used for the methods except the PF, and 600 for the PF. The David and Singer sequences mainly contain severe illumination variation which causes severe appearance changes. As we have stated previously, we adopted the LSH based appearance model which can deal with illumination change efficiently. Our tracker is expected to show much better performance in these two sequences compared to the other trackers. Tracking results over the David sequence is shown in Fig. 5. It is shown that, although severe illumination change occurred throughout the sequence, our method could consistently capture the object. The SSAMC could track the object at certain frames, but its tracking accuracy is lower than our method. The IUPF, UPF and PF methods frequently lose the object because they adopt the color based appearance model which cannot deal with the severe appearance variation in complex scenarios. Figure 6 shows the tracking results on the Singer sequence. The results are quite the same as that on the David sequence. Our method shows the best performance compared to the other four methods. The results on these two sequences demonstrate that our proposed tracking method could efficiently deal with appearance changes induced by severe illumination change.

Hockey. In this sequence, the hockey player is moving with many similar players wearing the same clothes. The results are shown in Fig. 7. When the object interacts with similar players, our track could give steady and accurate tracking result (#44 and #52). The PF tracker could also capture the object when interaction occurred, but its accuracy is worse than ours. The IUPF and UPF lose the object after interaction occurred, while the SSAMC frequently lose the object throughout the sequence.

4.2 Quantitative analysis