Figure/ground modeling combined with the context matching for visual object tracking

Abstract

The aim of this paper is to track objects in a sequence of surveillance video frames by providing accurate object contours. Video analysis is performed in four key phases: (i) modeling the object/background by means of the Gaussian Mixture Model and the Kalman Filter, (ii) extracting a rough contour for the object through separating the moving objects from the still background, (iii) finding the corresponding points and the object location using the shape context matching algorithm, and eventually, (iv) extracting the exact shape contour. The suggested methodology is accurate enough to track objects in terms of scale change, translation, rotation, and partial occlusion and it can be used in real-time applications. Results on different video sequences establish the capability of this technique for a variety of applications such as traffic control cameras, speedometers, CCTV security cameras, photography and etc.

Keywords

Context matching tracking gaussian mixture model kalman filter object/background modeling

1. Introduction

Visual object tracking is defined as finding the location of a favorite moving object in a range of video frames; in other words, the objective is to find a region in each frame which is occupied by the target object at any given time [1, 2]. The past thirty years have seen increasingly rapid advances in object tracking algorithms due to developments in computer technology, high quality video cameras, and growing need for automatic analysis of video frames [3]. Visual tracking is a major area of interest within the field of machine vision and it has been applied in many aspects including motion-based recognition, robot vision, video surveillance, human-computer interactions, image compression, etc. Although acceptable results have been reported under controlled situations with fixed cameras and rigid objects, these situations are quite limited in real-life applications. There are still numerous challenges in this issue, such as complex movements of objects, effects of noise, brightness variations, scale changes or rotations of the objects, partial occlusion as well as non-rigid changes of an object, complex environments, and the need for real-time analysis [4, 5, 6, 7, 8, 9, 10, 11]. An important issue in object tracking is choosing optimal, fast and reliable tracking algorithms [5].

The first serious discussions and analysis of the object tracking emerged during the 1980s considering the movement of objects as a continuous and smooth route and there was no abrupt appearance change. However, research has consistently shown that this simple assumption differs from what happens in the real world. Hereupon, recent algorithms aimed to consider abrupt appearance changes and even complicated conditions in which the object leaves the scene. In this regard, successful tracking systems should meet three important conditions [3]: (i) Robustness: the capability of tracking the intended object even in complex situations like cluttered background or complex object movement, (ii) Adaptability: the ability to adapt the system with the changes in the object shape, and (iii) Real-time processing: highly efficient algorithms together with fast equipment for live video streams.

Several approaches have been suggested in order to overcome these situations. These approaches are divided into the deterministic methods based on optimization solutions [6, 7, 8], and the probabilistic methods based on the posterior probability distribution estimation [9, 10, 11, 12, 13, 14]. In most of these tracking approaches, objects are assumed to be in an approximately rectangular or elliptical shape. However, many objects have more complex shapes; for example, the image of a hand with open fingers or the image of the body of some animals like snakes or scorpions cannot be represented by simple shapes. In some recent studies, attempts have been made to find more accurate shape estimate of complex objects [15, 16]. Ran and Malik [15] used figure/ground segmentation by means of the Conditional Random Field (CRF) to compute a mask by which they tracked the object. This technique provides a robust tracking system, which is appropriate for long video streams like a soccer match. However, they regrettably shown that the generalisability of such results to real-time applications is problematic, due to high computational complexities of the figure/ground segmentation.

The most popular tracking method is the active contour model proposed by Kass et al. [17]. The goal of this method is to get a tight contour which enfolds the object. In earlier studies, the contour energy was described by the gradient of the image. But the use of gradients made the algorithm sensitive to noise and resulted in low contrast images. Some techniques solved these problems by integrating the region based criteria into the contour models; however they still have a number of problems in use, including occlusion and cluttered background [18, 19].

Variations in the background (especially the brightness) is another important factor to affect the performance of the segmentation technique. So far, several algorithms present dynamic description for backgrounds, which improve the quality of the detection of moving objects. Some of these algorithms are modeling algorithms such as the Kalman Filter [20] and the Gaussian Mixture Model (GMM) [21, 22]. In these techniques, first, the background is modeled by which, an initial location for the moving object is estimated. Then, the context modeling algorithms determine the exact location of the preferred object [23]. In order to achieve this goal, the context matching algorithms first receive the image of the pattern and according to its features, they search for the target in another image. These techniques handle the background variations in this account.

This paper provides an effective procedure for tracking objects in video sequences, which is robust against translation, rotation, scale change and partial occlusion. In addition, due to its low computational complexity, it can be used in real-time applications and also in cluttered backgrounds. Instead of directly finding the exact contour of the object, this approach first incorporates the object/background modeling algorithm and moving object detection technique, to estimate the initial location of the object. As a result, the computational complexity is significantly decreased and the algorithm can work quite well in cluttered environments. In the second phase, the shape context matching algorithm is adopted to estimate the possible changes in the shape of the moving object including translation, rotation, and scale variations. Through this, the location of object contour is found with a good accuracy. Finally, in order to determine the more accurate contour, the Marching Squares method is employed to extract the boundaries of the intended object. Correspondingly, the methodological approach taken in this study is a mixed methodology based on figure/ground modeling and matching algorithms.

The overall structure of the study takes the form of four sections, including this introduction. Section 2 begins by laying out the steps of the proposed object tracking approach, and looks at how the modifications on the GMM, the Kalman Filter, and the context matching algorithm can improve the performance of the visual object tracking system. Section 3 presents the results of applying this tacking procedure to different videos. Finally, Section 4 gives a brief summary and critique of the findings and includes a discussion of the implication of the results to future research into this area.

2. The proposed approach

Figure 1.

The block diagram of the proposed approach. In each block, the functioning is shown in the normal font and the result is written in the boldface font.

2.1 The main framework

To date various methods have been developed and introduced to track objects in the video sequences. The shape context is one of the more practical descriptors which is fast and robust to small changes of the shape; thus a tracking method based on this descriptor can be used in real-time applications. This section presents the proposed approach integrating the figure/ground modeling and the shape context matching. The general framework includes four repeated phases for each frame. At first, the figure/ground model is found with the use of the Gaussian Mixture Model and the Kalman filter, through which the moving part of the image in determined. Then, a coarse contour is extracted using the pixel edge detection techniques. Following this, the shape context matching is recruited to localize the favorite object and determine its state in order to find the initial boundary, and finally an exact contour enclosing the preferred object is extracted by means of the Marching Squares. In fact, the objective of the first two phases is to extract a coarse contour of the object, and the goal of the two final phases is determining the location and the exact contour of the target.

Shape context is one of the most accurate matching algorithms in separating and detecting objects in images [24, 25]. Despite its high accuracy and capability of extracting the histogram characteristics, the performance of the shape context algorithm can be still affected in the presence of noises and other background disturbances. In fact, to keep the performance high, the shape context algorithm needs to be provided an initial approximate location of the object by other techniques. Hence, instead of searching the whole image, the algorithm searches only within this approximate location for the object; this can significantly reduce the effects of noises and disturbances and increase the accuracy and the speed. As mentioned earlier, in the target images, moving objects are considered as foregrounds and fixed objects are considered as background or disturbances. Once the background and fixed objects are removed, important information about the moving objects and other changes in the image can be obtained. Therefore, in the beginning, the fixed background is modeled and the changes and movements are detected using the initial model. Movement detection is one of the effective methods that can estimate the initial location of the object, without being involved in the conditions of brightness or color of the object. Since there exist various types of modeling systems for the background, one can choose an appropriate technique based on the application and the goal. The computational complexity of these systems is much smaller compared to that of techniques that use the object color characteristics for estimating the initial location of the objects. The procedure of the tracking algorithm of moving objects of our work is shown in Fig. 1, with respect to which, the following parts explain each of the main phases.

2.2 Foreground/background separation

Minimal techniques – as common methods for separating the background – create models for the background through the images that are continuously being updated. Background is considered as a part of an image which remains fixed and without considerable changes. In the pixel-based algorithms, each pixel in an image belonging to a video sequence is classified as either still or moving; and at the end, a collection of moving pixels determines the moving part represented by a Binary Movement Map (BMM). Several methods have been proposed for this purpose; nonetheless the GMM and the Kalman Filter have received considerable interest. The next part describes each of these two methods.

2.2.1 The gaussian mixture model (GMM)

One of the complicated cases in modeling the background is when the scenes contain repetitive movements such as movements of branches and leaves of trees or for example, presence of a moving flag. The GMM algorithm, proposed by Grimson and Stauffer [26], is defined for modeling background in outdoor scenes. In this method, for each pixel in the position $(x,y)$ the set of intensities in a sequence of time frames is defined as:

$\displaystyle\{X_{1},X_{2},\ldots\}=\{s(x,y,t),t=1,2,\ldots\}$ (1)

where $X_{t}=s(x,y,t)$ represents the brightness intensity of the pixel located at $(x,y)$ in the frame number $t$ , which is assumed to be generated by one of the Gaussian distributions. Hence, the probability of observing $X_{t}$ is computed as follows:

$\displaystyle p(X_{t})=\sum_{i=1}^{k}{\omega_{i,t}*\eta(X_{t},\mu_{i,t},\sigma% _{i,t})}$ (2)

where, $k$ is the number of the Gaussian components and $\omega_{i,t}$ , $\mu_{i,t}$ , and $\sigma_{i,t}$ are respectively, the mixture weight, the mean value and the covariance matrix for the Gaussian distribution $i$ in the frame $t$ and the weights are non-negative and add up to one for each $i$ . To avoid costly matrix inversion, it is assumed that the covariance matrix is represented as $\sigma_{i,t}=\sigma_{i,t}^{2}I$ with $I$ is an identity matrix. At frame $t$ , the weights are updated as follows [20]:

$\displaystyle\omega_{i,t}=(1-\alpha)\omega_{i,t-1}+\alpha.M_{i,t}$ (3)

In Eq. (3), $\alpha$ is a pre-specified learning rate. In addition, the matching function $M_{i,t}$ is one when $|X_{t}-\mu_{i,t}|\leqslant\lambda.\sigma_{i,t}$ ; and zero, otherwise ( $\lambda$ typically set to 2.5). The parameters of the Gaussian are updated as follows:

$\displaystyle\mu_{t}=(1-\rho)\mu_{t-1}+\rho X_{t}$ $\displaystyle\sigma_{t}^{2}=(1-\rho)\sigma_{t-1}^{2}+\rho(X_{t}-\mu_{t})^{2}$ (4)

In which, the value of $\rho$ is defined as follows:

$\displaystyle\rho=\alpha M_{k,t}/\omega_{k,t-1}$ (5)

In order to estimate the background model using the GMM, first the Gaussian distribution models are arranged in a descending order with respect to the rank $\omega_{k}/\sigma_{k}$ . In fact, the higher the value of $\omega/\sigma$ , the higher the possibility of the distribution model as the background; therefore, the models with higher probability for being the background are placed at the beginning. Then, the first $B$ distributions are selected to model the background. For a specified threshold $T$ , The value of $B$ is calculated as below:

$\displaystyle B=\mathop{\text{missing}}{argmin}\limits_{b}\bigg{\{}\sum_{k=1}^% {b}{\omega_{k}>T}\bigg{\}}$ (6)

For a new input image, the matching of each pixel against the Gaussian distributions are checked. If the matching distributions is among the first $B$ components, then it is considered as background. Otherwise the pixel is classified as foreground and the least probable Gaussian component is replaced by setting its mean, variance and weight respectively to the pixel intensity, some high value and a small value.

2.2.2 Kalman filter

Kalman Filter is an optimal predictor of the status of a process when the sequence of input samples has a normal distribution. The Kalman filter can be used for segmenting the background in a sequence of frames by modeling each pixel as a dynamical system. Here, it is assumed that the variations of the background is linear and the initial conditions are known. For a pixel at position $(x,y)$ of a frame number $t$ , consider $s(x,y,t)$ as the measured intensity and $\hat{s}(x,y,t)$ as the estimated intensity found by the Kalman filter from the pixel dynamical system [27]. In the foreground/background classification using the Kalman filtering, the approach is to identify a pixel in frame $t$ as the background when the difference between $s(x,y,t)$ and $\hat{s}(x,y,t)$ is small, and as the foreground once it is large. The system state for each pixel contains the estimated intensity $\hat{s}(x,y,t)$ and the estimated variety $\hat{\dot{s}}(x,y,t)$ while $s(x,y,t)$ is considered as the system input. Also, $\widetilde{s}(x,y,t)$ is the predicted intensity of the pixel at time $t$ . The predicted states are found from Eq. (7):

$\displaystyle\begin{bmatrix}\widetilde{s}(x,y,t)\\ \widetilde{\dot{s}}(x,y,t)\\ \end{bmatrix}=A\begin{bmatrix}\widetilde{s}(x,y,t-1)\\ \hat{\dot{s}}(x,y,t-1)\\ \end{bmatrix}$ (7)

And the estimated states are computed as below:

$\displaystyle\begin{bmatrix}\hat{s}(x,y,t)\\ \hat{\dot{s}}(x,y,t)\\ \end{bmatrix}=\begin{bmatrix}\widetilde{s}(x,y,t)\\ \widetilde{\dot{s}}(x,y,t)\\ \end{bmatrix}+K(x,y,t).\Bigg{(}s(x,y,t)-H\begin{bmatrix}\widetilde{s}(x,y,t)\\ \widetilde{\dot{s}}(x,y,t)\\ \end{bmatrix}\Bigg{)}$ (8)

where $A$ is the system matrix, $H$ is the measurement matrix, and $K$ is the Kalman filter gain matrix. The Kalman filter first makes a prediction and then, it modifies the predicted value of the system. In Eqs (7) and (8), the system matrix and the measurement matrix are considered constant:

$\displaystyle A=\begin{bmatrix}1&\quad 0.7\\ 0&\quad 0.7\\ \end{bmatrix},\quad H=[1\quad 0]$ (9)

And the gain matrix is found using the pre-determin-ed binary map for the input frame:

$\displaystyle K(x,y,t)=\begin{bmatrix}k_{1}(x,y,t)\\ k_{2}(x,y,t)\\ \end{bmatrix}$ (10) $\displaystyle k_{1}(x,y,t)=k_{2}(x,y,t)=\alpha.m(x,y,t-1)+\beta\big{(}1-m(x,y,% t-1)\big{)}$ (11) $\displaystyle m(x,y,t-1)=\left\{\begin{aligned} \displaystyle 1\quad\textit{if% }\ d(x,y,t-1)\geqslant T_{th}\\ \displaystyle 0\quad\textit{if}\ d(x,y,t-1)<T_{th}\end{aligned}\right.$ (12)

The parameter $T_{th}$ is a threshold and $d(x,y,t-1)$ is found as:

$\displaystyle d(x,y,t-1)=\|s(x,y,t-1)-\hat{s}(x,y,t-1)\|$

In addition in Eq. (11), $\alpha$ and $\beta$ are the rates of matching the filter with the changes of foreground and background, respectively. The pixel at frame $t$ is classified as the foreground when $d(x,y,t)$ is large and as the background when it is small, compared to the threshold $T_{th}.$

2.2.3 Binary movement map (BMM)

Once the pixels are classified as the moving or the background, a Binary Movement Map (BMM) is constructed in which moving pixels are shown in white and background pixels are represented by the black color. In the segmentation step, some pixels may be misclassified due to noises, which leads to plenty of holes and gaps in the body and the contour of the moving objects. Here, we used a median filter to reconstruct the edges and fill the holes and gaps to attain smooth surroundings in the BMM. Following this, the remaining disturbances need to be removed from the background. These disturbances could be very intense and cause serious difficulties in the matching phase. In order to remove these disturbances, those objects whose number of pixels are less than a pre-determined threshold are omitted. Thus, only big and more integrated objects will remain and the noise and disturbances will be attenuated in an acceptable manner. Since now, we simply refer to this corrected BMM as the BMM while the original one is called theinitial BMM.

Figure 2.

Edge detection of a rabbit image: (a) the image of a rabbit, (b) the extracted edges (red points), (c) the selected representative points.

2.3 Edge detection and classification

The next step is to select a set of edge points of moving objects and extract a rough estimation of the moving part. To do this, first the edge pixels of whole image are determined using one of the edge detection techniques such as the Canny method. Among these edges, only those that belong to the moving part are important and the others should be eliminated. This selection process is called the edge classification. To do this, we recruit the BMM as a mask to set on the image edge points. Hence, what remains are the edge points selected by the BMM, which belongs to the contours of the moving objects. A simple rectangle enclosing these edge points is called the rough contour. Figure 2a shows an image of a rabbit, as a simple example. Figure 2b illustrates the extracted edges of the rabbit image. Since number of the edge points is too large, next phase needs too many calculations to be performed to find the matched points; even though the accuracy increases. For this reason, a small number of the edge points will be selected as the representative points. Figure 2c shows the selected edge set of the rabbit image.

2.4 Edge pixel matching

Shape context is a feature descriptor proposed by Belongie and Malik [28] in 2000 based on the idea of measuring shape similarity and recovering point correspondences. The main objective of the shape context algorithm is to find a favorite pattern in a new image by searching for the best edge point matchings. It deserves to be mentioned that the object inside the image may be undergone some changes such as scale change, rotation or translation, which in turn makes much more difficult to find the best matches. In this part, the functioning of the matching algorithm is described by an example.

Consider a shape represented by a set $E$ containing $N$ sampled edge points from the internal or external contours of the object. A distance matrix $D_{N\times N}$ is constructed with the entry $d_{i,j}$ representing the Euclidean distance between the points $P_{i}$ and $P_{j}$ in $E$ :

$\displaystyle d_{i,j}=\|P_{i}-P_{j}\|$ (14)

Then, mean of the matrix $D$ is found according to which, $D$ is normalized as below:

$\displaystyle D=D/{\text{mean}(D)}$ (15)

Next, the entries of $D$ are quantized based on a comparison with a logarithmic multi-level threshold. To do this, first $R$ logarithmic levels are selected and sorted ascendingly; the larger $R$ , the higher the accuracy of the algorithm will be. Each of the entries of the matrix $D$ are compared to these levels and it is replaced by the number of levels that are larger than that entry value. In fact, the smaller the entry value, the allocated number will be larger.

In the next step, the angles between the points are determined in a manner similar to that for the distance and the angle matrix $\varphi$ is constructed. In Eq. (16) the entry $\varphi_{i,j}$ is the angle between the points $P_{i}$ and $P_{j}$ in $E$ :

$\displaystyle\varphi_{i,j}=\angle(P_{i},P_{j})$ (16)

Now, the values of the angles should also be quantized by considering a number of levels. This quantization is done as follows, where the $floor(x)$ rounds each element of $x$ to the nearest integer less than or equal to $x$ and $T$ is the number of angel levels:

$\displaystyle\varphi_{i,j}^{\text{quantized}}=floor\bigg{(}\frac{\varphi_{i,j}% \times T}{2\pi}\bigg{)}$ (17)

The distance and angle matrices are used to form a log-polar histogram for each point in the pattern. For example, consider the rabbit in Fig. 3a with the selected edge points in Fig. 3b. In order to form the histogram, divide the plane around each point into a grid of $R\times T$ bins in the log-polar discrete coordinates (Fig. 3c) whose origin is located at that point. Then the points in each bin is counted and considered as an entry of the $R\times T$ histogram matrix (Fig. 3d). Finally for any given edge point, there will be a $R\times T$ histogram defined to be the shape context of that point [29]. The fact is that the shape context is a robust, compact, and highly discriminative descriptor due to rich information in the distribution over relative positions.

Figure 3.

Extracting the log-polar histogram: (a) the object image, (b) the selected edge points, (c) the log-polar discrete coordinates, (d) the shape context for the sample point in Fig. 3b [28].

Figure 4.

Edge detection of a shape: (a) The search area, (b) the extracted edges, (c) the selected edge points.

Figure 5.

The matched points of the rabbit pattern. Shown the pattern in red, the true matched shape in the search area in green, and other shapes in white. Connection to the white parts are mismatches. (a) the shape context algorithm, (b) the shape context algorithm with the gradient angles cost function. (c) Determining the location of the intended shape in the target image.

Similar steps are followed for the new image to extract the shape contexts of the edge points. Suppose that the goal is to find the shape pattern of the rabbit in Fig. 4a. In so doing, at first the edges of the new image are extracted as in Fig. 4b and the representative points on the edges are determined as in Fig. 4c. Next, for every point of the pattern and the new image, the shape context is extracted. Following this, the distance between the shape context of the points of the pattern and those of the new image are found. For each point of the pattern, a point from the new image with the minimum distance of the shape contexts is selected as the matched point. Figure 5a shows how these matchings are performed. As it is clear, in most cases the algorithm found the matched points correctly; however, some errors also exist.

As it was mentioned, the shape context matching algorithm is designed for the points on the object edges. The Canny edge detector determine the gradient vector (containing the direction of the derivative vector) in addition to the location of the edge points, by which it is possible to reduce the matching errors in the shape context algorithm. Consider two points on the edges of the pattern and the target image with the gradient angles $\theta_{i}$ and $\varphi_{i}$ found by the Canny detector. The directional matching cost of these two points is defined as following:

$\displaystyle\text{Cost}_{\text{Direction}}=\|\theta_{i}-\varphi_{i}\|$ (18)

To use this measure to reduce the mismatch errors, first the context matching algorithm extracts the matched points, for which then the cost Eq. (18) is computed. If this measure is smaller than a threshold, these two points will remain matched; and if not, they will be removed from the set of matched points. Hence, a large number of incorrect matchings will be removed and the overall performance of the system will significantly improve (Fig. 5b). Once the intended shape is found, a rectangle enclosing the matched points will be drawn to represent the initial outlier (Fig. 5c).

2.5 The state extraction

On completion of finding the matched points, the general contour representing the boundary of the object should be estimated and finally the accuracy of this contour needs to be increased. To do this, first the contour of the pattern is determined. Then among the matched points, those with the most matches (matching distance less than a threshold) are selected, for which using the scale, translation and rotation operators, the state of the target object can be determined in the new image. This state helps to draw an initial contour for the found shape by transforming the pattern contour to the new image. Selecting a few best matched points reduces the computational complexity of the step. Although this method can determine the location and the contour surrounding an object, the results of the simulations show that the accuracy is not acceptable. Therefore, in this paper we determine the location and the initial contour of the intended object more accurately by intersecting this contour and the set of moving pixels. This leads to a more accurate estimate of the object location.

2.6 The exact contour

When the precise location of the object is determined, finally the Marching Squares method [30] is applied in order to draw an exact contour around the object and separate it from the surrounding environment. This is a simple method with low computational complexity that finds the boundaries of the object quite well.

3. The simulation results

In the previous section, the proposed algorithm for the visual object tracking was completely described. It was suggested to combine the shape context matching algorithm with the figure/ground modeling methods to achieve acceptable results. In this section, the simulation results of the algorithms are presented with several examples. It is demonstrated that the proposed approach is effective for tracking objects even in complicated backgrounds.

The first phase of the proposed algorithm is modeling the foreground/background and extracting the moving parts. An initial estimate of the object location should be a-priori known; the more accurate the initial estimate is, the more accurately context matching algorithm can identify and extract the matched points. In addition, by attenuating noise and disturbances, number of points which need to be examined by the matching algorithm will be reduced, and as a result the calculation load will decrease. In this paper, the first goal of the modeling is making the background robust against changes such as the environment brightness and the imaging noises. In order to remove the noise and also fill the gaps around and inside a moving object, a median filter was used to remove small sized set of pixels. In this subsection, the figure/ground modeling is evaluated and compared by three methods: (i) the initial BMM resulted from the GMM, (ii) the corrected BMM found from the GMM and (iii) the results of the corrected BMM constructed by the Kalman filter. In Fig. 6, the target object (the cell phone) is being moved by a hand in front of a camera. Here, the objective is to extract the moving parts (the cell phone $+$ the hand) from the background. Figure 7 shows the initial BMM found from the foreground/background modeling by means of the GMM. It is clear that the BMM contains a large number of noise and white spots. The existence of too much noise makes it difficult to estimate the initial location of the object and also it reduces the accuracy of edge detection techniques. Figure 8 illustrates the simulation results using the corrected BMM found from the GMM. As Fig. 8a shows, for a still scene, the BMM has not any noise and it is uniformly black. When a moving object enters the scene, the BMM is formed quite accurately without any extra noise (Fig. 8b). This modeling system has a significant effect on the performance of the next phases, especially in the phase of finding the contour of the target object. The Kalman filter can also be used to describe the background and identify the movement. This method has a lower computation complexity compared to the GMM method, and needs a smaller amount of memory. We selected $\alpha=0.0$ , $\beta=0.09$ and $T_{th}=3$ . Figure 9 shows that the Kalman filter can also lead to acceptable results.

Figure 6.

The movement scene: (a) the fixed background, (b) a frame containing moving objects.

Figure 7.

Object/background modeling using the GMM: (a) the initial BMM with no moving object, (b) the initial BMM with a moving object in the scene.

Figure 8.

Modeling the background using the masked edge points: (a) the corrected BMM when there is no moving object at the scene, (b) BMM in the presence of a moving object.

Figure 9.

The corrected BMM created by the Kalman filter in frames 212 and 250.

The shape context matching algorithm functions based on extracting log-polar histogram and looks for points in the new image matched to the pattern. In order to obtain a more accurate matching, the gradient angles are also used. Figure 5a showed the matching algorithm looking for the rabbit pattern in the new image, from which, it can be deduced that this algorithm has wrongly matched some points. Meanwhile, in Fig. 5b, the gradient angles cost function together with the context matching algorithm are used. It is evident that this technique has noticeably reduced the number of wrong matchings.

Table 1

The characteristics of the proposed approach

The parameter	The proposed method
Background modelling	GMM- Kalman filter using the corrected BMM
Noise reduction in background modelling	Noise reduction based on the remaining points
Restoration of the edges	Restoration the moving object by the median filter
Remaining noise in the background	Almost 0%
Matching algorithm	Improved shape matching context
Removing mismatches	Using the direction of the derivative vectors
Drawing the separation contour	Yes
Sensitive to scale change	No
Sensitive to translation	No
Maximum rotation angle for accurate detection	Around 30 degrees

Figure 10.

Defining the shape to be tracked and the environment: (a) the pattern of the glass, (b) the fixed background of the scene and, (c) a frame with the moving glass.

Figure 11.

Steps of finding the edge map: (a) moving part, (b) the BMM, (c) Image edge map, (d) rough contour, (e) pattern edge map.

Figure 12.

Process of determining the exact contour: (a) point correspondence with the shape context matching, (b) best matches selection, (c) finding the initial outlier around the target object (the yellow rectangular), the initial contour (the red rectangular), and the exact contour (the bold red curve).

Figure 13.

Tracking the favorite object in different frames. First row: the edges detection of the moving parts; second row: matching the points using the shape context algorithm. Third row: drawing the outline and contour around the target object.

Figure 14.

Tracking the favorite objects. First column: a vase with flowers, second column: a hand with open fingers, and the third column: a hairbrush.

Now, the functioning of the proposed algorithm is illustrated in a tracking application. The objective is to detect and track a “glass” moving by hand of a person in the target scene. The image of this glass is first given as a pattern to the tracking system (Fig. 10a). Figures 10b and c show the fixed background of the scene and a frame containing the moving object. The background scene displays a normal room with a complex environment and objects of different shapes. First, the moving part is separated from the fixed background by one of the figure/ground modeling (Fig. 11a) and the corrected BMM is found (Fig. 11b). The image edge map is obtained by means of the Canny method (Fig. 11c), which masking by the BMM gives the edge points of the moving parts, by which the rough contour is drawn (Fig. 11d). In addition, the edge map of the favorite pattern is determined (Fig. 11e).

The shape context matching algorithm is recruited to find the best matches in Fig. 11d with Fig. 11e. Then the mismatches are removed using the gradient angle cost function (Fig. 12a) and the best matches are selected (Fig. 12b) to determine the state of the object. By intersecting this outlier with the BMM, the initial contour would be extracted. Finally, by means of the Marching Squares method, the final contour of the target object is extracted (Fig. 12c). The proposed method has been applied to several complex objects. In Fig. 13 a glass is detected and tracked in different scenes and in Fig. 14 a vase containing flowers, a hand with open fingers and a hairbrush in different backgrounds are tracked.

The results of the simulations demonstrate the effectiveness of the proposed algorithm in tracking objects. In situations that the objects are rotating or in cases like the movement of a human hand with fingers opening and closing, the system still works well. The high accuracy is due to two factors: first, the figure/ground modeling with the corrected BMM results in a good estimate of the initial object location and reduces the surrounding noise. Second, using the gradient angel cost function, the mismatches are also removed. The simulations showed that the proposed system can detect and track the target object with a rotation of up to thirty degrees. Moreover, the system is robust to variations in the color and the structure of the background (such as being non-flattened or having a special pattern), which is resulted from the modeling and detecting the pattern independent of the background. The GMM is working well for multiple objects of interest even when they overlap each other and the tracking will be recovered if occlusion takes place. Meanwhile, the Kalman filter is faster and more accurate when the probability distributions are normal. Since the shape context descriptors contain rich information, they are robust against small rotation and partial occlusion. Table 1 briefly presents the characteristics of the proposed approach.

4. Conclusions and suggestions

The purpose of the current study was to propose a method for tracking objects in a sequence of video frames using the shape context matching algorithm. The main idea was using figure/ground modeling and image descriptors to determine the contour of the intended object in a video frame. The proposed approach was applied on several video sequences. The results of the simulations showed that the technique of this paper performs well in tracking objects with translation, rotation, scale change and partial occlusion. The tracking method accurately determines the contour of the object, accordingly it is appropriate for real-time applications. This paper had the following achievements: i. The GMM and the Kalman filter were combined with the context matching algorithm to detect the target object. Describing the background using the Kalman filter for high speed moving object is more appropriate, while the GMM works better in outdoor spaces or cluttered backgrounds. ii. In the foreground/background modeling, the noise was reduced and the unwanted small objects are omitted, which lead to more connected edge boundaries for the matching phase. iii. In the matching step, with the use of the gradient angel cost function the mismatches were noticeably reduced, which increases the accuracy of the resulted outline of the object. Both the Kalman filter and the GMM consider the intensity change of the background as normal distribution or mixture of Gaussian components. What is now needed is a study involving background with abnormal distributions such as Cauchy or Chebyshev. Furthermore, using color based detectors, may lead to less mismatches and therefore, the accuracy can be improved. Future studies on tracking multiple objects, considering multimodal noises, occlusions and nonlinear movement of the objects are recommended. In general, therefore, it seems that the present approach and its prospects are promising in order to achieve successful algorithms in tracking objects.

References

Ramadasan

Chevaldonne

Chateau

. Real-time SLAM for static multi-objects learning and tracking applied to augmented reality applications. Virtual Reality (VR). 2015 March; 267-268.

Han

Choi

. Shape context based object recognition and tracking in structured underwater environment. in Geoscience and Remote Sensing Symposium (IGARSS). 2011 July; 617-620.

Yang

Shao

Zheng

Wang

Song

. Recent advances and trends in visual tracking: A review. Neurocomputing. 2011; 74(18): 3823-3831.

Liu

, Shen

Feng

. Tracking objects using shape context matching. Neurocomputing. 2011; 83: 47-55.

Berclaz

, Fleuret

Turetken

Fua

. Multiple object tracking using k-shortest paths optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2011; 33(9): 1806-1819.

Ning

Zhang

Huang

Ren

Wang

Cai

. Spatially supervised recurrent convolutional neural networks for visual object tracking. Circuits and Systems (ISCAS). 2017 May; 1-4.

Zhang

Maei

Wang

. Deep reinforcement learning for visual object tracking in videos. ArXiv Preprint ArXiv:1701.08936.

Guo

Feng

Zhou

Huang

Wan

Wang

. Learning dynamic siamese network for visual object tracking. IEEE Int. Conf. Comput. Vis. 2017 October; 1-9.

Lin

Yen

. Visual object tracking via LDA. in Applied System Innovation (ICASI). 2017 May; 315-318.

10.

Abel

Kieritz

Becker

Arens

. Robust visual object tracking with interleaved segmentation. International Society for Optics and Photonics. 2017 October; 104410: 104410B.

11.

Wang

Liu

Xing

Zhang

. Visual object tracking with multi-scale superpixels and color-feature guided kernelized correlation filters. Signal Processing: Image Communication. 2018; 63: 44-62.

12.

Liu

Zhou

. Visual object tracking with online sample selection via lasso regularization. Signal, Image and Video Processing. 2017; 11(5): 881-888.

13.

Comaniciu

Ramesh

Meer

. Kernel-based object tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2003; 25(5): 564-577.

14.

Kwon

Lee

Park

. Visual tracking via geometric particle filtering on the affine group with optimal importance functions. Computer Vision and Pattern Recognition. 2009 June; 991-998.

15.

Ren

Malik

. Tracking as repeated figure/ground segmentation. Computer Vision and Pattern Recognition. 2007 June; 1-8.

16.

Yoon

Kim

Yoon

. Visual tracking via adaptive tracker selection with multiple features. European Conference on Computer Vision. 2012 October; 28-41.

17.

Kass

Witkin

Terzopoulos

. Snakes: active contour models. 1st Int. Conf. on Computer Vision. 1987 June; 259(268).

18.

Lankton

Tannenbaum

. Localizing region-based active contours. IEEE Transactions on Image Processing. 2008; 17(11): 2029-2039.

19.

Rathi

Vaswani

Tannenbaum

Yezzi

. Tracking deforming objects using particle filtering for geometric active contours. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2007; 29(8): 1470-1475.

20.

Santosh

Mohan

. Multiple objects tracking using extended kalman filter. Advanced Communication Control and Computing Technologies (ICACCCT). 2014 May; 1484-1488.

21.

Wang

Dai

. Adaptive background update based on mixture models of gaussian. in Information and Automation. 2009 June; 336-339.

22.

McLachlan

Peel

. Finite mixture models. John Wiley & Sons.

23.

Collins

Penn

Carpendale

. Bubble sets: Revealing set relations with isocontours over existing visualizations. IEEE Transactions on Visualization and Computer Graphics. 2009; 15(6): 1009-1016.

24.

Belongie

Malik

Puzicha

. Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2002; 24(4): 509-522.

25.

Pang

Liu

. Object tacking based on shape context features and particle filter. Joural of Information & Computational Science. 2012; 9(7): 1905-1911.

26.

Lee

. Effective gaussian mixture learning for video background subtraction. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2005; 27(5): 827-832.

27.

Gupta

Chaurasia

Shandilya

. Random-valued impulse noise removal using adaptive dual threshold median filter. Journal of Visual Communication and Image Representation. 2015; 26: 296-304.

28.

Belongie

Mori

Malik

. Matching with shape contexts. in Statistics and Analysis of Shapes. 2006; 81-105.

29.

Thayananthan

Stenger

Torr

Cipolla

. Shape context and chamfer matching in cluttered scenes. in Computer Vision and Pattern Recognition. 2003 June; 1.

30.

Sullivan

. Multiple material marching cubes algorithm. International Journal for Numerical Methods in Engineering. 2003; 58(2): 189-207.