Two-person interaction recognition from bilateral silhouette of key poses

Abstract

This work proposes a key pose based intelligent system for recognition of human interactions from video streams. In addition to interaction recognition, the task is useful for some of other applications like content based video retrieval. The main idea is to use the shape of the bilateral silhouette between the persons and analyze it using shape context descriptor, which is one of the popular shape descriptors in object recognition and matching tasks. At first, a dictionary from random samples for the whole classes is collected and the bilateral silhouette image is extracted for all samples and classes to train the low level classifier named frame classifier. Then, the frames of test sequence are compared with these samples and labeled as one class using frame classifier. Finally, a high level classifier is used to categorize the interaction as a function of predefined labels of frame sequence. We call this classifier as the sequence classifier. Because of probable errors in foreground extraction, some faults may occur in frame classification. Moreover, each interaction sequence is composed of two types of frames, which contain related or unrelated information about interaction. To tackle the problem, a normalized histogram of the frame labels is used as the action descriptor, which is robust against misclassification of some frames. This histogram is applied to a sequence classifier like random decision forests (RDF), Probabilistic Neural Network (PNN) or Support Vector Machine (SVM) to perform interaction recognition. Experimental results on SBU and UT-interaction dataset emphasize the privileged performance of the proposed method.

Keywords

Human interaction recognition bilateral silhouette key pose high level classifier low level classifier

1. Introduction

Human action recognition (HAR) is one of the important computer vision tasks due to its enormous applications like video content analysis and retrieval [9,12,16], human computer interaction (HCI) [41], healthcare systems [10] and autonomous video surveillance [13,24,44]. This is why the action recognition task has attracted a great deal of research works and researchers over the last decades. Human interaction recognition (HIR) is another type of HAR and here, the artificial intelligence system will focus on the interactions between two or more persons. Recognizing human interaction is more difficult than action recognition because of the occlusion of body parts during interaction and articulation ambiguity. While, in the most of action recognition datasets, the sequences are captured in front view of a person so that two hands are visible and there is no occlusion, in interaction recognition scenes, we usually lose half of each person’s body because we can see just one side of them during interaction. Moreover, the vastness of actions types and usual errors in interaction recognition is more than that of the action recognition. Another challenging problem of HIR is the shortage of large training databases. Since the most concentration of the researches has been on the Human action recognition, there are not enough datasets available for HIR. Most of the interaction datasets have been provided by only two actors which are not enough for a comprehensive dataset. We know that each person has his/her own motional characteristic. Therefore, in order to develop an all-inclusive dataset, we need more than two actors to play the same role. One useful dataset is SBU kinect interaction dataset, which is composed of 21 sets of two actors with 8 interactions: approaching, departing, pushing, kicking, punching, exchanging objects, hugging, and shaking hands from seven participants [43]. There are three important aspects, which make SBU a challenging dataset. The non-periodic action performed by actors is the first one, which makes hard to recognize them compared to periodic actions. The other aspects are similarity between body movements in different action categories and diversity of the actors that perform the specified actions with different execution speeds. This dataset is recorded with a $640 \times 480$ pixels resolution in a laboratory environment, which has a simple and static background. We have also used UT-interaction dataset, which is a challenging dataset containing six classes of interactions: shake hands, point, hug, push, kick and punch [35]. There are two sets of videos recorded in two different backgrounds, a parking lot (mostly static background with little camera jitter and slightly different zoom rate) and a lawn (slightly moving background e.g. tree leaves). The sample videos are taken with the resolution of $720 \times 480$ , where the height of a person is about 200 pixels. In this work, we use the bilateral silhouette shape between two persons, which provides perfect mutual information about the body pose of two actors and eliminates worthless information like backside of them. Figure 1 shows an example of bilateral silhouette shape for a boxing frame. As it is observed in the silhouette image, backside of the actors is eliminated because there is no significant information on the backside of an attacker person and a defender one. But, the informative body parts i.e. the gesture of attacker and defender is highlighted.

Fig. 1.

An example of bilateral silhouette for a boxing frame. To extract this shape from the foreground image, the actors are connected to each other by drawing a straight line from first actor’s head to the second one’s head. The space between the actors is highlighted as bilateral silhouette.

Afterward, shape context is used to compare shapes with pre-defined templates [3]. Selected frame is labeled as one of the classes and this procedure will be applied on all of the consecutive frames. Then, a sequence of labels is extracted and histogram of the labels in the sequence is computed. These histograms are classified using standard classifiers. To increase the recognition rate, classification is completed in two stages making use of frame classifier and sequence classifier. Sequence classifier is used because the frame classifier is not able to defeat the problems like bilateral silhouettes failure, classification errors and many other problems. This is why we can’t rely on frame classification. The key advantage of bilateral silhouette analysis is its low computational cost, because the system will analyze a 2D binary shape instead of 3D skeletons or color images. Speed and accuracy of the system depends on the dictionary size, image resolution and shape context matching parameters like the number of sampled points from boundary image. To speed up the algorithm, we need to create smaller dictionary of the poses or use fewer sample points for shape matching. On the other side, to achieve good performance on recognition and get a more accurate system, we should use an enlarged dictionary and more sample points. It means that there is a trade-off between real-time performance and accuracy. Unlike some of the state-of-the-art approaches, no pre-processing stages like motion segmentation and person tracking algorithms are needed. Removing such pre-processing stages helps us to prevent probable failures and errors and the feature extraction is done just in one stage. Usual techniques like Optical Flows, Oriented Gradients and Spatio-Temporal Interest Points have been frequently applied in action and interaction recognition tasks and in some of the studies, near-perfect results have been reported. But, the mentioned methods suffer from high computational complexity. Finally, we perform a number of experiments to evaluate the accuracy of shape context descriptor. We carry out experiments to evaluate the algorithm on SBU and UT dataset. We have tested three different types of classifiers in sequence classifier stage and the results are illustrated in confusion matrices. There are various types of cross-validation protocols such as Random subsampling, K-Fold and Leave-M-Out, used to split the action/interaction datasets into training and testing subsets. We choose Leave-One-Out (LOO) method in which, each experiment uses one example for testing and the remaining examples for training. LOO evaluation scheme is ideally suited for sparse datasets because it is important to train on as many training samples as possible in such datasets.

The remaining part of this paper is organized as follows: Section 2 briefly studies related works on interaction recognition and reviews the main idea of each state-of-the-art work. Section 3 presents the proposed method in details containing feature extraction and classification methods. Our experimental results on SBU and UT dataset are presented in Section 4. Section 5 concludes the paper and gives a summary of this study.

2. Related work

Many works on human action/activity recognition mainly focus on realistic datasets, which are collected from Internet or movies, while the others use artificial databases prepared for specific purposes. Accordingly, extracted features in existing related studies can be divided into three major categories based on the nature of the database: audio features, textual features and visual features. Audio features are often used in movie-based database where the sequences are parts of a movie. Textual features are exploited in realistic databases i.e. YouTube-based databases, which contain some tags and titles or movies with subtitles. Visual features are the most popular features and are used in any kinds of databases. As a subset of HAR, we can categorize the task of HIR based on image representation methods into two main classes: local representation and global representation [32]. In local representation, a bottom-up approach is employed in which the observed image is represented as a collection of independent patches. In this method, some interest points should be extracted to calculate the local patches around them. On the other hand, global representations use top-down mode, which captures the visual observation as a whole. In global representation, we have to locate the actor(s) in the first step. To do this, we can use background subtraction or person tracking methods. Then, the region of interest will be described as image descriptors. It is important to locate the actor(s) with a fine accuracy. As an exemplary global representation, recognition of two person interactions is performed using a hierarchical Bayesian network (BN) in [27]. A low level of the BN is used to estimate the tracked body parts like head, arms, legs and a high level of BN is exploited to estimate overall body pose. Afterward, processing the evolution of the poses for the multiple body parts is done by a dynamic BN. In another study, the visual motion patterns are segmented at first and a set of middle-level components are generated by clustering keypoint-based trajectories extracted from the video sequence. Secondly, spatio-temporal relationships between pairwise components are defined. These pairwise components and middle-level components are the final features, which describe the motion characteristics of a video sequence [42]. A two-level analysis framework is proposed in another work to perform human activity recognition with distributed camera sensors in different fields of view. In track-level analysis, the gross-level activity patterns are analyzed and in body-level analysis, person’s activity is analyzed in terms of the individual body parts’ coordination [28]. Natural language is a model used to define the relation between two person at a detailed semantic level and divid human interaction to single person actions composed of torso and arm/leg motions. The person’s action is explained in terms of $subject + verb + object$ and human interaction is represented in $cause + effect$ format, which is a user-friendly natural-language [26]. Another study proposed a method to extract the spatial semantics between the actors, including front, back, face to face, back to back, and left or right. They have exploited a new dataset, which is a collection of sequences of two person activities. These videos are captured from a far distance with different directions of view. They also used context-free grammar to recognize the interactions between persons [17]. Acceleration Measure Vector (AMV) is a new effective feature composed of direction and magnitude of motion and they use jerk as the temporal derivative of AMV [11]. In global representation mode, viewpoint is a very important factor, which makes it hard to apply a specific algorithm on various datasets. Unlike the global representation, local representation does not need some usual video processing steps like background subtraction and tracking. In this method, the image or video is summarized as descriptors, which are invariant to background clutter and robust against occlusions. As an example, a fusion of audio and visual features by the name of Audio-Visual Bag of Words is introduced by [21]. In the mentioned study, spatio-temporal interest points with histogram of oriented gradients (HOG) and histogram of optical flow (HOF) are exploited as visual features and the method also uses some audio features like Mel-frequency cepstral coefficients and some other simple features in the time domain. In another work, the problem of HIR on YouTube dataset is addressed by a fusion of textual information (tags), shape information (HOG) and motion information (HOF) [39]. Experimental results show an improvement in recognition rate when using a meaningful combination of the features. In fact, they take advantage of some structured textual information such as the title, description, and a set of associated tags. These kind of datasets are useful in some applications like video content analysis, automatic movie violence detection, etc. Some of the researches are based on spatial features like topological and directional relations between persons such as the person-centric method introduced in [30]. In this work, the person-centered descriptor employs attention (head orientation) and the local spatio-temporal context like HOG and HOF features in a specified neighborhood of each detected person. Then, they model global characteristics of interactions using the combination of head orientation with the relative location of people in each frame.

Several approaches combine local and global representations to improve the accuracy of recognition [14]. The method represents a model, which fuses large-scale global features and local spatio-temporal features. They used optical flow as the global feature after pedestrian detection. To describe the local features, they detected spatio-temporal interest points and computed gradient descriptors of these points.

There are also works taking advantage of dictionary and key frames to help the task of video-based datasets classification e.g. action recognition methodology introduced in [7]. Most of these key frame and key pose based approaches tackle the problems of key frame extraction using automatic methods. There are papers where the problem of key frame extraction is addressed by using visual features e.g. [37]. In this work, a method is proposed for key frame extraction based on intra-frame and inter-frame motion histogram analysis. In other words, the key frames are selected based on the complexity of motion. In another study, saliency-based visual attention model is introduced to extract key frames [20]. In a different category, a method of key frame extraction is proposed using dynamic Delaunay graph clustering by the use of an iterative edge pruning strategy on YouTube and 100 videos from the Open Video databases [19]. An earlier work, extract key frames using color histogram, moments of inertia and some visual features extracted from the correlation of RGB color channels [15]. The results reveal that the extracted key frames are close to the key frames drawn by human. We used a key pose extraction algorithm inspired by the work of [1]. This key pose extraction method is based on k-medoids clustering algorithm. The cluster medoids represent the common poses in each action and they have chance to be the key poses.

As a final point, it is realized that there are many researches concentrating on audio, visual and textual features, which are proposed in different applications of interaction recognition. The employed algorithms are different according to the application. For example realistic video categorization is quite apart from surveillance video classification or kinect video categorization. Since we use a top-down scheme, our work can be categorized in global representation group. So, major differences are expected with the studies introduced in local representation category such as [21] and [39]. The main workflow of the proposed methodology is almost like the one introduced in [7]. There are also significant differences from database and feature selection perspectives. They use a simple distance-based feature to perform action recognition, while we use shape context as a powerful descriptor on interaction recognition datasets. They use K-means clustering with Euclidean distance to extract key poses, but we use K-medoids key pose extraction method, which is more accurate and robust due to its ability to rank the potentiality of each key frame candidate [1].

Fig. 2.

An overview of our proposed approach.

3. Proposed method

3.1. Overview of methodology

In this part, the proposed algorithm will be introduced briefly in three main phases, which will be discussed in details in the next sections. Our proposed system is illustrated in Fig. 2 in a graphical abstract.

In the first step, a dictionary from the bilateral silhouettes is created (see Fig. 1). To achieve this end, some important key frames, which contain distinguished motion, are extracted. These key frames are extracted in two ways: automatic and manual.

In the frame classification stage, the extracted frames are compared with the dictionary using shape context matching methods and classified as one of these classes in (e.g. SBU) dataset: approaching (close), departing (far), pushing, kicking, punching, exchanging objects, hugging, and shaking hands, where each class label is specified with the numbers ${1, 2, \dots, 8}$ , respectively. The process is similar in the UT-interaction dataset. For each video, a sequence of these labels is generated, where the size of this sequence is equal to the number of the frames. Afterwards, the normalized histogram of the output sequence is computed and these histograms are ready to be classified using sequence classifier. Another important reason, which persuaded us to extract the histogram of output sequence, is variable length of each video sequence. By this way, the video sequence is described as a histogram, which will be ready to be classified in the next step.

As the sequence classification stage, these histograms are classified in the second classifier, which recognizes the label from the whole video. The sequence classifier will help us to classify variable-length videos in addition to defeat probable errors.

3.2. Bilateral silhouette extraction

The bilateral silhouette is a novel representation, which is robust to partial occlusions like one hand occlusion. The bilateral silhouette captures the space between the actors while the occluded body parts like left hand will be removed automatically (see Fig. 3).

Fig. 3.

Bilateral Silhouette is robust against partial occlusions, because the most important part of the scene is the space between the persons, not their occluded body parts.

Fig. 4.

Foreground extraction process using GMM clustering. There are some post-processing stages to extract the pure foreground image.

The method is also robust to small shape deformations. Shape context matching descriptor is insensitive to small perturbations as it is proven in [3]. Since we use shape context descriptor as the basis of the proposed approach, the algorithm will be robust against deformations due to the descriptor’s robustness. For example, small deformations of head does not affect while performing a kicking interaction. To extract bilateral silhouette, the background is removed and the actors are connected to each other by drawing a straight line, which connects two persons’ head. To draw the line, the topmost points of each person’s silhouette are connected to each other. Then, the spatial gap between the actors is highlighted as the bilateral silhouette, which aims to capture the relation between their actions. The foreground is extracted using a Gaussian Mixture Model (GMM) Clustering method, which gives a good result in slowly moving foreground objects. After the clustering stage, some morphological operations will be needed to get a pure foreground image. Then, the XOR image of the foreground and convex hull of the foreground image is computed as the bilateral silhouette (see Fig. 4).

3.3. Creating dictionary

Obviously, trying to find a particular data in some sort of summary is easier than searching a huge database. We have to summarize the video to save in computation time of data retrieval. Key frames extraction is the basis of video summarization and retrieval, which selects some of the most informative frames in a video to reduce the quantity of records required in video indexing. Video summarization can result some still images (key frames), or some moving images (video skims) [40]. In fact, the key frames are the selective salient frames, which contain semantic importance. On the other hand, video skims are selected meaningful video shots. Since organizing still images for further browsing is easier than video skims, we select key frames to form our dictionary. To achieve good performance on recognition, the selected frames must convey useful information and add a new shape of the desired action to the dictionary. Picking up of duplicated frames may cause a decrease in speed without any advance in accuracy. There are two ways to extract a meaningful dictionary from samples of whole classes: Automatically selected frames and manually selected frames.

In this work, we extract the key poses using both methods and compare the results of the methods. In the manual method, the computational complexity is less than the automatic methods. The selected frames should be different and number of frames per sample depends on diversity of human poses while performing interaction. To illustrate the efficiency of shape context based frame classifier, we separated training data from the test data, which means that the dictionary will not contain key frames from all of the samples. SBU dataset consists of 21 subsets in which each subset is performed by a different pair of actors. We selected about 40% of the subsets to learn the dictionary and no key frames are extracted from the remaining subsets of the dataset. Eight subsets for eight classes are selected to learn the dictionary (see Table 1). Similarly, on UT dataset, there are ten sequences for each class performed by different actors. We selected seq1 to seq4 of each class to learn our dictionary (see Table 1).

Table 1
Subsets used for training and testing dictionary

Database Subsets used for training dictionary Subsets used for testing dictionary

SBU interaction s01s02 s02s03 s02s06 s01s03 s01s07 s02s01

s03s04 s04s02 s04s06 s02s07 s03s02 s03s05

s06s02 s07s01 s03s06 s04s03 s05s02

s05s03 s06s03 s06s04

s07s03

UT-interaction Seq1 to Seq4 Seq5 to Seq10

Database	Subsets used for training dictionary	Subsets used for testing dictionary
SBU interaction	s01s02 s02s03 s02s06	s01s03 s01s07 s02s01
s03s04 s04s02 s04s06	s02s07 s03s02 s03s05
s06s02 s07s01	s03s06 s04s03 s05s02
	s05s03 s06s03 s06s04
	s07s03
UT-interaction	Seq1 to Seq4	Seq5 to Seq10

Fig. 5.

Some of important frames are selected for each class and the collection of these frames will form the dictionary. All of the key frames will participate in the classification stage.

The size of the dictionary used in frame classifier has been experimentally adjusted to 273 key frames in SBU dataset. We have also selected seq1 seq4 from each set in UT-interaction dataset, which will give 149 key frames. An example of dictionary for three samples is shown in Fig. 5.

Since we don’t have approaching and receding classes in UT-interaction dataset, we have modeled these classes. In fact, we have eliminated these frames which occur before and after the desired interaction (see Fig. 6). Using a distance criterion, we can recognize if the actors are standing far or close. We can also determine if they are getting closer or they are moving in the opposite direction.

In automatic key frame extraction, the per class key poses are obtained by k-medoids clustering with Euclidean distance similar to [1]. These key poses will apparently demonstrate important parts of the interaction sequence. Automatic key frame extraction is usually applied on the action databases, which have just one action per sample. On the other side, we know that occurring of interaction needs more than one actor and the actors should get close, interact and probably depart. We know that, in an interaction databases, each interaction consists of three kinds of actions: approaching, performing interaction, departing. This is what happens in reality. As a result, in order to obtain key frames from interaction databases (e.g. SBU), we need to eliminate approaching and departing key frames from each category. To overcome this problem, we use shape context descriptor to compare the key frames of the categories ${3, \dots, 8}$ with departing and approaching categories ${1, 2}$ and remove the frames, which have similar shape context descriptors. Shape context description process is followed by a matching technique, which is explained in Section 2.4. The automatic key frame extraction is implemented only on SBU dataset to show difference between the results of an exemplary automatic key frame extraction with the manual one. This key frame learning process is repeated individually for training the samples of each category in training subset to extract a set of k key frames for each interaction class. Number of the clusters (k) in SBU dataset is set to 40, which gives 320 words for the dictionary, but 220 key frames will remain after removing similar frames. Algorithm 1 illustrates the automatic key frame extraction method. Once the dictionary is made, each interaction sequence can be converted to a collection of key frames.

Algorithm 1

Finding key poses

3.4. Shape context based frame classification

Developing of the shape matching and recognition algorithms has gained great interest in recent years. An early study have introduced a 2D shape matching method to measure similarity between shapes, which is used in many applications like object recognition [3]. Because of its efficiency, the method has been extended to 3D shape context in [18]. The descriptor is invariant to translation, scaling and occlusions as it is indicated in [3]. It is also robust to small geometrical distortions and presence of outliers and can be made rotation invariant using local tangent orientation. Here is a brief review of shape context shape matching:

Fig. 6.

Approaching and departing classes are defined in UT-interaction dataset.

Fig. 7.

Shape context descriptor (down) is computed on the selected points from boundary image (up). The descriptor is a $60 \times 100$ matrix. 100 is number of the selected points in the boundary and 60 is number of regions defined in shape context log-polar histogram, which has twelve angles and five radius bins. In shape context, similar shapes have close descriptions.

In the first step, the bilateral silhouette is represented as a discrete set of points, which are sampled from the external or internal edges on the binary image: $\begin{matrix} (1) & P = {p_{1}, p_{2}, \dots, p_{n}}, p_{i} \in R^{2} \end{matrix}$

Therefore, after foreground extraction using current methods, contour of the bilateral silhouette is extracted using a robust edge detector like canny algorithm [6]. The sampled contour points should be selected always in the same order. For example, the topmost point will be considered as the first sample and the next points will be selected in a clockwise order. Then, a coarse histogram for each point $p_{i}$ is defined as $h_{i}$ (see Fig. 7): $\begin{matrix} (2) & h_{i} (k) = # {q \neq p_{i} : (q - p_{i}) \in bin (k)} \end{matrix}$

This histogram is called shape context of pi, which is computed in $n_{θ}$ uniformly spaced angle bins (direction), and $n_{r}$ uniformly spaced log-radius bins (length). Log-polar histogram gives more importance to the points, which are closer to the desired point than remaining ones and this is the most important advantage of it. In this work, we use five radius bins and twelve angle bins, which will create 60 regions for each boundary point. As a result, the output matrix will be $n \times 60$ where n is the number of selected points on the boundary. Sampling of boundary will reduce the computational complexity and by selecting an accurate sampling ratio, sampling process will not deteriorate the results.

After extracting shape context histogram, the cost of matching point $p_{i}$ on the first shape to point $q_{j}$ on the second shape is defined using chi-square distance: $\begin{matrix} (3) & C_{i j} = \frac{1}{2} \sum_{k = 1}^{K} \frac{{[h_{i} (k) - h_{j} (k)]}^{2}}{h_{i} (k) + h_{j} (k)} \end{matrix}$

Where, $h_{i} (k)$ and $h_{j} (k)$ are normalized histograms on $p_{i}$ and $q_{j}$ . Selected match from $C_{i j}$ should minimize the total cost of matching as follows: $\begin{matrix} (4) & H_{(π)} = \sum_{i} C (p_{i}, q_{π (i)}) \end{matrix}$

Subject to the constraint that the matching is one to one, i.e., π is a permutation [2]. In low level classification where each query frame is compared with the samples of dictionary, a simple voting scheme will be applied to select the most relevant action label to the query frame.

We select 100 points from the boundary, which are distributed uniformly. The output of shape context is the number of points matched to the desired point. To perform classification, each query frame is compared with the key frames of each eight classes (Fig. 5) and the number of matched points is saved. For each interaction in the dictionary, ten greatest values of the saved numbers are selected and averaged. The output label is the label with maximum quantity of these eight values. As a result, this step converts a video sequence to a sequence of labels for eight classes (see Algorithm 2). One significant aspect of our proposal is the computation of the output sequence histogram. The normalized histogram of the labels is computed to show the distribution of action labels in a video sequence. Figure 8 illustrates the normalized histograms for actions performed in Fig. 9. Samples in Fig. 9 are ideal samples and obviously there will be errors in low level classification step, but the second stage of classification will defeat these noises and misclassifications. For example in boxing class, some of the frames may be classified as giving something or punching class and misclassification percentage depends on dictionary richness.

Algorithm 2

Frame classification

Fig. 8.

Histogram of classified output labels for the samples in Fig. 9. Non-normalized histograms are represented to show number of labels.

Fig. 9.

Low level classification results for one sample per class. After extraction the output sequence, histogram of the labels will be computed as a feature.

3.5. Sequence classification

All of the samples in interaction datasets are composed of different interactions. For example, a kicking sequence is composed of approaching, kicking and receding classes with a partly specified distribution. So, we need another classifier to employ these data and recognize the interaction label from the histograms. In fact, recognizing an interaction in a video sequence using just one frame is an ineffective effort. Moreover, there may be errors in frame classification block between similar classes like hand shaking and giving object. In order to defeat noises and misclassifications, a sequence classifier is used in the next stage. To do this, three kinds of current classifiers are trained: random decision forests (RDF), probabilistic neural networks (PNN) and support vector machines (SVM).

Probabilistic Neural Networks: PNN is one of the well-known models used for supervised classification introduced in [38]. The main idea of PNN is based on the theory of Bayesian decision strategy and non-parametric estimation of probability density function (PDF) according to Parzen’s non-parametric estimator [29]. As a classifier, the most important advantage of PNN is its fast training process so that we can say there is no real training process. In this work, we employed PNN as the sequence classifier because of its insensitivity to wild points caused by frame classifier.

Support Vector Machines: SVM is the other supervised classification method applied in our approach based on [4]. Using kernel functions e.g. Radial Basis Functions (RBF), SVMs are able to classify more complex problems, which need non-linear separation boundaries. There are some advantages of SVM, which persuaded us to use this classification method in our approach:

Different kernel function can be defined to form a more flexible classifier, which can separate non-linearly Separable Problems.

SVM works well with fewer training samples in comparison with other classifiers. This is an important feature in video processing applications e.g. HIR, because the classifier will be trained to further classifications (after testing process), which makes test samples much larger than training samples.

Random Decision Forests: RDFs are predictive models designed for classification and regression tasks, which can use a set of binary rules to compute a target value [5]. In this study, we use random forests as sequence classifier, which can give acceptable results with a high classification speed.

4. Experimental results

In this section, we present the experimental results of our approach tested on SBU human interaction dataset and UT-interaction dataset. We don’t aim to compare our classification results on SBU dataset with the state-of-the-art studies (except [43], which uses the same dataset as we use), because the most important factor, which allows us to compare two algorithms is the ability of testing the methods on the same dataset. The experiments demonstrate the effectiveness of our proposed method for two-person interaction categories in SBU dataset. To the best of our knowledge, SBU dataset was only used in [43]. The mentioned research used depth images of the dataset as the main data. Since SBU dataset has RGB and Depth video sequences separately, we used RGB data as the main information. We have also tested and compared the results obtained in our experiments on UT-interaction dataset with recent results published on this dataset.

4.1. SBU interaction dataset

In a PNN classifier, a key parameter to control the learning process is spread value (σ). Picking a large spread value may cause loss of details and then, the model will not be able to fit the function accurately. On the other hand, if the selected spread value is too small, over fitting will occur. To find an optimal σ, a set of values from 0.1 to 4 were examined. From Fig. 10, one can see that the small spread values give better classification accuracy and the success rate slowly starts to fall as the spread value increases.

Fig. 10.

This graph shows the change in the recognition accuracy on the SBU dataset with respect to spread value in RBF kernel of PNN classifier. The highest accuracy is obtained when we set the spread value to 0.1 and the performance decreases fast when the value is larger than 0.1.

A PNN with spread value (σ) of about 0.1 had a good performance. The testing process was repeated 30 times for each value of σ. We selected 0.1 as σ step because with smaller steps (e.g. 0.01), there was not a significant change in recognition rate and it was quite stable.

Figure 11 contains the results of a LOO application for SBU dataset using PNN classification.

Fig. 11.

Confusion matrices of PNN classification using LOO cross-validation. The average of all categories we achieve in SBU dataset is 93.09% with manual dictionary extraction (left) and 80.86% with automatic dictionary extraction (right).

Fig. 12.

Confusion matrices of SVM classification using LOO cross-validation. The average of all categories we achieve in SBU dataset is 95.05% (left) using a third degree polynomial kernel in manual dictionary extraction, which is the highest recognition rate in our approach. Recognition rate in automatic dictionary extraction is 87.23% (right).

In each cross-validation round, one video sequence containing two particular actors are used as test set, while the rest of the video sequences are used to train the classifier. The results show the average performance (success rate %) of 93.09% and 80.86% for manual and automatic key frame extraction methods, respectively. Accuracy is defined as the ratio of the correct classified samples to all of the samples. The results are obtained after completion of about 150 training and testing trials.

We also train a one-versus-one multiclass SVM classifier using the LibSVM toolbox, which supports well-known kernel functions [8]. Best results were obtained by a third degree polynomial kernel. Figure 12 illustrates the results of SVM classification using a third order polynomial kernel.

The average interaction recognition accuracy for the eight classes of interactions with manually selected key frames is 95.05%, where the recognition rate achieves 87.23% with automatically extracted dictionary. As illustrated in Fig. 12, three classes are classified with a success rate of 100% and the other categories are classified with a correct recognition rate of over 91%, except boxing interaction. The worst performance was noticed in the last category, which is about 87%.

Fig. 13.

Confusion matrices of SVM classification using LOO cross-validation. The average of all categories we achieve in SBU dataset is 94.11% using a RBF kernel in manual dictionary learning mode (left) and 82.66% in automatic dictionary learning technique (right).

The average recognition accuracy of RBF kernel with manually selected key frames is 94.11%, which is close to the results obtained by polynomial kernel (see Fig. 13). In SVM with RBF kernel, receding is confused with approaching (the lowest recognition rate) since they have similar bilateral silhouettes. It can be observed from Figs 12 and 13 that the SVM classifier, which uses polynomial kernel, performs much better than the SVM classifier using RBF kernel. As explained before, various kernel functions have been developed for SVMs like neural network kernels, spline functions, polynomial functions with different degrees, RBF kernels, and Dynamic Time Warping (DTW) kernels. We have tested the effects of different kernels on the dataset using SVM classification.

Using RDF as a classifier, the method achieved accuracy of 93.17% using manually selected dictionary. The same setting as PNN and SVM classifiers (LOO cross validation) is used to test proposed methodology on interaction recognition using SBU interaction dataset. Most of confusions are due to similarity of the performed interactions and confused classes are similar to the confusions in SVM and PNN classifiers. Figure 14 indicates the results obtained by a RDF with 500 trees.

Fig. 14.

Confusion matrices of RDF classification using LOO cross-validation. The average of all categories we achieve in SBU dataset is 93.17% using 500 trees in manual dictionary learning mode (left) and 87.77% in automatic dictionary learning technique (right).

Our method outperforms the algorithm introduced by [43], which has the average recognition rate of 91.1% using MILBoost classifier, whereas our proposed method achieved a rate of 95.05% on the same dataset. Note that they use depth information and extract an articulated skeleton for each person, which is a more complex and time consuming feature in comparison with our 2D shape context feature. Then, they define some joint, plane and velocity features based on the extracted skeleton in the feature extraction stage. They model a MILBoost classifier for each action with true positive and true negative instances. The mentioned classifier is a well-known model in which each instance is classified by a linear combination of weak classifiers. This model is a more complex and exhaustive classifier than our two stage classifier, but we have concentrated on the feature extraction process to get a faster classification procedure, which may help us to perform real-time classification. In the classification procedure, some errors occur when receding action is confused with the approaching action (1st and 2nd classes). This is because of the similarity between some of transition frames while switching classes from far to close, or vice versa. It means that there is not decisive definition to separate the statements far and close. Since the confusion is caused in feature extraction block and it does not depend on the classifier type, we expect to have similar errors while using any type of classifiers. In some cases, confusion occurs between hugging and receding/approaching due to the similarity of the bilateral silhouette. In some cases, there are misclassifications between punching and boxing actions (4th and 8th classes). One possible reseaon is that some frames in the mentioned classes may not visually be separable. The base movement in these interactions is the first actor’s hand moving toward the second actor. The problem also happens between shaking and giving something actions (5th and 7th classes).

Fig. 15.

Confusion matrix for foreground silhouette images with manual dictionary extraction method.

Fig. 16.

Confusion matrices of set 1 and set 2 for UT-interaction dataset using RDF classifier.

To illustrate the effectiveness of the bilateral silhouette in comparison with the foreground silhouettes of the persons, we tested the shape context matching algorithm on pure foreground silhouette images. A new dictionary is defined using the foreground images and the low-level classification is done by matching the query foreground images with dictionary samples. In fact, the process is the same as the method explained before. But instead of bilateral silhouettes, the foreground binary image is exploited (left part of Fig. 1). The best recognition rate in sequence classifier was obtained by 2nd order polynomial kernel in SVM classifier, which is about 77%. The confusion matrix is reported in Fig. 15.

4.2. UT-interaction dataset

We have tested our proposed algorithm on UT-interaction dataset using PNN, SVM and RDF classifiers, but we have reported confusion matrices for the best results, which are obtained from the random forests classifier. All of the stages are similar to SBU dataset evaluation process and the validation details are same as the facts discussed in SBU dataset section. The average recognition accuracy for UT-interaction dataset using PNN classification with a LOO setup with the manually selected dictionary is 88.33% (90.25% for set 1 and 86.41% for set 2). Using SVM classifier, the average recognition accuracy with RBF kernel (offering highest results among other SVM kernels) is about 81.67% (84.2% for set 1 and 79.14% for set 2). These recognition rates are an average value of ten runs. Figure 16 illustrates the results of RDF classifier for set 1 and set 2, separately. In set 1 (left), the confusion occurs between classes 4 (boxing) and 5 (punching). In set 2, interaction 7 (shaking hands) is added to the above mentioned confusions. Obviously, this is because of similarity between performed interactions. The results demonstrate that the method can achieve 95% accuracy for set 1 and 90% accuracy for set 2.

Table 2
Comparison of the correct recognition rate in (%) for different methods using two datasets

Method SBU dataset UT-interaction dataset Avg

Dominating pose doublets [25] 91.67 81.67 86.67

Middle level components [42] – – 78.2

Structured learning [30] 84 86 85

Spatio-temporal interest points [23] 86 88 87

Dynamic bag of words [34] – 85 –

Poselet key-framing [33] 93.3 – –

Discriminative key-component model [36] 93.3 90 91.7

$Proposed algorithm + SVM (RBF kernel)$ 84.2 79.14 81.67

$Proposed algorithm + PNN$ 90.25 86.41 88.33

$Proposed algorithm + Random forests$ 95.05 90 92.5

Method	SBU dataset	UT-interaction dataset	Avg
Dominating pose doublets [25]	91.67	81.67	86.67
Middle level components [42]	–	–	78.2
Structured learning [30]	84	86	85
Spatio-temporal interest points [23]	86	88	87
Dynamic bag of words [34]	–	85	–
Poselet key-framing [33]	93.3	–	–
Discriminative key-component model [36]	93.3	90	91.7
$Proposed algorithm + SVM (RBF kernel)$	84.2	79.14	81.67
$Proposed algorithm + PNN$	90.25	86.41	88.33
$Proposed algorithm + Random forests$	95.05	90	92.5

Table 2 summarizes the results achieved by some of the state-of-the-art studies using UT-interaction dataset. Some of these studies are person-centric methods, which may need some complex pre-processing stages e.g. person detection and tracking, making the interaction recognition process too slow [25,30,36]. But we just need a simple foreground extraction as the pre-processing stage. Some of the proposed methods use bag of words algorithm combined with other features such as spatio-temporal features to represent the video sequences [23,34,42]. Such features are fast enough but they need a post-processing stage to refine the features, while our proposed shape context feature does not need any post-processing. These three methods have also used SVM-based classifiers, which is similar to our approach. The results show that our proposed algorithm, which gives the highest recognition rate on SBU dataset (95.05%), is also among the best results obtained by the state-of-the-art studies on UT-interaction dataset. The most common misclassifications are between boxing, pushing and shaking categories. The probable reason is visual similarity of the interactions, which have been discussed in SBU dataset.

5. Computational efficiency

Most of the action/interaction recognition applications require real time or at least a fast processing procedure. In terms of time complexity, in the proposed method, the frames will be compared to the dictionary using a point matching algorithm, which takes about 0.007 s to get the matching cost parameter ( $C_{i j}$ , see equation (3)). It also takes about 0.02 s to extract the shape context descriptor ( $h_{i} (k)$ , see equation (2)) from a bilateral silhouette. We also need a mean time of 0.011 s to extract a foreground silhouette and 0.006 s to make a post processing on the output image and convert it to a bilateral silhouette. Our evaluation system consists of a standard computer (Intel(R) Core I5 running with 2.50 GHz and 8 GB of RAM), running Windows 7 64-bit and an implementation using MATLAB. To compare the average computing time of the major components involved in different algorithms with our proposed algorithm, we can mention the HOG3D feature as an example of appearance-based features, which is a combination of Histograms of Gradients into orientation tensors introduced in [31]. The descriptor was computed with an average of 23 frames per second, which takes about 0.043 s per frame. In another study, a complete exemplary of motion-based features is defined, which is a multi-scale descriptor for human action recognition from a Pyramid of Accumulated Histograms of Optical Flow [22]. It is reported that it may take about 515.1 ms to compute the feature in the first step, which may be less than this value in the next steps. The authors have measured the following times for each frame: 3.1 ms for image resizing, 6.5 ms for optical flow computation, 15.9 ms for HOF computation and 2 ms for accumulation plus normalization. Note that since the computational complexity in the state-of-the-art studies about human interaction recognition was not discussed, we used the features in human action recognition to compare our feature extraction time. In fact, we just want to compare computational time of the different methods not the results obtained by them.

6. Conclusion and future work

In this paper, we introduced a bilateral silhouette based interaction representation and explored its capability in two person interaction recognition. A binary shape called bilateral silhouette has been extracted, which captures the essential information of each actor’s body pose. Also, a label has been assigned to each frame using a Shape Context matching method. The problem has been formulated as a supervised learning task using PNN, RDF and SVM. The bilateral silhouette based interaction recognition method achieved 95.05% for SBU dataset and 92.5% for UT-interaction dataset as the average rates. Experiments demonstrate the discriminative power of the new methodology for interaction recognition and show that our method can be a reliable solution to such video processing applications. The main strong point of our proposed method is the fast solution, which uses RGB data instead of Depth or Motion Capture information. It is obvious that using RGB images needs cheap and simple hardware, which is a simple camera and in the proposed method, the resolution and quality of the camera is not so important. Using RGB image is also of great potential to attain substantial computational saving compared with depth images. In the future study, we intend to discover the possibility of using similar features to speed up the process in order to have a prevision of the action. We can also use depth data from SBU interaction dataset to improve our classification accuracy.

References

Baysal,

M.C.

Kurt and

Duygulu, Recognizing human actions using key poses, in: 20th International Conference on Pattern Recognition, ICPR 2010, Istanbul, Turkey, August 23–26, 2010, pp. 1727–1730.

Belongie and

Malik, Matching with shape contexts, in: Proceedings of the IEEE Workshop on Content-Based Access of Image and Video Libraries, Hilton Head Island, SC, 2000, pp. 20–26.

S.J.

Belongie,

Malik and

Puzicha, Shape matching and object recognition using shape contexts, IEEE Transactions on Pattern Analysis and Machine Intelligence 24(4) (2002), 509–522. doi:10.1109/34.993558.

B.E.

Boser,

Guyon and

Vapnik, A training algorithm for optimal margin classifiers, in: Proceedings of the Fifth Annual ACM Conference on Computational Learning Theory, COLT 1992, Pittsburgh, PA, USA, July 27–29, 1992, pp. 144–152.

Breiman, Random forests, Machine Learning 45(1) (2001), 5–32. doi:10.1023/A:1010933404324.

Canny, A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence 8(6) (1986), 679–698. doi:10.1109/TPAMI.1986.4767851.

A.A.

Chaaraoui,

Climent-Pérez and

Flórez-Revuelta, Silhouette-based human action recognition using sequences of key poses, Pattern Recognition Letters 34(15) (2013), 1799–1807. doi:10.1016/j.patrec.2013.01.021.

C.-C.

Chang and

C.-J.

Lin, LIBSVM: A library for support vector machines, ACM Transactions on Intelligent Systems and Technology 2(3) (2011), Article ID 27.

L.-H.

Chen,

C.-W.

Su,

C.-F.

Weng and

H.-Y.M.

Liao, Action scene detection with support vector machines, Journal of Multimedia 4(4) (2009), 248–253. doi:10.4304/jmm.4.4.248-253.

10.

Cheng,

Liu,

Zhao and

Ye, Real world activity summary for senior home monitoring, in: Proceedings of the 2011 IEEE International Conference on Multimedia and Expo, ICME 2011, Barcelona, Catalonia, Spain, July 11–15, 2011, pp. 1–4.

11.

Datta,

Shah and

N.D.V.

Lobo, Person-on-person violence detection in video data, in: 16th International Conference on Pattern Recognition, ICPR 2002, Quebec, Canada, August 11–15, 2002, pp. 433–438.

12.

F.D.M.

de Souza,

G.C.

Chávez,

E.A.

do ValleJr. and

de A. Araújo, Violence detection in video using spatio-temporal features, in: SIBGRAPI 2010, Proceedings of the 23rd SIBGRAPI Conference on Graphics, Patterns and Images, Gramado, Brazil, August 30–September 3, 2010, pp. 224–230.

13.

Dedeoglu,

B.U.

Toreyin,

Gudukbay and

A.E.

Cetin, Surveillance using both video and audio, in: Multimodal Processing and Interaction,

P.M.

Maragos,

A.P.

Potam and

P.G.

Patrick, eds, Springer US, 2008, Chapter 10, pp. 1–13. doi:10.1007/978-0-387-76316-3_6.

14.

Dong,

Kong,

Liu,

Li and

Jia, Recognizing human interaction by multiple features, in: First Asian Conference on Pattern Recognition, ACPR 2011, Beijing, China, November 28–28, 2011, pp. 77–81. doi:10.1109/ACPR.2011.6166533.

15.

Ejaz,

T.B.

Tariq and

S.W.

Baik, Adaptive key frame extraction for video summarization using an aggregation mechanism, Journal of Visual Communication and Image Representation 23(7) (2012), 1031–1040. doi:10.1016/j.jvcir.2012.06.013.

16.

Gong,

Wang,

Jiang,

Huang and

Gao, Detecting violent scenes in movies by auditory and visual cues, in: Advances in Multimedia Information Processing – PCM 2008, 9th Pacific Rim Conference on Multimedia, Tainan, Taiwan, December 9–13, 2008, pp. 317–326.

17.

Jin,

Hu and

Wang, Human interaction recognition based on transformation of spatial semantics, IEEE Signal Processing Letters 19(3) (2012), 139–142. doi:10.1109/LSP.2012.2184091.

18.

Körtgen,

G.J.

Park,

Novotni and

Klein, 3D shape matching with 3D shape contexts, in: Proceedings of the 7th Central European Seminar on Computer Graphics, Budmerice, Slovakia, 2003.

19.

S.K.

Kuanar,

Panda and

A.S.

Chowdhury, Video key frame extraction through dynamic Delaunay clustering with a structural constraint, Journal of Visual Communication and Image Representation 24(7) (2013), 1212–1227. doi:10.1016/j.jvcir.2013.08.003.

20.

Lai and

Yi, Key frame extraction based on visual attention model, Journal of Visual Communication and Image Representation 23(1) (2012), 114–125. doi:10.1016/j.jvcir.2011.08.005.

21.

M.J.

Marín-Jiménez,

Muñoz-Salinas,

Yeguas-Bolivar and

Pérez de la Blanca, Human interaction categorization by using audio-visual cues, Machine Vision and Applications 25(1) (2014), 71–84. doi:10.1007/s00138-013-0521-1.

22.

M.J.

Marín-Jiménez,

Pérez de la Blanca and

Mendoza, Human action recognition from simple feature pooling, Pattern Analysis and Applications 17(1) (2014), 17–36. doi:10.1007/s10044-012-0292-8.

23.

M.J.

Marín-Jiménez,

Yeguas and

Pérez de la Blanca, Exploring STIP-based models for recognizing human interactions in TV videos, Pattern Recognition Letters 34(15) (2013), 1819–1828. doi:10.1016/j.patrec.2012.10.018.

24.

Mecocci and

Micheli, Real-time automatic detection of violent-acts by low-level colour visual cues, in: Proceedings of the International Conference on Image Processing, ICIP 2007, San Antonio, Texas, USA, September 16–19, 2007, pp. 345–348.

25.

Mukherjee,

S.K.

Biswas and

D.P.

Mukherjee, Recognizing interactions between human performers by ‘dominating pose doublet’, Machine Vision and Applications 25(4) (2014), 1033–1052. doi:10.1007/s00138-013-0589-7.

26.

Park and

J.K.

Aggarwal, Event semantics in two-person interactions, in: 17th International Conference on Pattern Recognition, ICPR 2004, Cambridge, UK, August 23–26, 2004, pp. 227–230. doi:10.1109/ICPR.2004.1333745.

27.

Park and

J.K.

Aggarwal, Recognition of two-person interactions using a hierarchical Bayesian network, in: First ACM SIGMM International Workshop on Video Surveillance, Berkeley, California, 2010.

28.

Park and

M.M.

Trivedi, Multi-person interaction and activity analysis: A synergistic track- and body-level analysis framework, Machine and Vision Applications 18(3–4) (2007), 151–166. doi:10.1007/s00138-006-0055-x.

29.

Parzen, On estimation of a probability density function and mode, Annals of Mathematical Statistics 33 (1962), 1065–1076. doi:10.1214/aoms/1177704472.

30.

Patron-Perez,

Marszalek,

I.D.

Reid and

Zisserman, Structured learning of human interactions in TV shows, IEEE Transactions on Pattern Analysis and Machine Intelligence 34(12) (2012), 2441–2453. doi:10.1109/TPAMI.2012.24.

31.

E.A.

Perez,

V.F.

Mota,

L.M.

Maciel,

Sad and

M.B.

Vieira, Combining gradient histograms using orientation tensors for human action recognition, in: Proceedings of the 21st International Conference on Pattern Recognition, ICPR 2012, Tsukuba, Japan, November 11–15, 2012, pp. 3460–3463.

32.

Poppe, A survey on vision-based human action recognition, Image Vision Computing 28(6) (2010), 976–990. doi:10.1016/j.imavis.2009.11.014.

33.

Raptis and

Sigal, Poselet key-framing: A model for human activity recognition, in: 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, June 23–28, 2013, pp. 2650–2657. doi:10.1109/CVPR.2013.342.

34.

M.S.

Ryoo, Human activity prediction: Early recognition of ongoing activities from streaming videos, in: IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, November 6–13, 2011, pp. 1036–1043.

35.

M.S.

Ryoo and

J.K.

Aggarwal, UT-interaction dataset, ICPR contest on semantic description of human activities (SDHA), in: IEEE International Conference on Pattern Recognition Workshops, Istanbul, Turkey, August 23–26, Contest Reports, 2010, p. 4.

36.

Y.S.

Sefidgar,

Vahdat,

Se and

Mori, Discriminative key-component models for interaction detection and recognition, Computer Vision and Image Understanding 135 (2015), 16–30. doi:10.1016/j.cviu.2015.02.012.

37.

Shao and

Ji, Motion histogram analysis based key frame extraction for human action/activity representation, in: Sixth Canadian Conference on Computer and Robot Vision, CRV 2009, Kelowna, British Columbia, Canada, May 25–27, 2009, pp. 88–92. doi:10.1109/CRV.2009.36.

38.

D.F.

Specht, Probabilistic neural networks, Neural Networks 3(1) (1990), 109–118. doi:10.1016/0893-6080(90)90049-Q.

39.

Sunyoung,

Seongho,

Hyeran,

Haejin and

Sooyeong, Human interaction recognition in YouTube videos, in: 8th International Conference on Information, Communications and Signal Processing, ICICS 2011, December 13–16, 2011, pp. 1–5.

40.

B.T.

Truong and

Venkatesh, Video abstraction: A systematic review and classification, Transactions on Multimedia Computing, Communications and Applications 3(1) (2007), Article ID 3.

41.

Wu,

Zhu and

Shao, One shot learning gesture recognition from RGBD images, in: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA, June 16–21, 2012, pp. 7–12. doi:10.1109/CVPRW.2012.6239179.

42.

Yuan,

Prinet and

Yuan, Middle-level representation for human activities recognition: The role of spatio-temporal relationships, in: Trends and Topics in Computer Vision – ECCV 2010 Workshops, Heraklion, Crete, Greece, September 10–11, Revised Selected Papers, Part I, 2010, pp. 168–180.

43.

Yun,

Honorio,

Chattopadhyay,

T.L.

Berg and

Samaras, Two-person interaction detection using body-pose features and multiple instance learning, in: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA, June 16–21, 2012, pp. 28–35. doi:10.1109/CVPRW.2012.6239234.

44.

Zajdel,

J.D.

Krijnders,

T.C.

Andringa and

D.M.

Gavrila, CASSANDRA: Audio–video sensor fusion for aggression detection, in: Fourth IEEE International Conference on Advanced Video and Signal Based Surveillance, AVSS 2007, Queen Mary, University of London, London, United Kingdom, September 5–7, 2007, pp. 200–205.