Sample weights determination based on cosine similarity method as an extension to infrared action recognition

3.2 Samples reweighting

For matching the first-order and high-order statistic of the coding action features, we reweight sample of the training sets in the low-dimensional RKHS, rather than the original space. In view of the problem that separately projecting the training and testing set into RKHS would lead to inconsistent data structures, we learned a projection for the training and testing set to ensure the original data structures of both. Our method learns a feature map, ψ represents transforming the original feature x to RKHS, X₁₂ indicates that there are 12 different action classes features as follows: x → ψ ( x ) ∈ H ψ ( X ) = [ ψ ( x 1 ) , ψ ( x 2 ) , . . . , ψ ( x 12 ) ](1)

We use simple, efficient linear kernel to construct the kernel matrix K, Y to represent new features that project original features into RKHS. P denotes the projection matrix, the projection matrix P is calculated by (4). New features representation can be calculated by the follow: Y = P T K(2)

If the training set sample and the testing set sample have k nearest neighbors, they will have the same label. The distance between samples with high similarity is relatively close, and the distance between samples with low similarity is far. Y_j and Y_l are used to represent the new features of the training and testing set after projection transformation. To keep the training and testing set data projected into the same structure in RKHS, we minimize the follow: 1 2 ∑ j , l = 1 n ∥ Y j - Y l ∥ 2 W jl(3)

Where L = D-W, L is the graph Laplacian matrix [29], D is a diagonal matrix, its ith diagonal element is calculated as the sum of ith row of W, X_j represents the new features. W is a symmetric adjacency characterizing the sample relationship. For the problem that there is a great within-class differences in the dataset of benchmark infrared action recognition, we used the cosine similarity between the samples of the training set and the unlabeled testing set to determine the weight of the training set samples for the first time. The matrix is obtained by cosine similarity as follows: W ij = { cos ine ( x i , x j ) , x j ∈ N k ( x i ) 0 , otherwise(4)

The weight of training set sample is determined by judging the cosine similarity between the training set samples and testing set samples. The sample weight with high similarity in the training and the testing set is increased, the sample weight with low similarity is reduced. Where N k ( x i ) is the k nearest neighbors of X_i, which belongs to the same class with X_i .  For the ith row and the jth column of W, if the sample X_j is the k nearest neighbor of X_i, W_ji is seted to the cosine distance between the two samples, otherwise W_ji seted to 0. This ensures that the distance between samples with high similarity is relatively close, the distance between samples with low similarity is far.

Therefore, by introducing the learned new features into (3), the trace of the matrix is denoted by tr. The equation (3) can be simplified to as follows: 1 2 ∑ j , l = 1 n ∥ P T k j - P T k l ∥ 2 W jl = ∑ j = 1 n P T k j D jj k j T P - ∑ j , l = 1 n P T k j W jl k l T P = tr ( P T KLK T P )(5)

Currently, the sample weight adjustment process of transfer learning related reference [21] doesn’t achieve the weight adjustment of high-dimensional video data. Moreover, its algorithm is applied in multiple domains, but infrared action recognition is a single domain. So it is impossible to directly apply its weight adjustment process to infrared action recognition. Firstly, we need to extract the high-dimensional feature data of infrared action video. Secondly, we reduce the dimension of the high-dimensional feature data to obtain the low-dimensional data before projection. In the reweighting samples stage, we learn a projection for the training and the testing set, and use the cosine similarity to calculate the weight of each training set sample after projecting the low-dimensional data. In summary, our goal is to project original features into RKHS, keep the new training and testing set features with similar data structures, and reweight for new features learned. Therefore, our objective function can be expressed as: min P tr ( P T K β LK T P ) + λ ∥ P ∥ 2 , 1 s . t . P T KHK T P = I(6) where β and λ represent compensation parameters, H = I n - ( 1 n ) I n , I_n represents a square matrix of n dimensions, all elements are equal to 1.

Finally, according to the constrained optimization theory, the Lagrange multiplier λ is introduce: ( K β LK T + λ G ) P = KHK T P Λ(7)

The decomposition of (6) is optimized to obtain the projection matrix P, where the parameters β and λ are both greater than zero. The matrix G is the sub-gradient of ∥P ∥ _2,1, P_i is the ith row of the P matrix, G is calculated by the follow: G ii = { 0 , P i = 0 1 2 ∥ P i ∥ , otherwise(8)

In this paper, we learn [21] proposed method abstracted feature vectors into graph vertices by: d = U i = 1 C knnsearch ( P r T K r ( i ) , P s T K s ( i ) , k )(9)d is the degree of vertex. The subscripts r and s represent the training and the testing set, respectively. It can be seen from Equation (9) that no training and testing set label data is used in this paper when k nearest neighbor calculation.

In this way, the training set sample weight α is calculated by the follow: α i = d i ∑ i d i(10) and it can be updated by the vertex degree obtained from (9). The training set data can be trained by SVM to obtain the model, and then the testing set can be classified by this model.

Input infrared action video, calculate iDTs features, use LLC coding, and reduce the dimensions by PCA to get the underlying feature representation.

4.1 Datasets

The InfAR dataset [4] consists of 600 human action videos taken by infrared thermal imaging cameras. The dataset contains videos of 12 different action classes: fight, handshake, handclapping, hug, jog, jump, punch, push, skip, walk, wave1 and wave2. Each action class was performed by 40 volunteers. Each action class contains 50 video clips, and each of which lasts an average of 4 seconds. The frame rate of the action video is 25 fps and the resolution is 293×256. The shooting scene of video is in line with the actual situation, containing one or multiple actions performed by one or several persons in each video, reflecting the complexity of the background, seasonal differences, within-class differences, and presence or absence of occlusion, etc.

Sample example diagram of 12 different action classes with simple and complex backgrounds in InfAR dataset is shown in Fig. 2. Each subgraph in Fig. 2, on the left is the sample with a simple background, on the right is the sample with a complex background. We mark the action being performed with a red box in the picture on the right to see more clearly from the Fig. 2. It’s important to note that we define a simple background and a complex background simple, which is not simply defined by several persons as background in each action video sample. Figure 2 is just to make it more intuitive that the details of each sample are different for the action of same class. There are no two identical samples, but some samples have very high similarity to other samples of the same class, and some samples have very low similarity to samples of the same class. In our algorithm, infrared video samples with many people as background can also be recognized as highly similar to other samples of the same class. Although there are several persons as background in these samples, they have less occlusion than the same class samples, the trajectories of actions are more obvious. The sample with low similarity is that the human is performing actions in video may be obscured by others, resulting in the action is not obvious. Normally, the proportion of these sample is less, but it exists in the actual situation, which cannot be removed directly.

OPEN IN VIEWER

Fig. 2

Example diagram of a simple background sample and a complex background sample. The image on the left of each action is a sample with a simple background, and the image on the right is a sample with a complex background. The complex movements in the background are marked in red boxes for clarity.

In order to further illustrate the sample differences of the same class action, Fig. 3 gives an example of a trajectory figure of sample with simple background and sample with complex background. According to the reference [24], the trajectory figure is the feature information figure of infrared action recognition. The trajectory figure of fight in Fig. 3(a) is more obvious, and the background in Fig. 3(b) is more complex than that in Fig. 3(a), because the others in the background are still moving, making the trajectories of the fight action less obvious. From the Fig. 3 we can see, there is a certain gap between the trajectories of the same type of action, indicating that there is indeed a difference between different samples of the same class.

OPEN IN VIEWER

Fig. 3

This is an example of two different sample trajectories of the same fight action. Trajectories of sample with simple backgrounds and sample with complex backgrounds: (a) simple and (b) complex. In the figure, the green represents the trajectories of human action, and the red dots are the trajectory positions in the current frame.

In our experiments, 50 video clips of each action category in InfAR dataset are randomly split into training and testing sets, 30 samples as training sets, and the rest samples as testing sets. Figure 4 shows the parameter settings of the REWS.

OPEN IN VIEWER

Fig. 4

The sensitivity of each parameter of our REWS is (a)–(e). The a, b, c, d and e represent the number of neighbor k, β,λ, dimensionality of subspace and number of iterations, respectively. It can be seen that compared with the value change of other parameters, the value of β is the most sensitive.

The a, b, c, d and e in the Fig. 4 represent the number of neighbor k, β, λ, dimensionality of subspace and number of iterations, respectively. For each evaluation, we repeat the experiments with the same setting 5 times and report the average accuracy. Accuracy represents the average precision of 12 classes action. It can be seen from the Fig. 4 that when they were taken 4, 0.7, 2, 150 and 7 respectively, the average precision of all action categories are up to 78.83%. It can be seen that compared with the value change of other parameters, the value of β is the most sensitive. If the value changes by 0.1, the accuracy will change greatly, because β determines the weight of the training set sample, which also reflects the importance of samples reweighting in the training sets.

In order to show the superiority of our proposed framework REWS, Fig. 5 shows two confusion matrices. We directly train the SVM as a baseline without reweighting of the training set. In Fig. 5(a) and (b) are the baseline and our proposed method, respectively. In the confusion matrix of each class action precision in InfAR dataset, the main diagonal is the precision of each action. It can be seen from the Fig. 5 that the proposed method has higher classification accuracy for almost all action classes. We draw the precision of each action into Fig. 7 to have a more intuitive contrast. As shown in Fig. 7 that the most obvious precision improvement is the punch and push actions. As can be seen from Fig. 2(g) and (h), punch and push are so similar that it is even difficult for human to tell them apart. But our proposed algorithm has a significant improvement over the baseline, further illustrating the effectiveness of our proposed method.

OPEN IN VIEWER

Fig. 5

The confusion matrices of the precisions of each action: (a) the baseline and (b) the proposed method. In the confusion matrix of each class action precision in InfAR dataset, the main diagonal is the precision of each action.

To reflect classification results of 20 testing set samples of each class action, we present the testing result distribution graph of each action. Figure 6(a) shows the distribution results of the SVM directly trained without the weight of the training set. Figure 6(b) shows the testing set result distribution of our proposed REWS. The total number of testing set samples in the two pictures are 240. Moreover, almost each type of action in Fig. 6(b) recognizes more correct samples than in Fig. 6(a).

OPEN IN VIEWER

Fig. 6

The testing result distribution graph of each action: (a) the baseline and (b) the proposed method. In the test result distribution diagram for each action, the main diagonal is the correct number for each action. The total number of testing set samples in the two pictures are 240.

Table 1 shows the comparison between REWS and other action recognition methods. For comparison purposes, all methods are implemented using the same experimental setup. The average precision of our method is 78.83%; it is improved compared with the baseline without samples reweighting 6.91%. Our method and CDFAG method extract the same trajectories features, after adding weights to all training sets, the average precision was 3.41% higher than CDFAG. We do not require to introduce visible light data for infrared action recognition, because in reality it is impossible to have two visible and visible datasets with the same action. So we don’t require time-consuming feature alignment and generalization process. Compared with the two-stream CNNs and 3D CNNs, the average precision is 2.17% and 1.33% higher, respectively. Besides, compared with the state-of-the-art deep learning method currently only 0.42% less, reaching the most advanced performance. Our approach achieved state-of-the-art results in infrared action recognition compared with hand-crafted and deep learning based methods.

OPEN IN VIEWER

Fig. 7

Comparison of the precisions for each action of SVM and REWS. The proposed method has higher classification ac-curacy for almost all action classes.

However, TSTDDs approach composed of hand-crafted and deep-learned features. The three-stream CNNs model they proposed requires a combination of trajectory constraint sampling (hand-crafted features) and pooling strategy (deep-learned features), which makes the model more complex and difficult to implement. In addition, the deep learning method requires time-consuming pre-training to obtain a robust model. Compared with TSTDDs approach, the approach REWS we proposed only needs to adopt an efficient method improved dense trajectory (hand-crafted features) to achieve the performance of TSTDDs. The proposed REWS has no complex image stream extraction process and is relatively simple and easy to implement. At the same time, the model we proposed don’t require time-consuming pre-training process [4–6], and can balance classification accuracy and time efficiency.

We execute our codes with MATLAB R2017b on a 64-bit Windows 10 PC with Intel i7 CPU 6-core 3.20 GHz and 16 GB RAM. The average time for each video clip feature extraction is 24.55 s, the LLC encoding requires 5.77 s. For sample reweighting of each training set, the average time is 9.43e-05 s.