Human action recognition using MHI and SHI based GLAC features and Collaborative Representation Classifier

3.1 Feature extraction

To encode the action features, the MHI and SHI are first derived from a depth action sequence and then the GLAC features are exploited from the resulting images. We discuss this in the following text. However, the well-known Motion History Image (MHI) gathers the description of body-segment movements (presented in depth frames) by contracting the depth sequence to a gray-level picture [20]. But, with inappropriate employment of the update function, the MHI can be unsuccessful to assure capturing utmost of the moving portions. Additionally, the knowledge about the motionless pose history domains, monotonous activities and monotonous unmoving poses is discounted in the MHI pattern [53]. Thus, we consider the Static History Image (SHI) to capture the aforementioned complementary components of the MHI. To generate the MHI and SHI images for each action video, we work with the 3-dimensional Motion Trail Model (3DMTM) [53]. The model provides a set of motion history images {MH1 _XOY , MH1 _YOZ , MH1 _XOZ } and a set of static posture history images {SH1 _XOY ,  SH1 _YOZ ,  SH1 _XOZ } corresponding to three 2D Euclidean planes. The motion update function φ _M  (x, y, t) and static posture update function φ _S  (x, y, t) are used in specifying the domains of moving and unmoving attitudes of subjects per activity completing respectively. These two functions are utilized for each depth frame investigated in the depth action video: φ M ( x , y , t ) = { 1 if P t > ζ M 0 otherwise φ S ( x , y , t ) = { 1 if d t - P t > ζ S 0 otherwise(1)

where (x,  y) represents pixel’s location and t represents amount of duration. Furthermore, d _t  = (d ₁, d ₂, d ₃, …  …   … , d _T ) is a depth map sequence whereas P _t  = (P ₁, P ₂, P ₃, …  …   … , P _T ) holds for a difference picture arrangement representing the absolute dissimilarity for a pair of depth maps. Moreover, both of these update functions want two threshold values; these are ς _M and ς _S for representing motion and stationary knowledge between sequential depth maps. So, the depth motion history image F _m  (x, y, t) is gained through employing motion update function φ _M  (x, y, t): F M ( x , y , t ) = { T if φ M ( x , y , t ) = 1 F M ( x , y , t - 1 ) - 1 otherwise }(2)

Furthermore, static posture history image (SHI) F _S  (x, y, t) is created by the static posture update function φ _S  (x, y, t) to reward for motionless domains for the entire depth video. This could be achieved by analogous technique like MHI: F S = { T if φ S ( x , y , t ) - 1 F S ( x , y , t - 1 ) - 1 otherwise(3)

It should be noted that the 3DMTM also provides the average motion history image (AMHI) and average static posture history image (SHI). In this work, we only consider the MHI and SHI as the AMHI and ASHI reduce the recognition accuracy [54]. Next, we extract gradient and curvature features (which are properties of human contours) from the MHI and SHI images using Gradient Local Auto-Correlation (GLAC) feature descriptor [14]. For an intuitive explanation, let us consider the I (x,  y) as a MHI/SHI. Then the magnitudes of the gradient vectors and the relevant orientation angles at each point of I (x,  y) are given by m = ( ∂ I ∂ x , ∂ I ∂ y ) and θ = arctan ( ∂ I ∂ x , ∂ I ∂ y ) respectively. Each spatial orientation of the gradient represented by θ is coded into D bins through voting weights to the nearby bins to construct a D-dimensional sparse vector f , which is known as gradient orientation vector (G-O vector shortly). With the utilization of the G-O vector f and the gradient magnitude, the Kth order auto-correlation function of gradients in a neighborhood can be expressed by:

F ( d 0 , … d K , a 1 … a K ) = ∫ w [ ( m ( r + b 1 ) , … , m ( r + b K ) ] gd o ( r ) gd 1 ( r + b 1 ) … gd K ( r + b K ) d r(4)

where b ₁, b ₂, …, b _k are translation vectors about the position vector r  = (x, y) ,  gd indicates the d-th component of g and w represents a weighting function. We in our experimentations shown in Section 4, K ∈ {0, 1}, a _1x,y  ∈ {± Δr,  0}  and w (.) = min(.) .

In K ∈ {0, 1}, the GLAC features are written as follows: 0 th order F 0 = ∑ r ∈ I m ( r ) gd 0 ( r )(5) 1 st order F 1 = ∑ r ∈ I min [ ( m ( r ) , m ( r + b 1 ) ] gd 0 ( r ) gd 1 ( r + b 1 ) ](6)

Since there are 4 individual configurations of (boldr, boldr + bolda ₁) as depicted by Fig. 2, the above GLAC feature dimension (boldF ₀ and boldF ₁) is configured by d = D + 4D ². Thus the obtained d-dimensional feature vector corresponding to the MHI _XOY is denoted by GMHI _XOY . Despite the fact that the dimension of GLAC feature vector is high, the vector generation is not so costly because of the sparseness of g. Equations (5 and 6) are utilized to a small number of non-zero components of g. A deeper discussion on the GLAC descriptor is represented in [14].

OPEN IN VIEWER

Fig.2

Configuration patterns for K ∈ {0, 1}.

From above, three feature vectors GMHI _XOY , GMHI _YOZ and GMHI _XOZ are obtained by employing the GLAC on the set of MHI. The obtained feature vectors are combined end by end to consist a vector GMHI = [GMHI _XOY ; GMHI _YOZ ; GMHI _XOZ ] to characterize a human action. Similarly, another feature vector GSHI = [ GHSHI _XOY ; GSHI _YOZ ; GSHI _XOZ ] can be constructed based on the set of SHI to represent the action through a complementary perspective. Finally, the above two complement feature vectors are combined to represent the action to enhance the discriminatory power of the recognition method.

3.2 Action classification

The l 2 -regularized Collaborative Representation Classifier ( l 2 -CRC) has been used successfully to recognize human actions in [25, 28]. As a result, after representing an action by combining the GMHI and GSHI feature vectors, we adopt an l 2 -CR C classifier to classify the action representation vector. For an intuitive description of the l ₂-CRC, let us consider an action dataset with C classes. When we arrange the training samples, we can gain a dictionary P = [ P ₁ , P ₂ , …  …  …, P _C ] = [ p ₁ , p ₂ , …  …  …, p _n] ∈ R^D × N, where D denotes the number of components in a sample, N indicates the entire number of learning examples in the process of action recognition. Also P _j ∈ R^D × M_j,(j = 1,2, ……C) represents a group of the training examples containing to the jth category and p _i  ∈ R ^D  (i = 1, 2, …… , n) indicates the particular learning action instance. Now, any query action sample S ∈ R^D can be expressed using matrix P as follows: S = P β(7)

Here β is a N × 1 vector related with coefficients equivalent to the learning examples. Solution for Equation (6) is not trivial since it is not usually determined [55]. Generally, the solution is gained as follows: β ˆ = β arg min { | | S - P β | | 2 2 + μ | | A β | | 2 2 }(8)

where A is Tikhonov regularization matrix [59] and μ denotes parameter named regularization. The term engaged with A confirms the utilization of earlier information of the solution by employing the method which is described by [56 –58], where the training instances which are very different from a query instance are assigned less weight rather than the learning instances which are very much analogous. Finally, the matrix A  ∈ R ^D×N is considered as follows: A = [ | | v - p 1 | | 2 0 0 0 | | v - p 2 | | 2 … 0 ⋮ ⋱ ⋮ 0 … | | v - p n | | 2 ](9)

In accordance with [60] vector β ˆ is evaluated by β ˆ = ( P T P + μ A T A ) - 1 P T S(10)

After that by utilizing the class information of all the learning instances, β ˆ could be split into C subsets β ˆ = [ β ˆ 1 ; β ˆ 2 ; β ˆ 3 ; … … … ; β ˆ C ] with β ^ j ( j = 1 , 2 , … … … , C ) . After splitting β ˆ the class label of the new instance S can be evaluated as class ( S ) = arg min j ∈ { 1 , 2 , … … , C } { q j }(11)

Where q _j is defined by q j = | | S - P j β ˆ J | | 2 .(12)

Algorithm 1 describes our recognition system concisely.

4.1 Results on the MSRAction3D Dataset

MSRAction3Ddataset [15] has 20 actions and each action was accomplished by 10 distinct persons 2 or 3times by fronting the depth sensor. The twenty human action classes are: “high wave (1)”, “horizontal wave (2)”, “hammer (3)”, “hand catch (4)”, “forward punch (5)”, “high throw (6)”, “draw x (7)”, “draw tick (8)”, “draw circle (9)”, “hand clap (10)”, “two hand wave (11)”, “side boxing (12)”, “bend (13)” “forward kick (14)”, “side kick (15)”, “jogging (16)”, “tennis swing (17)”, “tennis serve (18)”, “golf swing (19)”, “pick up and throw (20)”. All of these action types are considered in the context of gaming and include a diversity of movements related to arms, legs, torso, etc. In the dataset, inter-class resemblance amongst multiple action categories are observed, such as draw x and draw tick look analogous with a bit exception in one hand motion. To carry out an extensive assessment of the introduced approach, similar experimentations are taken such as previously adopted by [12 , 22]. Specifically, two different experimental settings (which are described in detail in the following sections) are established to evaluate our method.

First Experimental Setup and Results:

In the first experimental setup, the 20 actions are split into three different human action sets that are represented by Table 1. Three different test cases such as test one, test two and cross subject test are conducted for every action subset [15, 18]. In all the experiments, we set the optimal parameters D = 8 and Δr = 1 for the MHI and SHI based GLAC descriptors by utilizing the training examples through 5-fold cross validation technique. The number of spatial bins are also tuned and set to in the same manner. The l 2 -CRC parameter μ is set to 0.0001 through the 5-fold cross validation technique in the range of 0.00001∼10 on the learning samples. To enhance the computational easiness of the algorithm, Principle Component analysis (PCA) is employed to shrink the dimensions of the obtained action vector (in this setting, the dimension of feature vector is 3168). The PCA transform matrix is gained with the training feature set and then implemented to the test feature set. In all the experiments, the principle components that account for 99% of the entire variation are retained. The feature dimension utilized in each test case is reported in the corresponding accuracy table.

In test one , 1/3 examples of every subset are used for model learning, and the rest of the examples are used for evaluation. The individual and average recognition results corresponding to the AS1, AS2 and AS3 action sets are reported in Table 2. The proposed recognition method (i.e., fusion of GMHI and GSHI features) achieves an average recognition accuracy of 99.34%.

The recognition performances by employing the GMHI and GSHI features only are also shown in the table. There classification rates are inferior to the rates obtained through the proposed fusion method. Only the GMHI features achieve an equal accuracy (which is 99.33%) to the fusion method in AS3 action set. It should be noted in Fig. 3, the GMHI features can exhibit the promising recognition performance for several specific actions where the GSHI features cannot achieve the promising accuracy for those actions and vice versa. For example, in Fig. 3, the GMHI features show the higher recognition accuracy than the GSHI features to classify the high wave action in AS2. On the other hand, the hand catch action in AS2 is recognized more accurately using the GSHI features than using GMHI features. Thus, these two descriptors are complementary enough and by way of merging the GMHI vectors and the GSHI vectors, the whole recognition outcome is upgraded significantly on the situations while utilizing the GMHI features only or the GSHI features only.

In Test Two , the training stage uses 2/3 action instances of each subset and the system assessment turn utilize the residual instances. The recognition accuracies of the GMHI , GSHI methods and our approach are reported in Table 3. The fusion/introduced approach outperforms the GMHI and GSHI methods considerably for all action sets. The fusion method achieves 100% recognition accuracy for all action sets where the GMHI and GSHI are unable to achieve it. The GMHI features attain 100% classification accuracy only for the AS3 action set. The class specific classification accuracies are shown in Fig. 4 to more investigate the supremacy of the introduced fusion system.

OPEN IN VIEWER

Fig.3

Class specific accuracies in test one.

OPEN IN VIEWER

Fig.4

Class specific accuracies in test two.

In Test Three/Cross Subject Test , action samples executed by one half of the performers (1, 3, 5, 7, 9) are employed for model learning and the samples from rest performers are employed for the system evaluation. With Table 4, it could be demonstrated that the proposed strategy exhibits more improved recognition accuracies for all action sets than the GMHI and GSHI methods in the table. As an example, the introduced algorithm achieves over 7% higher recognition result than GMHI method and over 14% higher recognition rate than the GSHI method. The class specific accuracies for all actions sets with all employed methods are included in Fig. 5.

OPEN IN VIEWER

Fig.5

Class specific accuracies in cross subject test.

The performances of the introduced system are also assessed through extensive comparison with other methods that were evaluated on MSR-Action3D dataset by the similar experimental settings. The comparison of the average recognition accuracy (%) for the three test cases is shown in Table 5. The highest recognition result is highlighted through bold face. Clearly, it is noticeable that the recognition outcome of the proposed approach exhibits outstanding performance over all the approaches mentioned in the table. It should be observed that our algorithm attains the state-of-the-art accuracy of 99.34%, 100% and 97.3% in the test one, test two and cross subject test respectively. Especially for the most challenging cross subject test, the proposed approach beats the listed methods significantly, leading to 4.1% improvement over the second highest accuracy (93.2% in [42]). In addition, the recognition system shows superiority over the deep learning systems reported in [50]. The recognition outcomes by utilizing the GMHI and the GSHI features are also reported here. It is worth to mention that sometimes the integration of two complementary features increases the inter-class similarity and intra-class variation issues due to human body postures. For example, in Fig. 3, the fused features cannot increase the accuracy for the Hand catch action due to the above issue. But this case is irregular in feature integration and the accuracy improvement is Thus, we can be convinced that the features fusion works better.

Second Experimental Setup and Results

The experimental settings reported in [12, 22] is utilized here. All of twenty action classes are used beyond making action subsets and samples from one half of the actors (1, 3, 5, 7, 9) are engaged for model building and the samples obtained from rest actors are used for model assessment. The optimal parameters D = 8,  Δr = 1,  b _S  = 1 ×3 and μ = 0.0001 are determined in the same manner. The dimension of feature vector is minimized from 4752 to 85 by employing the similar technique as discussed in the first experimental setup. The comparison outcomes of the second experimental are illustrated in Table 6. Notice that we take a comparison of our system with the deep structured learning system described in [51]. The comparative recognition accuracies are all presented on their relevant papers. The recognition outcomes in Table 6 indicate that our system is even superior to the deep learning approach.

DHA dataset was firstly introduced by Lin et al. in [16]. In the dataset, action categories are increased from the Weizmann dataset [61], which was generally utilized for human action recognition using RGB action sequences. In fact, DHA dataset consists of twenty three (23) action classes, where the 1 to10 action types follow the similar descriptions in the Weizmann dataset [62] and the 11 to16 activities are added classes. The remaining actions (i.e., 17th to 23rd) belong to the groups of certain game actions. The 23 action groups include: “arm-curl (1)”, “arm-swing (2)”, “bend (3)”, “front-box (4)”, “front-clap (5)”, “golf-swing (6)”, “jack (7)”, “jump (8)”, “kick (9)”, “leg-cur (10)”, “leg-kick (11)”, “one-hand-wave (12)”, “pitch (13)”, “pjump (14)”, “rod-swing (15)”, “run (16)”, “skip (17)”, “side (18)”, “side-box (19)”, “side-clap (20)”, “tai-chi (21)”, “two-hand-wave (22)”, “walk (23)”. The dataset contains overall 483 depth action sequences, where an individual action sequence is executed by 21 performers (12 males and 9 females). In the dataset, inter-similarity among different types of action classes is found. For example, golf-swing and rod-swing actions have analogous motion segments with turning hands from one side up to the other side. Further analogous action couples could be observed in leg-curl and leg-kick, run and walk, etc. The set of entire 23 actions are utilized and samples obtained by one half of the whole actors (1 3 5 7 9 11 13 15 17 19 21) are employed for the model learning stage and the samples from remaining ones are employed for the evaluation session. The parameters D = 8, Δr = 1, b _S  = 1 ×2 and μ = 0.0001 are determined as optimal parameters. The feature dimension is reduced from 3168 to 252 to through the PCA algorithm. The recognition accuracy comparison in Table 7. Noticing that our method achieves 99.1% recognition accuracy (see Table 7), which is a remarkable outcome on the dataset. The recognition system achieves 100% recognition accuracy to recognize 21 actions among 23. The algorithm achieves 91% recognition for run and side-clap actions as they are 9% confused with other actions. More precisely, run is confused with rod swing and side-clap is confused with side. However, the proposed system outperforms all existing methods reported in Table 7.

The UTD-MHAD action dataset has 27 different actions accomplished by 8 players (4 females and 4 males). One subject repeats an action 4 times. There dataset contains 861 depth action sequences after deleting 3 unworthy sequences. The twenty seven (27) action categories cover: “right arm swiping to the left (1)”, “right arm swiping to the right (2)”, “right hand wave (3)”, “two hand front clap (4)”, “right arm throw (5)”, “cross arms in the chest (6)”, “basketball shoot (7)”, “right hand draw x (8)”, “right hand draw circle (clockwise) (9)“, “right hand draw circle (counter clockwise) (10)”, “draw triangle (11)”, “bowling (right hand) (12)”, “front boxing (13)”, “baseball swing from right (14)”, “tennis right hand forehand swing (15)”, “arm curl (two arms) (16)”, “tennis serve (17)”, “two hand push (18)”, “right hand knock on door (19)”, “right hand catch an object (20)”, “right hand pick up and throw (21)”, “jogging in place (22)”, “walking in place (23)”, “sit to stand (24)”, “stand to sit (25)”, “forward lunge (left foot forward) (26)”, “squat (two arms stretch out) (27)”. All the depth action sequences are recorded through a Kinect depth device and a inertial device (wearable). For depth sequences 1–21, the wearable device is kept on the actor’s wrist at right where the sensor is positioned on the actor’s thigh at right for the sequences 22th to 27th. Note that the dataset includes an extensive set of different human action categories. As instance, sport actions (e.g., bowling), hand gestures (e.g., draw x), daily activities (e.g., knock on door), and training exercises (e.g., arm curl). The whole of 27 action groups are employed in the experimental evaluation. Action samples captured by 50% of the total actors (1 3 5 7) are involved for the model learning and the samples gained from residual ones are utilized for the system evaluation. The parameters D = 8,  Δr = 1,  b _s  = 3 ×5 and μ = 0.0001 are determined as optimal parameters. The dimension of action feature vector is decreased from 23760 to 94 through the PCA algorithm. The comparison between our system and other existing systems are shown in Table 8.

It is worth mentioning that the comparison table shows our algorithm achieves 5.1% higher recognition accuracy than the best existing algorithm (the accuracy of the indicated algorithm is 84.4%) reported in [34]. To further clarify our algorithm, the confusion matrix is depicted in Fig. 8. In fact, the matrix provides clarification about the class specific accuracy and the misclassification status corresponding to individual action class.

Besides the overall accuracy, we evaluate the performance using the recall, specificity, precision and F1-score statistical measures on the MSRACTION3D, DHA [16] and UTD-MHAD [17] datasets to further examine our method. The statistical evaluations are considered on the dataset including all actions rather than using a subset of several actions. Thus, all of the statistical measures are calculated on the relevant confusion matrix (see Figs. 6–8) of MSRACTION3D, DHA and UTD-MHAD datasets, respectively. The results for the above three datasets are reported in Tables 9–11, respectively. All tables include the class-specific outcomes against a single measure as well as the average of the outcomes. The statistical analysis indicates that our approach can identify 100% of the negative samples correctly however, the positive samples are not being recognized accurately sometimes for all datasets. Consequently, we cannot achieve an overall accuracy of 100% for those datasets.

OPEN IN VIEWER

Fig.6

Confusion matrix on the MSRAction3D dataset on setting 2.

OPEN IN VIEWER

Fig.7

Confusion matrix on the DHA dataset.

OPEN IN VIEWER

Fig.8

Confusion matrix on the UTD-MHAD dataset.

The computational efficiency of the algorithm is measured through the running time of the major components involved in the algorithm and by the computational complexity of the major components. Indeed, the operation time varies from machine to machine, and hence the computational complexity is taken into account to perceive the efficiency of an algorithm.

Running time:

The proposed approach is tested on CPU platform with an Intel i5-7500 Quad-core CPU @3.41 GHz and a RAM of 16 GB. The processing time of proposed approach depends on the five major components such as 3DMTM based MHI/SHI generation, GMHI feature extraction, GSHI feature extraction, PCA based dimensionality reduction and l₂-CRC classification. The average running time (in millisecond) for the five components per action sample is represented in Table 12. The required time is observed on the MSRACTION3D [15] dataset where each action sample contains depth frames of 40 on average. Note that the total running time for the 40 frames is less than one second, i.e., 630.5±42.53 milliseconds. Consequently, it can be claimed that our recognition method can be applied for real-time operation to process more than 40 frames per second.

Computational Complexity:

The computational complexity of our method is similar to the method reported in [25], and the complexity is not as high as methods listed in Table 13. Although our method has same complexity with the method described in [25], our approach outperforms the method by 6.8% recognition accuracy with the same experimental setup (setting 1) on MSRACTION3D dataset. The proposed system is computationally more efficient compared to other methods included in Table 13.