A computationally efficient method for Human Activity Recognition based on spatio temporal cuboid and Super Normal Vector

Abstract

For the last three decades human activity recognition has shown a huge technological advancement due to less expensive RGB-D cameras and the increase in the large volume of video data. As a result of the increase in number of surveillance cameras, manual annotation becomes difficult and need for automatic recognition and annotation of video arises. In this paper, we introduce a computationally and storage efficient method for recognizing human activities from depth videos and a new frame selection method based on the mean value of motion energy. We extract normal vectors from the points in the boundary curve. Then polynormals are obtained by sequentially attaching the normals from a neighborhood of each of the points in the boundary curve. These polynormals from a spatio-temporal cuboid constructed from the input video and it is pooled to form the Super Normal vectors. These Super Normal vectors are the final feature vectors, which are given as input to the classifier. The classifier used is lib-linear SVM. The results on MSRAction3D dataset show that the algorithm we put forward is fast and the accuracy obtained is comparable with the existing methods. The method which we proposed here gives an accuracy of 88% while taking whole frames and 89.82% when frame selection method is applied. The proposed method is also tested on UTD-MHAD dataset.

Keywords

Motion energy depth videos frame selection boundary curves polynormal dictionary learning

1 Introduction

Now a days due to the invention of new cost effective cameras and the emergence of internet large amount of video data are prepared and shared through the internet or uploaded to social media sites. In order to effectively use these videos annotation is needed. Due to the huge quantity manual annotation is not possible. Also automatic monitoring of surveillance cameras is needed to respond to unusual events. Many of the unusual events require fast and immediate action. So we proposed a new Human activity recognition algorithm which is faster than many of the state of art methods. The aim of human activity recognition algorithms is to classify the videos into several categories based on the activities performed in those videos. Nowadays, computer vision, as well as artificial intelligence researchers, shows a great interest in Human Activity Recognition algorithms. There are a lot of applications for human activity recognition such as sign language recognition [1], human-computer interaction [1], video surveillance [1], content-based video retrieval [2], video Summarization [3] etc. Medical Health care [1] and Social care fields also get the advantage of human activity recognition algorithms. In medical and healthcare fields, we can use these algorithms to detect the fall of elderly people who are staying alone at home [4] and to monitor the activities of patients. In social care fields, we can detect the abnormal activities in public places like metro stations and air terminals.

Based on the difficulty level, we can categorize activities into four as (i) Gestures, (ii) Actions (iii) Interactions (iv) Group activities [5]. The motion of the part of the body comes under gestures, for example, moving the leg, raising the hand, etc. Actions are performed by single persons and are a combination of several gestures. Eg: waving, walking, punching, etc. Activities like kicking a ball, fighting between two persons, etc come under activities where the person interacts with the object or interacts with another person. Group activities are activities performed by two or more people for example group of person fighting, playing football, etc.

For every computer vision algorithms, the initial step is data acquisition. Sensor-based and Vision-based are the two categories of Human Activity Recognition algorithms based on data acquisition methods. In the first category, the person has to wear sensors. The sensors may be accelerometers, gyroscopes or goniometers. In the second category, actions are recognized from the videos collected from the visible light camera or the depth camera. The videos recorded using the normal visible light camera is affected by noise, illumination variation, etc. In order to solve those issues, we can use the depth camera. The three main benefits of depth camera are (i) it provides the 3D structure of the image which eases the recovery of postures and recognition of the activity. (ii) It can work even in darkness; because of the infra-red light used (iii) It is cost effective. Due to the above advantages, researchers are now showing greater interest in depth videos than normal RGB videos. In this paper, we present a computationally efficient method for human activity recognition based on Super Normal vectors extracted from the points in the boundary curves.

Section 2 brings out, various techniques for Human Activity Recognition and related works in human activity recognition. Detailed explanation about the proposed methodology is given in Section 3. The contributions of the proposed work are also highlighted in this chapter. The evaluation format, the dataset used and the outcome of the experiment is furnished in Section 4. This chapter also discusses the evaluation measures, quantitative and qualitative analysis of the proposed work also. In Section 5 we wind up the report by highlighting the advantage of the proposed method and points to the future scope of Human Activity Recognition.

2 Literature Survey

Traditional Video Descriptors based on Optical Flow and Gradient for representing RGB videos cannot be suitable for representing the depth video and hence cannot be used for classification of depth videos. So algorithms that are designed for the recognition of Human Activities from depth videos are based on skeleton joints, projected depth maps, cloud points, Normal Vectors, etc. Lu Xia et.al in [6] presented an algorithm for human activity recognition based on spatio-temporal interest points and depth cuboid similarity features. In [7] Eweiwe et al. discussed the relative location of the human joints, their correlation and velocities are utilized for the recognition of activity. In this method, joint features are extracted and converted to the spherical coordinate system to make it more robust to foreshortening effect. These features are then added to the histogram. In cross subject test they got an accuracy of 89.3% on MSRAction3D dataset. Several recently developed methods utilize the advantages of Normal Vectors which provide information about shape and structure. In [8] Oreifej et al. discussed an activity descriptor based on the Histogram of Normals in his paper. The temporal order is also encoded to achieve a satisfiable recognition accuracy of 88.89%.

In another Human Activity recognition method [1], extended surface normals of cloud points from a 3×3×3 neighborhood are concatenated to form the polynormals. The polynormals are encoded using an online sparse coding technique. They used a smaller dictionary of size 100x81. Then they extracted polynormals from an adaptive spatio-temporal grid constructed based on the motion energy and perform spatial average and temporal max pooling to get the Super Normal vectors. This pooling stage is a time-consuming step in this algorithm. The computed super normal vector is classified using LIBLINEAR SVM with a testing accuracy of 93.09% on the MSRAction3D dataset. Yang et al., in [9] replace the adaptive spatio-temporal grid with adaptive spatio-temporal cell and obtained an accuracy of 93.45%.

In [10], Chengjin Li et al. extended the concept of Super Normal vectors to get a better result. They computed the polynormal by concatenating 3 levels of surface normals. The bottom layer contains normals from the 3×3×3 neighborhood, the middle layer contains normals from the 2×2×3 neighborhood and the top layer contains normals from the 1×1×3 neighborhood. Chengjin Li et al. [9] uses a more discriminative sparse dictionary learning technique called the sparse constraint dictionary. So they achieve an accuracy of 94.55%. 3D points clouds based method for human activity recognition [11] achieves an accuracy of 91.64%. In this method, 3×3 Eigen values are taken at each point. Histogram of Oriented Principal components (HOPC) is a series of eigenvectors arranged in the descending order of their Eigen values. In [12], Wanqing Li et al. used bag of 3D points in the recognition of Human Activity.

A fast method for human activity recognition is introduced in [13] where, the depth motion maps are constructed by gathering the motion energy of projected depth map along three planes – front, side, and top. In [14] two architectures are introduced by Rui Yang et al. The first one is based on 2D-CNN and the second one is based on 3D-CNN. In the first method, the complex Human Activity Recognition problem has reduced to simple image classification task by stacking together the Depth Motion Maps of the entire frames. In the second method, a DMM cuboid is constructed to learn more temporal information than the first one. The accuracy obtained by them for the first method is 92.21% for the second method it is only 86.08% on MSR Action3D dataset. In order to reduce the computational complexity and to improve the recognition accuracy, a key frame extraction method was introduced by Cheche Xie et al. in [15]. In this method, frames are extracted based on structural distance similarity using a self adapted search algorithm. Depth Map and Posture data based Human Activity Recognition using deep CNN model, introduced by Aouaidjia Kamel et al. in [16] uses two descriptors - Depth Motion Image (DMI) and Moving Joint Descriptor (MJD). DMI represents the depth map sequence and MJD represents the body posture sequence. Another advantage of this method is that it uses only the front projection instead of the three projections. The accuracy they obtained for the MSRAction3D dataset is 94.51%.

In [17] also Depth Motion Map (DMM) is used for the recognition of Human Activity. The depth video is first projected to three Cartesian coordinate and the differences between the adjacent frames are accumulated as the depth map. The accuracy obtained for the MSRAction3D dataset is 92.3%.

Our proposed method is also based on Super Normal Vectors. We compute the normals for the points in the boundary curve of the person doing the activity. Then we concatenate the polynormals from an adaptive spatio-temporal cuboid to form the Super Normal vectors.

3 Human Activity Recognition based on Adaptive Cuboid and Super Normal Vector – Proposed Method

Our aim is to design and develop a space and time efficient system for Human Activity Recognition, which can work with depth videos efficiently so that it can be used in real time applications. In [1] X. Yang et al. computed polynormals from the cloud points in the depth sequence. Since each frame contains a large number of cloud points, extracting the normals from the cloud points and concatenating them from a neighborhood is a time-consuming process. In another work [19], J Park and et al have shown that the surface normals from the points in boundary curves are enough for modeling objects. Incorporating this knowledge in human activity recognition, we have reduced the computation time by considering only the points in the boundary curve. Analogous to many other existing human activity recognition algorithms the proposed algorithm also has 3 steps; Data Acquisition - Feature Extraction - Classification

Feature extraction step is again divided into six steps; Compute Motion Energy, Computing the Polynormals, Learning the dictionary, Adaptive Spatio-Temporal Cuboid construction, Aggregating Super Normal Vectors. The proposed method is summarized in the following diagram.

Fig. 1

Block diagram of proposed method.

3.1 Computing the motion energy

Different people do the same activity in different speed. So it is not flexible to divide the time axis equally. In order to overcome the above-said difficulty, it is desirable to pool the features that has the same activity status. Therefore, we divide the motion energy axis identically into four instead of the time axis.

First, we construct the projected maps by projecting each frame of the depth sequence into three Cartesian planes - front, top, and side. Then thresholding is applied on the difference between the projected maps of the two neighboring frames to form the binary map. Then, we compute the motion energy by taking the summation of the non-zero elements of the binary map using the formula [1] $ɛ (t) = \sum_{υ = 1}^{3} \sum_{j = 1}^{t - 1} sum (| I_{υ}^{j + 1} - I_{υ}^{j} | > η)$ (1) where, ɛ(t) is the motion energy of the t^th frame, η = 45 is the threshold, ν is the projection(front, top or side), j is the frame number, sum returns the number of non-zero elements in the binary map.

3.2 Computing the polynormals

Depth video can be considered as a hyper surface and this hyper surface can be represented by a collection of points having dimension ’m’. So all the ‘m’ dimensional points in the depth video should satisfy the equation [1] $F (x_{1}, x_{2}, \dots x_{m}) = f (x_{1}, x_{2}, \dots x_{m - 1}) - x_{m} = 0$ (2)

For depth sequences, dimension is 4, i.e., x, y, t and z. Therefore the value of m = 4. So each cloud point in the depth video should satisfy the equation [1] $F (x, y, t, z) = f (x, y, t) - 1 = 0$ (3)

Then the extended surface normal can be computed using the formula [1] $η = \nabla F = (\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}, \frac{\partial f}{\partial t}, - 1)$ (4)

The components of the surface normals are gradients along the three directions and a scalar (–1). This scalar makes the orientation of the normal align in the direction of gradient. The distribution of normal orientation is able to provide more information about the geometric cues than traditional gradient orientations. These normal vectors also give information about the motion cues. To preserve the correlation of each normal with its neighbors we form clusters of normals from a 3×3×3 spatio-temporal neighborhood. These clusters of normals are called polynormals. These polynormals are more robust to noise. The polynormals can be represented using the following formula [1] $p = {(η_{1}^{T}, η_{2}^{T}, \dots η_{L}^{T})}^{T}, n_{1}, n_{2}, \dots, n_{L} \in L$ (5) where L is the number of normals from the neighborhood. The size of the neighborhood is 3×3×3. So the value of L is 27. Thus each polynormal is a concatenation of 27 normals from three adjacent frames. These polynormals provide only discriminative information for human activity recognition. Computing the polynormals for all the cloud points requires a lot of computational power. Since surface normals from the boundary curves are enough for modeling pose of a human [19], it is clear that we can recognize human activity accurately using the surface normals from only the border line. Hence in the proposed method instead of using all the cloud points, we are considering only the points in the boundary curves for the polynormal computation and we ignore the remaining cloud points.

3.3 Learning the dictionary

After obtaining the polynormals, we encode it using sparse coding technique introduced in [21]. In traditional methods, after coding the features the low level features are ignored in the recognition stage. In the proposed method, we are taking the difference between the whitened polynormal and the dictionary word to include the low-level features also. Whitening transformation is done to transform the covariance matrix to the identity matrix. The problem of sparse coding or the basis pursuit can be solved using the formula [1] $min_{D, α} \frac{I}{N} \sum_{i = 1}^{N} (\frac{1}{2} | | p_{i} - D α_{i} | |_{2}^{2} + λ | | α_{i} | |)$ (6) where, D is the dictionary, d_k is the visual word, α is the coefficient of sparse decomposition and λ = 0.15 is a constant for regularizing the sparsity.

A large value of D will lead to a very low value to α. To avoid this condition, the value of d_k^Td_k is maintained to be less than or equal to 1.

3.4 Adaptive spatio-temporal cuboid construction

The extracted polynormals contain only the discriminative properties required for the classification of the video. So a Spatio-Temporal Cuboid is constructed from the depth sequences and incorporated with the polynormals to include the temporal order and the spatial layout globally. Apart from the traditional methods that use a grid covering the entire frame, we use a smaller grid which is large enough to cover the human in the video.

In order to construct the temporal cuboid, we first normalize the motion energy computed in the previous step in the range [0, 1]. Then a plot is drawn with normalized motion energy along the y-direction and frame index along the x-direction. We, then divide the motion energy axis identically into four segments with the normalized energy in the range [0–0.25, 0.25–0.5, 0.5 0.75, 0.75–1] and the frame numbers corresponding to the energy 0, 0.25, 0.5, 0.75, and 1 are noted as t₀, t₁, t₂, t₃ and t₄ respectively. A temporal cuboid is constructed by taking four successive segments {t₀–t₁, t₁–t₂, t₂–t₃, t₃–t₄} of the video based on the motion energy. Our adaptive spatio-temporal cuboid consists of this temporal cuboid and a 4×3 spatial grid.

3.5 Aggregating the super normal vectors

For each visual word in the constructed dictionary, we compute the weighted difference between the polynormal and the visual word. We have taken sparse coefficient as the weight. Then we compute the average pooling on the spatial level and max-pooling on the temporal level on these differences [1].

$u_{k} (t) = \frac{1}{N_{t}} \sum_{i ɛ N_{t}} α_{k, i} (p_{i} - {Dd}_{k})$ (7) where, u_k(t) represents the difference vector obtained after pooling the k^th visual word from the t^th frame [1] $u_{k, i} = max_{t = 1, \dots, T} u_{k, i} (t), for i = 1, \dots, M$ (8) where, u_k is the vector representation of the k^th visual word in the whole input video. i indicate the i^th component of the corresponding vectors. The final feature vector is the concatenation of vectors from all the k visual words and is denoted as ’U’ [1]. $U = {(u_{i}^{T}, \dots u_{K}^{T})}^{T}$ (9)

These vectors are termed as Super Normal Vectors and are fed to the classification stage as input.

3.6 Classification

Traditional classifiers may not show good accuracy in our proposed work because the number of classes and the number of polynormals is very high. So we use a technique introduced in [20] by R. Fan and et al. This method can effectively classify large sparse dataset. The accuracy we obtained is 88%. In order to remove the frames that lead to false classification we have introduced a frame selection method. This method reduces the computational time and storage complexity of the algorithm.

3.7 Frame selection

The depth video comprises of several frames, and hence it is possible that many frames may be redundant. In order to find out the irrelevant frames, we have proposed a new frame selection method based on the motion energy.

Indices of the frames selected for the feature extraction is computed using the formula $I = index (Δ ɛ) \forall Δ ɛ > mean (Δ ɛ) - ω$ (10) where, index returns the index of every motion energy whose value is greater than mean value, ω and the value of ω is experimentally set as 300 for MSRAction3D dataset.The motion energy is computed as the difference between the adjacent depth maps in three projected views. So each motion energy value indicates how much difference is between the adjacent frames. Smaller values of motion energy indicate there is only less motion in the next frame. In order to remove the frames having less motion, we have removed the frames having smaller values of motion energy. These selected frames are then used for feature extraction in both the training as well as in the testing stage.

The proposed method is summarized in the following algorithm.

Algorithm
1. Compute the motion energy using the formula
$ɛ (t) = \sum_{υ = 1}^{3} \sum_{j = 1}^{t - 1} sum (\| I_{υ}^{j + 1} - I_{υ}^{j} \| > η)$
2. Select the frames using the formula
$I = index (Δ ɛ) \forall Δ ɛ > mean (Δ ɛ) - ω$
where, Δɛ (t) = ɛ (t) - ɛ (t - 1)
3. Extract the boundary points of the human present in the selected frames. For every point do steps 2 to 4
4. Compute polynormals from the depth sequence
5. Compute the co-efficients α_i of sparse coding
6. For each grid and for each visual word do
a. Spatial average pooling
$u_{k} (t) = \frac{1}{N_{t}} \sum_{i ɛ N_{t}} α_{k, i} (p_{i} - {Dd}_{k})$
b. Temporal max pooling
$u_{k, i} = max_{t = 1, \dots, T} u_{k, i} (t), for i = 1, \dots, M$
c. Concatenate the vectors
$U = {(u_{i}^{T}, \dots u_{K}^{T})}^{T}$
7. Concatenate U to form Super Normal Vector
8. Classify the SNV using LIBLINEAR SVM

3.9 Contributions of the Proposed work

The contributions of our proposed work is summarized in this section. We have proposed a new Human Activity Recognition Algorithm by extracting Super Normal vectors from Boundary Curves. The computation time is decreased by 2.5 times than SNV [1]. The space complexity is decreased by 1.7 times than SNV [1]. We have proposed a new frame selection technique by modeling the motion energy.

4 Result analysis and discussions

4.1 MSRAction3D Dataset

We have tested our method on the MSRAction3D dataset. The MSRAction3D dataset contains depth videos of one person actions done by 10 subjects facing the camera. It contains 20 activity classes and each activity is done 2 or 3 times. The videos are recorded by placing the camera in front of the subject. There are a total of 567 videos in this dataset. The camera used for capturing the videos in this dataset is the Microsoft Kinect Camera.

4.2 Experimental setup for MSRAction3D dataset

For comparing the result with other methods, we have used the same experimental setup as [1]. For training, videos of odd subjects i.e., 1, 3, 5, 7, 9 are taken and the videos of the remaining subjects i.e., 2, 4, 6, 8, 10 are taken for testing. Therefore the train set contains 292 videos and the test set contains 275 videos. Some Activity Recognition Algorithms on MSRAction3D uses three subsets. So we tested our proposed method in these three subsets also. The first Activity Subset (AS1) contains videos of activity classes 2, 3, 5, 6, 10, 13, 18, 20, second Activity Subset (AS2) contains videos of activity classes 1, 4, 7, 8, 9, 11, 12, 14 and the third Activity Subset (AS3) contains videos of activity classes 6, 14, 15, 16, 17, 18, 19, 20. There are 226, 231, 226 videos in AS1, AS2 and AS3 subsets respectively. The AS1 and AS2 subsets, groups actions with similar movement, whereas, AS3 groups complex actions together.

Fig. 2

Confusion matrix of MSRAction3D.

4.3 UTD-MHAD Dataset

UTD-MHAD dataset contains 27 activity classes, each activity is performed by 4 females and 4 males. Each activity is repeated 4 times by each subject. Similar to the MSRAction3D dataset, the camera used is Microsoft Kinect. The camera is placed in a tripod stand at a distance of 3m from the subject. This distance ensure the visbility of the whole body of the subject.

4.4 Experimental setup for UTD-MHAD Dataset

We have used the same experimental setup as [18]. Videos of odd subjects are used for training and videos of even subjects are used for testing. The proposed method is also tested on two subsets with one containing videos action classes 1–12 and 14 to 20 and the other containing action classes 13 and 21–27.

Table 1
Accuracy of proposed system on MSRAction3D

MSRAction3D Accuracy (%)

Subset 1 89.62

Subset 2 84.96

Subset 3 97.32

Over all 89.82

MSRAction3D	Accuracy (%)
Subset 1	89.62
Subset 2	84.96
Subset 3	97.32
Over all	89.82

Table 2

Accuracy of the proposed system on UTD-MHAD dataset

	Accuracy (%)
Subset 1	79.28
Subset 2	96.83
overall	83.02

4.3 Performance comparison on MSR-Action3D with SNV [1]

The summary of the results is depicted in the following tables. The results reveal that the proposed method is computationally efficient in terms of time and space, comparable with the other existing methods.

4.4 Space comparison

We have computed the SNV descriptor for the whole dataset containing 567 depth videos. The space required for storing these descriptors is 1.22 GB for our proposed method without frame selection and 1.20 GB for our proposed method with frame selection, which is very less compared to the SNV [1]. The details of the space comparison is shown in table.

Table 3
Space comparision of proposed system with SNV [1]

Method SNV [1] Without frame selection With frame selection

Normals 729 MB 390 MB 252 MB

Descriptor 2.33 GB 1.22 GB 1.20GB

Method	SNV [1]	Without frame selection	With frame selection
Normals	729 MB	390 MB	252 MB
Descriptor	2.33 GB	1.22 GB	1.20GB

Table 4

Time comparision of proposed system with SNV [1]

Method	SNV [1]	Without frame selection	With frame selection
Computing polynormals	18 Mins	20 mins	9 Mins
Learning Dictionary	9 Mins	14 Mins	5 Mins
Computing Video	330 Mins	89 Mins	64 Mins
descriptor

Fig. 3

Time comparison.

4.5 Time comparison

Since time is a main constraint in Human Activity Recognition algorithms, we have measured the time consumed by our proposed method. The total time required for our proposed method without frame selection is 228 minutes and for our proposed method with frame selection is 134 minutes approximately. The time required for computing the motion energy is not included for comparison. The time is measured on a system having Intel i7 processor and 16GB RAM. The time consumed by the major steps are shown in table.

Table 5
Accuracy comparision of proposed system with HON4D [8] and SNV [1]

Method Accuracy

HON4D [8] 88.89%

SNV [1] 93.09%

Proposed method without frame selection 88%

Proposed method with frame selection 89.82%

Proposed method without frame selection 90.09%

and cloud points

Proposed method with frame selection 92.72%

and cloud points

Method	Accuracy
HON4D [8]	88.89%
SNV [1]	93.09%
Proposed method without frame selection	88%
Proposed method with frame selection	89.82%
Proposed method without frame selection	90.09%
and cloud points
Proposed method with frame selection	92.72%
and cloud points

Our aim is to recognize Human Activities faster without sacrificing the accuracy. For our proposed method without frame selection the accuracy is only 88%. When the frames having low motion energy values are removed we got a higher accuracy of 89.82%. The accuracy we obtained is 0.36% less than the SNV [1] when using sampled cloud points but we have reduced the computation time 1.7 times than SNV [1]. Table compares the accuracy we obtained with two other methods on MSRAction3D dataset.

5 Conclusion

We have proposed a fast and storage efficient method for recognizing human activities using the super normal vectors extracted from the boundary curves of human doing the activity. The objective of this work is to present a method which can recognize human activities from super normal vectors extracted from depth videos with minimum time and space complexities. In this paper, we have proposed a frame selection technique based on the motion energy. In this work, we adopted an adaptive spatio-temporal cuboid which can handle the time variations of actions in the same category effectively. The proposed method when applied on MSRAction3D dataset gives the result in very less time than the existing methods while consuming very less memory, proving that time and space complexities are substantially reduced by the proposed method.

References

Yang

and Tian

, Super normal vector for activity recognition using depth sequences, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 804–811, 2014.

Yao

, Liu

and Huang

, Spatio-temporal information for human action recognition, EURASIP Journal on Image and Video Processing2016(1) (2016), 39.

Gowda

S.N.

, Human activity recognition using combinatorial deep belief networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 1–6

Dubois

and Charpillet

, Human activities recognition with rgb-depth camera using hmm, in Engineering in Medicine and Biology Society (EMBC), 2013 35^th Annual International Conference of the IEEE. IEEE, 2013, pp. 4666–4669.

Ali

H.H.

, Moftah

H.M.

and Youssif

A.A.

, Depth-based human activity recognition: A comparative perspective study on feature extraction, Future Computting and Informatics Journal, 2017.

Xia

and Aggarwal

, Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera, in Proceedings of the IEEE.

, Miao

, Zhang

X.-P.

and Tian

, A hierarchical spatio temporal model for human activity recognition, IEEE Transactions on Multimedia, 2017.

Oreifej

and Liu

, Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 716–723, 2013.

Yang

and Tian

, Super normal vector for human activity recognition with depth cameras, IEEE Transactions on Pattern Analysis and Machine Intelligence39(5) (2017), 1028–1039.

10.

Shi

, Li

, Sun

, Hu

and Yin

, P-snv: Pyramid-super normal vector descriptor for human action recognition based on depth sequences, Journal of Information & Computational Science12(18) (2015), 7061–7070.

11.

Rahmani

, Mahmood

, Huynh

D.Q.

and Mian

, Hopc: Histogram of oriented principal components of 3d pointclouds for action recognition, in European Conference on Computer Vision, pp. 742–757, Springer, 2014.

12.

, Zhang

and Liu

, Action recognition based on a bag of 3d points, in Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, pp. 9–14, IEEE, 2010.

13.

Chen

, Liu

and Kehtarnavaz

, Real-time human action recognition based on depth motion maps, Journal of Real-time Image Processing12(1) (2016), 155–163.

14.

Yang

and Yang

, Dmm-pyramid based deep architectures for action recognition with depth cameras, in Asian Conference onComputerVision, pp. 37–49, Springer, 2014.

15.

Xie

, Bi

, Dong

and Jin

, Key frame extraction of skeleton joint based on kinect sensor, in 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress pp. 383–385, IEEE, 2018.

16.

Kamel

, Sheng

, Yang

, Li

, Shen

and Feng

D.D.

, Deep convolutional neural networks for human action recognition using depth maps and postures, IEEE Transactions on Systems, Man, and Cybernetics: Systems99 (2018), 1–14.

17.

Bulbul

M.F.

, Jiang

and Ma

, Human action recognition based on dmms, hogs and contourlet transform, in Multimedia Big Data (BigMM), 2015 IEEE International Conference on, pp. 389–394, IEEE, 2015.

18.

Liu

, Kong

and Wang

, Human activities recognition based on skeleton information via sparse representation, Journal of Computing Science and Engineering12(1) (2018), 1–11.

19.

Park

, Kim

, Baek

S.-Y.

and Lee

, An algorithm for estimating surface normal from its boundary curves, Journal of Computational Design and Engineering2(1) (2015), 67–72.

20.

Fan

R.-E.

, Chang

K.-W.

, Hsieh

C.-J.

, Wang

X.-R.

and Lin

C.-J.

, Liblinear: A library for large linear classification, Journal of Machine Learning Research9 (2008), 1871–1874.

A computationally efficient method for Human Activity Recognition based on spatio temporal cuboid and Super Normal Vector

Abstract

Keywords

1 Introduction

2 Literature Survey

3 Human Activity Recognition based on Adaptive Cuboid and Super Normal Vector – Proposed Method

3.5 Aggregating the super normal vectors

3.7 Frame selection

4 Result analysis and discussions

4.1 MSRAction3D Dataset

4.2 Experimental setup for MSRAction3D dataset

4.4 Experimental setup for UTD-MHAD Dataset

Table 1 Accuracy of proposed system on MSRAction3D MSRAction3D Accuracy (%) Subset 1 89.62 Subset 2 84.96 Subset 3 97.32 Over all 89.82

4.4 Space comparison

Table 3 Space comparision of proposed system with SNV [1] Method SNV [1] Without frame selection With frame selection Normals 729 MB 390 MB 252 MB Descriptor 2.33 GB 1.22 GB 1.20GB

References

Table 1
Accuracy of proposed system on MSRAction3D

MSRAction3D Accuracy (%)

Subset 1 89.62

Subset 2 84.96

Subset 3 97.32

Over all 89.82

Table 3
Space comparision of proposed system with SNV [1]

Method SNV [1] Without frame selection With frame selection

Normals 729 MB 390 MB 252 MB

Descriptor 2.33 GB 1.22 GB 1.20GB