Violence action recognition using region proposal in region convolution neural network

Abstract

Having an autonomous system to alarm for violence or suspicious incidence could greatly strengthen the security system. Such autonomous system could also be useful for other application such as patient monitoring, retail shop, and children surveillance. However, the current technology has not yet reach the level to effectively analyze the video since currently most video surveillance system could not understand the events happen in the video. Complex changes in environment caused by camera motion, dynamic scene such as crowds, changes in lighting intensity, viewing from different angles, wide variation in spatial (e.g. size of interest subject relative to video) and temporal (speed of the subjects in performing actions) make video analysis task a very challenging task. Even with these difficulties, researches in improving video analysis methods are still being actively explored. Some research approaches in violence incidence detection resembling the method used in detecting abnormal incidence. Instead of detecting whether an incidence have occurred, we attempt to build a model to detect the actions related to violence. In this paper, an online detection model is built to detect specific action related to violence actions. The model is built with reference of the image object detection (Faster-Region Convolution Neural Network, Faster-RCNN) and video action detection (Tube-Convolution Neural Network, TCNN).

Keywords

Online model video action detection

1. Introduction

Recently, there has been a lot of research works on video understanding related problem due to the increase demand of video analytics. The general tasks in this problems includes extracting video features representation [1, 2, 3], detecting actions [4, 5, 6], detecting anomalies [7, 8] and description tagging [9]. One of the motivations behind these researches is to enhance the surveillance system to autonomously detect specific event occurrence in video [6].

For detection tasks, it is commonly relates to video feature extraction, proposal generation, localization and classification [4, 10, 11]. Video action feature extraction itself is one of the popular research fields. In earlier approaches, various kind of hand-craft methods were proposed. However, hand craft features were considered not sufficiently discriminative [14, 15, 16], because they capture low level feature which may looks similar as the action classes grow. Recent approaches mostly employ the deep learning approaches to extract video action features [1, 3]. Deep learning approaches could be trained to extract both low and high level features [1]. Furthermore, most of the deep learn based classification approaches obtain better result than hand craft method [1, 3, 17]. After extracting the video features, a set of proposals can be generated to roughly capture the actions within the scene [4, 10] for the purpose of action localization and classification.

Attempts in detecting violence or suspicious action are still rather uncommon. Most of the available methods are implemented as abnormal or anomaly detection [8, 18, 19, 20]. However not all anomaly is related to violence. In some specific context such as ATM surveillance, the anomaly could be directly related to suspicious event [20]. Approach proposed in [21] is trained on Hockey fight dataset which directly related to violence detection.

In this paper, focus will be given onto specific suspicious actions such as kick, punch, hit, shoot and throw. The task is almost similar to the typical classification or detection tasks but focusing on action classes which related to violence action. The detection model designed mainly uses the region proposal network in faster-Region Convolution Neural Network (faster-RCNN) [10] with Tube-Convolution Neural Network (T-CNN) model. Faster-RCNN is proposed to detect objects from images, while T-CNN is proposed for offline action detection which requires full length video as input. Some minor change is made on the pooling to enable the model to work in online mode. In this paper, we modify the pooling operator to take both new input clips and previous pooled result as input. When consecutive clips are feed into the network, the strong activation from different clips will be pooled into a fixed size output. The result of this accumulated pooling should be similar to the original pooling proposed in T-CNN when full length video input is used as input. Even without full length video feature, the accumulated pooling feature should be able to capture some feature of full length action. This could help the model to recognize an action without full action video. In this paper, we described previous related work in Section 2. We further describe our proposed network in Section 3, and the results and discussions are presented in Section 4. Lastly , we conclude our work in Section 5.

2. Related work

Many efforts have been made on finding a good video action representation [1, 3, 12, 22, 23]. In earlier approaches, most features were represented by hand-crafted features. Spatial Temporal Interest Points features (STIP) [12] detects the Harris 3D corners at multiple spatial-temporal scales. Each point is described by Histogram of Gradient (HOG) and Histogram of Optical Flow (HOF). In 2013, Dense Trajectory [13] feature was proposed. Unlike STIP, Dense Trajectory samples the video trajectories of whole video frame with equally spaced x and y interval. The features are described by HOG, HOF and Motion Boundary Histogram (MBH). From their experiment, they found that dense sampling could outperform point based feature. MBH descriptor outperforms HOF partly due to the camera motion. In their later work, Improved Dense Trajectory (IDT) is proposed [24]. IDT incorporates the motion cancelation with MBH and human detector. Other action classification approaches often report IDT could improve their classification accuracy [17, 25, 26].

Most Hand-crafted features are low level features that are insufficiently discriminative [14, 15, 16]. More discriminative hand craft feature, Action Bank is proposed in [15] as high level hand craft feature. It works by matching input feature to a set of template action features in action bank. Tested on KTH and UCF sport dataset, action bank could perform better than the STIP. However, the mean accuracies of Action Bank and STIP on HMDB51 dataset are only 26.9 and 20.2% respectively. MotionLets proposed by [14] as mid-level feature which measures and rank the saliency motion feature performs better than other hand craft features , with the score on HMDB51 of 33.7%.

Other than Hand-crafted feature, deep learning action feature are also often used in recent video based problems [1, 9, 27]. Inspired by the success of Deep Learning in image based problem, efforts of adapting image deep learning model to video is proposed recently [1, 5, 28]. Different deep learning networks are proposed in [28] to extract feature from multiple frames at different stages. Multi-stream deep learning feature is also proposed to extract spatial and temporal feature separately [3, 29]. The spatial feature extractor often derived from existing pretrained ImageNet [30] deep learning model, while the temporal stream are trained to capture temporal feature. In [3], the temporal stream takes optical flow of the video as input. These models also contain fusion layers to integrate spatial and temporal feature together before feeding to classifier. Both [3, 29] shows that temporal features are necessary for action classification tasks. Convolution 3D (C3D) deep learning model is proposed by using 3D convolution filter kernels and 3D max pooling operators [1]. C3D model is much simpler compare to other video base deep learning models. It only uses 3D convolution filter, 3D max pooling and ReLU activation function. Similar to the findings in VGG-14 [26], C3D also use small filter kernel such as 3 $\times$ 3 $\times$ 3 produce better result compare to larger kernel. Larger filter kernel tends to distort the original image/video information at early layer of the network. From the Deconvolution analysis in Convolution 3D (C3D) [1] the low level feature such as moving edge, short changes and edge orientation are extracted at early convolution layers. Higher level feature such as body parts and trajectories are extracted at deeper layers.

Table 1
Performance summary of various video classifier and detection

Models classifier	Features	Year	Reported mean accuracy
			UCF101	HMDB51	J-HMDB
SVM- $\chi^{2}$ [29]	3-Stream CNN (Raw RGB, Optical Flow and MSDI)	2017	92.1	68.3	–
HC-MTL [16]	STIP (Harris3D) with HOG, HOF descriptor	2017	76.3	51.4	–
SSD [31]^(detection)	RGB frame and Optical Flow	2017	65.7	–	64.7
SVM [5]^(detection)	C3D features (contain additional layers for action box regression)	2017	94.4	–	67.2
SVM [3]	2-Stream CNN (RGB and Optical flow), FV $+$ IDT	2016	93.5	69.2	–
Support Vector Regression (SVR) [22]	IDT $+$ FV to form new representation, Video Darwin	2015	–	61.8	–
Linear SVM [17]	2-Stream CNN, IDT	2015	91.5	65.9	–
SVM [25]	Stacked Fisher Vector, Improved Dense Trajectories with HOG, HOF, and MBH descriptor.	2014	–	66.79	69.03
SVM-RBF- $\chi^{2}$ [2]	Dense Trajectories with HOG, HOF, MBH descriptors (this set of feature is referred as IDT)	2013	–	57.2	–
Linear-SVM [14]	Montionlets (combine with STIP and Action Bank)	2013	–	42.1	–
SVM [15]	Action Bank	2012	–	26.9	–
Non-linear SVM [12]	STIP	2008	–	20.2	–

Other than just classifying actions in videos, some approaches work on localizing (detecting) or generating proposal volumes to contain actions. Inspired by the success of image VGG network [23], Action Tublet Detector (ACT) [31] used VGG network with 2 convolutional layer branches. One layer is for scoring action purpose, while another one is for localizing the tube to contain the complete action. The drawback of this method is the challenge in the regression of the action tubelet when the actor is moving away from the initial location of the box.

An attempt to detect and localize action with actor’s pose in online manner is proposed in 2016 [6] by employing super pixel extraction and convolutional pose machine [32] estimator. The foreground and background super pixel are estimated to refine the estimation of actor’s poses. To classify the actions, they proposed the Structural-SVM which could increase the classification confidence over the time as more frames of the action are collected. However this model performance is largely affected by the initial prediction of actions, and would affect the overall accuracy of the localization and prediction.

The Tube Convolutional Neural Network (T-CNN) is proposed in [5]. Within T-CNN, Tube Proposal Network (TPN) is built from the C3D and proposal network. Similar to Faster-R-CNN (Region Proposal Network), the C3D features is used to estimate foreground and background class confidence along with a proposal bounding boxes estimation. The proposal boxes are either dropped or feed to the next stage based on the action score, and then processed by a set of fully-connected layers to localize the actions tubes. These tubes are linked with consecutive clips based on their overlapping criterion and actioness criterion. They also introduced Tube-of-Interest (ToI) pooling as a 3D version of Region of Interest (ROI) Pooling from Fast-RCNN [33]. ToI pooling samples the maximum response from changing sized tube features map and produce a fixed length action features. The tested classification score is around 7% higher than the original C3D classifier [1].

Violence scene detection method is proposed in [21]. The feature is represented in form of hand-craft STIP and Motion Scale Invariant Feature Transform (motion-SIFT). Extracted feature is classified by SVM. This model is trained and tested on their Hockey Fight dataset. Histogram of Gradient (HOG) also used in [20] with Random Forest Classifier to detect Abnormal event on their ATM data set. More related to scene anomaly detection [8] proposed a method to use Alexnet feature on auto-encoder. The auto encoder is trained with normal video data. During testing with anomaly video the auto encoder produces a distinguishable feature. This abnormal feature is detected with Gaussian distribution.

Figure 1.

The detection network used in the experiment (classification network in Fig. 4).

To account the shortage of video action annotations [34] proposed a model to localize actions in videos from images. Large sum of images related to actions are obtained from Google image search. Noisy images are eliminated with random walker algorithm by discovering outlier clusters. Image based object detector [35] is used to localize actors or action performed within images. The image action features is extracted with image CNN. Actions proposals detected in a video are ranked to obtain localized action videos. This ranking is worked by comparing the extracted Google image features with the image features extracted from the proposals detected in video.

Earlier dataset like KTH (6 actions), IXMas (13 actions) and Weizmann (10 actions) were regularly used as testing and training benchmark [36]. Most of the classifier including earlier hand-crafted model [12, 14, 15] could achieve mean accuracy higher than 90%. CAVIAR dataset was one of the early dataset introduced in 2003 and 2004 [37]. It is recorded from acting different scenarios such as walking alone, meeting with others, window shopping, entering and exiting shops, fighting, passing out and leaving a package in a public place from a fixed CCTV.

Recently introduced action dataset is more challenging and comes with more action classes. The current commonly used datasets in video classification and localization are HMDB51 [38], UCF101 [39] and YouTube [40]. These new dataset contain more complex actions. This complexity comes from moving camera, multiple actors, changing viewing angles and changing lighting. Some of the action classes may contain some similarities such as walking and running actions [16].

Figure 2.

The image above illustrate data flows in Region Proposal Network layers.

The HMDB51 consists of 51 different classes, mostly from movies. Each class contains at least 101 clips. It is one of the most challenging dataset available publicly. As shown in Table 1, testing accuracy on this dataset is usually lower than other datasets. J-HMDB dataset is the subset of HMDB51 which give more focus on the action of upper body part. UCF 101 is a large sport video dataset with 101 classes from 13320 videos. YouTube-8M (8 million videos) dataset contains 450000 hours of videos for 4716 classes from public video available in YouTube video database. This amount of video makes Youtube-8M the largest video database available.

More on violence related video dataset are available from VSD [41] and Visilab [21]. Visilab contributed a fighting dataset from hockey gameplay recording in 1000 short clips. Visilab prepared 100 fighting scenes dataset from movie accompanied by non-fighting scene for negative sample training. VSD dataset consists of both Hollywood (from 31 movies) and Youtube (86 videos) fighting scene video.

As a summary, different approaches to extract action representative features were introduced to recognize different actions. With more developed framework, these features have been used to perform more complex tasks. One of these tasks is the action detection task. This task generally involves feature extraction, regression capture foreground actions, and classification to identify the actions. Although deep learning related frameworks have been producing some improvement, they however require large amount of training datasets. Newer video action datasets mainly are taken from movies which contain more realistic samples. However most of them are not welly annotated.

3. Violence detection network

The proposed network is mainly derived from the T-CNN [5]. T-CNN is chosen as the base framework because of its achieved detection accuracy, speed and simplicity. In T-CNN, the proposal generation is performed through sliding window method. While in our proposed network, we replace the proposal generation network with Region Proposal Network (RPN) to speed up the proposals generation process. The RPN was proposed in Faster-RCNN [10] as image object proposal generation. The structure of action proposal and localization network is shown in Fig. 1. Furthermore the original T-CNN was proposed as offline detection model. In this paper, we modify the T-CNN’s pooling and tube linking operator to accumulate the convolution feature response. The modification is made at the classification network to allow the network to operate as online detection model. The structure of classification network is shown in Fig. 4.

3.1 Feature extraction

In this paper, we applied T-CNN to extract relevant features. Following the approach in T-CNN, Each pretrained C3D network take input of 8 frames clip. Each convolution layers in C3D mainly consists of Convolution, Pooling and Activation function. The C3D extraction network consists of 5 layers of convolution layers consisting of 3D filter kernels, followed by 3D max pooling and also ReLU activation function. The output of C3D features after the fifth convolution (Conv5b’s) serves as the main feature for proposals generation. Besides, the second convolution layer Conv2a feature is also used as localization and classification features.

3.2 Region proposal network

Region Proposal Network was proposed in faster-RCNN [10] as proposal generation network. The generated proposals provide rough estimation for foreground regions. Similar to the faster-RCNN, the RPN is used to generate action proposal in this model. The first layer of RPN in Fig. 2 (RPN-Conv/3 $\times$ 3) performs a 3 $\times$ 3 convolution on the Conv5b feature map resembling the operation of sliding window. The feature map comes in form of image of multiple channels. Each pixel on this image represents the response of a 16 $\times$ 16 patch from the original video. The output of this 3 $\times$ 3 filter layer is fed into two different 1 $\times$ 1 convolution layers, which are “RPN class score” layer and “RPN bbox pred” layer. “RPN class score” layer estimates the foreground/background score for different bounding boxes, while “RPN bbox pred” layer estimates the deltas to shift and scale bounding boxes to the location of the actions as shown in Fig. 2. RPN class score has the same spatial size as RPN-Conv/3 $\times$ 3 layer, but it contains 2 $\times$ $k_{\textit{bounding boxes}}$ . Every 2 channels of this feature map represent the foreground and background response for each bounding boxes. However these bounding boxes are only centered at the patch they come from. The layer of “RPN bbox pred” also has the same spatial size as “RPN-Conv/3 $\times$ 3”, but it contains 4 $\times$ $k_{\textit{bounding boxes}}$ . Every 4 channels of this feature map represent the transformation delta $(dx,dy,dw,dh)$ .

Based on the implementation of T-CNN and DarkNet [11] in object detection network, a number of bounding box size ratios are selected using K-means from ground truth annotation boxes. This will speed up the training process by initializing the bounding boxes at better width and height scale. The bounding boxes coordinate are represented as a 4-D vector $(x_{1},y_{1},x_{2},y_{2})$ representing upper left and lower right coordinate. In this experiment, coordinates are normalized to range of 0 to 1 in order to cope with changing sized feature maps and video input shape.

The “rpn bbox pred” layer estimates the deltas $(dx,dy,dw,dh)$ that transform the base bounding boxes width, height and position to roughly capture the actions. To obtain the proposal boxes, the bounding boxes are transformed as follows:

$\displaystyle\textit{Center}(X,Y)_{\textit{actual scale}}=$ $\displaystyle\quad(dx\times\textit{Width}_{\textit{bbox}},dy\times\textit{% Height}_{\textit{bbox}})+$ (1) $\displaystyle\quad(x,y)_{\textit{feature coordinate}}\times 16$ $\displaystyle\textit{Size}(W,H)=\left(e^{\textit{dw}}\times\textit{Width}_{% \textit{bbox}},e^{\textit{dh}}\times\textit{Height}_{\textit{bbox}}\right)$ (2)

Where $X, Y$ represent the center pixels location of a proposal box, while $W, H$ are denoted as width and height of the proposal box [10]. To train these RPN layers, the similar training method used in Faster-R-CNN [10] is used. After the video feature is extracted by C3D layers, the video feature spatial size is reduced by 2 ${}^{4}$ times due to the 4 Max Pooling layers within the C3D network. Therefore, each cell on Conv5 or RPN feature map represents a 16 $\times$ 16 pixel patch in the video. While the video temporal size is only reduced by 2 ${}^{3}$ times due to the spatial pooling in first pooling, the other 3 are 3D pooling. The pooling layer’s kernel size and temporal depth set to 2 to group pixel of 2 $\times$ 2 or 2 $\times$ 2 $\times$ 2 to 1.

When training RPN, $k$ -number of bounding boxes (black boxes) are generated for each cell (red) as show in Fig. 3. The overlapping score, Interest of Union (IoU) with a ground truth (green box) is calculated as follow:

$\displaystyle\textit{IOU}(\textit{bbox}_{i},\textit{ground truth}_{j})=$ $\displaystyle\quad\frac{\textit{Area}_{\textit{bbox},i}\cap\textit{Area}_{% \textit{ground truth},j}}{\textit{Area}_{\textit{bbox},i}\cup\textit{Area}_{% \textit{ground truth},j}}$ $\displaystyle\quad i\in\{1:\textit{Number of cell }\times k\}$ (3) $\displaystyle\quad j\in\{1:\textit{Number of ground truth}\}$ $\displaystyle\quad\textit{bbox}=\textit{the bounding from a cell}$

Figure 3.

Illustration of bounding boxes generated for a cell on the feature map.

A number of bounding boxes with best IOU are selected and the rest are treated as background. In T-CNN approach, the background is selected by comparing “RPN Cls Score” with the expected class feature. The prediction with high foreground confidence but overlapped badly with ground truth will be chosen as background. And the prediction that highly overlapped with a back-ground annotation box will be selected as hard negative. Some video may contain multiple actors performing the actions to be detected (e.g. kick, punch), while others may perform uninterested actions or background action (e.g. standing, sitting). Hard negative annotations are mainly annotating people that are performing background action. The hard negative helps RPN to differentiate between foreground and background action.

Besides, the number of selected background and foreground boxes must be equally distributed. Otherwise the network might learn to produce more negative prediction to achieve low loss instead of learning to predict correct result. This may happen as background appears much more often than foreground. To train “RPN Bbox Pred” layer, the transformation deltas are computed for each selected foreground boxes. The transformation deltas can be computed as follows:

$\displaystyle\!\!\!\!\!\!\!\!\!\!\!\!\delta_{x}=dx=\frac{\textit{ground truth % center }x-\textit{estimation center }x}{\textit{base bounding boxes horizontal% size}}$ (4) $\displaystyle\!\!\!\!\!\!\!\!\!\!\!\!\delta_{y}=dy=\frac{\textit{ground truth % center }y-\textit{estimation center }y}{\textit{base bounding boxes vertical % size}}$ (5) $\displaystyle\!\!\!\!\!\!\!\!\!\!\!\!\delta_{w}=dw=\ln\left(\frac{\textit{% ground truth width}}{\textit{base bounding boxes horizontal size}}\right)$ (6) $\displaystyle\!\!\!\!\!\!\!\!\!\!\!\!\delta_{h}=dh=\ln\left(\frac{\textit{% ground truth height}}{\textit{base bounding boxes vertical size}}\right)$ (7)

where $\delta_{x},\delta_{y}$ are the offset of bounding boxes of a cell towards the closest ground truth and $\delta_{w},\delta_{h}$ are the scaling factors to transform the bounding box’s size to the size of ground truth boxes in width and height [10]. These deltas are used to move ( $\delta x,\delta y$ ) and scale ( $\delta w,\delta h$ ) a selected foreground boxes to the closest ground truth. Only the selected foreground/background in “RPN Cls Score” layer and foreground boxes in “RPN Bbox Pred”, are contributing to the loss computation. The unselected cells or boxes are masked with ignore label or weighted with zero.

After training process, the prediction of the RPN layers can be used to generate proposal boxes. These proposal boxes capture the potential for classification. At each pixel on the RPN feature map, the estimation of bounding boxes are computed with Eqs (1) and (2). The RPN tends to generate large amount of proposals, therefore Non-Maxima-Suppression (NMS) is used to reduce overlapping proposals. It will remove highly overlapped boxes (when 2 boxes overlapped more than a threshold).

3.3 Localizing proposal boxes

After the proposal generation and filtering, the proposal bounding boxes coordinates are used to crop the feature map in Conv2a and Conv5b. This method was proposed in T-CNN as temporal skip pooling. It is used to recover the loss of temporal information in Conv5b by sampling Conv2 feature. These features are cropped out with the TOI Pooling layer based on the proposal boxes.

For each proposal boxes, the feature in C3D’s, Conv2a and Conv5b are cropped and Max-Pooled into fixed tube size of 1 $\times$ 128 $\times$ 8 $\times$ 8 $\times$ 8 and 1 $\times$ 512 $\times$ 8 $\times$ 4 $\times$ 4 feature map pair (following the caffe-C3D blob dimension format $N_{\textit{batch}}\times C_{\textit{channel}}\times L_{\textit{length}}\times H% _{\textit{height}}\times W_{\textit{width}})$ , where $N_{\textit{batch}}$ is the input batch size (or number of video clips used per iteration). $C_{\textit{channel}}$ is the feature channel (RGB video contains 3 channels, the channels size can change depend on the convolution kernel output). $L_{\textit{length}}$ is the number of frame within a batch or feature. $H_{\textit{height}},W_{\textit{width}}$ are the width and height of the feature map. The cropped Conv5b is a single frame feature map 1 $\times$ 512 $\times$ 1 $\times$ 4 $\times$ 4. It is duplicated 8 times to match the temporal size of the Conv2a. Each feature map pair is normalized, serialize and concatenated to form a $N_{\textit{proposals}}\times 8_{\textit{length}}\times 2_{\textit{channel}}% \times 8192$ feature vectors. The 1 $\times$ 1 convolution introduced in GoogleNet [42] is used to reduce the feature dimension into $N\times$ 8 $\times$ 1 $\times$ 8192. This dimension reduction is mainly used to reduce the number of parameters that need to be trained. This feature map consists of $N$ set of features for each proposal boxes. And each feature from a proposal tube has 8 set of 8192 feature vectors. Each 8192 long vector represent the feature for a single frame within the 8 frame clips. Therefore there will be $N\times$ 8 set of 8192 feature vectors feed into the fc6 to fc7. The last fully connected layer takes $N\times$ 8 set of 4192 feature vectors as input. It produces $N\times$ 8 set of 4D vector deltas to localize $N$ set of proposal boxes to form the action tube coordinates.

3.4 Action classification

The block diagram in Fig. 1 summaries the detection network up until localization network. After the obtaining the localized tubes, the feature within these tubes are used to classify the actions. Besides, the tubes from consecutive predictions may be generated from the same actions. These tubes are linked depending on overlapping score and average foreground score. Overlapping score comes from the overlap between last frame of first tube and first frame of second tube. The linked tubes are selected with the following equations:

$\displaystyle S\!=\!\frac{1}{m}\!\sum_{i=1}^{m}\textit{Actioness}_{i}\!+\!% \frac{1}{m\!-\!1}\sum_{j=1}^{m-1}\textit{Overlap}_{j,j+1}$ $\displaystyle\textit{Overlap}_{j,j+1}=\textit{IOU between last box in tube $j$}$ (8) $\displaystyle\phantom{\textit{Overlap}_{j,j+1}=}\textit{and first box in tube % $j+1$}$

Where $m$ represents total number of clips, actioness represents actioness score from i-th clip, while overlap represents IOU between last box in the tube $j$ and first box in tube $j+1$ . During training process, each localized tube coordinate is matched to closest ground truth. The tubes which overlap with foreground/background (IOU higher than 0.7) are selected. The selected tubes will be used to link tubes of 2 consecutive clips as proposed in T-CNN.

In T-CNN framework, localized tubes are generated from TPN network. Consecutive tubes from $m$ video clips are linked when $S$ score is greater or equal to a threshold (0.7). The linked tubes are used with Temporal of Interest (ToI) Pooling proposed in T-CNN [5] to poll a fixed feature size for the classifier to estimate the class scores. ToI Pooling is the extension of the Region of Interest (RoI) pooling proposed in [43]. RoI is normally implemented on image while ToI is applied on video frames. The ToI pooling takes localized tube coordinate and tube feature map to crop the tube features.

To adapt the T-CNN for online detection, the linked action score, overlap score and ToI pooled features are always saved into a buffer. When a tube is linked with previous clips, the new tube features is pooled into the buffer. Distinct features of an action is expected to accumulate in this buffer as an action is observed for a few clips. This pooled tube buffer feature is used as input to the classifier. A 1 $\times$ 1 convolution is used to reduce the Conv2a ( $N\times 128\times 8\times 8\times 8$ ) pooled feature’s temporal size to ( $N\times 128\times 1\times 8\times 8$ ). This reduced feature is concatenated with the pooled Conv5b feature to form a two channel features ( $N\times 2\times 8192$ ). One channel is from reduced Conv2a and another channel is from Conv5b. Another 1 $\times$ 1 convolution is used to reduce feature to size of ( $N\times 8192$ ). This feature is feed into fully connected layer (fc6 and fc7). A drop out layer is used to reduce the effect of overfitting and the last fully connected layer estimate the class score. The class score is converted to probability with using softmax function.

Table 2

Number of training clips used in the experiment according to action class

Phase	Number of foreground clips
	Kick	Punch	Hit	Shoot gun	Throw
Training action proposal	330	535	209	547	154
and localization
Total	1775

4. Result and discussion

4.1 Dataset

The dataset used in this experiment are taken from HMDB51, VSD’s movie fighting scene and hockey fighting scene. For this experiment, only 6 classes are manually annotated (kick, punch, hit, shooting, throw and background actions). The annotated videos are sliced into 8-frame clips. Each clip will have at least one or more foreground annotation. A total of 1785 clips are annotated. For each class, 10 random clips are removed from training dataset for testing. The standard test train ratio is not used in this experiment because of lacking of training sample. We show the details of the data in Table 2.

Table 3
Cross validation True/False Positive/Negative condition table

	Positive sample	Negative sample
Positive prediction (confidence score $>\alpha$ ) AND (IOU $\geqslant$ 0.5)	True positive	False positive	All positive prediction
Negative prediction (score $<\alpha$ ) AND (IOU $\geqslant$ 0.5)	False negative	True negative	All negative prediction
	Positive examples	Negative examples	Total

Additionally, if the IOU is below threshold (0.5), it is considered as False Positive if its score $>$ $\alpha$ and it is predicted as foreground. It is considered as True Negative for other cases of IOU below threshold.

The videos sequence is also randomized during the training. The trained action proposal network is used to extract the video features for localization network training. While the trained action localization network is used to extract the video feature for classification network training. The extracted features for classification network are shuffled and combined to form 355 batches dataset. Each batch contains the feature sample for 6 different action classes. Some action features are not used in the classification trainings to regulate the sample size for each action class.

4.2 Parameter setting

The network is implemented with C3D Caffe branch. The training is carried out separately (action proposal, localization and classification network) to ease the debugging process. The pretrained C3D weights on Sport 1M dataset is fine-tuned on the dataset while training action proposals. The proposal features are extracted to train the localization network. The features extracted are used to train the classification network.

The action proposal and localization networks are trained with single batched SGD optimization. The learning rate used is 0.001, momentum of 0.9, weight decay of 0.0005 following the standard setting [30]. The learning rate proposed in [30] is 0.001 but changed to 0.0001 in this experiment to prevent unstable learning loss. The weights are updated in every backward iterations (1775 weight update chances).

For classification network, the training batch size is set to 6. Using batched training produces more stable loss curve, but it reduces the number of back-propagations and weight updates. Adaptive Momentum (Adam) and Adaptive Gradient (Adagrad) optimization is used in training because of faster drop in training loss. The weights parameters are updated in every iteration.

4.3 Result evaluation metrics

4.3.1 Precision-Recall curve (PR curve)

The accuracy of the detection is commonly measured with Precision-Recall method used in PASCAL Visual Object Classes VOC Challenge [44]. Recall, $r$ parameter is defined as proportion of all positive samples above a certain confidence. Precision parameter is the portion of all prediction above rank $r$ , from positive class.

$\displaystyle\textit{measured recall},r(\alpha,\textit{IOU})=$ $\displaystyle\quad\frac{\textit{True positive}}{\textit{All positive examples}}$ (9) $\displaystyle\textit{recall level},\bar{r}=[0,0.1,0.2\ldots 1.0],$ (10) $\displaystyle\textit{measured precision},p(\alpha,\textit{IOU})=$ $\displaystyle\quad\frac{\textit{True positive}}{\textit{All positive % prediction}}$ (11) $\displaystyle\bar{p}=\textit{coressponding precision at recall}\bar{r}$ (12)

Precision-Recall ( $\bar{p}$ , $\bar{r}$ ) curve can be very noisy, hence the proposed method takes an extra step to interpolate the value [44].

$\displaystyle P_{\textit{interpolated}}(\bar{r})=\max p(\hat{r}),$ (13) $\displaystyle\quad\text{where }\hat{r}:\hat{r}\geqslant\bar{r},\textit{max % precision above rank level }\bar{r}$ $\displaystyle\textit{AP}=\frac{1}{11}\sum_{\bar{r}}P_{\textit{interpolated}}(% \bar{r})$ (14)

Figure 4.

Block diagram above show structure of action classifier.

Figure 5.

The testing Recall-Recall curve of the action proposal network.

Figure 6.

The action proposal testing ROC curve.

Figure 7.

Example output of the RPN during testing. Left column shows the foreground confidence (enlarged 16 times to match the video size).

Figure 8.

Some example of testing output. Images on left are proposal bounding boxes. Images on right are the localized proposals with non-maxima suppression. The localization manage to localize some proposals to better capture the actors.

Figure 9.

Failing cases of the RPN and localization.

Figure 10.

Training loss of the classification network.

Figure 11.

ROC plot and Recall-Precision curve of the overall detection accuracy.

The accuracy precision (AP) for a class prediction is calculated by finding the mean of the Precision with equally spaced Recall level. This metric is mainly used to compute the accuracy for a two class prediction (foreground and background). To compute the AP for multiclass task, the AP is evaluated separately. And the mean AP (mAP) is computed by taking the average of the AP computed for each class.

4.3.2 Receiver Operator Characteristic (ROC)

ROC Accuracy metric is also used in some papers of similar field [1], [4, 5]. Using back the similar confusion matrix (Table 3), the ROC are plotted with sensitivity and 1 – specificity.

$\displaystyle\textit{Sensitivity}=\textit{recall}=\frac{\textit{True positive}% }{\textit{All positive examples}}$ (15) $\displaystyle\textit{Specificity}=\frac{\textit{True Negative}}{\textit{All % Negative Examples}}$ (16)

The ROC is often related to another accuracy metric AUC which is computed from finding the area under ROC.

4.4 Experiment result

4.4.1 Action proposal

The Recall-Precision plot in Fig. 5 shows the accuracy of the foreground/background detection by varying threshold value, $\alpha$ . With low $\alpha$ , more prediction will be considered as positive; this will increase the Recall rate ( $x$ -axis). When increasing $\alpha$ , higher classification confidence is required to consider a prediction as a positive prediction. This made the model become more stringent and at the same time cause the Precision increase ( $y$ -axis). Without filtering low confidence boxes (Greedy RPN), the overall precision is lower compared to the filtered RPN (foreground confidence $\geqslant$ 0.7). In the test dataset, each class contain 10 samples. Therefore a random classifier would have a $\frac{5}{6}$ chance to correctly predict a foreground (precision $=$ $\frac{5}{6}$ ). While a random classifier could also correctly predict a background at precision $=$ $\frac{1}{6}$ . Figure 5 shows action proposal prediction is worse than random classifier when (recall $=$ 0.6.

The ROC curve plotted in Fig. 6 shows the relationship between true positive ( $y$ -axis) and false positive rate ( $x$ -axis). Similar to the AP Precision-Recall curve, the ROC plot is plotted with changing value of $\alpha$ . The low confidence filtering show better true positive rate and false positive rate (low left corner of the RPN ROC plot). However, the filtering also causes a slight drop in True Positive rate when the False Positive rate around 0.4. This shows that some of the foreground detection with slightly lower score are rejected by the filtering. But as a whole, the filtering increases the accuracy measured by both ROC’s AUC and Recall Precision – AP metrics. The ROC curves in Fig. 6 show the true positive rate $>$ false positive rate shows the prediction is better than random classifier.

The brighter region in the foreground confidence image (Fig. 7) is formed by averaging the confidence score of 12 boxes. This confidence map shows the location the proposal boxes spawn. Ideally, it should spawn near to an actor so the RPN proposal localization and the TPN localization could be much easier to predict. The trained RPN tend to capture human or motion within the scene. The green boxes are proposal boxes spawn from boxes higher or equal to 0.7. The RPN still captures the background actors as foreground, as shown in the fourth row of Fig. 7. This is most probably due to insufficient samples of dataset that come with background actors.

4.4.2 Localization and classification

The proposal network estimates the rough position of foreground action (left column in Fig. 8). These proposals are filtered by a non-maximal suppression to reduce overlapping proposals. The filtered proposal box is used to crop the C3D features for localization network. The example output of the localization network shown in right column in Fig. 8. These localized tubes are again filtered by another non maxima suppression to remove highly overlapping tubes. The localization’s performance is highly depending on the proposal estimations. Failure of proposal network in detecting the shooting action in Fig. 9 causes the failure in localizing the action.

The best testing accuracy achieved is from the classifier trained with AdaGrad optimization. Adam trained classifier has better accuracy at low recall rate (recall $<$ 0.2, higher probability threshold) in Fig. 11. However the biasness of the trained classifier at producing “Kick” prediction has lowered the overall accuracy. While for AdaGrad trained classifier, it can predict better at shoot gun and punch actions, but other classes’ prediction accuracy is lower.

For both trained classifiers, most prediction are predicted with low confidence score because they are still lacking of training samples. The mean ROC’s area under curve (AUC) achieved by AdaGrad and Adam trained classifier are 0.74 and 0.66 respectively. While the mAP for AdaGrad and Adam trained classifier are 0.39 and 0.34 respectively.

4.5 Discussion

The trained Action Proposal network can estimate the foreground actions, as shown in Fig. 7. The localization network manages to transform the proposal boxes to obtain better a better overlap IOU (in Fig. 8). The annotation was made with intention to capture certain foreground action. However, the network still detects humans in video instead of the actions, as shown in Fig. 7. This might be caused by the rectangle annotation that always contain human body regardless of the action performed and lack of hard negative training samples.

The classification network accuracy is rather low. This is most probably due to lack of training samples. From Fig. 11 both AP and ROC curves show class prediction is slightly better than random classifier except the background class where the ROC false positive $\cong$ true positive. The AP and ROC curve also show the “Kick” and “Shot Gun” action prediction is better than others. This could related to the number of sample used in training the action proposal (Table 2).

From the training loss in Fig. 10, it shows drop out could help in reducing the training loss at faster rate. The drop out disconnection might have emphasized the losses and updates of the connected weights. While training without drop out, most of the weight’s update hold a non-zero value. After normalizing, each weight update becomes smaller. The drop out also introduces more spikes in the training loss, because every iteration, the network is training a different network. The effect of drop out in reducing overfitting is not observable in this case because of low training sample.

Adam is slightly slower than Stochastic Gradient Descent (SGD) in computation speed, but its loss converging speed is the fastest. Regardless of the optimizer used in training, the loss plots in Fig. 10 shows the loss can still reduce with more training samples. When comparing Adam and AdaGrad training loss, there are some obvious spikes appear at the same training iteration. This shows that the training data at that training iteration is rather noisy or different from other samples.

Classification and detection of actions were conducted in others’ works. However annotation and benchmark result for specific violence related actions is not found or publicly available. Furthermore this experiment is conducted in small scale is not comparable with other general action classification benchmarks such as HMDB51.

5. Conclusion and future work

In this experiment, minor changes have been made on the T-CNN’s [5] tube linking and ToI pooling operator to accumulate the responses from consecutive tubes. The proposal and localization network capture some actors performing the actions (kicking, punching, hitting, shooting and throwing). From the testing result, it shows that the proposal can detect and differentiate foreground and background action with 0.6625 mean AP. However the classifier network training could not sufficiently converge the loss because of lacking of training sample. The best detection result obtained with mean AUC (area under ROC curve) and mAP (mean class precision in Recall-Precision curve) are 0.74 and 0.39 respectively.

Localization with rectangle boxes and tubes are simple. However it may include large area of background elements into foreground annotations. Annotating only part of the actor’s body that reflect an action. This may help the proposal network at capturing actions related parts instead of human body. Besides, annotating actor with rectangle box may include large portion of background.

In countering the issue of insufficient data, data augmentation is often used in machine learning. Augmenting video with frame striding was proposed in [29]. Other than that, the image augmentation approaches could apply for video, e.g. random horizontal flip and random patch cropping proposed in [30].

The whole network is separated into 3 parts (proposal, localization and classification). The trainings in later parts does not reflect back to earlier parts of the network. Training the network in End-to-end manner may yield a more accurate result.

Footnotes

Acknowledgments

This work is supported by Malaysia Ministry of Higher Education (MOHE) under Fundamental Research Grant Scheme (FRGS) (ref: FRGS/1/2016/ TK04/TARUC/02/1).

References

Tran

Bourdev

Fergus

Torresani

Paluri

. Learning spatiotemporal features with 3d convolutional networks. Proceedings of The IEEE International Conference on Computer Vision 2015.

Wang

Schmid

. Action recognition with improved trajectories. Proceedings of The IEEE International Conference on Computer Vision 2013.

Feichtenhofer

Pinz

Zisserman

. Convolutional two-stream network fusion for video action recognition. Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition 2016.

Gemert

Jain

Gati

Snoek

. APT: Action localization proposals from dense trajectories. BMVC 2015.

Hou

Chen

Shah

. Tube convolutional neural network (T-CNN) for action detection in videos. IEEE International Conference on Computer Vision 2017.

Soomro

Idrees

Shah

. Online localization and prediction of actions and interactions. IEEE Trans Pattern Anal Mach Intell 2018.

Christiansen

Nielsen

Steen

Jørgensen

Karstoft

. DeepAnomaly: Combining background subtraction and deep learning for detecting obstacles and anomalies in an agricultural field. Sensors 2016; 16(11): 1904.

Sabokrou

Fayyaz

Fathy

Moayed

Klette

. Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes. Comput Vis Image Und 2018.

Mazaheri

Zhang

Shah

. Video fill in the blank using lr/rl lstms with spatial-temporal attentions. 2017 IEEE International Conference on Computer Vision (ICCV) 2017.

10.

Ren

Girshick

Sun

. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems 2015.

11.

Redmon

Farhadi

. YOLO9000: better, faster, stronger. arXiv preprint 2017.

12.

Laptev

Marszalek

Schmid

Rozenfeld

. Learning realistic human actions from movies. IEEE Conference on Computer Vision and Pattern Recognition 2008.

13.

Wang

Kläser

Schmid

Liu

C-L

. Dense trajectories and motion boundary descriptors for action recognition. International Journal of Computer Vision 2013; 103(1): 60-79.

14.

Wang

Qiao

Tang

. Motionlets: Mid-level 3d parts for human motion recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2013.

15.

Sadanand

Corso

. Action bank: A high-level representation of activity in video. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2012.

16.

Liu

Nie

Kankanhalli

. Hierarchical clustering multi-task learning for joint human action grouping and recognition. IEEE Trans Pattern Anal Mach Intell 2017; 39(1): 102-14.

17.

Wang

Qiao

Tang

. Action recognition with trajectory-pooled deep-convolutional descriptors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015.

18.

Cheng

K-W

Chen

Y-T

Fang

W-H

. Video anomaly detection and localization using hierarchical feature representation and Gaussian process regression. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015.

19.

Lee

Nevatia

. Hierarchical abnormal event detection by real time and semi-real time multi-tasking video surveillance system. Mach Vision Appl 2014; 25(1): 133-43.

20.

Tripathi

Mittal

Gangodkar

Kanth

. Real time security framework for detecting abnormal events at ATM installations. Journal of Real-Time Image Processing 2016; 1-11.

21.

Nievas

Suarez

García

Sukthankar

. Violence detection in video using computer vision techniques. International Conference on Computer Analysis of Images and Patterns 2011; Springer.

22.

Fernando

Gavves

Oramas

Ghodrati

Tuytelaars

. Modeling video evolution for action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015.

23.

Simonyan

Zisserman

. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 14091556; 2014.

24.

Wang

Schmid

, editors. Action recognition with improved trajectories. Proceedings of the IEEE International Conference on Computer Vision 2013.

25.

Peng

Zou

Qiao

Peng

. Action recognition with stacked fisher vectors. European Conference on Computer Vision 2014; Springer.

26.

Zhao

Liu

Han

Hong

Tian

. Pooling the convolutional layers in deep convnets for video action recognition. IEEE Transactions on Circuits and Systems for Video Technology 2018; 28(8): 1839-49.

27.

Wang

Liu

Shen

. Order-aware convolutional pooling for video based action recognition. arXiv preprint 2016.

28.

Karpathy

Toderici

Shetty

Leung

Sukthankar

Fei-Fei

. Large-scale video classification with convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2014.

29.

Wang

Fang

. Three-stream CNNs for action recognition. Pattern Recognition Letters 2017; 92: 33-40.

30.

Krizhevsky

Sutskever

Hinton

. Imagenet classification with deep convolutional neural networks. Advances in neural Information Processing Systems 2012.

31.

Kalogeiton

Weinzaepfel

Ferrari

Schmid

. Action tubelet detector for spatio-temporal action localization. IEEE International Conference on Computer Vision (ICCV) 2017.

32.

Wei

S-E

Ramakrishna

Kanade

Sheikh

. Convolutional pose machines. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2016.

33.

Girshick

. Fast R-CNN object detection with Caffe. Microsoft Research 2015.

34.

Sultani

Zhang

Shah

. Unsupervised action proposal ranking through proposal recombination. Comput Vis Image Und 2017; 161: 42-50.

35.

Cho

Kwak

Schmid

Ponce

. Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015.

36.

Reddy

Shah

. Recognizing 50 human action categories of web videos. Mach Vision Appl 2013; 24(5): 971-81.

37.

“CAVIAR Test Case Scenarios”, 2003. [Online]. Available: http://homepagesinf.ed.ac.uk/rbf/CAVIARDATA1/. [Accessed: 28-Jul-2017].

38.

Kuehne

Jhuang

Stiefelhagen

Serre

. Hmdb51: A large video database for human motion recognition. High Performance Computing in Science and Engineering ’12: Springer; 2013. p. 571-82.

39.

Soomro

Zamir

Shah

. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12120402; 2012.

40.

Abu-El-Haija

Kothari

Lee

Natsev

Toderici

Varadarajan

, et al. Youtube-8m: A large-scale video classification benchmark. arXiv preprint 2016.

41.

Sjöberg

Ionescu

Jiang

Y-G

Quang

Schedl

Demarty

C-H

. The MediaEval 2014 Affect Task: Violent Scenes Detection. MediaEval 2014.

42.

Szegedy

Liu

Jia

Sermanet

Reed

Anguelov

, et al. Going deeper with convolutions. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition 2015.

43.

Girshick

. Fast R-CNN. Proceedings of the IEEE International Conference on Computer Vision 2015; 1440-1448.

44.

Everingham

Van Gool

Williams

Winn

Zisserman

. The pascal visual object classes (voc) challenge. International Journal of Computer Vision 2010; 88(2): 303-38.

Violence action recognition using region proposal in region convolution neural network

Abstract

Keywords

1. Introduction

2. Related work

Table 1 Performance summary of various video classifier and detection

3.1 Feature extraction

3.2 Region proposal network

3.4 Action classification

4.1 Dataset

Table 3 Cross validation True/False Positive/Negative condition table

4.3 Result evaluation metrics

4.3.1 Precision-Recall curve (PR curve)

4.4.1 Action proposal

4.4.2 Localization and classification

4.5 Discussion

5. Conclusion and future work

Footnotes

Acknowledgments

References

Table 1
Performance summary of various video classifier and detection

Table 3
Cross validation True/False Positive/Negative condition table