Framework for rare event detection using Artificial Neural Network based context free grammar

Abstract

Rare event detections are performed using spatial domain and frequency domain-based procedures. Omnipresent surveillance camera footages are increasing exponentially due course the time. Monitoring all the events manually is an insignificant and more time-consuming process. Therefore, an automated rare event detection contrivance is required to make this process manageable. In this work, a Context-Free Grammar (CFG) is developed for detecting rare events from a video stream and Artificial Neural Network (ANN) is used to train CFG. A set of dedicated algorithms are used to perform frame split process, edge detection, background subtraction and convert the processed data into CFG. The developed CFG is converted into nodes and edges to form a graph. The graph is given to the input layer of an ANN to classify normal and rare event classes. Graph derived from CFG using input video stream is used to train ANN Further the performance of developed Artificial Neural Network Based Context-Free Grammar – Rare Event Detection (ACFG-RED) is compared with other existing techniques and performance metrics such as accuracy, precision, sensitivity, recall, average processing time and average processing power are used for performance estimation and analyzed. Better performance metrics values have been observed for the ANN-CFG model compared with other techniques. The developed model will provide a better solution in detecting rare events using video streams.

Keywords

Artificial Neural Network Context-free grammar Rare event detection streaming video monitoring

1 Introduction

The modern world is full of surveillance cameras everywhere and they produce an enormous volume of data constantly. Monitoring the activities that are happening in front of numerous cameras and triggering an alarm for unusual events are merely a complicated task for the human. Therefore, some rare event detection procedures are used to find out the unusual events from a regular video stream. These procedures are majorly adopted based on the spatial domain or in the frequency domain. In this work, the spatial and frequency domains are enclosed by the CFG representation. Further, the input data are converted into graphs to be processed in ANN.

In general, grammar is used to define the programming structure of a computer language. In general, CFG [13] is the best mechanism for describing a structure mathematically in a simple and cherished way and is used widely in computer applications because of its recursive-definition nature. There are some predefined types in CFG such as, one-to-one, and one-to-many and the rules in CFG are free to use with any context. The ability to check the grammar with reversed rules makes it possible to use computer applications naturally. Languages generated by CFG are called Context-Free Languages, which are not entailed for handling adjectives and adverbs which are important for natural languages, instead, context-free language can handle nouns and verbs, which are important for computer programming languages. A CFG is defined by 4-tuple G = (V, Σ, R, S) where V is a finite set of elements, Σ is the finite set of terminals, R is the finite relation from V, and S is the initial variable. Presently, CFG is an unavoidable concept in computer science because the widely used Extensible Markup Language (XML) [7, 12] is completely dependent on CFG. Automating web-related tasks and web searching in these days are performed by CFG based XML. In this work, the concept of CFG is used to find rare event occurred in a video stream.

The aspire of developing ACFG-RED is to increase the quality of the video mining process by improving the parameters such as accuracy, precision, and recall. The algorithmic design of ACFG-RED is also constructed to improving the speed with reasonable power consumption. Accuracy is one of the important parameters used in mining and classification procedures. Accuracy is defined as the closeness of a metered value to the actual value. The value of accuracy is directly proportional to the quality of a predicting model or procedure. The prediction model provides the highest accuracy means, this procedure is said to be the best in its class. Precision is defined as the closeness between more than one measured values with each other. Precision is also directly proportional to the quality of a classification or prediction model. A recall is another important parameter to determine the quality of a classification procedure.

The rest of the paper is structured as follows. Section 2 discusses the overview of some existing related work in this field. Section 3 discusses the system model used in the proposed scheme. Section 4 presents the experimental setup of the proposed scheme in detail. Section 5 gives a comparative analysis of the proposed protocol regarding the accuracy, precision, recall, average processing time, and average processing power with related existing schemes. Finally, Section 6 concludes the paper.

2 Existing methods

Rare event detection is the process of finding abnormal events in a video stream or frame. To do that, in splitting, the key frames, and subsequent frames, nullifying constant background from active movement contents and analyzing the flow of the video by monitoring the frames. Some notable existing methods are used to find rare event detection from the videos. A clear study of the existing methods is performed and presented here to discuss their advantages and limitations. The existing methods taken here are a unified framework for event summarization and rare event detection from Multiple Views (ESRED-MV) [14], Recognition of composite human activities through CFG representation (RCHA-CFG), Towards Abnormal Trajectory and Event Detection in Video Surveillance (TAT-EDVS) [8] and abnormal event detection in video using Spatiotemporal Auto encoder (AED-VSA)

2.1 A unified framework for event summarization and rare event detection from multiple views

ESRED-MV [14] handles event summarization and rare event detection by converting the data as a graph editing problem in a single framework. The input video is represented as a graph in which each node represents different events. The events are detected by the spatial and temporal segmentation of the input video. The functional blocks of ESRED-MV are Graph-Structure Learning, Graph-Structure Editing, and Graph-Structure Matching. Spatial Decomposition is the first process in ESRED-MV; the events are preliminarily identified in different spatial locations. The input video is segmented as 10x10 pixel blocks, and then they are clustered based on the similarities of spatiotemporal activity patterns. The temporal decomposition is performed using the optical flow calculation. In this method, the optical flow calculation is performed by the Lucas-Kanade Method. After the temporal decomposition phase, learning edges are constructed and the energy minimization process is performed using Markov Chain Monte Carlo (MCMC) method. Then the even summarization process takes place which is used to detect the rare event detection achieved by matching the video-structure graph, sub-graph matching, and relation matching.

2.2 Recognition of composite human activities through context-free grammar representation

RCHA-CFG [19] operates based on Body-part layer and poses layer properties. Pixel-level, blob level, and object-level processing are done to extract information from raw data. A hierarchical mechanism proposed by Park and Aggarwal [20] is used to construct quantitative image features from the input frame. A gesture layer is constructed from body-part layer and pose layer. The predicates of RCHA-CFG are divided into three categories which include Temporal Predicates, Spatial Predicates, and Logical Predicates. These predicates are used to estimate the atomic and composite actions from the input frame. Since human activities have more probability to be recursive, the CFG representation is used here as a wise choice is used. The delegacies of these actions are constructed through the CFG which contains all sub-events and their correlations. Based on the time intervals and predicates, the composite actions are identified from the gesture layer.

2.3 Towards abnormal trajectory and event detection in video surveillance

TAT-EDVS [8] introduces fuzzing object trajectory analysis and pixel-based features for abnormal behavior detection. The pixel-based features are selected by the authors of TAT-EDVS stating that it is state-of-the-art technology in action rejection [16]. TAT-EDVS used an object tracking algorithm [17] and a group tracking algorithm [18] to extract complete movements of all objects and groups from a frame of the video. After this phase, TAT-EDVS uses Grid-based Analysis which consists of -tasks like Trajectory Snapping, Zone Discovery, and Trajectory based Anomaly Detection. Then abnormal trajectories are classified by the Trajectory Filtering process of TAT-EDVS. It provides a unified framework for comparison and also offers good results sub in finding rare events from videos.

2.4 Abnormal event detection in video using spatiotemporal auto encoder

AED-VSA [9] is acknowledged that the work is developed based on the inspiration of learning temporal regularity in video sequences [10]. It is a complete model that has a spatial feature extractor and temporal encoder/decoder to learn the temporal patterns of a set of frames. The main functional blocks of AED-VSA are Preprocessing, Feature Learning, Regularity score, and Anomaly Detection. The Feature Learning block performs essential tasks such as Auto-encoder, Spatial Convolution, Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), and Convolution LSTM. The Anomaly Detection Block includes processes threshold and event count. The application of Spatial Feature Extractor and Temporal Sequencer easily handles the problem of Anomaly Detection from a set of video frames. This work is analyzed here due to its efficiency in finding rare events from a set of video frames. The summary of the experiments carried out in existing methods and their merits/limitations are presented in the following Table 2.

Table 1
Notations used throughout the paper

Math symbol Description

G = {N, T, P, S} Definition of Context free grammar

∂ Difference Matrix

α, β Two successive pixels

e _ij edge represent bit of (i,j)th pixel

f1, f2 … fn Frames

e_tot = m × n Total number of edges in video sequence

∀_i To identify the edges by horizontal and vertical scanning

Γ (0, 0) Block representation

$\vec{e} x$ Edge vector representation

e ^(μ,v) tsb Beginning scope time of μth edge of vth frame

τ _F First frame of the edge e_(μ,v)

ζ Number of frames per second

τ _L Last frame of the edge e_(μ,v)

δ _(μ,v) Scope duration of edge e_(μ,v)

N Finite set of non-terminals

T Finite set of terminals

P Set of rules

S Start symbol

P : N→ (N ∪ T) * Production rule

G _d Define the direction

Ψ _ω String of the input video

Ψ_ω1, Ψ_ω2 … Ψ_ωn Substrings derived from Ψ_ω

Ψ _n Set of normal events string

Ψ _r Set of rare events string

Math symbol	Description
G = {N, T, P, S}	Definition of Context free grammar
∂	Difference Matrix
α, β	Two successive pixels
e _ij	edge represent bit of (i,j)th pixel
f1, f2 … fn	Frames
e_tot = m × n	Total number of edges in video sequence
∀_i	To identify the edges by horizontal and vertical scanning
Γ (0, 0)	Block representation
$\vec{e} x$	Edge vector representation
e ^(μ,v) tsb	Beginning scope time of μth edge of vth frame
τ _F	First frame of the edge e_(μ,v)
ζ	Number of frames per second
τ _L	Last frame of the edge e_(μ,v)
δ _(μ,v)	Scope duration of edge e_(μ,v)
N	Finite set of non-terminals
T	Finite set of terminals
P	Set of rules
S	Start symbol
P : N→ (N ∪ T) *	Production rule
G _d	Define the direction
Ψ _ω	String of the input video
Ψ_ω1, Ψ_ω2 … Ψ_ωn	Substrings derived from Ψ_ω
Ψ _n	Set of normal events string
Ψ _r	Set of rare events string

Table 2

Summary of existing methods-merits and limitations

Ref.	Techniques	Dataset	Validation	Merits	Limitations
[14]	ESRED: Spatio-Temporal Activity Pattern, MCMC	Subway Pattern Sequence,PETS 2006 &2007 sequences	Accuracy, Energy	Better Accuracy	High processing time
[19]	RCHA-CFG: Pixel-level, blob level and object level processing, hierarchical mechanism	Dedicated Data collected using Sony Handy Cam	Accuracy	CFG Representation	Handles Low Resolution only
[8]	TAT-EDVS: Object &Group Tracking, Spatiotemporal Auto-encoder	Vanaheim Dataset, Subway Dataset, Mind’s Eye Dataset	Accuracy	Better Accuracy	Not implemented for real-time process
[9]	AED-VSA: special feature extractor and temporal encoder / decoder, RNN, STIM	Avenue, Subway Entrance and Exit Datasets	Accuracy, Processing Time	Processing Speed	High Energy consumption

3 Proposed method

In this work, Artificial Neural Network-based Context-Free Grammar for Rare Event Detection (ACFG-RED) is designed with the functional modules such as Background Subtraction [5], Edge detection, Raster edge movement to Vector conversion, Edge movement vector representation in CFG and Knuth–Morris–Pratt pattern searching based Artificial Neural Network (KMP-ANN) [4]. ACFG-RED framework is designed to analyze and also find rare events from recorded video files as well as from real-time video streams.

3.1 Background subtraction

It is one of the important tasks in action recognition and video mining processes. Handling entire data from the video for the classification and recognition process is a huge job that consumes a lot of computational resources such as processing time and energy. The background subtraction process is a key task to separate constant or rarely changing data in a video. To be specific, a lot of video compression algorithms are designed based on the background subtraction technique to avoid storing and processing redundant data.

The proposed background subtraction process takes place in the spatial domain itself. This saves the conversion time between spatial and frequency domains. The input frames are monitored for considerable changes and the frames are treated as key frames for background subtraction. This process takes some time for initialization but after that, this procedure operates on the fly in real-time. Since ACFG-RED is intended to handle full HD videos, the frame resolution is fixed to 1280 x 1080 pixels and 24-bit color depth. One pixel refers to the compound of 8-bits for each Red, Green, and Blue shades. Therefore, a single full video frame contains 1382400 pixels and a frame is considered as a key frame when 40% (i.e. 552960 pixels) of the pixel values are changed from its predecessor frame. Then the common part between the successive key frames is recorded as the background and the portions are maintained as fixed instead of having recurring data. Whenever there is a huge change in the background, the change is registered as the new background, and the existing one is discarded. This process enables to handle different video daylight and night scenario backgrounds.

The background subtraction process performed in the frames and result image is given in Fig 1. It is also considered as a rare event in ACFG- RED. The background is illustrated dual-color representations of extreme white RGB (255,255,255) and Dark Black RGB (0,0,0) for ease of operation. The background data update takes place for every key frame, thus it is dynamic.

Fig. 1

Initial Frame, Subsequent Key Frame, Key Frames Overlapped, and Background Image.

3.2 Edge detection

The edge detection process is performed only in the foreground area of a picture frame. Detecting edges for video background is eliminated with the help of the background subtraction process. The purpose of edge detection is to collect the movement data from the input video. For edge detection, ACFG-RED uses horizontal and vertical fast scanning procedure. The values of two subsequent pixels are used to calculate the difference and to identify the edges. The pixel values are noted as an RGB matrix and differences in pixel values are calculated by Equation 1. $\partial {r, g, b} = | α {r, g, b} - β {r, g, b} |$ (1)

Where α, β are two successive pixels and ∂ is the difference matrix. The edge is referred if the value of any of the element r, g, b in ∂ is greater than the threshold value which is assigned as 7 for toll traffic videos. While scanning horizontally all vertical edges are marked and horizontal edges are marked during the vertical scanning. The Horizontal Scanning procedure starts from the left-top portion of the frame and scan through the image until the pointer reaches the right bottom. The edges are identified using the following equation in horizontal scanning. $\forall i = l \to r \forall j = t \to b : e_{ij} = 1 or 0 by \partial_{ij}$ (2)

Where e_ij edge represent a bit of (i, j) ^th pixel, l, t, r, and b are left, top, right, and bottom respectively.

The equation to find the edges in vertical scanning is given as Equation (3) $\forall i = t \to b \forall j = l \to r : e_{ij} = 1 or 0 by \partial_{ij}$ (3)

The detected edges for the video frame are highlighted and represented in Fig. 2. The edge detection process is performed in real-time during the video playing or streaming. Therefore, the continuous track record is maintained in memory to detect the movements in the video.

Fig. 2

Detected edges of video frame.

3.3 Raster edge movement to vector conversion

The movements are captured by the edges and the direction of movements is stored as the vectors rather than raster special Image. The vector conversion process reduces the memory requirement and improves the processing speed. The dimensions of the edges are used to determine the action in the video. If f₁, f₂ ⋯ f_n are the frames in which e₁, e₂ ⋯ e_m are the edges that exist in the frames, then the total number of edges in a video sequence is e_tot = m × n and they are represented as e_(1,1), e_(1,2) … e_(2,1), e_(2,2) ⋯ e_(m,n). The sizes of edges are represented individuallyas s_(1,1), s_(1,2), ... s_(2,1), s_(2,2), ... s_(m,n).Then the condition ∀i = b → c : ifs_(x,i) < s_(x,(i+1)) where 1 ⩽ b < c and b < c ⩽ n show that the object which is detected by the edge e_x is approaching the camera. Here s_x refers to the size of the edge e_x. In the toll traffic dataset is used in ACFG-RED, this condition is used to detect the incoming vehicles.

Similarly, the condition ∀i = b → c : ifs_(x,i) > s_(x,(i+1)) refers to outgoing vehicles. There will be no specific change in the size of the edges of vehicles while they are not moving. There is a criterion here in the conversion that the process should be rapid to achieve on the fly operation of ACFG-RED. Therefore, the frame is split into smaller grids as given in Fig 3. This splitting process helps to improve computational speed because the pixel-level edge movements are grouped into block level. It is admitted that the frame splitting process of the proposed procedure compromises some accuracy to improve the processing speed. It is also learned that the difference in accuracy is very negligible for detecting rare events from a video. The 1920x1080 frame is split into 144 blocks in such a way that both the height and width of the frame are divided with 12 grid lines. Each block will be of resolution 160x90 pixels. The blocks are represented as Γ_(0,0) → Γ_(11,11). The partitions of blocks and alignment are given in Fig 3. To explain the construction of the movement vector, the edges of the frame f₅₆₀ i.e. a video frame of t + 28 second is given in Fig 4, where t is the initial time of input video. The captured edge movements from f₄₈₀, f₅₀₀, f₅₂₀, f₅₄₀ and f₅₆₀ are given in Fig 5.

Fig. 3

Frame blocks alignment.

Fig. 4

Edges e_(34,560) and e_(40,560).

Fig. 5

Movements of Edges e_(34,560) & e_(40,560).

As per the observations the movement of the edge e₃₄ from block Γ_(4,10) to Γ_(6,5) is the storable entity as well as the movement of the edge e₄₀ from block Γ_(7,11) to Γ_(8,9) is also recordable. The edge vector of the edge e₃₄ is represented as ${\vec{e}}_{34} = {(Γ_{(4, 10)}, Γ_{(5, 9)} \dots Γ_{(6, 5)}), (f_{480}, f_{500} \dots f_{560})}$ and edge vector of edge e₄₀ is represented as ${\vec{e}}_{40} = {(Γ_{(7, 11)}, Γ_{(8, 10)} \dots Γ_{(8, 9)}), (f_{480}, f_{500} \dots f_{560})}$ .

Therefore, the positions of the edges in different video frames based on time can be accessed through the edge vectors. The generalized equation of an edge vector is as ${\vec{e}}_{x} = {(Γ_{(h_{1}, v_{1})}, Γ_{(h_{2}, v_{2})}, \dots Γ_{(h_{η}, v_{η})}), (f_{ɛ_{1}}, f_{ɛ_{2}} \dots f_{ɛ_{3}})}$ (4) where h₁, h₂ … h_η are horizontal frame-block partitions, v₁, v₂ ⋯ v_η are vertical frame partitions and ɛ₁, ɛ₂ … ɛ₃ are the frame numbers of a video. The scope of an edge vector begins from the first frame in which the edge is detected for the first time and ends with the frame in which the edge has vanished completely. This process enables to reuse of the edge number after a desired period thus the data type overflow is avoided in ACFG-RED. It is easy to get the scope of an edge from the edge vector representation. For example, the scope of edge e₄₀ can be obtained from its vector ${\vec{e}}_{40}$ as f₄₈₀ to f₅₆₀. Equations are given below to calculate the scope beginning time, ending time, and scope duration. $e_{(μ, ν)}^{tsb} = \frac{τ_{F}}{ς}$ (5)

Where $e_{(μ, ν)}^{tsb}$ is the beginning scope time of μ^th edge of ν^th frame, τ_F is the first frame of the edge e_(μ,ν) and ς is the number of frames per second of the input video. $e_{(μ, ν)}^{tse} = \frac{τ_{L}}{ς}$ (6) where $e_{(μ, ν)}^{tse}$ is the ending scope time of μ^th edge of ν^th frame and τ_L is the last frame of the edge e_(μ,ν) $δ_{(μ, ν)} = e_{(μ, ν)}^{tse} - e_{(μ, ν)}^{tsb}$ (7) where δ_(μ,ν) is the scope duration of edge e_(μ,ν)

Memory allocation for edge vector representation is based on the typical and the maximum number of frame tracks taken for and edge. In this case, for toll traffic data, the typical vector element size is 5 + 5 which is 5 numbers of blocks and 5 numbers of corresponding frames have been chosen. The maximum permitted edge movement vector size in ACFG-RED is 20 + 20. The size of a typical edge vector ${\vec{e}}_{x} = {(Γ_{(h_{1}, v_{1})}, Γ_{(h_{2}, v_{2})}, \dots Γ_{(h_{η}, v_{η})}), (f_{ɛ_{1}}, f_{ɛ_{2}} \dots f_{ɛ_{3}})}$ is allocated as follows.

Γ_(h₁,v₁) consumes 1 bytes in memory, bx1 is the horizontal partition of the block Γ and by1 is the vertical partition. Since the maximum number of partitions is set to 12 for a 1920x1080 resolution image in ACFG-RED, 4-bits are used to represent each horizontal and vertical partition. Therefore, one block represents consumes 1 byte in memory and a typical edge vector representation takes 5 bytes of memory for block storage. To store a single frame number, 8-bytes are used. To represent a typical 5 frame count edge vector, it requires 40 bytes of memory. A 4-byte memory space is used to store the edge number in the vector. Therefore, a typical edge vector will be of 5 + 40 + 4 = 49 bytes of memory. Since the maximum permitted element count is limited to 20 for an edge vector representation in ACFG, it will take 1 × 20 + 8 ×20 + 4 =184 bytes to store a big edge vector.

The size of an active edge movement region in a 1920×1080 pixel video frame will be around a 1540x480 pixel – highlighted in Fig 6. To hold such a raster image in memory, it will take 1540 × 480 × 3 =2217600 bytes in memory; here 1540 is the width, 480 is the height and 3 is to store the pixel color information in RGB. Thus, the vector representation of edge movements reduces a huge size of memory requirement in ACFG-RED.

Fig. 6

Active edge movement region.

3.4 Edge movement vector representation in context-free grammar

This part is used to find interrelated edges in a video to identify the objects. As per the grammar theory, a set of strings over a set of finite symbols is a language. A context-free grammar is defined as G ={ N, T, P, S }, where N is the finite set of non-terminal characters, T is the finite set of terminals where N∩ T = Ø, P is the set of rules such as P : N → (N ∪ T) ^* that is the production rule P which can have any right or left context and S is the start symbol. In ACFG-RED, the edge movement vectors are treated as non-terminal characters. The movement relationships between the edges are correlated as the production rules. The starting symbol is the first occurrence edge movement vector. As per the ACFG-RED definitions ${{\vec{e}}_{1}, {\vec{e}}_{2} \dots {\vec{e}}_{m}} \in N, P : Γ_{(h_{x}, v_{x})} \to Γ_{(h_{y}, v_{y})}$ (8) where Γ_{(h_x,v_x)} is the current block of the edge e_x and Γ_{(h_y,v_y)} is the succeeding block of the same edge in the next timestamp frame.

The video frame center point is calculated as $P_{c} = (\frac{h}{2}, \frac{v}{2})$ that is (960, 540) for an FHD (1920x1080) frame. This center point P_c is used to represent the movement direction of the edges and objects in the video. The movement directions in ACFG-RED are represented in a similar way used in the graphs. The point P_c refers to the center point of the graph (0, 0). The CFG used in ACFG-RED to define the directions is $G_{d} = ({A}, {+, -}, P, A), P : A \to A + | A \to A - | Ø$ (9)

Using this grammar, following direction strings are derived $A \to A + \to A + + \to + + = > (+, +)$ $A \to A - \to A + - \to + - = > (+, -)$ $A \to A + \to A - + \to - + = > (-, +)$ $A \to A - \to A - - \to - - = > (-, -)$

The movements categories set is defined as d_m ={ d₀, d₁, d₂, d₃ } where d₀ = (+ , +), d₁ = (+ , -), d₂ = (- , +) and d₃ = (- , -) as in Fig. 7.

Fig. 7

Movement Categories Set.

The Chomsky Normal form of the developed CFG is 0 ↦ A|B|C, A → AA, B → A + , C → A-. The movement calculation is performed using the following equation.

$\forall i = 1 \to n, Γ_{(h_{i + 1}, v_{i + 1})} = Γ_{(h_{i}, v_{i})} * d_{m} (x_{q}, y_{q})$ (10) where x_q and y_q are the x-axis and y-axis quotients respectively.

Since the video has no direct z-axis representations, the x and y coordinates are used to establish the z-axis visionary by the Euclidean range calculation of Equation 10. The edges with similar block shifts based on the time durations are considered as the member edges of an object in the input video frame. For example, as per Fig 5, the movements of edges e₃₄ and e₄₀ are following the same movements d₀ → d₃, it is easy to understand that the edges belong to the same object. The edge movement vectors are converted into strings using Equation 8 and 10 to apply Knuth-Morris-Pratt pattern matching.

3.5 Knuth–Morris–Pratt pattern searching based Artificial Neural Network

This part is used to find out the rare events in a video which is converted as edge movement vectors strings. In ACFG-RED, the KMP algorithm is added with ANN to update the knowledge base for rare event detection. The standard KMP [4] algorithm performs a sequential search to find the string matching.

ACFG-RED string search algorithm:

Let Ψ_ω is the string of the input video constructed using equation 8 and 10

Let Ψ_{ω
₁}, Ψ_{ω
₂} ⋯ Ψ_{ω
_n} substrings derived from Ψ_ω based on the edge scopes by equation 5

Let Ψ_n ={ Ψn₁, Ψn₂ ⋯ Ψn_m } is the set of normal event strings

Let Ψ_r ={ Ψr₁, Ψr₂ ⋯ Ψr_n } is the set of rare event strings

For i = 0 to n

Set REF = 0 (REF: Rare Event Flag)

Search Ψ_{ω
_i} for any member string of Ψ_r, set REF = 1 if so

If REF = 1, then report Ψ_{ω
_i} as a rare event string

Else

For j = 0 to m

set NEF = 0 (NEF: Normal Event Flag)

Search Ψ_{ω
_i} for any member string of Ψ_n, set NEF = 1 if so

If NEF = 1, then report Ψ_{ω
_i} as normal event string

Else call ANN to update Ψ_n and/or Ψ_r

End For [j]

End For [i]

End

The ANN is designed with the input layer in which the nodes receive the inputs such as edge movement direction, the time duration of edge movements. There are two hidden layers are assigned to operate with the input layers. The first hidden layer is indented to give weightage to the edge movement direction and the second hidden layer is indented to process the edge movement direction with the time duration. The output layer has two nodes for normal and rare event classifications. 18 nodesare placed in the input layer of ANN for edge movement directions { (+ , +) → (+ , +) }, { (+ , +) → (+ , -) }, { (+ , +) → (- , +) }, { (+ , +) → (- , -) }... .{ (- , -)→ (- , -) }, {hordedirections} and {movement duration}.The ANN architecture, ANN training phase, and overall flow diagram have been illustrated in Figs. 8, 9 & 10 respectively.

Fig. 8

KMP-ANN architecture.

Fig. 9

ANN training flow diagram.

Fig. 10

Overall flow diagram.

Table 3

Accuracy

		Accuracy in (%)
Timestamp	ESRED-MV	RCHA-CFG	TAT-EDVS	AED-VSA	ACFG-RED
T1	80.69	79.23	84.36	80.15	89.64
T2	81.11	79.32	84.93	79.08	88.38
T3	80.26	78.47	83.42	79.72	89.19
T4	81.68	78.67	83.82	80.07	89.14
T5	81.38	79.33	84.43	79.49	89.53
T6	80.77	79.52	84.05	79.9	88.59
T7	81.15	79.47	83.29	80.71	89.28
T8	81.1	78.03	84.24	79.43	89.46
T9	81.14	78.72	83.24	79.93	88.78
T10	80.78	78.71	83.93	79.96	88.99
Average	81.006	78.947	83.971	79.844	89.098

Table 4

Precision

Timestamp	ESRED-MV	RCHA-CFG	TAT-EDVS	AED-VSA	ACFG-RED
T1	80.62	79	84.31	80.78	89.94
T2	81.01	79.38	84.99	79.14	88.61
T3	80.27	78.2	83.58	79.61	88.96
T4	81.83	79.13	84.62	80.03	88.43
T5	81.75	79.9	84.76	79.01	89.94
T6	81.23	79.14	84.15	80.16	89.17
T7	80.28	79.5	83.44	80.75	89.38
T8	80.83	78.03	83.95	79.57	89.87
T9	80.76	79.21	83.33	79.89	89.44
T10	80.4	78.09	84.67	80.46	89.25
Avg.	80.898	78.958	84.18	79.94	89.299

Table 5

Recall

		Recall (%)
Timestamp	ESRED-MV	RCHA-CFG	TAT-EDVS	AED-VSA	ACFG-RED
T1	80.72	79.37	84.39	79.78	89.4
T2	81.16	79.28	84.88	79.04	88.2
T3	80.26	78.63	83.31	79.79	89.37
T4	81.59	78.4	83.3	80.09	89.71
T5	81.15	79	84.21	79.77	89.22
T6	80.49	79.75	83.98	79.75	88.15
T7	81.71	79.45	83.2	80.68	89.19
T8	81.26	78.02	84.44	79.35	89.13
T9	81.37	78.43	83.19	79.95	88.27
T10	81.01	79.06	83.43	79.66	88.78
Avg.	81.072	78.939	83.833	79.786	88.942

Table 6

Average Processing Time

		Average Processing Time (mS)
Timestamp	ESRED-MV	RCHA-CFG	TAT-EDVS	AED-VSA	ACFG-RED
T1	1201	1405	1441	925	838
T2	1238	1415	1445	949	835
T3	1218	1405	1464	931	835
T4	1235	1428	1443	915	858
T5	1231	1423	1468	931	825
T6	1207	1419	1433	926	848
T7	1202	1413	1432	923	844
T8	1225	1409	1455	919	854
T9	1228	1438	1448	946	826
T10	1210	1401	1452	923	824
Avg.	1219.5	1415.6	1448.1	928.8	838.7

4 Experimental setup

ACFG-RED is implemented using Visual Studio IDE [11]. Since the ANN training phase requires non-stop execution, it is performed in a leased HP ProLiant DL160 Gen9 Intel Xeon E5-2620v4 12 core cloud server. Some CCTV footage videos are used in the ANN training phase. The trained knowledgebase is used to test the ACFG-RED procedure with a video recorded in a Toll collection booth. The video is of 12 Minutes 29 seconds length with 1920x1080 FHD resolutions at 20 frames per second. The data rate of this video is 11611kbps with a total bit rate of 11785kbps. The storage size of the video is 1105413017 Bytes (1.02 GB). A dedicated User interface is designed to test the functionalities of ACFG-RED. The user interface is given in Fig. 11.

Fig. 11

ACFG_RED user interface.

5 Results and analysis

Accuracy, Precision, and Recall are the standard parameters to measure the quality of a data mining and classification procedure. Average Processing Time and Average Processing Power are measured along with the standard parameters for existing ESRED-MV, RCHA-CFG, TAT-EDVS, AED-VSA, and for proposed ACFG-RED. The performances of these procedures are measured for the entire video with 10 different timestamps and the results are described in Tables 3 to 7. As per the experiment results, the method ACFG-RED scored the highest accuracy average of 89.299 %. Proposed ACFG-RED consumes relatively lesser power than other methods for rare event detection process that shows its efficiency.

Table 7
Average Processing Power

Average Processing Power (mW)

Timestamp ESRED-MV RCHA-CFG TAT-EDVS AED-VSA ACFG-RED

T1 833 702 691 980 638

T2 834 719 689 985 638

T3 835 703 696 983 640

T4 822 716 681 994 637

T5 830 703 687 998 648

T6 827 712 680 984 638

T7 828 708 692 999 647

T8 826 705 694 984 632

T9 824 711 693 990 638

T10 832 710 691 988 635

Avg. 829.1 708.9 689.4 988.5 639.1

5.1 Accuracy

Accuracy is the prime metric of any classification procedure. Accuracy is calculated using the formula $\frac{(TP + TN)}{(TP + TN + FP + FN)}$ , where TP is True Positive, TN is True Negative, FP is False Positive and FN is False Negative. Accuracy is directly proportional to the quality of a classification algorithm. The accuracy values for ACFG-RED along with existing procedures have been measured and given in Table 3.

Based on the observations, the Accuracy averages of ESRED-MV, RCHA-CFG, TAT-EDVS, AED-VSA, and ACFG-RED are 81.006, 78.947, 83.971, 79.844, and 89.098 respectively. The proposed ACFG-RED method has secured the highest average Accuracy in finding rare events from the input video also described in Fig 12.

Fig. 12

Accuracy Graph.

5.2 Precision

Precision is one of the standard metrics of classification and mining procedures. Precision is calculated as $\frac{TP}{TP + FP}$ . The precision values are measured for existing methods and proposed methods are tabulated in Table 4 and the comparison graph is shown in Fig. 13.

Fig. 13

Precision Graph.

5.3 Recall

Recall otherwise known as true positive rate refers to the successful retrieval of relevant events among the total number of events. A recall is calculated by the formula $\frac{TP}{TP + FN}$ . It is one of the standard metrics used to represent the quality of a mining procedure. The measured recall values are given in Table 5 and the comparison graph is given in Fig 14. The ACFG-RED procedure gets the Recall average of 88.942% which is the highest value of the category.

Fig. 14

Recall Graph.

5.4 Average processing time

The average processing time of ACFG-RED for processing 24 frames is 838.7mS which is less than a second. The measured Average Processing Time values are given in Table 6 and the comparison graph is given in Fig 15.

Fig. 15

Average Processing Time Graph.

5.5 Average processing power

The processing power consumption refers to the electrical power consumed by a process that is measured in Watts. The processing powers consumed by existing and proposed methods are given in Table 7. The comparison graph is given in Fig 16.

Fig. 16

Average processing power graph.

6 Conclusion

This work presents a framework for detecting rare events from video using CFG and ANN. The developed model ACFG-RED has been implemented and tested using real-time video recorded at Toll Collection Centre. The performance of the proposed model was computed and compared with other similar existing models. The proposed framework can handle both stored video files as well as real-time streaming videos for rare event detection with better efficiency. The lesser power consumption and higher processing speed make ACFG-RED more suitable for detecting rare events in real-time CCTV surveillance systems. Further, the proposed rare event detection module is also used in unmanned CCTV monitoring units.

Footnotes

Acknowledgments

The authors are grateful to the Department of Science & Technology, New Delhi, India (SR/FST/ETI-371/2014) for their financial support. Also express their sincere thanks to SASTRA Deemed University, Thanjavur for extending infrastructural support to carry out the work.

References

and Chang

, Video anomaly detection and localization via multivariate gaussian fully convolution adversarial autoencoder, Neurocomputing 369 (2019), 92–10.

Ramachandran

and Sangaiah

A.K.

, Unsupervised deep learning system for local anomaly event detection in crowded scenes, Multimedia Tools and Applications, 2019.

Janjua

Z.H.

, Vecchio

, Antoini

and Antonelli

, IRESE: An intelligent rare-event detection system using unsupervised learning on the IoT edge, Engineering Applications Of Artificial Intelligence 84 (2019), 41–50.

Kalnoor

and Agarkhed

, Detection of Intruder using KMP Pattern Matching Technique in Wireless Sensor Networks, Computer Science 125 (2018), 187–193.

Chen

, Wei

, Yang

, Li

, Wang

and Yang

M.-H.

, Spatiotemporal GMM for Background Subtraction with Superpixel Hierarchy, IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Access 40 (2018), 518–1525.

Kumar

and Shrimankar

D.D.

, F-DES: Fast and Deep Event Summarization, IEEE Access 20 (2018), 323–334.

Joosten

S.J.C.

, Parsing and Printing of and with Triples, Relational and Algebraic Methods in Computer Science (2017), 159–176.

Cosar

, Donatiello

, Bogorny

, Garate

, Alvares

and Bremond

, Towards Abnormal Trajectory and Event Detection in Video Surveillance, in IEEE Transactions on Circuits and Systems for Video Technology IEEE Access 27 (2017), 683–695.

Chong

Y.S.

and Tay

Y.H.

, Abnormal Event Detection in Videos Using Spatiotemporal Autoencoder, in International Symposium on Neural Networks – Advances in Neural Networks – ISNN 2017, (2017), 189–196.

10.

Hasan

, Choi

, Neumann

, Roy-Chowdhury

A.K.

and Davis

L.S.

, Learning temporal regularity in video sequences, IEEE Access (2016), 733–742.

11.

Ritchie

, Introduction to Visual Studio in, Practical Microsoft Visual Studio 2015 (2016), 1–25.

12.

Sukumaran

, Samuelsson

and Forslow

, Method for sharing service identity among multiple client devices in a real-time communication network, Inc, https://patents.google.com/patent/US9246924B2/en, 2016.

13.

Poulding

, Alexander

, Clark

J.A.

and Hadleyb

M.J.

, The optimisation of stochastic grammars to enable cost-effective probabilistic structural testing, Journal of Systems and Software 103 (2015), 296–310.

14.

Kwon

and Lee

K.M.

, A Unified Framework for Event Summarization and Rare Event Detection from Multiple Views, IEEE Access 37 (2015), 1737–1750.

15.

Kwon

and Lee

K.M.

, A unified framework for event summarization and rare event detection, IEEE Access (2012), 1266–1273.

16.

Wang

, Kläser

, Schmid

and Liu

C.-L.

, Action recognition by dense trajectories, in CVPR 2011, IEEE Access (2011), 3169–3176.

17.

Chau

D.P.

, Brémond

, Thonnat

and Corvée

, Robust mobile object tracking based on multiple feature similarity and trajectory filtering, VISAPP 2011 – Proceedings of the Sixth International Conference on Computer Vision Theory and Applications,Vilamoura, Algarve, Portugal, (2011), 569–574.

18.

Zaidenberg

, Boulay

, Garate

, Chau

D.P.

, Corvee

and Bremond

, Group interaction and group tracking for video surveillance in underground railway stations, International Workshop on Behaviour Analysis and Video Understanding (ICVS), Sophia Antipolis, France, (2011).

19.

Ryoo

M.S.

and Aggarwal

J.K.

, Recognition of Composite Human Activities through Context-Free Grammar based Representation, IEEE Access (2006), 1–8.

20.

Park

and Aggarwal

J.K.

, Semantic-level Understanding of Human Actions and Interactions using Event Hierarchy, IEEE Access (2004), 66–78.

		Average Processing Power (mW)
Timestamp	ESRED-MV	RCHA-CFG	TAT-EDVS	AED-VSA	ACFG-RED
T1	833	702	691	980	638
T2	834	719	689	985	638
T3	835	703	696	983	640
T4	822	716	681	994	637
T5	830	703	687	998	648
T6	827	712	680	984	638
T7	828	708	692	999	647
T8	826	705	694	984	632
T9	824	711	693	990	638
T10	832	710	691	988	635
Avg.	829.1	708.9	689.4	988.5	639.1